Epics PV Archiver

Epicsrchiver Help: WARNING OUT-OF-DATE Needs fixing!
The current version of Epics PV Archiver is 2.1 at
Epics PV Archiver Setup and Management

This page gives a few details for setting up and managing the Epics PV Archiver system. Installation instructions are in the source kit.

As described in the Overview, the Epics PV Archiver has two main data stores for PV data, each controlled by a separate running process on the server:

Both processes will write log files to /var/log/pvarch.

The command-line pvarch process

The main administrative interface to the PV Archiver is the command line program pvarch. Typical usages of this program are:
Command Meaning
Status / General Information
pvarch -h shows a status message.
pvarch status shows cache and archiving status, some recent statistics.
pvarch check print # of archived PVs in past 10 minutes. Should be >1!
pvarch list prints a list of recent data archives
pvarch save create backup files of recent data archives
pvarch clean clean out old files in the web data directory
The Archiving Process
pvarch start start the archiving process
pvarch stop stop the archiving process
pvarch restart restart the archiving process
pvarch next restart with 'next run' of data archives
The Caching Process
pvarch cache start start cache process (if it's not already running)
pvarch cache stop stop cache process
pvarch cache restart restart cache process
pvarch cache status show # of PVs cached in past 60 seconds
pvarch cache activity show list of PVs cached in past 60 seconds
Adding and removing PVs
pvarch add_pv add a PV to the cache and archive
pvarch add_pvfile read a file of PVs to add to the Archiver
pvarch drop_pv remove a PV from cache and archive

Deadtime and Deadbands: setting how often a PV is recorded

While the caching process saves current values for all PVs as fast as it can, it is not necessary to store all changes to PVs. Two parameters set how often values are archived: the deadtime, and the deadband. Both of these can be set for each PV.

The deadtime for a PV sets how long to wait after archiving a PVs value before archiving another value for this PV. Let's take as an example a deadtime of 10 seconds. If the PV hasn't changed in longer than 10 seconds, then a change to the value will be archived as quickly as possible. But if the value changes again within 10 seconds, the change will not be recorded immediately. Instead, the change will be seen and the archiver will wait until 10 seconds has elapsed since the last time the value was recorded. This way, if multiple changes happen during that 10 seconds, the only value recorded will be the one held after 10 seconds has elapsed -- all intermediate changes will be forgotten.

One thing to note, though, is that the time of the last change will be recorded properly. Thus, if a value changes, changes again 1 second later, and then stays fixed, the archiver will record the changes as taking place 1 second apart.

The deadband for a PV sets how big a fractional change must be in order to be archived.

When PVs are added to the database, typical deadtimes are one or a few seconds, depending on whether the PV has a type 'double' -- which can easily change trivial amounts -- or a type of 'int','enum', or 'string', which change less often. For this setup, the default deadtime for PVs of type 'double' or 'float' is seconds, while the default deadtime for all other PVs is 1 seconds. Typical deadbands are set small enough to be not actually effective.

Rotating databases, cron jobs

The archiving process writes data to a single database (more details on database layout is below). It is expected and encouraged to have several of these archive databases, each one representing a 'run' for a fixed time duration, typically around a weeks to a month.

That is, you will want to periodically start archiving into a new database. Using pvarch next will tell the Archiver to make the "next" database in the series of archives and to start collecting into that new database. All PVs archived and all their settings in the old database will be moved into the new one. Using pvarch list will show a list of the most recent 'runs'.

Looking up a PV's value for plotting or data retrieval will seamlessly span multiple databases, so you don't have to worry about how often you start a new run. Because of this, putting pvarch next in a cron table to run once a week or once a month, or at some other frequency, is recommended. I would not recommend using a new database more often than once a day -- once a month or every few weeks seems about right.

Also, because both pvarch start and pvarch cache start will do nothing if there is a successfully running archiving and caching process, you can also put these two commands in cron tables, running more often to ensure that the caching and archiving are running.

Another potential use of a cron job can be to clean out the data files created for the data plotting and viewing. These are temporary files stored in {(config.data_dir}}. To clean old files in this directory, you can use the command


pvarch clean

Finally, to backup the databases with simple mysql dump files, use pvarch save. This will save the two principle databases: the master database, , and the currently active archive database. You can specify additional databases to save as command-line arguments:


pvarch save 

Example crontab files for these tasks are included in the source distribution.

Database layout

This Epics PV Archiver has two main databases that it uses. The main or master database, named , holds the cache of values, status information about running processes, the history of archive 'runs', settings for Instruments and Alerts, and the data for related PVs. The archive database holds the archived values for the PVs as well as data about how to archive the data.

At any one time, one archive database is 'current', meaning that it is the one be written to, so the database in use is a sequentially number database, of the form , , .... Each databases holds a set of data for a specific time period -- a 'run'. Data retrieval uses the full set of archive databases, and looks up older spanning archive databases as necessary.

The assumption is that the Python programming interface will be used to access data in the PV Archive. While the data is stored with MySQL databases, using simple SQL queries to retrieve data from the PV Archive is slightly non-trivial. Still, for completeness, a partial description of the databases is provided here:

Structure of the Master Database:

The master database, , is a fairly simply database. There are several tables, but none of them is very complicated, and the logic that joins them is fairly simple. The tables are:
TableDescriptionData Columns
cache holds the cached data for all PVs. id, pvname,type, value, cvalue, ts, active
cvalue is the 'character string value', ts is the timestamp, active holds whether the PV is actively cached.
runs holds info about archiving 'runs'. id, db,notes, start_time,stop_time
db is the name of the current archive databases.
info status information about running 'cache' and 'archive' processes id, process, status, db, datetime, ts, pid
process is either 'cache' or 'archive'.
requeststemporary storage for requested changes to the cache database id, pvname, action
action is 'add', 'drop', 'suspend', or 'ignore'. The Caching process looks here for new values to include.
alertsdata about Alerts id, pvname, name, mailto, mailmsg, compare, trippoint, timeout, active, status
pairsdata for Related PVsid, pv1, pv2, score
instrumentslist of instrumentsid, name, station, notes
station holds the ID from the stations table.
stationslist of stations id, name, notes
instrument_pvsdata to map PVs with instruments id, pvame, inst
instrument_positionsdata for saved positions for instrument id, name, inst, active, ts

Structure of the Archive Database:

The archive databases, with names like , is is slightly more complicated than the master database, in an effort to make an efficient system for archiving thousands of PVs. There is a main PV table that holds information about the PVs being archived, including their data type, archiving deadtime and deadband, and which Data Table to use for data storage. There are 128 Data Tables , , .... . Because many PVs rarely change, while others change very frequently, it was found to be inefficient for each PV to have its own data table, or to have one data table for all archived data. Instead, when adding a PV to the archive, the PV name is hashed to give a number between 1 and 128, and that determines which data table to use. Of course, multiple PVs then store to any single table, so a way to identify the PV in the data table is needed. When looking up data for a PV, which table is read from needs to be determined (the table name is stored in the PV table, so this is very fast) but then only that one table needs to be read, eliminating more than 99%% of the data in the archive.

The PV table has the following columns and meanings:

Column Description
id integer ID
name PV name
description description of PV
data_table name of corresponding data table
deadtime deadtime for archiving
deadband deadband
graph_hi default high value for plotting range
graph_lo default low value for plotting range
graph_type default type for plotting: 'normal','log','discrete'
type PV data type
active Whether PV is actively being archived.

In contrast, the data tables are much simpler, with columns of pv_id (that is the id from the PV table), time, and value

For further details, simply explore the MySQL databases. Epics PV Archiver Overview

The Epics PV Archiver saves Epics PVs into a MySQL database and provides a web interface for looking up, plotting, and managing the archiving process. The Archiver can easily accommodate thousands of PVs and save and retrieve data for many years.

The PV Archiver provides Alerts, which will send an email notification when a particular PV goes out of an acceptable range. It also provides a concept of an Instrument, which is a group of PVs for which you may want to save and look up positions.

Caching and Archiving Processes

The Epics PV Archiver uses two running processes: Normally, you do not need to think very much about these two processes except that they both need to be running. The Status Page will show the current status of the two processes, including the number of PVs updated in the cache and the number archived recently. For both of these, there should be a substantial amount of activity.

Adding PVs to the Archive

There are a few ways to add PVs to the archive, all of them very easy.

First, you can use the command line program pvarch, with


   ~> pvarch add_pv MyPV.VAL
You can add multiple PVs on a single line:

   ~> pvarch add_pv MyPV1.VAL  MyPV2.VAL
This will add both PVs, and also set them as "related PVs", which will show up in the plot view page. Second, you can put a list of PVs in a file, either one per line or multiple PV names per line (separated by a space or comma), and run

   ~> pvarch add_pvfile PVList.txt
This is probably most useful on initial setup or when adding a large set of new PVs. As with pvarch add_pv, putting multiple PVs on a single line will make the "Related".

The third way to add PVs is to put them in one of the template files for the web status pages (See the Template Page for details). The next time the corresponding web page is accessed, the system will see that the PV is not included, will initially report 'Unknown', and request it to be added to the Cache and Archive processes.

For all these methods, the PV will not actually be added unless it is a valid Epics PV that can be accessed from the server. In some cases, you may wish to record remote PVs -- you'll want to set EPICS_ADDR_LIST accordingly.

Adding Motor PVs

When adding a PV for a Motor (that is, one that has a record type of 'motor' from the Epics Motor Record), all of the following fields will automatically be added, and all pairs of PVs will be set to be related: This subset of all the fields of the motor record are thought to be enough to reconstruct "where a motor was" and/or "what happened to the alignment questions".

Related PVs

As alluded to above, there is a concept that some PVs can be "related" to others. The Archiver uses a simple scoring method for pairs of PVs, and pairs of PVs that have a non-zero score are said to be related. In addition, any time a pair of PVs are plotted together, the score is increased, so that they will 'move up' in the list of related PVs.

When PV are added to the archive, pairs of PVs will get an initial pair score of 10 under the following conditions:

The list of related PVs for any PV can also be managed through the Settings / Admin page, where you can adjust the score for any pair of PVs.

Web Pages

Controlling the layout and details of the main Web Status Pages is discussed on the Web Page Template Page. Epics PV Archiver Alerts

Alerts are used to send an email when a PV goes into an 'Alert' condition. The alert condition in the Epics Archiver is set by the user, and does not need to be related to Alarm states for Epics Process Variables. Alerts are set for a single PV -- currently they cannot depend on complex conditions of PVs (say, PV1 is greater than 5 AND PV2 is not 0), though an Epics calc record might be able to provide such logic.

To set or change an alert, you will be challenged for the administrator password. As with all administrative tasks for the Archiver, this is a security measure so that only trusted users can set up emails sent from your system. To set up Alerts, a mail server and "from address" must be configured at installation time.

Use the main Alert Page to see a list of currently defined alerts. From this page you can select a "View/Change" link to modify any of the settings for a particular alert.

Setting an Alerts

In addition to the PV name, each Alert has a label to describe the alert, and so that multiple alerts can be set on a single PV (one person may want an alert when a valve closes, while another person may want an alert when it opens!). The label will be used in the email sent when the Alert condition is met.

Each alert consists of 3 pieces: a PV's value, a 'test condition' and a 'trip point'. When the PV's value meets the condition for that trip point, the Alert is said to be in the 'Alarm' state. The possible test conditions are:

As an example, if a PV's value is 1.8, the test condition is 'greater than', and the trip point is 2.0, the Alert will be in the 'OK' state. If the PV's value changes to 2.1, the Alert will be in the 'Alarm' state.

An email notice is sent to the specified addresses when the Alert changes from 'OK' to 'Alarm'. If the PV's value changes again, but still stay in the 'Alarm' state or if it stays in the 'Alarm' state indefinitely, NO more emails will be sent. In order for another message to be sent, the Alert would have to change back to an 'OK' condition', and then again to an 'Alarm' condition. In addition, each Alert has a 'Time Out' value (in seconds). Once an email has been sent for an alert, no email will be sent for that alert for this time, even if the Alert goes from 'OK' to 'Alarm'. This is provided to avoid multiple emails for a value that may rapidly fluctuate about its trip point.

Customizing and Formatting an Alert email

By default, the content of the sent email will look something like this:

Hello,
   An alarm labeled 'XXX went non-zero'
   was detected for PV = 'XXX.VAL'
   The current value = 1.0. This is
   not equal to the trip point value of 0.0
You can customize this message on the Alerts page, putting in any text you want. Since you may want to include some of the the name and value of the PV, and the alert label, condition, and trip point, you can use the following simple template format for the mail message to automatically get these values: With the last item, you can get the value for other PVs in the mail message. For this to work, the other PV must already be added to the Cache and Archive.

Thus, you could customize the message to be:


Hello,
   An alarm labeled %%LABEL%%
   was detected because %%PV%% = %%VALUE%%.

   Other PV Values at the time of this alert were:
        Valve 1 : %%PV(Valve1.VAL)%%
        Current : %%PV(Current.VAL)%%

To get a more complete report. The mail message will always have a Subject line that starts with [Epics Alert], so you can set up auto-forwarding and mail sorting rules, and will always include a web link to the Plot page for the PV.

Managing Alerts

From the web page for Alerts, each can be set to be inactive and then be re-activated later. This can be useful to temporarily suppress messages for PVs that are not always critical. Alerts can also be deleted completely. Epics PV Archiver Plotting and Viewing

Plotting and Viewing of PV Data is fairly straightforward using the Plot page. From this page, you can choose one or two PVs to plot, and the time range for the data to be plotted. You can also explicitly set the range for the y-axis (ordinate), and whether to use normal or log scale for the y-axis.

There are two methods for setting the time range (abscissa). First, you can select the 'Time From Present' button and the time length from the drop down menu (ranging from 15 minutes to 1 month). Second, you can give an explicit time range and select the 'Data Range' button. Here, the format needs to be "YYYY-mm-dd HH:MM:SS" for both dates. Clicking on the small buttons will pop-up a calendar than can be used to chose the date and time for these time ranges.

The plot should be shown automatically, and labeled well enough to interpret the data. In addition, links are provided for:

The data format for these data files is a three column ASCII file with headers marked by lines starting with #. The header will include some information about time range, and for ENUM PVs, the legend for mapping integer ENUM values to strings. The data rows (one per time) will be of the form

YYYYMMDD HHMMSS UNIXTS   Value

That is, a 4 digit year, a 2 digit month, a 2 digit day, 1 space, a 2 digit hour, a 2 digit minute, a 2 digit minute, 1 space, a floating point Unix timestamp (seconds since epoch), 1 space, and the data value. Epics PV Archiver Templates and Web Page Layout

The web status pages are automatically generated from the template files in the 'web template directory'. For this installation, the directory is %(template_dir)s . There, you will find a README file (which mostly repeats the instructions here), a unix makefile, a file named pages.py, a file named FileList, and possible several other files, each containing the template for an individual 'Notebook Tab' web page.

Each template file can list many PVs, typically grouped logically, and will be shown as a separate tab in the status page, with the pages title. The template format for each of these template files is detailed below.

Ordering of Pages / Overall Structure

The master file FileList in the template directory contains the list of Title, Template File pairs:


Storage Ring | storage_ring.txt
Vacuum System | vacuum.txt

and so on, with Title and File name separated by '|'. This will be used to make the 'Notebook Tabs' in the status page. For this example, there will be two tabs; 'Storage Ring' and 'Vacuum System'.

To add or change the Tab titles or template files, alter FileList and run 'make'. This will generate pages.py which will be loaded by the web scripts.

You do NOT need to run 'make' when you change a template file: these are read on the fly. You do need to run 'make' when you want to add a new tabbed page, change the titles for each tab, or change the order in which they appear.

Template Formatting:

The status pages contain an HTML table, with each row corresponding to one or more Epics PVs. The general idea is to list the PV(s) to be shown. A simple example would be:

 S:SRCurrentAI.VAL

This will generally get mapped to a table row that has

   Description               Current_Value_with_Link_to_Archive

The description shown can be found automatically from the PVs own description field, if available. If it is not available, the PV name will be used. In either case, it can be set explicitly set by separating with a '|':

 S:SRCurrentAI.VAL | Ring Current

In addition, the formatting can be explicitly set as well:

 S:SRCurrentAI.VAL | Ring Current | %%.2f

For ENUM PVs, the ENUM string will be shown as the Current_Value, but it's not uncommon for integer PVs to hold a 'status' field with values '0' and '1' (for example, the 'Stations Searched' PVs at the APS!) For these, there a special 'yes/no' format can be used to map 0 to No, 1 to Yes, and any other value to Unknown.

To specify the formatting but use the default description, use


 S:SRCurrentAI.VAL | | %%.2f

You can put multiple PVs on a single line, separated by commas:


XXX:m1.VAL, XXX:m2.VAL

Which will list many PVs on a single line. You'll probably want to set the description explicitly:


XXX:m1.VAL, XXX:m2.VAL | Sample Motors (chi, theta)

Other formatting options: a section heading line in red can be given with [Title], a horizontal rule with ----, and an empty row with <> Several examples are provided in the files 'GSE_pageN.txt' And, again, the template files can be edited 'on-the-fly'.