| The current version of Epics PV Archiver is 2.1 at |
As described in the Overview, the Epics PV Archiver has two main data stores for PV data, each controlled by a separate running process on the server:
| Command | Meaning |
| Status / General Information | |
| pvarch -h | shows a status message. |
| pvarch status | shows cache and archiving status, some recent statistics. |
| pvarch check | print # of archived PVs in past 10 minutes. Should be >1! |
| pvarch list | prints a list of recent data archives |
| pvarch save | create backup files of recent data archives |
| pvarch clean | clean out old files in the web data directory |
| The Archiving Process | |
| pvarch start | start the archiving process |
| pvarch stop | stop the archiving process |
| pvarch restart | restart the archiving process |
| pvarch next | restart with 'next run' of data archives |
| The Caching Process | |
| pvarch cache start | start cache process (if it's not already running) |
| pvarch cache stop | stop cache process |
| pvarch cache restart | restart cache process |
| pvarch cache status | show # of PVs cached in past 60 seconds |
| pvarch cache activity | show list of PVs cached in past 60 seconds |
| Adding and removing PVs | |
| pvarch add_pv | add a PV to the cache and archive |
| pvarch add_pvfile | read a file of PVs to add to the Archiver |
| pvarch drop_pv | remove a PV from cache and archive |
The deadtime for a PV sets how long to wait after archiving a PVs value before archiving another value for this PV. Let's take as an example a deadtime of 10 seconds. If the PV hasn't changed in longer than 10 seconds, then a change to the value will be archived as quickly as possible. But if the value changes again within 10 seconds, the change will not be recorded immediately. Instead, the change will be seen and the archiver will wait until 10 seconds has elapsed since the last time the value was recorded. This way, if multiple changes happen during that 10 seconds, the only value recorded will be the one held after 10 seconds has elapsed -- all intermediate changes will be forgotten.
One thing to note, though, is that the time of the last change will be recorded properly. Thus, if a value changes, changes again 1 second later, and then stays fixed, the archiver will record the changes as taking place 1 second apart.
The deadband for a PV sets how big a fractional change must be in order to be archived.
When PVs are added to the database, typical deadtimes are one or a few seconds, depending on whether the PV has a type 'double' -- which can easily change trivial amounts -- or a type of 'int','enum', or 'string', which change less often. For this setup, the default deadtime for PVs of type 'double' or 'float' is seconds, while the default deadtime for all other PVs is 1 seconds. Typical deadbands are set small enough to be not actually effective.
That is, you will want to periodically start archiving into a new database. Using pvarch next will tell the Archiver to make the "next" database in the series of archives and to start collecting into that new database. All PVs archived and all their settings in the old database will be moved into the new one. Using pvarch list will show a list of the most recent 'runs'.
Looking up a PV's value for plotting or data retrieval will seamlessly span multiple databases, so you don't have to worry about how often you start a new run. Because of this, putting pvarch next in a cron table to run once a week or once a month, or at some other frequency, is recommended. I would not recommend using a new database more often than once a day -- once a month or every few weeks seems about right.
Also, because both pvarch start and pvarch cache start will do nothing if there is a successfully running archiving and caching process, you can also put these two commands in cron tables, running more often to ensure that the caching and archiving are running.
Another potential use of a cron job can be to clean out the data files created for the data plotting and viewing. These are temporary files stored in {(config.data_dir}}. To clean old files in this directory, you can use the command
pvarch clean
Finally, to backup the databases with simple mysql dump files, use pvarch save. This will save the two principle databases: the master database, , and the currently active archive database. You can specify additional databases to save as command-line arguments:
pvarch saveExample crontab files for these tasks are included in the source distribution.
At any one time, one archive database is 'current', meaning that it is the one be written to, so the database in use is a sequentially number database, of the form , , .... Each databases holds a set of data for a specific time period -- a 'run'. Data retrieval uses the full set of archive databases, and looks up older spanning archive databases as necessary.
The assumption is that the Python programming interface will be used to access data in the PV Archive. While the data is stored with MySQL databases, using simple SQL queries to retrieve data from the PV Archive is slightly non-trivial. Still, for completeness, a partial description of the databases is provided here:
| Table | Description | Data Columns |
| cache | holds the cached data for all PVs. | id, pvname,type, value, cvalue, ts, active |
| cvalue is the 'character string value', ts is the timestamp, active holds whether the PV is actively cached. | ||
| runs | holds info about archiving 'runs'. | id, db,notes, start_time,stop_time |
| db is the name of the current archive databases. | ||
| info | status information about running 'cache' and 'archive' processes | id, process, status, db, datetime, ts, pid |
| process is either 'cache' or 'archive'. | ||
| requests | temporary storage for requested changes to the cache database | id, pvname, action |
| action is 'add', 'drop', 'suspend', or 'ignore'. The Caching process looks here for new values to include. | ||
| alerts | data about Alerts | id, pvname, name, mailto, mailmsg, compare, trippoint, timeout, active, status |
| pairs | data for Related PVs | id, pv1, pv2, score |
| instruments | list of instruments | id, name, station, notes |
| station holds the ID from the stations table. | ||
| stations | list of stations | id, name, notes |
| instrument_pvs | data to map PVs with instruments | id, pvame, inst |
| instrument_positions | data for saved positions for instrument | id, name, inst, active, ts |
The PV table has the following columns and meanings:
| Column | Description |
| id | integer ID |
| name | PV name |
| description | description of PV |
| data_table | name of corresponding data table |
| deadtime | deadtime for archiving |
| deadband | deadband |
| graph_hi | default high value for plotting range |
| graph_lo | default low value for plotting range |
| graph_type | default type for plotting: 'normal','log','discrete' |
| type | PV data type |
| active | Whether PV is actively being archived. |
In contrast, the data tables are much simpler, with columns of pv_id (that is the id from the PV table), time, and value
For further details, simply explore the MySQL databases. Epics PV Archiver Overview
The Epics PV Archiver saves Epics PVs into a MySQL database and provides a web interface for looking up, plotting, and managing the archiving process. The Archiver can easily accommodate thousands of PVs and save and retrieve data for many years.The PV Archiver provides Alerts, which will send an email notification when a particular PV goes out of an acceptable range. It also provides a concept of an Instrument, which is a group of PVs for which you may want to save and look up positions.
First, you can use the command line program pvarch, with
~> pvarch add_pv MyPV.VALYou can add multiple PVs on a single line:
~> pvarch add_pv MyPV1.VAL MyPV2.VALThis will add both PVs, and also set them as "related PVs", which will show up in the plot view page. Second, you can put a list of PVs in a file, either one per line or multiple PV names per line (separated by a space or comma), and run
~> pvarch add_pvfile PVList.txtThis is probably most useful on initial setup or when adding a large set of new PVs. As with pvarch add_pv, putting multiple PVs on a single line will make the "Related".
The third way to add PVs is to put them in one of the template files for the web status pages (See the Template Page for details). The next time the corresponding web page is accessed, the system will see that the PV is not included, will initially report 'Unknown', and request it to be added to the Cache and Archive processes.
For all these methods, the PV will not actually be added unless it is a valid Epics PV that can be accessed from the server. In some cases, you may wish to record remote PVs -- you'll want to set EPICS_ADDR_LIST accordingly.
When PV are added to the archive, pairs of PVs will get an initial pair score of 10 under the following conditions:
To set or change an alert, you will be challenged for the administrator password. As with all administrative tasks for the Archiver, this is a security measure so that only trusted users can set up emails sent from your system. To set up Alerts, a mail server and "from address" must be configured at installation time.
Use the main Alert Page to see a list of currently defined alerts. From this page you can select a "View/Change" link to modify any of the settings for a particular alert.
In addition to the PV name, each Alert has a label to describe the alert, and so that multiple alerts can be set on a single PV (one person may want an alert when a valve closes, while another person may want an alert when it opens!). The label will be used in the email sent when the Alert condition is met.
Each alert consists of 3 pieces: a PV's value, a 'test condition' and a 'trip point'. When the PV's value meets the condition for that trip point, the Alert is said to be in the 'Alarm' state. The possible test conditions are:
An email notice is sent to the specified addresses when the Alert changes from 'OK' to 'Alarm'. If the PV's value changes again, but still stay in the 'Alarm' state or if it stays in the 'Alarm' state indefinitely, NO more emails will be sent. In order for another message to be sent, the Alert would have to change back to an 'OK' condition', and then again to an 'Alarm' condition. In addition, each Alert has a 'Time Out' value (in seconds). Once an email has been sent for an alert, no email will be sent for that alert for this time, even if the Alert goes from 'OK' to 'Alarm'. This is provided to avoid multiple emails for a value that may rapidly fluctuate about its trip point.
Hello, An alarm labeled 'XXX went non-zero' was detected for PV = 'XXX.VAL' The current value = 1.0. This is not equal to the trip point value of 0.0You can customize this message on the Alerts page, putting in any text you want. Since you may want to include some of the the name and value of the PV, and the alert label, condition, and trip point, you can use the following simple template format for the mail message to automatically get these values:
Thus, you could customize the message to be:
Hello,
An alarm labeled %%LABEL%%
was detected because %%PV%% = %%VALUE%%.
Other PV Values at the time of this alert were:
Valve 1 : %%PV(Valve1.VAL)%%
Current : %%PV(Current.VAL)%%
To get a more complete report. The mail message will always have a Subject
line that starts with [Epics Alert], so you can set up
auto-forwarding and mail sorting rules, and will always include a web link
to the Plot page for the PV.
There are two methods for setting the time range (abscissa). First, you can select the 'Time From Present' button and the time length from the drop down menu (ranging from 15 minutes to 1 month). Second, you can give an explicit time range and select the 'Data Range' button. Here, the format needs to be "YYYY-mm-dd HH:MM:SS" for both dates. Clicking on the small buttons will pop-up a calendar than can be used to chose the date and time for these time ranges.
The plot should be shown automatically, and labeled well enough to interpret the data. In addition, links are provided for:
YYYYMMDD HHMMSS UNIXTS ValueThat is, a 4 digit year, a 2 digit month, a 2 digit day, 1 space, a 2 digit hour, a 2 digit minute, a 2 digit minute, 1 space, a floating point Unix timestamp (seconds since epoch), 1 space, and the data value. Epics PV Archiver Templates and Web Page Layout The web status pages are automatically generated from the template files in the 'web template directory'. For this installation, the directory is %(template_dir)s . There, you will find a README file (which mostly repeats the instructions here), a unix makefile, a file named pages.py, a file named FileList, and possible several other files, each containing the template for an individual 'Notebook Tab' web page.
Each template file can list many PVs, typically grouped logically, and will be shown as a separate tab in the status page, with the pages title. The template format for each of these template files is detailed below.
Storage Ring | storage_ring.txt Vacuum System | vacuum.txt
and so on, with Title and File name separated by '|'. This will be used to make the 'Notebook Tabs' in the status page. For this example, there will be two tabs; 'Storage Ring' and 'Vacuum System'.
To add or change the Tab titles or template files, alter FileList and run 'make'. This will generate pages.py which will be loaded by the web scripts.
You do NOT need to run 'make' when you change a template file: these are read on the fly. You do need to run 'make' when you want to add a new tabbed page, change the titles for each tab, or change the order in which they appear.
S:SRCurrentAI.VALThis will generally get mapped to a table row that has
Description Current_Value_with_Link_to_ArchiveThe description shown can be found automatically from the PVs own description field, if available. If it is not available, the PV name will be used. In either case, it can be set explicitly set by separating with a '|':
S:SRCurrentAI.VAL | Ring CurrentIn addition, the formatting can be explicitly set as well:
S:SRCurrentAI.VAL | Ring Current | %%.2f
For ENUM PVs, the ENUM string will be shown as the Current_Value, but it's not uncommon for integer PVs to hold a 'status' field with values '0' and '1' (for example, the 'Stations Searched' PVs at the APS!) For these, there a special 'yes/no' format can be used to map 0 to No, 1 to Yes, and any other value to Unknown.
To specify the formatting but use the default description, use
S:SRCurrentAI.VAL | | %%.2f
You can put multiple PVs on a single line, separated by commas:
XXX:m1.VAL, XXX:m2.VAL
Which will list many PVs on a single line. You'll probably want to set the description explicitly:
XXX:m1.VAL, XXX:m2.VAL | Sample Motors (chi, theta)
Other formatting options: a section heading line in red can be given with [Title], a horizontal rule with ----, and an empty row with <> Several examples are provided in the files 'GSE_pageN.txt' And, again, the template files can be edited 'on-the-fly'.