Data pullers

Data pullers are external programs, used to pull state from the equipment as fast as possible. Data pullers are usually used in high-load environments.

Configuration

Data pullers are defined in Universal Controller configuration, section [datapullers] as <name> = <cmd>, where:

  • name data puller name (unique)
  • cmd data puller startup command

e.g.

[datapullers]
dp1 = /opt/dp1/datapuller -c /opt/dp1/config.yml
dp2 = /opt/dp2/datapuller -c /etc/dp2.ini
dp3 = datapuller3

Execution

All data pullers are started automatically at Universal Controller start. Data pullers can be stopped or restarted later using UC API functions or CLI (eva uc datapuller).

Universal Controller always tries to kill data puller and its child subprocesses on stop / shutdown. However to avoid critical situations, data puller SHOULD exit manually, if STDOUT / STDERR are not available any longer (standard programs do that by default).

Data exchange

Basics

Data puller is a program, or a shell script, written in any language, which repeatedly pulls the equipment, collects and processes data and then sends it to stdout. Data pulling should be done in an infinite loop. Start / stop operations are handled by Universal Controller.

When stop (SIGTERM signal) is received, data puller has 1 second to shut down itself gracefully, otherwise it (and its child processes) will be forcibly killed.

If data puller process exits or crashes, it will be automatically restarted. The automatic restart is performed with 1-second delay.

The idea of data pullers is developing small programs in low-level programming languages to let EVA ICS receive only modified data. Data pullers are good for Ethernet/IP and Modbus slave equipment, as well as for SNMP-enabled equipment without a support of SNMP traps or any other equipment, which is unable to send any state events.

Health check

If there are no events, data puller SHOULD send .ping or empty string to STDOUT, otherwise it will be considered as dead after period equal to a default controller timeout and automatically restarted:

.ping

Data output

Data puller should collect data and output to STDOUT update commands in format, equal to UC UDP API:

<item_id> u <status> <value>
# e.g.
sensor:tests/test1 u 1 777.555

If status or value need to be skipped, they should be set to “None”:

sensor:tests/test1 u None 777.555

Data pullers SHOULD:

  • collect and output all available data on startup
  • during the execution, send to Universal Controller only modified data
  • manually handle logic mapping, e.g. converting Modbus registers or Ethernet/IP tags to EVA ICS items states.

Logging

Data puller can send messages to Universal Controller logs. If stdout line starts with .log, it’s considered to be the log command:

.log <level> <message>
# e.g.
.log warning Some warning information

Where level is the level of the message (debug, info, warning, error or critical, first letter is enough e.g. w for warning).

Data puller can also print messages to STDERR, they will be automatically logged with error level.

Environment variables

All cvars and other EVA ICS core variables are available in the datapuller process system environment.

Pulr

Pulr is a free data pulling tool, 100% compatible with EVA ICS. Refer to Pulr documentation about the tool configuration, here are few remarks how to use it:

  • In Pulr configuration, set output: eva/datapuller to output data in EVA ICS native format.
  • Set beacon value less than Universal Controller default timeout, otherwise the Pulr subprocesses will be killed and continuously restarted.
  • All fields in process sections must have set-id param, to convert pulled data IDs into EVA ICS item IDs (<unit|sensor>:<group>/<id>)
  • set-id must end with either .status or .value to set the proper EVA ICS items state fields. If no ending is specified, item state value is updated.

Pulr supports SNMP (v2), Modbus (TCP/UDP) and Ethernet/IP (Allen-Bradley compatible) field buses.