Data pullers

Data pullers are external programs, used to pull state from equipment as fast as possible. Data pullers are usually used in high-load environments.

Configuration

Data pullers are created either with “eva uc datapuller” CLI command or with corresponding UC API methods.

Execution

All data pullers are started automatically at Universal Controller start. The data pullers can be stopped or restarted later using UC API functions or CLI (eva uc datapuller).

Universal Controller always tries to kill a data puller and its child subprocesses on stop / shutdown. However, to avoid critical situations, data puller SHOULD exit manually if STDOUT / STDERR are not available any longer (standard programs do this by default).

Data exchange

Basics

Data puller is a program, or a shell script, written in any language, which repeatedly pulls the equipment, collects and processes data and then sends it to stdout. Data pulling should be done in an infinite loop. Start / stop operations are handled by Universal Controller.

When stop (SIGTERM signal) is received, data puller has 1 second to shut down itself gracefully, otherwise it (and its child processes) will be forcibly killed.

If a data puller process exits or crashes, it will be automatically restarted. The automatic restart is performed with 1-second delay.

The idea of data pullers is developing small programs in low-level programming languages to let EVA ICS receive only modified data. Data pullers are good for Ethernet/IP and Modbus-slave equipment, as well as for SNMP-enabled equipment without a support of SNMP traps or any other equipment, which is unable to send any state events.

Health check

If there are no events, a data puller SHOULD send .ping or empty string to STDOUT, otherwise it will be considered as dead after the period equal to the data puller timeout (or the default controller timeout) and automatically restarted:

.ping

Additional parameter “event-timeout” asks UC to restart the data puller if there are no events from it.

Data output

Data puller should collect data and output update commands to STDOUT in format, equal to UC UDP API:

<item_id> u <status> <value>
# e.g.
sensor:tests/test1 u 1 777.555

If the status or the value need to be skipped, they must be set to “None”:

sensor:tests/test1 u None 777.555

Data pullers SHOULD:

  • collect and output all available data on startup
  • during the execution, send to Universal Controller only modified data
  • manually handle logic mapping, e.g. converting Modbus registers or Ethernet/IP tags to EVA ICS items states.

Logging

Data puller can send messages to Universal Controller logs. If a stdout line starts with .log, it is considered to be a log command:

.log <level> <message>
# e.g.
.log warning Some warning information

Where level is the level of the message (debug, info, warning, error or critical, first letter is enough, e.g. w for warning).

The data puller can also print messages to STDERR, they will be automatically logged with error level.

State reporting

A data puller can report custom state messages (e.g. “loading”, “working”, “having a problem”) with lines starting with “.state”:

.state <message>
# e.g.
.state almost ready

Environment variables

All cvars and other EVA ICS core variables are available in the datapuller process system environment.

Pulr

Pulr is a free data pulling tool, 100%-compatible with EVA ICS. Refer to Pulr documentation about the tool configuration, here are few remarks how to use it:

  • In Pulr configuration, set output: eva/datapuller to output data in EVA ICS native format.
  • Set beacon value to less than Universal Controller default timeout, otherwise the Pulr subprocesses will be killed and continuously restarted.
  • All fields in process sections must have set-id param, to convert pulled data IDs into EVA ICS item IDs (<unit|sensor>:<group>/<id>)
  • set-id must end either with .status or with .value to set the proper EVA ICS items state field. If no ending is specified, the item state value is updated.

Pulr supports SNMP (v2), Modbus (TCP/UDP) and Ethernet/IP (Allen-Bradley compatible) field buses.