Collectd Plugin for ESTP over nanomsg

The ESTP is Extensible Statistics Transmission Protocol. The protocol is intended to become standard protocol for submission statistics data by nanomsg applications. More info can be found here:

http://github.com/estp/estp

The plugin allows both to receive data from nanomsg for storing by collectd and to publish collectd-provided statistics to nanomsg socket. So it's capable to send data between collectd instances too

Building

Ubuntu

For the impatient (tested for Ubuntu Precise):

apt-get install collectd-dev libsensors4-dev
make CFLAGS="-I/usr/include/collectd"
sudo make install

If this doesn't work, longer instructions follow. (It assumes you have nanomsg installed)

You need collectd headers somewhere, to build the plugin. In ubuntu they are at /usr/include/collectd in the collectd-dev package. The collectd headers may be ./configure'd with bigger number of libs installed than what collectd-dev depends on. So if you see No such file or directory you need to install some other library's headers (*-dev package) to build the plugin.

To compile you need CFLAGS="-I/usr/include/collect" parameter to make as it's the directory where include files are installed to.

Other Systems

There are three cases:

1. There is package that installs header files. Instructions are mostly same as with ubuntu, just you may need to adjust package names and directory

2. You compile collectd from source. Just set the environment variable CFLAGS="-I/path/to/collectd-sources/src" for make

3. You need to fetch collectd sources of the correct version and configure them, its very recommended that arguments to ./configure are same, that used to compile collectd. The process is roughly following:

   wget http://collectd.org/files/collectd-v4.10.1.tar.gz tar -xzf collectd-v4.10.1.tar.gz cd collect-v4.10.1 ./configure --prefix=/usr ... cd .. make CFLAGS="-Icollectd-v4.10.1/src make install

Configuration

The following is example config, which accepts data:

LoadPlugin nanomsg_estp
<Plugin "nanomsg_estp">
  <Socket Subscribe>
    Bind "tcp://*:6001"
  </Socket>
</Plugin>

Any nanomsg address can be specified for Bind.

The following is example config, which sends data from collectd to another collectd instance or any other processing application:

LoadPlugin nanomsg_estp
<Plugin "nanomsg_estp">
  <Socket Publish>
    Connect "tcp://host.example.com:6001"
  </Socket>
</Plugin>

Note: if you have write filters enabled, do not forget to add "nanomsg_estp" plugin to your "write" target.

The Connect and Bind directives both can be used in any socket types, and can be used multiple times. Pull socket can be used instead Subscribe, likewise Push socket can be used instead Publish. However, this is rarely useful in practice. For more information refer to nanomsg documentation.

Command-line

If you want to play with plugin from a command-line, it's possible both to submit data to collectd, and receive data from it.

First, let's prepare config:

LoadPlugin nanomsg_estp
<Plugin "nanomsg_estp">
  <Socket Subscribe>
    Bind "tcp://127.0.0.1:6001"
  </Socket>
  <Socket Publish>
    Bind "tcp://127.0.0.1:6002"
  </Socket>
</Plugin>
<Chain "PostCache">
    Target "write"
</Chain>

Then you can send data using the following command-line:

nanocat --pub --connect tcp://127.0.0.1:6001 --delay=0.2 --data="ESTP:localhost:myapp:instance1:value: $(TZ=UTC date +"%Y-%m-%dT%H:%M:%SZ") 10 100"

I assume that you on unix and have date utility. On Windows you may need to write date yourself. When submitting by hand, you may use shorter form:

nn_pub -l6001 -d0.2 -D"ESTP:localhost:myapp:instance1:value: $(TZ=UTC date +"%Y-%m-%dT%H:%M:%SZ") 10 100"

See the ESTP spec for the description of the data format. To read the data use the following:

nanocat --sub --connect tcp://127.0.0.1:6002 --format=ascii

This will show both the data sent from other nanomsg clients to collectd, and the data produced by collectd itself. As well as you can connect to any nanomsg client directly without collectd intermediary. Shorter form:

nn_sub -Al6002

You may filter values. To filter values from localhost only:

nn_sub -Al6002 --subscribe "ESTP:localhost:"

Note: that if you use --format=ascii or --ascii or -A option, you will see ESTP extensions garbled (e.g. many messages generated by collectd itself do have extension that stores collectd-specific info). You may use --format=quoted to have them correctly escaped.