InfluxDB UDP

Jesse's Software Engineering Blog

Jun 03

Database Administration / InfluxDB, PHP

InfluxDB is a database for storing time series data in a distributed environment. With zero dependencies and a robust querying language, InfluxDB is just beginning to grow its presence in the open source database market. One great use case for InfluxDB is storing application analytic data. With the ability to communicate with the server using UDP an application can be built to “send and forget” performance data to be analyzed over a period of time.

Installing InfluxDB

To install InfluxDB on Ubuntu 14.04 (64 bit) simply:

wget https://s3.amazonaws.com/influxdb/influxdb_latest_amd64.deb
sudo dpkg -i influxdb_latest_amd64.deb

Use like any other service

sudo service influxdb start
sudo service influxdb status

Other server installation instructions can be found in the InfluxDB documentation.

Once installed the admin interface can be pulled up in a browser. Be sure to use root/root for user/password and set the localhost port value to 8086.

http://localhost:8083/

Once logged in create a database: phpudp

Enabling UDP For InfluxDB

By default InfluxDB does not enable UDP communication. After the installation the config file will be in one of these locations

/opt/influxdb/shared/config.toml
/usr/local/etc/influxdb.conf

Open the config file and update the [input_plugins.udp] section to

enabled = true
port = 4444
database = "phpudp"

NOTE: There is a one to one relationship between a port and a database. When you are ready to write to other databases look at the [[input_plugins.udp_servers]] section to assign different databases to different ports.

Sending Data Over UDP With PHP

PHP has a variety of methods for communicating with a UDP socket connection. I prefer to use the stream_* methods as they work with other PHP resource functions such as the file manipulation functions (fwrite, fclose, etc.) and require no additional extensions to be loaded.

InfluxDB is a schemaless database so the data points sent are arbitrary. For example to track load time for routes there could be data like

$data = [
    'route' => '/games/football',
    'code'  => 200,
    'time'  => 134
];

Next define the required fields for a data point in InfluxDB and add the data

$packet = [
    'name' => 'collection',
    'columns' => array_keys($data),
    'points' => [
        array_values($data)
    ]
];

The name is the collection, or table, name, which does not have to be created before hand. The columns are the column names and points are the data points. Notice the points is an array and can accepts multiple sets of data as long as all sets share the same column names.

Now simply open the UDP socket and send the data

$socket = stream_socket_client("udp://127.0.0.1:4444");
stream_socket_sendto($socket, json_encode([$packet]));
stream_socket_shutdown($socket, STREAM_SHUT_RDWR);

Create a connection to the correct port, encode the data, and send. Again notice that the json_encode() takes an array therefore multiple packets could be sent, each packet being for a different collection and having different column names, and each containing multiple data points.

After running the above code the data points should be visible in the InfluxDB admin UI via the following query.

select * from /.*/ limit 1

Monitoring UDP Traffic

The benefit of using UDP is that there is none of the overhead associated with TCP i.e. making connections and ensuring that the data is received. Therefore it can be difficult to track down problems when data is not correctly being received. One way to monitor UDP traffic is with the tcpdump command

sudo tcpdump -i lo -p -n -s 1500 udp

This allows monitoring of UDP packets on the localhost, so for example

14:54:42.941144 IP 127.0.0.1.58457 > 127.0.0.1.4444: UDP, length 99

The output displays where the packet was sent and the length.

Conclusion

Using InfluxDB as a time series data aggregator can be extremely useful data applications which can benefit from UDP connections. InfluxDB is a great replacement for older technologies such as Graphite. With InfluxDB’s robust query language and web API, it is easy to store various types of data and access it easily using a variety of different languages.

JessesNet />