Exported all error and discard statistics for igb, ixgbe, i40e and fm10k (Physical Functions and Virtual Functions) drivers as metrics.
Export virtual interface drop and error counters as metrics.
DPDK Keep Alive Core Status is exposed as a metric and a notification is generated in the case of failure to notify fault management applications
Export Link status as a metric and generate an notification in the case of a change
SNMP Support – TODO
MIB Support – Will Support standard MIBs
Failure events must be detected within 10ms
Collection interval should be configurable.
DPDK provides a number of features to expose statistics and events relevant to DPDK interfaces and packet processing cores. A collectd read plugin should take advantage of these features to expose the key telco KPI statistics that allow you to monitor the performance of DPDK interfaces.
In terms of stats, DPDK provides two functions to access packet-forwarding statistics. Firstly the generic statistics API, and secondly through the extended statistics (xstats) API. The first API returns generic aggregate packet-forwarding statistics; that is statistics that are common across all NIC drivers such as: packets received, packets sent, bytes received, bytes sent, etc. Although valuable information, modern NICs collect and expose much more detailed information which cannot be accessed through the generic statistics interface. The xstats API was designed to transparently expose this information. The extended statistics API allows each individual NIC Driver to expose a unique set of statistics. The xstats API also exposes the generic stats API metrics. The collectd plugin will take advantage of the xstats API to expose all the DPDK interface metrics.
Due to the large number of stats and the stringent timing requirements for events failure detection, a separate plugin will be developed for DPDK events.
DPDK Keep Alive or KA is a feature that acts as a heartbeat/watchdog for DPDK packet processing cores, to detect application thread failure. The feature supports the detection of ‘failed’ DPDK cores and exposes the packet processing core state through a POSIX shared memory Object. This feature ensures that the failure of a DPDK packet processing core does not result in a fault that is not detectable by a management entity.
Collectd plugins to export DPDK stats
A read plugin ‘dpdkstat’ gathers DPDK packet-forwarding statistics with collectd. The plugin itself uses a DPDK process to retrieve the stats from DPDK. DPDK allows an application to launch different types of DPDK processes as part of its multi-process support. The two types of processes supported to date, include:
- “Primary processes, which can initialize and which have full permissions on shared memory
- Secondary processes, which cannot initialize shared memory, but can attach to pre- initialized shared memory and create objects in it.
Standalone DPDK processes are primary processes, while secondary processes can only run alongside a primary process or after a primary process has already configured the hugepage shared memory for them” .
The dpdkstat plugin is implemented as a DPDK secondary process, allowing a DPDK primary process to configure and initialize the system and forward packets. The secondary process then reads the existing configuration, and can access the same hardware as the primary process.
DPDK fork off a separate process
Currently, the DPDK library doesn’t support API to de-initialize the DPDK resources allocated on the initialization. It means, the collectd plugin will not be able to release the allocated DPDK resources (locks/memory/pci bindings etc.) correctly on collectd shutdown or reinitialize the DPDK library if primary DPDK process is restarted. The only way to release those resources is to terminate the process itself. For this reason, the plugin forks off a separate collectd process. This child process becomes a secondary DPDK process which can be run on specific CPU cores configured by user through collectd configuration file (“Coremask” EAL configuration option, the hexadecimal bitmask of the cores to run on). More information on EAL configuration options can be found in collectd.conf man page.
During the implementation of the dpdkstat plugin a number of issues were identified. These complications were addressed by adding features to the DPDK EAL, and implementing extra functionality in the dpdkstat plugin. These issues include:
1. Secondary Process Start-up
When a DPDK primary process starts it configures the system, writing a configuration file. The default location for the configuration file is “/var/run/.rte_config”. When a DPDK secondary process starts it reads the configuration file, as written by the primary process (see Figure 7).
Figure 7: rte_eal_primary_proc_alive()
Figure 7 Shows how rte_eal_primary_proc_alive() can be used to check if a DPDK primary process is currently running
When implementing dpdkstat, DPDK did not have any way to probe if a primary process was running, or if the configuration file was valid. A DPDK primary process holds a write-lock on the configuration file, while a secondary process takes a read-lock. This file-locking has been used to implement a function to check if a primary process is currently running. The file-lock held by the primary has the added functionality that if the primary process terminates unexpectedly the kernel will remove the file-lock, which allows the secondary process to detect that the primary has went down.
A function rte_eal_primary_proc_alive(const char *file) was added to the DPDK environment abstraction layer (EAL), allowing a secondary process to query primary liveliness. The file parameter allows a secondary process to specify which primary configuration file it would like to monitor. This function solves both the issue of detecting when a primary process dies, as well as if there is currently a primary process running.
2. Primary and Secondary Interaction
When a primary process terminates, the secondary process is required to quit and release the read-lock it holds on the configuration file so that a primary process can then be re-launched, and re-initialize the DPDK EAL. The problem that had to be solved was that there is no DPDK function to release all EAL resources, instead the process must exit and the kernel cleans up the resources.
In the context of the dpdkstat plugin, this means that rte_eal_init() cannot be called in the collectd process, as it would be mandatory to quit the collectd process in order to re-initialize the DPDK EAL. To solve this issue a helper-process is created, and any DPDK functions are called in this process instead of the collectd process. The process can be terminated or spawned as needed, without affecting the collectd process.
Figure 8 shows how the collectd process creates a shared memory object and spawns the helper process. The helper process checks if a DPDK primary is alive using the rte_eal_primary_proc_alive(). When a primary is alive, the number of ports and number of statistics per port are written to the shared memory, and the DPDK helper process quits.
Figure 8: dpdkstat start-up
At this point, the collectd process can unlink the shared memory, and re-allocate a new region of shared memory that is large enough to contain the names and values of all the statistics available in the currently running primary process (see Figure 9).
Figure 9: dpdkstat prepare for reading values
The shared memory is now large enough to contain all the values to be dispatched by collectd and the helper process is alive. When collectd calls the read function of the dpdkstat plugin, it posts a semaphore which the helper process is waiting on. This kicks the DPDK helper into action, reading the statistics and writing the values into shared memory. When finished writing to the shared memory the DPDK helper posts a different semaphore, indicating that the collectd thread may now continue and dispatch the statistics contained in shared memory to collectd (see Figure 10).
Figure 10: dpdkstat reading values
This concludes one iteration of the dpdkstat plugin, checking if a primary is alive, configuration of the shared memory, reading statistics and pushing values to collectd.
3. Plugin initialization time
Since helper process re-spawning requires some synchronization with main collectd process, and plugin internal state changes are driven by read callback calls, initial plugin start up time depends on plugin read interval. First five plugin read cycles are utilized on initialization activities and during that time no DPDK port statistics data are submitted.
Also if plugin is running and the number of DPDK ports is increased, internal buffers have to be resized. That requires three read cycles and no port statistics are submitted in that time.
4. DPDK port visibility
When network port controlled by Linux is bound to DPDK driver, the port will not be available in the OS. It affects the SNMP write plugin as those ports will not be present in standard IF-MIB. Thus addition work is required to be done to support DPDK ports and statistics.
Multi process support
The collectd multi-process support is achieved using dpdk-procinfo application and collectd exec plugin (see reference section for more details on those application). The interaction between those applications is described on figure 11.
Figure 11 Collectd exec plugin/dpdk-procinfo interaction
The collectd exec plugin forks off the dpdk-procinfo application and reads all statistics from its STDOUT (in specific format described in collectd exec plugin documentation). By default, the dpdk-procinfo doesn’t support the sending of DPDK statistics into STDOUT which is understandable by collectd exec plugin. For this reason, the dpdk-procinfo application needs to be fixed and extended. The two new options have been added to the dpdk-procinfo to support the collectd exec plugin format:
1) “--host-id” - The host id used to identify the system process is running on.
2) “--collectd-format” to print statistics to STDOUT in expected by collectd format. In this case the DPDK port statistics are sent to STDOUT in the following format understandable by collectd exec plugin:
PUTVAL <host-id>/dpdkstat-port.<port-id>/<counter-type>-<xstats-name> N:<xstats-value>
1) dpdk-procinfo - http://dpdk.org/doc/guides/tools/proc_info.html
2) collectd exec plugin - https://collectd.org/documentation/manpages/collectd-exec.5.shtml
- L2/L3 statistics are only supported;
- The DPDK primary process application should use the same version of DPDK that collectd DPDK plugin is using.
Since, the network devices bound to DPDK are not visible to the OS, addition work on is required for SNMP agent to support DPDK ports.
MIB name and location.
Alarms, events, statistics considerations
The monitoring solution should not have an adverse impact on the overall platform – in terms of cpu, memory, io… footprint. It should also not significantly impact the performance any primary applications running on the system
Expected plugin initialization time should be taken into account during testing.
The plugin has been tested on Intel NIC’s only.
The following table outlines possible impact(s) the deployment of this deliverable may have on the current system.
System Impact Description
Recommendation / Comments
The following assumptions apply to the scope specified in this document.
The following exclusions apply to the scope discussed in this document.
The following table outlines the key dependencies associated with this deliverable.
DPDK 16.04 or later
Only DPDK 16.04, 16.07, 16.11 have been tested
Monitoring DPDK multiple instances or DPDK applications used in a container
It's recommended that you use DPDK's procinfo application executed through the exec plugin to allow you to monitor multiple DPDK applications or DPDK applications running in a container
- Create new user and add it to docker group (for example):
useradd -m -b /home <username>
usermod -a -G docker
service docker restart
2. Modify collectd.conf file exec plugin configuration by adding the following lines (as example):
Exec "romank:docker" "/usr/bin/env" "HOME=/home/<username>" "/usr/bin/docker" "exec" "-i" "71d3c65ac38e" "../dpdk/app/proc_info/dpdk-procinfo" "-c 0x000000000500" "--proc-type=secondary" "-n 4" "--socket-mem" "1024,0" "-w 04:10.3" "--" "--xstats" "/bin/bash”
After building and configuring the container collectd can be run.
- docker is a usergroup.
- "/usr/bin/env" "HOME=/home/<username>" is mentioned here because by default docker is looking for its configuration files in /root directory.