NF Development
Overview
The openNetVM manager is comprised of two directories: one containing the source code for the manager and the second containing the source for the NF Lib. The manager is responsible for assigning cores to NFs, maintaining state bewteen NFs, routing packets between NICs and NFs, and displaying log messages and/or statistics. The NF_Lib contains useful libraries to initialize and run NFs and libraries to support NF capabilities: packet helper, flow table, flow director, service chains, and message passing.
Currently, our platform supports at most 128 NF instances running at once with a maximum ID value of 32 for each NF. We currently support a maximum of 32 NF instances per service. These limits are defined in onvm_common.h. These are parameters developed for experimentation of the platform, and are subject to change.
NFs are run with different arguments in three different tiers–DPDK configuration flags, openNetVM configuration flags, and NF configuration flags–which are separated with --
.
DPDK configuration flags:
Flags to configure how DPDK is initialized and run. NFs typically use these arguments:
-l CPU_CORE_LIST -n 3 --proc-type=secondary
openNetVM configuration flags:
Flags to configure how the NF is managed by openNetVM. NFs can configure their service ID and, for debugging, their instance ID (the manager automatically assigns instance IDs, but sometimes it is useful to manually assign them). NFs can also select to share cores with other NFs and enable manual core selection that overrides the onvm_mgr core selection (if core is available), their time to live and their packet limit (which is a packet based ttl):
-r SERVICE_ID [-n INSTANCE_ID] [-s SHARE_CORE] [-m MANUAL_CORE_SELECTION] [-t TIME_TO_LIVE] [-l PACKET_LIMIT]
NF configuration flags:
User defined flags to configure NF parameters. Some of our example NFs use a flag to throttle how often packet info is printed, or to specify a destination NF to send packets to. See the simple_forward NF for an example of them both.
Each NF needs to have a packet handler function. It must match this specification: static int packet_handler(struct rte_mbuf* pkt, struct onvm_pkt_meta* meta);
A pointer to this function will be provided to the manager and is the entry point for packets to be processed by the NF (see NF Library section below). Once packet processing is finished, an NF can set an action coupled with a destination NF ID or destination port ID in the packet which tell the openNetVM manager how route the packet next. These actions are defined in onvm_common:
ONVM_NF_ACTION_DROP
: Drop the packetONVM_NF_ACTION_NEXT
: Forward the packet using the rule from the SDN controller stored in the flow tableONVM_NF_ACTION_TONF
: Forward the packet to the specified NFONVM_NF_ACTION_OUT
: Forward the packet to the specified NIC port
Skeleton NF
In order to provide a baseline implementation for the development of NFs, a skeleton is provided in openNetVM/examples/skeleton/
. This skeleton outlines all required files and functions within any standard ONVM NF, and their respective uses. These functions, which are declared in the function table or control execution, include:
main:
int main(int argc, char *argv[])
Responsible for initializing the function table and local NF context. Starts and stops the NF, and handles command-line arguments.setup
static void setup(struct onvm_nf_local_ctx *nf_local_ctx)
Allows the NF to initialize non-local data or perform necessary instructions which must be executed before receiving packets or messages.action
static int action(struct onvm_nf_local_ctx *nf_local_ctx)
Continually called while the NF is running, allowing the user to perform actions or checks which are packet or message-independent.packet_handler
static int packet_handler(struct rte_mbuf *pkt, struct onvm_pkt_meta *meta, struct onvm_nf_local_ctx *nf_local_ctx)
As described previously, the handler function will handle each packet upon arrival.message_handler
static void handle_msg(void *msg_data, struct onvm_nf_local_ctx *nf_local_ctx)
Like the packet handler, the message handler function will handle messages from other NFs upon arrival.
For those who wish to use the skeleton as a training mechanism, inline commentary will provide additional clarification on the purpose of each function and where edits must be made throughout the files. The NF also includes various customary helper-functions for those seeking a baseline template; these functions can be removed if not applicable.
To demonstrate the management of data, the NF contains high-level functionality that counts the number of packets received and the time which the NF has been running. This data is stored within a state struct initialized in main and stored within the local context. The NF will print the current time and number of packets on a delay which is specified by the user at the command line (see /examples/skeleton/README.md
for further details). To observe packets being counted, you can easily send packets from the speed_tester NF by following the instructions on the :ref:`OpenNetVM Examples Page <examples>`_ (you are looking to create a linear NF chain, but the circular NF chain instructions will provide context on how to use the speed_tester PCAP files). If you are looking for introductory comprehension, tracing through the skeleton NF is a logical place to begin. For developing more advanced NFs, template usage requires only a few deletions.
NF Library
The NF_Lib Library provides functions to allow each NF to interface with the manager and other NFs. This library provides the main communication protocol of the system. To include it, add the line #include "onvm_nflib.h"
to the top of your c file.
Here are some of the frequently used functions of this library (to see the full API, please review the NF_Lib header):
int onvm_nflib_init(int argc, char *argv[], const char *nf_tag, struct onvm_nf_info** nf_info_p)
, initializes all the data structures and memory regions that the NF needs run and communicates with the manager about its existence. Fills the passed double pointer with the onvm_nf_info struct. This is required to be called in the main function of an NF.int onvm_nflib_run(struct onvm_nf_info* info, void(*handler)(struct rte_mbuf* pkt, struct onvm_pkt_meta* meta))
, is the communication protocol between NF and manager, where the NF provides a pointer to a packet handler function to the manager. The manager uses this function pointer to pass packets to the NF as it is routing traffic. This function continuously loops, giving packets one-by-one to the destined NF as they arrive.
Advanced Ring Manipulation
For advanced NFs, calling onvm_nf_run
(as described above) is actually optional. There is a second mode where NFs can interface directly with the shared data structures. Be warned that using this interface means the NF is responsible for its own packets, and the NF Guest Library can make fewer guarantees about overall system performance. The advanced rings NFs are also responsible for managing their own cores, the NF can call the onvm_threading_core_affinitize(nf_info->core)
function, the nf_info->core
will have the core assigned by the manager. Additionally, the NF is responsible for maintaining its own statistics. An advanced NF can call onvm_nflib_get_nf(uint16_t id)
to get the reference to struct onvm_nf
, which has struct rte_ring *
for RX and TX, a stat structure for that NF, and the struct onvm_nf_info
. Alternatively NF can call onvm_nflib_get_rx_ring(struct onvm_nf_info *info)
or onvm_nflib_get_tx_ring(struct onvm_nf_info *info)
to get the struct rte_ring *
for RX and TX, respectively. Finally, note that using any of these functions precludes you from calling onvm_nf_run
, and calling onvm_nf_run
precludes you from calling any of these advanced functions (they will return NULL
). The first interface you use is the one you get. To start receiving packets, you must first signal to the manager that the NF is ready by calling onvm_nflib_nf_ready
.
Example use of Advanced Rings can be seen in the speed_tester NF or the scaling example NF.
Multithreaded NFs, scaling
NFs can scale by running multiple threads. For launching more threads the main NF had to be launched with more than 1 core. For running a new thread the NF should call onvm_nflib_scale(struct onvm_nf_scale_info *scale_info)
. The struct scale_info
has all the required information for starting a new child NF, service and instance ids, NF state data, and the packet handling functions. The struct can be obtained either by calling the onvm_nflib_get_empty_scaling_config(struct onvm_nf_info *parent_info)
and manually filling it in or by inheriting the parent behavior by using onvm_nflib_inherit_parent_config(struct onvm_nf_info *parent_info)
. As the spawned NFs are threads they will share all the global variables with its parent, the onvm_nf_info->data
is a void pointer that should be used for NF state data.
Example use of Multithreading NF scaling functionality can be seen in the scaling_example NF.
Packet Helper Library
The openNetVM Packet Helper Library provides an abstraction to support development of NFs that use complex packet processing logic. Here is a selected list of capablities that it can provide:
Swap the source and destination MAC addresses of a packet, then return 0 on success.
onvm_pkt_mac_addr_swap
can be found hereCheck the packet type, either TCP, UDP, or IP. If the packet type is verified, these functions will return 1. They can be found here
Extract TCP, UDP, IP, or Ethernet headers from packets. These functions return pointers to the respective headers in the packets. If provided an unsupported packet header, a NULL pointer will be returned. These are found here
Print the whole packet or individual headers of the packet. These functions can be found here.
Config File Library
The openNetVM Config File Library provides an abstraction that allows
NFs to load values from a JSON config file. While NFLib automatically
loads all DPDK and ONVM arguments when -F
is passed, a developer can
add config support directly within the NF to support passing additional
values.
NOTE: unless otherwise specified, all DPDK and ONVM arguments are required
onvm_config_parse_file(const char* filename)
: Load a JSON config file, and return a pointer to the cJSON struct.This is utilized to launch NFs using values specified in a config file.
onvm_config_parse_file
can be found hereAdditional config options can be loaded from within the NF, using cJSON. For further reference on how to access the values from the cJSON object, see the cJSON docs
Sample Config File
1 {
2 "dpdk": {
3 "corelist": [STRING: corelist],
4 "memory_channels": [INT: number of memory channels],
5 "portmask": [INT: portmask]
6 },
7
8 "onvm": {
9 "output": [STRING: output loc, either stdout or web],
10 "serviceid": [INT: service ID for NF],
11 "instanceid": [OPTIONAL, INT: this optional arg sets the instance ID of the NF]
12 }
13 }
Running Groups of NFs
Additionally, a developer can run the run group script to deploy multiple network functions, including linear or circular chains of multiple NFs, from a JSON config file. An example config file can be found here. “NF Name” indicates the example NF to be run and must be the name of the NF folder in the examples folder.
Optional “globals” in the config file include:
“TTL”: specifies number of seconds for the NFs to run before shutdown. If no timeout is specified, the NFs will run until a raised error or a manual shutdown (Ctrl + C).
“directory”: specifies a directory name. A directory will be created (if it does not already exist) for the output log files.
“directory-prefix”: a directory will be created with the prefix + timestamp. If no directory name or directory-prefix is specified, the default name of the created directory will the be the timestamp. Output from each NF will be continuously written to the corresponding log text file within the created or pre-existing directory. Format of the log file name will be: “log-NF name-instance ID”.
To track the output of a NF:
This script must be run within the /examples folder: