Network Service Benchmarking Introduction
Network Services Benchmarking (NSB), part of the OPNFV Yardstick project, oﬀers service providers, OEMs, software vendors, and systems integrators a framework for characterization and benchmarking of virtual network functions (VNF), network functions virtualization infrastructure (NFVI), and network services. It utilises the Yardstick framework, to characterise a wide range of workloads, and also provides facilities for benchmarking your NFVi. NSB combines a number of open source components including: OPNFV Yardstick (test framework), OPNFV Barometer (NFVi metric collection), OPNFV SampleVNFs (reference VNFs), OpenStack, Open vSwitch, DPDK and collectd
The Need for Community-Driven Benchmarking
Across the industry, the deployment of virtualized networks and functions has been challenged by a lack of common standards and industry-accepted benchmarks for conformance to carrier-grade requirements. This has made it difficult for communications service providers (CoSPs) and other stakeholders to eﬀectively evaluate the network solutions on the market, compare how diﬀerent vendor solutions may impact their existing network requirements, and estimate total cost of ownership (TCO) when it comes to network transformation.
At the same time, more eﬀective tools to characterize network performance—along with the ability to dimension network workloads and model the impact of stress vectors such as trafc throughput—would give independent software
vendors (ISVs), original and telecommunications equipment manufacturers (OEMs/TEMs), and systems integrators (SIs) the ability to make (or implement) best-in-class products that meet the needs of network operators, and to bring those solutions to market more quickly.
To meet these challenges, Intel contributed the Network Services Benchmarking (NSB) project along with industry partners and the OPNFV community. The NSB framework features were added to the Yardstick tool to support both network functions virtualization infrastructure (NFVI) and virtualized network functions (VNFs) characterization with the goal of facilitating deterministic and repeatable benchmarking. NSB has the ﬂexibility to help CoSPs, ISVs, VNF vendors, OEMs/TEMs, and SIs fnd performance bottlenecks and evaluate VNF/NFVI solutions using standard benchmarks, a key factor in enabling network functions virtualization (NFV) deployments to help networks become more cost-efcient and agile. NSB also added representative workloads under SampleVNF project to demonstrate the characterization and benchmarking using Yardstick tool
What is in NSB?
VNF characterization and benchmarking is done in three different execution environments - bare metal (i.e. native Linux environment), standalone virtual environment and managed virtualized environment (e.g. OpenStack etc.) to help quantify the cost of virtualisation and orchestration.
Yardstick Framework extensions
A number of extensions were made to the Yardstick framework to help support NSB, such as additional traffic generator support, new contexts (environments) and support for VNFs
Testcase examples are given for running RFC2544 and RFC3511 tests with different VNFs, traffic profiles and scaling factors (CPU, memory, ports, etc)
NSB can use OPNFV SampleVNFs, and allows support to be added for commercial VNFs as well. Available SampleVNFs are CGNAPT, vFW, vACL, vPE, Router and Prox.
NSB characterisation provides simultaneous views of Network, NFVi and application metrics for a given test scenario.
Traffic generator support
NSB has the capability to interact with external traffic generators (both hardware and software based) for triggering and validating the traffic according to user defined profiles. There is currently support for Ixia, Trex, Spirent and PROX
NSB Testing Methodologies
1. Test cases characterization methodology enable benchmark across multiple environments:
- Bare metal test cases
- Standalone virtualization test cases
- Managed virtualization test cases
2. One tool for both VNF and NFVi bench-marking:
- Test cases for NFVi characterization
- Test cases for VNF characterization
3. Standardized test cases in NSB for testing VNFs:
- IETF standards:
- RFC-3511 for firewall
- 3GPP standards for vEPC testing
4. NSB test cases features includes:
- Scale-up testing for ports and cores
- Scale-out testing for ports and cores
- Reuse the stack in heat context to speed up testing
5. Traffic generator portfolio support in NSB for VNF and NFVi characterization:
- Open-source traffic generators
- Commercial traffic generators
- Spirent landslide
Want to Get involved in the NSB project?
Join our weekly meetings:
Contact Deepak S if you want to contribute to this project
Contribute to our code base
How to contribute to Yardstick
How to contribute to sampleVNF
Presentations and Demos
- Intel Developer Zone:
- NSB Quick Start Guide
- Network Virtualization Europe: NFV Characterization - Best Known Methodologies
- ONS Europe Conference Sept 2018:
Demo: Need to upload to youtube
PPT: Network Service Benchmarking
- OpenStack Summit Vancouver 2018:
- OPNFV Plugfest Euphrates Release Presentation
- NSB Overview:
- Sample VNF in OPNFV:
- NSB Hands-On Lab Training (from Out Of The Box Network Developers summit):
- NSB Demo by Aarna Networks:
- NSB Presentation at OpenStack 7th Birthday: