Page tree
Skip to end of metadata
Go to start of metadata

Network Service Benchmarking Introduction

Network Services Benchmarking (NSB), part of the OPNFV Yardstick project, offers service providers, OEMs, software vendors, and systems integrators a framework for characterization and benchmarking of virtual network functions (VNF), network functions virtualization infrastructure (NFVI), and network services. It utilises the Yardstick framework, to characterise a wide range of workloads, and also provides facilities for benchmarking your NFVi. NSB combines a number of open source components including: OPNFV Yardstick (test framework), OPNFV Barometer (NFVi metric collection), OPNFV SampleVNFs (reference VNFs), OpenStack, Open vSwitch, DPDK and collectd


The Need for Community-Driven Benchmarking

Across the industry, the deployment of virtualized networks and functions has been challenged by a lack of common standards and industry-accepted benchmarks for conformance to carrier-grade requirements. This has made it difficult for communications service providers (CoSPs) and other stakeholders to effectively evaluate the network solutions on the market, compare how different vendor solutions may impact their existing network requirements, and estimate total cost of ownership (TCO) when it comes to network transformation.


At the same time, more effective tools to characterize network performance—along with the ability to dimension network workloads and model the impact of stress vectors such as trafc throughput—would give independent software
vendors (ISVs), original and telecommunications equipment manufacturers (OEMs/TEMs), and systems integrators (SIs) the ability to make (or implement) best-in-class products that meet the needs of network operators, and to bring those solutions to market more quickly.


To meet these challenges, Intel contributed the Network Services Benchmarking (NSB) project along with industry partners and the OPNFV community. The NSB framework features were added to the Yardstick tool to support both network functions virtualization infrastructure (NFVI) and virtualized network functions (VNFs) characterization with the goal of facilitating deterministic and repeatable benchmarking. NSB has the flexibility to help CoSPs, ISVs, VNF vendors, OEMs/TEMs, and SIs fnd performance bottlenecks and evaluate VNF/NFVI solutions using standard benchmarks, a key factor in enabling network functions virtualization (NFV) deployments to help networks become more cost-efcient and agile. NSB also added representative workloads under SampleVNF project to demonstrate the characterization and benchmarking using Yardstick tool


What is in NSB?

Methodology

VNF characterization and benchmarking is done in three different execution environments - bare metal (i.e. native Linux environment), standalone virtual environment and managed virtualized environment (e.g. OpenStack etc.) to help quantify the cost of virtualisation and orchestration.

Yardstick Framework extensions

A number of extensions were made to the Yardstick framework to help support NSB, such as additional traffic generator support, new contexts (environments) and support for VNFs

Testcases

Testcase examples are given for running RFC2544 and RFC3511 tests with different VNFs, traffic profiles and scaling factors (CPU, memory, ports, etc)

VNFs

NSB can use OPNFV SampleVNFs, and allows support to be added for commercial VNFs as well. Available SampleVNFs are CGNAPT, vFW, vACL, vPE, Router and Prox.

KPI collection

NSB characterisation provides simultaneous views of Network, NFVi and application metrics for a given test scenario.

Traffic generator support

NSB has the capability to interact with external traffic generators (both hardware and software based) for triggering and validating the traffic according to user defined profiles. There is currently support for Ixia, Trex, Spirent and PROX






NSB Testing Methodologies


NSB Features:

1. Test cases characterization methodology enable benchmark across multiple environments:

  • Bare metal test cases
  • Standalone virtualization test cases
    • SR-IOV
    • OVS-DPDK
  • Managed virtualization test cases 
    • Openstack 
    • Kubernetes

2. One tool for both VNF and NFVi bench-marking:

  • Test cases for NFVi characterization
  • Test cases for VNF characterization

3. Standardized test cases in NSB for testing VNFs:

  • IETF standards: 
    • RFC-2544,
    • RFC-3511 for firewall
  • 3GPP standards for vEPC testing

4. NSB test cases features includes:

  • Scale-up testing for ports and cores
  • Scale-out testing for ports and cores
  • Reuse the stack in heat context to speed up testing

5. Traffic generator portfolio support in NSB for VNF and NFVi characterization:

  • Open-source traffic generators
    • T-Rex
    • PROX
    • Packet-gen
  • Commercial traffic generators 
    • IxNetwork
    • IxLoad
    • Spirent landslide


Want to Get involved in the NSB project?

Yardstick

Join our weekly meetings:

SampleVNF

Contact Deepak S if you want to contribute to this project


Contribute to our code base

Repository 

Yardstick (https://github.com/opnfv/yardstick)

sampleVNF (https://github.com/opnfv/samplevnf)

Gerrit

https://gerrit.opnfv.org/

JIRA Projects

Yardstick

sampleVNF

JIRA

https://jira.opnfv.org

Release Planning

Yardstick Releases

sampleVNF Releases

How to contribute to Yardstick

Getting started with Yardstick

How to contribute to sampleVNF

https://wiki.opnfv.org/display/SAM/Get+started+as+a+SampleVNF+developer


Presentations and Demos