Page tree
Skip to end of metadata
Go to start of metadata

Please note that this page is a draft page aiming to prepare the Danube release...(image src:

Key Summary T Created Assignee Status

Testing project integration


Describe topologies that can be deployed with Open Stack
  • Plug-in - execute native vswitchperf testcases; results processed and reported by yardstick
  • Heat - vswitchperf runs in trafficgen mode; Yardstick launches VNFs and configures flows to test OVS performance in real world scenarios
  • Results dashboard
  • New traffic profiles iMIX genome, Flow based traffic, TCP
  • New test cases - QoS, VM-VM, RFC2544 features (pkt loss profile, …), RFC2889 LAN switching, Spirent tests (vlan, traffic, b2b, …), STP
  • NIC support for 40GE, accelerated OVS
  • Traffic generator support - Ixia scripts, Ixia SDN/NFV test suites, TCP support, Python APIs, Replay, Spirent virtual traffic gen

Are there Vsperf APIs provided for any features?


  • Integration via yardstick plugin CLI.
  • Yardstick calls Qtip's test API to run benchmark tests.
  • Further details will be discussed  face to face during Barcelona Open Stack Summit.


  • bottlenecks call yardstick to run test case and return test results.
  • Yardstick supports reusing keep-deploy context.
  • Yardstick supports flavor-create in test case setup stage. Yardstick should support creating flavor in release-D.
  • The keep-deploy context issue should be solved first, then we solve the flavor-create action support issue.


  • Yardstick API aligns with Functest API in release-D. 
  • For reporting and dashboard, we can do more things in release-D with Functest.

Test Cases

  • Describe realistic deployment scenarios that can be deployed with Open Stack and tested using Yardstick and integrated testing features. 
  • Describe detailed test topology
  • Describe traffic profiles 
  • Provide an accurate description of what is being tested, details of metrics measured, relevant specifications / references (what part of spec is actually implemented)
  • Provide suitability and accuracy of Test Tool/s (traffic generators, ...)
  • Provide interpretation of results

Yardstick generic test cases

Add reliability/availability test cases:

Compute (examples):
  • Processor availability (Error free processing time)
  • Memory availability (Error free memory time)
  • Processor mean-time-to-failure
  • Memory mean-time-to-failure
  • Number of processing faults per secondNetwork:
  • NIC availability (Error free connection time)
  • Link availability (Error free transmission time)
  • NIC mean-time-to-failure
  • Network timeout duration due to link failure
  • Frame loss rate 

Storage (examples):
  •  Disk availability (Error free disk access time)
  • Disk mean-time-to-failure
  • Number of failed storage read/write operations per second 

Feature projects (to be confirmed)

  •  KVM
  • IPV6
  • FDIO
  • OVS
  • HA

Dovetail test cases

There may be test cases and test infrastructure features that Yardstick delivers to Dovetail for OPNFV compliance testing.

Framework Features

1. Heat integration method

  • Use Heat templates to deploy the test workloads.
  • Heat context should support more function if necessary.

2. Future vision

As part of release D lets provide a clear vision for the future with motivations and how we will work with other OPNFV and upstream projects (Models, OSM, etc.) 

3. Yardstick supports local deployment with influxDB and grafana


  • run 3 dockers including yardstick, influxDB and grafana simultaneously using docker compose
  • Add datasourses and specific dashboard to grafana using API
  • Add multiple graphs or tables to one dashboard if necessary using API
  • config yarstick docker via puppet

4. Yardstick custom vm image upgrade

5. Yardstick-key replaced by dynamic generation

6. Yarstick install unified (pip/setup)

7. Unit test complement 

Since there would be some improvements in architecture in release-D, so it's better to complement some necessary unit test.

8. REST API support

  • Support REST API to execute yardstick test case or test suite
  • Support REST API to call for yardstick service
  • Introduce REST API to use yardstick dispatcher service to push data to yardstick's influxdb database
  • Api document enable via swagger
  • Add yardstick concurrent support for REST API
  • simplify the logic between command and REST API

9.  HA testing framework improvement  (should discuss with HA team)

  • Enable HA testing results shown on Grafana
  • Other improvements on testing result accuracy, flexibility and usability of HA testing

10. log logistic simplified and unified

  • logger level unified
  • CI log only show the level of info
  • output debug log into artifacts


1. Yardstick / Functest common feature improvements and evolution

  • reporting (consistancy, reuse of same icons, dockerization, ...) 
    • refactor the code to reuse the same resource and use dynamic web server.

2. Grafana improvement

  • Improve Yardstick main dashboard and Yardstick scenario dashboard
  • Dashboard for new HA test cases.
  • Add thresholds for test cases.
  • Visualize the result with scenario view.
  • move grafana config from repo to artifacts.


1. On-boarding materials for developers/testers, new test projects, new feature projects

 Document should be updated with detailed tutorial about common usage. 
  •  A dev guide and a user guide, like functest 
  • for user, we need a friendly and easy guide. for developer, we can provide more professinal guide.  As Frank mentioned,  we should give guide about how to  interpret result in grafana.
  •  Additional, update the video of Yardstick Installation.  the old one show the installation of yardstick in Virtual Env, Not docker way.sure so I 'd like to organize some meeting for training video.

2. 'apidocs' update how to update.

Fix display issue in release-D.

VNF on-boarding / deployment                                                                                           

  • Collaborate with Functest  for a consistent way to on-board VNFs using different MANO stacks  
  • Perhaps VNF deployment could be done by Functest through an API we develop together (details should be dicussed in the testperf meeting)
  •   Yardstick calls API to for deployment
  •   Functest calls API to run a qualification / load test

Backlog of features from Colorado  

  • We need to list a backlog in jira like functest do

Restful API for Danube (draft)

/yardstick/env/actionprepareYardstickEnvPOSTto fetch openrc file and load imagesdone in yardstick api
 createInfluxDBContainerPOSTto create influxDB container and config databasedone in yardstick api
 createGrafanaContainerPOSTto create grafana container and config grafanadone in yardstick api
to run a docker container
 restartContainerPOSTto restart a docker container 
to stop and remove a docker container
to get status of a docker container
to get information of a docker container
get a list of all test cases
done in yardstick cli
/yardstick/testcases/{name} GET
get information of a test case
done in yardstick cli
/yardstick/testcases/samples/actionrunTestCasePOSTto run a test case under samples directorydone in yardstick api
/yardstick/testcases/release/actionrunTestCasePOSTto run a test case under release directorydone in yardstick api
 createTestCasePOSTto create a test case 
 deleteTestCaseDELETEto delete a test case 
 updateTestCasePUTto update a test case 
/yardstick/testsuites GET
get a list of all test suites
/yardstick/testsuites/{name} GET
get information of a test suite
to run a test suite
done in yardstick cli
 createTestSuitePOSTto create a test suite 
 deleteTestSuiteDELETEto delete a test suite 
 updateTestSuitePUTto update a test suite 
/yardstick/scenarios GET
get a list of all scenarios
/yardstick/scenarios/{name} GET
get information of a scenario




to get the results

exist in yardstick api



to push the test result data into test database

to get scenario logs
/yardstick/logs/actionpushLogPOSTto push a log file 
/yardstick/dashboard/actioncreateDashboardPOSTto create a dashboard 
 deleteDashboardDELETEto delete a dashboard 
to add a dataSource
 deleteDataSourceDELETEto delete a dataSource 
to config a exist dataSource
to create a image which the test case need
  • No labels