Page tree
Skip to end of metadata
Go to start of metadata


Please note that this page is a draft page aiming to prepare the Danube release...(image src: openclipart.org)

http://etherpad.opnfv.org/p/yardstick_release_d

Loading
Key Summary T Created Assignee Status
YARDSTICK-405 new reliability/availability generic test cases in network Epic Nov 04, 2016 Unassigned Open
YARDSTICK-382 support the integration of Vsperf Story Nov 01, 2016 Jing Lu Open
YARDSTICK-523 Support plugin list Story Dec 30, 2016 Jing Lu Open
YARDSTICK-232 Test Case: OPNFV_YARDSTICK_TC041_DPDK pktgen throughput and packet loss Story Mar 14, 2016 Oktávián Papp Open
YARDSTICK-386 new reliability/availability generic test cases in compute Story Nov 01, 2016 Unassigned Open
YARDSTICK-404 VNF on-boarding / deployment Story Nov 01, 2016 Unassigned Open
YARDSTICK-381 yardstick support testing projects integration in Danube Epic Nov 01, 2016 Rex Lee In Progress
YARDSTICK-469 Create API to update data source of certain dashboard Task Dec 09, 2016 Unassigned Open
YARDSTICK-467 Create API to add data source to certain dashboard Task Dec 09, 2016 Unassigned Open
YARDSTICK-466 Add API to update a dashboard of Grafana according to dashboard name Task Dec 09, 2016 Unassigned Open
YARDSTICK-468 Create API to delete data source of certain dashboard Task Dec 09, 2016 Unassigned Open
YARDSTICK-465 Add API to delete a dashboard of Grafana according to dashboard name Task Dec 09, 2016 Unassigned Open
YARDSTICK-464 Create API to create a dashboard of Grafana Task Dec 09, 2016 Unassigned Open
YARDSTICK-463 Create API to push a log file to certain directory Task Dec 09, 2016 Unassigned Open
YARDSTICK-461 Add API to push test result data into InfluxDB Task Dec 09, 2016 Unassigned Open
YARDSTICK-460 Add API to get information of certain scenario according to scenario name Task Dec 09, 2016 Unassigned Open
YARDSTICK-458 Add API to create a custom test suite Task Dec 09, 2016 Unassigned Open


Testing project integration

Vsperf 


Describe topologies that can be deployed with Open Stack
  • Plug-in - execute native vswitchperf testcases; results processed and reported by yardstick
  • Heat - vswitchperf runs in trafficgen mode; Yardstick launches VNFs and configures flows to test OVS performance in real world scenarios
  • Results dashboard
  • New traffic profiles iMIX genome, Flow based traffic, TCP
  • New test cases - QoS, VM-VM, RFC2544 features (pkt loss profile, …), RFC2889 LAN switching, Spirent tests (vlan, traffic, b2b, …), STP
  • NIC support for 40GE, accelerated OVS
  • Traffic generator support - Ixia scripts, Ixia SDN/NFV test suites, TCP support, Python APIs, Replay, Spirent virtual traffic gen

Are there Vsperf APIs provided for any features?

Qtip

  • Integration via yardstick plugin CLI.
  • Yardstick calls Qtip's test API to run benchmark tests.
  • Further details will be discussed  face to face during Barcelona Open Stack Summit.

Bottlenecks

  • bottlenecks call yardstick to run test case and return test results.
  • Yardstick supports reusing keep-deploy context.
  • Yardstick supports flavor-create in test case setup stage. Yardstick should support creating flavor in release-D.
  • The keep-deploy context issue should be solved first, then we solve the flavor-create action support issue.

Improvements

  • Yardstick API aligns with Functest API in release-D. 
  • For reporting and dashboard, we can do more things in release-D with Functest.

Test Cases

  • Describe realistic deployment scenarios that can be deployed with Open Stack and tested using Yardstick and integrated testing features. 
  • Describe detailed test topology
  • Describe traffic profiles 
  • Provide an accurate description of what is being tested, details of metrics measured, relevant specifications / references (what part of spec is actually implemented)
  • Provide suitability and accuracy of Test Tool/s (traffic generators, ...)
  • Provide interpretation of results

Yardstick generic test cases

Add reliability/availability test cases:


Compute (examples):
  • Processor availability (Error free processing time)
  • Memory availability (Error free memory time)
  • Processor mean-time-to-failure
  • Memory mean-time-to-failure
  • Number of processing faults per secondNetwork:
  • NIC availability (Error free connection time)
  • Link availability (Error free transmission time)
  • NIC mean-time-to-failure
  • Network timeout duration due to link failure
  • Frame loss rate 

Storage (examples):
  •  Disk availability (Error free disk access time)
  • Disk mean-time-to-failure
  • Number of failed storage read/write operations per second 

Feature projects (to be confirmed)

  • SDNVPN
  •  KVM
  • IPV6
  • FDIO
  • OVS
  • HA

Dovetail test cases

There may be test cases and test infrastructure features that Yardstick delivers to Dovetail for OPNFV compliance testing.

Framework Features

1. Heat integration method

  • Use Heat templates to deploy the test workloads.
  • Heat context should support more function if necessary.

2. Future vision

As part of release D lets provide a clear vision for the future with motivations and how we will work with other OPNFV and upstream projects (Models, OSM, etc.) 

3. Yardstick supports local deployment with influxDB and grafana

 

  • run 3 dockers including yardstick, influxDB and grafana simultaneously using docker compose
  • Add datasourses and specific dashboard to grafana using API
  • Add multiple graphs or tables to one dashboard if necessary using API
  • config yarstick docker via puppet

4. Yardstick custom vm image upgrade

5. Yardstick-key replaced by dynamic generation

6. Yarstick install unified (pip/setup)

7. Unit test complement 

Since there would be some improvements in architecture in release-D, so it's better to complement some necessary unit test.


8. REST API support

  • Support REST API to execute yardstick test case or test suite
  • Support REST API to call for yardstick service
  • Introduce REST API to use yardstick dispatcher service to push data to yardstick's influxdb database
  • Api document enable via swagger
  • Add yardstick concurrent support for REST API
  • simplify the logic between command and REST API

9.  HA testing framework improvement  (should discuss with HA team)

  • Enable HA testing results shown on Grafana
  • Other improvements on testing result accuracy, flexibility and usability of HA testing


10. log logistic simplified and unified

  • logger level unified
  • CI log only show the level of info
  • output debug log into artifacts

Dashboard

1. Yardstick / Functest common feature improvements and evolution

  • reporting (consistancy, reuse of same icons, dockerization, ...) 
    • refactor the code to reuse the same resource and use dynamic web server.

2. Grafana improvement

  • Improve Yardstick main dashboard and Yardstick scenario dashboard
  • Dashboard for new HA test cases.
  • Add thresholds for test cases.
  • Visualize the result with scenario view.
  • move grafana config from repo to artifacts.

Documentation

1. On-boarding materials for developers/testers, new test projects, new feature projects

 Document should be updated with detailed tutorial about common usage. 
  •  A dev guide and a user guide, like functest 
  • for user, we need a friendly and easy guide. for developer, we can provide more professinal guide.  As Frank mentioned,  we should give guide about how to  interpret result in grafana.
  •  Additional, update the video of Yardstick Installation.  the old one show the installation of yardstick in Virtual Env, Not docker way.sure so I 'd like to organize some meeting for training video.

2. 'apidocs' update how to update.

Fix display issue in release-D.

VNF on-boarding / deployment                                                                                           

  • Collaborate with Functest  for a consistent way to on-board VNFs using different MANO stacks  
  • Perhaps VNF deployment could be done by Functest through an API we develop together (details should be dicussed in the testperf meeting)
  •   Yardstick calls API to for deployment
  •   Functest calls API to run a qualification / load test

Backlog of features from Colorado  

  • We need to list a backlog in jira like functest do

Restful API for Danube (draft)

 ActionMethodDescriptionStatus
/yardstick/env/actionprepareYardstickEnvPOSTto fetch openrc file and load imagesdone in yardstick api
 createInfluxDBContainerPOSTto create influxDB container and config databasedone in yardstick api
 createGrafanaContainerPOSTto create grafana container and config grafanadone in yardstick api
/yardstick/docker/action
startContainer
POST
to run a docker container
 
 restartContainerPOSTto restart a docker container 
 
stopContainer
POST
to stop and remove a docker container
 
 
showStatus
GET
to get status of a docker container
 
 
showInformation
GET
to get information of a docker container
 
/yardstick/testcases
 GET
get a list of all test cases
done in yardstick cli
/yardstick/testcases/{name} GET
get information of a test case
done in yardstick cli
/yardstick/testcases/samples/actionrunTestCasePOSTto run a test case under samples directorydone in yardstick api
/yardstick/testcases/release/actionrunTestCasePOSTto run a test case under release directorydone in yardstick api
 createTestCasePOSTto create a test case 
 deleteTestCaseDELETEto delete a test case 
 updateTestCasePUTto update a test case 
/yardstick/testsuites GET
get a list of all test suites
 
/yardstick/testsuites/{name} GET
get information of a test suite
 
/yardstick/testsuites/action
runTestSuite
POST
to run a test suite
done in yardstick cli
 createTestSuitePOSTto create a test suite 
 deleteTestSuiteDELETEto delete a test suite 
 updateTestSuitePUTto update a test suite 
/yardstick/scenarios GET
get a list of all scenarios
 
/yardstick/scenarios/{name} GET
get information of a scenario
 

/yardstick/results

 

GET

to get the results

exist in yardstick api
/yardstick/results/action

pushData

POST

to push the test result data into test database

 
/yardstick/logs/{id}
 GET
to get scenario logs
 
/yardstick/logs/actionpushLogPOSTto push a log file 
/yardstick/dashboard/actioncreateDashboardPOSTto create a dashboard 
 deleteDashboardDELETEto delete a dashboard 
/yardstick/dashboard/datasource/action
addDataSource
POST
to add a dataSource
 
 deleteDataSourceDELETEto delete a dataSource 
 
configDataSource
PUT
to config a exist dataSource
 
/yardstick/utils/action
createImage
POST
to create a image which the test case need
 
  • No labels