Page tree
Skip to end of metadata
Go to start of metadata

Characterize vSwitch Performance for Telco NFV Use Cases: Team Meetings

Meeting Logistics

Time

Wednesday PT 0800.

IRC

freenode https://freenode.net/

channel: #opnfv-vswitchperf http://webchat.freenode.net/?channels=opnfv-vswitchperf

 

MeetBot http://meetbot.opnfv.org/meetings/opnfv-vswitchperf/2017/  

 

Audio

Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/2362828999

Or iPhone one-tap :
US: +16699006833,,2362828999# or +16465588656,,2362828999# 
Or Telephone:
Dial(for higher quality, dial a number based on your current location): 
US: +1 669 900 6833 or +1 646 558 8656 or +1 877 369 0926 (Toll Free) or +1 855 880 1246 (Toll Free)
Meeting ID: 236 282 8999
International numbers available: https://zoom.us/zoomconference?m=Xn-Kas4jq2GuyfbCCKYpwvi6FpHEYX8n




vSwitch Performance Characterization Meeting Minutes

MEETING MINUTES NOW STORED @ http://ircbot.wl.linuxfoundation.org/meetings/opnfv-vswitchperf/


Jan 25 2017

August 31st 2016

Action items

Updates

  1. vhost-cuse support has been removed
  2. vloop-vnf (http://artifacts.opnfv.org/vswitchperf/vnf/vloop-vnf-ubuntu-14.04_20160823.qcow2) has been updated with:
    1. Ethtool - required for vanilla OVS multiqueue configuration

    2. only 1 NIC is configured by default to speed up boot with 1 NIC setup

    3. security updates applied

  3. multi-vms patch for testing multiple VMs in series and in parallel is under review.
  4. multi arch support for vsperf patches up for review
  5. Vanilla OVS multiqueue patch merged to master
  6. Guest drive configuration options - merged to master
  7. Bugfix for MoonGen Incorrect RX_MBPS Calculation

MoonGen

 

"Its not ready for a release yet, but behind the scenes Andrew is working a new version of MoonGen with all kinds of new features that will be able to be integrated into VSPERF.   There is a little more development to be done and stability testing.  Also, there have been some minor changes on how to configure MoonGen's traffic profile that would today break MoonGen as it is today within VSPERF.  So there will be some work done here locally before he pushes the new version to his MoonGen stable branch within Github.  More details forthcoming in the upcoming weeks...

 

Lastly, I see Christian opened (thanks!) a Jira ticket for tracking an effort to add latency measurements for MoonGen within VSPERF - JIRA-383.  I just assigned this to myself.  I probably won't have time to work on this particular issue over the next week but I will make it a priority in the near future if its deemed more important than other outstanding issues". -  Bill Michalowski

 

Background load

Question on background Load - we  have tests for CPU and memory load using stress and stress ng - do we need tests that take background traffic load into account. Ref: https://www.net.in.tum.de/fileadmin/bibtex/publications/papers/IFIP-Networking-2016-FLOWer.pdf and original paper https://www.net.in.tum.de/fileadmin/bibtex/publications/papers/MoonGen_IMC2015.pdf

 

Latency

priotity to address this for release D in terms of being able to measure and report latency distributions. 

 

Yardstick

Simple integration for release D complete - phy2phy type deployment and using vsperfs traffic gen capabilities

Need to think about integration through the plugin arch for release D as well as more complex deployment that build on the initial integration.

 

http://ircbot.wl.linuxfoundation.org/meetings/opnfv-vswitchperf/2016/opnfv-vswitchperf.2016-08-31-15.01.html 

August 24th 2016 - No Meeting

August 17th 2016

Action items

  1. Maryam to create a wiki where we can start collecting features for OPNFV release D
  2. vsperf community to add stories for the august sprint by EOD Fri 19th Aug 2016 (sprint will be kicked off Monday)
  3. vsperf community to add features to this wiki over the next 3 weeks


  1. Sprint
    1. Maryam set the last sprint to complete so any in progress stories have been moved to the new sprint
    2. any stories that were "resolved" were also closed
    3. August sprint open to add stories till EOD Fri 19th Aug 2016 
    4. sprint will be started after this.
    5. proposal that we start doing monthly sprints and planning at the end of each month for the upcoming sprint 
    6. AGREED
    7. proposal that we start doing monthly sprints and planning at the end of each month for the upcoming sprint
    8. agreed 
  2. release D planning 
    1. ACTIONMaryam to create a wiki where we can start collecting features for OPNFV release D 
    2. ACTIONvsperf community to add features to this wiki over the next 3 weeks 
    3. planning to start in 3 weeks time 
  3. Moongen
    1. clean up in progress 
  4. Xena 
    1. IMIX feature under consideration
    2. multi queue for OVS implementation underway, some tuning required 
    3. https://tools.ietf.org/html/rfc6985
    4. the link is to the imix genome 
  5. Colorado
    1. branch cut 
    2. branch to be tagged soon. 

August 10th 2016

Dev update

  • LTD LTP split complete and merged
  • hugepage allocation on the fly complete and merged.

Xena Status:

  • One patch pushed

Moongen Status:

  • bugfixes
  • documentation updates

Spirent Status:

  • patch to harmonize trial/test terminology

August 3rd 2016 

Dev update

  • LTD LTP split complete and patch is up for review.
  • testpmd crash debug still underway.
  • doc updates in progress.
  • support for allocating hugepages ion the fly has been pushed to gerrit for review.

Xena Status:

  • Continuing to get internal ci up and running - vsperf stable now.
  • Starting to look at release D feature planning.

Spirent Status:

  • Setting up a vsperf env locally.
  • final testing on the fwding patches in progress
  • trial/test testing to follow soon.

Moongen Status:

  • Bug fixing in progress: patch for fixed 64b packet assumption and for the rx & tx %.
  • Starting to look at integration with CI.

Trial/test terminology

  • Agreement that the number of trials SHOULD NOT be configurable.
  • Agreement that only the number of tests SHOULD be configurable.
  • Should decide for release D if we report out on the average.

Apex

  • now supports vsperf installation

Actions

Questions:

July 27th 2016 - No meeting, Email update


MoonGen status:

  • Still have the problem regarding running phy2phy / MoonGen with VLAN tagged frames (OvsVanilla not forwarding them).
  •  CI work: 
    • phy2phy_tput - running this locally without issue.   TX rate reported greater than 100% in certain cases.  will open a Jira ticket... should be an easy fix
    • back2back - Again, running it locally just fine
    • phy2phy_tput_mod_vlan -  not run yet, but from the name there will be VLAN tagged frames so the above issue could be a problem
    • phy2phy_scalability- test not tried yet.
    • pvp_tput pvp_back2back -test not tried yet, but received instructions from Christian on how one configures the virtual machine
    • pvvp_tput pvvp_back2back - Same status as pvp_back2back


Xena Status:

  • Xena still stable in my extended runs.

Development:

  • Christian working with Mars to get his hugepage patch in so I can finish mine.
  • testpmd issue debug underway.
  • DPDK 16.07 support integrated with vsperf.
  • ixnet configuration document updated on how to setup a shared drive for the results.

July 20th 2016

Actions from last weeks meeting

  • Maryam to send on CI information to Bill - done 
  • cleaning up naming in Jira still due will be completed in the next few days 

Moongen Update 

  • Having issues with vlan id, Vlan tag is set to zero so traffic doesn't flow through the system 
  • Bill will create a Jira to get the issue fixed 
  • Maryam will share with DPDK team and see if they are experiencing issues with Vlan tag 

Xena Update

  • Running full runs with VSPERF and they are stable 
  • Only one open issue/bug which should be fixed soon 
  • Overall release C is in a good state 
  • OVS app control has been merged 
  • Martin suggested to submit a test case to show case the work for Release D

Dev Update 

  • Started integration with new results databases ,patches will be submitted shortly 
  • URL where we send the results will change 
  • Results will be graphed automatically once completed 
  • Scripts moved from Releng to OPNFV docs, now causing issues with nightly run, currently been fixed

Documentation Update

  • One outstanding test case left match action that needs to be ported to our documentation 
  • Feature list will be updated as part of C release 
  • All documentation needs to be completed by August 15th 

July 6th 2016

Meeting cancelled due to 4th July holiday in the US 

June 29th 2016 

No meeting due to low attendance

June 22nd 2016 

Summary of Actions

  • ACTION: Simona to follow with committer list and how many reviewers are required to push a patch 
  • ACTION: Simona to post Christians presentation to wiki 

Dev Updates

  • Working on vsperf/yardstick integration.
  • Created a simple yardstick scenario, which deploys virtual machine with VSPERF and executes vsperf in trafficgen mode. It means, that vsperf configures IXIA in accordance with specified test parameters and VSPERF configuration and executes the traffic.
  • Currently experimenting with manual configuration of OVS in OpenStack installation to correctly route traffic from IXIA ports towards VNF (VM with loopback application)

 Moongen Update

  • Not much time spend on moongen this last week, due to Redhat summit but still on target for end of June
  • its functional at the moment just need to do some clean up before submitting patches 

Xena Update

  • Last of patches were merged today 
  • This week burst traffic will be pushed ,endcap decap patch will also be pushed this week 
  • On target to complete committed features for code freeze 
  • Potentially multi queue may be at risk will know more in the coming days 

Spirent Update 

  • Question raised by Sridhar, How do you add committers to review ?
  • Is there an official list of people that can review 
  • ACTION: Simona to follow with committer list and how many reviewers are required - Complete 
  • Srihdar to upload patches in the coming days 

Presentation by Christian

  • Topic VSPERF Namespace/Veth peer port prototype 
  • ACTION: Simona to post presentation to wiki 

June 15th 2016 

Release C 

  • Confirmed with David test freeze date is 7th July 

Dev Updates 

  • Martin working with Yardstick and looking for integration point 
  • tcl scripts are being ported to phython (work is just kicked off) 
  • Pod 3 nearly setup , Maryam  will post info re:Pod 3 to the wiki

Moongen Update

Xena Update

  • Still stuck due to build issue , waiting to push patches through 

OPNFV Summit

  • Maryam encouraging folks to attend the Intel booth 
  • no formal meet up planned for VSPERF 

Agenda topics for upcoming calls

  • Christian planning on presenting next week 
  • Backlog grooming for the week after next 29th June 
  • Maryam to share back to back test results next week via mail 

June 8th 2016 

Summary of Actions

  • ACTION: Simona to update Jira tickets with new version for Colorado and remove label - complete 

Pod 3 Update:

  • node 5 is setup 
  • All instructions are up on the wiki 
  • Only ixia test left to run

Xena Update

  • Currently doing 40gb testing - investing issue with Xena2544.exe
  • Also working on back to back testing  

Moongen Update

  • Moongen continuous traffic gen is now merged and running 

  • Patches to be submitted soon

Spirent Update

  • Opened a new Jira ticket for clean up 
  • Maryam to follow up on open tickets 

Release C 

  • Key release dates ,feature Freeze 1st July, Stable Branch created August 15th , Release planned for 22nd September
  • ACTION: Simona to update Jira tickets with new version for Colorado and remove label 

June 1st 2016 

Summary of Actions

  • ACTION: Sridhar to upload presentation to the wiki 
  • ACTION: Maryam to send link to Wiki
  • ACTION: Simona to follow up with Dave on Code Freeze date - Complete 


Pod 3 Update:

  • Jack has replaced all of the servers in POD 3 
  • CI job is up and running
  • Maryam currently working on setting up sandbox  

Xena Update

  • Currently doing 40gb testing 
  • Also working on back to back testing  

Moongen Update

  • Problem with short run where things don’t calibrate, Andrew and Bill troubleshooting the issue

Spirent Update 

  • Working on support scripts for 2889

OPNFV Summit 

  • Larry and Al attending
  • If anyone else is travelling please let Maryam know 

 Presentation 'Changes to RFC 2899'

  • Sridhar presented slides on  changes to RFC 2899 
  • ACTION: Sridhar to upload presentation to the wiki 
  • ACTION: Maryam to send link to Wiki

 Back to Back Testing 

  • Maryam shared results of back to back test  just ran the test for Phy to Phy  for 64 byte packets 
  • Please refer to Maryam's presentation for summary and detail of results

May 25th 2016 

Summary of Actions 

  • ACTION: Maryam to run vfio test and post results to JIRA 
  • ACTION: Sridhar to give a presentation next week on the proposed changes to results for RFC 2899 
  • ACTION: Maryam to collate results for back to back tests from the nightly build so we can see how consistent the results are 
  • ACTION: Maryam to follow up with David McBride on code freeze date 

Committers nominations 

  • Congrats to Christian, Bill, Martin and Sridhar on their promotion to vsperf committers 

Dev update 

  • Yardstick ramp up still ongoing 
  • working on installing yardstick 

Xena update 

  • Still looking to bottom out on the continuous pvp results
  • ACTION: Maryam to run vfio test and post results to JIRA 
  • Christian has some patches and fixes for bidirectional traffic to be pushed soon. 

Spirent update 

  • ACTION: Sridhar to give a presentation next week on the proposed changes to results for RFC 2899 

Moongen Integration 

  • B2B tests working 
  • Bill is doing some general cleanup and refactoring
  • First patches to be pushed 1st week in June 
  • Automated configuration/translation from vsperf params to Moongen params is now implemented also 

Back to back testing 

  • ACTION: Maryam to collate results for back to back tests from the nightly build so we can see how consistent the results are 
  • question, should the back to back test start at line rate for a particular time, maintain the minimum interframe gap and if there's packetloss, modify/reduce the rate run for the same period of time till the back to back value is found 
  • that's one way of doing it. B2B test could also start at a low rate and work up. Or could be a binary search... 
  • issues with the binary search in our current world... 
  • TBD after we look at results next week 
  • ACTION: Maryam to follow up with David McBride on code freeze date 


May 18th 2016 

Summary of Actions 

  • Action: Simona to check with Maryam if Yardstick intro presentation has been posted to our wiki 
  • Action: Once the sandbox is ready I need some help/a volunteer to integrate and test the latest set of patches that Mircea Dan from IXIA provided us with for RFC5481 testing
  • Action: Christian volunteered to help with updating User guide, will follow up with an email to Maryam 
  • Action: Simona to include lastest update from David (Release manager) in meeting minutes - Complete see notes below

 Dev Update

  • Support of DPDK v16.04 has been merged.
  • Ramping up on yardstick. Putting HW in place for first installation.
  • Yardstick intro was provided at the OVS4NFV call.

CI Update

We've been having issues with the Hard drive connected to POD 3, it was causing some odd reboots mid tests. The Hard drive has been replaced and we're running an build at the moment which seems stable.

POD 3:

  • Jack will be upgrading the servers in POD3 from Haswell to Broadwell EP. Once this is done I will setup the SFQM platform as a sandbox env in the meantime for vsperf.
  • Once the sandbox is ready I need some help/a volunteer to integrate and test the latest set of patches that Mircea Dan from IXIA provided uswith for RFC5481 testing.

 Moongen Update 

  • Moogen readout next week ? Bill’s update: haven’t merged anything yet, plan to have code submitted end of May timeframe would be better to postpone review until then
  • New version of Moongen on the Github with few fixes not too different from previous version 
  • Plan to have code submitted by end of May 
  • Plan to have everything completed by mid-June 

Xena Update

  • Waiting for response from Maryam on continuous results 
  • Did some full runs of Vsperf with Xena in the lab found some issues with receive numbers, plan to have fix in next few days 

 Spirant Update

  • Working on closing old Jira tickets 
  • RFC 2889 created setup changes, need to share with the team to get approval with from the team before going ahead with making the changes 

Rel C 

  • New release date being proposed Sept 22nd 
  • Which would mean feature code freeze will move to July 1st 
  • Not official yet but its being proposed to TSC next week

Colorado Schedule link to wiki  here

 Here is summary from David Release manager on rationale of moving the Release to September

  1. More granular milestones.
  2. Schedule test definition early in the process to avoid having test implementation on top of test execution.
  3. Schedule installer integration with OpenStack and running Functest smoke tests as early as possible.
  4. Separate feature freeze from scenario integration and installer freeze.
  5. Separate test and infra development/updates from test execution.
  6. Schedule as much time as possible for test==>debug==>fix iterations following scenario integration.
  7. Take into account resource unavailability in July/August due to vacations.  Particularly for the FuncTest and Yardstick teams, which are on the critical path.

May 4th 2016

Summary of Actions 

  • Action: Bill to add a Jira ticket for non aggregated results
  • Action: Maryam to follow up on decision for database tool

Moongen Update:

Xena Update:

  • Packaging together continuous patch, once all working will publish the patch

Spirent Update:

  • Tickets that are under review that need to be closed, Sridhar needs to figure out what needs to be done to close them out

Dev Update: 

  • Few patches merged this week
  • Some issues resolved on Pod 3
  • Thanks to Christian for making the updates to huge page configuration
  • Bug fix for VSPERF on Ubuntu now resolved

Rel C 

  • Still need to add timings for all of our tests
  • Updates for VSPERF completed for Scenario wiki and test case wiki
  • Next week we will focus on Release C Health Check (focus will be what we have had planned and what’s the current state)

Test Case Review

 

April 27th 2016

Summary of Actions

  • Action: Sridhar to send Jira ticket numbers
  • Action: Maryam to revert back with any details on if we plan on providing multi-queue in the future
  • Action: Prevaze to send Maryam an email in relation to getting new updates included for VSperf HW acceleration

Moongen Update

·         Can now invoke RFC2544 test next step is to parse the data 

Xena Update 

·         VSPERF 273 Xena Port IP Assignment (currently set to static IP assignment) currently in review

Spirent Update 

·         Sridhar having issues with some old Jira tickets that need to be closed (Sridhar to follow up re:what action is needed to close them out)

Multi-queue

 

April 20th 2016

Summary of Actions

  • Action: Maryam to add steps to the deck make it easier to follow
  • Action: Maryam to send on link with details
  • Action: Maryam to write up Matchaction performance test
  • Action: All please go to ethepad and help with defining tests as listed in the etherpad

Reviewed Actions from last week

  • Need to move update from yardstick to next week no one available to join this week
  • Al has made relevant changes to slides 

Moongen Update

  • Not much progress this week 

Xena Update

  • b2b tests have been merged 
  • Trying to get integration decapsulation not working with Xena, currently having some issues 

Spirent Update

  • Need to add Jira tickets for test cases 

Dev Update

  • Few bug fixes submitted 
  • reboot causing problems 
  • waiting on wildcat passes for Pod 3 
  • due in on Friday 22nd 
  • port for vhost user have been sent for review 

Vm2vm presentation review

  • Maryam updated the slides based on feedback from last week 
  • question do you accommodate CPU pinning ? (see action referenced above for follow up)

Rel C update 

  • Maryam looking for help filling out collateral that has been requested 
  • Maryam looking for help filling out scenario wiki content too 
  • Colorado testing proposal first wiki 
  • scenario second wiki 

Test results Api

  • All testing teams are trying to standardize testing results 
  • looks like Kibana is being proposed
  • potentially we will have some integration/migration work to do with whatever tool they decide to use

Test Definitions

  • it’s been a while since we had added new definitions 
  • feel free to suggest review of any of the existing tests 
  • We would like to start running integration test as part of our merge gate for VSPERF 
  • if anyone has any suggestions of tests you would like to see running on the commit gate 
  • thank you Al for adding overlay networking tests 
  • Etherpad Link https://wiki.opnfv.org/display/vsperf/Vswitch+Test+List 
  • We do have a Test framework backlog in Jira 
  • Need to revisit P1 and look at what is realistic for Rel C 


April 13th 2016

Summary of Actions:

  • ACTIONlooking for someone to work with Yardstick on creating a scenario 
  • ACTIONMaryam to get Ana to do a read out on one of our calls, maybe next week?
  • ACTION: Al to add Maciek feedback to slides

Review actions from last week 

  • CI not up and running hardware due 22nd April
  • ACTIONfor folks to review open reviews

Dev Update

  • SR-IOV is under review 
  • Pod 3 CI is down having issues with Ixia licenses 
  • should be resolved in the next few days 

Moongen update Bill

  • Current focus is on integration

Xena updates Christian

  • Throughput patch is already merged.
  • Back 2 back patch pushed today.
  • More testing and JIRA bug tracking.

Yardstick update

  • Maryam met with the Yardstick team 
  • Maryam has put together a gold deck for VSPERF link to VSPERF WIKI where golden deck is located https://wiki.opnfv.org/display/vsperf/VSperf+Home
  • Maryam will be going through the deck with Yardstick team tomorrow 
  • Maciek would like to use presentation as reference for Fdio  

Al presented vm to vm methodology update

  • ACTION: Al to add Maciek feedback to slides

April 6th 2016

ACTION: Maryam try to join Yardstick call to help identify work that needs to be completed for integration - DONE

ACTION: Mars to align with Apex team and determine the work that needs to complete and add stories to JIRA - DONE

Mars has spoken to the Apex Team and they are willing to support this activity.

ACTION: investigate parallelizing the nightly build. Add a JIRA issue.

Dev update

  • SR-IOV VM loopback test implemented for vsperf.

Traffic gen update

Moongen: 

  • Running tests that last typically last between 60-120 seconds, 3-5mins.
  • Dialled down the acceptable packet loss to 1% as there were some inconsistent results.

Xena

Reworking throughput patch in gerrit.

Doing so additional testing.

B2B - need to discuss what values to return for this test in addition to the back to back value.

Spirent

Cherry picking patch to latest master. 

Readouts:

vsperf ci.

IETF 95 update.

Mar 30th 2016

Action Items

ACTION: Maryam circle back with Jack on status of Pod 3

ACTION: All go in and have a look at the open reviews

ACTION: Simona to locate Colorado label and modify existing tickets that are going to be part of Release C

ACTION: Simona to create Colorado Wiki and add content from etherpad

ACTION Maryam need to schedule meetings with other projects 

ACTION: Maryam try to join Yardstick call to help identify work that needs to be completed for integration

ACTION Simona start re-organizing JIRA around the features in our etherpad

ACTION: Mars to align with Apex team and determine the work that needs to complete and add stories to JIRA 

 

Review actions from March 23rd

Traffice Gen Updates

Moongen update:

  • no new updates for the last week 
  • have been running moongen with new scripts and its looking robust 

Spirent update:

  • Sridhar not on present on the call 
  • patches submitted for review 

Xena update:

  • Update sent by email from Christian: Packing up first patch for 2544 throughput methods implementation. Just had other priorities this week so I will have this ready soon.
  • Working on some cleanup of the socket API methods (continuous / burst). I had to debug some socket issues where sockets were not closing correctly causing exceptions to be raised on __exit__ calls.
  • Trying to debug an issue with continuous traffic and 64 byte throughput with 100% transmit flow. The Xena stops recording receive stats and requires me to reboot my bare metal system to recover. Still looking into this.

Dev Update

  • OVS 2.5 merged in requested patches to fix the vhost user issues 
  • New RFC 5481 scripts from IXIA - still open needs to be tested 
  • Martin having issues with SR/IOV test ,VM seems to be dropping packets 
  • Christian added if some contributors to the project have time I would appreciate feedback on VSPERF-256 for the documentation issues I see related to correct hugepage setup.

Rel C Update

  • Dave McBride new Release manager sent out specific requirements re: MS1 which is due April 8th
  • We discussed our project scope for C release and identified dependencies on the following projects  Moongen, Yardstick, and Fuel
  • Mars indicated that he would like to work on the Apex installer or Compass 
  • ACTION: Mars to align with Apex team and determine the work that needs to complete and add stories to JIRA 
  • Possible work across all the testing teams streamlining the results DB 

PTL role 

Mar 23rd 2016

Actions:

ACTION: Simona to create Jira ticket to create best practice guide when upgrading to new version of VSPERF – Complete (Jira ticket VSPERF 269)

ACTION: Al to send out email with details to join the IETF benchmarking working group – Complete

ACTION: Maryam add to the agenda for next week quick readout re: POD how to get access to the pod etc for new committers 


Rel C updates 

  • Maryam commented that we can start VSPERF integration without SW traffic gen 
  • MS1 is due April 8th  agreed to dedicate next week’s call to MS1 Planning

OPNFV Hackfest

  • Joint meeting with OVS4NFV.
  • Opportunity to help OVS4NFV testing,
  • Blocker – POD 3.

POD3

  • Need OpenPGP credentials from Jack to access

Dev Updates 

  • VIFO support added for VSPERF 
  • Integrated test support for OVS 

Traffic Gen updates 

Xena update:

  • ready to go, just need to do some final testing 
  • Testing continuous traffic.
  • Working on how to push the patches upstream and what to do with 2544.exe
  • Kudos to Christian and Dan

IETF draft 

  • Thanks to AL for submitting the draft 
  • now need to work on the presentation 
  • Maciek and Pierre agreed to join the working group

Establish communication between Yardstick and VSPERF frameworks 

  • Maryam shared discussion with Yardstick, questions raised and answers (see attached)
  • we need to consider our questions for the yardstick team, if anyone has any suggestions please send on 

PTL role for VSPERF 

  • Maryam stepping down and is looking to the community if anyone is interested in taking up the opportunity to lead the project 

OPNFV Summit

  • OPNFV summit is on in Berlin, if interested in submitting a proposal for VSPERF deadline is 31st March 
  • Maryam suggested perhaps some of the Moongen work would be a good candidate  and the other traffic gens 
  • working on a deck for VSPERF which gives folks an overview and adding new updates/features 

BMWG

  • Maryam to present remotely April 7th.
  • Draft has been moved forward-> call for Adoption to add it to the BMWG charter

Mar 9th 2016

  • Agenda:
  • Overview considerations when bench-marking of Virtual Switches
  • Project Dev Status
  • Traffic Gen Updates
  • Rel C Planning
  • ACTIONS
  • [ACTION] Maryam to add CI overview to agenda, Wednesday 16th March
  • [ACTION] Bill to create an updated installation / usage version of the tool as the one I presented several weeks ago is now obsolete.
  • [ACTION] Simona to check with Maryam if we have a best practice guide when upgrading to new version of VSPERF
  • [ACTION] All please provide input to Rel C planning etherpad or confirm acceptance (see link below to etherpad)
  • [ACTION] Maryam to provide an update next week on help that is being provided by Ixia
  • Benchmarking Overview
  • Al gave an overview of Benchmarking for Virtual Switches
  • Also presented the slides at Test projects - weekly coordination meeting, main aim to have it adopted by the wider community
  • Traffic Gen updates
  • Xena Update
    • Trying to run VSPERF on Redhat plus fix Xena bugs, testing of this starting today
  • Moognen Update
    • Re-write of LUA wrapper done by Andrew more detail on github
    • User guide is currently obsolete, plan to stabilize the new version of Moongen before creating the new user guide, should have something for next week.
    • Changes can be viewed at Andrews github : https://github.com/atheurer/MoonGen/tree/dpdk2.x-opnfv
  • Spirent Update
    • Challenge to update to new VSPERF code', Simona to check with Maryam if we have a best practice guide for upgrading to new version of VSPERF

       

      * Release C updates

  • Indicated our intent to be part of REL C as planned on March 8th
  • Now need to focus on assessing our impacts to other projects such as installers, CI, Infrastructure
  • Please provide input directly to etherpad link: https://etherpad.opnfv.org/p/VSPERF_C_Planning
  • Meeting with IXIA for packet capture was productive, Plan is to move some TCL to python

Mar 2nd 2016

  • Agenda:
  • VPP Overview
  • Project Dev Updates
  • Traffic Gen Updates
  • Test Priority List for Release C
  • ACTIONS
  • [ACTION] Maryam to add CI overview to agenda in two weeks, Wednesday 16th March
  • [ACTION] Bill to create an updated installation / usage version of the tool as the one presented several weeks ago is now obsolete.
  • [ACTION] Simona to send details via email of top Priorities for Rel C - Complete
  • [ACTION] All to review and agree final REL C proposal and or provide feedback for any changes/updates
  • Traffic Gen updates
  • Xena Update
    • working on XML file configuration
  • Moognen Update
    • Well done to Andrew and Bill, fantastic progress has been made in last few weeks.
    • Andrew Theurer has made significant changes to the tool. This is in regards to the Lua wrapper around the DPDK engine, in particular the file opnfv-vsperf.lua which instruments RFC 2544 like testing.
    • The changes include but are not limited to: Multi (concurrent) device support. As many pairs of ports as the user wants -Multi queue support for transmit -Hardware tx method is default.
    • All bugs with hw tx should now be avoided (with multi queue support) -Major clean up and reorganization of code -Test parameters are now in file -no command line arguments -Tx rate is calibrated before testing to never be above requested rate
    • -Uni/bidirectional logic handled in connections[] array -Use log: facility that comes with MoonGen -Too low tx rate detection allows N attempts instead of aborting on first failure
    • The tool is now ported from DPDK 1.71 to DPDK 2.2
    • Changes can be viewed at Andrews github : https://github.com/atheurer/MoonGen/tree/dpdk2.x-opnfv
  • Spirent Update
    • Undecided whether to push two scripts or merge into one, however there potentially could be overlap. Maryam suggested to push scripts as they are and a decision can be made as part of the review.

       

      * Release C updates

  • Discussed milestones and C planning, see link to WIKI for details https://wiki.opnfv.org/releases/releaseplanning
    • MS1 due next week 8th March
  • More servers have been requested to be set up for testing, Maryam to follow up on status next week #link https://wiki.opnfv.org/get_started/intel_pod3

Feb 24th 2016

  • Agenda:
  • Review Action Items from Feb 17th
  • Project Dev Updates
  • Traffic Gen Updates
    • IETF-95 Update
  • Rel B Status
  • Test Priority List for Release C
  • ACTIONS
  • [ACTION] ALL can you please review patches and abandon those that are no longer valid
  • [ACTION] All please notify Maryam or Al if you plan on attending hackfest
  • [ACTION] : Maryam to update Jira with new tickets for Rel C
  • [ACTION] : All Draft tests on etherpad please review and revert back to Maryam and Al with your feedback
  • [ACTION] : All please ensure your Jira tickets that have been completed are closed in JIRA
  • [ACTION] : Bill to advise on mailing list for Moongen
  • Traffic Gen updates
  • Xena Update
    • working on XML file configuration
  • Moognen Update
    • Success in getting 40gb supported, moongen now upgraded to DPDK 2.2
    • Started working on integrating into VSPERF infrastructure
  • Spirent Update
    • Need to fix issues with older version and then update scripts

       

      * ITEF-95 Update

  • Al updated Jira ticket deadline has been extended, Maryam to add some updates to the draft
  • look at possibility of doing remote presentation
  • Release B updates
  • Official External launch announcement March 1st
  • Tag added to stable Brahmaputra branch for VSPERF
  • Release C updates
  • Potential collaboration between OVS and VSPERF on new feature (ability to classify packets and give priority) submitted by AL.
  • Could use VSPERF as test suite to measure the benefit
  • Please send on any agenda items you would like to discuss next week

Feb 17th 2016

  • Agenda:
  • Review Action Items from Feb 10th
  • IETF-95 Update
  • Project Dev Updates
  • Hackfest March
  • Traffic Gen Updates
  • Rel B Update
  • Test Priority List for Release C
  • ACTIONS
  • [ACTION] All can you let Maryam know if you plan on attending hackfest
  • [ACTION] Maryam to add review feedback on test results to the backlog for Rel C
  • Traffic Gen updates
  • Xena Update
    • working on some bugs, having issues with results generation, continue to find issues while testing
  • Moognen Update
    • Fix implemented for 0% and 1% loss, difference of variance is now closer
  • Spirent Update
    • newtestcase.rst, according to the comments in done. I’ll relook at this document next week to enhance it to cover other cases (other attributes of testcase) apart from the traffic generator.
    • Made initial commit on JIRA-VSPERF:237. Will continue to fine tune the scripts.
    • Planning to add scripts for multicast benchmarking (3918). Should create Jira ticket for the same soon.
  • Rel B updates
  • Primary release date on target for Feb 25th
  • Only outstanding item, publish dashboard results
  • Received some good feedback, need to add definition for tests
  • Need to review unstable results, B2B frame test need to define what we are plotting
  • Rename scalability OVS/DPDK tests
 

 

  • Test priority list for release C
  • Reviewed updates and new tests on etherpad https://etherpad.opnfv.org/p/vSwitchTestsDrafts, please review and add feedback
  • Suggested we have a cut off point for review perhaps one week and then we push to gerrit
  • Please send on any agenda items you would like to discuss next week

Feb 10th 2016

  • Agenda
  • Review Action Items from Feb 3rd
  • IETF-95 Update
  • Project Dev Updates
  • Hackfest March 14-15
  • Traffic Gen Updates
  • Rel B Update
  • Test Priority List for Release C
 

 

  • ACTIONS
  • [ACTION] Maryam to investigate MPLS support in OVS.
  • [ACTION] Al to open a JIRA ticket to Update the Internet draft, etc.
  • {ACTION} Al to combine the VM2VM-related tests, then JIRA ticket Priority 1 for Rel C [DONE]
  • IETF-95 Update
  • Need to sync Internet Draft with the list of tests as of Brahmaputra Release
  • Traffic Gen updates
  • Moongen Update
    • made 3 bug fixes to the tool, found missing packets causing tests to fail
  • Spirent update
    • Sridhar not on the call, no update
  • Xena Update:
    • Amit finishing up final integration to meet traffic gen spec
  • Ixia (RFC 5481)Update:
    • made progress on SW upgrade to support packet capture on the Pod in Shannon
    • some progress on PDV calculations for results report.
  • Rel B updates
  • Primary release date pushed to Feb 26th
  • Any project that is ready will go out on Feb 26th, projects that are not ready will go out March 28th
  • Documentation: still open question about the new templates proposed by Chris, room for some flexibility perhaps we can continue to use what we have for this release, as far as we know.
  • Hackfest
  • Will be held in Santa Clara March 14-15th
  • Both Maryam and Al attending, would like to know who else is attending so we can plan demos and breakout sessions
  • Registration is free, link https://www.regonline.com/Register/Checkin.aspx?EventID=1802426
  • It’s a good opportunity to get input from folks that don’t attend project meetings
  • Test priority list for release C
  • Al added Jira tickets for new tests
  • When you combine the VM2VM-related tests, then Priority 1, need wiki update and JIRA ticket
  • Voting for Rel C tests are still welcome
  • If folks are happy with current top 4, please send a mail to Al confirming
  • link to Test Review Scratchpad etherpad: https://etherpad.opnfv.org/p/vSwitchTestsDrafts, welcome test contributions for match action and VM2VM.
  • New Draft Test at the Scratchpad (etherpad):
    • Overlay Network performance described in general for VXLAN, GRE, GENEVE
    • feedback that this approach works, saves time in review.
  • link to VM2VM Presentation https://wiki.opnfv.org/wiki/vswitch_readouts?&#vsperf_project_readouts

Feb 3rd 2016

  • Agenda:
  • Traffic Gen update.
  • Dev update.
  • Test Priority discussion.
  • VSPERF results.
  • VM2VM discussion
  • ACTIONS
  • [ACTION] Maryam to create wiki for tests.
  • [ACTION] Maryam to investigate MPLS support in OVS.
  • Test Generator Update
  • Xena: Proving out implemented methods and working on/around Xena incompatibility with Python 3.
  • Spirent: Implemented changes to allow TestCenter to execute using the REST APIs, removing the need for a local client.
  • General dev updates
  • Final Report or release B is being worked on.
  • Reporting % Frameloss from the traffic gen - support in progress - PLEASE NOTE (Spirent/Xena integrators) watch out for this patch.

Jan 27th 2016

  • Agenda:
  • Moongen Overview
  • Project Dev Updates
  • Traffic Fen Updates
  • Test Priority List for Release C
  • Review Open Actions
  • ACTIONS
  • [ACTION] Maryam to post test results report
  • [ACTION] Maryam to post list of integration tests/tests from OVS developers to ether pad
  • [ACTION] Sridhar add a "how to add a new test case" document
  • [ACTION] All Action all to provide feedback to Al on Top 6 tests for Rel C, deadline pushed out to Feb 6th https://etherpad.opnfv.org/p/vSwitchPerfNewTestList
  • Moongen Overview
  • Notes taken during the presentation
  • Red Hat have integrated Moongen with an internal test framework. but mostly Bill's repo mirrors Moongen @ github, A detailed README exists on github
  • Additional setup information can be found @ https://raw.githubusercontent.com/bmichalo/MoonGen/opnfv/MoonGenSetUp.html
  • github repo can be found @ https://github.com/bmichalo/MoonGen/tree/opnfv, the relevant branch for vsperf from Bill's repo is the opnfv branch
  • Today Moongen uses DPDK 1.7.1, it would be a "nice to have" to upgrade to DPDK 2.2 but this is up to the Moongen maintainers.
  • Bill can't run Moongen with DPDK 2.1 or DPDK 2.2, but this isn't an issue for us, we should just use Moongen as a standalone traffic generator
  • Examples/opnfv-vsperf.lua is a script that Andrew and Bill wrote modelled on other scripts in the example dir and it runs an RFC2544 test
  • The script does a binary search, to find the throughput value for the specified traffic loss %
  • Bill added a validation run to the script to ensure that the throughput value found in the binary search is indeed the x% frameloss throughput value
  • If validation fails the script continues to run the binary search until the validation phase passes
  • Traffic Gen updates
  • Spirent update
    • Sridhar should be pushing some patches to vsperf will use the spirent REST API.
    • Sridhar to follow up on Jira access with Aric
    • Sridhar has created a document for adding a new test case
    • Sridhar to migrate the ppt to rst and push to gerrit
  • Xena Update:
    • Xena code is written - it just needs to be tested
    • Christian is going to test it - working on installing vsperf

Jan 20th 2016

  • Agenda:
  • Moongen Overview
  • Project Dev Updates
  • Traffic Gen Updates
  • Test Priority List for Release C
 

 

  • ACTIONS
  • [ACTION] Bill Setup separate BlueJeans call for Moongen overview
  • [ACTION] Maryam to post traffic gen guide to Gerrit as a draft
  • [ACTION] Maryam to post list of integration tests from OVS developers to ether pad
  • [ACTION] Maryam and Sridhar add a "how to add a new test case" document
  • [ACTION] All to provide feedback to Al on Top 6 tests to start defining priorities for Release C
  • Moongen Overview
  • Moved out to next week, due to presentation issue
  • Bill action to setup separate bluejeans session next week
  • Traffic Gen updates
  • Spirent update
    • Brian has reviewed the documentation and provided feedback through email
    • Sridhar working on enabling b2b and other RFC2544 tests
    • Sridhar working on RFC3918 – implementation patches will be posted soon
    • Action: action add a "how to add a new test case" document – Sridhar and Maryam
  • Xena Update:
    • Action: Maryam to post traffic gen guide to gerrit as a draft
    • Action: Maryam To post list of integration tests/tests from OVS developers to ether pad
    • Xena developers working on burst and cont traffic methods... there are some missing return parameters that Xena needs to enable access to
  • Test priority list for release C
  • Action all to provide feedback to Al on Top 6 tests to start defining priorities for Release C

Jan 13th 2016

  • Agenda:
  • Readout From Ralph - is your SDN switch ready for the real world ?
  • CI
  • Test Prioritisation
  • Moongen Integration with VSPERF
  • Traffic Gen

*ACTIONS

  • [ACTION] Simona include Al’s bugs in next Sprint
  • [ACTION] All Review Test Prioritisation list and pick top 6 and email to Al

*Milestone E

  • Test Generator Integration
  • Moongen integration with vsperf
    • Bill will go through installation tutorial of Moongen at next week’s meeting
      • User guide it 2/3rds complete, available for folks can try it out
  • Sridhar on Spirent Integration: Sridhar has written 2544 script, more coming
    • RFC 2544 b2b test implemented and under review
  • Traffic gen Xena: update from Rick good progress integrating RFC2544 test, next looking at Latency, jitter and frameloss testing, pick-up next Monday again

-

Jan 6th 2016

  • Agenda:
  • Nightly Build
  • Milestone E
  • Test Generator Integration
  • ACTIONS
  • [ACTION] Al & Brian have volunteered to review user Guides, and other docs (design docs and Release docs, ignore LTD Spec)
  • [ACTION] Pierre to help with upgrading firmware (RFC 5481)
  • [ACTION] Maryam to provide link to list of tests for next release
  • [ACTION] AL to prioritise tests by working with community and work test spec development
  • Milestone E
  • Successfully reached and reported to Debra, link https://wiki.opnfv.org/releases/brahmaputra/release_plan/vsperf_milestone_e_report
    • Any Questions on the report please send them to the mailing list
  • Creation of Stable Release B Branch deadline tomorrow 7 Jan
  • New defined way of working on Stable branch - link https://wiki.opnfv.org/releases/stablebranch
  • Again, any questions please send to mailing list on Stable Branch work methods
  • Next Milestone, Jan 12, no changes allowed after this date
  • Final Milestone - Jan 19 - All Documentation must be available for Review, link http://artifacts.opnfv.org/vswitchperf/docs/all/index.html
  • Need volunteers to review user Guides, and other docs (design docs and Release docs, ignore LTD Spec)- Al and Brian have agreed to complete this task see AR above
  • Review means read through and consistency checks, pick up JIRA tickets
  • Jan 26, All Docs complete, All tests will be conducted on the various Pods on stable Releases, till Feb 1.
  • Test Generator Integration
  • New Epic on Moongen with Bill, he will come up with Documentation for Using MoonGen
  • New work on Dummy Traffic Gen will help with MoonGen and Xena
  • Documentation on how to integrate a traffic gen with vsperf will help with MoonGen and Xena integration.
  • Amit on Xena: implemented part of vsperf traffic gen, need to add connect disconnect
  • Sridhar on Spirent Integration: RFC 2544 and RFC3849 scripts to support testing with Spirent
  • Test Prioritization and Definition – Maryam to provide link to existing link<MT> Link https://etherpad.opnfv.org/p/vSwitchPerfNewTestList
  • RFC 5481 scripts
    • risk in disturbing Pod 3, Firmware installed locally, Pierre has agreed to help with upgrade of the firmware
    • Details on Percentile calculation for RFC 5481 nailed down - one preferred method in RFC 2330 will be used

Minutes also available at https://wiki.opnfv.org/meetings/vsperf

Dec 2nd 2015

  • Agenda:
  • Project dev updates
  • Traffic Gen updates
  • CI update
  • Inactive committers
  • OPNFV summit
  • PDV to be collected in RFC5481 format and IXIA
  • ACTIONS
  • [ACTION] Add vloop_vnf to documentation as the reference image to use with vsperf.
  • [ACTION] Maryam to establish an initial roadmap including this capability as a feature.
  • [ACTION] Maryam to establish an initial test priority list to review with the community.
  • [ACTION] Maryam to include path finding and integration activity for a reference pipeline to the roadmap.
  • [ACTION] Gene to update vloop_vnf https://jira.opnfv.org/browse/VSPERF-133 and push new image to artifacts.opnfv.org
  • [ACTION] Gene to migrate our commit gate checks and establish a basic Jenkins job that runs nightly on POD3.
  • Project dev updates
  • OPNFV test dashboard integration is well underway
  • vloop_vnf integration with vsperf complete – need some updates to the vloop_vnf image and need to push the updated image to artifacts.opnfv.org
  • Traffic Gen updates
  • Xena: Amit mentioned that they’ve had couple of discussions about how to use existing xena code in vsperf. Amit has written small python scripts to perform individual tasks, Now the next step would be to use these small scripts in vsperf template. Amit is having a working session with Dan tomorrow that will hopefully result in having Xena take over the integration effort.
  • Moongen:
    • Looking for a volunteer to help Andrew with this.
    • [ACTION] Maryam and Billy to help facilitate the integration work.
  • CI update
  • [ACTION] Gene to send on update
  • CI Nightly Build Steps:
    • Clone, install and build vsperf dependencies.
    • check if IxNet server is running and calls basic phy2phy_tput testcase
    • Run all test cases that are registered with the OPNFV test Dashboard and push the results to the dashboard.
  • Inactive committers
  • I’ve kicked off the process to remove inactive committers, so we can promote contributors to committer status.
  • PDV to be collected in RFC5481 format and IXIA
  • Al has looked at them.
  • tcl script and a python file. it looks like the tcl script needs to be dropped into the right place within vsperf and tying in the python script into vsperf.
  • Maryam will look at integrating these scripts in the next sprint

Nov 18th 2015

Agenda

  • Rel B requirements for installers
  • Project Updates - Xena, Spirent, Moongen
  • DUT Topology discussion.

DUT Topology discussion

  • Agreed at OPNFV conference to remain with current naming.

Rel B requirements

  • Compass is a possibility for installer for rel-C.
  • For Rel-B it is possible to have a simple code install on a "clean base" host - like a git clone or untar of the vsperf source code.

Dev updates

  • Can now use Spirent TestCentre traffic generator for RFC2544.
  • More tests/scripts to follow
  • [ACTION] Brian Castelli will add Jira tickets for these tasks
  • Still trying to get lic servers running on POD3

vnf_loop vnf

  • work on integrating this in progress

moongen

  • unlikely to be in a position to write vsperf integration code for Rel-B (unless help given)
  • RFC2544 with %loss is working in moongen itself.
  • no cost for licencsing or distribution of same (MIT lic.?)
  • [ACTION] Andrew Th. to verify licenseing requirements
  • moongen is distributed in src. May need to pkgd in order to deploy on distro.

Xena

  • hoping for eng resource to be allocated shortly.
  • [ACTION] billy put on agenda for 2wks from now for Ralph Daniels (Sprient) to present on slide deck (on vsperf ML) on OpenFlow vSwitch testing.
  • The slide deck for Ralph's presentation was sent to the mailing list on 2-Nov
 

AOB

  • We should try some push-back on Rel-B installer requirements and target Rel-C as originally committed.

Nov 11th 2015

  • No meeting due to OPNFV conference.

Nov 4th 2015

  • [ACTION] for ALL: Please review VSPERF feature wish list email and send on any features that we should include for next release.

Note no meeting November 11th

AGENDA for Next Meeting:

  • 20 mins DUT Topology discussion.
  • 20 mins OPNFV Summit update
  • 10 mins project updates.
  • Discussion around unified/common denominator API for Traffic Gens
    • Q: In VSPERF, what is the purpose of the common template(Traffic Item + API), is it to try to get the same results across the different traffic gens or something else?
      • A: The intention was to hide the implementation details and complexity in terms of parameters passed to the traffic gen behind a simple object that could describe the basic parameters (such as packet size, bi/uni directional traffic…) that we want to configure for a test. Today we pass a set of parameters that we want to modify to the generic traffic gen API. They get translated into the appropriate parameter by the underlying traffic gen implementation and all other parameters are set to their defaults.
    • We agreed that we will document the implicit settings of Trafficgens supported by VSPERF in a separate document under docs.
    • We need to standardize the set of input and output to and from traffic gens – Added this to the VSPERF design summit and will bring a proposal to the weekly meeting after the summit.
  • Traffic Gen integration update
    • Xena: Rick has been using the RFC2544 Linux executable. Rick can run the suite itself and run it in an automated env. Targeting the next (non-release B) VSPERF open feature development window for integration.
    • Spirent: Looking for another resource to continue the integration work.
    • Moongen: [ACTION] Andrew to send on update
  • CI Integration
    • [ACTION] Gene to provide the update
  • PDV to be collected in RFC5481 format and IXIA:
    • We received the initial scripts and will spend some time looking at them.
  • Deployment naming discussion
    • To be covered at the design summit.
  • VSPERF breakout session
  • VSPERF feature wish list – REMINDER LOCK DOWN 1st of Dec 2015 is there anything we need today for release B.

Oct 28th 2015

  • AGENDA for November 4th:
    • 20 mins DUT Topology discussion.
    • 20 mins Discussion around unified/common denominator API for Traffic Gens.
    • 10 mins project updates.
    • 10 mins Deployment naming discussion.
    • new tests discussion continuation – next session.
  • Traffic Gen integration update
    • Xena: Rick is working with the ported RFC2544 implementation.
    • Spirent: First set of patches upstreamed to vsperf. Lays the ground work for future tests. Throughput tests is what has been implemented thus far. Brian is also looking at a new solution that makes the implementation even simpler than what it is today.
    • Moongen: Bidi and latency testing have been enabled but not pushed yet.
  • CI Integration
    • [ACTION] Gene to provide the update
  • Documentation integration with RELENG
  • IETF-94 and vsperf bmwg draft
    • Presented an update of our draft at IETF 94 last night – it was well received – separate update email to follow shortly.
  • PDV to be collected in RFC5481 format and IXIA:
    • We received the initial scripts and will spend some time looking at them – could be a good topic for the Group breakout session at the OPNFV design summit.
  • New tests
    • Discussed some possibilities:
      • Scalability Tests adding More VMs in succession and building a profile
      • East-west across the same node [intra node]
      • Overlay Networking Tests: VXLAN/GRE performance testing, encap, decap, encap and decap?
      • (ACLs/Firewall config)Match action performance testing? The cost of the different actions supported by a vSwitch?
    • Some More possibilities to be discussed:
      • Classifying L2, L3 and L4 traffic Profile Tests.
      • Stream/bulk Data transfer "unidirectional stream" performance
      • request & response/transaction rate tests?
      • Performance testing with Mirroring enabled on the switch?
      • TCP Max connections per second, Max # of active sessions, Max transactions per second
      • IPv6??
      • Best of N and Worst of N Tests from Maciek
      • Deactivation tests?
      • vSwitch in QOS mode/load balancer
  • Deployment naming discussion
    • Should we stick with:
      • P2P, PVP, PVVP …
    • Or should we migrate to:
      • T1n – VSWITCH Onboard port Forwarding ( either P2P port forwarding or flow based )
      • T1u – VSWITCH Forwarding with user space L2 forwarding
      • T1v – VSWITCH Forwarding with one VM ( either switch based port forwarding or flow based )
      • T2v – VSwitch Forwarding with two VMs ( either switch based port forwarding or flow based with DNAT on VM1)
  • VSPERF feature wishlist (LOCK DOWN 1st of Dec 2015 is there anything we need from this today?)
    • VLOOP_VNF integration with VSPERF.
    • Single port traffic gen support (today we just use 2 ports)
    • Bypass/SR-IOV setup support.
    • VXLAN/overlay tests (when we have them defined).
    • VM 2 VM tests.
    • L2FWD integration with the VNF as an option instead of test-pmd or Linux Bridge.
    • test-pmd integration as an option for Vanilla OVS VNFs instead of Linux Bridge.
    • a vSwitch/VNF None option to allow us to bypass their configuration and leverage our traffic gen implementations (for Yardstick).
    • IMIX specification as an option to the traffic generator using https://tools.ietf.org/html/draft-morton-bmwg-imix-genome-02
    • pcap replay as an option for traffic gens.
    • Detailed benchmark reporting and graphing.
    • RFC5481 Latency measurements
    • RFC2889 tests implementation

Oct 14th 2015

  • Readouts
    • Morgan Richomme opnfv-test-dashboard
    • Cisco Readout on learnings when characterizing the performance of vSwitches next week. (14th October 2015)
  • Traffic Gen integration update
    • Xena: Sync up with Dan regarding integration work complete. [ACTION] Billy and Dan to send out list of open questions.
    • Spirent: Brian continuing work on integration.
    • Moongen (Andrew T): [ACTION] Andrew to send status.
  • CI Integration
    • [ACTION] Gene to send out more info
  • Documentation integration with RELENG
    • [ACTION] Ram to address Aric’s comments in the review. https://gerrit.opnfv.org/gerrit/#/c/2533/
    • Really good progress made on this front, Ram added all the relevant rst index files and restructuring of the documentation. A big thanks to Ram for looking after this.
  • IETF-94 and vsperf bmwg draft
    • Updated draft submitted to gerrit, it has also been reviewed and pushed. A big thanks to Al for looking after this.

TOPICs not discussed this week.

AGENDA for October 21st:

  • 20mins project updates.
  • 20mins new tests discussion.
  • 10mins Deployment naming discussion
  • 10mins CI discussion.

Oct 7th 2015

 

* JIRA and TOIT/VSPERF development update

  • Features under development
    • PVP and PVVP enablement for Vanilla OVS – in Review
    • PVVP for vhost cuse and user – merged to master
    • Traffic Gen integration update
      • Xena: Initial python development looking at the integration with VSPERF has been kicked off. But there seem to be missing params that could affect the results of a test. [ACTION] Billy and Rick to setup a session to discuss issues & questions.
      • Spirent: Brian has been integrating Spirent with VSPERF and has traffic passing through the DUT. Brian gave a demo of the integration efforts so far. Thanks and great progress Brian!
      • Moongen: MoonGen update (Andrew T): Bidirectional traffic is working, latency measurement has been added, however, latency stats for bidirectional traffic is incorrect (seeing negative numbers). There is also now support to specify the frame size for the test. Changes haven’t been committed to github yet, but be will soon. Also, you can now specify a %frame loss as a pass/fail threshold (default is 0).
  • CI Integration
    • Gene is working on installing the Jenkins Client on the 2nd server attached to the VSPERF DUT in POD3.
  • Documentation integration with RELENG
    • [ACTION] Ram to send on what support/help is needed here to progress this issue/story.
  • IETF-94 and vsperf bmwg draft
    • Al is working through the LTD issues in JIRA to start looking at this.
    • New tests and spec developments should be added to the IETF Draft.
    • A demo would be very well received at the BMWG session.
  • PDV to be collected in RFC5481 format and IXIA:
    • Sync up with IXIA planned.
  • OVS-NFV collaboration
    • Need to investigate possible collaboration points in terms of testing the vswitch.
  • YARDSTICK and VSPERF joint design session
    • Billy installed Yardstick and OpenStack and has been studying the Yardstick framework (it is not using/an extension of Rally).
    • Area for collaboration seems to be to leverage vsperf’s traffic gen integrations for yardstick.
  • Readouts
    • Cisco Readout on learnings when characterizing the performance of vSwitches next week. (14th October 2015)
  • Packet Re-ordering
    • Spirent is compliant with RFC5481
    • Need to check if IXIA is compliant with RFC5481 also.
    • We should enable this as a configuration option for VSPERF. [ACTION] Maryam to add JIRA issue for this (won’t be addressed till post release B).
  • New tests
  • 20 mins scheduled for October 21st vsperf weekly sync up.

Sept 23rd 2015

  • JIRA MILESTONE C 25th Sept 2015.
    • [ACTION] Gene to add any CI stories - In progress. Waiting on POD3 access to install jenkins.
  • [ACTION] Andrew, Brian and Rick to add any traffic gen integration stories they would like to see as part of release B.
    • Brian – DONE.
    • Andrew - Will do.
  • [ACTION] Maryam to update wiki with additional goals for Release B. - In progress
  • [ACTION] Billy supply vsperf status update to RelB planning.
  • TOIT/VSPERF development update
    • Features under development
      • PVP enablement for Vanilla OVS – in progress
      • Sysmetrics patch has been uploaded to gerrit.
      • Frame modification tests implementation – patch up for review for almost 2 weeks, Maryam will push under 1-week rule when she returns.
  • Traffic Gen integration update
    • Xena:
      • Some training complete. Talking to Xena on Monday. Update next week.
  • Spirent:
    • Brian C, RFC2544 running to completion. Needs some work to gather results.
    • [ACTION] Brian C, will investigate possibility of setting up Spirent Traffic Gen VM was on POD3.
  • Moongen:
    • bidirectional functionality enablement in progress
    • Via Bill M, Andrew still working on bidi functionality. Andrew will update next week.
  • CI Integration
    • Proposed triggers:
      • 1. Run pylint on the code.
      • 2. Run a short VSPERF sanity check.
      • Require jenkins to +1 before commit allowed.
    • [ACTION] Maryam/Billy Input from Maryam/Billy on 2. required.
    • [ACTION] Maryam/Billy Input from Maryam/Billy on 1. required.
  • Documentation integration with RELENG
    • Patch posted to gerrit from Aric and Fatih, need to review and add TODOs to JIRA.
    • Gene thinks further action on this is blocked on pylint and vsperf sanity check above.
    • [ACTION] Gene to find out from RELENG is .jjb files should be in vsperf or releng repo.
  • PDV to be collected in RFC5481 format and IXIA:
    • Do we need to define a loss threshold?
    • ACTION: Al M. Add to LTD Test Definition for a test such as 'Re-ordering at the throughput level. Simple, Pass/Fail test no-reordering or there was some re-ordering'.
    • Via Al M, Expecting script soon from Ixia to measure PDV.
    • Via Gurpreet. Spirent has a re-ordering detection/measurement built-in to other tests such as it's RFC2544 Throughput.
    • ACTION Gurpreet will check if this feature is compliant to a specific RFC.
    • ACTION Billy add re-ordering to next weeks agenda.
  • POD 3 update
    • [ACTION] Brian C to verify VPN access via second server which has been hooked up back to back with the server in POD 3 in preparation for Moongen installation.
  • VSPERF Release B participation
    • Profile test and RFC2544 reset test will need to be pushed to stretch goals. [ACTION] Wiki to be updated
  • OVS-NFV collaboration
    • Need to investigate possible collaboration points in terms of testing the vswitch.
    • Would be good to ensure that dpdk-enabled ovs deployed in rel-b is testable with vsperf.
  • YARDSTICK and VSPERF joint design session
    • [ACTION] Billy and Maryam to ramp on Yardsticks framework and analyse the opportunity for collaboration.
    • Meeting provisionally set for the 30th of Sept. – More details to follow.

Sept 15th 2015

  • [ACTION] Maryam to follow up with Trevor about setting up a server where Moongen can run connected back to back to the DUT – Done a board has been setup back to back with our server in POD3
  • [ACTION] Maryam to post wiki instructions on log in details to the new server.
  • [ACTION] Billy - TOIT High Level Design wiki addition to https://wiki.opnfv.org/vsperf/test_methodology - In progress
  • [ACTION] Al - Test Spec Overview wiki addition to https://wiki.opnfv.org/wiki/ltd_test_spec_overview - In progress, only new tests left to go in.
  • [ACTION] Maryam to invite Morgan to give an overview of the testing dashboard at the vsperf weekly.
  • JIRA MILESTONE C 25th Sept 2015.
    • [ACTION] Maryam and Billy to add any generic/feature stories. - In progress
    • [ACTION] Gene to add any CI stories
    • [ACTION] Al to add any spec update stories that we think we need to lock down for release B - In progress
    • [ACTION] Andrew, Brian and Rick to add any traffic gen integration stories they would like to see as part of release B.
      • Brian – DONE.
      • Maryam to send out JIRA instructions - Done sent out by Al, Thanks Al.
      • [ACTION] Folks make sure you have permissions to create and progress a ticket, if there are any issues, ping me and the linux helpdesk.
    • [ACTION] Maryam to update wiki with additional goals for Release B. - In progress
  • TOIT/VSPERF development update
    • Features under development
      • PVP enablement for Vanilla OVS – in progress
      • update sysmetrics implementation – in progress – patches should be up for review soon.
      • Frame modification tests implementation – patch up for review.
  • Traffic Gen integration update
    • Xena:
      • Amit is in the process of familiarizing himself with the Xena RFC2544 program and Rick and Amit are planning a follow up training session with Dan from Xena to review the new Linux-based Xena2544 program.
    • Spirent:
      • Created the Jira ticket for Spirent integration.
      • Ported setup to the Spirent VCT POD.
    • Moongen: bidirectional functionality enablement in progress

       

      * CI Integration

  • CI Plan – Gene updated the wiki.
  • In terms of what kind of test we want to run as part of our commit gate, it looks like some verification in terms of pylint/ ReSTructured text validation as well as one of the simple performance tests. [ACTION] update JIRA with the tests we’d like to start running as part of our CI.
  • We need to figure out the triggers.
  • Need to run pylint on the codebase as is - [ACTION] Billy to submit JIRA ticket for this.
  • Documentation integration with RELENG
    • Patch posted to gerrit from Aric and Fatih, need to review and add TODOs to JIRA.
    • [RECOGNITION] Thanks Aric and Fatih for helping us kick start this.
  • PDV to be collected in RFC5481 format and IXIA:
    • Do we need to define a loss threshold? We do need to define a threshold.
    • Do we need to look at or take re-ordering into account? I don’t assume so because we aren’t calculating IPDV, we are calculating PDV? In the case of PDV this is correct. We should take this into account when
  • POD 3 update
    • Second server has been hooked up back to back with the server in POD 3 in preparation for Moongen installation.
    • [RECOGNITION] Thanks Michael and Trevor for completing this setup.
    • Will need some support to get moongen up and running on the second server.
    • Traffc gen VNF will also go here.
    • [ACTION] Maryam to update POD wiki with new server login info.
  • VSPERF Release B participation
    • Profile test and RFC2544 reset test will need to be pushed to stretch goals. [ACTION] Wiki to be updated
  • OVS-NFV collaboration
    • Need to investigate possible collaboration points in terms of testing the vswitch.
  • New tests
    • TODO – next meeting
  • YARDSTICK and VSPERF joint design session
    • [ACTION] Billy and Maryam to ramp on Yardsticks framework and analyse the opportunity for collaboration.
    • Meeting provisionally set for the 30th of Sept. – More details to follow.

FYI Folks I’m out of office next week, as such Billy will be leading the meeting next week.

Sept 9th 2015

  • [ACTION] Maryam to follow up with Aric about JIRA emails/updated to committers and contributors Status: In Progress, Pinged the helpdesk. Issue of progressing an issue has been resolved for Gene, Al to try to progress a story before we close.
  • [ACTION] Al to updated the LTD to discuss sampling and details of PDV – Done.
  • [ACTION] Maryam to follow up with Trevor about setting up a server where moongen can run connected back to back to the DUT – In Progress We can setup a board back to back with the DUT, will be kicked off in the next couple of days.
  • [ACTION] Billy - TOIT High Level Design wiki addition to https://wiki.opnfv.org/vsperf/test_methodology - In progress
  • [ACTION] Al - Test Spec Overview wiki addition to https://wiki.opnfv.org/vsperf/test_methodology - In progress
  • JIRA
    • [ACTION] Maryam and Billy to add any generic/feature stories.
    • [ACTION] Gene to add any CI stories
    • [ACTION] Al to add any spec update stories that we think we need to lock down for release B.
    • [ACTION] Andrew, Brian and Rick to add any traffic gen integration stories they would like to see as part of release B.
    • [ACTION] Maryam to update wiki with additional goals for Release B.
  • TOIT/VSPERF development update
    • Features under development
      • PVP enablement for Vanilla OVS – in progress
      • update sysmetrics implementation – in progress – patches should be up for review soon.
      • Frame modification tests implementation – in progress – patches should be up for review soon.
  • Traffic Gen integration update
    • Xena: Migration of RFC2544 exe to Linux executable has been completed.
    • Spirent:
      • Ramp on building and configuration complete.
      • Brian is going to migrate his work to the Spirent POD in Sunnyvale.
      • Brian also set the Goal of doing a demo next week (No Pressure (tongue)).
    • Any traffic gen integration support should be directed to the tech-discuss mailing list with the VSPERF tag.
    • Gene suggested setting up a traffic gen working group for folks who are trying to integrate a traffic generator with VSPERF. If people are interested in participating please reply to the minutes [ACTION] Gene to send out some more info.
  • CI Integration
    • There was a missing link for the PVVP deployment scenario, L2FWD requires DNAT - -> DONE – merged to master
    • CI Plan – Gene will publish soon.
  • Documentation integration with RELENG
    • On hold, as a new tool chain is being deployed for OPNFV projects.
    • We should consider adding pre-commit gates that run some ReSTructured text validation whenever the rst files are modified – [ACTION] Talk to Fatih about this
  • PDV to be collected in RFC5481 format and IXIA:
    • Good analysis put together by Billy to advise Vendors on this, additional comments provided by Al. http://lists.opnfv.org/pipermail/opnfv-tech-discuss/2015-September/004900.html
    • L2FWD module also collects stats, so we can monitor what’s going on in the VM.
    • Looking at the packet captures is a good first step towards building the right algorithms to capture PDV. The Algorithms should be built to do PDV calculations on the fly and present the results in histograms.
  • VSPERF Release B participation
    • [ACTION] Maryam to add a summary feature table to the wiki. – DONE – Added goals, time plan.
    • JIRA MUST be updated with all our highlevel stories for release B, by COB Friday.
  • New tests
    • TODO – next meeting
  • POD 3 update
  • We should be able to hook up a server back to back with our DUT so that we can setup Moongen for the POD.

Sept 2nd 2015

  • [ACTION] Maryam to follow up with Aric about JIRA emails/updated to committers and contributors Status: Open, Pinged Aric, waiting for response. Will ping him again and CC Gene and Al
  • [ACTION] Al to updated the LTD to discuss sampling and details of PDV – In progress.
  • [ACTION] Maryam to follow up with Trevor about setting up a server where moongen can run connected back to back to the DUT - Open
  • [ACTION] Billy - TOIT High Level Design wiki addition to https://wiki.opnfv.org/vsperf/test_methodology - In progress
  • [ACTION] Al - Test Spec Overview wiki addition to https://wiki.opnfv.org/vsperf/test_methodology - In progress
  • [ACTION] Volunteer for Test Projects status update tomorrow – to give status update for vsperf. – Al Volunteered, Thank you Al.
  • TOIT/VSPERF development update
    • Features under development
      • Affinitization for Vanilla OVS – best without affinitization
      • PVP enablement for Vanilla OVS – in progress
      • update sysmetrics implementation – on hold
      • Frame modification tests implementation – in progress

         

        * Traffic Gen integration update

  • Xena: RFC test was successfully deployed under Linux. [ACTION] Dan to give a demo next week.
  • Moongen:
    • Andrew tested 40Gb adapter and it initializes successfully but sw rate control for the 40Gb NIC still needs to be implemented for the adapter. Moongen typically uses HW/SW rate control.
    • Andrew experimented with the latency measurements but this has not been integrated with the RFC2544 script yet.
    • bidirectional traffic generation is also a feature that needs to be looked at.
  • Spirent: environment has been setup and ramp up on the scripts is in progress.
  • Gene suggested setting up a traffic gen working group for folks who are trying to integrate a traffic generator with VSPERF. If people are interested in participating please reply to the minutes [ACTION] Gene to send out some more info.
  • CI Integration
    • There was a missing link for the PVVP deployment scenario, L2FWD requires DNAT - -> JIRA ticket added for this, code to be submitted soon.
    • The CI plan to be published soon and some commits not long after.
  • New tests
    • We need to start addressing some of the testing gaps in our test specification and look at new deployment scenarios and any other test categories that we need to develop tests for, either from a standard or from a relevant use case.
    • I implore the VSPERF community to start adding tests to the test_spec and pushing to gerrit for review. [ACTION] VSPERF Community. Tests can be reviewed or discussed at the weekly call.
    • This is our current list of tests:
      • Throughput tests
        1. Test ID: LTD.Throughput.RFC2544.PacketLossRatio
        2. Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification
        3. Test ID: LTD.Throughput.RFC2544.Profile
        4. Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime
        5. Test ID: LTD.Throughput.RFC2544.BackToBackFrames
        6. Test ID: LTD.Throughput.RFC2889.Soak
        7. Test ID: LTD.Throughput.RFC2889.SoakFrameModification
        8. Test ID: LTD.Throughput.RFC6201.ResetTime
        9. Test ID: LTD.Throughput.RFC2889.MaxForwardingRate
        10. Test ID: LTD.Throughput.RFC2889.ForwardPressure
        11. Test ID: LTD.Throughput.RFC2889.AddressCachingCapacity ? [ACTION] Maryam add a JIRA ticket to define a new category for this test.
        12. Test ID: LTD.Throughput.RFC2889.AddressLearningRate ? [ACTION] Maryam add a JIRA ticket to define a new category for this test.
        13. Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering ? [ACTION] Maryam add a JIRA ticket to define a new category for this test.
        14. Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding ? [ACTION] Maryam add a JIRA ticket to define a new category for this test.
    • Packet Latency tests
      1. Test ID: LTD.PacketLatency.InitialPacketProcessingLatency
      2. Test ID: LTD.PacketDelayVariation.RFC3393.Soak
    • Scalability tests
      1. Test ID: LTD.Scalability.RFC2544.0PacketLoss
      2. Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability
    • Coupling between control path and datapath Tests
      1. Test ID: LTD.CPDPCouplingFlowAddition
    • CPU and memory consumption
      1. Test ID: LTD.CPU.RFC2544.0PacketLoss
  • Documentation integration with RELENG
    • Ram made some great progress here. The bash scripts and yml files have been setup and configured, and some testing is underway. More to follow next week.
  • PDV to be collected in RFC5481 format and IXIA:

August 26th 2015

  • [ACTION] Maryam to follow up with Aric about JIRA emails/updated to committers and contributors Status: Open, Pinged Aric, waiting for response. Will ping him again and CC Gene and Al
  • [ACTION] Al to updated the LTD to discuss sampling and details of PDV.
  • [ACTION] Maryam to setup a POD3 reservation wiki. Status: Closed.
  • TOIT/VSPERF development update
    • Features under development
      • PVP vhost-cuse merged to master
      • CPU and memory stress tests – merged to master.
      • Patch to affinitize DPDK_OVS threads – merged to master
      • Bug fix for pkt_size and duration selection from command line – merged to master
      • Migration of all documentation to docs/ and conversion to ReStructuredText – merged to master. Next steps: integration with RELENG, TV is looking at this
      • PVP enablement for Vanilla OVS – in progress
      • Affinitization for Vanilla OVS – in progress
      • update sysmetrics implementation – in progress
      • Frame modification tests implementation on hold till Billy returns..
    • Traffic Gen integration update
      • Xena: Rick is working on validating the workaround Dan provided for testing RFC2544 through th sshd server.
      • Moongen: author started adding support for 40Gb Intel NICs Xl710, and Andrew is working on adding the latency aspect to the RFC2544 script.
        • [ACTION] Maryam to follow up with Trevor about setting up a server where moongen can run connected back to back to the DUT
      • Spirent: New contributor onboard – Brian Castelli, welcome Brian.
    • CI Integration – In progress.
      • Setting up 2 separate instances back to back this week and running vsperf locally.
      • looking at integrating the Jenkins jjb with toit.
      • Gene setup a test methodology wiki https://wiki.opnfv.org/vsperf/test_methodology it’s a WIP.
      • Gene added the diagrams with Lucid Chart, an open source online service. The documents are public and can be exported in Visio format if additional editing is required.
      • TOIT High Level Design – [ACTION] Billy
      • Test Spec Overview – [ACTION] Al
  • PDV to be collected in RFC5481 format and IXIA:
    • Firmware on IXIA chassis attached to POD3 has been upgraded.
    • the POD3 TCL clients on the server have been upgraded and tested – previous tests pass.
    • PDV within a single flow is what we are looking to collect. We need an idea of the delay distribution to determine how best to apply the stats. We will probably have to play around with the bucket settings to see how the distributions are affected. This will be documented by Al in the LTD.
  • POD 3 update
  • POD 3 Scheduling – Booking wiki Setup https://wiki.opnfv.org/wiki/pod3_booking_calendar
  • Only VSPERF contributors and committers should be requesting access to the POD to run vsperf tests.
  • The POD is not setup for performance testing right now, it’s a validation environment – will need to tweak BIOS and grub configuration as well as RAM on the system to set it up for performance.
  • YARDSTICK and VSPERF invite for a joint design session (Meeing Invitation attached)
    • When: August 31st, UTC 14h00
    • Goal: Establishing communication between YARDSTICK and VSPERF frameworks
  • [pharos][functest][yardstick][qtip] Weekly Test Project Sync and Inform
    • VSPERF Demo 27th August 2015
  • VSPERF Release B participation

August 19th 2015

  • Recognitions/Shout outs
    • Al Morton, Pierre Lynch, Bogdan Tenea , and Trevor Cooper for driving and supporting the RFC5481 PDV collection with IXIA.
  • Actions
    • [ACTION] Al update LTD and IETF draft with the correct terminology in terms of throughput/maximum forwarding rate for benchmarking methodology. Status: Closed,
    • [ACTION] Gene to share abstracts for OPNFV Summit/talk names – Status: Closed, abstract in the minutes below.
    • [ACTION] Maryam PVP Demo to be setup on POD 3 for people to be able to run – Status: Closed, Please see POD3 update below
    • [ACTION] Maryam to follow up with Aric about JIRA emails/updated to committers and contributors Status: Open
  • TOIT/VSPERF development update
    • Features under development
      • PVP vhost-user merged to master.
      • RFC 2544 scalability test Merged to master.
      • PVP vhost-cuse will be pushed to gerrit this week.
      • Frame modification tests implementation on hold till Billy returns..
      • CPU and memory stress tests – tests to be pushed to gerrit this week.
      • Migration of all documentation to docs/ and conversion to ReStructuredText - pushed to gerrit for review. Next steps: integration with RELENG
    • CI Integration – In progress, Gene is looking at this.
  • PDV to be collected in RFC5481 format and IXIA:
    • Details of the methodology to orchestrate the time stamping has been locked down: start and stop the packet capture from the TCL scripts, then we should be able to extract the pcap file and do some post processing on it. We will need some help from IXIA on interpreting the time stamps but after that the calculation of PDV should be straight forward.
  • YARDSTICK and VSPERF invite for a joint design session (Meeing Invitation attached)
    • When: August 31st, UTC 14h00
    • Goal: Establishing communication between YARDSTICK and VSPERF frameworks
  • OPNFV Summit Call for Participation Gene’s Abstracts
    • Test Automation with OPNFV: Test Automation with OPNFV ( Gene Snider, Huawei Inc. ) - Leveraging the assorted test infrastructures developed by the OPNFV can be quite challenging. This talk will discuss various components of the OPNFV test infrastructure and their deployment in house by OPNFV members. Topics to be covered will include an overview of Gerrit and Jenkins, the use of Jenkins Job Builder to manage automation and how to create and use a virtual machine infrastructure for OPNFV testing. Initially this is intended to include the OCTOPUS and VSPERF projects, the functest project will be covered to the extent that it is integrated with the releng process.
    • Creating VNFs with vmbuilder: Creating VNFs with vmbuilder ( Gene Snider, Huawei Inc. ) - The process of creating a minimum bootable runtime image to provide virtual network functions is not well documented or well understood. In this presentation Gene Snider will present a step by step tutorial on creating a minimal virtual machine suitable for hosting VNF solutions. This presentation will cover scriptable solutions for building and packaging custom VNF solutions.
  • Readout/discussion this week: VSPERF Release B participation: slides and plan can be found here https://wiki.opnfv.org/wiki/vswitch_release_plan
    • Brahmaputra target release date for February 2nd, 2016
    • Please review the backlog to see if there are stories that are missing that need to be added in.

August 12th 2015

  • Actions
    • [ACTION] Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework --> WIP, Gene is looking at this
    • [ACTION] Gene to share abstracts for OPNFV Summit/talk names.
    • PDV to be collected in RFC5481 format - investigate if it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators - Status: Open – See below
      • [ACTION] Maryam to get the model of the IXIA chassis connected to POD3 – Closed. Chassis is Optixia XM2, Version: IxNetwork 7.30.917.27 GA, IxOS 6.70.1050.16 GA .
      • [ACTION] Andrew Theurer to check availability for a Moongen Demo/readout – closed Demo/Readout August 12th 2015
    • [ACTION] Maryam PVP Demo to be setup on POD 3 for people to be able to run – In progress
  • TOIT/VSPERF development update
    • Features under development
      • PVP vhost-user reworked and pushed to gerrit.
      • PVP vhost-cuse will be pushed to gerrit this week.
      • Vanilla OVS merged to master.
      • Simple perf test – merged to master
      • Frame modification tests implementation on hold till Billy returns..
      • CPU and memory stress tests investigation kicked off.
      • RFC 2544 scalability test is being reworked to the new definition in the LTD.
    • Integration of Xena, Moongen & Spirent traffic generators
      • Xena – Work around for RFC2544.exe for Linux using Xena Web presented by Dan Amzulescu dsa@xenanetworks.com, if anyone is looking for assistance with this please contact Dan directly for instructions.
      • Moongen: Please see Moongen RFC2544 script below.
      • Spirent – no update [ACTION] Maryam to check in with Gurpreet
    • CI Integration – In progress, Gene is looking at this.
  • PDV to be collected in RFC5481 format and IXIA:
    • Al has been investigating this and it is possible with some IXIA equipment which have a HW capture buffer. We can capture a certain number of packets for a particular flow/stream (through filtering) with IXIA and should be able to calculate the PDV for packets by processing the timestamps on the packets.
    • Version of IXOS installed on the IXIA chassis connected to POD 3 doesn’t have the capabilities we need to do this – investigating the option of upgrading IXOS.
  • OPNFV Summit Call for Participation Abstract
    • Virtual switches have been long recognized as an essential component of the virtual network in the data centre, and are growing in popularity in the telecoms sector. Exploding traffic is creating extraordinary demand on the network and with that, exceptional performance requirements for the interfaces into Virtual Network Functions in the NFVI. One such interface is the virtual switch, thus it is crucial to be able to characterize its performance in order to determine its capabilities and suitability for deployment in Telco NFV environments. This talk looks at the work undertaken by the OPNFV VSPERF project to define a test suite for characterizing the performance of the virtual switch and any considerations that must be taken into account when doing so. http://events.linuxfoundation.org/cfp/proposals/6770/6242
    • 2 Proposals from Gene for OPNFV – one relating to releng and vsperf test automation, Gene to share more details.
  • Readout this week: VSPERF Release B participation: slides and plan can be found here https://wiki.opnfv.org/wiki/vswitch_release_plan - Only did a quick overview, discussion Item for next week.
  • Open discussion:
    • IMIX traffic (IPv4, TCP and IPv6) and Mobility MIX in OPNFV – OPNFV Testing problem we need to resolve with the community.

August 5th 2015

  • Actions
    • TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: Closed.
    • [ACTION] Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework --> WIP, Gene is looking at this
    • Andrew (Xena) offered a readout on the Xena traffic generator –[ACTION] Billy to follow up - Open.
    • PDV to be collected in RFC5481 format - investigate if it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators - Status: Open
      • Al has been investigating this and it is possible with some IXIA equipment which have a HW capture buffer.
      • [ACTION] Maryam to get the model of the IXIA chassis connected to POD3.
      • Andrew Theurer joined in the discussion and mentioned that it is possible Moongen could also be extended/modified for this purpose. Andrew gave a good overview of the Red Hat contributions to Moongen and future work, please see Moongen update below.
      • [ACTION] Andrew Theurer to check availability for a Moongen Demo/readout.
  • TOIT/VSPERF development update
    • Features under development
      • PVP pushed to gerrit, being reworked.
      • Vanilla OVS pushed to gerrit.
      • Simple perf test – merged to master
      • Frame modification tests implementation ongoing.
      • vhost-cuse support for PVP deployment implementation ongoing.
      • CPU and memory stress tests investigation kicked off.
    • Integration of Xena, Moongen & Spirent traffic generators
      • Xena are dedicating some resources to port their pre-canned RFC2544 tests from an exe process to a Linux process. (few months away)
        • No definite timeframe, there’s a potential workaround that we can use – need to do an analysis on the cost of doing this work now..
      • Moongen – update below.
    • Gurpreet update on integrating the Spirent traffic generator with VSPERF - Open.
    • CI Integration – In progress, Gene is looking at this.
  • Moongen
    • Moongen is a fully scriptable high-speed packet generator built on DPDK and LuaJIT. It can saturate a 10 Gbit/s connection with 64 byte packets on a single CPU core while executing user-provided Lua scripts for each packet. Multi-core support allows for even higher rates. It also features precise and accurate timestamping and rate control. (https://github.com/emmericp/MoonGen)
    • Andrew Theurer mentioned that Red Hat are looking at modifying/extending it in an attempt to emulate/conform to the VSPERF LTD. This will lower the barrier of entry to VSPERF and is useful for prototyping/beta testing.
    • Andrew has been looking at, modifying and contributing to Moongen (modifying existing tests, and added an implementation for a 10Gb adapter, trying to enable a 40Gb adapter at the moment).
    • Red Hat are interested in implementing the VSPERF Spec in Moongen and integrating it with VSPERF.
  • OPNFV Summit Call for Participation Abstract
    • Virtual switches have been long recognized as an essential component of the virtual network in the data centre, and are growing in popularity in the telecoms sector. Exploding traffic is creating extraordinary demand on the network and with that, exceptional performance requirements for the interfaces into Virtual Network Functions in the NFVI. One such interface is the virtual switch, thus it is crucial to be able to characterize its performance in order to determine its capabilities and suitability for deployment in Telco NFV environments. This talk looks at the work undertaken by the OPNFV VSPERF project to define a test suite for characterizing the performance of the virtual switch and any considerations that must be taken into account when doing so.
    • http://events.linuxfoundation.org/cfp/proposals/6770/6242
  • IETF Draft LTD Summary
    • [Action] update LTD and IETF draft with the correct terminology in terms of throughput/maximum forwarding rate for benchmarking methodology – Al volunteered.
    • Put this on hold and try to implement the tests and getting results – to build a compelling story.
    • Should aim continually update the spec in the IETF meeting cycles.

       

      * Gerrit patches Status (https://gerrit.opnfv.org/gerrit/#/q/project:vswitchperf ):

  • vnfs: Enable PVP using vhost-user – to be reworked
  • test_spec: Add LTD.CPDPCouplingFlowAddition
  • test_spec: Clarify LTD.CPU.RFC2544.0PacketLoss
  • test_spec: Clarify LTD.Throughput.RFC2889.BroadcastFrameForwarding – Merged to master
  • src/dpdk: Enable building of vhost-user in src/dpdk. – merged to master
  • Support vswitch selection from the command line. – merged to master
  • Add a simple performance test that sends a continuous stream – merged to master
  • test_spec: Clarify LTD.PacketLatency.InitialPacketProcessingLatency – Merged to master
  • Vanilla OVS support implementation – Merged to master
  • Readout this week: VSPERF VM loopback demo on POD3. [ACTION] Maryam PVP Demo to be setup on POD 3 for people to be able to run.
  • Open discussion:
    • IMIX traffic (IPv4, TCP and IPv6) and Mobility MIX in OPNFV – OPNFV Testing problem we need to resolve with the community.

July 29th 2015

  • Actions
    • TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: open (commit permission issues)
    • Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework --> WIP, Gene is looking at this
    • PDV to be collected in RFC5481 format - Maryam to investigate if this it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators - Status: Open
    • Andrew (Xena) offered a readout on the Xena traffic generator – Billy to follow up - Open.
    • Rick to follow up with Dan on Moongen.
  • TOIT/VSPERF development update
    • Features under development
      • PVP pushed to gerrit, please see patch review list below.
      • Vanilla OVS landing end this week or start of next.
    • Integration of Xena & Spirent traffic generators
      • Xena are dedicating some resources to port their pre-canned RFC2544 tests from an exe process to a Linux process.
      • No update on Spirent - Open
    • Gurpreet update on integrating the Spirent traffic generator with VSPERF - Open.
    • CI Integration – In progress, Gene is looking at this.
  • IETF Draft LTD Summary
    • Maryam & Al presented the draft to IETF on Thur 23rd (https://wiki.opnfv.org/_media/wiki/benchmarking_vswitches_in_opnfv_1.6_aspect_4_3.pdf), presentation was very well received and the BMWG is intrested in seeing how our work progresses. We got some invaluable feedback from Scott Bradner regarding the terminology we are using when we refer to throughput and maximum forwarding rate for our soak tests. For Throughput you need to stop transmitting for a period of time to the DUT, then measure what the device is pushing. Whereas in the soak tests what we are measuring is the DUT max forwarding rate.
  • Patches open for review(https://gerrit.opnfv.org/gerrit/#/q/project:vswitchperf ): Please review folks.
    • test_spec: Clarify LTD.Throughput.RFC2889.BroadcastFrameForwarding
    • vnfs: Enable PVP using vhost-user
    • test_spec: Clarify LTD.PacketLatency.InitialPacketProcessingLatency
    • src/dpdk: Enable building of vhost-user in src/dpdk.
    • test_spec: Add LTD.CPDPCouplingFlowAddition
    • Support vswitch selection from the command line.
    • Add a simple performance test that sends a continuous stream
  • Open discussion:
    • IMIX traffic (IPv4, TCP and IPv6) and Mobility MIX in OPNFV - Investigation underway
  • Readout next week: VSPERF VM loopback demo on POD3.

July 22nd 2015

  • TOIT/VSPERF development update
    • Features under development
      • Bidirectional traffic merged.
      • Frame Modification (push VLAN tag) merging this week.
      • PVP & Vanilla OVS landing end this week or start of next.
    • POD3
      • At least one community member has logged on used POD3 successfully.
    • Integration of Xena & Spirent traffic generators
      • Andrew (Xena) gave a brief demo of Xena in respect of the latency test discussions below. Also said Linux support (using mono) was about 6wks away - Open
      • No update on Spirent - Open
      • RH also considering moongen (dpdk based app, University in Munich) as an off-DUT software trafficgen.
  • IETF Draft LTD Summary
  • Slowpath Latency & Fastpath Latency: good input from all contributers. Billy to submit clarified versions - Open.
  • PDV to be collected in RFC5481 format – Maryam to investigate if this it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators – Status: Open
  • Open discussion
    • IMIX traffic (IPv4, TCP and IPv6) and Mobility MIX in OPNFV – Investigation underway
    • Andrew (Xena) offered a readout on the Xena traffic generator - Open.
 

 

July 15th 2015

  • Actions From Last week
    • Billy to draft a new test for profiling packet loss. Status: closed, merged to master.
    • Add Test report section to every test case in the LTD and specify the fields that should be reported. We should also expand the LTD to report platform specific info that we should gather. Status: closed, merged to master.
  • Future action – Cover SDN flow controller tests. Add a Jira ticket for this, Maryam. Status: Closed, VSPERF-54.
  • TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: open
  • Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework --> open till Gene comes back. Status: open.
  • TOIT/VSPERF development update
    • Features under development:
      • Bidirectional functionality ready to be pushed to gerrit.
      • xunit parsing patch merged to master.
      • Frame modification (vlan tag pushing) under review in gerrit. https://gerrit.opnfv.org/gerrit/#/c/996/
      • pvp deployments (PVP is in progress)
      • Enabling OVS kernel Datapath (in progress)
    • POD3 has been setup to allow folks to login and run VSPERF https://wiki.opnfv.org/get_started/pod_3_-_characterize_vswitch_performance
    • Gurpreet update on integrating the Spirent traffic generator with VSPERF – Open.
    • RH/Xena traffic gen being integrated with vsperf.
    • Gene sw traffic gen integration
      • evaluating netexpect – experiments so far have had disappointing results (nowhere near 10Gb)
      • CI – integration with Jenkins (exploration/discussion underway).
  • vloop VNF – uploaded to the large object archive (Project Milestone) Kudos to Gene!
  • DPDK 2.0 Build
    • TV is to push any additional fixes – Status: Closed.
  • Test definitions under review:
    • PDV to be collected in RFC5481 format – Maryam to investigate if this it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators – Status: Open
      • We should add a JIRA ticket for meshed tests – Status: Closed, VSPERF-55.
  • Open discussion:
    • IMIX traffic (IPv4, TCP and IPv6) and Mobility MIX in OPNFV – Investigation underway

July 8th 2015

  • Future action – Cover SDN flow controller tests. Add a Jira ticket for this, Maryam. Status: open, need to check if this is covered by Yardstick
  • TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: open
  • Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework --> open till Gene comes back. Status: open.
  • TOIT/VSPERF development update
    • Features under development:
      • Back to back test merged to git.
      • Bidirectional functionality ready to be pushed to gerrit.
      • xunit parsing patch under review in gerrit.
      • Frame modification (vlan tag pushing) tests ready to be pushed to gerrit.
      • pvp and pvvp deployments (PVP is in progress)
      • Enabling OVS kernel Datapath (still on hold)
    • POD3 is being setup to allow folks to login and run VSPERF (Traffic Gen issues being debugged and worked through).
    • Gurpreet update on integrating the Spirent traffic generator with VSPERF – Open.
    • Gene sw traffic gen integration – is this something that is still on the horizon.
      • evaluating netexpect.
      • CI – integration with Jenkins (exploration/discussion underway).
  • DPDK 2.0 Build
    • TV is to push any additional fixes – Status: Open.
  • Test definitions under review:
    • PDV to be collected in RFC5481 format – Maryam to investigate if this it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators – Status: Open
      • We should add a JIRA ticket for meshed tests – Status: Open.
  • Open discussion:
    • IMIX traffic (IPv4, TCP and IPv6) and Mobility MIX in OPNFV – Investigation underway
    • The LTD should specifically mention the tuples that should be used when matching a flow (variations include: (inport, outport), Network 5 tuple, Network 13 tuple) – Status: Closed – A flow classification entry has been added to the LTD.

July 1st 2015

  • Future action – cover SDN flow controller tests. Add a Jira ticket for this, Maryam. Status: open.
  • TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: open
  • Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework --> open till Gene comes back. Status: open.
  • TOIT development update
    • Features under development:
      • Back to back test ready to be pushed to gerrit.
      • Bidirectional functionality ready to be pushed to gerrit.
      • xunit parsing patch to be pushed to gerrit.
      • Frame modification tests – Have a working solution, needs some more testing before integrating Openflow rules into VSPERF.
      • Enabling OVS kernel Datapath (still on hold), pvp and pvvp deployments (still ironing out some issues.)
    • POD3 is being setup to allow folks to login and run VSPERF (CentOS 7 has been installed, we are just configuring the system in use for VSPERF).
    • Gurpreet update on integrating the Spirent traffic generator with VSPERF – Open.
  • IETF Draft LTD Summary
    • Al has pushed the latest version of the draft to gerrit – some final touches to be made.
    • Isolation and reporting should be included in the presentation.
    • Should be finalized Friday 03 July 2015.
  • DPDK 2.0 Build
    • Maryam will publish the basic build fixing patches -> Status: Done, patches merged onto master
    • TV is to push any additional fixes – Status: Open.
  • LTD New additions:
    • Test results section.
    • Test Summary section.
    • New Latency profile test.
  • Test definitions under review:
    • PDV to be collected in RFC5481 format – Maryam to investigate if this it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators – Status: Open
    • RFC2889 Forwarding test – should be renamed to max forwarding tests and reference RFC2889.
    • We should add a JIRA ticket for meshed tests.
  • Open discussion:
    • IMIX traffic (IPv4, TCP and IPv6) and Mobility MIX in OPNFV – Investigation underway
    • The LTD should specifically mention the tuples that should be used when matching a flow (variations include: (inport, outport), Network 5 tuple, Network 13 tuple) – Status: Open

June 24th 2015

  • Actions From Last week
    • Gurpreet to publish slides on “Overture vSwitch Performance” to the project wiki.
    • Gene to submit JIRA bug for relative paths issue with VSPERF. ? Maryam to take this as Gene is on holiday. Status: closed.
    • Maryam to update ltd to indicate RFC2889 address caching and learning tests are applicable to MAC learning switches only. Status: closed.
  • Future action – cover SDN flow controller tests. Add a Jira ticket for this, Maryam. Status: open.
  • Billy to draft a new test for profiling packet loss. Status: open.
  • TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: open
  • Al to publish Isolating Functions in the System Under Test on the wiki. Status: open
  • Add Test report section to every test case in the LTD and specify the fields that should be reported. We should also expand the LTD to report platform specific info that we should gather.
  • Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework --> open till Gene comes back. Status: open.
  • Update the meeting link or remove from the wiki, Status:Done.
  • TOIT development update
    • Features under development: Final testing of Back to back test is in progress, xunit parsing patch to be posted soon, frame modification tests (we’re still exploring how to implement this test), enabling OVS kernel Datapath (On hold), pvp and pvvp deployments (ironing out some issues.)
    • POD3 is being setup to allow folks to login and run VSPERF (encountered some remote control issues and we are working on resolving this).
    • Gurpreet update on POD11?
    • Gurpreet update on integrating the Spirent traffic generator with VSPERF – Gurpreet has ramped up on the code and gone throught the High level design document and it’s just implementation that remains.
  • IETF Draft LTD Summary
    • Al has pushed the latest version of the draft to gerrit – needs to be rebased against the latest master.
    • Submission date July 6th
  • DPDK 2.0 Build
    • Maryam will publish the basic build fixing patches this week.
    • TV is to push any additional
  • Test definitions under review:
    • LTD.Throughput.RFC2544.SoakFrameModification
      • 0% packet loss to be taken out
      • Metrics collected to be modified to match LTD.Throughput.RFC2544.Soak.
      • PDV to be collected in RFC5481 format – Maryam to investigate if this is possible.
    • LTD.Throughput.RFC2544.BacktoBack – no modifications
  • Open discussion:
    • IMIX traffic (IPv4, TCP and IPv6) – we should expand the LTD to include IMIX traffic to tests as a traffic type.
    • The LTD should specifically mention the tuples that should be used when matching a flow (variations include: (inport, outport), Network 5 tuple, Network 13 tuple)

June 17th 2015

  • Actions From Last week
    • Gene to submit JIRA bug for relative paths issue with VSPERF. ? AR Maryam to take this as Gene is on holiday.
    • Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework --> open till Gene comes back.
    • Maryam to investigate RFC2889 and whether it’s simple related to MAC learning. If it is, tests should indicate that they are only applicable to vswitches that support MAC learning. -> Update ltd to indicate they are applicable to MAC learning switches.
      • Investigation CLOSED, the 2 tests related to address caching and address learning are related to MAC learning frames.
      • OPEN: Maryam to update test cases with the relevant comments.
    • Future action – cover SDN flow controller tests. Add a Jira ticket for this, Maryam
    • Maryam to look into how we can achieve/implement the RFC 2889 tests
      • Tests can be conducted with IXIA or Spirent traffic generator.
    • Billy to draft a new test for profiling packet loss. Status: open.
  • Packet loss ratio tests
    • Tests that run for a short period of time should be updated to test 0% packet loss.
    • Tests that run for a long period of time (soak tests) should have the 10||-7% packet loss ratio.
      • Action TV to update these tests
  • TOIT development update
    • Features under development: Back to back tests, frame modification tests, enabling OVS kernel Datapath, pvp and pvvp deployments.
    • POD3 is being setup to allow folks to login and run VSPERF.
    • Gurpreet mentioned we might be able to also get setup on POD11.
    • Gurpreet is looking at integrating the Spirent traffic generator with VSPERF.
    • Ideally we want to find a POD where we are connected to all the VSPERF supported traffic generators.
  • IETF Draft LTD Summary
    • Really good progress being made here, Al added in a 3x4 matrix within which we can track the coverage of our tests and identify gaps.
    • The draft has been merged to master.
  • DPDK 2.0 Build
    • TV is making good progress on this front.
    • validated the suggested fixes from Maryam.
    • currently has connectivity issues with gerrit and can’t push fixes. ? ACTION TV to contact opnfv-helpdesk@rt.linuxfoundation.org
  • Readout
    • Isolating Functions in the System Under Test: Al led a discussion on how we can try to get consistent test results without too much variation.
    • ACTION: Slides to be published on the wiki.
    • Action Maryam work with Al on making additions to the diagrams.
  • Other ACTIONS:
    • Gurpreet to publish slides on “Overture vSwitch Performance” to the project wiki.
    • Homework for next week - Billy to nominate tests for discussion.

June 10th 2015

  • VSPERF (TOIT) Update
    • Code has been +2’d and merged to master.
    • We are looking at getting POD3 in the Portland Lab setup for folks to be able to go in and run VSPERF and provide feedback on the framework.
    • We need to investigate if we can setup a test bed in another OPNFV lab that has Spirent equipment. Gene mentioned that the Huawei Lab could be a suitable candidate.
    • Gene is looking at integrating a SW packet generator with VSPERF. Gurpreet mentioned that Spirent could do the same.
  • IETF Summary LTD Draft
    • Good progress being made here, Al is going to push a revised version of the Draft.
    • We need to add in to our spec and this draft how we plan on isolating our measurements so that the results of tests can be repeatable.
      • This will be a discussion item for next week, please see Homework.
    • We need to put some slides together for the Next IETF meeting and socialize our work in order to get it adopted as a working group draft.
  • Homework:
    • 'Considerations on Benchmarking VNFs and their Infrastructure'
  • Actions
    • Gene to submit JIRA bug for relative paths issue with VSPERF.
    • Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework
    • Maryam to investigate RFC2889 and whether it’s simple related to MAC learning. If it is, tests should indicate that they are only applicable to vswitches that support MAC learning.
    • Maryam to look into how we can achieve/implement the RFC 2889 tests.
    • Billy to draft a new test for profiling packet loss.
  • Packet loss ratio tests
    • Tests that run for a short period of time should be updated to test 0% packet loss.
    • Tests that run for a long period of time (soak tests) should have the 10||-7% packet loss ratio.
    • A new test to be drafted to give a packet loss profile for a vSwitch.

June 3rd 2015

  • TOIT Update
    • Code has been pushed for Gerrit review last Friday. [Billy O'Mahony]
    • Huawei currently playing with same. [Gene Snider]
      • Gene is keen to integrate Spirent, Huawei traffic generators as well as some open source VM-based Traffic Generators.
  • Readout on Overture vSwitch Performance Test
    • Gurpreet gave a readout on some performance tests using a low-end COTS system. [Gurpreet]
    • [AR] Gurpreet will post these slides on the VSPERF wiki
  • AOB:
    • Billy asked for volunteers to go on as 'Reviewed-by' for test case reviews. Gene, TV, Gurpreet and Al all agreed.
  • Homework:
    • Next week the following two tests will be reviewed:
      • LTD.Throughput.RFC2889.AddressCachingCapacity
      • LTD.Throughput.RFC2889.AddressLearningRate

May 27th 2015

  • TOIT Update
    • Code is expected to be pushed to OPNFV Gerrit at the end of this week. [Billy O'Mahony]
  • IETF Summary LTD Draft
    • On track to meet draft submission deadline [Al Morton]
    • Some figures require editing to reduce column count as per IETF documentation norms.
  • AOB
    • Gene Snider asked if TOIT would support other traffic generators. Billy said this was a core requirement of TOIT.
    • Gurpreet asked for some time to present some information of VNF testing at the next meeting.
    • [AR] Gurpreet to contact Billy to agree presentation duration.
  • Reviewed 2 LTD test cases:
    • LTD.Throughput.RFC6201.ResetTime:
      • Definition of 'reset' should include both a ungraceful forced reset by the OS (e.g. kill -9) and a graceful restart as per the virtual switches documentation. A test implementation should support both reset methods as an argument to the test.
      • Make explicit that 'average' refers to the average over n trials of the test under identical conditions.
    • LTD.Throughput.RFC2544.SystemRecoveryTime:
      • Clarify that the pre and post overload traffic should be 75% of the RFC2544 maximum throughput.
      • Increase the latency averaging interval
    • [AR] Billy will make and submit these changes to Gerrit.

May 20th 2015

  • TOIT Update
    • Framework integration with low level components is almost complete. We are starting the port to python 3.3.3.
  • IETF Summary LTD Draft
    • Progress made on adding in deployment scenarios to the first draft.
  • Jenkins Build:
    • [Action] TV Rao to push draft patch to gerrit for review so we can try to replicate the encountered issues.
  • Reviewed 2 LTD test cases:
    • LTD.Throughput.RFC2544.Soak: some clarification is required on where 10||-7% packet loss figure came from [action] investigate this. Latency should also be collected as part of this test. Time intervals should be defined for the long/soak tests so that we can determine the time segment where anomalies occur.
    • LTD.Throughput.RFC2544.PacketLossRatioFrameModification: decision made to change the default modification to adding/modifying a VLAN tag.
    • Overall there is consensus for a new section to be added to test cases that specifies what is to be documented in the test report in terms of metrics and other parameters.

Maryam is out of office till June 8th, 2015 Billy O Mahony is providing cover on VSPERF

May 13th 2015

  • LTD Summary for IETF
    • Maryam and Billy are setup with XML mind
    • Al started a great draft for summarizing the LTD to highlight to the IETF how we’ve been using the BMWG RFCs.
    • [ACTION] Maryam to push the draft to gerrit for review (https://gerrit.opnfv.org/gerrit/#/c/555/ )
    • [ACTION] Al to send on XML to RFC text online generation tool – DONE.
  • vloop_vnf
    • Has been pushed to master.
  • Discussion Backlog
    • Billy to nominate a test case to discuss at the weekly call next week.
  • Open Gerrit Reviews
    • TestSpec: Add LTD.Latency.Slowpath
    • testspec: cleanup + update tests with new metrics
    • TestSpec: Add RFC2889 Broadcast Frame Latency test
    • TestSpec: Modify soak tests to measure PDV
    • TestSpec: Modify soak tests to measure PDV
    • TestSpec: Modify LTD access permissions

         I encourage VSPERF committers and contributors to review these and provide their feedback.
      


      Thanks Sothy for the very informative readout on the Performance of OVDK + port and flow mirroring.

May 6th 2015

  • TOIT Update
    • 1st development stream (Implementing the stubbed out skeleton framework) Almost complete
    • 2nd development stream (porting the existing low level components like vswitchs and vnfs… to the new directory structure while maintaining functionality/the ability to run tests) – only one component left.
    • 3rd development stream (path-finding porting the existing framework from python 2.7 to 3.x while maintaining the ability to run tests etc…) – Done.
  • Discussion Backlog
    • I would like to propose we use part of this slot to discuss some of the test cases and their parameters that are a little unclear in the LTD. For example for Frame modification tests we recommend:
      • Modifying the Ethernet source or destination MAC address.
      • Modifying/adding a VLAN tag.
      • Modifying/adding a MPLS tag.
      • Modifying the source or destination ip address.
      • Modifying the TOS/DSCP field.
      • Modifying the source or destination ports for UDP/TCP/SCTP (Recommended).
      • Modifying the TTL.

          Which is the most appropriate/applicable for our case?
        


        I Suggest we review LTD.Throughput.RFC2544.SoakFrameModification as the first item for discussion.

  • I think the next item for discussion I would recommend is the tuple parameters to use to match on for each of the tests. i.e. what do we consider a flow.
  • Updating our pre-commit Jenkins Build
    • TV Rao volunteered to do this.
    • Maryam to send TV the details.

April 29th 2015

Attendees: Eric Dubé, Maryam Tahhan, Al Morton, Billy O’Mahony, TV Rao, gsingh, Dave Neary

Issues with GotoMeeting resulted in a meeting over IRC. So only a couple of topics covered this Week.

  • IETF - draft submission of LTD
    • 3 Volunteers: Al Morton, Billy O’Mahony and Maryam Tahhan
    • [Action: Al Morton] to have a look at the MD open and the ease of conversion to XML, we can create a simple/base doc (next week or so) and work from there.
  • TOIT update: at the moment we have 3 development streams:
    • 1st stream is porting the existing tests we had into the agreed on directory structure - this is going well, we have all the low level components (vswitches/trafficgens/vnfs) bar one moved and old toit still functioning.
    • 2nd stream we have is implementing a stubbed out skeleton framework into which the low level components can be plugged in. We’re making good progress on this front also.
    • 3rd stream which is more of a path finding stream, is taking the exisiting code and porting it from python 2.x to python 3 the methodologies and lessons learned from here can be used to port the skeleton code to python 3 also...
    • As it stands we're on track to making our first code drop in 4 weeks’ time.
  • Opens for next meeting:
    • vloop_vnf
      • Does it bypass the network stack in the guest?
      • Are we concerned that the network stack is a bottle neck?
      • Does code exist for this? If so can we push to gerrit for review?
      • Is it a proprietary VNF, or is it something we plan on developing?
    • Creating a backlog for general discussion items for the weekly meeting.

Action Items

  • Gene Snider to share notes through email or maybe do a wiki entry on the tools that were demo’ed last week.
  • Gurpreet to share a timeslot that suits to read out on DPDK enabled vSwitch performance

April 22nd 2015

Open Questions

  • LTD Summary for IETF:
    • Are we at a point where we can start this?
    • How do we get started?
    • What’s the best way to do this? My suggestion would be etherpad, or a draft patch in gerrit that we can work on, Any recommendations for a way of working on this?
  • vloop_vnf
    • Does it bypass the network stack in the guest?
    • Are we concerned that the network stack is a bottle neck?
    • Does code exist for this? If so can we push to gerrit for review?
    • Is it a proprietary VNF, or is it something we plan on developing?

Action Items

  • Gene Snider to share notes through email or maybe do a wiki entry on the tools that were demo’ed last week.
  • Gurpreet to share a timeslot that suits to read out on DPDK enabled vSwitch performance

Project Update

  • TOIT
    • The refined TOIT HLD can be found at https://gerrit.opnfv.org/gerrit/#/c/225/, Please feel free to download the patch and review.
    • Development has started and we plan on submitting our first patches to VSPERF in 5 weeks’ time. (will update on progress every week)
  • VNFs
    • Can be shared with Aric (Gardner) to push to artefact.opnfv.org so that our Jenkins jobs can access them.

General discussion

  • The generated benchmark reports need to document the minute details of the system setup including the vNIC version as performance can vary greatly between different versions.
  • Spirent’s virtual traffic generator supports RFC testing – great enabling tool, but important to note we can’t get the same performance measurement accuracy as the HW generator.
  • Need to keep the p->v use case that bypasses the vswitch (pNIC to vNIC) on our minds also.

April 8th 2015

Project Update

  • TOIT:
    • We had a quick walkthrough the TOIT High Level Design (HLD) document.
    • This HLD is available for review at: https://gerrit.opnfv.org/gerrit/#/c/225/ and covers the high level architecture for TOIT.
    • Work is continuing on the HLD to define the interfaces and the various classes, this will be published shortly for review.
  • Merged Tests to test spec:
    • Throughput.RFC2889.ForwardingRate
    • Throughput.RFC2889.ForwardPressure
    • Throughput.RFC2889.AddressCachingCapacity
    • Throughput.RFC2889.AddressLearningRate
    • Throughput.RFC2889.ErrorFramesFiltering
    • Throughput.RFC2889.BroadcastFrameForwarding
  • Opens:
    • Need some more clarification on [JIRA] (VSPERF-39) Add simple L2 forwarding module
    • Item for Discussion: How we can advertise our work to other benchmarking communities (ETSI_NFV, IETF Benchmarking Methodology Group)?
  • A new regular slot to feature in the weekly meeting going forward, this slot will be focused around what else is going on in the benchmarking community (ETSI_NFV, IETF Benchmarking Methodology Group). Al Morton kindly agreed to lead the discussions on this.
  • Suggested agenda Items for next week:
    • Eugene Snider to lead a discussion around a plug n play packet generator.
    • Clarification on [JIRA] (VSPERF-39) Add simple L2 forwarding module
    • Item for Discussion: How we can advertise our work to other benchmarking communities (ETSI_NFV, IETF Benchmarking Methodology Group)?
    • Patch submission process walkthrough.

March 25th 2015

Project Update

  • Commit gate issue:
    • Patches pushed to Gerrit were being -1'd by Jenkins CI as a result of a failure in the DPDK build issue on the CentOS 6 machine the verification job was running on. This is because the version of DPDK being built was too old and not ported to the kernel version on the platform running the test.
    • Solution:
      • Patch pushed to master to use a newer version of DPDK (1.8) and also to use a newer tag on the OVS master branch.
      • Requested that the verification job runs on a CentOS 7 or Ubuntu 14.04
  • New Contributors:
    • Tv Rao tv.rao@freescale.com
    • Kin-Yip Liu Kin-Yip.Liu@caviumnetworks.comkliu@cavium.com
  • TOIT
    • High Level Design doc will be pushed for review early next week.
  • Merged Tests:
    • Throughput.RFC2544.SystemRecoveryTime
    • Throughput.RFC2544.BackToBackFrames
    • Throughput.RFC2544.ResetTime
    • Throughput.RFC2544.Soak
    • Throughput.RFC2544.SoakFrameModification

March 11th 2015

Project Update

  • New Project Topic and email tag: VSPERF. (based on teh JIRA project)
  • 2 tests pushed to master:
    • Throughput.RFC2544.PacketLossRate
    • Throughput.RFC2544.PacketLossRateFrameModification
  • Three tests have been rebased and open for review:
    • Throughput.RFC2544.SystemRecoveryTime
    • Throughput.RFC2544.BackToBackFrames
    • Throughput.RFC2544.Soak
  • TOIT
    • Undergoing redesign to fit the prescribed directory structure
  • Vote on promoting Al Morton from contributor to committer has passed.
  • Branching Models:
    • general consensus is to continue development on master and worry about a release when the time comes.
  • CI validation
    • we need to figure out what pre and post commit tests we would like to have for the project.
  • JIRA:
    • still no notifications being received on ticket creation [Action: Maryam to follow up with LF]

Feb/11/2015

Mailing List

TOIT

  • Code is being cleaned up and prep'ed for release
  • Slides to be sent out by Stephen
  • Feedback given that tests should have a configurable duration.

Admin/Infrastructure point of contact for the project: Maryam Tahhan

Patch reviews

  • One +1 for the intro. Need another committer to +1 to progress that test spec section to the next phase
  • Review cycle to be reduced to 2 days rather than 3 days
  • Features to be tested patch rework started today by Maryam Tahhan
  • Owner column added to patch review wiki (https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review) so that other contributors can also re-work patches.
  • Patch reviews and rework should be conducted in the order specified in the table (top to bottom).

Project Leader

  • Will need to select one when the selection process is hardened.

Feb/04/2015

Minutes:

  • Stephen Finucane gave an overview/demo of TOIT
    • TOIT is an OVS test integration framework written in Python that Intel would like to share with the community for re-use for this project. [Action: Stephen F. to send out slides].
    • The framework is generic and licensed under a permissible apache license.
    • Right now it supports OVS and IXIA: IxOS and IxNetworks but the framework can be easily extended to include other traffic generators and switches.
    • Community feedback was very positive, an initial commit for the phy-phy test cases is to be made to gerrit. [Action: Stephen F.]
    • Out of cycle sessions will be setup to walkthrough the code.

Project updates:

  • Initial code reviews for test spec are open. The test spec has been broken down into sections and committed to gerrit for review.
    • Question: why not a single commit? Answer: This is to avoid continuous reviewing sections of a doc that we approve.
    • In an effort to co-ordinate reviews, so sections of the test spec get pushed to gerrit, open reviews will be documented @https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review
    • The suggested review cycle is 3 Working days.
  • Governance model roles for committed and contributor have been updated to the wording from the TSC-Charter (https://wiki.opnfv.org/wiki/vswitchperf_governance_model)
  • OPNFV lab HW spec for the project can be found here: https://wiki.opnfv.org/wiki/vswitch_work_environment
    • Question, what happens if we want to get different NICs…? [Action: Maryam to follow up with Trevor Cooper]
  • Tools:
    • JIRA has been setup.
      • Project name is VSPERF.
      • Please make sure you can access the project and can see the associated issues.
      • Gerrit now enforces a JIRA tag for commits.
  • Opens:
    • Do we need a separate mailing list?
    • It’s difficult to follow the project updates via the current/generic opnfv-tech-discuss mailing list. [Action: Maryam to follow up with Linux Foundation]
  • General Wiki updates:
    • Coding Guidelines : OPNFV Coding Guidelines
    • OPNFV Developer Resource Page : Getting started with your account, Gerrit and Git
    • HW Spec (specified above)
    • Open Gerrit reviews (specified above)
    • Governance model wording (specified above)
  • Open Gerrit review for repo structure (https://gerrit.opnfv.org/gerrit/#/c/9/ )
    • The repo structure review has received a single +1 from one committer. [Action: A committer to review and if applicable provide a second +1 so the patch can committed to our repo]

Jan/21/2015

21 Attendees

Summary:

  • Repo structure proposal review update:
    • Aihua has made the first commit that will allow us to review the repo structure using Gerrit. The URL for review is https://gerrit.opnfv.org/gerrit/#/c/9 .
    • Timeline for this review: All comments must be submitted by: Tuesday 27th January 2015. [Action: vswitchperf committers]
  • Governance model review:
    • Lots of good questions here and discussion about the model.
    • The following updates need to be added to the model [Action: Maryam]
      • Role of reviewer to be defined
      • Integrate features/bugs with JIRA upfront
        • This would mean implicit addition of JIRA tags to commits to ensure they are associated with a planned item, or a fix for a known issue…
        • Will allow us to have agreement on changes upfront
        • Make sure it’s Gerrit enforced.
    • Follow up with Linux Foundation to make sure committers/reviewers, contributors and maintainers have the appropriate +1, +2 options for code review. [Action: Maryam/Mike]
  • Test-bed infrastructure
    • When will we have access to the lab?
      • Access not available yet, work is ongoing to get the lab fully operational.
      • Part of the lab will be dedicated to the vSwitch characterization project.
      • Project specific requirements are necessary to provide an accurate date for accessibility to the Lab. [Action: Maryam] Put together a proposal for community review at the next weekly sync.
  • Git Tags:
    • What tags will we use for various projects?
      • This will be dependent on the switches we use. The Suggestion would be to use the tag for latest stable release of a particular switch and take the appropriate tags (that the switch is compatible with) for any other projects the switch is dependent on. For example the current stable release for OVS is 2.3.1 and it’s compatible with DPDK 1.7.1… [Action: Maryam] Propose tags for community review at the next weekly sync.
  • Tools update
  • Test spec update:
    • The test spec has been updated with Aihua, Srini and Al’s comments. We propose uploading the test spec to Gerrit in sections for review, for e.g. each test will be a separate review in Gerrit. General sections like the intro and methodology will be submitted as one patch.

Jan/14/2015

Attendees: Approx. 25 attendees

Notes:

  • Opens
    • [Action]: Mike to put in place IRC channel for next week’s meeting
    • Where do we store artifacts like VM images? Consensus that they really do not belong in GIT….
    • [Action]: Mike to raise this at Performance and Test group meeting on how to resolve as it won’t be a problem specific to this project.
    • Raised today (Jan 15th); P&G group have taken an action to make a proposal on how to handle this project-wide
  • Comments on Repo Structure proposal
    • Need directory for test specification (+ 1 by Aihua)
    • Plan not to include the source code in the repo but just to clone the Tags (+1 by Aihua)
    • Other comments: included in Aihua’s Google Docs review
    • [Action]: Aihua to present status of review at next week’s project meeting
  • Project Repo and Development tools
    • Git Review in conjunction with Gerrit is set up as part of repo creation by Linux Foundation
    • This is being used in Huawei (GitReview). Operationally similar to GitPush. No objections to using this

Project Development Tools – what candidates?

Code review: Gerrit… no objections

Bug tracking:

  • Could use GitHub bug tracker (assuming we are building on GitHub – need to confirm with Linux Foundation) Alternative could be to use Bugzilla.
  • Bryan Sullivan has experience of GitHub Bug tracker – seems to be pretty effective +1
  • Bugzilla is acceptable fallback if GitHub bug tracker not available (Bryan again).
  • Need to ensure we can enforce a mechanism that makes it clearly visible what bugs are being fixed (Gene Snider)
  • Ideally we should not re-invent the wheel, let’s use as much default from Linux Foundation as possible, unless consensus exists that it’s not acceptable
  • Test Specification draft
    • Maryam presented the structure of the test spec
      • Bryan: is there anything there from QoS perspective? No….Need to get feedback from community here.
      • Al Morton (AT&T): Happy to contribute to some parts of this. Mike to add Al as a contributor [done]
      • Al to send out links to some IETF benchmarking/performance RFCs and drafts [done]
  • Governance structure
    • We need to agree a Governance and Review structure!
    • We are doing some trailblazing here from OPNFV perspective which may serve the benefit of the whole community
    • [Action]: Maryam to provide a detailed proposal to the mailing list on a Governance Model
  1. can now invoke and RFC2559 test next step parse the data (Simona, 15:10:06)

Action: Bill to add a Jira ticket for non aggregated results

Action: Maryam to follow up on decision for database tool

  1. DIcussion on Background Load - we need to account for all of it (acm, 15:33:23)
  2. https://www.net.in.tum.de/fileadmin/bibtex/publications/papers/IFIP-Networking-2016-FLOWer.pdf (mtahhan, 15:33:42)
  3. https://www.net.in.tum.de/fileadmin/bibtex/publications/papers/MoonGen_IMC2015.pdf - original paper (marst, 15:36:53)
  4. Figure 3 shows a bi-modal distribution (acm, 15:42:35)
  • No labels