Page tree
Skip to end of metadata
Go to start of metadata

Preparation & useful links

brainstorming: https://etherpad.opnfv.org/p/functest_release_C

Test strategy release C (Morgan): https://wiki.opnfv.org/download/attachments/2925495/Test%20Strategy%20for%20C%20release%20%28Morgan%29.pptx?api=v2

Proposal Tiers (Jose): https://wiki.opnfv.org/download/attachments/2925495/Proposal%20Functest%20Tiers%20Colorado.pptx?api=v2

Espoo Meetup minutes: http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2016/opnfv-testperf.2016-03-30-06.02.html

Introduction for Flash test (Nikolas): https://wiki.opnfv.org/download/attachments/2925495/first_presentation.pptx?api=v2

 

Post mortem

Global Testing post-mortem: https://etherpad.opnfv.org/p/Colorado-Testing-postmortem

Some graphs based on the validated scenarios (stored from reporting)

Colorado Release Time plan

Please Note

  • the dates have been modified by Functest and have not been validated. They correspond to what something reasonable according to Functest contributors 
  • OPNFV Plugfest 9-13/5
  • OPNFV Summit 20-23/6

See Colorado for more details

Summer vacation planning

WhoWhenComments
morgan_orange15/7-5/8 
CG_Nokia1/7 - 31/7

In Ireland 1st July -> 7th July
Back in Finland 8th July

viktor_t11/7 - 31/7, 8/8 - 10/8 
Jose1-25/8Summer vacation in Spain.
juhak11/7 - 7/8

 

JuhaHaapa11/7 - 7/8 
ollivier13/7-5/8 
lhinds22/7 - 08/8 

Colorado Functest test cases

Colorado test cases are declared in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml and in the Test DB (http://testresults.opnfv.org/test/api/v1/projects/functest/cases)

TierTest caseProjectCI loop

Estimated

duration

0: healthcheckhealthcheckfunctestmerge/daily/weekly2
1: smoke


vping_sshfunctestmerge/daily/weekly2-4
vping_userdatafunctest1-2
tempest_smoke_serialfunctest15-20
rally_sanityfunctest10
2: sdn_suiteodlfunctestdaily/weekly1
onosfunctest1
oclovno?
3: featurespromisepromisedaily/weekly1
doctordoctor5
bgpvpnsdnvpn1
odl-sfcsfc 
onos-sfconosfw 
dominodomino 
coppercopper 
parserparser 
multisitemultisite 
4:Openstackrally_fullfunctestweekly60-90
tempest_fullfunctest60-90
5:VNFvimsfunctestweekly30

We introduced the notion of Tiers in order to assign a category to the test cases and ultimately save time in CI.

Project management

Coordinated schedules with other projects and captured cross project dependencies in JIRA

  • Dependencies to SDNVPN project:  FUNCTEST-181 - Getting issue details... STATUS   SDNVPN-19 - Getting issue details... STATUS   SDNVPN-38 - Getting issue details... STATUS
  • Dependencies to Promise project:  FUNCTEST-162 - Getting issue details... STATUS   PROMISE-35 - Getting issue details... STATUS
  • Dependencies to ONOSFW project:  FUNCTEST-182 - Getting issue details... STATUS  (still needed linked dependencies)
  • Dependencies to OVNO project:  FUNCTEST-183 - Getting issue details... STATUS  (still needed linked dependencies)
  • Dependencies to Doctor project: not defined yet
  • Additional projects to be discussed (Copper? Policy?) 

 

Established a concrete project time plan in place that accommodates dependencies and resource availability

Described in the tables above and below.

 

Clarified dependencies, must include upstream dependencies, impacting ability to execute on future milestones

The only upstream tools used in Functest are Openstack/Tempest and OpenStack/Rally projects and they do not impact any milestone as we use fixed stable versions that are always in line with the continuous OpenStack releases.

 

In addition:  please include a link to a JIRA filter and/or dashboard that includes your documented JIRA issues.

Dashboard: https://jira.opnfv.org/secure/RapidBoard.jspa?rapidView=59

List of JIRA issues:

Loading
Key Summary T Created Updated Due Assignee Reporter P Status Resolution
FUNCTEST-272 Replace system calls between python scripts Improvement May 30, 2016 May 21, 2017 Jun 30, 2016 Jose Lausuch Cedric Ollivier Minor Open Unresolved
FUNCTEST-271 Replace coreutils system calls by os library Improvement May 30, 2016 May 21, 2017 Jun 30, 2016 Jose Lausuch Cedric Ollivier Minor Open Unresolved

 

Colorado content for Functest

Note: most of the task have only 1 owner and can be done in parallel. 

test evolutionnew testframework evolution
ODL (Colum, Cedric, Ragha, Juha)Security Group (David)Dashboard evolution to Kibana (Juraj) changed DB backend from Mongo DB to Elastic search to include native analytics functions (Juraj)
vIMS (Valentin)security suites (Luke)Test management refactoring - slicing the test suites from Health to VNFs (Jose)

Tempest evoution (2 levels smoke and advanced)

back to upstream / upstream contribution

new VNFs (Bertand)Add trust indicator to optimize test runs (Morgan)

Rally evolution

back to upstream / upstream contribution

ARM suite (Ciprian)new unit test on API (Serena)
Onos evolutionFlashtest (Nikolas)

Add swagger for automatic documentation of the API (Serena)

 Opencontrail / Ovno (with OVNO contributor)Generate automatic test reporting (Morgan)
  Refactoring of Jenkins page (May)
  API evolution (include version, scenario, success criteria, trust indicator.. (Morgan)
  Add web admin frontal to declare/modify/remove new test cases/pods/projects in DB (Mongo then Elasticsearch) (Ragha)
   

 

Feature project Matrix

 

ProjectProject declared in test DB

Test cases declared

in test DB

JIRA createdHistory

Link to test dir

in project repo

Test duration

evaluation (min)

success criteriaPart of release criteria

Dependecies

(installers/scenarios)

promiseyesyesyes> Brahmaputrayes1PASSYesinstaller: fuel, joid
doctoryesyesno> Brahmaputrayes5PASSYesinstaller: apex
sdnvpnyesyesyes*> Brahmaputrayes1PASSYesscenario: bgpvpn
copperyesyesyesColoradotests10PASSYesinstaller: all scenarios of JOID, APEX
modelsnononoColorado?10PASSYesinstaller: all
moonyesyesyesColorado?5?PASSYesscenario: moon
movienononoColorado?5?PASSTo be decidedscenerio:os-odl_l2-*, Installer:tbd
multisiteno (Not yet)no

Test cases ready in OpenStack repository,

provide plugin to tempest

tempest can discover the test cases

yesColorado?10?PASSTo be decided

Use preconfigured multisite testbed

for no installer support multisite installation automation yet.

parsernonoyesColorado?5PASSTo be decidedscenerio:os-onos-nofeatue-ha, installer:compass
dominoyesyesyesColorado?5PASSTo be decidedscenerio:os-onos-nofeatue-ha, installer:compass
Parser + Open-O
ONOSFW yesyesyes> Brahmaputrayes (in 1-2 weeks)~ 10PASSYes

installer: all

scenario: os-onos-sfc (vnffg)

 

How to Functest

Why did we create tiers?

How to know which test I can run?

How to manually start Functest test?

How to select my list of tests for CI?

How test constraints are defined?

How do I declare my project in the Test DB?

How do I push my results in the database?

Where can I find the documentation on the test API?

How to exclude Tempest case from default Tempest smoke suite?

How do I know the Functest status of a scenario for Colorado?

 

  • No labels