Preparation & useful links
brainstorming: https://etherpad.opnfv.org/p/functest_release_C
Test strategy release C (Morgan): https://wiki.opnfv.org/download/attachments/2925495/Test%20Strategy%20for%20C%20release%20%28Morgan%29.pptx?api=v2
Proposal Tiers (Jose): https://wiki.opnfv.org/download/attachments/2925495/Proposal%20Functest%20Tiers%20Colorado.pptx?api=v2
Espoo Meetup minutes: http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2016/opnfv-testperf.2016-03-30-06.02.html
Introduction for Flash test (Nikolas): https://wiki.opnfv.org/download/attachments/2925495/first_presentation.pptx?api=v2
Post mortem
Global Testing post-mortem: https://etherpad.opnfv.org/p/Colorado-Testing-postmortem
Some graphs based on the validated scenarios (stored from reporting)
Colorado Release Time plan
Please Note
- the dates have been modified by Functest and have not been validated. They correspond to what something reasonable according to Functest contributors
- OPNFV Plugfest 9-13/5
- OPNFV Summit 20-23/6
See Colorado for more details
Summer vacation planning
| Who | When | Comments |
|---|---|---|
| morgan_orange | 15/7-5/8 | |
| CG_Nokia | 1/7 - 31/7 | In Ireland 1st July -> 7th July |
| viktor_t | 11/7 - 31/7, 8/8 - 10/8 | |
| Jose | 1-25/8 | Summer vacation in Spain. |
| juhak | 11/7 - 7/8 |
|
| JuhaHaapa | 11/7 - 7/8 | |
| ollivier | 13/7-5/8 | |
| lhinds | 22/7 - 08/8 |
Colorado Functest test cases
Colorado test cases are declared in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml and in the Test DB (http://testresults.opnfv.org/test/api/v1/projects/functest/cases)
| Tier | Test case | Project | CI loop | Estimated duration |
|---|---|---|---|---|
| 0: healthcheck | healthcheck | functest | merge/daily/weekly | 2 |
| 1: smoke | vping_ssh | functest | merge/daily/weekly | 2-4 |
| vping_userdata | functest | 1-2 | ||
| tempest_smoke_serial | functest | 15-20 | ||
| rally_sanity | functest | 10 | ||
| 2: sdn_suite | odl | functest | daily/weekly | 1 |
| onos | functest | 1 | ||
| ocl | ovno | ? | ||
| 3: features | promise | promise | daily/weekly | 1 |
| doctor | doctor | 5 | ||
| bgpvpn | sdnvpn | 1 | ||
| odl-sfc | sfc | |||
| onos-sfc | onosfw | |||
| domino | domino | |||
| copper | copper | |||
| parser | parser | |||
| multisite | multisite | |||
| 4:Openstack | rally_full | functest | weekly | 60-90 |
| tempest_full | functest | 60-90 | ||
| 5:VNF | vims | functest | weekly | 30 |
We introduced the notion of Tiers in order to assign a category to the test cases and ultimately save time in CI.
Project management
Coordinated schedules with other projects and captured cross project dependencies in JIRA
- Dependencies to SDNVPN project: FUNCTEST-181 - Getting issue details... STATUS SDNVPN-19 - Getting issue details... STATUS SDNVPN-38 - Getting issue details... STATUS
- Dependencies to Promise project: FUNCTEST-162 - Getting issue details... STATUS PROMISE-35 - Getting issue details... STATUS
- Dependencies to ONOSFW project: FUNCTEST-182 - Getting issue details... STATUS (still needed linked dependencies)
- Dependencies to OVNO project: FUNCTEST-183 - Getting issue details... STATUS (still needed linked dependencies)
- Dependencies to Doctor project: not defined yet
- Additional projects to be discussed (Copper? Policy?)
Established a concrete project time plan in place that accommodates dependencies and resource availability
Described in the tables above and below.
Clarified dependencies, must include upstream dependencies, impacting ability to execute on future milestones
The only upstream tools used in Functest are Openstack/Tempest and OpenStack/Rally projects and they do not impact any milestone as we use fixed stable versions that are always in line with the continuous OpenStack releases.
In addition: please include a link to a JIRA filter and/or dashboard that includes your documented JIRA issues.
Dashboard: https://jira.opnfv.org/secure/RapidBoard.jspa?rapidView=59
List of JIRA issues:
Colorado content for Functest
Note: most of the task have only 1 owner and can be done in parallel.
| test evolution | new test | framework evolution |
|---|---|---|
| ODL (Colum, Cedric, Ragha, Juha) | Security Group (David) | Dashboard evolution to Kibana (Juraj) changed DB backend from Mongo DB to Elastic search to include native analytics functions (Juraj) |
| vIMS (Valentin) | security suites (Luke) | Test management refactoring - slicing the test suites from Health to VNFs (Jose) |
Tempest evoution (2 levels smoke and advanced) back to upstream / upstream contribution | new VNFs (Bertand) | Add trust indicator to optimize test runs (Morgan) |
Rally evolution back to upstream / upstream contribution | ARM suite (Ciprian) | new unit test on API (Serena) |
| Onos evolution | Flashtest (Nikolas) | Add swagger for automatic documentation of the API (Serena) |
| Opencontrail / Ovno (with OVNO contributor) | Generate automatic test reporting (Morgan) | |
| Refactoring of Jenkins page (May) | ||
| API evolution (include version, scenario, success criteria, trust indicator.. (Morgan) | ||
| Add web admin frontal to declare/modify/remove new test cases/pods/projects in DB (Mongo then Elasticsearch) (Ragha) | ||
Feature project Matrix
| Project | Project declared in test DB | Test cases declared in test DB | JIRA created | History | Link to test dir in project repo | Test duration evaluation (min) | success criteria | Part of release criteria | Dependecies (installers/scenarios) |
| promise | yes | yes | yes | > Brahmaputra | yes | 1 | PASS | Yes | installer: fuel, joid |
| doctor | yes | yes | no | > Brahmaputra | yes | 5 | PASS | Yes | installer: apex |
| sdnvpn | yes | yes | yes* | > Brahmaputra | yes | 1 | PASS | Yes | scenario: bgpvpn |
| copper | yes | yes | yes | Colorado | tests | 10 | PASS | Yes | installer: all scenarios of JOID, APEX |
| models | no | no | no | Colorado | ? | 10 | PASS | Yes | installer: all |
| moon | yes | yes | yes | Colorado | ? | 5? | PASS | Yes | scenario: moon |
| movie | no | no | no | Colorado | ? | 5? | PASS | To be decided | scenerio:os-odl_l2-*, Installer:tbd |
| multisite | no (Not yet) | no Test cases ready in OpenStack repository, provide plugin to tempest tempest can discover the test cases | yes | Colorado | ? | 10? | PASS | To be decided | Use preconfigured multisite testbed for no installer support multisite installation automation yet. |
| parser | no | no | yes | Colorado | ? | 5 | PASS | To be decided | scenerio:os-onos-nofeatue-ha, installer:compass |
| domino | yes | yes | yes | Colorado | ? | 5 | PASS | To be decided | scenerio:os-onos-nofeatue-ha, installer:compass Parser + Open-O |
| ONOSFW | yes | yes | yes | > Brahmaputra | yes (in 1-2 weeks) | ~ 10 | PASS | Yes | installer: all scenario: os-onos-sfc (vnffg) |
How to Functest
How to know which test I can run?
How to manually start Functest test?
How to select my list of tests for CI?
How test constraints are defined?
How do I declare my project in the Test DB?
How do I push my results in the database?
Where can I find the documentation on the test API?
How to exclude Tempest case from default Tempest smoke suite?
How do I know the Functest status of a scenario for Colorado?