Background for OPNFV Release Cycle
Until Danube we may consider testing activities below. Release management usually considers only a subset of test projects as release criteria.
Moreover, we only got resources as CI Labs or community Labs as of Danube. In Danube, some tests require stable labs for long duration tests (1 week campaign, destructive tests,..).
Discussion of these tests could be found in https://wiki.opnfv.org/display/bottlenecks/Stress+Testing+over+OPNFV+Platform.
Previous releases show that it is currently impossible to plan long duration test campaigns on the "target" release/master due to a late availability and a lack of stability of the candidate versions.
It is thus planned to consider stable release for robustness/stress testing.
Another CI Chain for Stable Release
For Euphrates, the testing group would like to consider a new way to proceed, described as follow:
The test activities will thus deal with 2 versions of OPNFV in parallel
- version N for the release validation
- version N-1 for the qualification
As a consequence, 2 CI chains must be maintain per installer:
- Master for release validation
- Stable (version N-1) for qualification/long duration tests
With a 6 months cadence, it is not realistic to perform robustness/stress/long duration test on Master. On the stable chains, a planning shall be created.
We assume that unlike Master, reinstallation shall be triggered according to this planning, it means that some scenarios may be used during several weeks without reinstallation.
Generic scenarios must be considered first.
CI Chains summary
- Master managed as today
- Stable must be maintained, reinstallation triggered after a test campaign. Allocation of resources to be organized within the testing group
Test Planning for the Euphrates Release (Proposal)
Testing arrangement is proposed below.
2 projects share a timeslot for joint testing, i.e.,
- either run test cases simultaneously, e.g., Storage and CPU testing in parallel,
- or one project call the other to do some tests, e.g., Bottlenecks call Yardstick to do tests.
Use OSA/Otaca to do the deployment considering currently resource limitation and to form common base for OPNFV installations.
- Access to Intel-18 to locate the failures and root causes
- Access to Intel-18 to show the testing reports
- Feed back bugs that only exist in long duration tests
Each timeslot is assigned with 2 weeks to guarantee we do thorough test and to be able obtain useful results
- The testing workflow should be triggered by CI system
- Re-installation on demand (CI still needed for clean re-installation but re-installation not automated to allow long duration tests + troubleshooting)
- Reports for the executed tests. It'd better have a unified report page, e.g., based on CI system, based on testing DB, etc.