This page is intended to provide an overview of the test cases provided by each test project in order to come to a decision about which ones will be part of the release criteria, and therefore of the daily runs on CI PODS.
IMPORTANT: Not being part of the release criteria does not mean not being part of the release. Those test cases that won't be part of the criteria, can be still run on non CI PODs and released in Colorado.
The intention of this is to have defined daily jobs for testing and estimate how long it will take to run a complete verification after a deployment.
Test Case Name | Approx. Duration (min) | Success Criteria | Part of Release Criteria? | Dependencies | Tier | CI |
|---|---|---|---|---|---|---|
| FUNCTEST | ||||||
| healthcheck | 2 | PASS | Yes | - | 0 | daily |
| vping_ssh | 2-4 | PASS | Yes | - | 1 | |
| vping_userdata | 1-2 | PASS | Yes | - | 1 | |
| tempest_smoke | 15-20 | 100% PASS | Yes | scenario: os-* | 1 | |
| rally_sanity | 10 | 100% PASS | Yes | scenario: os-* | 1 | |
| security_groups | ? | PASS | Yes | scenario: os-* | 1 | |
| odl | 1 | PASS | Yes | scenario: odl_l2, odl_l3 | 2 | |
| onos | 1 | PASS | Yes | scenario: onos | 2 | |
| ocl | ? | PASS | Yes | scenario: ocl | 2 | |
| tempest_full_parallel | ~60-90 | 80% PASS | to be decided | scenario: os-* | 4 | weekly with threshold set to 3h max use trust indicator |
| rally | 90 | 90% PASS | to be decided | - | 5 | |
| vIMS | 30 | PASS | to be decided | - | 6 | |
| Feature projects integration: | ||||||
| promise | 1 | PASS | Yes | installer: fuel, joid | 3 | daily |
| doctor | 5 | PASS | Yes | installer: apex | 3 | |
| sdnvpn | 1 | PASS | Yes | scenario: bgpvpn | 3 | |
| copper | 10 | PASS | Yes | installer: all | 3 | |
| models | 10 | PASS | Yes | installer: all | 3 | |
| moon | 5? | PASS | Yes | scenario: moon | 3 | |
| movie | 5? | PASS | To be decided | scenerio:os-odl_l2-*, Installer:tbd | 3 | |
| multisite | 10? | PASS | To be decided | scenerio:os-odl_l2-*, Installer:tbd | 3 | |
| parser | 5 | PASS | To be decided | scenerio:os-onos-nofeatue-ha, installer:compass | 3 | |
| domino | 5 | PASS | To be decided | scenerio:os-onos-nofeatue-ha, installer:compass Parser + Open-O | 3 | |
| security | ? | PASS | To be decided | installer:all | 3 | |
Test Case Name | Approx. Duration (min) | Success Criteria | Part of Release Criteria? | Dependencies | Tier | CI |
|---|---|---|---|---|---|---|
| YARDSTICK | ||||||
| OPNFV_YARDSTICK_TC001_NW PERF | 5 | TO BE DECIDED | YES | - | 0 | daily |
| OPNFV_YARDSTICK_TC002_NW LATENCY | 5 | TO BE DECIDED | YES | - | 0 | |
OPNFV_YARDSTICK_TC005_STORAGE PERFORMANCE | 5 | TO BE DECIDED | YES | - | 0 | |
| OPNFV_YARDSTICK_TC010_MEMORY LATENCY | 5 | TO BE DECIDED | YES | - | 0 | |
| OPNFV_YARDSTICK_TC014_PROCESSING SPEED | 15 | TO BE DECIDED | YES | - | 0 | |
OPNFV_YARDSTICK_TC008_NW PERF, PACKET LOSS EXTENDED | 20-25 | TO BE DECIDED | YES | - | 3 | weekly with threshold set to 3h max |
OPNFV_YARDSTICK_TC009_NW PERF, PACKET LOSS | 20-25 | TO BE DECIDED | YES | - | 3 | |
| OPNFV_YARDSTICK_TC011_PACKET DELAY VARIATION BETWEEN VMs | 5-10 | TO BE DECIDED | YES | - | 3 | |
| OPNFV_YARDSTICK_TC012_MEMORY BANDWIDTH | 15 | TO BE DECIDED | YES | - | 3 | |
| OPNFV_YARDSTICK_TC024_CPU LOAD | 20 | TO BE DECIDED | YES | - | 3 | |
| OPNFV_YARDSTICK_TC037_LATENCY, CPU LOAD, THROUGHPUT, PACKET LOSS | 20 | TO BE DECIDED | YES | - | 3 | |
| OPNFV_YARDSTICK_TC038_LATENCY, CPU LOAD, THROUGHPUT, PACKET LOSS | 25 | TO BE DECIDED | YES | - | 3 | |
| Feature projects integration: | ||||||
| sfc | ? | PASS | YES | installer:apex scenario:*-sfc | 1 | daily |
| ha | 10 | PASS | YES | scenario:*-ha installer:fuel | 1 | |
| ipv6 | 10 | PASS | YES | scenario:nosdn installer:compass- | 1 | |
| nfv for kvm | 5 | PASS | YES | scenario:*-kvm | 1 | |
| parser | 5 | PASS | YES | all | 1 | |
| promise | 1 | |||||
| vnffg | 5 | PASS | to be decided | use case and test palns | 1 | |
| sdnvpn | 1 | |||||
| multisite | 10? | PASS | to be decided | vlan & POD assignment | 1 | |
| domino | 5 | PASS | to be decided | test plans | 1 | |
Testing projects integration:
Test Case Name | Approx. Duration (min) | Success Criteria | Part of Release Criteria? | Dependencies | Tier | CI |
|---|---|---|---|---|---|---|
| Vsperf | ? | scenario: *-ovs | 2 | daily | ||
| Cperf (to be decided) | ? | scenario: *-odl-l3 | 2 | |||
| StorPerf | ? | 3 | weekly 3h max |
| test case name | Approx Duration(min) | success criteria | Part of release criteria | dependencies | CI | |
|---|---|---|---|---|---|---|
| Bottlenecks | ||||||
| rubbos | opnfv_bottlenecks_tc1001: rubbos_1-1-0-1 | 20 | PASS | yes | os-odl_l2-nofeature-ha/ os-onos-nofeature-ha | daily |
opnfv_bottlenecks_tc1002: rubbos_1-2-0-1 | 20 | PASS | yes | os-odl_l2-nofeature-ha/ os-onos-nofeature-ha | daily | |
opnfv_bottlenecks_tc1003: rubbos_1-1-1-2 | 20 | PASS | yes | os-odl_l2-nofeature-ha/ os-onos-nofeature-ha | weekly | |
| more… | ||||||
| vstf | opnfv_bottlenecks_tc2001: Tu-1 | 10 | PASS | yes | scenarios with ovs (liyiting@huawei.com) | daily |
opnfv_bottlenecks_tc2003: Tu-3 | 10 | PASS | yes | scenarios with ovs (liyiting@huawei.com) | daily | |
opnfv_bottlenecks_tc2011: Ti | 15 | PASS | no(need special phy env) | |||
opnfv_bottlenecks_tc2021: Tn | 15 | PASS | no(need special phy env) | |||
| feature projects integration | ||||||
| prediction | ||||||
| multisite | opnfv_multisite_tc1001:* | 10 | PASS | tbd | PODS? vlans? | |
| parser | opnfv_parser_tc1001:* | 5 | PASS | tbd | ||
| domino | opnfv_domino_tc1001:* | 5 | PASS | tbd | ||
| vswitch | Test Case Name | Packet Sizes | Approx. Duration (min) | Success Criteria | Part of Release Criteria? | Dependencies | Comments |
|---|---|---|---|---|---|---|---|
| VSPERF | |||||||
| ovsdpdk | phy2phy_tput_p2p | 64, 128, 512, 1024, 1518 | 00:17:24 | Test runs successfully. We only report performance today | Doesn't impede the release | Until Moongen is integrated our dependency is on IXIA connected to POD 3 | Can be reduced significantly if we iterate over less packets In addition we have other tests that can be bound to 1 minute. These aren't running in the CI today but can be enabled easily. |
| back2back_p2p | 64, 128, 512, 1024, 1518 | 00:25:21 | |||||
| phy2phy_tput_mod_vlan_p2p | 64, 128, 512, 1024, 1518 | 00:28:20 | |||||
| phy2phy_scalability_p2p | 64, 128, 512, 1024, 1518 | 00:17:16 | |||||
| pvp_tput_pvp | 64, 128, 512, 1024, 1518 | 00:18:46 | |||||
| pvp_back2back_pvp | 64, 128, 512, 1024, 1518 | 00:36:56 | |||||
| pvvp_tput_pvvp | 64, 128, 512, 1024, 1518 | 00:36:14 | |||||
| pvvp_back2back_pvvp | 64, 128, 512, 1024, 1518 | 00:50:51 | |||||
| ovs | phy2phy_tput_p2p | 64, 128, 512, 1024, 1518 | 00:28:11 | Test runs successfully. We only report performance today | Doesn't impede the release | Until Moongen is integrated our dependency is on IXIA connected to POD 3 | Can be reduced significantly if we iterate over less packets |
| back2back_p2p | 64, 128, 512, 1024, 1518 | 00:42:35 | |||||
| phy2phy_tput_mod_vlan_p2p | 64, 128, 512, 1024, 1518 | 00:27:54 | |||||
| phy2phy_scalability_p2p | 64, 128, 512, 1024, 1518 | 00:28:04 | |||||
| pvp_tput_pvp | 64, 128, 512, 1024, 1518 | 00:29:31 | |||||
| pvp_back2back_pvp | 64, 128, 512, 1024, 1518 | 00:43:49 | |||||
| pvvp_tput_pvvp | 64, 128, 512, 1024, 1518 | ||||||
| pvvp_back2back_pvvp | 64, 128, 512, 1024, 1518 | 00:45:31 |
Test Case Name | Approx. Duration (min) | Success Criteria | Part of Release Criteria? | Dependencies |
|---|---|---|---|---|
| STORPERF | ||||
| Initialization | 1 day | Installation Cinder Volumes | ||
| SNIA Latency | 4h - 1d | Steady state reached | Initialization | |
| SNIA IOPS | 4h - 1d | Steady state reached | Initialization | |
| SNIA Throughput | 4h - 1d | Steady state reached | Initialization | |
| CPERF (new project, tests under-dev) | ||||
CBench ("hello world") | 20 | TBD perf number | No | odl_l3 |
| NSTAT | 60 | TBD perf number | No | odl_l3 |
2 Comments
Fatih Degirmenci
Jose Lausuch can we move this page to a better place so we only list the releases on right hand side?
Morgan Richomme
Jose Lausuch Fatih Degirmenci I moved the page to the Colorado testing page under testing