Page tree
Skip to end of metadata
Go to start of metadata

This page is intended to provide an overview of the test cases provided by each test project in order to come to a decision about which ones will be part of the release criteria, and therefore of the daily runs on CI PODS.

IMPORTANT: Not being part of the release criteria does not mean not being part of the release. Those test cases that won't be part of the criteria, can be still run on non CI PODs and released in Colorado.

The intention of this is to have defined daily jobs for testing and estimate how long it will take to run a complete verification after a deployment.

Test

Case

Name

Approx.

Duration

(min)

Success

Criteria

Part of

Release

Criteria?

DependenciesTierCI
FUNCTEST
healthcheck2PASSYes-0daily
vping_ssh2-4PASSYes-1
vping_userdata1-2PASSYes-1
tempest_smoke15-20100% PASSYesscenario: os-*1
rally_sanity10100% PASSYesscenario: os-*1
security_groups?PASSYesscenario: os-*1
odl1PASSYesscenario: odl_l2, odl_l32
onos1PASSYesscenario: onos2
ocl?PASSYesscenario: ocl2
tempest_full_parallel~60-9080% PASSto be decidedscenario: os-*4

weekly

with threshold set to 3h max

use trust indicator

rally9090% PASSto be decided-5
vIMS30PASSto be decided-6
Feature projects integration:
promise1PASSYesinstaller: fuel, joid3daily
doctor5PASSYesinstaller: apex3
sdnvpn1PASSYesscenario: bgpvpn3
copper10PASSYesinstaller: all3
models10PASSYesinstaller: all3
moon5?PASSYesscenario: moon3
movie5?PASSTo be decidedscenerio:os-odl_l2-*, Installer:tbd3
multisite10?PASSTo be decidedscenerio:os-odl_l2-*, Installer:tbd3
parser5PASSTo be decidedscenerio:os-onos-nofeatue-ha, installer:compass3
domino5PASSTo be decided

scenerio:os-onos-nofeatue-ha, installer:compass

Parser + Open-O

3
security?PASSTo be decidedinstaller:all3

Test

Case

Name

Approx.

Duration

(min)

Success

Criteria

Part of

Release

Criteria?

DependenciesTierCI
YARDSTICK
OPNFV_YARDSTICK_TC001_NW PERF5TO BE DECIDEDYES-0daily
OPNFV_YARDSTICK_TC002_NW LATENCY5TO BE DECIDEDYES-0

OPNFV_YARDSTICK_TC005_STORAGE PERFORMANCE

5TO BE DECIDEDYES-0
OPNFV_YARDSTICK_TC010_MEMORY LATENCY5TO BE DECIDEDYES-0
OPNFV_YARDSTICK_TC014_PROCESSING SPEED15TO BE DECIDEDYES-0

OPNFV_YARDSTICK_TC008_NW PERF, PACKET

LOSS EXTENDED

20-25TO BE DECIDEDYES-3

weekly

with threshold set to 3h max

OPNFV_YARDSTICK_TC009_NW PERF, PACKET

LOSS

20-25TO BE DECIDEDYES-3
OPNFV_YARDSTICK_TC011_PACKET DELAY VARIATION BETWEEN VMs5-10TO BE DECIDEDYES-3
OPNFV_YARDSTICK_TC012_MEMORY BANDWIDTH15TO BE DECIDEDYES-3
OPNFV_YARDSTICK_TC024_CPU LOAD20TO BE DECIDEDYES-3
OPNFV_YARDSTICK_TC037_LATENCY, CPU LOAD, THROUGHPUT, PACKET LOSS20TO BE DECIDEDYES-3
OPNFV_YARDSTICK_TC038_LATENCY, CPU LOAD, THROUGHPUT, PACKET LOSS25TO BE DECIDEDYES-3
Feature projects integration:
sfc?PASSYES

installer:apex

scenario:*-sfc

1daily
ha10PASSYES

scenario:*-ha

installer:fuel

1
ipv610PASSYES

scenario:nosdn

installer:compass-

1
nfv for kvm5PASSYESscenario:*-kvm1
parser5PASSYES

all

1
promise    1
vnffg5PASSto be decided

use case and test palns

1
sdnvpn    1
multisite10?PASSto be decided

vlan & POD assignment

1
domino5PASSto be decided

test plans

1

Testing projects integration:

Test

Case

Name

Approx.

Duration

(min)

Success

Criteria

Part of

Release

Criteria?

DependenciesTierCI
Vsperf?  scenario: *-ovs2daily
Cperf (to be decided)?  scenario: *-odl-l32
StorPerf ?   3

weekly

3h max


 test case name

Approx

Duration(min)

success criteria Part of release criteriadependenciesCI
Bottlenecks      
rubbos

opnfv_bottlenecks_tc1001:

rubbos_1-1-0-1

20PASSyes

os-odl_l2-nofeature-ha/

os-onos-nofeature-ha

daily
 

opnfv_bottlenecks_tc1002:

rubbos_1-2-0-1

20PASSyes

os-odl_l2-nofeature-ha/

os-onos-nofeature-ha

daily
 

opnfv_bottlenecks_tc1003:

rubbos_1-1-1-2

20PASSyes

os-odl_l2-nofeature-ha/

os-onos-nofeature-ha

weekly
 more…     
vstf

opnfv_bottlenecks_tc2001:

Tu-1

10PASSyes

 scenarios with ovs

(liyiting@huawei.com)

daily
 

opnfv_bottlenecks_tc2003:

Tu-3

10PASSyes

scenarios with ovs

(liyiting@huawei.com) 

daily
 

opnfv_bottlenecks_tc2011:

 Ti

15PASSno(need special phy env) 
 

opnfv_bottlenecks_tc2021:

 Tn

15PASSno(need special phy env) 
       
feature projects integration    
prediction      
multisiteopnfv_multisite_tc1001:*10PASStbd

PODS?

vlans?

 
parseropnfv_parser_tc1001:*5PASStbd  
dominoopnfv_domino_tc1001:*5PASStbd  
vswitch

Test

Case

Name

Packet Sizes

Approx.

Duration

(min)

Success

Criteria

Part of

Release

Criteria?

DependenciesComments
 VSPERF      
ovsdpdk






phy2phy_tput_p2p64, 128, 512, 1024, 151800:17:24Test runs successfully. We only report performance today






Doesn't impede the release






Until Moongen is integrated our dependency is on IXIA connected to POD 3








Can be reduced significantly if we iterate over less packets

In addition we have other tests that can be bound to 1 minute. These aren't running in the CI today but can be enabled easily.












back2back_p2p64, 128, 512, 1024, 151800:25:21
phy2phy_tput_mod_vlan_p2p64, 128, 512, 1024, 151800:28:20
phy2phy_scalability_p2p64, 128, 512, 1024, 151800:17:16
pvp_tput_pvp64, 128, 512, 1024, 151800:18:46
pvp_back2back_pvp64, 128, 512, 1024, 151800:36:56
pvvp_tput_pvvp64, 128, 512, 1024, 151800:36:14
pvvp_back2back_pvvp64, 128, 512, 1024, 151800:50:51
        
ovs






phy2phy_tput_p2p64, 128, 512, 1024, 151800:28:11Test runs successfully. We only report performance today






Doesn't impede the release






Until Moongen is integrated our dependency is on IXIA connected to POD 3








Can be reduced significantly if we iterate over less packets

In addition we have other tests that can be bound to 1 minute. These aren't running in the CI today but can be enabled easily.

back2back_p2p64, 128, 512, 1024, 151800:42:35
phy2phy_tput_mod_vlan_p2p64, 128, 512, 1024, 1518 00:27:54
phy2phy_scalability_p2p64, 128, 512, 1024, 151800:28:04
pvp_tput_pvp64, 128, 512, 1024, 151800:29:31
pvp_back2back_pvp64, 128, 512, 1024, 151800:43:49
pvvp_tput_pvvp64, 128, 512, 1024, 1518 
pvvp_back2back_pvvp64, 128, 512, 1024, 151800:45:31

Test

Case

Name

Approx.

Duration

(min)

Success

Criteria

Part of

Release

Criteria?

Dependencies
STORPERF
Initialization1 day  Installation
Cinder Volumes
SNIA Latency4h - 1dSteady state reached Initialization
SNIA IOPS4h - 1dSteady state reached Initialization
SNIA Throughput4h - 1dSteady state reached Initialization
CPERF (new project, tests under-dev)

CBench

("hello world")

20TBD perf numberNoodl_l3
NSTAT60TBD perf numberNoodl_l3
     
  • No labels

2 Comments

  1. Jose Lausuch can we move this page to a better place so we only list the releases on right hand side?

  2. Jose Lausuch Fatih Degirmenci I moved the page to the Colorado testing  page under testing