Status of proposed tests

2018-06-25

Mandatory Tests

Installer

Scenario

vping_userdata

vping_ssh

stress.ping

tempest.osinterop

tempest.compute

tempest.identity_v3

tempest.image

tempest.network_api

tempest.volume

tempest.trunk_ports

tempest.ipv6_api

security.patrole

ha.nova_api

ha.neutron_server

ha.keystone

ha.glance_api

ha.cinder_api

ha.cpu_load

ha.disk_load

ha.haproxy

ha.rabbitmq

ha.database

Apex

nosdn-nofeature

fraser

PASS

FAIL

Unable to render Jira issues macro, execution error.

PASS

FAIL

HA proxy processes don't recover

PASS

Compass

nosdn-nofeature

fraser

PASS

FAIL

cannot connect to 192.168.11.xxx

PASS

total: 200

PASS

FAIL

Unable to render Jira issues macro, execution error.

PASS

Fuel

nosdn-nofeature

master

PASS

FAIL

SLA failure:
10.725534 > 5.000000

UNSTABLE

most of the time the test detects 11 keystone processes of which only 9 recover

Is the keystone process query correct?

PASS

FAIL

SLA failure:
12.149971 > 5.000000

PASS

FAIL

SLA failure:
12.336285 > 5.000000

rabbitmq-server not found
in the host environment

FAIL

SLA failure: 13.879296 > 5.000000

2018-06-25

Installer	Scenario	stress.ping	tempest.compute	tempest.identity_v2	tempest.identity_v3	tempest.image	tempest.network_api	tempest.volume	tempest.trunk_ports	security.patrole	ha.controller_restart	ha.rabbitmq	ha.database	ha.neutron_l3_agent	tempest.network_scenario	vnf.vEPC	vnf.vIMS	tempest.bgpvpn	snaps.smoke
Apex	nosdn-nofeature stable/fraser	FAIL	PASS	FAIL - only v3 used skip mechanism needed or just part of reviewer instructions (does not need to be automated) Unable to render Jira issues macro, execution error.	PASS	PASS	PASS	PASS	PASS	PASS	FAIL IPMI info missing	PASS	PASS	PASS	PASS	FAIL passes on Apex daily loop	PASS	not supported	FAIL test_volume_type_with_encryption test_volume_type_with_qos_and_encryption passes on Apex daily CI loop
	odl stable/fraser	FAIL	FAIL	FAIL - only v3 used skip mechanism needed or just part of reviewer instructions (does not need to be automated)			FAIL	FAIL	FAIL	FAIL	FAIL IPMI info missing	PASS	PASS	not supported: l3 agent not installed	FAIL	FAIL	FAIL	FAIL	FAIL
	nosdn-nofeature master	FAIL	PASS	not supported	PASS	PASS	PASS	PASS	PASS	PASS	FAIL IPMI info missing	PASS	PASS	PASS	PASS	PASS	FAIL VM in error state: "no valid host found" passes on Apex daily CI loop	not supported	FAIL
Compass	nosdn-nofeature stable/fraser	PASS create stack rate 5/5	PASS	SKIP	PASS	PASS	PASS	PASS	PASS skip: 9 pass: 43	PASS skip: 3 (dvr is not enabled) pass: 132 test_create_distributed_router ... skip test_show_distributed_router ... skip test_update_distributed_router ... skip	PASS	PASS	PASS	FAIL SLA failure: 9.023325 > 5.000000	FAIL fail: 5 No IPv4 addresses found	PASS	PASS	not supported	FAIL fail: 1 pass: 120 test_ssh_client_fip_after_reboot ... fail
Compass	odl stable/fraser	PASS create stack rate 5/5	PASS	SKIP	PASS	PASS	PASS	PASS	PASS skip: 9 pass: 43	PASS skip: 5 (l3_agent_scheduler extension not enabled) pass: 130 test_create_router_on_l3_agent ... skip test_delete_router_from_l3_agent ... skip test_list_routers_on_l3_agent ... skip test_create_high_availability_router ... skip test_update_high_availability_router ... skip	PASS	PASS	PASS	not supported: l3 agent not installed	FAIL fail: 2 (False is not true : ip address 192.168.11.114 is reachable) test_update_instance_port_admin_state test_update_router_admin_state	PASS	PASS	not supported	FAIL fail: 2 pass: 119 test_create_network_router_admin_user_to_new_project ... fail test_create_network_router_new_user_to_admin_project ... fail
Fuel (x86)	nosdn-nofeature master	PASS	PASS	FAIL - only v3 used skip mechanism needed or just part of reviewer instructions (does not need to be automated) Unable to render Jira issues macro, execution error.	PASS	PASS	PASS	PASS	PASS skip: 9 pass: 43	PASS skip: 3 (dvr is not enabled) pass: 132 test_create_distributed_router ... skip test_show_distributed_router ... skip test_update_distributed_router ... skip	PASS	FAIL rabbitmq located in msg0x VMs https://docs.opnfv.org/en/stable-fraser/submodules/fuel/docs/release/installation/installation.instruction.html#automatic-installation-of-a-baremetal-pod	FAIL database located in dbs0x VMs https://docs.opnfv.org/en/stable-fraser/submodules/fuel/docs/release/installation/installation.instruction.html#automatic-installation-of-a-baremetal-pod	FAIL l3-agent in gtw0x VMs https://gerrit.opnfv.org/gerrit/#/c/58333/	FAIL - Success: 3 - Skipped: 0 - Expected failures: 0 - Unexpected success: 0 - Failures: 2 test_update_instance_port_admin_state ... fail test_update_router_admin_state ... fail	FAIL auth problem passes on FUEL daily CI loop	FAIL auth problem deployment clearwater-opnfv timed out	not supported	PASS

Installer

Scenario

vping_ssh

vping_userdata

tempest.osinterop

tempest.ipv6

tempest.multi_node_scheduling

tempest.security

tempest.vm_lifecycle

ha.nova_api

ha.neutron_server

ha.keystone

ha.glance_api

ha.cinder_api

ha.cpu_load

ha.disk_load

ha.haproxy

sdnvpn.subnet_connectivity

sdnvpn.tenant_separation

sdnvpn.router_assoc

sdnvpn.router_assoc_fip

Apex

nosdn-nofeature

stable/fraser

PASS

FAIL

passes on Apex daily loop

FAIL

passes on Apex daily CI loop

FAIL

PASS

FAIL

PASS

FAIL

PASS

FAIL

IPMI info missing

PASS

not supported

Fuel

nosdn-nofeature

PASS

FAIL

PASS

FAIL

PASS

FAIL

PASS

not supported

Historical results below

2018-05-23 - limited results due to failed CI jobs

Installer	Scenario	vnf.vIMS	snaps.smoke	stress.ping	tempest.compute	tempest.identity_v2	tempest.identity_v3	tempest.image	tempest.network	tempest.volume	tempest.trunk_ports	tempest.bgpvpn	security.patrole	ha.controller_restart	ha.rabbitmq	ha.neutron_l3_agent	ha.database
Apex	nosdn-nofeature	FAIL Exception: Unable to create a number	PASS	FAIL	PASS	FAIL - only v3 used skip mechanism needed Unable to render Jira issues macro, execution error.	PASS	PASS	FAIL - Success: 19 - Skipped: 0 - Expected failures: 0 - Unexpected success: 0 - Failures: 1 test_mtu_sized_frames ... fail	PASS	PASS	not supported	PASS	FAIL	FAIL SLA failure: 31.512733 > 30.000000	FAIL stack create failed inner exception: file not exist	PASS
Apex	odl	FAIL	FAIL		PASS	FAIL	PASS	PASS	FAIL	PASS	PASS	FAIL	FAIL	FAIL	FAIL	l3 agent not installed	PASS
Compass	nosdn-nofeature
Compass	odl	FAIL	FAIL fail: 2 pass: 117	FAIL failed to create stacks	PASS	SKIP	PASS	PASS	FAIL fail: 3 pass: 17	PASS	PASS skip: 9 pass: 43	FAIL No .testr.conf file was found	PASS skip: 5 pass: 130	PASS	FAIL No rabbitmq-server process	FAIL No neutron-la-agent process	PASS
Fuel (x86)	nosdn-nofeature	FAIL deployment clearwater-opnfv timed out	PASS	PASS	PASS	SKIP	PASS	PASS	FAIL fail: 1 pass: 19 test_mtu_sized_frames ... fail	PASS	PASS skip: 9 pass: 43	FAIL No .testr.conf file was found	PASS	FAIL IPMI information missing	FAIL No rabbitmq-server process	FAIL failed to create stack with shade	FAIL No mysql process
Fuel (arm)

2018-05-16

Installer	Scenario	smoke.tc001	smoke.tc002	ha.tc009	ha.tc010	ha.tc011	vnf.tc001	stress.tc001	sdnvpn.tc000	security.patrole	tempest.neutron_trunk_ports
Apex	nosdn-nofeature	FAIL 0 failing tests 76 success 19 skipped: skip: Identity api v2 is not enabled skip: AccountQuotasTest skipped as swift is not available => implement mechanism for specifying skippable tests Unable to render Jira issues macro, execution error.	FAIL 1 failing test: PortConfigError: Cannot locate network with name - external in project - admin	FAIL ValueError: context not found for server 'node0.LF-8419b4c1' IPMI information missing?	FAIL missing ssh key in Yardstick FIX: https://gerrit.opnfv.org/gerrit/#/c/57401/	FAIL Unable to find network with name or id 'net04_ext' Env variables not passed correctly FIX: https://gerrit.opnfv.org/gerrit/#/c/57265/	FAIL failed to download file (temporary problem?)	FAIL TypeError: cannot concatenate 'str' and 'NoneType' objects => OS_TENANT_NAME not defined in OS credentials on Apex Unable to render Jira issues macro, execution error.	-- not supported by scenario --	PASS	-- not yet run --
Apex	odl	FAIL 0 failing tests 76 success 19 skipped: skip: Identity api v2 is not enabled skip: AccountQuotasTest skipped as swift is not available => implement mechanism for specifying skippable tests Unable to render Jira issues macro, execution error.	FAIL 1 failing test: PortConfigError: Cannot locate network with name - external in project - admin	FAIL ValueError: context not found for server 'node0.LF-8419b4c1' IPMI information missing?	FAIL missing ssh key in Yardstick FIX https://gerrit.opnfv.org/gerrit/#/c/57401/	FAIL Unable to find network with name or id 'net04_ext' Env variables not passed correctly FIX https://gerrit.opnfv.org/gerrit/#/c/57265/	FAIL out of floating IPs	FAIL TypeError: cannot concatenate 'str' and 'NoneType' objects => OS_TENANT_NAME not defined in OS credentials on Apex Unable to render Jira issues macro, execution error.	FAIL FIX: https://gerrit.opnfv.org/gerrit/#/c/57263/	PASS	-- not yet run --
Compass	nosdn-nofeature	FAIL 1 failing tests 75 success 19 skipped fail: No IPv4 address found	FAIL 2 failing tests fail: Guest does not have a console available. (HTTP 404) fail: network_id are not the same	PASS	PASS	FAIL SLA failure: 10.010200 > 5.000000	PASS (nosdn-nofeature) FAIL failed to download files	FAIL	---	PASS 0 failing tests 132 success 3 skipped skip: dvr is not enabled	PASS 0 failing tests 43 success 9 skipped skip: vlan, vxlan, gre skip: Advanced image is required to run this test
Compass	odl	FAIL 0 failing tests 76 success 19 skipped	FAIL 1 failing tests fail: <snaps.domain.network.Router instance at 0x7f4f262d6290> is not None	PASS	PASS	---	FAIL failed to download files	FAIL No valid host was found	---	FAIL
Fuel (x86)	nosdn-nofeature	FAIL 0 failing tests 76 success 19 skipped	PASS 119 success	FAIL IPMI information missing	FAIL Ssh is still unavailable: SSHError	FAIL failed to create stack with shade	FAIL failed to download files	PASS	---	PASS 135 success	PASS 0 failing tests 43 success 9 skipped skip: vlan, vxlan, gre skip: Advanced image is required to run this test
Fuel (arm)

2018-05-09

Installer	Scenario	smoke.tc001	smoke.tc002	ha.tc009	ha.tc010	ha.tc011	vnf.tc001	stress.tc001	sdnvpn.tc000	security.patrole	tempest.neutron_trunk_ports
Apex		FAIL 0 failing tests 76 success 19 skipped: skip: Identity api v2 is not enabled skip: AccountQuotasTest skipped as swift is not available	FAIL 1 failing test: PortConfigError: Cannot locate network with name - external in project - admin	FAIL ValueError: context not found for server 'node0.LF-8419b4c1' IPMI information missing	FAIL Ssh is still unavailable: SSHError	FAIL Unable to find network with name or id 'net04_ext' Env variables not passed correctly? Dovetail needs to pass the env file. We (Xudan) have seen this before.	PASS (nofeature) ? (odl)	FAIL TypeError: cannot concatenate 'str' and 'NoneType' objects	FAIL Unknown test case or tier 'bgpvpn', or not supported by the given scenario 'unknown'	PASS
Compass	nosdn-nofeature	FAIL 1 failing tests 75 success 19 skipped fail: No IPv4 address found	FAIL 2 failing tests fail: Guest does not have a console available. (HTTP 404) fail: network_id are not the same	PASS	PASS	FAIL SLA failure: 10.010200 > 5.000000	PASS (nosdn-nofeature) FAIL failed to download files	FAIL	---	PASS 0 failing tests 132 success 3 skipped skip: dvr is not enabled	PASS 0 failing tests 43 success 9 skipped skip: vlan, vxlan, gre skip: Advanced image is required to run this test
Compass	odl	FAIL 0 failing tests 76 success 19 skipped	FAIL 1 failing tests fail: <snaps.domain.network.Router instance at 0x7f4f262d6290> is not None	PASS	PASS	---	FAIL failed to download files	FAIL No valid host was found	---	FAIL
Fuel (x86)	nosdn-nofeature	FAIL 0 failing tests 76 success 19 skipped	PASS 119 success	FAIL IPMI information missing	FAIL Ssh is still unavailable: SSHError	FAIL failed to create stack with shade	FAIL failed to download files	PASS	---	PASS 135 success	PASS 0 failing tests 43 success 9 skipped skip: vlan, vxlan, gre skip: Advanced image is required to run this test
Fuel (arm)

Space shortcuts

Page tree

Historical results below