Test case detail reports come from existing test projects that implemented the respective tests. A given report may or may not be suitable or sufficient for CVP purposes. This section is WORK IN PROGRESS.
Below are some known information about the status of these reports, as we make preliminary analysis of FuncTest and Yardstick test cases.
As our certification reports come from the report of test cases, we have to check those test cases one by one. And the most important data is the test case details. Test case details include its duration, test case names , etc. So i make a table to show the details of testcases.
Test-case | duration | sub-testcase | details on pass | details on fail |
healthcheck | ||||
tempest_smoke_serial | Y | Y | N | Y |
vping_ssh | Y | N | N | N |
vping_userdata | Y | N | N | N |
rally_sanity | Y | Y | N | N |
Tempest_full_parallel | ||||
rally_full |
Well, yardstick is much simpler than functest, it has same result format for each test case.
The most important part of yardstick test result is the metrics of every sequence which we do not need it at all.
We just need to know that the test case pass or fail.
The Dovetail Test Run Summary Report is to aid the determination of test result for CVP.
Shown below is a sample output of current version of implementation. It is WORK IN PROGRESS.
The report content needs more information to match , here is the candidate:
ComplianceDovetail Report
Version: 1.0
TestSuite: proposed_set
Result dashboard: <link to test result dashboard>
CertifyValidation ID: 9b2a4144-21f8-7c2a-987b-8f2c125c2df2
Create Date: 2016-11-01 20:27:21 UTC
Upload Date: 2016-11-01 20:27:21 UTC
Duration: 514 seconds
Status:
Pass Rate: 0% (0/1) #passed tests/ total tests
Compliance: NO
Overview:
dovetail.ipv6.tc001(2/3) # passed sub tests/ total sub tests
-objective: VIM ipv6 operations, to create/delete network, port and subnet
- - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network pass
- - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port fail
- - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet pass
Assessed test areas:
-vimoperations results: <link to document section with details> pass % </link >
-networkoperations results: <link to document section with details> pass % </link >
-networkipv6 results: <link to document section with details> pass % </link >
-networkvpn results: <link to document section with details> pass % </link >
-networkbgp results: <link to document section with details> pass % </link >
etc ...
vimoperations: Pass rate %
- <link to spec>dovetail.test.identifier1 </link >: <result(PASS/FAIL)> <link to test logs>
- <link to spec>dovetail.test.identifier2 </link >: <result> <link to test logs>
- <link to spec>dovetail.test.identifier3 </link >: <result> <link to test logs>
networkoperations: Pass rate %
- <link to spec>dovetail.test.identifier1 </link >: <result> <link to test logs>
- <link to spec>dovetail.test.identifier2 </link >: <result> <link to test logs>
- <link to spec>dovetail.test.identifier3 </link >: <result> <link to test logs>
networkipv6: Pass rate %
- <link to spec>dovetail.test.identifier1 </link >: <result> <link to test logs>
- <link to spec>dovetail.test.identifier2 </link >: <result> <link to test logs>
- <link to spec>dovetail.test.identifier3 </link >: <result> <link to test logs>