Dovetail Test Case Detailed Reports

Test case detail reports come from existing test projects that implemented the respective tests. A given report may or may not be suitable or sufficient for CVP purposes. This section is WORK IN PROGRESS.

Below are some known information about the status of these reports, as we make preliminary analysis of FuncTest and Yardstick test cases.

As our certification reports come from the report of test cases, we have to check those test cases one by one. And the most important data is the test case details. Test case details include its duration, test case names , etc. So i make a table to show the details of testcases.

• duration: whether result has a duration of executing time
• sub-testcase: whether result contains any sub-testcase
• details on success: whether result contains details of sub-testcase, like testcase name, parent testcase, etc,  when pass
• details on fail: whether result contains details of sub-testcase when fail

FuncTest Test Report Status

Yardstick Test Report Status

Well, yardstick is much simpler than functest, it has same result format for each test case. 

The most important part of yardstick test result is the metrics of every sequence which we do not need it at all.

We just need to know that the test case pass or fail.

Dovetail Test Run Summary Report

The Dovetail Test Run Summary Report is to aid the determination of test result for CVP.

Shown below is a sample output of current version of implementation. It is WORK IN PROGRESS.

current style:

candidate style:

The report content needs more information to match , here is the candidate:

ComplianceDovetail Report

Version: 1.0 

TestSuite: proposed_set

Result dashboard: <link to test result dashboard>

CertifyValidation ID: 9b2a4144-21f8-7c2a-987b-8f2c125c2df2 

Create Date: 2016-11-01 20:27:21 UTC

Upload Date: 2016-11-01 20:27:21 UTC

Duration: 514 seconds  

Status:

Pass Rate: 0% (0/1)          #passed tests/ total tests

Compliance: NO

Overview:

dovetail.ipv6.tc001(2/3)     # passed sub tests/ total sub tests

-objective: VIM ipv6 operations, to create/delete network, port and subnet

- - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network pass

- - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port fail

- - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet pass

Assessed test areas:

-vimoperations results: <link to document section with details> pass % </link >

-networkoperations results: <link to document section with details> pass % </link >

-networkipv6 results: <link to document section with details> pass % </link >

-networkvpn results: <link to document section with details> pass % </link >

-networkbgp results: <link to document section with details> pass % </link >

etc ...

vimoperations: Pass rate % 

- <link to spec>dovetail.test.identifier1 </link >: <result(PASS/FAIL)> <link to test logs>

- <link to spec>dovetail.test.identifier2 </link >: <result> <link to test logs>

- <link to spec>dovetail.test.identifier3 </link >:  <result> <link to test logs>

networkoperations: Pass rate %

- <link to spec>dovetail.test.identifier1 </link >: <result> <link to test logs>

- <link to spec>dovetail.test.identifier2 </link >: <result> <link to test logs>

- <link to spec>dovetail.test.identifier3 </link >:  <result> <link to test logs>

networkipv6: Pass rate %

- <link to spec>dovetail.test.identifier1 </link >: <result> <link to test logs>

- <link to spec>dovetail.test.identifier2 </link >: <result> <link to test logs>

- <link to spec>dovetail.test.identifier3 </link >:  <result> <link to test logs>