Page tree
Skip to end of metadata
Go to start of metadata

Dovetail Test Case Detailed Reports

 

Test case detail reports come from existing test projects that implemented the respective tests. A given report may or may not be suitable or sufficient for CVP purposes. This section is WORK IN PROGRESS.

Below are some known information about the status of these reports, as we make preliminary analysis of FuncTest and Yardstick test cases.

As our certification reports come from the report of test cases, we have to check those test cases one by one. And the most important data is the test case details. Test case details include its duration, test case names , etc. So i make a table to show the details of testcases.

  • duration: whether result has a duration of executing time
  • sub-testcase: whether result contains any sub-testcase
  • details on success: whether result contains details of sub-testcase, like testcase name, parent testcase, etc,  when pass
  • details on fail: whether result contains details of sub-testcase when fail

FuncTest Test Report Status

Test-case

duration

sub-testcase

details on pass

details on fail

healthcheck

    

tempest_smoke_serial

Y

Y

N

Y

vping_ssh

Y

N

N

N

vping_userdata

Y

N

N

N

rally_sanity

Y

Y

N

N

Tempest_full_parallel

    
rally_full    

Yardstick Test Report Status

Well, yardstick is much simpler than functest, it has same result format for each test case. 

The most important part of yardstick test result is the metrics of every sequence which we do not need it at all.

We just need to know that the test case pass or fail.

Dovetail Test Run Summary Report

The Dovetail Test Run Summary Report is to aid the determination of test result for CVP.

Shown below is a sample output of current version of implementation. It is WORK IN PROGRESS.

current style:

candidate style:

The report content needs more information to match , here is the candidate:

ComplianceDovetail Report

Version: 1.0 

TestSuite: proposed_set

Result dashboard: <link to test result dashboard>

CertifyValidation ID: 9b2a4144-21f8-7c2a-987b-8f2c125c2df2 

Create Date: 2016-11-01 20:27:21 UTC

Upload Date: 2016-11-01 20:27:21 UTC

Duration: 514 seconds  

 

Status:

Pass Rate: 0% (0/1)          #passed tests/ total tests

Compliance: NO

Overview:

dovetail.ipv6.tc001(2/3)     # passed sub tests/ total sub tests

-objective: VIM ipv6 operations, to create/delete network, port and subnet

- - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network pass

- - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port fail

- - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet pass

Assessed test areas:

-vimoperations results: <link to document section with details> pass % </link >

-networkoperations results: <link to document section with details> pass % </link >

-networkipv6 results: <link to document section with details> pass % </link >

-networkvpn results: <link to document section with details> pass % </link >

-networkbgp results: <link to document section with details> pass % </link >

etc ...

vimoperations: Pass rate % 

- <link to spec>dovetail.test.identifier1 </link >: <result(PASS/FAIL)> <link to test logs>

- <link to spec>dovetail.test.identifier2 </link >: <result> <link to test logs>

- <link to spec>dovetail.test.identifier3 </link >:  <result> <link to test logs>

 

networkoperations: Pass rate %

- <link to spec>dovetail.test.identifier1 </link >: <result> <link to test logs>

- <link to spec>dovetail.test.identifier2 </link >: <result> <link to test logs>

- <link to spec>dovetail.test.identifier3 </link >:  <result> <link to test logs>

 

networkipv6: Pass rate %

- <link to spec>dovetail.test.identifier1 </link >: <result> <link to test logs>

- <link to spec>dovetail.test.identifier2 </link >: <result> <link to test logs>

- <link to spec>dovetail.test.identifier3 </link >:  <result> <link to test logs>




  • No labels

1 Comment

  1. Leo wang what does Duration here mean? if it's the whole testsuite running time? does it include the docker downloading time?