Page tree
Skip to end of metadata
Go to start of metadata

For Colorado we created an automatic reporting page. This page is dynamically created through a cron job (update every day) and is based on the results stored in the Test DB.

You can access this reporting page: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-compass.html

There is one page per installer. For each installer we list the scenarios run from CI since 10 days.

The scenario summary including the scenario name, its status, a scoring of last 4 iterations and a scoring of last 10 days.

The scenario is detected through one test case that is supposed to be run all the time. If for any reason, a scenario crashed before running this test , then the scenario will not be referenced.

The number of cases differ from one scenario to another (and is sometimes also different from one installer to another). It is due to the dependencies declared by the case owner (e.g. onos case has no sense on no onos scenarios). The test criteria have been discussed in Test Release Criteria

The results are all coming from the database and may come from different pods (bm or virtual). The results are be pushed from CI to the database if the pod id declared in the test DB: http://testresults.opnfv.org/test/api/v1/pods

The scoring is a basic calculation: we consider the results for each test case and we sum the scoring of each case to give a scenario scoring.

Points are attributed as follow:

NumberStatusCommentPointPercent
1test never reached the success criteria since 10 days.0

0%

(0/6)

2

test never reached the success criteria in the last 4 iterations,

but success at least once within the last 10 days.

1

16.7%

(1/6)

3

test reach the success criteria once in the last 4 iterations.

 

2

33.3%

(2/6)

4

test reach the success criteria twice in the last 4 iterations.

 

3

50.0%

(3/6)

5

test reach the success criteria three times in the last 4 iterations,

but never success four times in a row in the last 10 days.

4

66.7%

(4/6)

6

test reach the success criteria four times in a row in the last 10 days,

but failed at least once in the last 4 iterations.

5

83.3%

(5/6)

7test reach success criteria all in the last 4 iterations.6

100%

(6/6)

As an illustration if we consider the scenario os-odl_l3-nofeature-ha(source 28/10/2016), we can see that the scoring is 3/4 in the last 4 iterations, and 7/9 in the last 10 days.

So its point is 4, and the status is number 5.

As for the trend, since we the reporting page update every day, we record the every day percent. So one dot on the trend image means the percent for that day. If we connect all this dots, we get the trend image.

Now on the 28/10 12:07 UTC, we list the last 4 results stored in the database, the 4th one was FAILED.

{
"results": 
[
{"project_name": "yardstick", "stop_date": "02:00:22", "case_name": "scenario_status", "build_tag": null, "pod_name": "huawei-virtual2",
 "installer": "compass", "scenario": "os-odl_l3-nofeature-ha", "trust_indicator": {"current": 0, "histories": []}, "version": "colorado", 
"details": "", "criteria": "SUCCESS", "_id": "581171861d2c6e000ab2721f", "start_date": "2016-10-27"
}, 
{"project_name": "yardstick", "stop_date": "20:35:35", "case_name": "scenario_status", "build_tag": null, "pod_name": "huawei-virtual3", 
"installer": "compass", "scenario": "os-odl_l3-nofeature-ha", "trust_indicator": {"current": 0, "histories": []}, "version": "colorado", 
"details": "", "criteria": "SUCCESS", "_id": "581279931d2c6e000ab27519", "start_date": "2016-10-27"
}, 
{"project_name": "yardstick", "stop_date": "04:37:08", "case_name": "scenario_status", "build_tag": null, "pod_name": "huawei-virtual1", 
"installer": "compass", "scenario": "os-odl_l3-nofeature-ha", "trust_indicator": {"current": 0, "histories": []}, "version": "colorado",
"details": "", "criteria": "SUCCESS", "_id": "580fd65c1d2c6e000ab26ed1", "start_date": "2016-10-26"
}, 
{"project_name": "yardstick", "stop_date": "04:29:17", "case_name": "scenario_status", "build_tag": null, "pod_name": "huawei-virtual1", 
"installer": "compass", "scenario": "os-odl_l3-nofeature-ha", "trust_indicator": {"current": 0, "histories": []}, "version": "colorado", 
"details": "", "criteria": "FAILED", "_id": "580e7a961d2c6e000ab26b5c", "start_date": "2016-10-25"
}
]
}

Note that if the scenario is validated (full scoring), we save the name, installer and date. So if scenario is not run anymore and/or regression occured we keep the trace of success.

By default we enable them into the automatic reporting.

 

  • No labels