Page tree
Skip to end of metadata
Go to start of metadata

This wiki page is to describe the process to integrate Qtip into Yardstick. It also serves as the procedure for other projects to follow in order to integrate with Yardstick.

Introduction

In OPNFV Colorado release, we have successfully integrate StorPerf into Yardstick, Yardstick treats StorPerf as a plugin and manage plugins via yardstick plugin CLI.

For more information on Yardstick plugin, please visit:

    https://wiki.opnfv.org/display/yardstick/How+to+install+a+plug-in+into+Yardstick

    http://artifacts.opnfv.org/yardstick/colorado/3.0/docs/userguide/index.html#document-08-yardstick_plugin

Qtip as a Yardstick plugin

Qtip has a flexible architecture to allow different deployment modes: standalone mode and agent mode[1]. The Yardstick-Qtip integration will used Qtip in the agent mode. That is , Qtip will act as a collector and reporter driven by Yardstick. Yardstick will act as a test runner to generate test results and trigger Qtip agent on the completion of test[2]. 

The integration procedure can follow a way similar to what we did in the StorPerf integration case. 

1. Deliver Yardstick and Qtip

Yardstick and Qtip are both delivered as Docker containers from https://hub.docker.com/u/opnfv/

 

2. Launch Qtip from Yardstick

We can use yardstick plugin CLI to manage Qtip Docker container.

Whether to launch Qtip will be determined by checking the existence of OS environ variable 'QTIP'. If it exists, Qtip will be launched by using Yardstick CLI 'yardstick plugin install'.


There are a collection of files that yardstick plugin CLI uses to manage plugins. Not all files used in build are listed here, these are the primary files.

plugin/qtip.yaml

A plugin configuration file for Qtip. This file is contains plugin name and plugin deployment information.

yardstick/resources/scripts/install/qtip.sh

Install script for launching Qtip Docker container.

yardstick/resources/scripts/remove/qtip.sh

Uninstall script for removing Qtip Docker container and resetting the environment that effected by the install script.

 

 


There are two yardstick plugin CLI available:

  • yardstick plugin install will invoke 'yardstick/resources/scripts/install/qtip.sh' to pull qtip Docker image,  configure and run the Docker container.
  • yardstick plugin remove will invoke 'yardstick/resources/scripts/remove/qtip.sh' to delete Qtip Docker container.


3. Yardstick interacts with Qtip

After Yardstick launching the Qtip container, Yardstick will execute the Qtip dedicated 'Qtip-PoC' test suite on the SUT and collect test data (metrics, hardware/software info, etc).

When ‘Qtip-Poc’ test suite is completed, Yardstick will do a 'post-test' process to collect data from Database or 'yardstick.out' and then post test data to Qtip via Qtip API.

Qtip agent will parse these data, calculate the QTIP performance index (QPI) and synthese a QPI report.

After the QPI is calculated, Yardstick will fetch the QPI report via Qtip API and push QPI data to database and dashboard for storage and visualization.

Yardstick-Qtip integration

 

4. Qtip API demand

To implement the workflow above, several Qtip API (RESTful preferred) are demanded.

Below is a sample:

Sample URLMethodSample payloadDescription

/api/v1.0/job

POST

{

   task_id: ******,

   testsuite_name: "Qtip_PoC",

   testcases:[

   "tc002": {

                 {"rtt": {"ares": 2.113}},

                 {"rtt": {"ares": 1.586}}

                },
  "tc005": {

                  {"read_bw": 3034, "read_iops": 758.51, "read_lat": 1308.38},

                  {"read_bw": 38586, "read_iops": 602.92, "read_lat": 1648.27}

                }],
  "env": {....................... } 

 ...................
}

Post Yarstick test data in JSON format to Qtip and trigger Qtip agent to calculate QPI.

Test data contains metrics and environment information.

Metrics data are key-value pairs. The 'key's are the test case No.

Content of the 'value's shown here is just a sample.

A task-id should be returned for querying the QPI result.

 GET

{
  "status": "successful",

   "project_name": "Qtip", 

"task_id": "#",

  "data": {

                 "QPI": "#",  

                 ...................

               }
}

or

{

   "status": "failed",

   "project_name": "Qtip",

   "task_id": "#",

    "error_msg": "*****************"

}

Fetch Qtip QPI result reporting.

A task-id will be used to refer a Qtip job.

The returned JSON should contains a job status (successful or failed) indicating if the QPI calculation is completed.



  DELETE 

 Terminate the currently running job.


5. Yardstick demand

From Yardstick point of view, a 'post-test' process need to be added at the end of Yardstick test suite execution. The main purpose of adding such a 'post-test' module is to provide flexibility for handling test results.

This 'post-test' process will act as a service, the content can be extended and customised. To use this service, a python file is required.

In the Qtip integration case, this 'post-test' process will be used to collect all test cases' data related to the executed 'Qtip-PoC' test suite, this 'post-test' process will also trigger data-posting via Qtip api and fetch QPI result from Qtip.


[1] https://wiki.opnfv.org/display/qtip/Architecture

[2] QTIP design spec

6 Comments

  1. Jing Lu the calculation of QPI is a synchronous procedure which means it could be included in the response to the yardstick test result POST. So sample 2 may not be necessary.

    1. Thanks for the feedback. I will amend the API sample as you suggested.

  2. I think method GET is necessary. But the status could not be a intermediate state, I suggest it returns "successful" and "failed" to show the result of QPI caculation.

    request:

    {

    "project_name": "Qtip"

    "task_id": ***

    }

    respond:

    {

        "status": "successful",

        "project_name": "Qtip"

        "task_id": ***

         "data": { "QPI": ***}

    }

    or

    {

       "status": "failed",

        "project_name": "Qtip"

        "task_id": *,

         "error_msg": *****************

    }

     

     

      1. I've seen this, and I agree with you that zhihui's idea is more reasonable.

  3. Jing Lu as suggested in last Yardstick meeting, I've updated QTIP architecture and introduce a even more simplified plugin mode. Could you please update the proposal on yardstick part?

    cc Kubi