Page tree
Skip to end of metadata
Go to start of metadata

Definition

Availabilitythe proportion of time a system is in a functioning condition.

Robustnessthe degree to which a system or component can function correctly in the presence of invalid inputs or stressful environmental conditions.

Reliability: the ability (probability) of a system or component to function under stated conditions for a specified period of time.

Introduction

For Danube release, OPNFV testing group decides to extend the OPNFV testing scope by introducing basic stress test cases. Principles, requirements and test cases are discussed in Etherpad. Bottlenecks and Yardstick projects jointly implements some of the test cases while Bottlenecks project acts as the load manager calling yardstick to do the tests. These tests which are not Danube specific are aiming at testing the system for its breaking points and provide level of confidence of the system to users.

Stress testing is a software testing activity that determines the robustness of software by testing beyond the limits of normal operation. Stress testing is particularly important for  "mission critical" software, but is used for all types of software. Stress tests commonly put a greater emphasis on robustness, availability, and error handling under a heavy load, than on what would be considered correct behavior under normal circumstances.

 from wikipedia

Generally, stress testing tries to break the system in either positive or negative way. It tests for heavy loads and breaking points, i.e., the limit/bottleneck for the system. However, stress testing does not break the system for pleasure. By providing PASS or FAIL result. It will provide level of confidence of the system to users.

A) It also allows observation of how system react to the failures. Additional purpose behind this madness is to make sure that the system fails and recovers gracefully - a quality known as recoverability.  A list of questions could be raised to examine it    recoverability:

  1. What is the first thing crashed, how and why?
  2. Does it save its state or does it crash suddenly? 
  3. Does it just hang and freeze or does it fail gracefully? 
  4. Could the system/component recover smoothly?
  5. On restart, is it able to recover from the last good state? 
  6. Does it print out meaningful error messages to the user, or does it merely display incomprehensible hex codes?
  7. Is the security of the system compromised because of unexpected failures? 
  8. The list goes on. 
B) Moreover, apart from providing PASS or FAIL result. Stress testing can also provide more detailed result with reliability, the probability that system will survive during a given time interval. The reliability could be another indication of the level of confidence for the system.  

Stress Testing for OPNFV

For OPNFV,  stress test could find breaking points for deployments and feedback bugs. In this way to accelerate the maturity and promote the level of confidence of the platform.

More could be found in Euphrates Testing needs in Testperf proposed by Morgan requesting for long duration pod for stable release qualification.

Test Cases in Discussion

In the Etherpad, first stress test cases are discussed in Danube. A list of these test cases are summarized below which are focusing on performing data-plane traffic and life-cycle event testing.

(The test cases descriptions in dark color are abandoned and proposed by Bottlenecks projects)

Categories

Test Case

Description

Data-plane Traffic

for a virtual or bare metal POD

TC1 –Determine baseline for throughput

  • Initiate one v/b POD and generate traffic
  • Increase the package size
    • Throughput, latency and PKT loss up to x%

TC2 - Determine baseline for CPU limit

  • Initiate one v/b POD and generate traffic
  • Decrease the package size
    • Measure CPU usage up to x%

Life-cycle Events

for VM pairs/stacks

TC3 – Perform life-cycle events for ping

  • Spawn VM pairs, do pings and destroy VM pairs
  • Increase the number of simultaneous live VMs
    • latency and PKT loss up to x%
    • Testing time, count of failures, OOM killer?

TC4 – Perform life-cycle events for throughput

  • Spawn VM pairs, generate traffic and destroy VM pairs
  • Serially or paralleled increase the package size
    • Max load and PKT loss vs load
    • Throughput, latency and PKT loss up to x% for either pair

TC5 – Perform life-cycle events for CPU limit

  • Spawn VM pairs, generate traffic and destroy VM pairs
  • Serially or paralleled Decrease the package size
    • Measure the CPU usage up to x%

 

These test cases are illustrated below to better understand the testing mechanism behind them.

During Euphrates Release, more stress tests are planned (DRAFT, DETAILS ARE UNDER DISCUSSION):

A.  https://etherpad.opnfv.org/p/yardstick_release_e

  1. Scale-out test
  2. Scale-up test

B. StorPerf & VSPerf

  1. Scale out until maximum throughput reached
  2. Test of different Nova schedulers
  3. Run VSPERF and record numbers
  4. Run StorPerf and record numbers
  5. Run both at the same time and compare numbers.

C. Bottlenecks

  1. Implementation of TC3 based on 'lazy creation' of VM pair while measure the dataplane traffic

Implementation of Stress Testing

For Danube, 2 test cases has been implemented and merged into OPNFV CI pipeline.

  • (TC1) Baseline testing for throughput: measure the system throughput under increasing traffic stress for a single user (a virtual or bare metal POD).

  • (TC3) Life-cycle event for ping: by concurrently increasing the number of life-cycle events of multi-users (VM pairs/Stacks), it measure the stability of the system under large number of concurrent requests/traffic.

(TC1) Baseline testing for throughput

Implementation Mechanism

Raw Data

 

(TC3) Life-cycle event for ping

Implementation Mechanism

Raw Data

Num of Stack  12  16  20
 demo0-a739cc95228 secs28 secsdemo0-51a3c47e271 secs25 secsdemo0-b5436968305 secs42 secs
 demo0-140007f4303 secs29 secsdemo0-c32d6136251 secs31 secsdemo0-8ef0cf99362 secs33 secs
 demo0-31c3bd47419 secs25 secsdemo0-f279f41e238 secs28 secsdemo0-f83d563a225 secs39 secs
 demo0-28410038279 secs24 secsdemo0-809f9803129 secs39 secsdemo0-c8893740209 secs25 secs
 demo0-3a13f557443 secs27 secsdemo0-1c277b6e316 secs20 secsdemo0-f7e7c460299 secs36 secs
 demo0-ac6dbe45418 secs27 secsdemo0-9335d944245 secs24 secsdemo0-3d969441204 secs28 secs
 demo0-6c7624aa271 secs21 secsdemo0-cb969ba3138 secs35 secsdemo0-754d034c362 secs22 secs
 demo0-30e4a687399 secs20 secsdemo0-a5d1337c146 secs29 secsdemo0-19fa74f0170 secs39 secs
 demo0-7b9c6cad135 secs34 secsdemo0-82b6ef21220 secs20 secsdemo0-f1dee3f4343 secs22 secs
 demo0-824d4098136 secs32 secsdemo0-e0e32c78179 secs22 secsdemo0-a369add3307 secs45 secs
 demo0-530f94fe290 secs29 secsdemo0-f70d81b1112 secs44 secsdemo0-f5563f25299 secs42 secs
 demo0-b2a2942d362 secs22 secsdemo0-c5dbd4f1113 secs44 secsdemo0-c92bd889321 secs40 secs
    demo0-4522322b279 secs33 secsdemo0-99f0665c205 secs28 secs
    demo0-1665df83268 secs27 secsdemo0-ac007ea2311 secs25 secs
    demo0-65e974bf127 secs26 secsdemo0-682e54a0345 secs38 secs
    demo0-869c4cbc320 secs29 secsdemo0-c98826b1277 secs20 secs
       demo0-b2586df1198 secs35 secs
       demo0-e617af13299 secs31 secs
       demo0-d180f8b2281 secs22 secs
       demo0-ff700296326 secs24 secs

Throughput Soak Test for Lazy Creation Life-cycle Events

Implementation Mechanism

Measurements and Criterion

  • Using netperf under UDP protocol
    • generates traffic by random packets sizes
  • Measurements
    • Measure CPU usage both Traffic-gen VM and forwarder VM
    • Measure throughputs for both up and down link
    • Measure Sum Packet Loss Rate
    • Measure Latencies/average latency
    • VM pair creation time
  • Criterion
    • Labeled with Fail if Packet Loss (for either VM Pair) beyond x% for y testing time
    • Labeled with Fail if there is VM failure for y testing time

(TC4) Perform life-cycle events for throughputs (More details will be added later)

Implementation Mechanism

The testing procedure is similar to the throughputs soak test except:

  1. each VM pair is created when the throughput of proceeding VM pair reaches its limit
  2. and the VM pairs are detroies together when the system failed to create VM pairs or packet loss rate raise to x%.

Measurements and Criterion

  • Measurements
    • Measure CPU usage both Traffic-gen VM and forwarder VM
    • Measure throughputs for both up and down link
    • Measure Sum Packet Loss Rate
    • Measure Latencies/average latency
    • VM pair creation time
  • Criterion
    • Labeled with Fail if Packet Loss (for either VM Pair) beyond x% for y VM pairs
    • Labeled with Fail if there is VM failure for y VM pairs

Upstream Stress Testing Projects

Upstream stress testing projects deal with different systems/components

Materials

OPNFV Paris Plugfest (#3)

      Demo for OPNFV Plugfest Paris

      Stress Test Demo for Plugfest Paris.pptx

OPNFV Summit 2017 - Main Summit Talk

      Under Stress - How OPNFV React Beyond Breaking Points.pptx

Page viewed 280 times by 18 users since Jul 04, 2017

  • No labels