Page tree
Skip to end of metadata
Go to start of metadata

Project: OPNFV - Base system functionality testing (Functest)

  • Project Name: Functest
  • Project Repository: functest
  • Project Category: Integration & Testing
  • Project IRC chan: #opnfv-functest

Quick links

 

 

Project description

The  Functest project provides comprehensive testing methodology, test suites and test cases to test and verify OPNFV Platform functionality that covers the VIM and NFVI components.

This project uses a "top-down" approach that will start with chosen ETSI NFV use-case/s and open source VNFs for the functional testing. The approach taken will be to

  • break down the use-case into simple operations and functions required.
  • specify necessary network topologies
  • develop traffic profiles
  • develop necessary test traffic
    • Ideally VNFs will be Open Source however proprietary VNFs may also be used as needed.

This project will develop test suites that cover detailed functional test cases, test methodologies and platform configurations which will be documented and maintained in a repository for use by other OPNFV testing projects and the community in general. Developing test suites will also help lay the foundation for a test automation framework that in future can be used by the continuation integration (CI) project (Octopus). Certain VNF deployment use-cases could be automatically tested as an optional step of the CI process.

The project targets testing of the OPNFV platform in a hosted test-bed environment (i.e. using the OPNFV test labs world wide). It will leverage output of the different installer projects.

The key objectives are:

  • Define tooling for tests
  • Define test suites (SLA)
  • Installation and configuration of the tools
  • Automate test with CI
  • Provide API and dashboard functions for Functest and other test projects

A dedicated page for all the testing projects (Functest, Yardstick, Pharos, ...) is available here.

Scope

Functest delivers a functional testing framework along with a set of test suites and test cases to test and verify the functionality OPNFV platform.

The testing framework (tools, test-cases, etc.) is used by the CI framework for the purpose of qualifying the OPNFV platform on bare metal servers. In this context, OPNFV Tester will use open source VNF components. Functional testing includes

  • Testing the basic VIM functionality that includes tenant, user CRUD operations, VNF Image CRUD operations etc.
  • Testing the VIM functionality to support VNF operations (create, modify, grow, shrink, destroy)
  • Testing the VIM functionality to support basic VNF network connectivity
  • Testing the inter working between the VIM and the SDN controller.
  • Testing the SDN controllers
  • Testing the NFVI functionality as a black box to ensure that it meets the VIM requirements.

The project requires the following components:

  • OPNFV Lab setup with complete access to a set of Bare metal servers for Controller and Compute nodes (as defined by BGS project (OPNFV stack) and Pharos project (hardware)); associated switches and routers.
  • OPNFV platform software bundle from the repository that includes several upstream software components dealing with specific scenarios

Functest is

  • installer neutral
  • Controller neutral

Tooling

Tooling and test cases are installed on a docker file since Brahmaputra release.

Functional tests shall be

  • independent from the installer (Apex, Compass, Fuel,  Juju)
  • automated and integrated in CI

Functest versus Release

Testcases

Healthcheck

Testcase

Availability since

Average

duration

Comment

Success criteriaBlocking (y/n)
connection_checkDanube1'

This test case verifies the OpenStack connectivity

100% passedy
api_checkDanube2'

This test case verifies the retrieval of OpenStack clients: Keystone, Glance, Neutron and Nova and may perform some simple queries. When the config value of snaps.use_keystone is True, functest must have access to the cloud's private network.

100% passedy
snaps_healthcheckDanube
(replaces previous healthcheck from Colorado)
4' (TBC)basic tests of OpenStack main functions y


Smoke

Testcase

Availability since

Average

duration

Comment

Success criteriaBlocking (y/n)

vping_ssh

Brahmaputra

2'

Spawning 2 VMs, SCP ping script from the docker file, SSH on VM 1 to run the script, check that the Ping is OK

test passedy

vping_userdata

Arno

2'

basic hello world example, spawning 2 VMs, then Ping between the 2 VMs using cloudinit mechanism

test passedy

tempest_smoke_serial

Arno

25'

Smoke suite (165 tests)

100%

case exclusion must be motivated

 

rally_sanity

Arno

15'

Based on default Rally scenario.

100%

case exclusion must be motivated

 
refstack_defcoreDanubeTBDSubset of Tempest used for OpenStack certification ("Powered by OpenStack")TBD 

ODL

Arno

1'

21 L2 tests

Minor change for Brahamputra to fix Neutron/Keystone IP Address

100% passed 

ONOS

Brahmaputra

1'

20 L2 and L3 tests

100% passed 
snaps_smokeDanube7'

This test case contains tests that setup and destroy environments with VMs with and without Floating IPs with a newly created user and project. Set the config value snaps.use_floating_ips (True|False) to toggle this functionality. When the config value of snaps.use_keystone is True, functest must have access to the cloud's private network.

100% passed 

Features

Testcase

Availability

Duration

Comment

Success criteria

Promise

Brahmaputra

1'

Resource reservattion (fuel and joid only)

100% passed

Doctor

Brahmaputra

8'

Fault monitoring (Apex only)

100% passed

BGPVPN

Brahmaputra

20'

candidate: First tries on fuel but scenario not validated for Brahmaputra

100% passed
Security testsColorado5'Part of security project (temporarily hosted in Functest repo) run on apex for Coloradotest run
CopperColorado5'on Apex/Fuel100% passed
MoonColorado5'

1 error in ODL due to implementation choice, see release note

on Compass

100% passed
MultisiteColorado5'

Specific Tempest tests

Specific POD

On Fuel

100% passed
DominoColorado1'

On Joid

Possible to deploy manually but no full automation

 
ODL-sfcColorado?  
ONOS-sfcColorado15'  
ParserColorado3'On Fuel HA 
orchestraDanube?  

 

Components

Testcase

Availability

Duration

Comment

Success criteria
tempest_full_parallel

Colorado

?

 

80% passed

rally_full
Colorado? 90% passed

VNF

Testcase

Availability

Duration

Comment

Success criteria

cloudify_ims

Brahmaputra

30'

Orchestrator: 5

VNF dpeloyment: 15'

Tests: 10'

Deployment of clearwater vIMS done by Cloudify pre-study

Test introduced in CI in Colorado

Orchestrator and VNF deployed

tests run

orchestra_imsDanube?  
opera_imsDanube?  

vPE

?

 

pre-study

 

vPE perfo

?

 

pre-study

 
vEPC, vBBU, RRH, vUEPlanned D Release pre-study 

 

Study on VNF deployment

Tips

Short story on Functest versus the other test projects and the installers

First of all we try to be as agnostic as possible.

we see OPNFV as a black box.

We automatically run functest-{installer}-master after a successful fresh installation of {installer}
In functest (and in yardstick) we have internal test cases and companion test projects
the functest companion projects dealing with functional testing are

  • doctor
  • copper
  • ovno
  • policyTest
  • promise
  • sdnvpn

companion project means that we wil onboard them in Functest CI process. After a fresh install, functest will be started (a docker file) running several tests one after the others (e.g. https://build.opnfv.org/ci/view/functest/job/functest-fuel-master/lastSuccessfulBuild/consoleText) including companion tests in the near future.
each test may need specific env, so functest can install tooling and triggers scripts to setup env
we will also create a filtering mechanism to run tests only when possible (assuming that by default it is always possible) but no need to run odl suite towards an OPNFV including onos

In functest we developed the code for our internal tests (scripts, tests, integration of upstream test suites,...)

The companion projects are in charge of their scripts, their code, their scenario...
we mutualize the CI env (running on target CI production labs), the tooling (some projects need for instance to run upstream ODL suite, functest install once robot and mininet for all of them, Rally may test module used by feature project (eg; congress/copper)), the reporting, the documentation

Then jenkins functest-{installer} job will produce results that will be pushed in the Test collection DB
before Release B we will look carefully to the jobs (success criteria for Functest for B-Release is 4 successfull run on CI of the onboarded projects)

  • if errors in integration or internal cases => we will blame ourselves
  • if erros in companion projects => we will report the issues to the member of the feature projects

So you can see that we did not do specific stuff for 1 installer.
We have some adaptations of course in our scripts (e.g. to retrieve the credentials - not the same way on the different installers) but we do not create a plugin for a given installers, we have a docker file.
Functest (as yardstick) docker file could even be almost used towards any OPNFV-like solution (some adaptations on the env setup)

Git

Dealing with Branches

Best practice => Cherry Pick from Master to Stable useful patches as described in Stable Branch

If you forgot that (it may happen and we experienced that for SR1) as Master and stable may be considered as the same, you may have to merge Master on Stable...and then winter is coming...so a procedure that works:

  • git clone ssh://<Your_ID_assuming_you_have_the_merge_rights>@gerrit.opnfv.org:29418/functest functest-clean
  • git checkout stable/arno
  • git merge master
  • git push origin HEAD:refs/for/stable/arno.

Key Project Facts

Project: Base System Functionality Testing Project (functest)
Project Creation Date: January 20, 2015
Project Category: Integration & Testing
Lifecycle State: Incubation
Primary Contact: Cedric Ollivier (cedric.ollivier@orange.com)
Project Lead: Cedric Ollivier (cedric.ollivier@orange.com)
Jira Project Name: Base System Functionality Testing Project
Jira Project Prefix: FUNCTEST
Mailing list tag: [functest]
IRC: Server:freenode.net Channel:#opnfv-functest
Repository: functest

Committers:
Morgan Richomme <morgan.richomme@orange.com>
Jose Lausuch <jalausuch@suse.com>
Cedric Ollivier <cedric.ollivier@orange.com>
Helen Yao <helanyao@gmail.com>
Serena Feng <feng.xiaowei@zte.com.cn>
Juha Kosonen <juha.kosonen@nokia.com>
Valentin Boucher <valentin.boucher@kontron.com>
Viktor Tikkanen <viktor.tikkanen@nokia.com>
Mei Mei <meimei@huawei.com>
Linda Wang <wangwulin@huawei.com>

Additional contributors:
Georgios Paraskevopoulos <georgepar.91@gmail.com>
Romanos Skiadas <rom.skiad@gmail.com>
Michael Polenchuk <mpolenchuk@mirantis.com>
Cristina Pauna <cristina.pauna@enea.com>
Steven Pisarski <s.pisarski@cablelabs.com>

Link to TSC approval of the project: http://meetbot.opnfv.org/meetings/opnfv-meeting/2015/opnfv-meeting.2015-01-20-14.57.html

Link(s) to approval of additional committers:

http://lists.opnfv.org/pipermail/opnfv-tech-discuss/2015-April/001971.html
http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2015/opnfv-testperf.2015-09-29-13.00.html
http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2016/opnfv-testperf.2016-03-01-08.00.html
http://ircbot.wl.linuxfoundation.org/meetings/opnfv-functest/2016/opnfv-functest.2016-10-11-08.01.html

  • Additional Contributors
    • Frank Brockners (fbrockne@cisco.com)
    • Ryota Mibu (r-mibu@cq.jp.nec.com)
    • Hideyasu Hayashi (hideyasu.hayashi@okinawaopenlabs.org)

Tester Journey

  • No labels