Page tree
Skip to end of metadata
Go to start of metadata

Data updated weekly

NEW as of:  3/31/2020

Consolidated requirements (RM, RA-1) for NFVI and VNF are in RI Chapter 3:  

For Baldy, our goals are:

  • Ensure RI-1 implementation matches RM and RA-1 requirements (config and deployment)
  • Ensure RC-1 suites validate both Application (OpenStack) and CNTT (config, capabilities. Extensions, etc) requirements

Summary:

  • Total: 84 requirements
    • 50/84 MUST - 60%
      • MUST:
        • 25/50 - are in Functest 50% (Cedric will check, for excample )
        • 6/50 - covered by a combination of Performance tools
        • 4/50 - covered by Yardstick HA tests ( general Yardstick Test Case info here)
        • 1/50 - covered by Yardstick IPv6 connectivity (#40 Dual stack v4 + v6)  Is this a MUST?? Is it covered by Functest? yes currently disabled.
        • 1/50 - covered by Yardstick compute Capacity (#12)
        • Total - 37 MUST Requirements covered or In-progress
        • More Work:
          • 3/50 - Need new tool e.g., Network topology discovery and/or Neutron Resilience in Yardstick #31, Undercloud encap in #83 and #84
          • 1/50 - Functional checks (#29 VXLAN,)
          • Total - 4 MUST Requirements
        • Feedback: Note Clarify under status
          • 1/50 - Not Testable (#34)
          • 3/50 - Clarification Needed (#12, #53, #71)
          • 6/50 - Clarification Needed (Security, 77, 78, 79, 80, 81, 82)
          • 1/50 - Duplicates (#26, Should, covered by Functest, and #39, Must: Support for Network Options, such as OVS, SR-IOV, DPDK)
          • Total - 11 MUST Requirements (some overlap with "More Work" category)
      • 30/84 SHOULD/MAY - 40%
  • Categories: Functional, High Availability (HA), Performance,  (NFR replaced with something meaningful).
  • Testable: either Testable, Requirement Not Testable but Verifiable (Verifiable), Not Testable
  • CRUD: Create, Read, Update, and Delete
  • BLUF:
    • HDV and SDV prototypes being uploaded to Repo - will aid with baremetal validation gaps
    • 23 of 50 (46%) of MUST requirements reviewed for test interpretation, 100% Testable <<< ALL Interpretations and Testable Verdicts Under Review
    • Additional pass to be made by Functest PTL to verify coverage
    • To address 50% coverage gap (all in progress):
      • Vet Functest coverage
      • Confirm requirement is testable 
      • On-board existing OPNFV projects (6 under review)
      • Add HDV and SDV verification tools to address baremetal and manifest validations
      • Add NFVI and VIM verification from Spirent (on-boarding to XTesting)

RequirementDescription

Test-Req Interpretation

(new)

Testable?

(new)

CatalogStatusMUSTCategoryRI-1 Implemented (Confirmed)RC-1 Coverage (Today)

RC-1 Project To On-board

(Proposed)

Notes
1req.gen.ost.01must use OpenStack APIs.

Validate API function properly for Opentack service like keystone, Nova, Neutron, glance, Cinder, Swift etc.  

TestableGeneralin FunctestMustFunctional
Functest

2req.gen.ost.02must support dynamic request and configuration of virtual resources through APIs.Validate CRUD works properly for Openstack APIs. TestableGeneralin FunctestMustFunctional
Functest

3req.gen.cnt.01should consist of stateless service components. However, where state is required it must be kept external to the components.

General
Should




4req.gen.cnt.02should consist of service components implemented as microservices that are individually dynamically scalable.

General
Should




5req.gen.scl.01should support policy driven auto-scaling.

General
Should




6req.gen.rsl.01must support resilient OpenStack components that are required for the continued availability of running workloads.Validate Openstack API still works properly for rainy day scenario like compute reboot, service restart, etc.TestableGeneral - Resiliency testingmissingMustHA

Yardstick TC025 and/or TC045Also TC046-TC054
7req.gen.rsl.02should support resilient OpenStack service components that are not subject to req.gen.rsl.01.

General - Resiliency testingmissingShouldHA

Yardstick TC025 and/or TC045Also TC046-TC054
8req.gen.avl.01must provide High Availability for OpenStack components.Validate Openstack service are configured in HA mode, either active-active or active-standby mode.VerifiableGeneral - HANeeds Configuration checkMustHA

Yardstick (provides evidence)Also TC046-TC054
9req.inf.com.01must provide compute resources for VM instances.Validate VM CRUD works properly.TestableInfrastructurein FunctestMustFunctional
Functest

10req.inf.com.02should include industry standard hardware management systems at both HW device and platform level

Infrastructure - baremental testingmissing in Functest. Captured in Telco TCs Baremetal - validations. Should




11req.inf.com.03should support symmetrical CPU multi-processing with shared memory access as well as multi-threading.

Infrastructurein FunctestShould

Functest

12req.inf.com.04must be able to support multiple CPU SKU options to support Base, Network Intensive, and Compute Intensive infrastructure profiles.Validate hardware deployment and Openstack flavor series support Base, Network Intensive, and Compute Intensive host profile VerifiableInfrastructureCLARIFY - Needs Configuration checkMustPerformance

Yardstick TC055This requirement seems to indicate running a deploy on mutliple hardware versions / SKUs representing base, compute, network intensive.  As currently written, this may not be well understood or testable.
13req.inf.com.05must support Hardware Platforms with NUMA capabilities.Validate hardware deployment, Openstack settings, and flavor series support NUMA alignment settings. TestableInfrastructure - baremental testing

VSPERF TCs exist.

missing or incomplete in Functest. Captured in Telco TCs Baremetal - validations. 

MustPerformance
VSPERF
See TST009, Annex
14req.inf.com.06must support CPU Pinning.Validate hardware deployment, Openstack settings, and flavor series support CPU Pinning settings. TestableInfrastructure - baremental testing

VSPERF TCs exist.

missing or incomplete in Functest. Captured in Telco TCs Baremetal - validations. 

MustPerformance
VSPERF
See TST009, Annex
15req.inf.com.07must support different hardware configurations to support Base, Network Intensive, and Compute Intensive infrastructure profiles.Validate hardware deployment, Openstack settings, and flavor series support Host Aggregate and Availability Zone for Base, Network Intensive, and Compute Intensive infrastructure profilesTestableInfrastructure - baremental testing

VSPERF TCs exist.

missing or incomplete in Functest. Captured in Telco TCs Baremetal - validations. 

MustPerformance
VSPERF
See TST009, Annex
16req.inf.stg.01must provide shared Block storage for VM Instances.Validate Openstack Block Storage service, e.g.,  Cinder API CRUD, works properlyTestableInfrastructurein FunctestMustFunctional
Functest

17req.inf.stg.02must provide shared Object storage for VM Instances.Validate Openstack Object Storage service, e.g. Swift API CRUD, works properlyTestableInfrastructurein FunctestMustFunctional
Functest

18req.inf.stg.03may provide local file system storage solution for VM Instances.Validate Openstack Object local file storage service, e.g., local disk service, works properlyTestableInfrastructure - baremental testingmissingShould (May)




19req.inf.stg.04may support Software Defined Storage (SDS) that seamlessly supports shared block storage, object storage and flat files.

Infrastructure - baremental testingmissingShould (May)




20req.inf.stg.05should be able to accommodate VNFs that store back into its image through use of hypervisor attached volumes.

Infrastructurein FunctestShould

Functest

21req.inf.stg.06should make the immutable images available via location independent means.

Infrastructurein FunctestShould

Functest

22req.inf.stg.07should provide high-performance and horizontally scalable VM storage.

Infrastructure - performance testingmissingShould




23req.inf.stg.08should allow use of externally provided large archival storage for its Backup / Restore / Archival needs.

Infrastructure - baremental testingmissing in Functest. Captured in Telco TCs Baremetal - validations. Should




24req.inf.stg.09should make available all non-host OS / Hypervisor / Host systems storage as network-based Block, File or Object Storage for tenant/management consumption.

Infrastructurein FunctestShould

Functest

25req.inf.ntw.01must provide virtual network interfaces to VM instances.Validate Openstack Network service, e.g., Neutron API CRUD, works properlyTestableInfrastructurein FunctestMustFunctional
Functest
Instantiation of a VM with network connectivity verifies this.
26req.inf.ntw.02must include capabilities for integrating SDN controllers to support provisioning of network services, from the OpenStack Neutron service, such as networking of VTEPs to the Border Edge based VRFs.Validate Openstack Neutron network plugins, e.g., Open vSwitch, SR-IOV, etc,, as well as service plugins, e.g., FWaaS (Firewall as a Service), LBaaS (Load Balancer as a Service), and VPNaaS (VPN as a Service), etc. TestableInfrastructurein FunctestMustFunctional
Functest

27req.inf.ntw.03must support low latency and high throughput traffic needs.Validate Openstack Neutron network service support low latency and high throughput, .e.g, DPDK or SR-IOV.  Testableperformance testingVSPERF, PROX, and NFVBench coverage.MustPerformance

VSPERF

PROX

NFVBench


28req.inf.ntw.04should support service function chaining.

InfrastructuremissingShould




29req.inf.ntw.05must allow for East/West tenant traffic within the cloud (via tunnelled encapsulation overlay such as VXLAN or Geneve).Validate Openstack east/west traffic between VMs that located in compute nodes in the same data center Testable - Validate connectivity using tunnels or encaps between VMs in diff hostsInfrastructuremissingMustFunctional



30req.inf.ntw.06should support Distributed Virtual Routing (DVR) to allow compute nodes to route traffic efficiently.

InfrastructuremissingShould




31req.inf.ntw.07must support network resiliency.Validate network resiliency, e.g., each server is dual homed to TOR (Leaf/Access) switches with redundant (2x) connections. The Leaf switches are dual homed with redundant connections to spinesTestable - Validate network connectivity using Open Souce mapping toolsInfrastructure - resiliency testingmissingMustHA

Yardstick TC045 Tests Neutron Failures

Some of the options we have:

  1. Run LLDPtool on compute and control nodes – easiest to implement, at least for CNTT pods. But, it can get complex, if the number of nodes are more.
  2. LLDP on TORs - Reuse the solutions that used by ‘data centers’ – opensource and scalable. It may take slightly more time to bring it CNTT.
32req.inf.ntw.08The NFVI Network Fabric should embrace the concepts of open networking and disaggregation using commodity networking hardware and disaggregated Network Operating Systems.
Not TestableInfrastructuremissingShould




33req.inf.ntw.09The NFVI Network Fabric should embrace open-based standards and technologies.
Not TestableInfrastructuremissingShould




34req.inf.ntw.10The NFVI Network Fabric must be capable of supporting highly available (Five 9’s or better) VNF workloads.Validate NFVI Network Fabric must be capable of supporting highly available (Five 9’s or better) VNF workloads.Not Testable in reasonable time frames 5 Nines = 53 minutes downtime PER YEAR, average
Infrastructure - performance testingmissingMustHA



35req.inf.ntw.11The NFVI Network Fabric should be architected to provide a standardised, scalable, and repeatable deployment model across all applicable NFVI sites.
VerifiableInfrastructureArchitecture and Standards not testable.Should




36req.inf.ntw.12The SDN solution should be configurable via orchestration or VIM systems in an automated manner using openly published API definitions.
VerifiableInfrastructure - SDNmissing or incomplete. All captured in ODL?Should



Orchestration is beyond NFVI and OPNFV scope.
37req.inf.ntw.13The SDN solution should be able to support federated networks.
??Infrastructure - SDNmissing or incomplete. All captured in ODL?Should




38req.inf.ntw.14The SDN solution should be able to be centrally administrated and configured.
??Infrastructure - SDNmissing or incomplete. All captured in ODL?Should




39req.inf.ntw.15must support multiple networking options for NFVI to support Base, Network Intensive, and Compute Intensive infrastructure profiles.Validate Openstack neutron support network options, e..g, Open vSiwtch SR-IOV, DPDK, according to Base, Network Intensive, and Compute Intensive infrastructure profilesTestableInfrastructureCLARIFY - missing or incompleteMustFunctional


This is covered by Functest elsewhere, what is different about this requirement?

See req.inf.ntw.02,

#26

40req.inf.ntw.16must support dual stack IPv4 and IPv6 for tenant networks and workloads.Validate VM able to communicate with each using IPv4 or IPv6. Testable - VerifiableInfrastructuremissing or incompleteMustFunctional

Yardstick TC027
41req.inf.ntw.17should use dual stack IPv4 and IPv6 for NFVI internal networks.

Infrastructuremissing or incompleteShouldFunctional



42req.inf.acc.01should support Application Specific Acceleration (exposed to VNFs).

Infrastructuremissing in Functest. Captured in Telco TCs Openstack -
VNF Interoperability - validations 
ShouldFunctional



43req.inf.acc.02should support NFVI Acceleration (such as SmartNICs).
TestableInfrastructureComparison  testing with VSPERF, PROX, NFVBenchShouldPerformance

VSPERF, PROX, NFVBench
44req.inf.acc.03should not rely on SR-IOV PCI-Pass through to provide acceleration to VNFs.Q: What Kind of Acceleration ??????InfrastructureClarifyShouldPerformance


SAME AS #58 req.int.acc.02
45req.vim.01must allow infrastructure resource sharing.Validate Openstack support multi-tenants and tenants can create VMs in the same compute server.TestableVIMin FunctestMustFunctional
Functest

46req.vim.02should support deployment of OpenStack components in containers.

VIMin FunctestShould

Functest

47req.vim.03must allow VIM to discover and manage NFVI resources.

Validate Openstack placement API works properly to place VM to a specific server, host aggregate or Availability zone.

TestableVIMin FunctestMustFunctional
Functest

48req.vim.04must support Enhanced Platform Awareness (EPA).UnclearTBDVIMin FunctestMustFunctional
Functest

49req.vim.05must include image repository management.Validate Openstack is able place VM to a specific server, host aggregate or Availability zone.TestableVIMin FunctestMustFunctional
Functest

50req.vim.06must allow orchestration solutions to be integrated with VIM.Validate Openstack orchestration work properly, e.g., Horizon.TestableVIMin FunctestMustFunctional
Functest

51req.vim.07must support a multi-tenanted environment.Validate Openstack support multi-tenants and projects CRUD.TestableVIMin FunctestMustFunctional
Functest

52req.vim.08must support resource tagging.Validate Openstack tagging, e.g., instance tagging, or neutron network tagging.TestableVIMin FunctestMustFunctional
Functest

53req.vim.09must support horizontal scaling.Validate Openstack horizontal scaling, e.g, adding more servers to cluster.TestableVIM - performance testing ??
missingMust

Functional (Scaling function)



Yardstick TC057 ?? no doc??

PROX ???

NFVBench??

Adding Idle Servers to a POD??
54req.int.api.01must provide Control API endpoints to cloud platform core services.Validate API function properly for Opentack service like keystone, Nova, Neutron, glance, Cinder, Swift etc.  TestableInterface & APIin FunctestMustFunctional
Functest

55req.int.api.02must provide GUI access to tenant facing cloud platform core services.

Validate Openstack horizon GUI dashboard works properly

TestableInterface & APIin FunctestMustFunctional
Functest

56req.int.api.03must provide APIs needed to discover and manage NFVI resources.

Validate Openstack placement API works properly to place VM to a specific server, host aggregate or Availability zone.

TestableInterface & APIin FunctestMustFunctional
Functest

57req.int.acc.01should provide an open and standard acceleration interface to VNFs.
??? What is the "standard" acc;leration interface???Interface & APIin FunctestShould

Functest
??? What is the "standard" acc;leration interface???
58req.int.acc.02should not rely on SR-IOV PCI-Pass through for acceleration interface exposed to VNFs.Q: What Kind of Acceleration ??????InfrastructureClarifyShould



SAME AS #44 req.inf.acc.03
59req.tnt.gen.01must support multi-tenancy.Validate Openstack support multi-tenancy, e.g., users and projects CRUD works properly. TestableTenantsin FunctestMustFunctional
Functest

60req.tnt.gen.02must support self-service dashboard (GUI) and APIs for users to deploy, configure and manage their workloads.Validate Openstack horizon dashboard GUI works properly.TestableTenantsin FunctestMustFunctional
Functest

61req.lcm.gen.01must support zero downtime expansion/change of physical capacity (compute hosts, storage increase/replacement).Validate Openstack hardware rainy day scenario works properly without service interruption, e.g., during server rebooting, storage replacement. TestableLCM - performance testingmissingMustHA

Yardstick TC025TC025 measures downtime for host reboot
62req.lcm.adp.01should allow for “cookie cutter” automated deployment, configuration, provisioning and management of multiple NFVI sites.

LCMuse airshipShould




63req.lcm.adp.02must support hitless upgrades of software provided by the cloud provider so that the availability of running workloads is not impacted.Validate Openstack software rainy day scenario works properly without service interruption. TestableLCM - performance testingmissing or incomplete. Use airship?MustHA

Yardstick?


64req.lcm.adp.03should support hitless upgrade of all software provided by the cloud provider that are not covered by req.lcm.adp.02. Whenever hitless upgrades are not feasible, attempt should be made to minimize the duration and nature of impact.

LCMuse airshipShould




65req.lcm.adp.04should support declarative specifications of hardware and software assets for automated deployment, configuration, maintenance and management.

LCMuse airshipShould




66req.lcm.adp.05should support automated process for Deployment and life-cycle management of VIM Instances.

LCMuse airshipShould




67req.lcm.cid.02should support integrating with CI/CD Toolchain for NFVI and VIM components Automation.

LCMuse X-testingShould




68req.asr.mon.01must include integration with various infrastructure components to support collection of telemetry for assurance monitoring and network intelligence.Validate Openstack telemetry works properly, using solutions like Aodh, ceilometer, gnocchi etc.TestableAssurancein FunctestMustFunctional
Functest

69req.asr.mon.02should support Network Intelligence capabilities that allow richer diagnostic capabilities which take as input broader set of data across the network and from VNF workloads.????Assurance - operationCLAIRFYShould




70req.asr.mon.03must allow for the collection and dissemination of performance and fault information.

Validate Openstack performance and fault info are collected properly, using solutions like prometheus or nagios, etc. 


TestableAssurance - operationin combination of Barometer and VSPERFMustFunctional and Performance

VSPERF and Barometer combinedCurrent metrics possible are via collectd-openstack plugin extensions, libvirt metrics, OVS & DPDK metrics. More performance metrics required need to be clarified and integrated.
71req.asr.mon.04The NFVI Network Fabric and Network Operating System must provide network operational visibility through alarming and streaming telemetry services for operational management, engineering planning, troubleshooting, and network performance optimisation.Validate alarming and telemetry for baremental server or physical network works properly, using solution like prometheus or nagios. TestableAssurance - operationCLARIFY - alarming and reporting on barometer via prometheus already available via barometer work. Requirements not clear to find out whats needed of physical networkMustFunctional

alarming and reporting on Barometer via prometheus already available via barometer work.

See #70 for overlap

Expecting some input from sunku ranganath

Alarming, reporting, visibility of baremetal nodes, NIC interfaces & virtual network (vswitch, dpdk, libvirt) available through barometer project. However physical network stats from switch and routing telemetry, are not currently available via barometer

72req.sec.gen.01must provide tenant isolation.Validate Openstack tenant or project can only access  CRUD its own resource but not other tenant or project resourceTestablesecurityin FunctestMustFunctional
Functest

73req.sec.gen.02must support policy based RBAC.Validate Openstack API are projected with Role based access using policy yaml file or embedded code. Testablesecurityin FunctestMustFunctional
Functest

74req.sec.gen.03must support a centralised authentication and authorisation mechanism.Validate Openstack authentication and authorization works properly using solution like keystone or LDAP.Testablesecurityin FunctestMustFunctional
Functest

75req.sec.zon.01must support identity management (specific roles and permissions assigned to a domain or tenant).Validate Openstack identity services work properly using solution like keystone or LDAP.Testablesecurityin FunctestMustFunctional
Functest

76req.sec.zon.02must support password encryption.Validate Openstack password encryption services work properly using solution like barbican.Testablesecurityin FunctestMustFunctional
Functest

77req.sec.zon.03must support data, at-rest and in-flight, encryption.Validate sensitive information in Openstack API calls and logs are encrypted properlyTestablesecurityCLARIFY requirments for Encryption Strength -  missingMustFunctional



78req.sec.zon.04must support integration with Corporate Identity Management systems.Validate Openstack integration with corporate identity management system using solution like keystone integrated with LDAP.TestablesecurityCLARIFY requirments for Encryption Strength -  missingMustNFR



79req.sec.cmp.02must comply with all applicable standards and regulations.Validate Openstack API and logs are fully compliant with Security Policy and RequirementsTestable + Verifiable, but Impossible without a complete listsecurityCLARIFY - missing in Functest. Captured in Telco TCs SecurityMust???


Compliance with "All applicable Standards and Regulations" beyond OPNFV's scope.
80req.sec.cmp.03must comply with all applicable regional standards and regulations.Validate Openstack services are fully compliant with applicable region standards and regulations, e.g., General Data Protection Regulation (GDPR) in EU. Testable + Verifiable, but Impossible without a complete listsecurityCLARIFY - missingMust??


Compliance with "All applicable Regional Standards and Regulations" beyond OPNFV's scope.
81req.sec.ntw.01must have the underlay network include strong access controls that comply with ISO 27001 and adhere to the V1.1 NIST Cybersecurity Framework.Validate Openstack services are fully compliant with ISO 27001  and adhere to the V1.1 NIST Cybersecurity Framework, e.g., achieve Cloud Security Alliance (CSA) STAR Certification.Testable or VerifiablesecurityCLARIFY - missingMustFunctional


What are the export limitations and Open Source tool limitations to test Strong encryption?

ISO 27001 Ten easy steps??

Commercial tools available, at cost

NIST Framework Core

The core of the framework has five functions – identify, protect, detect, respond and recover. These functions can be thought of as outcomes and aligned with them are 22 categories, 98 subcategories, 125 outcomes and 287 informative references (controls).


82req.sec.ntw.02must have all security logs stored in accordance with ISO27001.Validate Openstack logs are fully compliant with ISO 27001, e.g., achieve Cloud Security Alliance (CSA) STAR Certification.TestablesecurityCLARIFY - missingMustFunctional


What are the export limitations and Open Source tool limitations to test Strong encryption?

ISO 27001 Ten easy steps??

Commercial tools available, at cost

83req.sec.ntw.03must have the underlay network incorporate encrypted and/or private communications channels to ensure its security.Validate Openstack underlay network support encryption or private channels using solution like VPN or IPSec tunnels.Testable or VerifiablesecuritymissingMustFunctional


Capture traffic and confirm headers?
84req.sec.ntw.04must configure all of the underlay network components to ensure the complete separation from the overlay customer deployments.Validate Openstack tenant traffic is separated from underlay network traffic using solution like vlan, vxlan, etc.Testable or VerifiablesecuritymissingMustFunctional


Capture traffic and confirm headers?

  • No labels