Page tree
Skip to end of metadata
Go to start of metadata

Overview

This will be the 'software' counterpart of this work. The validation of the software configuration, should be performed against the requirements, that is preferably defined in a machine-readable format. Hence, this project relies heavily on an implementation of PDF 2.0 defined in CNTT. 


Team

NameAffiliationRemarks

Sridhar K. N. Rao

Sridhar Rao

SpirentCIRV-SDV Lead

Parth Yadav

Parth Yadav

Delhi UniversityLFN Intern

Ashwin Nayak

NITKLFN Intern

Shivam Balikondwar

Shivam Balikondwar

PICTStudent Volunteer 

Implementation of PDF 2.0 in JSON

This implementation will be "SPECIFIC" to support software validation, and multiple installers/installations (Airship, TripleO, Fuel, etc). In this implementation, there will be parameters that are not part of CNTT-PDF. The implementation has two parts:

  1. PDF-Template in JSON.
  2. Site-specific PDF in JSON

The PDF-Template includes "extrapolation" information, which will help to create site-specific pdfs. The template will be filled by the user and extrapolated by the tool to create complete site-specific PDF.

The below figure summarizes the organization of the template.


Architecture

There will be two versions with single code-base. 

Version-1

The below figure provides the software architecture of the SDV. The user only runs 'valid' program, and select one or more validations.

Version-2

The containerized version.

Testcases

The initial testcases can be found here: http://testresults.opnfv.org/test/#/projects/cirv 

Pre-Deployment Software Validation: Hyperlinks


The below table and excel-sheet summarizes the links for which the connectivity is to be checked. Along with the connectivity, the 'Latency' can also be noted.

PS: The table (sources) and the excel (images) are specific to Airship. For other deployments, the components and corresponding repo-names and tags may vary.

Images

Airship-Images.xlsx

Sources


Sl. No.ComponentRepository-NameTagsURL to use
1Neutronopenstack-helm/neutrond2abe39d498f48c4721e26aca19e81189bc8891bhttps://opendev.org/openstack/openstack-helm
2helm-toolkitopenstack-helm-infra/helm-toolkit
03580ec90afa162c166661acf27f351b83565375
https://opendev.org/openstack/openstack-helm-infra
3novaopenstack-helm/novad2abe39d498f48c4721e26aca19e81189bc8891bhttps://opendev.org/openstack/openstack-helm
4openvswitchopenstack-helm-infra/openvswitchd0b32ed88ad652d9c2226466a13bac8b28038399https://opendev.org/openstack/openstack-helm-infra
5calico, libvirt, mariadb, memcached, rabbitmq, postgresql, ceph-client, ceph-mon, ceph-osd, ceph-provisioners, openstack-helm-infra/< >03580ec90afa162c166661acf27f351b83565375https://opendev.org/openstack/openstack-helm-infra
6apiserver, calico/etcd,
controller-manager, coredns, etcd, haproxy, ingress, proxy, scheduler, promenade
charts/< >64807416b71958e31156ef7a50e169813acc4e15 https://opendev.org/airship/promenade
7barbican, cinder, glance, heat, horizon, keystone, tempestopenstack-helm-infra/< >52c132b9356695bf455ae25ec78cef9f532f9fe8https://opendev.org/openstack/openstack-helm
8tillercharts/tiller50384e47c762438b9e39abe4677f3c29f3c09184 https://opendev.org/airship/armada
9calico-utility, ceph-utility, compute-utility, etcd-utility, mysqlclient-utility, openstack-utility, postgresql-utility charts/<>9c2038cb59bfbb3ff5c3bbf11c7001d621437b98https://opendev.org/airship/porthole


Pre-Deployment Software Validation: Configuration

Apart from ensuring that the requirements are met, this validation helps in minimizing/eliminating any deployment errors, drives test-automation, and checks for consistency to achieve efficient automation. The below picture summarizes the scope (in red dashed rectangle) of the software validation.

This software validation can be: (a)  'Pre-Deployment' Software Validation (b) NW Links Validation  (c) Post-Deployment Software Validation


The below table summarizes validation of different configurations. The table uses Airship and its configurations as an example. 


Sl-NoValidation-Parameterexpected valueFile To look at  (Applies to AIRSHIP ONLY)"Key" to match (Applies to AIRSHIP ONLY)Requirement Level
1OSubuntu
xenial
gobal/software/charts/ucp/drydock/maas.yaml
type/cntt/profiles/host/*
default_os:
image:
Mandatory
2HugePage Configurationsize: 1G
count: 32 (minimum)
profiles:
type/cntt/profiles/hardware/*
site/<site_name>/profiles/hardware/*
hugepages:
    dpdk:
        size:
        count:
Mandatory
3CPU Isolation Ex: 4-43, 48-87same as abovecpuset:
    kvm:
    vcpu_pin_set
Mandatory
4Openstack Versionocata/pike/steinNo Single Explicit Location.
global/software/config/versions.yaml includes the mention of the version
osh:
    <service_name>:
        <service_component>:*ocata*
Mandatory
5Openstack Services ?ingress-controller, ceph-config,, mariadb, rabbitmq, memcached, keystone, radosgw, glance, cinder, compute-kit, heat, horizontype/cntt/software/manifests/full-site.yaml
global/software/manifests/full-site.yaml

Mandatory ?
6Virtual-Switch


openvswitch
openvswitch-dpdk

Config only - Version as supported by OSH
site/<podname>/software/charts/osh/openstack-compute-kit/chart-group.yaml
chart_group:
    - openvswitch-dpdk
    - neutron-ovsdpdk
Mandatory
7Version of the Manifests1.7site/<site-name>/site-definitions.yamlrevision:Mandatory
8Node-Names ?list of all the names.site/<site_name>/baremetal/nodes.yamlname:Mandatory ?
10bootstrap protocol ?pxeprofiles:
type/cntt/profiles/hardware/*
site/<site_name>/profiles/hardware/*
bootstrap_protocol: Mandatory ?


Sl-NoValidation-Parameterexpected valueFile To look at  (Applies to AIRSHIP ONLY)"Key" to match (Applies to AIRSHIP ONLY)Requirement Level
1Hypervisorkvmglobal/software/charts/osh/openstack-compute-kit/nova.yamlvirt_type: Optional
2Container Enginedockerglobal/software/config/versions.yamlrepositories:
    docker:
Optional
3Container Management kubernetesNANAOptional
4k8S components proxy, container-networking, dns, etcd, haproxy, coretype/cntt/software/manifests/full-site.yaml
global/software/manifests/full-site.yaml

Optional
5Ceph Components mds, mon, osd, rgw, mgrtype/cntt/profiles/genesis.yaml
------------------
type/cntt/software/config/endpoints.yaml
labels:
    dynamic:
        <ceph-*>: enabled
-------------
data:
    <tenant-ceph-*>:
Optional
6K8S NetworkingcalicoNo single explicit location.
Optional
7LMA Components - Client Sidepromethues-openstack-exporter
fluentbit
type/cntt/software/manifests/full-site.yaml
global/software/manifests/full-site.yaml

Optional
8LMA Components  - Server Sideinfra-logging, infra-monitoring, infra-dashboardstype/cntt/software/manifests/full-site.yaml
global/software/manifests/full-site.yaml

Optional
9Special NIC Driversixgbe or i40ebaremetal/bootactions/*.yamlThe complete file.Optional
10Users


Optional
11UCP Componentsceph-update, ceph-config, core, keystone, divingbell, armada, deckhand, drydock-scaled, promenade, shipyard, type/cntt/software/manifests/full-site.yaml
global/software/manifests/full-site.yaml

Optional
12Tenant Ceph Components tenant-ceph (mgr, mon, rgw)type/cntt/profiles/genesis.yaml
------------------
type/cntt/software/config/endpoints.yaml
labels:
    dynamic:
        <tenant-ceph-*>: enabled
-------------
data:
    <tenant-ceph-*>:
Optional
13Post-Dep N/W Config Script


Optional
14Site-Typecnttsite/<site-name>/site-definitions.yamlsite_type:Optional
15Version of the Manifests1.7site/<site-name>/site-definitions.yamlrevision:Optional
16OVS DPDk ConfigurationNon-Datapath Threads,
PMD Threads
OVS-Bridge name
profiles:
type/cntt/profiles/hardware/*
site/<site_name>/profiles/hardware/*
--------------
type/cntt/network/common-addresses-ovsdpdk.yaml
cpu_sets:
    dpdk-lcore-mask:
    pmd-cpu-mask:
--------------
bridge_for_ovsdpdk

Optional


Network-Links Validation

Some of the options we have:

  1. Run LLDPtool on compute and control nodes – easiest to implement, but, it can get complex if the number of nodes are more.
  2. LLDP on TORs - Reuse the solutions that used by ‘data centers’ – opensource and scalable. It may take slightly more time.

We propose to use Option-1. The first problem is topology representation. If PDF 2.0 does not include this in its description, we will use DOT-Specified network graph .

Post-Deployment Software Validation : State


Sl. No.Validation TypeDescription
1Container or POD StatusEnsure container status is OK. Detect failed containers and raise an error
2Required Container ports are openEnsure relevant docker containers are up and running, with ports open to listen
3ntpVerify all deployed nodes have their clock synchronized
4OVS-DPDK ConfigurationValidates OVS DPDK PMD cores from all NUMA nodes
5RabbitMQ LimitsMake sure the rabbitmq file descriptor limits are set to reasonable values
6Service StatusDetect services status on the target host and fails if we find a failed service.
7SELinuxEnsure we don’t have any SELinux denials on the system
8TLS configurationEnsure endpoints are secured.
9Storage (Ceph) Health CheckHealth should be OK

Post-Deployment Software Validation: Security


Sl. No.Validation TypeDescription
1Security: Right File Permissionshttps://docs.openstack.org/security-guide/checklist.html
2Security: Right Configurations


Readouts


SDVState readouts

ReadoutAuthorSlidesNotes
SDVState announcement at LFN Mentorship Project PresentationParth Yadav





  • No labels

7 Comments

  1. Sridhar Rao What do you mean by focus on pre-deployment?  If this is validation of the software deployment, wouldn't that take place after the stack has been deployed? Or is the red-lined box to check access to the upstream sources, repos, images, etc.?

  2. I don't think this HDV testing is closely bounded to installers? It could be just some verification tools directly communicate with the hardware. it has nothing to do with any specific installer.

  3. Lincoln Lavoie  There are two aspects - (a) one is check access to upstream sources/repos/etc..  and (b) Check the configuration in manifests if it is meeting the requirements. Both (a) and (b) are done before deployment of the cloud. 

  4. Sridhar Rao that sounds very specific to Airship or only installing from pure open source. How would such a verification take place on a vendor platform?

  5. Lincoln Lavoie I don't think so. Both validating access to repos and configurations are generic - nothing airship specific, for example, it applies to TripleO too.  I the above table, for configuration, the Mandatory ones are not airship specific.  Optional ones, yes. In addition, the expected values provided are 'examples'. 

  6. My first thought is that we should have clear separation of the precheck/validation at the CIRV level and precheck/validation at the installer level. Of course, if cntt/opnfv adopts an upstream format standard, we can leverage the validation tool from the upstream. The source/repo link verification can be generic and implemented in the cirv level fairly straightforward.  The software configuration table is what confuses me. The parameter key and path, files to look, etc., listed in the table, are installer bound. I understand they are meant to be examples, but I don't see how it is feasible for the CIRV to implement the validation w/o knowing which installer is to be used or the implementation details of the specific installer. My thought is that these type of checks, we should define an interface/plugin, and delegate the actual work to the installer. Airship 1.0, for example, has a service called Pegleg, does the manifest validation. So we can add a simple plugin in CIRV to invoke Airship Pegleg cmd (which will change to AirshipCtl cmd in version 2.0).

  7. James, correct me if I'm wrong, pegleg validation is for syntax - it does not validate against the requirement. I cannot provide it requirements and ask it to validate the manifests - isn't it? The goal is to validate the 'Software Configuration' against the requirements. If you look at the 'mandatory' rows, they all apply to any installer - OS, CPU Isolation, Hugepage, OPenstack-Version, OVS_DPDK and SRIOV Configurations, etc.  CNTT will provide requirements- when, how and where you validate is an open problem.  If this validation is implemented by Installers, I agree with you that we need NOT do at the CIRV. I haven't come across any installer doing it - including Airship.