Page tree
Skip to end of metadata
Go to start of metadata

This page is the original Clover project proposal. Please refer to the project wiki main page for up to date information: Clover Home.

Project Name:

  • Proposed name for the project: clover

  • Proposed name for the repository: clover

Project description:

  • Monolithic infrastructure slows development  and constrains scalability and reliability, so the industry has been trending on micro-services based architecture [micro-services]. Growth of micro-services, however, often leads to ‘container sprawl’: service-to-service communication becomes unwieldy and hard to manage; integration and testing permutations quickly multiply; uniform policy enforcement becomes a lot harder…These issues can be esp. prominent in service provider and NFV use cases. A dedicated service layer (Service Mesh) agnostic to individual app and deployment and built on top of k8s’s existing pod and service constructs and orchestrator functions, is designed to overcome these challenges.
  • The service providers found operating a large set of micro-services challenging without a common set of tools that give them visibility and traceability. This challenge is esp. relevant when applications are developed independently (by different teams, vendors or communities), or e.g. legacy software components are containerized and brought in to the mix. A common set of reusable services, such as monitoring, logging, tracing…, agnostic to development and deployment environments is needed. NFV specific use cases may demand additional common utility services that are outside of the general computing applications. Such reusable services may include network tracing, nat, dpi etc.
  • Continuous delivery – a core benefit of cloud native applications is that its components (micro-services) can be updated live independently. Currently CI/CD pipeline in OPNFV largely stops at CI level. Building safe, low risk, platform independent and at-scale continuous deployment is an important challenge facing NFV use cases. We need to address it in collaboration with xci/releng and upstream e.g. cncf cross-cloud ci.
  • NFV specific use cases introduce new challenges to solve that are different from the general cloud native cases. E.g. network needs for gateway (intermediate systems) common in NFV use cases, potential network performance issues, security issues, edge related challenges, in collaboration with edge related projects, enhancements or additional common services for NFV and necessary interfacing with other OPNFV or LFNF projects (e.g. ONAP).
  • To meet the above challenges, the Clover project intends to integrate and test a cloud native computing framework for NFV use cases, based on CNCF and other upstream components and cloud native computing community’s best methodologies and tools.

  • Cloud native computing technology can be very beneficial to NFV in terms of elasticity, scalability, dynamism, resilience,  simplified operation, among others. In the area of edge NFV, it also has the benefit of being more lightweight. Although cloud native may not be suitable in all use cases, it can clearly be an important addition to the OPNFV solutions. So best practice solutions to the challenges identified above are critically important for supporting cloud native technology in NFV.

  • Clover will require Kubernetes, which can be the output of other OPNFV projects, e.g. container4nfv or XCI or other k8s scenarios. It will also be transparent to Clover if Kubernetes runs natively in bare metal or on Openstack virtual machines or other underlying cloud computing platforms (like AWS, GCE, …).


  • Existing OPNFV projects support Kubernetes and containers as a foundation.  Clover intends to solve problems that arise from adopting cloud native computing in scale, to better support cloud native computing for NFV virtual functions. We've identified a micro-services service mesh framework, common agnostic reusable services (logging, tracing, monitoring,...), nfv specific requirements (e.g. light-weight edge support, gateway mode support, security enhancements, additional nfv common services), continuous deployment that is optimal for cloud native micro-service based deployments... and make these capabilities easier to use for NFV.

  • Gap analysis and upstream engagement as necessary

  • It will be integrated with OPNFV XCI or classic CI/CD, upstream initiative e.g. cross cloud ci, and be extended towards continuous deployment.

  • Testing will be implemented by automation with supported platforms and integrated with OPNFV testing infrastructure. Sample cloud native use cases will be developed or dopted for fulfilling testing and validation needs. Will support live debugging and tracing.

  • Client console may also be integrated/developed for ease of use.

  • Foundation/infrastructure layer support of Openstack or Kubernetes (VIM and orchestration) or Docker are out of scope. We rely on existing OPNFV or other projects, such as container4nfv or XCI, for bringing up the prerequisite foundation.

  • Edge NFV is a common use case for containers, we will demonstrate suitability to support edge use cases, but we do not intend to cover the actual edge systems as in, e.g. ENFV project or the Multi-Access Edge project proposal. Clover intends to solicitate collaboration ideas with these projects.

  • Deliver the solution in a k8s- scenario.

  • Future extensibility of the project will depend on user and developer community feedbacks.


  • We intend to develop fully automated unit test and integration test that can be incorporated in OPNFV testing infrastructure.

  • QA and test resources can be existing OPNFV Pharos labs, future OPNFV Lab-as-a-Service, virtual machines in a developer’s laptop, or common cloud services such as aws and gce.


  • Introductory and developer oriented documentations are to be developed.


  • Clover depends on Kubernetes and other projects in CNCF and other upstream communities.

  • Depends on either an OPNFV project or XCI scenario or another tool to bring up common k8s support or Openstack support.

  • Depends on some Pharos community or Lab-aaS computing resources, although we intend to make developers able to use a local laptop or public cloud services as well.

  • Container4nfv (aka OpenRetriever) is a comprehensive container and orchestration project in OPNFV. Clover can use the output of container4nfv and is complimentary in scope (OpenRetriever's Scope).There are also other k8s scenarios in OPNFV.

  • Collaborations with AUTO, Models, on related deliverables

  • Additional dependencies (or gaps) in areas such as storage, edge cases, performance tools etc may become more concrete when we investigate further.
  • Names and affiliations of the committers

  • Names and affiliations of any other contributors


    • Andrew Theurer (Red Hat)

    • … being populated

Planned deliverables:

  • A new scenario based on k8s integrating micro-service oriented core services
    • Service mesh
    • common reusable services: Monitoring, logging, tracing
    • Networking (from infrastructure)
    • Additional services
  • Cloud native CD and testing linkage with opnfv dev infrastructure
  • Automated testing of resiliency, scalability …
  • Best practices for cloud native application development
  • Example use cases, sample and test containers
  • Documentation

  • Upstream patches to relevant upstream projects to address gaps identified and solved

Proposed Release Schedule:

  • The first release planned: Fraser

    • Tentative deliverables for the Fraser release

      • Clover core services

      • Development and test platforms

      • Documentations

      • CI/CD pipeline

      • At least one use case and test cases

  • Will this align with the current release cadence

    • Yes

Key Project Facts

Project Name: Clover (clover)

Repo name: clover

Lifecycle State: Incubation

Primary Contact: Wenjing Chu,

Project Lead: Stephen Wong,

Jira Project Name: Same as Project name : clover

Jira Project Prefix: [10 Characters max [A-Z] ] CLOVER

mailing list tag [Should match Jira Project Prefix] Clover


*Link to TSC approval: TBD

Link to approval of additional submitters: TBD

  • No labels


  1. I think it would be useful to differentiate the idea of cloud native applications from the compute footprint of containers. I believe that it is possible to fulfill the promise of cloud native application architecture with VMs too - the steps to cloud native are, for me:

    • Automated deployment
    • Continuous Integration with a comprehensive automated test suite
    • Comprehensive tracing and monitoring, including log aggregation
    • Componentization of the aplication to isolate elements that can scale independently
    • Separation of stateful (scale out) components from stateful components
    • Integration of application resiliency into the application (both for stateless components and for storage)
    • Management of distributed applications (policy-driven fault and lifecycle management, upgrades, root cause analysis, etc)
    • Continuous deployment pipeline with deployments to different deployment profiles from the same source tree

    There is more, but these are progressively more work, and build on each other. They are also mostly independent. Once you get to the end of the list, with resilient, componentized applications, and the ability to scale out on demand and easily promote development and test deployments to production, it does not matter so much whether you are running on VMs or containers.

    There are a few considerations that affect every step here - resource isolation (security, multi-tenancy, performance) which are not a step of their own, but are vitally important.

  2. Wenjing, congratulations for Clover Project! Would you present the dependency at Auto meeting 6am PT next Monday?

    1. Thanks Tina. It'll be a good topic to discuss at some point after we get through the initial crunch time.

  3. It seems, that first we need to agree on what do we call "Cloud Native". I know two definitions, the CNCF one from the CNCF Charter and the 12 Factor App one. Both of them are different from Dave Neary-s list (wink) .

    Do you already have project meetings? I could not find this info on the wiki (actually I could not find the project wiki). I would be happy to join.

    1. Welcome to the clover project, Gergely. We had some discussion on the semantics of cloud native in the project proposal presentation: Clover-4.pdf.

    2. Thanks Gergely Csatari for your comment - my list is not, I think, a definition of what it means to be cloud native, but it is, in my opinion, a collection of steps that application developers need to get through to be cloud native. If we look at the CNCF definition, "Microservices oriented" implied, for me, componentized, with a separation of stateful and stateless function to allow them to scale independently, and comprehensive tracing, monitoring, testing, and CI & CD pipeline. "Dynamically managed" implies management of distributed applications, high quality monitoring and fault management, and integration of well-established resiliency architecture patterns in applications. So I don't think my list comes from nowhere, or is incompatible with CNCF or 12factor.

    3. It is not clear if all networking services can (or should) be implemented with full compliance to 12-Factor requirements, for example, 12-Factor VII, "export services via port binding" generally implies that the underlying network and transport services are IP (only) and protocols typically use the common transport protocols over IP (UPD, TCP, SCTP, ...). While many protocols in networking elements indeed do use IP stack underneath and therefore their specific Service Access Points (SAPs) can be implemented while meeting the constraints of this requirement especially for the most control and dataplane protocols, it is not universally true. Networking applications and protocols, especially at edge of the networks ALSO implement protocols and services that implicitly operate at lower layers than IP. While it would in theory be possible to encapsulate all the lower layer protocols to IP, some entity in the path would need to be available to perform this function, and the overhead would increase and performance would decrease as a result. In addition to control and management planes, many networking applications still need to process packets at the lower than L3, which has even more severe performance implications than C+M plane protocol SAPs would have. This specific requirement works well for "services on network", but does not necessarily work well or at all for "network services" (which NFV is all about). Even for protocols on top of IP which are "compliant" with this constraint, there is often additional requirements, such as multiple (overlapping) IP address spaces, or need to know which physical/logical interface the associated packets originated from (i.e. lower layer context awareness).  It might be advisable to examine the whole list of the 12-factor requirements in the context of the network service instances before just saying that is to always apply.

      1. Hi,

        Thanks for the comments, this will be a fun project to participate (smile)

        Wenjing Chu:

        • Can you add the presentation to the wiki page? It good information and nice figures (and I can not edit the wiki).
        • So we can already state, that we use the CNCF definition of Cloud Native and focus on Micro-services and continuous delivery.
        • Inside the micro services domain the focus is on
          • service mesh
          • dynamic request routing (even of of telco protocols?)
          • circuit breaker,
          • service discovery
          • load balancing (even of telco protocols?)
          • global control and policy mechanisms
          • Tracing
          • Common utility services (as far as I understood these are some kind of platform services)
        • In CD the focus is on
          • Enhancing the OPNFV CI/CD systems
        • As I see on some slides there are already different open source projects selected. Won't be the scope of this project to discuss about the different options to use for the different problems?

        Dave Neary: Now with the slideset it is more clear (smile)

        Pasi Vaananen: I think the full 12 factor app cloud nativenes is not a possible target for all telco applications. The original 12 factors were the best practices for Heroku. In my opinion at the moment the CNCF definition of cloud native is a better target and as I see Clover is also targeting this.

        1. Thanks, I just felt that it was worth pointing out some of the issues in our context as this keeps coming up as "definition".

          FYI: ETSI NFV ISG EVE WG is working on classification helps for their view of the "cloud native" VNFs in EVE 011 document, present draft can be accessed from here as one reference:

          1. Yepp I forgot the ETSI references. There is also IFA029 ( what is a study about the affects PaaS and cloud native to the ETSI NFV architecture.

  4. This is a great idea seeing you guys presenting at the OPNFV meeting yesterday at Spirent. I saw lots of potential for Telco usecases with container service chaining using Istio and also some usecases about Policy enforcement. How can we VMware can join this initiative to promote Clover becoming an upstream project in OPNFV ?

    1. You are absolutely welcome to join! Most (almost all, in fact) of the team will be going to the CNCF/KubeCon in Austin on December, and we have already talked about having a face to face meeting there at the event. So if you are also attending, you are most welcome to join our meeting. Let me include you also in the email thread.

      Otherwise, we will start IRC meeting shortly after (probably next year). Will post the time and channel on the Clover home page as well as sending email out to opnfv-tech-discuss.

      Thanks for wanting to participate. Looking forward to working together on Clover soon.

      1. Thanks Stephen- unfortunately I would not be able to make it this year for Kubecon- keep me posted what is going on with Clover and Istio-

        1. I have some discussion with AT&T and Orange where they trying to integrate with Istio on the Edge M-CORD. I think it is good to mention your project Clover to them so we can better do coordination and adding more nice use-cases in combining ONAP Clover and CORD Edge network, what would you think Stephen. If we can have a meeting once a month or twice a month with Clover and Istio, I am more than happy to support from VMware perspective what can we help Clover to become an upstream project- do love pretty much the idea you guys proposing.

          1. That would be great, Tuan. Let's coordinate after CNCF/KubeCon. Thanks!

  5. Stephen Wong I'm interested in contributing to log aggregation, comprehensive Tracing and monitoring. I'm also going to KubeCon this year and will have presence in Kubernetes Contributor Summit(pre KubeCon). Please let me know how can I join Clover team discussion in KubeCon.

    1. That would be great! Let me add you into that email thread. We will likely meet on or after Wednesday (Dec. 6th) afternoon. Looking forward to meeting you.

      1. Yes, sure. 
        Looking forward to meet you.