Page tree
Skip to end of metadata
Go to start of metadata



Before going into details of the advantages and challenges of the current release model employed by OPNFV and proposed one, it is important to highlight again that XCI brings in a new development model based on CD which has its own advantages and challenges. 

This development model puts certain expectations to the developers to ensure the master branch is releasable at all times.

Apart from having expectations on the developers, this development model needs to be supported by the model employed by the release which should be lighter and with less overhead. Having a lighter release model is important in order to enable the community to push things out regularly and with a faster pace to bring small and frequent updates and more importantly working software to the users, focusing on the speed, the agility, and the quality. This will also benefit the other developers and the CI since everyone will have working version of the software to base their work on. 

Current Release Model vs New Development Model

The current release model employed by OPNFV follows traditional point release model which is good in providing stability due to the fact that it uses stable versions of upstream components. However, it lacks the agility that is needed for CD way of working and has somewhat heavier expectations on developers. Apart from what it requires from developers, it also structures the development phase in a way that certain things can't happen before the others, slowing down and even blocking them.

A delay in earlier phases has impacts on the rest of the process, impacting the later phases/milestones by cutting the time from them in order to give more time for the activity/milestone that caused the delay. This has potential to introduce slip in the entire release schedule. Apart from delaying the current release, it also limits the ability to start working on the next release. What is even worse than these is not the delay in the release date but release something that's not tested according to criterias set by the community, resulting in a release that is not generally known for its quality but for its content.

What is summarized above is based on the experiences from the past OPNFV releases; they have shown that OPNFV puts relatively more importance to time and scope compared to quality.

The CD based development model aims to fill this gap by having frequent releases, keeping the time commitments and the quality high but at the same time giving projects chance to catch up with the releases later on. This is enabled by giving full ownership, responsibility and accountability to projects when it comes to what they make available to the users as release and when they want it. This then enables projects to release their work when they are confident in them and what they intend to release fulfills the criteria instead of waiting for next major release or maintenance release. (if you missed the train, jump onto the next one which will depart shortly.)

The advantages explained here is generally about how and when to do the release. But the idea is not only about releasing but also not-releasing as well. If the projects feel they need work to do due to concerns with the quality or documentation, they can decide not to release something now and wait for later since the next one will happen soon anyway. 

Proposed Way Forward

The proposal to support projects that strives to apply CD is to have 2 tracks initially that are run in parallel as explained below.

  • stable track: based on the current OPNFV release model and with no or minimum changes, making it possible for projects to mark/tag releases as usual and release them.
  • latest track: based on CD and supported by XCI, allowing projects to mark/tag certain versions as "latest X" and giving opportunity for projects to pick one of them to be part of regular OPNFV release when the release date approaches.

The reasons for having 2 tracks running in parallel are

  • to keep the impacts on projects that want to continue working in traditional development and release model at minimum by not changing settled/known way of working.
  • to give the possibility to the projects that want to apply CD and take part in XCI to find if the new way of working really works for them by trying it out and potentially feeding the learnings back to the processes to improve it, again without impacting the traditional model.
  • to make it possible for the projects to participate in both tracks.

Having both tracks running in parallel will have no or very little overhead on the projects that do not take part in XCI. (Please note that the test projects are an exception as they need to support both tracks. These projects need to keep an eye on both tracks, ensure none of the tracks are impacted due to something that is done for the other one and adapt as needed.)

The releases done in this model and marked as "latest X" mentioned on above scenarios do not exactly mean to have releases like how OPNFV does today with its official releases. They are simply tags on a specific version that informs potential users about the availability of a working version. (In CI Evolution terms, this can be seen as another confidence level; the version which is tagged as "latest X" passed all the possible/available testing in OPNFV and now it gets promoted to be used by users, essentially extending the CI to the users.)

Please note that how the tags "latest X" will look like is subject to discussion.

Putting Both Models Together

The diagram below tries to illustrate how the OPNFV Releases would look like when they are put next to each other.

Several things need to be clarified in above diagram.

As it is visible, the straight line on top tries to show the current release model with its milestones and official releases. It is called as "stable track" on this page.

The dotted line below tries to show the proposed release process that is running in parallel to traditional one. It is called as "latest track" on this page.

The 4-pointed starts in "latest track" are the tags applied by the projects to make certain versions available to their users as their intermediate releases as summarized in the previous section. They can be named like latest monthly or latest weekly releases for example. These versions are also candidates for the official OPNFV releases; the projects need to determine which version should be part of the official OPNFV release and pass this information to Release Manager. The chosen version then gets tagged by official release tag and the projects will not need to wait for release to happen in order to continue with the work for the upcoming release.

Milestones in Latest Track

Another important point to highlight regarding "latest track" is that it does not contain any milestones like "stable track" since the activities that are carried out as part of certain milestones in stable track are already built in/embedded to the CD based development model. (thus the dotted line for latest track in the figure)

Here are the details regarding milestones that are strictly followed by stable track  and how they are done in latest track.

  • uplifting to new version of OpenStack or other components: there will be no uplift as XCI will always be on the master. Bringing in new versions will be part of regular SHA bumps that is planned to be done regularly. (eg. starting with monthly and then bi-weekly and perhaps weekly over time - the frequency of SHA bumps requires discussion within the community and also depends on the stability of upstream masters. )
  • testing: 
    • healthcheck is part of XCI (CI Evolution). Periodic jobs for upstream components and scenario patchset verification will be subject to healthchecks and SHAs for the components/scenarios do not pass this will not be pinned/will not be taken into XCI.
    • daily tests are also there right from the beginning. If a version of upstream component gets bumped in XCI, it will directly be taken through daily testing which will include functest initially and yardstick in near future.
    •  test framework/case readiness is part of XCI right from the beginning rather than having a separate milestone and bringing in new versions of them to XCI will be natural by applying promotion principles. (SNAPS already uses XCI which will proposed to be extended to other test projects.)
    • stress tests that are not currently run on stable track could become part of XCI over time run as part of latest track.
  • feature freeze: there will be no "feature freeze" as the development & testing is continuous. When the release date approaches, projects can point to one of the candidate versions as version to release and move on. (it could be L8, L9, or L10 on the diagram above.)
  • stable branch: The mechanics of this obviously requires more thinking and it is again up to projects to decide if they want to branch off from whatever candidate version they picked as the one they release. XCI will not enforce when they can branch. XCI will and must always work no matter if projects branch off & tag their work before XCI gets tagged.
  •  test complete will disappear as the testing will continuously run if there is something there to test. 
  • documentation needs to be kept up to date at all times since users can pick anything and everything that is tagged as intermediate release. It is important and beneficial for projects to treat their users and treat them well. This also has additional benefits since the users are not only the people outside of OPNFV, cloning repositories and trying things out. One of the users of XCI is the stable track. Having documentation early is important for that track as well. 
  • jira issues are similar to the documentation. The backlog must always be kept up to date so users can take informed decisions if they think of attempting one of the intermediate releases project makes available.

The prerequisite to tag versions as intermediate release is same as current OPNFV release criteria; baremetal deployment passing functest (& yardstick) daily and having the accompanying documentation. Please note that the 4 consecutive successful runs will not be a valid release criteria for latest track due to applying promotion concepts.

XCI vs OPNFV Projects/Scenarios vs Upstream Projects

The relations between XCI, OPNFV Projects/Scenarios and Upstream Projects in latest track from release point of view can be seen from the diagram below.



The way how the development is proposed to be structured makes it possible to release XCI itself and the scenarios independently from each other - like how upstream projects do their releases; independent from OPNFV.

Another point to mention here which makes the releases independent is the relation between XCI and the OPNFV scenarios; OPNFV projects are upstream to XCI.

The details of proposed way of onboarding projects/scenarios to XCI and the way of working is available on Onboarding Projects/Scenarios to XCI.

Releasing Scenarios

In CD based development, the scenarios will be fully owned by the projects as oppose to how it is done today; the ownership and the responsibility to develop the scenarios, tag intermediate releases, branch the repositories, and promote an intermediate release as official OPNFV release are fully controlled by the projects themselves. This means that all the work that is required to develop a scenario and put it through XCI will be done in Git repositories of the corresponding projects, not in XCI repository. The development work will be supported by XCI to provide fast feedback to the projects, giving projects the opportunity to catch and fix faults earlier than what it is today. (The details/mechanics of the proposed way of developing scenarios is documented on Onboarding Projects/Scenarios to XCI.)

A very high level list of steps are

  • project develops a scenario and scenario is put through XCI continuously
  • the documentation is done in parallel to the development work
  • scenario fulfills the requirements to be tagged as intermediate release (tests pass, documentation is available, known issues logged to JIRA) and project has confidence
  • project tags the version and makes it available as intermediate release

Having frequent intermediate releases makes it possible to get the software in the hands of users early and often to establish feedback channels between developers and users. Apart from that, this helps upstream projects to gather additional feedback from OPNFV, essentially improving the overall quality of the component they are developing.

Finally, making a scenario part of official OPNFV release will be as simple as passing a tagged version to Release Manager as version to release. No additional check is needed here since all the prerequisites will already be fulfilled long before due to gaining the right to tag a version as intermediate release.

Releasing None-Scenario Based Projects

This document mainly focuses on scenarios and their relation to XCI but none-scenario based projects (eg test projects) will be impacted by and possibly benefited from the new way of working.

One thing that must be highlighted here is that if we see OPNFV Platform Testing (daily runs) as the production environment for OPNFV Test Projects, one cay say that OPNFV Test Projects work in (kind of) Continuous Deployment model. This means that whenever a change comes into a test project, it gets merged to master after (limited) CI checks, resulting in the use of that version during platform testing, which does not always produce good results due to not having necessary checks in place. (lack of CI coverage) 

By utilizing what is provided by XCI and following the accompanying release model, the Test Projects will have chance to adapt themselves to this way of working by employing similar approach; the versions to use within established XCI loops must have been verified properly, tagged, and made available for consumption to XCI and users in general. This might seem like slowing the test projects down but in reality, it helps increasing the overall speed of OPNFV by limiting the faults that slip into platform testing and ensuring we focus and troubleshoot on testing the platform instead of the test frameworks or the test cases. At the same time, the availability of the test frameworks and test cases will be improved along with the quality of them.

XCI is beneficial for requirement projects that do their development work directly upstream as well. If these projects do the development in upstream, XCI can pull down those components from master version and run the test cases that are developed within OPNFV or in upstream. The projects do not need to wait for their contributions to become available to OPNFV through traditional release model; they will directly be available when their contribution gets merged to master in upstream. (or even before the contribution gets merged to master in upstream by providing patchset verification jobs for the upstream projects the contribution is done.)

Releasing XCI

As summarized in different occasions, XCI is a service provided by OPNFV Infra, bringing a framework into developers' and users' hands to do their work more effectively and faster. So the XCI itself will need to be tagged in order to ensure the users who want to use the XCI to get the scenario running can do that. 

This is based on the approach we have been following with OPNFV CI since the beginning: OPNFV CI is agnostic to what runs on top of it and it only provides Jenkins jobs, wrapper scripts, and utilities that are developed, maintained and released as part of Releng. 

All the OPNFV projects are on OPNFV CI, using Releng by following certain guidelines such as having and scripts in project repositories which get executed by the jobs created by Releng. ( and are symbolic names and they are actually CI API.) Projects have little worry over the machine where Jenkins is installed, the Jenkins itself, or the build servers. These are all provided by Releng.

Because of this approach, anyone can duplicate what we do in CI in their labs. They don't need OPNFV Jenkins or jobs or wrapper scripts since what matters here is what is developed by the projects. (Even though it is possible to start from scratch, some organizations prefer to replicate OPNFV CI which is relatively straightforward due to the approach explained above. 

Similar to the and scripts delivered by projects, the roles for the scenarios created by the OPNFV projects can be consumed by anyone who is using Ansible like how XCI consumes ODL Ansible role for os-odl-nofeature and os-odl-sfc scenarios.


Q: Do we have to follow CD and use XCI?

It is entirely up to the projects to follow CD and use XCI. It is also important to mention that it is possible for the projects to join to CD & XCI initiative later on, when they feel confident about XCI.

Q: Will XCI be available for all the projects by default and the new release model enforced on them?

XCI will only be available to the projects who want to participate in XCI initiative.

This means that the CI support provided by XCI and the ability to integrate and test the scenarios using the master versions of upstream components will only be available to the ones who join since the projects need to make necessary adjustments within their projects, follow guidelines, and make their artifacts available to be integrated and tested in XCI.

It is important to highlight that not participating in XCI does not mean what is made available by XCI or the participating projects are only available to these projects. All the projects are welcomed to use whatever is made available by XCI, such as sandbox. In fact, none-participating projects will benefit from XCI indirectly; the version of the platform they will be integrating their features will be more stable and the testing projects that test their scenarios will already be proven to be working. But this doesn't make the CI aspects better for them. They will still be following a release behind and they will be impacted by the usual platform version uplifts since they are not integrating with the latest release.

As noted in corresponding chapters, CD & XCI way of working and the accompanying release model will not be enforced on non-participating projects. The development and release work will remain same for these projects.

Q: Our project have ongoing commitments to OPNFV Installer(s). Do we have to forget about our commitments in order to go CD and use XCI? 

Participating in XCI does not mean any project to forget about their commitments or enforce them to use XCI. If projects want to continue with current way of working as well, they are free to do so.

If they don't want to take part in XCI, it is again their decision. XCI is provided a service to the OPNFV and other communities by the OPNFV community and it is up to them to use it or not.

As proposed above, the impacts on the existing release model is at the minimum, not requiring projects to stop working the way they have been working. The only impacted projects are the ones that take part in XCI. And if these projects decide to solely work on XCI then again it is project's decision that they stop supporting traditional release model and installers.

Q: How mature is XCI?

XCI is under heavy development currently and there are many things that are changing on a daily basis.

XCI currently has

  • sandbox
  • partial implementation of CI Evolution
  • scenarios os-nosdn-nofeature, os-odl-nofeature and os-odl-sfc, the last 2 ones being under development

So, what is currently made available by XCI should be seen as early drop to start collecting feedback from OPNFV projects, upstream communities, and the users.

XCI will be released as part of Euphrates as "experimental" along with the scenarios onboarded to XCI.

Q: Can we take part in "stable" and "latest" tracks both at the same time?

If the projects believe they have the time and resources and they can support both "stable" and "latest" tracks, it is possible to participate in both at the same time.

But it is important to highlight that this will put demand on the project itself more rather than XCI.

Q: It seems the CD based development model and proposed release model only applicable to mature projects. Is that so? 

The applicability of CD & XCI is not about the maturity of the project itself but it is about the how the projects would like to work.

It is possible for a newly created project to start working with CD & XCI right from the beginning without joining to traditional release model and vice versa.

So the decision to take part in XCI or stay in traditional model is entirely left to the projects since this is a commitment that must be made by the projects, not by the XCI or others. This is due to the fact that no one can say that "if you go this track, your life will be miserable" or "this track is better than that" since both tracks have advantages and challenges. The only thing we can do at this phase is to document what the projects should expect from both tracks, the requirements on them and what they will get in the end so they can take informed decisions.

Q: This model seems to be a better fit to the projects that release scenarios.

Even though the document mainly focuses on scenarios and the projects that work on developing scenarios, CD is applicable to different types of projects as they all can get different benefits from applying CD and using XCI.

XCI will be helpful to none-scenario based projects more than some might think of. For example, if we compare how the test projects are released today and how XCI already started working with the test projects, this will be more clear.

One of the very first users of XCI is SNAPS. SNAPS uses XCI for their patchset verification/gating to ensure what they do doesn't break things for others, especially when their project is used during daily platform testing. This basically means that the test projects will have real CI Pipelines, running proper testing for their frameworks/test cases, not just unit test, static analysis and so on. 

The proposal is to extend the scope of the CI for test projects even further and have additional testing for them so instead of always using latest from master (or latest built docker image) during daily platform testing, we can start using promoted versions of the test frameworks/cases. (CI Evolution) You can see the proposal from this link: - This will be presented to testing community based on our learnings with SNAPS PoC.

So, XCI will enable test projects to be compatible with the upcoming releases of upstream components and will help them run additional testing since the focus of XCI is on doing true-CI, which is fundamentally different than what we are doing with OPNFV CI.

Q: How difficult it is to work in CD and use XCI?

Onboarding to XCI requires initial investment which might be seen as a big work but it is not different from starting a new scenario.

Once the initial development and integration is done, the cost to maintain a scenario going forward will relatively be cheaper comparing to how it is done today with a condition that the projects keep an eye on the feedback they get from XCI and keep their scenario working at all times. This is due to that rather than doing big adaptations for upstream component version uplifts (such as OpenStack) during the development phase of a release, they need to do small changes to keep things working against the master. There is no uplift cost but there needs to be constant attention which is what CI requires anyways.

XCI is designed and developed in a way to be as transparent as possible, meaning that the developers will have the guidelines to do their work and tools supporting it, instead of slowing or even blocking them. Whatever they do will directly be put through the CI pipeline and they will get the feedback that is tailored for them and the change they proposed.

Q: Do we have to work against master? Can we use stable version of a component?

The intention with/concern of XCI is to work with master since XCI doesn't try to duplicate or replace what is done by the current OPNFV CI supporting current development way of working with the use of installers.

However, XCI framework makes it possible for projects to use different versions of one or more all upstream components for their scenario and this only impacts their scenario. If a project works with 2 scenarios, they can even use different versions of the upstream components for these 2 since the level of granularity is on scenario level.

This capability is important to have the ability to change versions of components or even sub-components to pick a later version that is not yet verified by XCI or trying out a patch that's not even merged to master yet. Because of this, going back is also possible. This means that if a project prefers to start developing their scenario to onboard to XCI using stable versions of upstream components, it is supported. But the expectation from the projects to switch to master versions as soon as possible.

The reason for having the possibility to override versions is that XCI merely brings things together, from upstream or from OPNFV projects in certain way that is CI-able and that's it. Whatever user tells XCI is passed to whatever component XCI utilizes and the rest is done by that component.

Not having support for stable versions in XCI will also reduce the resource needs so we can have enough capacity to provide timely feedback and have time for additional testing.

Q: When can we mark a version as "latest" and have it available is an intermediate release? What are the criterias?

The criteria to mark a version as "latest" is same as the current release criteria for OPNFV;

  • The scenario deploys on baremetal and passes functest/yardstick
  • The documentation is complete
  • Known issues/bugs are logged to JIRA


Q: Doesn't CD emphasize any version from master releasable at all times? If so, why do we need to mark versions as "latest X"?

That is correct but in order for us to experience with CD and find what works for us best, we need to take a slightly conservative approach. This is also visible in the proposal; starting monthly intermediate releases, followed by bi-weekly and weekly, finally reaching to true-CD.

Once the participating projects feel ready and confident, there is no reason for them not to issue more frequent releases or announce that any version users can pick from their project's master branch is expected to work.


Q: What will be the artifacts for my project if we take part in XCI?

There will be no traditional artifacts (RPMs/ISOs) at this phase. The artifacts that will be released by the projects are

  • Ansible Roles for the scenarios
  • Documentation

Q: What about establishing fixed set of components for OpenStack and other upstream projects and their versions?

For the current release based model, what is expected is logical, especially for test projects.

But with XCI, the variation or constant change is on purpose since we will be working against trunk and nothing can prevent the change from happening. What we guarantee is XCI will always provide "working" version of the "composed platform" that is based on the "latest version of the components" that are included in that version. So, XCI provides CI which is more extensive, structured, and aimed to add value, both for the projects that work on composing the platform/scenarios and none-scenario based projects such as test projects. (A proof of this is what SNAPS is doing, they use XCI to verify their framework.)

Q: Will this really work?

OPNFV is not the first one who tries to work in CD. In fact CD has been around for quite a while and worked well for many different organizations.

However, no one can guarantee that the textbook definition of CD will work for OPNFV. We need to find what works and what doesn't for us and fine tune the way of working that suits us well and does not limit the benefits we will get from CD. So it is important for OPNFV to not stand still and continue trying new ways of doing things in order to have progress.

The only thing one can promise at this phase is that the journey to CD will be challenging but immensely rewarding and fun.

  • No labels