WORK IN PROGRESS!
This document is still a draft and is not committed yet.
Semantic versioning (or semver) is a specification for versioning software components that provide public APIs. If a project provides python libraries or provides a REST interface, it should be using semver in order to make the management of dependencies by projects that use multiple software components possible.
For example Functest uses a lot of other software components (opnfv projects or external software components) and relies on these components to follow semver.
One important note is that strict semver versioning is generally never used for versioning large program releases (such as OPNFV release) because it will be very difficult to comply to the semver versioning rules with so many constituent projects and a release model that is independent of the content of individual constituent project APIs. A simple example to prove the point is that a program release could decide to cut a new major release (thus incrementing the major version) that includes projects that have a new version but do not break any public API backward compatibility - which normally forbids these projects to increment their majors number based on semver rule #1.
Python specific considerations
Python projects can decide to follow best practices on how to version their code and use common versioning libraries such as pbr. Although these are recommendations, OPNFV projects can still decide to not version their code and not use pbr for the euphrates release.
pbr is a widely adopted python library for managing versioning in openstack projects. It is used by almost every package in openstack: nova, neutron, keystone…
Such projects can manage their semver compliant versions using pbr.
Pbr is a setuptools plugin which makes it easier to manage versions in python in a way that works with python packaging and installation tools.
Pbr also ties versioning to the way git tags are used. For example, a project that uses pbr will never have to hardcode its version in the source code since the pbr library can automatically deduct the version from the repo git tags (in the case of a git clone) or from standard installation metadata (in case of a python package deployment).
I’m not going to describe all the features of pbr here, suffice to say that it is a mature and widely adopted versioning library that every python project should use (it has a few pitfalls but overall works pretty well).
One major side effect of using pbr is that git tags used for versioning need to follow a particular syntax in order to be “recognized” by pbr as a version tag.
Examples of git tags that are recognized by pbr as version tags (the pbr string representation is the version string returned by the pbr library and that is typically used for identifying the component version at runtime, e.g. --version on the CLI):
|git tag||pbr string representation||used for|
Examples of git tags that are not seen as versioning tags by pbr:
PEP 440 (Version Identification and Dependency Specification) provides recommendations on how python packages should be versioned.
While semver provides strict rules and semantic on each component of a version, PEP 440 complements it by defining how 2 version strings can be "compatible".
PEP 440 also defines a syntax for the version string and although that syntax is not fully compatible with semver, both specifications still use the same concepts and provide a smaller fully compatible version syntax.
GIT Tagging Restriction
One key characteristic of git is that git tags have a global scope at a repository level.
This means that for example you can’t have the same tag for 2 different commits or for commits in different branches of the same repository.
The direct consequence of this is that if the OPNFV release picks a particular tag format, that tag format will have an impact on all other tags existing on that repo – including repository specific tags.
A project user is any software or team that makes use of a project deliverable which can be binaries or libraries under the form of a standard packaging format (PyPI, git clone, docker container, VM image, PyPI, rpm, deb...)
A given version of a project can therefore generate multiple deliverables that all have the same version but under different format.
A popular project can have a large number of users and can be used by any combination of the following:
- other OPNFV projects
- one or more OPNFV releases
- one or more external projects (external to OPNFV)
As such, it is important to note that the OPNFV release is not the only consumer of a project and that each user has its own development and release cycle! For example, an OpenStack project which is tied to the OpenStack release cycle might use an OPNFV project.
Furthermore, a project can be used by one user as a library (e.g. pip install from PyPi package) and by another user in binary form (e.g. a docker container or rpm or VM image).
As an example, an OPNFV project X could very well be used by another OPNFV project Y at version V1 in source code form (this is hidden inside the project Y dependencies metadata) and at the same time contribute directly to the same OPNFV release (than project Y) using a different version V2 in binary form.
It is therefore critical that one single user does not impose in any way how a project should version its code.
Project Specific Versioning
With CD (Continuous Delivery), projects can evolve at a much faster rate than an OPNFV release.
Projects that do not participate in CD can keep using the “latest” tag to version their builds from master and use the official OPNFV tags for versioning their OPNFV release builds.
Projects that want to follow a CD model will need to produce and track a potentially very large number of project specific builds, some of which can be promoted as the next "stable" build to be used for all CI testing without waiting for the release branch to be in place (this is done at MS7 so pretty late in the release cycle).
Each of these builds needs to be versioned as they cannot all be versioned with the same tag (e.g. container image tag with “latest”). With git, each of these versions needs to be associated a git tag - which we can call a project specific tag.
Example of typical project using project specific tags (not all tags and not all possible branches are shown):
When compared to the pace of OPNFV release, the number of project specific tags may easily outnumber the number of OPNFV release tags (for example the OpenStack neutron project has well over 100 project specific tags for about a dozen OpenStack releases)
Because of the above git tagging restriction, project specific tags and OPNFV release tags need to be "compatible".
Docker Container Tags
For projects that generate Docker containers, Docker container images have a tag that is not necessarily the same as the git tag used to build the image (for example, "latest" or "stable" are common Docker container tags even when there are no "latest" or "stable" tags in git).
The “latest” container tag is convenient to use (e.g. if you don't care which version to use, just pick latest) but does not allow to trace easily an image to a git commit and therefore should be used with care and should not be used for identifying official images.
The most common method for naming docker container tags is to use the git tag used to build the container. As a result, the dockerhub folder for each project will have a mix of images corresponding to both OPNFV release builds and images corresponding to project specific builds. It is therefore important to use git tags that helps distinguish these 2 types of images to avoid confusion and issues related to the “wrong” container image being used.
Current versioning and tagging (before euphrates release)
Very few projects use project specific git tags, instead relying only on OPNFV release tags.
The good thing about these tags is that they clearly the release name along with a sub-release number and they hardly impose any constraint on how project can tag their project specific version (for those project that want to do so).
The downside is that the release number must be inferred from the release name initial character (e.g. c = 3).
Euphrates versioning and tagging (current TSC approved state)
Projects can still prefer/decide to not use project specific tags and continue relying only on OPNFV release tags. This time the release tags have changed to reflect the release number following the commonly used 3-number version syntax (major.minor.patch):
The main problem for projects that want to have their own project specific tags and follow semver/PEP 440 at this point is that they will be severely limited and constrained by the OPNFV release cycle given that semver versioning is now controlled by the OPNFV release for the entire project repository:
- project specific tags cannot use the 3-number version as they would conflict with OPNFV release tags
- which limits them to
- either use dev snapshots versions in a way that "works" with the OPNFV release tags if they want to use versioning libraries such as pbr
- or use non standard version tags and abandon any possibility to use pbr and face hardship versioning their artifacts in public repositories (which require standard version formats)
- major version increase must follow the OPNFV release major at all time (even if there is no API compatibility change, in violation of semver rules)
- any new deliverable to an external user before the next OPNFV release date must use dev snapshot versions
Example of project using dev shapshots as project specific tags that are in sync with OPNFV release tags:
The use of non standard version tags for project specific tags makes it very difficult to publish artifacts for external publication/consumption. PyPI for example pretty much requires all published python packages to follow the 3-number versioning syntax.
The only option would then be to only publish artifacts that have an OPNFV release tag but those are severely constrained by the OPNFV release schedule (e.g. the next 5.0.0 build available publicly cannot be used before the OPNFV 5.0.0 release is cut). Moreover the proposed OPNFV RC tags cannot be used externally either since their syntax is not semver compatible.
New revised proposal for tagging OPNFV releases and project repositories
In order to allow strict semver versioning on all projects that will follow CD, it is imperative that projects use semver compatible git tags that reflect the nature of their change history.
This implies that OPNFV release tags cannot use 3-number semver tags for tagging release builds. Instead it is safer to use a distinguishable format (that is not viewed by pbr as a valid version tag) for tagging all OPNFV releases.
- Projects that want to follow CD can use strict semver versioning without interfering with OPNFV release tags
- The tags for container images in dockerhub will clearly differentiate images related to OPNFV release from those related to project specific builds
- Projects can support more easily multiple independent users - even external users that have a release cycle different from OPNFV
- Projects will be able to follow industry best practice in versioning their code
- Does not force the use of developer snapshots (e,g, "188.8.131.52") for tagging project-specific builds - since not all project specific builds are developer snapshots (some project specific builds can be "official" versions for external users)
- No impact for projects that do not wish to follow the CD model
Note that the format of the OPNFV release tags used so far (until danube, e.g. "danube.1.0") were actually perfect in that regard as the tags were not recognized by pbr as versioning tags. If the OPNFV release number needs to be displayed in the release tag, to avoid any redundancy, one could use for example "opnfv-5.0.0" as the release git tag. This has the added benefit of clearly distinguishing OPNFV release images from project specific images in DockerHub (e.g. "opnfv/functest:opnfv-5.0.0" vs. "opnfv/functest:2.9.0")
When the project does not use project specific tags:
Example of project tagging when OPNFV release tags do not prevent projects to use strict semver versioning (not all project specific tags are represented, not all potential project specific branches are represented either):
Tagging Stable Builds for CD
(Discussion and options will be evaluated after the git tag decision is taken for OPNFV releases)