Page tree
Skip to end of metadata
Go to start of metadata

OPNFV CI Infrastructure

The information presented here is yet to be discussed within Octopus Team and should be seen as input to discussion unless noted otherwise.

There are several tools and hardware resources that make it possible for us to develop/establish CI Framework for OPNFV.

Most of the tools needed for CI are already provided by Linux Foundation. We need to setup/configure/utilize these tools.

OPNFV hardware resources will be delivered to LF during coming week(s) so first CI POCs can/will be done on community donated hardware if things don't move on fast enough.

We also need to identify what other tools/resources we need with the help from wider OPNFV Community and make them available with the help from LF.

Please note that the information on this page will be detailed/evolved while we move on.

Overview

Below diagram tries to capture some of the tools/hardware resources needed for Octopus and does not mean how the things work.

See this page for CI Flows.

Tools/Resources

Tool/Resource

Name

Status

Tasks Left to Do

Notes

CI Engine

Jenkins

Pending configuration

See below

Link to general info

SCM

Git

Pending configuration

See below

Link to general info

Code Review

Gerrit

Pending configuration

See below

Link to general info

Artifact Repository

TBD

TBD

See below

Link to general info

Build Servers

N/A

TBD

See below

N/A

Test/Deployment Servers

N/A

TBD

See below

N/A

Tasks

This section lists tasks to be handled in order to setup and configure tools and hardware resources.

Tasks that are not strikethroughed are pending.

Jenkins

Summary

Priority

Responsible

Notes

Create CI Service/Functional User

High

LF

Will be used for connecting LF/community Jenkins slaves, running jobs, voting in Gerrit, mail notification, chat, etc.
Suggested id: octopus

Configure CI Service/Functional User in Jenkins

High

LF

SSH keys etc. setup on Jenkins.

Configure Jenkins master/apply configuration to Jenkins Global Configuration

High

LF

Whatever is necessary.

Create domains

Medium

LF

Whatever is necessary.

Install Jenkins Plugins and configure them

High

LF

Please see below or click here for initial list of plugins.

Connect Jenkins to Gerrit

High

LF

Need to be able to trigger jobs on Gerrit events.
Please see here.

Connect/configure slaves

High

LF

Can/will start with echoing "Hello World!" by jobs running on master.
And then connect/use community(Ericsson/Intel BGS) hosting until LF resources are ready.

Create pools/attach labels on slaves so build and test/deployment resources are properly used

High

LF/Octopus Team

N/A

Configure mail notification

Low

LF

This could be handled like opt-in if people don't want to be bugged.

Make Jenkins talk/chat

Low

LF

Like it is done with #opnfv-gerrit chatroom.
Please see here and here.

Create basic jobs to evaluate what can be done without proper HW

High

Octopus Team

Create and run some jobs to see how urgent to get real HW for real Gerrit Verification flow etc.

Create basic jobs to verify the Gerrit connection

High

Octopus Team

Do simple commits and check if things are triggered/voted on Gerrit/mail notification sent.

Decide/agree which repos to monitor/trigger jobs on

High

Octopus Team/OPNFV Community

We start with the repos with some code/activity in them, such as BGS.

Git/Gerrit

Summary

Priority

Responsible

Notes

Configure CI Service/Functional User in Gerrit

High

LF

SSH keys etc. setup on Gerrit.
Also it should be granted to Gerrit stream events.
Please see here.

Configure Mail Notification

High

LF

This could be duplicate to Jenkins Mail notification and can be opted in.
But in general, people should be notified if a certain commit passed Gerrit verification so they can go and do the code review on Gerrit.

Enable Git/Gerrit hooks for large file check

Medium

LF/Octopus Team/OPNFV Community

This is still under discussion but it would be good if we can start with some initial setting so we don't mess up with the size of the repos right in the beginning.
This obviously requires discussion/agreement with other OPNFV projects who have Git repos.
Please see here.

Do simple commit/push/wait for verification/review stuff

High

Octopus Team

In order to evaluate if things are setup fine. Similar to Jenkins basic job creation task above.

Artifact Repository

Summary

Priority

Responsible

Notes

Agree on if we need artifact repository or not

High

Octopus Team/OPNFV Community

SSH keys etc. setup on Gerrit.
Also it should be granted to Gerrit stream events.
Please see here.

Decide which artifact repository to use with the guidance from LF

High

Octopus Team/LF/OPNFV Community

Swift/Artifactory/Nexus

Guide OPNFV Community regarding artifact repository selection

High

LF

Given that LF already hosts repositories for other OSS projects.

Decide and create repositories

Medium

Octopus Team/LF

We can start with "a" repo and evolve later on.

Decide retention policy

Medium

Octopus Team/LF

If storage becomes the bottleneck, we may need to start cleaning up based on the policy.

Acquire/provide storage if storage becomes the bottleneck even after enforcing the the retention policy

Low

OPNFV Management

This needs to be evaluated while we move on and raised by the community.

Hardware Resources

Summary

Priority

Responsible

Notes

Configure/connect/evaluate Ericsson hosting until LF resources arrive

High

LF/Octopus Team/Ericsson OPNFV Contributors

This requires discussions.

Configure/connect/evaluate Intel hosting until LF resources arrive

High

LF/Octopus Team/Intel OPNFV Contributors

This requires discussions.

Assist Ericsson/Intel OPNFV Contributors to securely connect hosting to Jenkins

High

LF

This requires discussions.

Configure/connect LF resources

High

LF/Octopus Team

N/A

Jenkins Job Builder

During Octopus Team Meeting on 2015-02-16, it has been suggested to give a try and perhaps use OpenStack Jenkins Job Builder for OPNFV CI.

OpenDaylight also uses Jenkins Job Builder for Jenkins Jobs so this is a common way of handling Jenkins related things. As OPNFV uses OpenStack and OpenDaylight as upstream projects for the components themselves, using/doing CI in a similar way is also logical.

As a first step, LF will help us setting up some simple jobs using Jenkins Job Builder and they will be put in Octopus repo on OPNFV Gerrit. Then we will continue with the rest of the evaluation but this looks quite promising as the things stand.

Jenkins Job Builder helps us to develop/maintain/handle Jenkins job configurations like the source code; changes to jobs go through regular Gerrit review process and makes things visible to greater OPNFV Community. Having job configurations version controlled has its own benefits which is not needed to be listed here. Deploying those jobs could also be automated.

Apart from all the good things listed in previous sentence, it also relieves us from creating/modifying/maintaining Jenkins jobs from Jenkins GUI, potentially reducing chance of mistakes and annoyance of clicking things around.

Here are some links

List of Plugins to Install

This is the initial list of plugins to install and new ones will probably be added based on Octopus/Community feedback/needs.

The ones that are marked as low priority can be skipped for the timebeing to see/evaluate the load on Jenkins master when other plugins are installed. If the load looks fine, they can be installed as well.

Plugin

Purpose

Responsible

Priority

Link

SSH Slaves Plugin

Connecting slaves to Jenkins with SSH

LF

High

SSH Slaves Plugin

Git Plugin

Use Git as SCM

LF

High

Git Plugin

Gerrit Plugin

Integrate Gerrit with Jenkins

LF

High

Gerrit Plugin

Gerrit Trigger Plugin

Integrate Gerrit with Jenkins to trigger jobs

LF

High

Gerrit Trigger Plugin

Parameterized Trigger Plugin

Create jobs that can trigger other jobs and take action depending on result

LF

High

Parameterized Trigger Plugin

Multijob Plugin

Create complex/hierarchical jobs

LF

High

Multijob Plugin

Job DSL Plugin

Create jobs programmatically

LF

Medium

Job DSL Plugin

Workflow Plugin

Create workflows

LF

Low

Workflow Plugin

Robot Framework Plugin

Publishes/stores Robot FW test reports

LF

High

Robot Framework plugin

Throttle Concurrent Builds Plugin

Throttle concurrent builds

LF

High

Throttle Concurrent Builds Plugin

Email-ext Plugin

Send customized emails

LF

Medium

Email-ext Plugin

Instant Messaging/Jabber Plugin

Make Jenkins talk/chat

LF

Low

Instant Messaging Plugin, Jabber Plugin

eXtreme Feedback Plugin

Show quick build status

LF

High

eXtreme Feedback Plugin

Build Pipeline Plugin

Create and visualize pipelines

LF

Medium

Build Pipeline Plugin

Green Balls

Use Green Balls for successful builds

LF

High

Green Balls

Plot Plugin

Create plots

LF

Low

Plot Plugin

Priority Sorter Plugin

Adjust priorities of jobs

LF

Medium

Priority Sorter Plugin

Dashboard View Plugin

Create dashboards

LF

Medium

Dashboard View

Static Code Analysis Plugin

Provide Static Code Analysis utilities

LF

High

Static Code Analysis Plugin

Artifact Deployer Plugin

Deploy artifacts from workspace to output directories

LF

Medium

ArtifactDeployer Plugin

Promoted builds plugin

Mark builds as promoted

LF

Medium

Promoted Builds Plugin

Build Failure Analyser

Present causes of failures

LF

Low

Build Failure Analyser

Build Timeout Plugin

Set timeout so builds don't get stuck

LF

High

Build Timeout Plugin

EnvInject Plugin

Inject env variables to builds

LF

High

EnvInject Plugin

Git Parameter Plugin

Choose revisions or tags

LF

Medium

Git Parameter Plugin

Job Config History Plugin

Track job configuration history

LF

High

JobConfigHistory Plugin

Log Parser Plugin

Parse console logs

LF

High

Log Parser Plugin

Mask Passwords Plugin

Mask passwords

LF

High

Mask Passwords Plugin

Post Build Task Plugin

Do post build tasks

LF

High

Post Build Task Plugin

Show Build Parameters Plugin

Show parameters used for a certain build

LF

High

Show Build Parameters plugin

Timestamper Plugin

Add timestamps to console

LF

High

Timestamper Plugin

Delivery Pipeline Plugin

Visualize delivery pipeline

LF

Low

Delivery Pipeline Plugin

Robot Framework Plugin

Publishes/stores Robot FW test reports

LF

High

Robot Framework plugin

  • No labels