Linking different jobs to create pipelines in Jenkins

With the emergence of trends such as continuous deployment and delivery, the continuous integration server is not limited to integrating your products, but has become a central piece of infrastructure. Many of these jobs depend on the other jobs to be successfully completed. So you would want to proceed with a certain action only if the previous action is completed or do a different action. For example, consider below series of actionable tasks related to a project:

  1. Stop the server.
  2. Remove Website.
  3. Drop SQL Database.
  4. Clean up the file system.
  5. Create SQL Database.
  6. Create Website.
  7. Start the server.

So you can either create either a massive job or create small jobs which are linked together. If you go with the creating many small jobs, you can re-execute that job(read as redo that action), and not do the previous related jobs to it. In other words, you can start from a failing step rather than from very initial stage. So its always advisable to create multiple small jobs and link them together to create a pipeline.

In this blog post, we’ll learn how to create linking between different jobs or projects in Jenkins. For our learning purposes, we have created two small jobs named ‘Job 01’ and ‘Job 02’ in Jenkins server:

Two Small Jobs for creating linking.JPG
Two Small Jobs for creating linking

These two jobs contain nothing but a simple echo statement echoing the name of the job. Here are few of the steps required to create a very basic job.

One to One Relationship between Jobs

The first approach is pretty simple: one job triggers another one after successful completion. To achieve it, you can either configure a post-build action starting the next job (downstream) or configure the trigger of the current build (upstream). Both ways are totally equivalent. In order to this, let’s go to job 02 and select configure from dropdown:

Select Configure from black traingle next to job name.png
Select Configure from black traingle next to job name

Now go to ‘Build triggers’ and select ‘Build after other projects are built’. Under ‘Projects to Watch’ to select ‘job 01’:

build-trigger-options

Let’s select ‘Trigger only if build is stable’ so that ‘Job 02’ is executed only when ‘Job 01’ is successfully completed. After this, select save and apply. Now when you build ‘Job 01’ and see the log output, it should be something like this:

Job 01 build output after configuring upstream for job 02.JPG
Job 01 build output after configuring upstream for job 02

If you look at last 2nd line, you can see that the it started a new build of ‘Job 02’. Since it contains a link, instead of going to dashboard, you can directly click on the link to go to job. It will contains a new section named ‘Upstream Projects’ and would have listed ‘job 01’ as the build. Again from build history pane, you can see the recently build of this ‘Job 02’.

Using this one-to-one dependency approach is probably the simplest way to orchestrate jobs on Jenkins.

One to Many Relationship between Jobs

You can also create a master job and associate various child jobs to it. So let’s say we have ‘Job 01’ as master job and ‘Job 02’, ‘Job 03’, ‘Job 04’ as child jobs of it. Again, we’ll use the same method to create links between jobs as above and initiate build.

Job 01 build output after configuring upstream for job 02, job 03 and job 04.JPG
Job 01 build output after configuring upstream for job 02, job 03 and job 04

A Special Note

Please keep below points in mind, before going for one-to-many relationship:

  1. If one of the child job fails, other child jobs will still be triggered. They may or may not have some inter-dependency later in the flow. Off course, you can use the success of multiple jobs as the trigger as well. So you need to plan it carefully.
  2. There is no order defined for calling the child jobs. This is because Jenkins threads are asynchronous in nature.

Configuring custom relationships between Jobs

Let’s suppose your flow is something like this:

              /-> Job 02 -\
  Job 01 -> *                * ->  Job 04 -> Job 05
              \-> Job 03 -/

We can use the ‘Join Plugin’ for same. This plugin allows a job to be run after all the immediate downstream jobs have completed.   In this way,  the execution can branch out and perform many steps in parallel, and then run a final aggregation step just once after all the parallel work is finished. You can go to Manage Jenkins -> Manage Plugins -> Available tab and search for this plugin:

Install Join Plugin from Manage Plugins.JPG
Install Join Plugin from Manage Plugins.JPG

Select and install it.

Now Let’s go to ‘Job 01’ and go to post-build section. In build other projects, add ‘Job 02’ and ‘Job 03’. Then add join trigger from post build steps and enter ‘Job 04’. This will ensure that ‘Job 04’ is executed only when both ‘Job 02’ and ‘Job 03’ are completed. Again, you can use this plugin to create multiple diamond shape dependencies like above. So this helps in allowing creating complex pipelines.

 

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s