We are the Dev Teams of
  • brands
  • ebay_main
  • ebay
  • mobile

Taming the Hydra (part 2)

by Marc Günther
in How We Do It


Our weapon of choice

Sorry to keep you waiting. I hope Guy with just two heads didn’t eat his complete keyboard by now. :)

So in Part 1 we were looking for some kind of flexible templating mechanism. Some we had to reject right away, but following are the interesting alternatives we stumbled upon….

Internal development

We already had two home grown mechanisms, one based on the Jenkins REST API, and one based on Groovy system scripting (which is very powerful). Both turned out to be too inflexible, too slow, too hard to use and modify, and also too inflexible. Nevertheless, we used them for several years.

  • Pros:
    • written by us
  • Cons:
    • hard to modify, need to edit config.xml manually
    • no flexibility at all. All jobs have to be exactly identical
    • unmaintainable and way too slow
    • written by us

CloudBees Template Plugin

A possible solution could have been CloudBees, which provides a Template Plugin as part of their Jenkins Enterprise offering.

  • Pros:
    • GUI: Provides a nice configurable configuration UI for each template, making it very easy for developers to fine-tune their jobs.
  • Cons:
    • XML: it’s again all based on writing config.xml by hand. We completely lose the existing configuration UI. And which sane person would willingly write XML by hand?
    • no inheritance: all templates are completely independent of one another. Which means we still have to copy and paste all the common stuff between all the different templates.
    • licensing model: CloudBees only offers the plugin as part of a big bundle on a subscription basis, based on the maximum number of executors. As we were planning to move to Docker to spawn executors on demand, this wasn’t acceptable for us.

Luckily, some quick duckduckgoing turned up an alternative so promising that we didn’t look much further.


The Jenkins Job-DSL-Plugin allows you to define jobs using its own Groovy domain-specific language (the Job DSL, also see the Reference). This one is way easier to read and write than XML, and it has the advantage of being a full-blown programming language, meaning it can do anything

  • Pros:
    • very actively developed and maintained
    • supports lots of Jenkins plugins
    • easily extensible for not yet supported plugins (which we did)
    • Groovy DSL gives you the full power of a (very cool) programming language, instead of just a templating engine
  • Cons:
    • we still lose the existing Jenkins configuration UI

We have a winner

So what can we do with Job-DSL?

Job-DSL-Plugin Examples

The following DSL snippet defines a very simple hello-world job:

freeStyleJob("hello-world") {
  steps {
    shell 'echo "hello world"'

Hello World, DSL style…

The important thing here is, that this is not some static declaration language, but actually a Groovy program. It doesn’t look like a program, because Groovy syntax allows you to omit semicolons and most parentheses, but when we add them back in, we get this more unreadable, but otherwise identical program:

freeStyleJob("hello-world", {
    shell('echo "hello world"');

So freeStyleJob is actually a method with two arguments (a String and a Closure), steps is a method with a single Closure argument, and shell is a method with one String argument.

When you execute this program (using a special build step provided by the job-dsl-plugin), it will generate a job called hello-world, which outputs “hello world”.

The next example defines a job that actually does something useful:

freeStyleJob("my-cool-project") {
  scm {
    github('mobile-de/my-cool-project', 'master', 'ssh', 'github.internal.domain')
  triggers {
  steps {
    maven 'clean install'

When executed, this will create a job which will checkout the specified repo, and run mvn clean install. It will also trigger on commits to the master branch.

These were some very simple examples. The job-dsl-plugin offers a whole lot more of these DSL commands, which allow you to configure your job to your hearts desire. And if some obscure plugin configuration is not (yet) supported, you can easily extend the DSL yourself, or output the relevant XML directly (using nice Groovy syntax to generate the XML).

To summarize: The DSL allows you to create every kind of job that you could also create by clicking around in the “Configure” page in Jenkins, but without the clicking around. And it might also be important to mention, that it does NOT allow you to create any new kinds of jobs with sudden magic capabilities, that you couldn’t create before. It’s still the plain old Jenkins jobs that you know and love.

Treat job configurations as source code

So where does this all lead us? Well, it radically changes the way you think about Jenkins job configurations. It allows you to treat them as source code, as programs. This idea is so important, that I feel it necessary to repeat it:

With Job-DSL-Plugin, your Jenkins job configurations become source code

The possibilities of this are endless. You have the full power of a programming language, with conditions, loops, inheritance, etc at your disposal. You can put it in version control. Well, you can do all the things you want to do. No more limitations, unsupported plugins, strange behaviour, ….

To bring up an analogy familiar to anyone: The difference of this approach as compared to all those other methods is similar to the difference between a set of static html pages and a dynamic webapp. Both output html pages, but the latter is overly more powerful.

Use the force, Luke…

So what can we do with this new power we have been granted? Here is a more interesting example:

@Grab(group='org.kohsuke', module='github-api', version='1.59')

def gitHost = 'github.internal.domain'
def gitURL  = "https://${gitHost}/api/v3"
def gitAuth = '...'
def orgName = 'mobile-de'

def githubApi = org.kohsuke.github.GitHub.connectToEnterprise(gitURL, gitAuth)

def maven_goals = 'clean install'

githubApi.getOrganization(orgName).repositories.values().each { repo ->
  // begin - actual job definition
  freeStyleJob(repo.name) {
    scm {
      github(repo.fullName, repo.masterBranch, 'ssh', gitHost)
    triggers {
    steps {
      maven maven_goals
  // end - actual job definition

using GitHub API to iterate over all repositories

Again, we define the same kind of Maven job, but this time we loop over every repository in the given GitHub organization, and create one such job for each repository (running this took 18sec to create 209 jobs). And if we need to change the job definition here, we simply run it again, and all the jobs are updated in place.

We want more

Now, this was already better and faster than at least one of our two existing inhouse mechanisms, but now that we had seen the light, we wanted a lot more flexibility.

Developers should be able to:

  • choose from different kinds of jobs (packaging, testing, merging, deployment, …) for their repository
  • fine-tune the chosen job (eg. choose JDK version, special build steps, etc…)
  • and even have multiple jobs for each repository.

And of course, everything should be fast, easy to use, fully automatic and extensible/customizable.

The vision takes shape

With The Vision from last weeks episode on our mind, we started hacking away on the above code snippet (yes, that was actually the start of it all) and split it up into three parts:

  • the mechanism itself: The github loop became what is now known as “The Job Generator” and lives in its own repository.
  • the similar job configuration parts: The “actual job definition” from above code snippet we extracted into separate files, and put them into a sub directory (called templates) inside the generator.
  • the specific job configuration parts: The parameters (eg. maven_goals from above) to fine tune the jobs we extracted into a config file (called jenkins.yml), which lives not in the generator, but in the repository of the job itself.

My naming mistake: Despite all I have just said about the DSL job definitions not being mere templates but actual programs, I made the mistake of calling them templates all over the place. It is now too late to change that, but I think this has strongly (and wrongly) influenced the perception of the expressive power of these job definitions by our fellow developers.

Where we are now

The current status of our Job Generator, as seen from a users perspective. The overall architecture and technical implementation details will be explained next week in part 3.

Ease of use

All configuration happens through the jenkins.yml file (something like a .travis.yml file) in the repository, where you can choose which job definition (I mean template, aahrg) to use, and also configure the options for that template (like which JDK or NodeJS version to use, setting environment variables, injecting pre/post build steps, etc…)

This file lives in the master branch of the repository, and changes to it will be applied as soon as you git push it back to the server. No other way of configuring a job is necessary nor possible.

template_name: multi-branch
show_forks: true

  LANG: de_DE.UTF-8
  DEBUG: info

jdk_version: jdk8
nodejs_version: '0.12.7'

Example ‘jenkins.yml’ file, which sets jdk and nodejs version, some env variables, and uses the ‘multi-branch’ template, which creates jobs for every branch, and here has been told to also create jobs for all forks.

The available options are linked on every job page in Jenkins.

Automatic and Fast

The generator is triggered by github/gitorious webhooks, so responds very fast to all events. No need to run it manually. For example, if your template creates a separate job for each branch in the repository, then as soon as you create and git push a new branch, the generator kicks in, and creates the job for that new branch (likewise for deletion or changes to jenkins.yml).

Customization / Extensibility

There are several templates available in the generator by now, some are quite generic and configurable and used by lots of repos (for example a template for artifacts that follow the standard Maven release process), others are only used by one repo (for example the template for our iOS App).

In case there is no template that does what you need, you can either create a new one, or you can try to modify an existing one, for example by introducing new configuration options into it. We provide a sandbox to play around to test your changes, and regression tests for all existing templates and options to make sure your change didn’t break existing behaviour.


The Job Generator” allows:

  • developers to easily create and configure the jobs for their repositories, without the usual copy/paste problems, while still allowing
  • our team to maintain and change all these jobs from one central place.

Tune in next week for the implementation details…

The other parts of the series:

- Part 1: The vision
- Part 3: The implementation

jenkins, agile, automation, devops