We are the Dev Teams of
  • brands
  • ebay_main
  • ebay
  • mobile

Taming the Hydra (part 3)

by Marc Günther
in How We Do It

Jenkins LogoHow to take care of a large Jenkins installation and still keep your sanity

Part 3: The Scary Details

"Sometimes maintaining our Jenkins infrastructure reminds me of fighting the fabled Hydra. Every time you slash a problem in one of the jobs, in the meantime ten more jobs have sprouted, each of which will pose their own problems in the future."

Sorry, it’s been a long week since we looked at the possible solutions in Part 2. So, without losing more time, let’s dive right into the implementation details of our current solution, the “Job Generator”. First, we look at the templates, and how they work.

The simplest template

As an example, let’s start with the most simple “job template” that actually does something useful (BTW, don’t confuse our usage of the term “job template” with the Job-DSL-Plugin’s Template Jobs, which is something entirely different):

mobile_job(config) {
  steps {
    mobile_maven(config.maven_default_goal ?: 'clean install')
  publisher {


This, as you might have guessed, defines a job which (by default) calls “mvn clean install”, and also archives jUnit results.

How to use

Templates live in our “Job Generator” repository in a special folder and are referenced by name.

In order to use the above template, a developer would put a jenkins.yml file in her repository, which specifies the template’s name. Let’s say the template is called “very-simple-maven-job”. The jenkins.yml file would then look like this:

template_name: very-simple-maven-job

As soon as this file is committed and pushed, our system will execute the template, which creates the corresponding job, and then run the job.

Understanding the DSL

If you know Job DSL Syntax, you might recognize some parts (notably the steps and publisher sections and the archiveJunit command). You will also notice some unknown commands (mobile_job and mobile_maven), and a strange variable called config.

And if you really know the DSL syntax, you will notice that some important parts are missing completely (eg. the job’s name, and the complete scm section).

So what’s up with these? The interesting thing here is that the generated job does much more than catches the eye. It also checks out the repository (obviously, otherwise it would be quite useless), triggers on SCM changes, sets some basic options like console timestamps and colorized Maven output, lets the user choose JDK/Node.js version, define HTML reports, environment variables, pre/post build steps and what not…

The template can do all of this, because our “Job Generator” provides two important things on top of the Job-DSL-Plugin. It:

  • extends the DSL with our own commands (mobile_job, mobile_maven), and
  • provides a way to configure the job through the jenkins.yml file (the config variable).

Let’s tackle these one by one.


We define our own DSL commands using Groovy Monkey Patching. For example, mobile_maven is simply calling the original maven command with all the options we typically use in all jobs:

StepContext.metaClass.mobile_maven = { String my_goals, Map my_props = [:] ->
  maven {
    my_goals.split(/\s+/).each { goals(it) }
    mavenInstallation('Maven 3.2.5')
    mavenOpts('-Xmx1024m -Xms512m')
    property('maven.test.failure.ignore', 'true')

“mobile_maven”: calling Maven with some useful default settings

The mobile_maven command takes two parameters: a list of goals and a list of properties. The whole thing is defined in the “Job Generator” itself, and can be used by every template. We provide several more of these commands:

  • mobile_job: a normal freeStyleJob, with all the useful options I mentioned above.
  • mobile_list_view: a Jenkins List View, with our default columns already pre-configured
  • mobile_git: a Git section which knows about our two Git servers
  • mobile_maven: see above
  • mobile_jdk: our available JDK installations
  • and some more, for example, build steps to trigger deployment into our testing systems using Autodeploy, …

Side note: The term “mobile” here does not refer to the normal English word (like in mobile phone), but rather the name of our company mobile.de, which is short for “automobile” and pronounced rather differently in German. Try to guess how we call our real mobile apps for Android and iOS…


OK, now that we have a way to avoid duplicated code and ease maintenance by extending the existing DSL, how do developers fine tune their jobs?

If you remember the very-simple-maven-job template from far above, the Maven step was called like this:

steps {
  mobile_maven(config.maven_default_goal ?: 'clean install')
} ...

The first argument to mobile_maven is the Maven goal to execute, which here is taken from the config.maven_default_goal variable (defaulting to “clean install”). So where does this variable come from? Easy enough, it’s taken from the jenkins.yml file.

Let’s say our developer would like to change that Maven call to “mvn clean deploy”. She would make the following change to her jenkins.yml file:

template_name: very-simple-maven-job
maven_default_goal: clean deploy

“jenkins.yml” with added Maven goal

And again, as soon as this change is pushed, the “Job Generator” will update the existing job, changing the Maven goal in there, and trigger the job.

The config variable

To summarize, this variable is a normal key/value map, and available to every template. Its contents come mainly (but not only) from the jenkins.yml of the repository in question. So whatever you put into that file can be used by the template by referencing config.whatever. As yaml is a hierarchical data format, you can even use lists and maps as values.

In addition, the generator itself sets some hardwired values that you can not change. These are mostly meta-information about the current repository, like the repository name, clone URL, organization, list of branches and forks, default branch name, description, etc.

To solve the mystery above of how the generated Jenkins job gets its name and SCM section, let’s look at the definition of our mobile_job command. It contains stuff like this:

JobParent.metaClass.mobile_job = { Map<String, Object> config, Closure closure ->
  freeStyleJob(config.job_name) {
    description config.job_description
    mobile_jdk config.jdk_version
    scm {
      mobile_git config.repo_clone_url

“mobile_job”: Usage of the config variable

Here the freeStyleJob is initialized with config.job_name, and the SCM section with config.repo_clone_url, both of which are automatically provided, hardwired values. On the other hand, config.jdk_version needs to be set from the jenkins.yml file (if unset, it defaults to something reasonable like Java 7).

The overall architecture

Let’s look at all the pieces of the system. By now we have seen:

  • templates
  • extension commands
  • jenkins.yml and the config variable

We have not yet seen:

  • the actual generator job itself.

The “Job Generator” Jenkins Job

This is a normal Jenkins job (what the Job-DSL-Plugin calls a Seed Job), which contains a Process Job DSLs build step (provided by the Job-DSL-Plugin) that executes our Groovy code.

The generator job always only processes one repository at a time, and is parameterized to take the name of that repository. It is triggered by web hooks from our GitHub and Gitorious servers, which fire on every change to every repo (well, all pushes and branch creations). This means it runs a lot, but so far this hasn’t been a problem, as it only takes about 10 seconds to finish.

What exactly does it do? Let’s see how a git push propagates through the system…

The life of a Git commit

As shown in the image below, when somebody pushes ① a change to one of our GitHub/Gitorious repositories, the following will happen:

  • the generator job is triggered by a web hook ② from the Git server
  • the generator job then processes that (and only that) repo:
    • it reads ③ the jenkins.yml file from the repo
      • (if the file doesn’t exist, the generator stops here)
    • and puts its contents into the config variable
    • it then looks for a template called config.template_name
    • and executes it, passing the config variable in its binding.
    • the template then creates/updates/deletes ⑤ the corresponding jobs/views/folders, using the DSL commands described above
      • (most of the time, nothing has changed, so the generator will not touch any jobs at all)
  • the relevant job(s) then get triggered ⑥ by the usual web hook with a 15-second delay

Job Generator architecture
How the ‘my-cool-project’ repository (upper left) gets its corresponding job (upper right)

The “vision” revisited

Did we succeed in implementing The Vision from part 1? To recap, we were looking for a mechanism that allows us to:

  • administer the similar parts of the jobs in one central place…: Yes, we have the templates and the extension commands, both of which live in the Job Generator repository.
  • keep the specific configuration parts of the jobs separate: And these would be the jenkins.yml files, which are stored in each individual repository.

So I would say in theory we definitely succeeded. But how did it work out for us in practice? After all, the difference between practice and theory is usually bigger in practice than in theory.

In part 4 we will look at the problems we encountered along the way, like speed, deletion of jobs, agreeing on what the templates should do, creating them and making changes without breaking everything, how to structure Jenkins views, some nice goodies we came up with, and improvements we would like to do, like speed and the “meta-generator”.

The other parts of the series:

- Part 1: The vision
- Part 2: The alternatives

jenkins, agile, automation, devops