Lessons learned about Jenkins

Jenkins is not always the easiest tool to work with, sometimes getting more in the way than helping. This page attempts to document some issues I've run into during my time with Jenkins. Some solutions are applicable to systems in general, some to other CI daemons. It is intended to be continually updated, which has happened once so far.

Documentation is rather poor

This is something I think anyone who has tried to do anything more than just the basics has run into. Much of it stems from the fact that most functionality in Jenkins is provided by plugins, and the plugins (even official ones) depend on the maintainers, which are usually not as invested as the maintainers or Jenkins itself.

The goal of the core project seems to be to automatically generate most (all?) documentation so that it doesn't matter if it comes from the core or from a plugin. This still has quite a ways to go before it will cover more than just the bare basics.

Solution: There isn't really one... read the source, use whatever docs are avaliable, create spike solutions

Console output is slow

Streaming logs "live" to the Jenkins master may cause significant slowdowns in builds that produce large amounts to logs.

Solution: Don't send large amounts of logs to the master. Pipe the output of commands run on slaves to a file, and then archive that file instead. There is an issue in Jenkins issuetracker to resolve this problem with external logging mechanisms: JENKINS-38313

Pipelines

Pipelines are slower than the identical freestyle job

One reason this may be is because pipelines are more "durable" than freestyle jobs. If the Jenkins crashes during execution of a pipeline, it can be resumed, most of the time. Not so with freestyle jobs.

The way this is implemented is by writing every step taken to disk, much like a journalling file system. This means that if you have lots of steps in your pipeline, there will be lots of extra writes to disk, and possibly communication to the master. If your disk is slow (NFS) this might have a noticable performance impact.

Solution: Since LTS 2.73 (+ related plugins) you can override the durability setting as outlined in the linked article.

options { durabilityHint("PERFORMANCE_OPTIMIZED") }

(No, this is not really documented anywhere) Now documented through the declarative snippet generator. On the pipeline level, this will override the default durability to be essentially the same as of that for freestyle jobs.

Testing pipelines is hard

When we build pipelines-as-code this can quickly become a problem. JenkinsPipelineUnit has a solution, allowing you to mock out many things. It does not support declarative pipelines however. I have however not had any luck with it, prefering declarative pipelines wherever possible.

There is some help to be had for declarative pipelines however, the jenkins-cli.jar (accessible from /cli on a master) allows you to run the declarative-linter. This seems to be the same thing that Jenkins runs before it starts to execute declarative pipelines. It is fairly trivial to set up a shell script to run a file through this linter. I personally set up a hotkey in my editor (IntelliJ) to run such a shell script on the current file. If working with a repository which has mostly pipelines, I also like to set up a 'build' job that runs all pipelines through the linter. This can guard against bad commits if you have a build-before-commit strategy.

The best strategy to minimize this problem is however to not put any logic in the pipelines that don't explicitly has to be there. In general, the thing we want out of our pipelines is reporting and perhaps some conditionally running stages. Everything else can move out into more easily testable scripts. I prefer python in the general case and gradle if I want up-to-date checks.

It should be noted that pipelines that 'build' code (Ant/Maven/Gradle, etc) should use the build tool as far as possible, since this minimizes the differences seen by the CI daemon and developers building on their own machines. Prefably there should only be one way to build it.

Solution (opinionated): Reduce complexity through logic extraction out of the pipeline.

Passing back information from called processes

Passing in information is usually straightforward, command line arguments. This is a one way street however. For passing information back we have a couple of options:

• Pipes
• Files
• Return codes

Pipes: stdout: bat, sh, powershell all have the returnStdout parameter, which when set to true will mean that the step will return everything printed to stdout when finished. It will also prevent this from being logged in the build log (console output). It is usually wise to .trim() the output in the pipeline.

stderr can be redirected to stdout in bat & sh by 2>&1, but usually it is useful to let stderr go to the build log instead. Example: Pythons logging lib goes to stderr if nothing else is specified.

Other pipes can be used as well, but on *nix at least, these really just work like files for our purposes.

The obvious drawback here is if you cannot control stdout fully for your process you need to do filtering in the pipeline, which is less than ideal.

There is also a little gotcha with bat: echo of the commands is ON by default, as in all .bat scripts. The simple way to prevent this is by prepending @ to your bat lines, which suppresses the echo for that line only. @echo off can be used as the first line of larger blocks to suppress echo of the commands for that entire block (file, really).

Files: Perhaps the most obvious way is to let the process write to a file which you then read in the pipeline. The drawback here is that there is disk IO for something that should perhaps only live in primary memory. When doing this, let the pipeline pass in the path of the file to write to, removes the potential for de-synced names across files.

If you want to save the passed information, this is probabaly the way to do it, since it is easy to archive or stash the file for later.

Return codes: bat, sh, powershell all support returnStatus, which if set to true makes the step return the status code. This also prevents the pipeline from failing if the status code is non-zero.

Return codes can be used as a way to pass information back to the pipeline, even if this is somewhat standards breaking. If you only wish to pass a few bits of information, like flags, or a small integer, this can be used effectively. On most POSIX systems, this value might be truncated to 8 bits (mod 256), which limits the uses. Windows uses 32-bit integers for return codes.

Other ways are possible, through a DB, over the network etc. But in most cases these are overkill.

Solution: Use what best fits your situation. Files are simple, pipes are fast (no disk IO), return codes can be used to drive your collegues insane.