Blog post
October 7, 2019

Rebuilding Jenkins to avoid logjams

Automated code building and testing is transformative. It speeds up the coding process by allowing everyone to simultaneously write and deploy code, while continuously testing to ensure bugs don’t gum up the works. A problem comes, however, when your job queue is completely occupied with builds and automated tasks, preventing other important jobs from being quickly completed. 

Stephen Ball, Engineering Team Lead, addressed this issue by recently updating Jenkins, our open source automation system, to accommodate multiple server nodes.  

“Jenkins now separates our scheduled jobs from our builds and deployments so they never block one another,” explained Stephen. “It also spins up new nodes for each build and deployment—meaning we don’t need to worry about conflicts or jobs slowing one another down.” 

One huge benefit of this is being able to allocate a data pipelines for our Business Intelligence team for doing massive data processing jobs without fear of it impacting the engineering team or our customer-facing platforms.

Installing Jenkins is not quite as easy as pie

Our newly installed Jenkins, according to Stephen, required a lot of trial and error. Though Jenkins can be tricky to install, it has a large, well-established support community for sharing knowledge. And the extensive and ever-growing catalog of plugins ensures finding the right tools to update and enhance any system. 

Once under the hood, adding and updating Jenkins plugins, Stephen took the opportunity to re-write all our Jenkins build and deployment pipelines. “Doing this, I came to realise our projects all follow the same basic structure. But changes introduced in more recent projects were often not ported back to older ones.” 

To correct this, Stephen wrote shared functions that could be used on all projects. “Now when we make a change we don’t need to edit all 50 services. For example: to change the way our Slack channel messages are posted when a build is made, we only need to edit one shared function to change the Slack messaging for every service pipeline.”

The same is true of the deployments, according to Stephen. “We had the same complicated script duplicated for every project. Now it’s one script. So making changes across all projects is a lot simpler.”

Why Jenkins suits us

Jenkins is not the only automated system out there. “We looked at various others,” stated Stephen, “Being a finance company, though, we decided any remotely hosted solutions, could have potential compliance issues.” Jenkins also had the benefit of being open source, supported by a large community of developers and offering enough plugins to make it customisable to our specific needs.

The downside of Jenkins is you need a Stephen

Using Jenkins, code is easier to maintain and debug, builds take less time, teams are more tightly integrated, and product development can be truly agile. The downside is that Jenkins doesn’t install and maintain itself. You need someone, like Stephen, to set time aside from product development to install or periodically test the system to make sure it’s working properly.

Not every business has a Stephen, unfortunately. And those that do not prioritise installing and updating an automated system, like Jenkins, often end up reverting to their previous, labour-intensive building and testing. 

For us, however, the benefits of updating and expanding our Jenkins automation translates into huge, long-term benefits for all of LendInvest.

 

Interested in joining our growing product team? Check out what roles we have available here.

Leigh Rimmer
Posted by Leigh Rimmer
Your privacy matters
We use cookies to offer you a better experience. To understand what data we collect you can access our privacy policy and cookies policy. You can also adjust your or confirm your consent by clicking the accept button.

The cookies we use:

These are required for the operation of our website. For example, they support you to log into secure areas. You may disable these by changing your browser settings, but this may affect how the website functions.
These allow us to collect information such as how you navigate our website or if you experience errors which we can use to improve your journey.
These recognise you when you return to our website so we can personalise your content and improve your visit.