Welcome to the very first peek into the software kitchen of Selligent. We’re proud to guide you around and let you have a taste of our daily development concoctions. Obviously, as a shop that thrives on scrum values, we are also looking forward to improving ourselves based on any feedback that we might get.
That being said, let’s agree that software is not about the number of bits and bytes that you throw at a problem. Technical excellence is just an empty box, if it comes without a plan to go to production and solve actual use cases. It’s a bug’s lair, waiting to trap the next customer ticket.
The path to successful delivery is paved with a solid Software Development LifeCycle, an SDLC. For a company the size of Selligent, you better make that SDLC scale.
To illustrate why, I took a snapshot of our Git repository in the past year (until Feb 21, 2019), which shows that over 100 people touched at least one line of code. The most active engineer even committed code 800 times.
And while we do trust our developers, the sad truth is that the industry standard forecasts a whopping 15 to 50 errors per 1000 lines of code. Well, our codebase contains two million lines. With a total of 14K commits over that year, we were exposed to two new possible bugs, 35 times per day – without even going to production.
So how to subdue a volatile giant like that? There’s an easy answer: automate! Automate builds, automate test execution, automate deployment… automate going to the bathroom, if possible!
To achieve this (except for the bathroom, of course), we rely heavily on Microsoft Azure DevOps. In our current setup, code that ends up in production has passed +17K tests, including unit, integration and end-to-end tests plus after-deploy spot checks, all under the scrutinous watch of Azure DevOps.
Notably, our engineers hatch their code onto this eventful pipeline, pledging accountability at every gate along the way.
With delivery on autopilot, it sounds like we’re up for a walk in the park, doesn’t it? Almost. To continue exploring the room for improvement, a dedicated scrum team is tasked with the custody of the SDLC. This team – which not incidentally baptized themselves “Inception” – aims for one paramount goal: fast feedback.
Code taking weeks to get into production? Waiting an hour for a build to complete? Features that break a week after initial tests? With angelic patience, yet relentless tenacity, Inception weeds out the overhead of our processes and fosters a DevOps ecosystem, to enjoy rather than to fight. In their ideal world, “the process” is a whistling canary on your shoulder. Or a very silent one, when things are off.
In their secret weapons arsenal are the little gems that hide between the cracks of Azure DevOps. Let’s highlight this with a real-life scenario.
A developer wrapping up a task will typically create a Pull Request (PR), a bunch of code soliciting for feedback: Will the code build? Does QA approve? A rigid set of conditions defines whether the code is ready for master – you know, that cozy place where production code goes to live happily ever after.
Still, every now and then, the master used to contain broken features. But how is that possible, if all incoming code is under scrutiny?
During quality checks, QA and development make an impact assessment of the code changes. The result is a trade-off between completeness and test execution speed. So occasionally, a developer will shift a block in our Jenga tower without anyone noticing the falling brick on the other side.
Given unlimited resources, QA would run the full suite of tests on every code change. However, until quantum computing materializes, we’ll have to find other ways.
Inception tackled it this way:
In the UI, it would look like this:
Notice how they didn’t solve the issue of broken code on master, but they did tighten the feedback loop. A developer will now see the functional state of master right on his PR, which allows him to go in and fix that before he would throw another set of bricks at the already-fallen Jenga tower.
Now we get to the most interesting part of the story. Inception was able to solve this problem by two preconditions.
First: they recognized the value of the PR feedback feature on Azure DevOps.
Second: they created a so-called “Function” in Azure to do the heavy lifting. An Azure Function is basically a piece of code that can run from anywhere because it doesn’t need a server context. In this case, the code would fetch the latest test results from master, and report those back to the caller.
So, silver linings aside, this is where “the cloud” really shines. Due to our deep integration into the Azure ecosystem, any engineer can whip up a hosted resource like that, no internal ticket involved. Just make sure to keep an eye on the spending forecast and get some pennies from your wallet.
And since Inception practices what it preaches, they also conceived a proper pipeline, just to maintain this piece of code. This includes Unit Tests, staging deployment, integration test, and approval to go into production.
Rock on!
On a closing note: it’s probably not a coincidence that the driving force behind the PR feedback was the same engineer who championed with 800 commits on our product codebase in the past year. Yeah, Blaise Braye, I’m talking about you…
Marigold: where relationships take root.
