Shift left: the theory and concepts behind our On-Demand Test Environment
At Bold Penguin, we take developer experience very seriously. We also take software reliability very seriously. As our team has grown, we noticed code was spending a lot more time in our testing environments. Our automated delivery pipeline makes it easy for us to move from left to right very confidently, but any bugs or issues introduced into the pipeline led to traffic jams. Older things needed additional work to go to production which caused delays in getting newer things into the pipeline.
Not only was this causing us to deliver value to our customers more slowly, it also started to create a negative feedback loop. Because the time between writing code and shipping was increasing, we found ourselves creating larger pull requests which increased the surface area of testing which led to even more time in our lower environments which led to larger Pulls Request etc. Our current systems made sense when we were a much smaller team but we knew we had to make changes if we were going to scale effectively.
Our automated delivery pipelines are triggered off any changes to the master branch. Any updates are immediately compiled and deployed to our internal testing environment (Alpha). At this point, these changes will either be approved to go to the next environment, or a subsequent PR will need to be merged into master to revert the changes.
As we’ve grown we have found that our velocity has started to work against us. The one-way nature of our automated delivery pipelines has caused us to be uncomfortable with putting changes into our test environment out of fear that one of those changes will introduce undesired side effects and essentially blocked our pipeline.
One of the ways we are approaching our pipeline slowdown problem is by introducing tooling that allows us to run quickly provisionable test environments. On-demand environments are spun up for our front-end applications that get created automatically when pull requests are open. They exist outside of our normal pipeline and exist only as long as they are necessary. By getting deployed versions of our code publically accessible, we get the opportunity for feedback before code gets merged in. Engineers reviewing the code don’t have to pull down every pull request to test locally and our product team can ensure we are solving the correct problems. We even share these builds with our customers to get early feedback on new features!
Here’s an example:
When we push a PR, a new Github Action triggers.
When it finishes, it creates a deployment on the Pull Request.
Clicking the View deployment button brings you to a live, on demand test environment.
There are a few pieces that make the magic happen.
So in a nutshell:
Since introducing on-demand test environments, we have increased our developer productivity and happiness, gotten more feedback sooner, and have started to speed back up. We’re thinking a lot about how we can continue to scale our processes and tooling to empower our ever growing team’s demands. If these sound like problems you’re interested in helping us solve, we’d love to talk with you! Visit https://www.boldpenguin.com/careers to see our latest openings.
Bold Penguin's CTO Dishes on the Amazon Insurance Accelerator
Penguins teams unite to build on their respective heritages of success
Partnership will allow thousands of agents across the U.S. with the ability to quote and bind comprehensive cyber coverage in minutes.