"I'll just push this fix to production really quick." We've all heard someone say that - or even said it ourselves. We've all been there: under the gun, a critical issue breathing down our necks. Of course it's after hours on a Friday and you had plans. Those plans are now ruined and worse, it's all riding on you. The pressure in that moment is real, the embarrassment of having a broken user experience, a site or service down, our phones blowing up with alerts and stakeholders all banging on the door.
These are our nightmares, the worst moments in our careers. The times we wish we could forget. These are also the proving grounds, the times when the disciplines of good software engineering are the most important. These are also the times when "cowboy coding" is the most appealing and rides rampant, even among the best software engineers.
What is cowboy coding?
Cowboy coding is a derogatory term for going in guns blazing and quick fixing an issue. It feels good in the moment, to ride in and solve the problem all by yourself. To make that two line change and get it in production inside of five minutes. But the costs involved are high and often subtle. Organizations can lionize developers who can perform this way when the stakes are highest. Unfortunately what is easily missed is that high stakes development breeds more high stakes development. Foregoing good practices for the quick fix often leads to more quick fixes.
What are good processes?
The best software development is full of "boring" processes. Boring doesn't mean dull, it means routine. Routine feature planning, routine code reviews, routine testing, and routine releases. These are exciting because of their impact, not because your team crossed their fingers and it magically and unexpectedly all worked. The most exciting, thrilling parts of being a software engineer are not the times when everything hits the fan, instead they are:
- That Tuesday afternoon where you had a breakthrough on a hard problem.
- The pair programming session where you solved an elusive bug.
- The Friday morning code review conversation that really took code quality to the next level.
- The five star client feedback where you could hear just how pleased they are with what you've built.
Cultivating these kinds of environments takes work. It takes discipline, especially when stakes are highest. The rewards make it totally worth it.
What to do when the stakes are highest?
So you're there, in the middle of the worst bug of your career. Your boss is asking when it'll be fixed, clients are emailing, your spouse wants to know when you'll be home. What do you do in that moment? That moment is the one all of your processes were built for.
1) Call in help
Usually this is our last instinct when things are rough, especially if it's our fault. We don't want to burden someone else with our bug and the intensity of the moment. But this is why having a team is invaluable. When the stakes are highest, you need another set of eyes evaluating the problem holistically. Someone who can back you up on your solution and thinks through things from a developer's perspective. Fight against the urge to figure this out alone and call in help. You'll be glad you did.
2) Don't skip code review
Someone may say "code quality doesn't matter when we're experiencing an outage." I get where they're coming from, it feels like a nice to have, not a must have when the stakes are high. However this is the time when code review matters most. Subtle bugs and unintended consequences sneak in when you're under pressure to deliver a quick solution. It doesn't have to take long, but get a teammate's eyes on the code to make sure one issue doesn't doesn't turn into two.
3) Test before releasing
I don't mean just test the fix, I mean test everything. Run your testing suite. Manually test. Do whatever it takes to be sure everything is functioning as it should before you release your fix. Not testing leads to cascading bugs. This is even more true in urgent situations where a fix is needed immediately. Take the extra time to make sure this fix actually fixes the issue and doesn't break anything else.
4) Release with confidence
You've had two team members asses the problem and the fix. You've had another code review your proposed solution. All tests have passed and QA is done. Now you can release using your normal release process, communicate with the stakeholders, clients, boss and team, then head home. You can be confident that your work really solved the problem,
5) Do a retrospective
Take time after all the fires are out to sit down with your team and evaluate what happened. Talk about your successes in handling the problem, any failures and what led to the bug in the first place. Change your processes as necessary. Thank the people who helped you. Make the changes necessary to move towards "boring", routine, software development.
Wrapping it up
If these instructions sound familiar, that's good. They should be your normal software development process, not an aberration. The only thing that should change in emergencies is your timeline, not your disciplines. Like any discipline, the hardest times are a proving ground. Decide now, before the emergency comes, how you and your team will respond. Put the right processes in place today to minimize the number of emergencies that crop up. Do your part to stamp out normalization of cowboy coding on your team. Instead praise people who stick to the process, ask for help, and do things the right way when it's hardest to do so. It will pay dividends, not only in code quality but in quality of life for everyone involved.