It's not what DevOps mean!
Don't have gate keepers in you release cycle. It increases the costs and tries to sabotage the organization.
This is not going to be ordinary post. Recently I encountered below image on twitter. Oh boy, you would be surprised how many organizations have pipelines likes this. Sit, let me tell you a story.
What’s wrong with this?
So many things are awful with this approach that I don’t know where to start.
Instead to empower engineers to solve problems and deliver value to customers. People build pipeline that does more gate keeping and hurts entire organization.
Process like this is only hurting the organization. In case you have to solve some high critical bugs it requires too much coordination, small things requires multiple teams to coordinate.
Imagine if you have many teams and how release would look like in this environment. Nothing gets done and team struggles to deliver, and costs to deliver just goes up.
When you have gate keepers, it means that there is underlying issue in the application and it causes a fear of releasing application. Because of that, they put one person in charge so it can intervene immediately. Every release brings risk to their organization that it will break. That's why people use gate keeping like this. It's not correct approach, it's fear that creates this!
Why it happens?
Processes are in place, every release brings bugs and you don't have way to verify that release is success.
You don't have tests, or any other way to verify that API is working.
You don't have way to recover from failed release and entire app fails.
You don’t have way to notify people when app fails.
Instead to rollback release, your app fails.
That's why these processes exists and they hide root cause!
Somebody adds new feature, they have to wait for DevOps engineer to ship it. They can’t close their ticket and move forward, customer has to wait longer and engineer will keep switching context. Same problem is with 2 week cycles and shipping code after 2 weeks.
People get comfortable with this and don’t challenge it. But I encourage you to challenge bad processes and change them, that will bring so much value to engineers and users at the same time.
Hear me out, yes, some features have to be released at once. I’m not against it, but most value can be shipped daily.
How to do it?
Whole idea about DevOps is to reduce time to deliver value while still delivering the high quality work.
Gitlab has good definition of the DevOps, and image above is from that blog. It says:
DevOps practice enable software development (dev) and operations (ops) teams to accelerate delivery through automation, collaboration, fast feedback, and iterative improvement.
A lot of pipelines are everything else except this. You need to increase confidence of your team in regular releases. Allow them to trigger pipeline which is safe:
Build and test application
Deploy application
Run tests to verify it’s up and processing everything
If test fail, rollback
If test pass, great, notify them
Keep monitoring application with automation
It’s not easy to build proper deployment strategy. But it would remove people from the process and let automation handle everything. That way, people know when releases pass that app works since tests passed. And they can continue working on next task, they can release multiple times per day without gate keeping.
This actually greatly builds on top of how you design your applications, as we covered in earlier posts:
Observability is important because that way you intervene when things break. And you know this much faster, you don’t wait for your customers to tell you that.
Fault tolerance is important, so you can recover from errors when they happen.
Final thought
Challenge bad processes for the love of God. They are not good, and this guys that said this on reddit see this. But you would be surprised how many people don’t care to change them.
Thanks for reading once again, this was more my rant. In the future, I will definitely write one post which will give more example how pipelines can empower engineers and how to build those release strategies.