DataOps: keeping the data flowing with Model-driven Operations

DataOps: Prologue – why is it so hard?

If you’ve ever lived DataOps, you’ll know that it’s a challenge at the best of times. A day in the life of a typical data engineering team involves securing, releasing, debugging and stabilising complex and oftentimes fragile data pipelines. These pipelines can involve many source applications and intermediaries, and troubleshooting them under management pressure when it’s all going wrong is stressful.

There are so many complex layers to modern data platforms that need to be cared for that the team can often feel overwhelmed. Sometimes it feels like it isn’t unusual for 80% of the sprint to be focused on low-level plumbing and fixing technical debt.

Nobody likes technical debt and with less than 20% of the team’s work time available for actual product development, the DataOps team is often on the receiving end of management frustration. We’ve all been in That review call—you know the one I mean: why are things taking so long? Why is there so little to show? Why is your cadence so poor?

I want you to step out of that difficult and uncomfortable conference call, just for a moment. Stop trying to defend and justify yourselves; instead, reflect.

Exegesis: doing less with more

If even 20% of the team’s distraction is removed, the team’s productivity, in theory at least, doubles. And so I introduce you to Model-driven Operations. Model-driven Operations models the relationships between components, their interactions and interdependencies. This paradigm has slowly been gathering momentum in the Kubernetes community. Some of you may already be using, or thinking of using, Kubernetes operators to help automate deployment of trade tools like Apache Spark and Flink.

Model-driven Operations, however, is a compute paradigm that goes much further than point deployment automation of an individual software component. With model-driven operations we can model the complete solution. 

This means going beyond deployment. It also means going beyond individual components. We can also model the day-0 to day-N tasks associated with the solution. These tasks go beyond deployment and integration, to daily operational management tasks such as certificate renewal, password rotation, post-mortem snapshotting, performance reporting, backup and restore, high-availability failover, disaster recovery testing and DRP execution.

This all sounds time consuming! Well, perhaps at first. But investing time in building an automation model of your team’s operational tasks can and will yield a return on investment, improving quality, repeatability, predictability and most importantly, productivity.

Synthesis: Model-driven Operations for good

Let’s reimagine that sprint review call. Management is seldom happy—we all know that! But after investing some story points on codifying a few of the team’s operational tasks into an automation model, the team’s productivity has measurably increased. Perhaps productivity has doubled – or more! And management has voiced far less unhappiness lately.

As the DataOps team has more new projects completed and in production, management has more success stories to share. It’s starting to feel like the Sword of Damocles no longer hangs above the team. For the first time ever, the team has had the same scrum master for more than two quarters. Things are looking up!

Envision the situation: with less time spent on mundanity and extinguishing fires, the team can all spend more time in the flow state doing what they love to do best – productive, net-new engineering.

Epilogue: make the happy ending real

Let’s make it real.

• Want to make this happen? Learn more about Model-driven Operations, Juju and charmed operators.
• Contact us to find out about our managed application services and how they can help your data engineering teams to double down on productivity.
• Further reading: Data Lab Architectures