Apr 26, 2024 9:32:21 AM
There are multiple ways to be agile, but they all have common values and principles. However, the basic idea for agile development is to do the development incrementally in small iterations, deliver often, and get constant feedback to ensure you are developing the right things. Communication and collaboration are at the core of agile development.
We argue that most things in DevOps or DataOps are just natural evolution or a continuum from agile development.
The agile manifesto has four core statements:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
In this blog post, we’ll go through these statements and how they are reflected in DataOps.
About this DataOps blog post series
To get an overview of the DataOps methodology and its key principles, tools, and best practices, we set out to create a series of blog posts focusing on different aspects of DataOps.
We'll publish the blog series over the coming weeks. If you don't want to wait, you can also download the whole story as a whitepaper.
Individuals and interactions over processes and tools
When we state that individuals and interactions over processes and tools, we would be really careful not to take that as an instruction not to have the needed processes and tools in place. In DataOps, tools and processes are one of the main contributors to the development. However, a well-performing team only performs well with proper means of internal and external communications. Even the best processes and tools won't help you if the product does not match what was expected.
There is an extra layer of communication in data development as you have to have well-established communication channels to the technical side of the business applications, but also with the subject matter experts and other relevant stakeholders.
Well-performing DataOps team should have all the processes and tools in place. As a matter of fact, we've never seen so disciplined and process-oriented teams than well-organized and performing DataOps teams.
Working software over comprehensive documentation
Development should always focus on producing working software over comprehensive documentation, but this does not mean that the documentation is non-existent. It means that the documentation should be on the correct level.
You should remember that in a correctly organized DataOps team, there is also the 'Ops' part. This means that whoever – within the team – is responsible for the production support or operations of the data platform should be able to sort out the possible issues or at least be able to delegate the problems to the correct address. When you organize your Ops with circulating production support within the team, it will become in their best interest to develop good enough data products that maintaining and running the platform does not mean constantly extinguishing fires. It also means that the documentation has to be on the correct level, or the issues bounce back to the actual developer who built it.
We can – and should – expect a certain level of professionalism from the data developers and engineers. It doesn’t mean that the documentation has to be on a level that a random IT guy can sort out the problem based on the documentation, but a data engineer familiar with the technologies used, should be.
Customer collaboration over contract negotiation & Responding to change over following a plan
Customer collaboration over contract negotiation and responding to change over following a plan go hand in hand. The basic idea is that the team should provide the output as a service. When required, we should be able to correct the course in the right direction without heavy contract negotiations or hanging on a plan that everyone understands is obsolete. That said, we still should have a plan and contracts and not just start doing random stuff!
Achieving agile success: time, experience, and support from the organization
It takes time and experience to get an agile team to perform well. It also requires new thinking and actions from the surrounding organization. If the relevant stakeholders around your product do not understand or are not able to support agile delivery, it is hard to succeed. It is important to have the team members working full time or at least 80% for the product to avoid bottlenecks on the development.
It usually pays off to hire an agile coach for at least a couple of months to get the team on the right track. And when you do so, don't break up the team without a good reason: it always takes time to get a new team to perform well, even when the individuals are experienced. It doesn't really matter what agile methodology you choose as long as it fits your requirements.
In conclusion, agile principles, emphasize the significance of effective communication, collaboration, flexibility, and a focus on delivering working and good quality data products.
By prioritizing individuals and interactions, working software, customer collaboration, and responsiveness to change, you create a strong foundation for successful data development.
Remember, agility is not a one-time achievement but a continuous journey of improvement.
Stay committed to these principles,
adapt as needed, and nurture an agile mindset within your team and organization.