Should chaos engineering practice be made part of Devops CI/CD pipeline!!!

Answering this question with just Yes or No would be tough.  At first glance it looks like straight forward and possible. There are tools like Gremlin, Litmus or Chaos Monkey to simulate different types of failures, and then integrate these tools into your pipeline so that they are automatically run as part of your testing and deployment process. However, Its not as simple. Let us deep dive to see what are the challenges for including these fault injection experimentations into CD pipeline.

We need to start with basics of Chaos Engineering and the purpose of Devops to arrive at the detailed answer. We must start with the official definition:  

Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.

Next we need to look at the principles. Principles of chaos engineering is well defined to help users understand the practice in detail. it includes:

Good understanding of the system

Including stakeholders

Formulating hypothesis

Preparing experiments

Planning for game day

Running experiments

Monitoring

Collecting metrics

Increasing blast radius 

Automating the fault injections using Devops pipeline would miss certain aspects of principles stated above. Monitoring the system behaviour on a scheduled day to run these experiments together with all the stakeholders is critical to the success of this practice. The learning that comes from live monitoring and analysis along with other stakeholders provides better perspective on the issues that are uncovered. It also helps in detecting the weak points in the system. 

Simple fault injections whose results are easily guessable without the need of monitoring may be considered as better candidate for CD pipeline. But there are not many such simple faults. Even simplest faults injected can manifest beyond imagination to become complex issue within the ecosystem. Monitoring system at every layer with every possible means is important while flats are being injected. Today’s distributed systems are complex with multiple layering and interdependent modules. Same yard scale cannot be used in every stage. 

Game day planning includes activities that cannot be automated easily. The fire drill concept, team gathering, and war room setup are some of the activities that must be experienced by every chaos engineering practitioner. These rehearsals build confidence to deal with the real life issues. Loosely based name sake automation could fail the preparedness objective. The real life chaotic surprises requires greater people collaboration skills to spring back at the earliest proving resiliency. Mere dependency on the rigid CD process without the possibility of eternal evolving would not yield better results. 

After every iteration of the fault injection exercise, teams must plan for increasing blast radius and learn more. Automation with same input criteria/parameters for every run would fail the objective of continuous improvement. Adding variable time in increasing manner is not tough automating. However, that requires complex logic inclusion and becomes unsustainable sooner than you think. Devops CD pipelines are not designed for handling such highly variable, time consuming and human intervention demanding process. 

Time is crucial parameter and with chaos engineering practice, every minute variation could increase possibilities of uncovering the new issue. The main motto of automation is to save on time. However, chaos engineering requires uses time as varying parameter to alter the characteristics of the fault injection and subject the system to fail using time as catalyst. Also, running time consuming process in Devops CD process is not recommended. This slows down the deployment process and fails the modularity aspect with increasing dependency. 

Conclusion

The above details with definition and principle establishes that this practice is nothing but detailed experimentation and not just mere testing that can be simply automated. Chaos engineering is about uncovering the inherent chaos in the system using unique approaches in every trial. Without the variable parameters, human collaboration, and real time monitoring, the process may not serve the real purpose of chaos engineering. 

Chaos Engineering Clarity

My recent conversations with technical community made me feel that Chaos Engineering(CE) concepts are being misunderstood. This being emerging practice in the industry, very soon deeper understanding would eventually clear all the confusions. There are comprehensive documentation written about this practice already defining its objectives, principles, and implementation. These documentation are widely available through various open source and commercial CE tools available in market. 

Technical folks new to this practice are having doubts and confusions on this practice. I believe further clarities need to be provided on the CE practice widely. I have gathered many questions through my interaction with technical teams. I'm trying to address some of them in this blog post.

Before I begin, you should read through the chaos definition. As per wikipedia:

Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production

Is chaos engineering practice against compliance and regulation? 

No! In fact, the CE experiments help in meeting security and compliance standards by uncovering the hidden issues. Compliances are setup generally for ensuring data privacy, data protection, security, country/regions specific laws and following standard processes. 

The objective of chaos engineering is to have deeper understanding of IT systems and eliminate or minimize the application/service outage. The tests by injecting faults are supposed to be done in a controlled manner where rollback plan is also part of it. 

As far as the tools usage, just like any third party tools that are used in the applications, CE tools also required to be complaint with security and licensing standards of an organization. The scans that enterprises mandate usually checks for all the CVEs in third party tools.

Is chaos engineering replacement for security related tests like static and dynamic/penetration test practice?

The general CE practice is not meant to replace the security tests like static, dynamic or interactive application security tests (SAST, DAST or IAST). Both security testing and CE practice includes injecting fault into the system. The method of analysing the system by trying to break is same in both types of practices but objectives differ. 

I see attempt being made to extend CE practice to cover certain types of security testing. There is even a new name Security Chaos Engineering coined for this purpose.  They use the same fault injection approaches to inject security flaws into the system. However, one should not confuse general CE practice with security testing. Their objectives are different even if the same tool is used for both types of practices. 

Is chaos engineering suitable only for cloud infrastructure?

Any type of infrastructure that hosts your applications can be considered. Whether it is cloud, containerized, virtual machines or physical servers infrastructure. The widely available examples in the documentation of CE provides more examples on  cloud infrastructure. That doesn't mean it is only applicable for cloud infrastructure. 

Can it be made part of automation through CI pipelines?

Automation of injecting faults at various layers like application, network, and computing resources is quite possible. However, CE process require observability through instrumentation which is better if carried out manually together with all stakeholders. Automating the systems breaking process could fail the objective of deeper understanding on the system and its behaviour when faults are deliberately induced. 

The crucial phase of CE practice is to have a game plan day (mock drill) where faults are injected and systems behavior is captured. This would help the teams to be prepared when such real time situation arises in future. 

How is chaos engineering different from load testing or performance testing?

Load testing objective is to determine the system performance under load and benchmark it. This helps teams in understanding the system capacity and design the system accordingly. The scalability requirement of the system is identified and fulfilled with the help of performance testing. 

This is entirely different from the objective and approaches of CE practice. However, CE practice can include certain load testing tools to further test the behavior of the system under the load while fault is injected. This apart, there is no other connection between CE practice and performance testing. They remain different in its objectives and approaches. 

Is it necessary to use chaos engineering tool?

Tools help you to get started quickly on achieving the CE objectives. There are various open source and commercial tools available in the market. These tools provide make readymade fault injections available to you with easy setup. Most of the tools comes packaged with resource exhaustion tests (RAM, Disk, I/O), network test(Latency), Infrastructural(pods, containers, VM, server) or application level tests. 

Writing these tests from scratch requires lot of time and effort. I would recommend to use one of such tools if you are new to CE practice. Once you are proficient, you can start building your own tool. 

Hope this post helps you. Thanks for reading and look forward to your feedback.

References

http://principlesofchaos.org/

https://arxiv.org/abs/2006.04444

https://www.ibm.com/cloud/architecture/architecture/practices/chaos-engineering-principles/