Chaosblade: introduction to chaos Engineering (I)

brief introduction

With the popularity of microservices and container technology , With the help of Kubernetes
The container choreography ability of , The difficulty of deploying a distributed system is getting lower and lower . But what follows is an increasingly complex system , And increasingly difficult system reliability testing , Sometimes it's just an interface failure , It may lead to the avalanche of the whole system . In the avalanche , It is also very difficult to find the interface that initially failed , Because mistakes are reported everywhere .

To solve these problems , In addition to continuously reducing the coupling of services , Establish a powerful monitoring system and set the fuse 、 Strategies such as current limiting , Then chaos engineering appeared

What is chaos Engineering

According to the principle of chaos Engineering （PRINCIPLES OF CHAOS ENGINEERING）[1]： Chaos engineering is the subject of experiments on distributed systems ,
The purpose is to build the ability and confidence of the system to resist out of control conditions in the production environment .

The development of large-scale distributed software systems is changing software engineering . As an industry , We soon adopted the practice of improving development flexibility and deployment speed . But with these advantages , An urgent problem has arisen , That's it ： How much confidence do we have in complex systems put into production ？

Even if all individual services in the distributed system are running normally , But the interaction between these services can also lead to unpredictable results . These unpredictable results ,
It is compounded by rare and destructive events that affect the production environment , These distributed systems are inherently chaotic .

Before abnormal behavior occurs , Identify these weaknesses throughout the system . These weaknesses include the following forms :
Incorrect rollback settings when the service is unavailable Retry storm caused by improper timeout setting Service interruption due to downstream dependent traffic overload Cascading failure at single point of failure, etc
Before these weaknesses are exposed to users through the production environment , We must actively find these important weaknesses . And a method is needed to manage the inherent chaos of these systems ,
Increase our confidence in the deployment of production environments by increasing flexibility and speed , Although the complexity of the system is caused by these deployments .

Based on experience and systematic methods, the problems caused by the scale growth of distributed systems are solved ,
And build the ability and confidence of the system to resist these events . Understand the characteristics of distributed systems by observing their behavior in controlled experiments , We call it chaos engineering .

ChaosBlade[2] The blade of chaos , Alibaba is an open source experiment injection tool that follows the principle of chaotic engineering and chaotic experimental model , Is an internal project
MonkeyKing
Open source projects , It is based on Alibaba's fault test and drill practice in recent ten years , Combining the best ideas and practices of the group's businesses . It aims to help enterprises improve the fault tolerance of distributed systems , And provide business continuity guarantee in the process of enterprise going to the cloud or migrating to the cloud native system .

Currently supported scenarios are ： Basic resources 、Java application 、C++ application 、Docker Containers and Kubernetes
platform . The project encapsulates the scenario into a separate project according to the domain implementation , It can not only standardize the implementation of scenarios in the field , And it is very convenient to expand the scene horizontally and vertically , By following the chaotic experimental model , Realization chaosblade cli Unified call .

The project experience is excellent , Only one executable is needed blade, You can complete experiments in various scenes （ Some complex scenes such as Java
Applied experiments , You need relevant tool kits ）, At the same time, if you don't want to download chaosblade tool kit , Quick experience chaosblade, You can pull docker
Mirror and run , Experience in the container .

Project on 2020 year 5 month 27 The latest was released on the th v0.6.0[3] edition , The whole practice of this series of articles will also be based on this version and the repaired version of this version
v0.6.x Conduct .

ChaosBlade-Operator

ChaosBlade-Operator yes ChaosBlade Of Kubernetes
Platform experiment scenario implementation , It is also the main tool used in this series of articles . It passes the chaos experiment Kubernetes The standard CRD Mode definition , Easy to use
Kubernetes Resource operations to create 、 to update 、 Delete the experiment scenario , Including the use of kubectl、client-go
And so on , And you can also use the chaosblade cli Tool execution .