[Translation] Chaos Net + SkyWalking: Better observability for chaos engineering

作者：Wang Ningxuan2022年1月31日

特邀文章,原载于PingCAP博客,作者：Wang Ningxuan

Chaos Mesh是一个开源的云原生混沌工程平台.你可以使用Chaos Meshto easily inject faults,Simulate abnormal situations that may occur in reality,Thereby discovering potential problems in the system.Chaos MeshA Chaos Dashboard is also provided,Allows you to monitor the status of chaos experiments.然而,This dashboard does not allow you to observe how failures in the experiment affect the service performance of the application.This prevented us from further testing the system and finding potential problems.

Apache SkyWalkingis an open source application performance monitor（APM）,专门用于监控、Track and diagnose cloud native、Container-based distributed system.It collects events that happened,and then displayed on its dashboard,Lets you directly observe the type and number of events that occur in the system,and how different events affect service performance.

Use both when you are in Chaos ExperimentSkyWalking和Chaos Mesh时,You can observe how different failures affect service performance.

This tutorial will show you how to configure itSkyWalking和Chaos Mesh.You will also learn how to use both systems to monitor events,And observe in real-time how the chaos experiment affects the service performance of the application.

准备工作

在你开始使用SkyWalking和Chaos Mesh之前,你必须.

• 根据SkyWalking配置指南,建立SkyWalking集群.
• 使用Helm部署Chao Mesh.
• 安装JMeter或其他Java测试工具（to increase the service load）.
• If you just want to run a demo,根据本指南配置SkyWalking和Chaos Mesh.

现在,You are completely ready,We can cut to the chase.

第1步：访问SkyWalking集群

安装SkyWalking集群后,You can access its user interface（UI）.然而,No services are running at this time,So before starting monitoring,You have to add a service and set up a proxy.

在本教程中,我们以Spring Boot,Take a lightweight microservice framework as an example,Build a simplified presentation environment.

1. 参照此文件,在Spring Boot中创建一个SkyWalking演示.
2. 执行命令kubectl apply -f demo-deployment.yaml -n skywalkingto deploy the demo.

完成部署后,你可以在SkyWalkingThe user interface observes real-time monitoring results.

注意：Spring Boot和SkyWalkinghave the same default port number：8080.Be careful when configuring port forwarding,否则,You may have a port conflict.例如,你可以通过使用kubectl port-forward svc/spring-boot-skywalking-demo 8079:8080 -n skywalking这样的命令将Spring Boot的端口设置为8079,以避免冲突.

第2步：部署SkyWalking KubernetesEvent exporter

SkyWalking KubernetesEvent exporter能够观察、Filter and sendKubernetes事件到SkyWalking后端.然后SkyWalkingCorrelate events with system metrics,And shows an overview of when and how the metric was affected by the event.

If you want to deploy with one line commandSkyWalking Kubernetes Event Explorer,请参考该文件,以YAML格式创建配置文件,Then customize the parameters in the filter and exporter.现在,你可以使用命令kubectl apply来部署SkyWalking Kubernetes Event Explorer.

第3步：使用JMeterto increase the service load

To better observe changes in service performance,你需要增加Spring Booton the service load.在本教程中,我们使用JMeter,A widely adoptedJava测试工具,to increase the service load.

使用JMeter在localhost:8079上进行压力测试,and add five threads,Continue to increase the service load.

Apache JMeter的用户界面

打开SkyWalking仪表板.你可以看到,Access rate is100%,The service load reaches approx5300个呼叫（CPM）.

SkyWalking仪表板

第四步：Inject faults through the chaos net and observe the results

完成上述三个步骤后,You can use the Chaos Dashboard to simulate stress scenarios,Observe changes in service performance in chaos experiments.

User interface for Chaos Net

The stress under three chaotic conditions is described below,Changes in service performance.

CPU负载.10%;内存负载.128 MB

The first chaotic experiment simulated was lowCPU使用率.To show when the chaos experiment starts and ends,Click the toggle button on the right side of the dashboard.To understand whether the experiment is applied to or recovered from the system,Move the cursor over the green dash.

in the time period between the two short green lines,Service load dropped to 4929CPM,But it returned to normal after the chaotic experiment ended.

Service load changes in the first chaos condition

CPU负载.50%;内存负载.128 MB

当应用程序的CPUload increased to50%时,Service load dropped to 4,307 CPM.

The second chaotic condition的Service load changes

CUP LOAD:100%;内存负载.128 MB

当CPU使用率为100%时,The service load drops to only when there is no chaos experiment40%.

The third chaotic condition的Service load changes

摘要

由于LinuxThe process scheduling under the system does not allow a process to occupy all the timeCPU,所以即使在CPUunder extreme conditions of full load,部署的Spring Boot Demo仍然可以处理40%的访问请求.

通过结合SkyWalking和Chaos Mesh,You can clearly observe when and to what extent a chaos experiment affects application service performance.This combination of tools allows you to observe service performance under various extreme conditions,Thereby enhancing your confidence in the service.

在所有PingCAPThanks to the tireless efforts of engineers and community contributors,Chaos Mesh在2021Year has been greatly developed.In order to continue to upgrade our support for the majority of users,Learn more about user experiences in Chaos Engineering,We would like to invite you to participate 这次调查,Give us your valuable advice.如果你想了解更多关于Chaos Mesh的信息,欢迎你加入GitHub上的Chaos Mesh社区或我们的Slack讨论（#project-chaos-mesh）.如果你在使用Chaos Meshfound anybugor lack of functionality,You can ask usGitHub仓库Submit your pull request or issue.