Build a reliable, scalable and maintainable application system

Current application systems , Usually, there is a large amount of data 、 The data complexity is relatively high and the data is rapidly changing , Such a system can be called a data intensive system , Data intensive systems are usually built on different module components , Different components are used according to different functions , Each component cooperates with each other to form a powerful 、 Application systems that meet different demand scenarios , For example, an Internet system usually consists of a structure similar to the following ：

database ： The main products are mysql, For storing and persisting data , One time data can be accessed many times .

• Cache ： The main products are Redis、Memcached Cache data that needs to be accessed frequently as well as complex calculations and operations , Used to speed up the return of data , Reduce database pressure , Expand system concurrency .
• Indexes ： The main products are ElasticSearch、Solar, Used to support the search function , Support users to search elements and filter data with keywords .

In addition, other modules will be involved

• Streaming ： Continuous data generation , Fast consumption data is required for data calculation .
• The batch ： Regularly calculate and process a large amount of historical data

As functional requirements become more complex , The system requirements are more and more extensive , A single component cannot meet all the data processing and storage requirements , So we need to decompose the task according to the requirements , Combine appropriate technical components to efficiently complete certain functions , Multiple components are organically combined through application layer code , Our technical requirements are also relatively high , It also brings higher technical challenges . The more complicated situation is that the number of users and business become more and more complex in the process of use , There are bound to be more difficult problems .

such as , In case of partial failure of the system , How to ensure the correctness and integrity of data ？ When the system is degraded , How to design to provide users with a consistent user experience ？ When the system load increases , How to extend the system ？ These issues need to be taken into account in an Internet distributed system .

Data intensive application systems focus on three issues that are extremely important to most software systems ：

• reliability

When something unexpected happens, like hardware 、 Software failure 、 Human error, etc , The system shall be able to operate normally , Performance may be reduced , But the function is still correct . Hardware failure , The usual approach is to add hardware redundancy to reduce the system failure rate , With the increase of data volume and application computing demand , More applications can run on large-scale machines , The resulting hardware failure rate increases linearly . So it is a new method to tolerate multi machine failure by software fault tolerance , Or at least a powerful complement to hardware fault tolerance . Software problems , Faults are more difficult to predict , Because nodes are associated by software , Therefore, it will often lead to more system failures .

• Extensibility

With the amount of data 、 Growth in scale, such as traffic or complexity , The system should be able to match this growth in a reasonable way .

• Maintainability

as time goes on , Development of new functions 、 New team members can quickly participate in system development and operation and maintenance . It mainly focuses on three design principles of software system ： Operability 、 simplicity 、 Evolvability .