From programmers to large-scale distributed architects, where are you (2)

Write this article in order to know more about your technical ability , And share it with everyone , See where your technical level is .

Limited personal ability , Based on what I understand ： From programmers to large distributed architects , Where the hell are we .

Please point out the improper description , Brother, you can go to a higher level ！！！

I just drew it before Microservice system architecture diagram Let's explain one by one .

The last one tells of ：Java Programmers practice hard   Forging Foundation

      From programmers to large distributed architects , Where on earth are you （ One ）

This one goes on ……

1. Distributed service governance scheme

1.1. Briefly describe a few nouns

What is service governance ？

     Read it in reverse ： Governance Services . A distributed service governance scheme is a scheme that governs distributed services . The emphasis is still on “ service ”, Therefore, the premise of governance is that there are complex business services , We need a whole set of solutions to solve this problem .

What is a distributed architecture ？

     The construction of a complete set of distributed architecture is like building an automatic production workshop . Newly added “ Business system N” When deployed to this family, you will have a complete set of Ability （ Service governance capabilities 、 Automation 、 Big data capability and so on ).

Cloud and distributed architecture on application

     In recent years, I often hear or talk about “ On the cloud ” This word ,“ Application on the cloud ” and “ Distributed architecture ” At first glance, the two have nothing to do , In fact, it matters a lot . Take Alibaba cloud as an example ： You will add “ Business system N” Deploy to alicloud , What kind of service do you need? Just buy it with money ？ Or just do it “ Industry solutions ”. This third-party cloud is also called Public cloud ; Yes “ Male ” There is “ private ”, Private cloud Is the enterprise's own unique cloud ; When the two mix up, they become A hybrid cloud . Cloud can be regarded as a complete set of distributed architecture + infrastructure . How to get on the cloud , Apply for resources first （ Software resources 、 Hardware resources and network resources ）, Then deploy business services to this distributed architecture , So as to realize the unified management of all resources pluggable and services , That's why people like to do “ On the cloud ” One of the reasons .

This article is for readers to do positioning and follow-up learning with goals , Don't expand too much here .

1.2. Technical implementation plan

Two commonly used distributed service governance schemes ：RESTful and RPC Distributed service governance scheme .

The most essential difference between the two ：

    RESTful Emphasize that everything is regarded as a resource , Operate on resources ;

    RPC Is a remote procedure call , It emphasizes the interaction between client and server .

RESTful Distributed service governance technology solution

Technical implementation plan ：SpringCloud

RPC Distributed service governance technology solution

Technical implementation plan ：Dubbo + Zookeeper

Neither of the two schemes provides a complete set of service governance technology implementation , Link tracking technology also needs to be added 、 Monitoring alarm system 、 Log analysis system, etc ​.

Link tracking system

In a distributed environment, a business process involves calls between multiple services , It is difficult to locate the fault quickly through the system log in the traditional way , The fault can be located quickly through link tracking . The link tracking system has the function of fast fault location , And Visualization （ Such as performance index ）、 Dependency optimization （ Sort out service dependencies and optimize ）、 Data analysis （ Such as the user's behavior path , Many application scenarios are analyzed ）.

Monitoring alarm system

​ General monitoring includes flow monitoring 、 Abnormal monitoring 、 Resource utilization 、 Request delay, etc . When the monitored data exceeds the specified range , The alarm will be notified by email or telephone , You can also view the monitoring data on the visual platform , The person who receives the alarm notification will deal with the corresponding problems .

Log analysis system

Log analysis is the operation and maintenance engineer to solve the system failure , The main means of finding problems .

The service governance technology is contained in the red box

2.DevOps( Integration of development, operation and maintenance )

For high availability , Clustering business services is a normal thing , But every time it needs to be changed, it needs to be deployed to several different servers , Do repetitive work and are likely to make mistakes , It's a very smooth thing to complete the deployment in one day . Do you really want to have automated deployment ？ In fact, automation has long existed in many industries , Now many industries have entered the era of intelligence .

DevOps The integration of development, operation and maintenance is not to let development do operation and maintenance , Instead, the development and operation and maintenance are organically combined through some mechanisms 、 Efficient unification , To be a whole , So as to eliminate the gap between the development team and the operation and maintenance team , Effectively improve the efficiency of R & D, operation and maintenance of application services .

key technology

1. kubernetes（ Automatic operation and maintenance platform ）--  platform

2. Jenkins Pipeline（ Assembly line ）  -- Assembly line

3. docker（ Containers ）  --  The parcel

There are two main roles involved ： Developer （developer） and Operations staff （Operation）

  &  

The gap between the two is actually solved through container deployment , The operation and maintenance personnel only need to know how to deploy the container image , You don't need to know the details .

Continuous integration （CI）： From code submission by programmers to automatic package generation docker Mirror image , And deploy to the specified test environment

Continuous delivery and deployment （CD）： After passing the test , The operation and maintenance personnel will docker Deploy the image to the build environment

The whole process can be in k8s See... In the platform , It also supports the rollback operation of project deployment , More details will not be carried out one by one .

3. Big data access

I often hear big data “ Kill ”, Even if the same platform is ripe , It's annoying to cross major platforms . such as ： I searched for sports equipment on Taobao or JD , Then you will receive various sports equipment recommendations on various platforms . This kind of intelligent push is also called retarded push , such as ： I read a certain specification of a certain commodity , This specification is used to search products again , In fact, this is not the result I want . Big data is a double-edged sword , Well done , Maybe mentally retarded , Even violate the provisions of the law .

Point by point summary

• Big data platform ：Hadoop

• Data sources ： database , Cache middleware , File store , Data warehouse and so on .

• Data collection mode ： Real time acquisition and Offline collection

• Data acquisition technology ：kafka,flume,sqoop wait , Different data sources use different collection middleware technologies

• Data storage technology ：Hbase,HDFS wait

• Data processing ： real-time processing and Offline batch processing , Corresponding to different technology stacks

• Data services ： Provide services and realize value , Such as ： Precision marketing , Browse , Analysis and Retrieval , Download, etc.

• Platform management technology ：Apache Ambari（ supply 、 management 、 Monitoring etc. ） and Zookeeper（ Platform configuration and scheduling ）

4. other

4.1. Intranet DNS

advantage ： Facilitate memory and service migration IP Replacement only requires modification DNS that will do ;

shortcoming ：DNS Services can become performance bottlenecks and affect availability .

4.2. Network area and access control

Business services are deployed in the Intranet environment , The main reasons are safety problems and IP The question of resources .

Intranet environment It can also be split into multiple areas ：DMZ District 、 Business area .DMZ It's called the isolation zone , There are firewalls for external and internal isolation , An area that separates the external network from the internal network . It is often used to deploy system services connected by external network . Intranet business area , As the name suggests, it is to deploy business-related services .

Unified network access ： The service is deployed in the Intranet environment , Every system providing external services needs to have an entrance , For the sake of network security and better management, there is a unified network access , The system that needs to provide Internet services can apply for network access resources .

Unified outgoing network ： When the intranet system needs to access Internet resources , Control through unified outgoing network .

4.3. Safety technology

Network security

• Intranet deployment , Intranet, VPC

• Unified access network

• Application firewall WAF： From all kinds of common attacks （XSS attack 、 Injection attack 、DDOS Attack, etc ）

• Use HTTPS

• The operation and maintenance personnel must access the intranet server through the intranet fortress machine

wait

Coding security

• browser Cookie  Encryption and expiration

• Inter service communication encryption

• Log in various verification methods such as explosion-proof image verification code （ Such as ： One time encryption of email or SMS verification code ）

• Data anti crawler robot verification

• Important information data desensitization （ Such as ： cell-phone number , full name , Id no. , Card number, etc ）

wait

Server security

• Confirm the database by penetration test 、 Application and other security , Common technology ：nmap、burp suite、sqlmap etc. .

• Security of container deployment

• Common middleware security

And many other security technologies , Security issues should be considered from the code to the whole platform .

4.6. Other nagging

As the business grows , There will continue to be technical problems , The business problem , Development issues , Operational problems , Business response speed and so on . Avoid making wheels repeatedly , Need to pull away from common capabilities ; Different businesses continue to develop and form different software platforms , Function boundary division and data island between platforms , We need to introduce domain driven design idea and middle platform idea to solve the corresponding problems , But at the beginning of the project, object-oriented software design method is still recommended , Don't look around and say DDD Just join the fun , Domain Driven Design in 2014 It's not new that books have been published for years , Over design is better than hierarchy More conducive to development and maintenance .

​

Read the previous article ： From programmers to large distributed architects , Where on earth are you （ One ） 

Reprint indicates the source ！！！