Six years of technology iteration, challenges and exploration of Alibaba's globalization and compliance

​

author ： Li Weijie ( Luo Jie )      Alibaba global business platform team

​

Global technology is rooted in global business , After five stages of evolution , It has gradually developed into a relatively independent technology system within Alibaba Group . This article will first focus on the challenges and technical practices of the infrastructure layer of globalization .

One 、 Business development history

1.1  Business background

from 1999 Alibaba company was founded in , The globalization of Alibaba Group has begun , The first business unit of the company Alibaba.com That is, global business , Follow up 09 The company was founded in 10 Anniversary AliExpress Beta Version Online , It marks the globalization trend of Alibaba TO C Time , later 16 in “ Next 20 year , Alibaba group should serve 20 Billion consumers 、 create 1 Billion jobs , help 1000 Profits of 10000 small and medium-sized enterprises ”, Companies are also acquiring Lazada（ Southeast Asia 6 countries ）、Daraz（ South Asia 5 countries ）、Trendyol（ Turkey ） And other overseas e-commerce companies , Thus, globalization is officially opened as the three strategies of Alibaba Group （ consumption 、 Cloud computing 、 Globalization ） One of the great sailing times .

2016 After Alibaba's globalization related business acquisitions were completed in , Alibaba Group's global business layout has initially taken shape ：

​

• The above figure shows the countries currently covered by Alibaba's global business / region , You can see the key business countries / The region spans Asia 、 o 、 The three continents of the United States , Differences in business demands lead to obvious differences in technical solutions , An end-to-end technical solution cannot perfectly support all countries / region , But the hierarchical combination of differentiation / Customization has been proved feasible by practice , This is for us
  【 Standardization of the system 】
  Put forward a request ;

• The era of extensive harvesting has passed , In the era of refined operation , Dealing with user experience / Compliance regulation , Technical solution deployment closer to users , It is the foundation of local experience , This is for us
  【 Lightweight of the system 】
  Put forward a request ;

• With the deepening of the digital era , Digitization / Intellectualization is influencing and changing all aspects of human society more and more profoundly . As a global business , Whether our users come from developed countries or developing countries , Let the numbers / Intelligence helps users live a better life , It is always our goal , And this is also for us
  【 The intellectualization of the system 】
  Put forward a request .

1.2  Iterative process of technical system

In response to the above business demands , The global technology system is officially incubated from the group technology system , After five stages of evolution, it gradually developed into a relatively independent technology system within Alibaba Group .

• Stage 1 , Based on domestic Taobao 、 Tmall 、 Search and push and other team systems , stay 6 A full set of support has been set up in six months Lazada New e-commerce kernel system based on .

• Stage two , Customize the e-commerce kernel system accordingly , Set up a full set of support Daraz New e-commerce system .

• Stage three , Combine this e-commerce core with AE The system is deeply integrated , At the same time, Taobao was introduced 、 Excellent system solutions from tmall and other teams , It can support local + The internationalization of cross-border transaction mode is the embryonic form of Zhongtai .

• Stage four , Based on the above fusion version , Merge Lazada、Daraz、 Tmall Taobao overseas , Complete the internationalization of the middle Technology Branch 4 close 1 action , Finally formed the present 1 A middle platform support N A new global architecture for sites .

• Stage five , The open source strategy of International China and Taiwan has begun to land , after 1 More than years to 2021 year 11 In June, the open source of the full link between China and Taiwan was completed , The global business and the respective closed-loop iteration situation of the middle office have formed .

• Stage six , The future has to , Coming soon .

Next , We will use a series of articles , Explain clearly to you the challenges and responses of the global technology system . In this paper , We will first share with you the challenges and technical practices of globalization infrastructure .

Two 、 Challenges at the infrastructure level of globalization

From two aspects of e-commerce website services, buyers, sellers, customers and website management , In addition to meeting the performance of users accessing websites around the world 、 Availability and other basic requirements , In the context of globalization, global deployment has also been added 、 Legal compliance 、 Data isolation and other requirements , These requirements make our infrastructure construction meet new challenges , Here are some examples ：

• Global deployment ： Whether considering the user experience , Or consider regulatory compliance , The global deployment of infrastructure is the basic capacity that global businesses must build , The infrastructure deployed globally also directly determines many specific architectural forms of the global technology system , At the same time, the construction and maintenance of the globally deployed infrastructure itself is also a huge challenge .

• performance ： Here, the performance refers to the delay of user request processing , user
  The shorter the delay from initiating a request to receiving a response , Represents better performance
  . And global Internet services have natural challenges in terms of delay , That is, the physical distance is longer , The computer room may be in the United States , Users may be in Australia . Our test data shows that American users request the general network of American Internet services RTT yes 10ms within , And the Russian user requested the western US computer room RTT stay 150ms To 300ms Not between , This directly leads to the user's full screen load time will be more 1 Second , and 1 Seconds will cause the conversion rate to decrease , Even the loss of users .

• Usability ： There are also cost challenges in serving global users , This challenge will also bring challenges in system availability . If availability is guaranteed only from a local perspective , Then we need to build dual computer rooms in each local area to ensure high availability , But in this way, the idle resources of computer rooms in other areas cannot be used , The overall cost will also be very high . And we 7*24 Hour availability requires a global perspective , therefore , If we can achieve disaster recovery in different places around the world , It can be within the acceptable range of cost , Better take into account the user's availability .

• Data consistency ： The challenge of data consistency refers to when data is shared by many users around the world and many users will read and write , How to ensure data consistency ？ give an example ： The scene of global buying and global selling , Buyers create orders in local data centers , Sellers maintain orders in their local data center , If it is the same order and the buyer and seller are in different data centers , How to ensure consistency in reading and writing ？ When the global data centers have disaster recovery for each other , There will also be more reading and writing , How to ensure data consistency .

In addition, in recent years, with the upgrading of globalization deployment architecture , The stock computer room is gradually migrated to the cloud computer room , The expansion of new businesses and the deployment architecture of compliance are based on cloud , Global businesses have been running on the cloud , At the same time, the cloud also provides richer 、 flexible 、 Unlimited infrastructure capabilities .

On top of the cloud infrastructure , We have practiced the multi-mode deployment and disaster recovery architecture applicable to overseas , Used to solve the user experience 、 Usability 、 Data consistency issues

; And by defining the compliance architecture , It fully meets the requirements of national laws for business compliance ; meanwhile , Through the concept of cloud native architecture, it defines how to use cloud products to reshape the process of software development .

The following will be combined with the challenges of globalization , Overseas deployment and disaster recovery architecture 、 Data compliance 、 Cloud native architecture illustrates the practice of globalization and compliance from three perspectives .

3、 ... and 、 Cloud based sea landing practice

3.1  Overseas deployment and disaster recovery

3.1.1  Alibaba cloud infrastructure

• IAAS layer ：
  Relying on Alibaba cloud's globally consistent infrastructure , We have built a global 6 A large area 、13 Physical machine room 、17 A logic room (AZ) The infrastructure of overseas digital commerce , While enjoying flexible resource capacity, there is no need to be in multiple countries / Regional deployment and maintenance of data centers .

• PAAS layer ：
  Relying on various middleware of Alibaba cloud / Cloud products are deployed globally , So as to solve a series of technical challenges of globalization from top to bottom .

​

3.1.2  Global deployment architecture

Based on the two business models of this to this and cross-border , We have two deployment architectures, namely, remote deployment and local deployment , At the same time, we often need to deploy Multi Country and multi site services in a regional computer room , Thus, a multi tenant architecture is derived , The following will introduce in detail how we live in different places 、 Double living in the same city 、 Practice on single area multi tenancy .

Regionalize and live more in different places

AliExpress Our core demand is the global purchase of e-commerce & Sell all over the world , In addition, we should also consider the network delay when users visit nearby 、 Disaster recovery scenarios . Under the constraints of this multi regional deployment scenario and core requirements , The general principle of regional deployment is very clear , That is different from Amazon as well as Lazada Local site mode , Data consistency must be guaranteed between different regions . For example, when buyers and sellers from different regions trade , It is necessary to ensure the consistency of shared data ; When disaster recovery occurs in other places , After the user area is migrated , It is also necessary to ensure the consistency of unified users of services in different regions .

• The network layer ：
  According to DNS Resolve to the nearest computer room IDC, Arrive at the unified access layer of the machine room .

• Access layer ：
  It is necessary to bridge a unified routing layer to rectify the strong consistency of user ownership , That is, call the routing service at the access layer , Query the user's ownership and realize cross machine room scheduling , To achieve the purpose of users jumping across machine rooms .

• Service layer ：
  For strongly consistent data , For example, payment 、 Trading, etc , The user ownership of the unified routing layer needs to be guaranteed , That is, if the unified routing layer is wrong , that MSE The layer also needs to call the service back to the computer room where the user belongs correctly across the computer room for consumption ; At the same time, for the consistency of shared data , It is necessary to expand the cross machine room service call function of central reading and writing ; In short , stay MSE The layer needs to realize the cross machine room call function according to the user ownership or the consumption of the central machine room .

• Database layer ：
  We extend its plug-in , Realize the write ban function , It is also the bottom line of user attribution errors and strong data consistency guarantee , That is, if the user's home area is inconsistent with the actual call area , We will protect it from writing , To avoid dirty data writing between different regions .

• Data synchronization layer ：
  The data between the central machine room and the regional machine room is synchronized in two directions , Ensure the data consistency of remote disaster recovery , Avoid data loss after user area change .

This pair of this city double living

differ AliExpress Global trading business ,Lazada/Daraz The business is more focused on Southeast Asia , Local trading is adopted Local to Local Pattern , Therefore, this is the city to City dual active deployment architecture . 

As the name suggests, the construction of disaster tolerance of double living in the same city is two in a city IDC Carry out disaster recovery construction , The goal is in a IDC It can quickly switch to another one after failure IDC On , Ensure business availability . Adopt dual cell deployment architecture , With the help of unitization, the self closing loop isolation of flow in the unit is realized , Database usage RDS Three node enterprise edition to ensure its high availability . Once a fault is found, disaster recovery , Can flow from the inlet 、 Unified access layer, etc. quickly switch to another IDC On , Ensure business availability .

Multi tenant Architecture

The basic feature of global business is multi region 、 multicurrency 、 Multilingual , In order to realize the refined implementation of business strategy , Based on these dimensions , Business units can be determined , In order to realize the isolation between each business unit and standardize the metadata of business region , It is necessary to put forward the concept of tenant oriented to business operation area , At the technical architecture level, it is necessary to provide a unified multi region oriented tenant architecture standard . Business volume of each business unit 、 There are certain differences in business forms , Therefore, the deployment architecture can provide two forms of physical isolation and logical isolation of tenants , Configuration isolation needs to be provided in the technical architecture 、 Data isolation 、 Flow isolation capability , We need to maintain a unified tenant model in the definition of tenants .

Establish the mapping relationship between business unit and technical architecture based on unified tenants , Thus, the development of business unit dimension can be realized by tenants 、 test 、 Deploy 、 R & D activities such as operation and maintenance , Reduce the coupling between business units in the process of R & D activities , Improve the independence of business units .

​

Based on the design concept of multi tenant architecture , The internal working principle of the operating state is divided into the following core parts ：

• 【 Flow dyeing 】 Request identification on the end , Determine what tenant traffic , And dye the flow

• 【 Precise location 】 Based on traffic coloring , And the service routing capability of the access gateway layer , Accurately locate the physical cluster where the tenant is located

• 【 Link transparent transmission 】 Within a single service instance of the cluster , It is necessary to solve the problem of transparent transmission of tenant objects , And keep up with downstream synchronization 、 Transparent transmission of tenant information during asynchronous interaction

• 【 Resource isolation 】 During the execution of internal business logic , Operation on any resource , Need to consider the isolation problem , Such as configuration , data , Flow, etc

3.1.3  Global disaster recovery solutions

• Region Level and network are not available ：
  Machine room level unavailable , The external network entrance cannot reach the physical machine room or the physical machine rooms cannot be interconnected .

• Service level unavailable ：
  Extranet / The intranet connectivity is normal , Service not available .

• The data tier is not available ：
  DB/ Cache not available .

• Network disaster tolerance ：
  The user's first hop is outside the network route （ If the cell network is abnormal, we basically have no operation space ）, In the next 2->N jump , We can build the switching capability of network operators respectively （ many CDN Manufacturers are related to each other ）, Link switching capability of machine room （Region Level tangency ）, The switching ability of the operator at the entrance of the machine room （IDC Network teams are interconnected ） And other means to try disaster recovery .

• Access layer disaster recovery ：
  When the traffic arrives at Alibaba cloud machine room , After entering the internal gateway routing layer , According to user granularity 、Api The granularity level and other dimensions are used for real-time traffic correction , Second level to take effect . When the network and gateway products are normal , Access layer disaster recovery is the daily norm to be applied 、 The disaster recovery plan with the most rehearsals .

• Disaster recovery at the service level ：
  For some strong center services , For example, inventory 、 Sales and other single area deduction Services , It also needs to build its disaster preparedness capacity .

• Data layer disaster tolerance ：
  For multi active architecture , In ensuring a single data Master On the basis of , Ensure that data will not be dirty during disaster recovery . For compliance scenarios , Consider some Region No sensitive data , Realize the compliance disaster tolerance capability under limited scenarios .

3.2  Global data compliance practice

3.2.1  Introduction to global compliance

​

For Internet e-commerce platforms , The overall risk compliance area is very broad , Differences in risk and compliance areas , Their respective contents are roughly shown in the above figure . Compliance generally involves the following ： Data compliance 、 Intellectual property infringement 、 Product content security 、 Interactive content security 、 Technology export compliance 、APP Compliance, etc , These contents are also the focus of current supervision , In addition to data compliance , Other compliance issues mainly focus on individual business scenarios , For example, intellectual property infringement 、 Commodity content security mainly exists in the commodity domain , Interactive content security mainly lies in the communication between buyers and sellers 、 Live broadcast and other scenes .

The focus of compliance work is data compliance , Almost run through all scenarios of e-commerce platforms , Any problem involving data processing , Can be related to data compliance , At the same time, data compliance due to its regulatory sensitivity , For the platform, it belongs to business fuse risk .

3.2.2  Data compliance requirements and deployment architecture

Close the scope according to personal data , Cross border business is generally divided into regionalized architecture schemes 、 There are three kinds of privacy data closure schemes and personal data closure schemes . The company adopts the independent unit closure scheme for its business .

​

3.2.3  Local storage solutions

Data compliance often faces a direct regulatory demand ： Data local storage （ Departure is not allowed  or  Keep it locally for future reference ）. Even some sensitive businesses have higher regulatory requirements , There will be situations where the public cloud is not allowed to be used or the public cloud resources are highly secure and independent . To this end, we need to have the ability to build a complete infrastructure to meet the needs of compliance station construction .

​

3.3  Application architecture cloud prototype

Cloud native technology is good for organizations in the public cloud 、 In new dynamic environments such as private and hybrid clouds , Build and run scalable applications . Representative technologies of cloud Nativity include containers 、 Service Grid 、 Microservices 、 Immutable infrastructure and declarative API etc. . These technologies can build good fault tolerance 、 A loosely coupled system that is easy to manage and easy to observe . Combined with reliable automation means , Cloud native technology makes it easy for engineers to make frequent and predictable major changes to the system .

For global technology research and development , In addition to running the business on the cloud , We also need to further start from the challenges and pain points of our own business research and development , Combine cloud native technology and related architecture concepts to solve business research and development 、 The efficiency of operation and maintenance .

3.1  Challenges faced by traditional application architecture

chart   Traditional application architecture mode

The figure above describes the software delivery process under the traditional application architecture . From the whole process , Applications play the role of research and development objects 、 Carrier of delivery 、 Containers in operation , All the capabilities expected by software capabilities are through declarative references in the application source code , And complete the overall delivery of software through unified construction , This process can be called

Rich application of software （Fat Application） deliver

.

Because the globalization platform originally evolved from the actual business system （ transaction 、 marketing 、 payment .....）, So the platform is no exception , It continues the traditional application architecture mode . But as the platform itself evolves , And the development of business diversity on the platform , Both from the organizational structure and business structure have brought great impact , The main challenges are as follows ：

• The application architecture is not sustainable ：
  In the rich application delivery mode , In the process of software production , There is always a single point —— application , When the content supported by the application becomes larger and more complex , Then it will be the key point affecting the efficiency of R & D , It is also the biggest challenge that affects the sustainability of the entire international platform architecture .

• R & D delivery uncertainty ：
  The global platform and business layered R & D model are inconsistent in purpose and change rhythm . To solve the difference between the two , It will lead to the application itself becoming bloated and corroded , So it brings great uncertainty and unpredictability to the daily R & D iteration .

• Lack of standards for operation and maintenance capacity ：
  As the complexity of the application itself increases , The matching operation and maintenance capacity will also increase , And currently advocated DevOps idea , Also derived a lot of related products and tools , But the standards of these products and tools are not unified , And then cause scattered and miscellaneous 、 There is no unified product import , As a result, the operation and maintenance efficiency and understanding cost continue to increase .

​

For the challenge above , Cloud native technology provides us with new ideas for solving problems ：

• Container arrangement technology ：
  Through the cloud native container orchestration technology, the traditional software delivery process is evolved into the combined delivery of various container orchestrations , Split a single application delivery into multiple modules, and flexibly arrange and deliver , So as to promote the evolution of global application delivery system .

• The deliverables are mirrored ：
  Applications are no longer the only object of research and development , But to build a mirror R & D system , Ensure the certainty of the delivered content based on the invariance of the image , And realize the mirroring of platform capabilities , Have an independent and stable R & D system .

• Unified operation and maintenance standards ：
  With the help of clouds IaC/OAM etc. GitOps idea , Converge and define the application operation and maintenance standards under the cloud through a unified model . And redefine the business organization SRE, Query through a unified perspective 、 analysis 、 Measure the current situation of application operation and maintenance capabilities and resource usage .

3.2  Global cloud native architecture practice

3.2.1  Cloud based native application architecture

Combined with the cloud native problem-solving ideas mentioned above , We abstract the overall global R & D delivery process , It is used to support the upgrading of global application architecture in a broader sense . In the process , We also fully integrate the advanced technologies in cloud primitives , And applied to the scene of globalization ：

• IaC：
  Provide a unified R & D infrastructure statement paradigm . In order to better decouple the platform from business dependencies , Reduce the cognitive cost of the platform , What we apply to the site IaC The hierarchical abstract standard is defined , Define infrastructure standards around globalization scenarios , From specifications 、 Log collection 、 probe 、hook、 Publish strategies, etc. for unified convergence , Reduce service access IaC cost .

• OAM：
  Provide the definition of unified application model . Depending on OAM Separation of development and O & M concerns 、 Platform independent and highly scalable 、 Modular application deployment and operation and maintenance , We define business and platform application-oriented Standards , So as to better link application developers 、 Operations staff 、 Application infrastructure , Make the delivery and management process of cloud native applications more consistent .

• GitOps：
  Provide business R & D continuous delivery capability . Cloud based GitOps Declarative ideas , External dependent components can be integrated from capabilities to operation and maintenance control and uniformly declared in the project , And then only need to be based on unity GitOps The standard carries out the declaration and definition of dependent capabilities , Thus, the delivery and control of component capabilities are handed over to the underlying GitOps engine , Improve the integrity and sustainability of the entire software system .

• ACK：
  Provide a unified resource scheduling engine . We are based on Alibaba cloud ACK Container services , Use the powerful container orchestration it provides 、 Resource scheduling 、 And automatic operation and maintenance , Realize the delivery of different business module functions for different environments , And traffic scheduling based on the upper layer , Realize business on-demand deployment , Dispatch on demand .

• Container arrangement ：
  adopt ACK Container flexible orchestration technology has successfully upgraded the global application architecture , Integrate business logic and infrastructure 、 Platform capabilities 、 Public rich clients are completely isolated in the R & D state , In the running state, business processes and operation and maintenance processes are relatively completely isolated through lightweight containers , Improve the overall application R & D delivery efficiency and the stability of business form .

​ chart   Integration of application architecture in container state

The practice of container arrangement is emphasized here . In the process of upgrading the global application architecture, three types of containers have been derived , As shown in the figure above ：

• Infrastructure container （Base Container）, It includes the operation and maintenance container 、 The capabilities of the infrastructure on which applications such as gateway containers depend ;

• Temporary container （Temporary Container）, The container does not have any lifecycle , Its function is to pass its own R & D products Pod The shared directory under is integrated into the main application container and business container , Complete the integration of the whole capability and be used , It is mainly composed of platform containers ;

• Business container （Business Container）, This container is the same as the main application container , Have a complete life cycle , And through gRPC Complete the communication with the main application , It mainly consists of categories 、 Multilingual and other rich client containers .

3.2.2  Based on the original operation and maintenance system of cloud

​ chart   Operation and maintenance system in global application architecture

Combined with the upgrading of application architecture , Globalization has also upgraded the operation and maintenance system of applications . With the help of cloud native architecture and IaC Declarative references to standards ,  Globalization unifies the use of various application operation and maintenance capabilities , And through the strong ability of infrastructure, the efficiency is improved ,  Including but not limited to ：

• Application of release :  Intelligent release decision 、 Upgrade in place 、 Rolling upgrade 、 Release in batches

• Elastic capacity :  Auto elastic 、 Timing flexibility 、CPUShare

• Mass operation and maintenance ： Restart the application container in place 、 Container replacement 、 Log cleaning 、JavaDump

• Lightweight containers :  The operation and maintenance container is independent 、Sidecar layout

• Multi container delivery deployment :  Port conflict 、 Process conflict 、 File directory sharing

• Observability and stability :  Application life cycle 、 Start abnormal diagnosis 、 White screen 、 Container perspective monitoring

​ chart   Cloud resources BaaS turn

In this operation and maintenance system , Globalization has also introduced cloud native BaaS Ability .BaaS Provide a complete set of end state oriented 、 Separation of concerns (Application、Infrastructure) Solutions for , Through the production of Alibaba cloud and middleware resources 、 Measurement and billing 、 Identity Authorization 、 The process of consumption , take IaC Provide end-to-end experience as an entry . Globalization by introducing BaaS Ability ,  Realized SRE Unified measurement management of application cloud resource usage ,  At the same time, R & D personnel can pass BaaS Realize the consistent declarative use of multiple resources ,  Greatly reduce the cost of use .

​ chart  Java Process lifecycle normalization

In order to improve the application self-healing ability in the cloud native environment ,  We are also unified Java Apply to K8s Pod Lifecycle specification in , Start the application 、 function ( Survival and readiness )、 Standardized definition at different stages such as stopping service ,  And pass IaC and SDK Open to business use , Realization Java Consistent binding of application lifecycle and container lifecycle . 

Four 、 Summary and prospect

Technology serves the business , Global technology is rooted in global business , stay “ Let there be no difficult business in the world ” Business direction , We still have a lot of things to do . Again , Although after years of construction , There are also many imperfections in the global technology system , There are also many technical challenges to overcome , We are still on the way .

​