当前位置：网站首页>Analysis on the wallet system architecture of Baidu trading platform

Analysis on the wallet system architecture of Baidu trading platform

2022-07-05 09:45:00 【Baidu geek said】

Reading guide ： Baidu APP It contains cash 、 Activities 、 Virtual and other asset information , Distributed in Baidu APP In each business line in the , It is difficult for users to return information , And users' awareness of Baidu's assets is not high . After my wallet is built , Gather Baidu APP All user asset information in , It solves the problem that it is difficult for users to return visits , Establish user Baidu APP Asset cognition . This article mainly introduces the wallet from 0 To 1 The process of building 、 Various problems encountered and corresponding solutions , It's about throwing a brick at a jade , Hope to bring thinking and help to readers .
The full text 6082 word , Estimated reading time 16 minute

One 、 background

Baidu APP Enjoy daily life 2 Billion +、 Monthly living 6 Billion + user , At Baidu APP Many users' asset information is generated at every moment in . At present , Baidu APP It contains cash assets 、 Active assets 、 Virtual assets and other asset information , Distributed in Baidu APP In each business line in the . Each business line provides users with independent asset information , Low correlation between business lines , It is difficult for users to return information , It is not conducive to the formation of users' understanding of Baidu APP Overall recognition of asset information . A system is urgently needed , Can gather Baidu APP All user asset information in , Support unified summary of user asset information 、 Exhibition , Close the user return visit portal , Establish user Baidu APP Asset cognition , Upgrade Baidu APP User experience , My wallet came into being .

Two 、 Business Introduction

Baidu APP My wallet in the personal center , It mainly gathers the asset information of Baidu users , Carry out unified management 、 Exhibition , At the same time, view the assets of each business line 、 Use equal capacity access , It is convenient for users to return visit quickly 、 Find personal asset information . Wallet in Baidu APP Four Vajra positions and wallet entry address are exposed in the personal center , Four Vajra bits support configuration , The operation personnel can make decisions according to the activities in different periods 、 Marketing, etc. are configured with different exposed business parties , It is convenient for users to return visit quickly . At the same time, the four Vajra positions support the exposed display of business data , Enable users to intuitively perceive personal asset information .

Click on the wallet entry , You can access the wallet homepage . The cash of the main exposed users in the home page 、 Withdrawal activities 、 Return to cash 、 Card voucher 、 Virtual currency 、 Quick entry and other information , At the same time, it supports operation configuration marketing 、 Activities . For cash balance information , We need user authorization , After the user is authorized, query whether the user is in Du Xiao man 、 Balance of Baidu flash payment , Enter the details page , Can be authorized separately 、 View details . Withdrawal amount for activities , Wallet drives Baidu APP All businesses involving the withdrawal amount are connected to the wallet , Unified management , Users can view the amount details , The amount details include two parts , The first part is the business of returning user balance information , Users can click to jump to a specific business party to view details , The other part is to summarize all businesses of Baidu cash withdrawal center , Support users to click to jump , Establish unified management of users' cash balance 、 Cognition of the visit . For virtual currency data , It can be dynamically displayed according to the balance quantity of each virtual currency , It is convenient for users to quickly know their own virtual currency information , Click the specific virtual currency , You can jump to the business party details page to view details 、 Recharge and other operations , It also supports jumping to the wallet unified virtual currency details list , View user details 、 Monthly summary and other information . For business party access , The wallet provides a variety of access methods ：API Data synchronization 、 Real-time query 、 Configure jump , The business party can choose according to its actual situation .

△ chart 2-1 Wallet display

3、 ... and 、 System business architecture

△ chart 3-1 WALLET business architecture diagram

The overall system business architecture of the wallet is shown in the figure above 3-1 Shown , Wallet service is mainly for C End user 、 Service access party 、 Operating personnel . in the light of C End user , Wallet provides Baidu APP The first level entrance of the personal center and the front page of the wallet are two core entrances , Front end use talos Framework implementations . After the user reaches the logical layer through the personal Center entrance , First, determine whether the user has hit the user cache , This cache is used to identify whether the user has assets , If the cache is hit and there are assets , Then read the data cache and return the data . The user enters the wallet through the wallet homepage , According to the modules accessed , The routing adapter is configured according to the rules , Select different data rules to load data 、 obtain , After data acquisition, aggregate and return according to rules . To prevent service exceptions from affecting page rendering , Provide abnormal solution , When the business side interface is abnormal, use the cached data to cover the bottom , When an internal exception occurs, the system returns the background file .

For different business parties , We provide a variety of access methods , Exposed business at the primary entrance of a personal center , We provide push 、 Pull mechanism , That is, the business party synchronizes the wallet when the user data changes , The wallet pulls the latest user data on time ; For wallet homepage 、 List page business , We provide real-time query , For businesses that use wallets to divert traffic within the business , We support H5、scheme、talos、 Multiple jump capabilities such as in the terminal . Configuration mainly includes basic configuration 、 Extended configuration 、 Flow control configuration , Implement different adapters for upstream calls .

Operators can use B End operation tools , To configure 、 Modify the access business party information , You can also manage the wallet homepage display 、 Operating activities 、 Marketing, etc , Relevant flow control can also be configured 、 Monitoring and other information . Underlying services , We used the database commonly used in Baidu 、Redis、 Message queue 、CDN And other basic services , The basic service is operated and maintained by a unified department , The stability is reliable .

Four 、 Technical details

1、 Asset data synchronization

Wallet building , Faced with a thorny problem ： Baidu APP Asset information is distributed in different business lines , Information is isolated from each other , System heterogeneity . If you push the business side to directly expose API Query data , The business side interface is required to meet the high requirements of the wallet QPS、 Flat bottom sound 、 high stability , At the same time, there may be some special situations in the business line , Plus holidays 、 Sudden increase in traffic during special periods such as activities , It will seriously affect the wallet data display , And then affect the user experience . There are some small business users , But it must provide performance that meets the needs of the wallet , This puts additional pressure on the business side . If all business parties are required to synchronize the full amount of data to the wallet , Facing the business line data push 、 Wallet mass data storage 、 Data accuracy, reconciliation, etc .

For the above series of questions , We have defined the principle of access to the wallet ：① The business side exposed in the broken shell display space of the personal center , Asset data must be synchronized to the wallet , The wallet service internally ensures the interface availability and degradation scheme ;② The front page of the wallet displays data , Support the business party to select whether to synchronize data to the wallet , If the data is not synchronized , Need to provide satisfaction QPS、 Flat sound, etc API For wallet to call ;③ Wallet secondary page 、 Third level page data , The business party is required to provide query API Or landing page , Query the business data in real time for display , Ensure that the detailed data is accurate .

1.1 Data synchronization

For the user asset data synchronized by the business party , We only store the latest data of users , Consider the data magnitude problem , The wallet does not store historical snapshots . The wallet defines the interface specification for business party data push , The business party updates the data , It is necessary to inform the wallet to pull the latest data in time . In order to prevent the sudden increase of traffic when the business party pushes data to the wallet sever The impact as well as improve the wallet server The throughput of , Introduce message queue for cutting edge , That is, when receiving the business party's asset data change request , After being put into the message queue, the business party notification is returned , wallet server Internal consumption queue messages , Pull the latest data from the business party according to the configuration information to update . For user asset details , Wallet definition interface specification , The business party will provide the calling address to the wallet after implementation , Wallet configuration , Call in real time when the user accesses .

therefore , We designed the following data synchronization scheme ：

△ chart 4-1 Business data synchronization

① When user assets change , The business party calls the wallet interface to notify the information change ;

② When the wallet is notified , Put it in the message queue ;

③ The consumption task obtains the messages in the message queue for consumption processing ;

④ According to the configuration , Pull the latest information of the business party , Update to wallet store ;

When consuming news , wallet server Batch processing , For the same user information of the same business party within a certain period of time , Pull the data only once , Reduce service pressure on the business side . When something goes wrong , Will be carried out in 3 Retries pull . wallet server Controls whether to pull 、 Flow control , When the service of the downstream business party is abnormal , Stop pulling in time 、 Reduce the pull flow , Reduce or eliminate the impact of downstream systems on wallets .

1.2 Real-time query

Some business parties do not push data , We define the interface specification , The business party implements , After implementation, report the wallet configuration call method . For such business parties , It is not allowed to configure to display in the personal Center , Prevent the non-standard interface performance of the business party from dragging down the overall user experience . The real-time query interface is mainly divided into balance 、 Paging details interface , The wallet is unified format Show after . After the balance interface query returns , Asynchronous write Redis Cache , It is used to show the details of the next access business interface exception , If the subsequent access business interface is normal , Update with the latest data Redis data .

User detailed paging data includes paging detailed data and monthly summary data （ Monthly total revenue and total expenditure ）, The downstream of the wallet call is split into two interfaces ： Paging details interface and monthly summary interface , Provide front-end display pages to be encapsulated into an interface , Reduce front-end interaction complexity . Call the downstream paging interface to return user detailed paging data in real time , The monthly summary interface returns the summary data of the month to which the current details belong . The wallet service is summarized internally according to the interface data of the business party's detailed page , In order to reduce the number of requests to business parties , The wallet determines the paging data , If the returned data is the same month , Then only the monthly summary interface of the business party is requested , Subsequent pages are no longer requested , If the returned data is scattered in two months , Request the business party to interface twice a month , If the returned data is scattered in three months （ contain ） above , Then only the monthly data of the starting and ending months are requested , The intermediate month data is internally calculated by the wallet according to the detailed data .

2、 Multi level cache

Baidu login account system , user id It has exceeded billions , There are nearly 100 million users with assets , At the same time, the wallet will be in Baidu APP The personal center displays some businesses , I.e. primary entrance , It is expected to bring an average of 10000 QPS、 The peak value exceeds 100000 QPS Traffic , Special time points may be higher . In order to prevent service downtime on Baidu APP Produce unintended effects , Therefore, it is necessary to provide a complete cache scheme and degradation plan for the data displayed in the broken shell . To improve the high throughput of the system , We decided to cache all the level-1 entry data to Redis, Access traffic is read-only Redis, If you encounter Redis When abnormal , Return the bottom data , Don't check DB（ Prevent crushing DB）.DB Data is used to Redis Recovery in the event of an extreme crash Redis Data is used when .

Analyze the number of users with assets , There is a characteristic of users who enter the personal center ： Most users have no exposed asset data in the personal center , Abstract into a sparse matrix , This makes us design , Consider two levels of caching ： The first layer determines whether the user has asset information , The second layer cache stores user asset data . So we designed the following two-layer cache scheme .

△ chart 4-2 Sequence diagram of wallet two-layer cache scheme

Most of the user traffic will be blocked by the first layer cache , Design a reasonable first layer cache data structure , Has become a key to the system . We investigated hyperLogLog、 The bloon filter 、roaringBitmap And so on , After analysis and experiment, it is found that ,hyperLogLog in pfadd The operation itself can meet the performance requirements ,key The size of is 12k Also meet , But because of pfadd The needle itself is for a uid, It can only be operated once , So it is not suitable for this scenario hyperLogLog; The bloom filter is being tested 20 Billion data , About% of the memory is used 3G, It takes up a lot of memory , be based on redis The bloom filter has no distributed data capability , Essentially right redis Strong dependence on , Risk exists .

roaringBitmap Meet the requirements in storage space , According to the actual scene of the wallet , Improved bucket sorting and calculation rules , According to the user id Conduct sharding Points barrels , Use in barrels uid Of hash Value corresponding bitmap Site marker status . Experimental data validation ,3000 Fragmentation ,8 Units of calculation ,1000W experimental data , Storage takes up 500M+, Miscalculation rate 2.14%, There is 2.14% Users without asset information will hit the second layer cache , The overall pressure on the second layer cache is controllable .

3、 Read / write separation

The wallet service undertakes large read and write traffic , To prevent mutual influence , We split the read / write traffic into different services . An exception occurs when the user accesses the read service , It does not affect the business party's push write , When the business party pushes and writes the service exception , No effect C End user access . Reading services mainly undertake C End users through the personal center and wallet home page 、 Second level page access traffic , Cache relevant data accessed by users , The lifting system is flat . The writing service is mainly used to carry out business, pull business data and C End user authorization information , At the same time, it undertakes message queue consumption processing and interaction with downstream business parties . Between reading and writing services , adopt RPC Interface call for interaction .

△ chart 4-3 Read / write separation

4、 The data is consistent

Business data exposed in the personal center , For the business parties exposed in the access user center , We need the business party to push the wallet in time when the user's assets change , To ensure that the data the user sees in the wallet is the same as the data viewed in the portal provided by the business party . But this is not necessarily the case , For example, the business push service is abnormal 、 The network jitter , Both may lead to data inconsistency between the two parties . therefore , We propose a push-pull data consistency solution . That is, when the asset information of the business party changes , Push your wallet on time , When the user enters the personal Center , Trigger to pull the latest asset information of the user . At present , We are only responsible for the business party whose asset information has been pushed to the wallet , Pull the latest asset information of the user , Prevent pulling the full volume of business party data to bring too much invalid pull traffic to the business party with a small user level, resulting in service collapse . For the internal system , Because only Redis Cache data in , If there are some exceptions when synchronizing user information Redis And DB The data in is inconsistent , We use timed tasks , Pull every morning DB The day before （DB Change the time ） Changed user information , And Redis Compare information , When the data is inconsistent , Pull the latest data of the business party , to update Redis data , Achieve T+1 Reconciliation , Solve the problem of inconsistent data in the system .

For the business side displayed in the wallet , There will also be service exceptions 、 The problem that the latest information of users cannot be obtained due to network jitter . The plan we adopted , For normal request business party result data , Write the result information to Redis, If the next access to the business interface is abnormal , Use Redis Data as the bottom of the story , Give priority to ensuring the normal display of data . If the downstream interface of subsequent requests is normal , Then use the successful data update Redis data , Make sure Redis Quasi real time data .

△ chart 4-4 Sequence diagram of wallet data consistency scheme

5、 Configuration change

The beginning of wallet design , It is faced with how to support the business party to quickly access and support the operators to quickly configure the user display interface 、 Activities 、 Marketing and other information . Common configurational capability , It is the first choice to solve such problems .

Configuration is mainly divided into two parts ： Access configuration and display configuration . Through analysis, we found that , Access of different business parties 、 Show different needs , So we abstract the common information , Generate general basic information , For business personalized information , Use the extended model to record the special business configuration . Summarize the basic configuration of the business party + After extended configuration , Put in Redis In cache . For wallet display configuration , The underlying design shows the general configuration , For example, display name 、 copy 、 Jump Links 、 picture Url、 Background map Url, For special presentation needs , Configurable to extended information , Such as exposure Icon、 Show the amount , The configuration information is also put into Redis In cache .

At the beginning of the project , Configuration information is added and changed frequently , Risk of revision errors , Lead to C The end user experience is impaired . So we designed the version number + The white list scheme is used for large-scale control , The new version release needs to be associated with the whitelist users , After the configuration takes effect, users in the white list shall be verified online , Step by step full flow after verification . When modifying the configuration , Save the historical change full data , Used to rollback when the configuration information is abnormal .

Because the wallet is highly dependent on the configuration information of the business party , If Redis When abnormal, it will affect the stability of the wallet , So we synchronously configure the configuration information to Baidu cloud control platform GCP in , As Redis The bottom of the abnormal situation . The cache validity period can be set to be permanently valid , Synchronize updates when changing configuration Redis, You also need to synchronize updates GCP To configure , Issue in time .

6、 Database design

Due to the large number of users , The user asset information involved will be synchronously amplified at the database level , Limited by MySQL Database stand-alone processing capacity , So split the data , Different data is placed on different machines , It is convenient for the expansion of the machine . In the design of data model , The table fields of the database are user id As a sub table field （shardingKey）, This is done by the user id Navigate to specific libraries and tables , Because all tables of the entire asset information base are segmented according to uniform rules , The rules of sub table are consistent , Ensure that the same user can be in the same library , So you can use database transactions . Adopt the mode of separate database and table , When the database storage bottleneck is encountered later , It can be easily expanded horizontally .

5、 ... and 、 summary

This article focuses on Baidu APP The whole technical details of personal wallet construction , In the process of project promotion and Implementation , Also encountered many difficulties , The main difficulty lies in the high availability of the system 、 High stability 、 Fast support for service access . Sort out the difficulties and technical key points , Grasp the key issues , Subtract the system reasonably , Reduce system complexity , The rapid propulsion system goes online , Follow up iterations , Create the ultimate user experience . follow-up , In terms of business, it will continue to promote the rapid access of business lines , Allow users to view... In their wallet 、 Manage Baidu's full asset information , Continue to promote stability in technology 、 Reliability construction , Bring more user traffic to the business side , Provide users with better user experience .

——————END——————