Turn the evolutionary path of push

One 、 brief introduction

from 0 Let's talk about it push The iterative process of the system , And the solution of common problems encountered .

seeing the name of a thing one thinks of its function ,push It is a way to send messages to users with the help of manufacturer channels , Generally used for user recall and activity touch , and IM（ Instant messaging ） There are still some differences , Don't go over here .

1.1  A term is used to explain

Business attributes ：  operating 、 Business 、 Function analogy send

Push range ：  Monthly living 、 Total quantity 、 Directional push 、 Personalized push

Target end ： Usually Android 、ios client

passageway ： millet 、 Huawei 、 meizu 、apns Wait for the permanent connection of the mobile phone manufacturer

token：  Unique identification for equipment , from APP It generates itself

devicetoken： Unique identifier for push , Generally provided by the manufacturer

Push volume ： Number of push messages

Arrival rate ： The number of messages reaching mobile phones / Push volume

Click through rate ： User click rate / Push volume

1.2  Existing architecture

• The existing architecture supports background push 、 Business push and personalized recommendation push .

Background push

：

There are usually standard formats , It is characterized by short time 、 A lot of pushing , such as 8 Click the second kill activity .

Business push

 ：

It is usually triggered by business , It is characterized by strong real-time 、 High priority , Such as order payment message .

Personalized push

：

Often related to user portraits , It is characterized by complex strategies 、 The content is diverse , Risk control management is required , I guess you like the recommendation strategy .

Two 、 The process of project iteration

2.1  origin ：PM Want to push operation activities

• step ：

• 1）PM Export a part of the user set from the big data platform

• 2）RD Write program call push Interface

• problem ：

• 1）N individual PM There is demand. ,RD..........

• 2）8 There is a sudden situation ,9 Click for a wave

• 3） Every weekend we have activities , push

• Solution ：

• 1） Build a backstage , Support according to users ID Upload , Liberate and develop resources

• 2） Support push by time , Support copywriting

• 3） Support for Android 、IOS Split push

• Remaining problems ：

• Problem description ：

• PM Upload a data set of users who have browsed the mobile phone category , Too much data , Upload timeout

• Solution ？

• PS： The number of users is about 1000w about +, about 300M Documents on the left and right ？

• Tips ：

• 1） The upload time is about 1 About minutes ,  You need to contact the operation and maintenance department to set the longest link time , otherwise nginx Will actively disconnect

• 2） Upload by synchronization , Change to the way of progress bar , Let the uploader see the progress

• 3）  Uploading and data processing are separated （ We were uploading , Side parsing files is slow ）

2.2  Major festivals , Hope to be able to notify the recent active users

2.2.1  Push in real time

• Problem description ：

• Major festivals , Push all users 、 Monthly living 、 Weekly live data , Every time it's just a different copy ,PM All need to run big data systems , Efficiency is too low , Data of the day are not available , Average push needs 1 More than a hour .

• requirement ：

• 1）1 Billion data can be pushed in an hour

• 2） To cover users in a certain period （ Like a month ）

• 3） Support preview , Support pause

• analysis - Data volume （ With 2000w Take monthly living as an example ）：

• 1）  The total number of users is recognized as nearly 3 Months （90 God ） Users who have visited the Internet

• 2）  It is estimated that the number of all equipment is 5000w about

• 3）  The estimated space occupied is 5G

• analysis - performance （ With 2000w Take monthly living as an example ）：

• 1）  Old system push The average of QPS yes 2000

• 2） 2000W/2000/60/2=83~=1 Hours 20 minute , I hope it can be in 12 Push finished in minutes （ One hour push 1 Billion target ）

• Difficulty analysis ：

• 1）  Data should be quasi real-time , How to calculate quasi real time

• 2）2000 Millions of data 12 Push finished in minutes ,QPS~=2.7w,  How to improve performance 13.5 times （2k Upgrade to 2.7w Concurrent ）

• Solution ：

• 1）  Quasi real time data

• Real time reception kafka Log message , Merge the cleaned data every minute

• 2） Data elements that need to be stored

• a)  User token Information （ Notice that it's not devicetoken）

• b)  this token Active time （ Time stamp ）

• 3) User data storage selection

Final choice

redis Of zset

For storage

4） How to improve sending performance

First, analyze the reason why it was slow before ：

1）  Single thread send

2）  Limited by the manufacturer's channel , Single interface takes time 100ms+(IOS passageway )

Solution ：

1） Distinguish Android 、IOS Send separately , The original program is only responsible for starting from redis After getting the data, assemble it into a fixed structure （ Simple splicing operation is fast ）

2） Push data to MQ in （ Don't have to MQ Do you ？）

3） Multiple consumer subscribers , Consumption ( Easy to expand ), By manufacturer   Channel push out .

Be careful ：IOS passageway , We use pushy Open source tools , Under certain circumstances, messages cannot be pushed continuously , Need to check regularly , Recreate the channel

• The final effect ：

• push Push the QPS achieve 3w+, While the push ability is improved , It also caused the following problems .

2.2.2  The business server cannot bear the instantaneous traffic

Problem description ：

When push After the push ability of ,  The problem of instantaneous access of users follows

1） Instantaneous flow peak , Lead to more timeouts

2） Some requests reach the performance bottleneck , Overtime increases , The page doesn't open ~, See the picture below

terms of settlement ：

1） The easiest way ： With the machine

2） Business interface multithreading 、 Service governance , Peak elimination （ratelimit）

3）app Core functions increase cache , Ensure that there will be no white screen

4） Reduce active paths .（ commonly push Will land on an activity page . But normally open push, Will first enter the home page , Then jump to the activity page . to push The message adds a special burial point , If this is the case push news , Just jump to a specific page , Reduce intermediate links .）

2.3 AB laboratory

• Problem description ：

• One night 9 Click to push an operation class push, I found that the click through rate is super high , Excellent copywriting ？ Or peak traffic ？

• requirement ：

• 1） There are multiple push files , The system can select the best click through rate to push ？

• Solution ：

• 1）  Join in AB Measured ability , First push a few users , according to AB The effect of , Preferential push

2.4  Multi channel approach

• The new problem ：

• Before Android channel, we only had Xiaomi channel + A push （ The reach rate of a push is average , Make a base ）,  If we push messages to China for mobile phones , It is also difficult to reach through the Xiaomi channel .

• requirement ：

• 1） I hope we can connect all the channels of major manufacturers

• 2） The pushed message can be optimized according to the channel in which the user logs in last

• 3） Don't slow down

• Solution ：

• 1）  build tokens service , can
  Batch
  determine devicetoken The last manufacturer to use （ You need to rely on the forwarding client to report ）

• 2）  Store by database and table

• 3）  Data hot standby to cache

• effect ：

• 1）  The statistics of that year can be improved 10% The achievement rate of

• 2.5  Real-time monitoring

• General monitoring dimensions include ：

• product line ： around 、 Find a beautiful machine and so on

• client ： Android 、IOS

• indicators ：  send out 、 arrive 、 Number and proportion of clicks

• Data comparison ：  Templates 、 cycle

• passageway ：  millet 、 Huawei 、vivo、apns

• 

3、 ... and 、 summary

present situation ：

1）  Push monthly living 10 minute

2）  Support pause 、 preview , View the amount of push data in real time

3）  Support in advance AB See the effect

4）  Support not online , Wechat notification

5）  Support anti disturbance

6）  Support priority and manufacturer high-quality channels

Increase speed ：

Preloading + cache + Multithreading + Reasonable data structure + Batch processing + Rational layout + Special burial point

A compromise ：

Asynchronous upload 、 Current limiting control 、 Degraded processing 、 Layered decoupling 、 Notice of compensation

Author's brief introduction

Wang Jikuan , Transfer platform business leader .

Turn to the technology learning and exchange platform of R & D center and industry partners , Regularly share front-line practical experience and cutting-edge technical topics in the industry .

Official account 「 Transfer technology 」（ Comprehensive ）、「 Big turn FE」（ Focus on FE）、「 around QA」（ Focus on QA）, More dry goods practices , Welcome to share ~