Introduction to asynchronous task capability of function calculation - task trigger de duplication

author ： Gradually meaning

Preface

Whether in the field of big data processing , In the field of message processing , Mission systems have a critical capability - Task trigger de duplication . This ability can be used in some scenarios that require high accuracy （ For example, in the field of finance ） It's essential . As Serverless Task processing platform ,Serverless Task Such guarantees are also required , At the user application level and in its own system, the two dimensions have the exact trigger semantics of tasks . This article mainly introduces the technical details of the asynchronous task function of function calculation aiming at the topic of message processing reliability , It also shows how to use this capability provided by function calculation in practical applications to enhance the reliability of task execution .

Talking about task de duplication

When discussing asynchronous message processing systems , The basic semantics of message processing are unavoidable topics . In an asynchronous message processing system （ Mission system ） in , The simplified processing flow of a message is shown in the following figure ：

chart 1

The user issues a task - Enter the queue - The task processing unit listens and obtains messages - Scheduling to actual worker perform

In the process of task message flow , Any component （ link ） Possible downtime and other problems will lead to incorrect message delivery . The general task system will provide up to 3 Three levels of message processing semantics ：

• At-Most-Once： Ensure that the message is delivered at most once . When the network partition appears 、 When system components are down , Message loss may occur ;

• At-Least-Once： Ensure that the message is delivered at least once . The messaging link supports error retry , The message retransmission mechanism is used to ensure that the downstream must receive the upstream message , But in the case of downtime or network partition , May cause the same message to be delivered multiple times .

• Exactly-Once Mechanism can ensure that the message is sent exactly once , Accurate once does not mean that there is no retransmission in the case of downtime or network partition , Instead, retransmission does not change the state of the receiver , It is the same as the result of one transmission . In actual production , It often depends on retransmission mechanism & The receiver removes the duplicate （ idempotent ） To achieve Exactly Once.

Function calculation can provide Task distribution Of Exactly Once semantics , That is, in any case , Repeated tasks will be considered by the system as the same trigger , Then the task distribution will be carried out only once .

Binding graph 1, If you want to do a heavy task , The system must provide at least two dimensions of support ：

1. System side support ： Task scheduling system itself failover It does not affect the correctness and uniqueness of message delivery ;

2. Provide users with a mechanism , It can be combined with business scenarios , Trigger the whole business logic + Perform to heavy .

below , We will combine the simplified Serverless Task System architecture , Let's talk about how function calculation achieves the above capabilities .

Function to calculate the implementation background of asynchronous task trigger de duplication

The task system architecture of function calculation is shown in the following figure ：

chart 2

First , The user calls the function calculation API Issue a task （ step 1） Access to the system API-Server in ,API-Server After verification, the message is sent to the internal queue （ step 2.1）. An asynchronous module in the background monitors the internal queue in real time （ step 2.2）, Then call the resource management module to get the runtime resources （ step 2.2-2.3）. After getting the runtime resources , The scheduling module sends the task data to VM Level client （ step 3.1）, And the client forwards the task to the actual user running resource （ step 3.2）. In order to guarantee the two dimensions mentioned above , We need support at the following levels ：

1. System side support ： Steps in 2.1 - 3.1 in , Of any intermediate process Failover Only one step can be triggered 3.2 Implementation , That is, the user instance will be scheduled to run only once ;

2. User side application level de duplication capability ： It can support users to repeatedly execute steps 1, But it will only be triggered once step 3.2 Implementation .

The system side is upgraded gracefully & Failover When the task distribution to re guarantee

When the user's message enters the function computing system （ That is, complete step 2.1） after , The user's request will receive HTTP Status code 202 Of Response, Users can think that they have successfully submitted a task . Enter from the task message MQ rise , Its life cycle is determined by Scheduler maintain , therefore Scheduler Stability and MQ The stability of the system will directly affect Exactly Once The implementation of .

In most open source messaging systems （ Such as MQ、Kafka） Generally, it provides the semantics of message multi copy storage and unique consumption . Function to calculate the message queue used （ At the bottom RocketMQ） Same thing , Underlying storage 3 The replica implementation eliminates the need to focus on the stability of message storage . besides , The message queue used by the function calculation also has the following features ：

1. The uniqueness of consumption ： Each message in each queue is consumed , Will enter “ Invisible mode ”. In this mode , This message is not available to other consumers ;

2. The actual consumer of each message needs to update the invisible time of the pattern in real time ; When the consumer is finished , Delete the message to be displayed .

therefore , The whole life cycle of messages in the queue is shown in the following figure ：

chart 3

Scheduler Mainly responsible for message processing , Its task mainly consists of the following parts ：

1. Calculate the scheduling strategy of the load balancing module according to the function , Listen to the queue for which you are responsible ;

2. When a message appears in the queue , Pull the news , And maintain a state in memory ： Until the message consumption is completed （ The user instance returns the result of function execution ） front , The visible time of continuously updating messages , Make sure that the message does not appear in the queue again ;

3. When the task is completed , Show delete this message .

In the scheduling model of queues , Function calculation is used by ordinary users “ Single line ” Management mode of ; That is, all asynchronous execution requests of each user are isolated from each other by an independent queue , And by a Scheduler Fixed responsible . This load mapping relationship is managed by the load balancing service calculated by the function , As shown in the figure below （ We will introduce this part in more detail in the following articles ）：

chart 4

When Scheduler 1 In case of downtime or upgrade , A task has two execution states ：

1. If the message has not been delivered to the user's execution instance （ chart 2 Step in 3.1 ~ 3.2）, So when this one Scheduler The responsible queue is used by others Scheduler After picking up , The message will reappear after the consumption visible period , therefore Scheduler 2 The message will be retrieved again , Do the following triggering .

2. If the message has already started executing （ step 3.2）, When the news is in Scheduler 2 After it reappears in , We rely on users VM Medium Agent Conduct status management . here Scheduler 2 The corresponding Agent Send execution request ; here Agent It is found that the message already exists in memory , Then the execution request will be ignored , And inform you of the execution results through this link Scheduler 2, And then complete Failover The recovery of .

User side business level distribution de duplication is realized

The function computing system can accurately consume each message under a single point of failure , However, if the user side repeatedly triggers function execution for the same business data , Function calculation cannot identify whether different messages are logically the same task . This often happens in network partitions . In the figure 2 in , If the user calls 1 Timeout occurred , There are two possible situations at this time ：

1. The message did not reach the function computing system , The task was not submitted successfully ;

2. The message has reached the function calculation and is queued , Task submitted successfully , However, due to the timeout, the user cannot know the information about the successful submission .

In most cases, the user will retry the submission . If it's No 2 In this case , Then the same task will be submitted and executed many times . So function computation needs to provide a mechanism , Ensure the accuracy of the business in this scenario .

Function calculation provides TaskID This task concept （StatefulAsyncInvocationID）. The ID Globally unique . Each time a user submits a task, he can specify such a ID. When a request timeout occurs , Users can retry indefinitely . All repeated retries will be verified on the function calculation side . Function calculation is used internally DB Right task Meta Data storage ; When there is the same ID This request will be rejected when entering the system , And back to 400 error . At this point, the client can know the submission status of the task .

In practical use, it is indicated by Go SDK For example , You can edit the following code to trigger the task ：

import fc "github.com/aliyun/fc-go-sdk"

func SubmitJob() {
    invokeInput := fc.NewInvokeFunctionInput("ServiceName", "FunctionName")
    invokeInput = invokeInput.WithAsyncInvocation().WithStatefulAsyncInvocationID("TaskUUID")
  invokeOutput, err := fcClient.InvokeFunction(invokeInput)
    ...
}

And submitted a unique task .

summary

This paper introduces function calculation Serverless Task For the technical details of task trigger de duplication , In order to support scenarios that have strict requirements on the accuracy of task execution . In the use of Serverless Task after , You don't have to worry about the... Of any system components Failover, Each task you submit will be executed exactly once . To support the distribution and de duplication of business side semantics , You can set the global uniqueness of a task when submitting it ID, Use the capabilities provided by functional computation to help you de reprocess tasks .

Previous recommendation

Asynchronous task processing system , How to solve the problem of long and time-consuming business 、 High and difficult questions ？

---

1 minute Serverless Build real websites at top speed

Use Serverless 1 Minutes to easily build your first personal website ！

Free Quota , Pick up ！ Xiaobai can also set up a station at a high speed ： No need to consider the server and website source code , We provide you with free computing resources , Operation and maintenance management server . Complete the scene experience during the activity , I have the chance to get Tmall supermarket 10 Yuan voucher .（ Suggest pc The experience ）

Activity time ： 2022 year 6 month 20 Japan -7 month 1 Japan （ Collect during working days ）

Q & a group ： Nail search “44700570”

Experience address ： Click to read the original text or Scan QR code

Click on here , Direct experience ！