What is a distributed timed task framework?

I am a 3y, A year CRUD Ten years of experience markdown The programmer ‍ It is known as a high-quality eight part essay player all the year round

It has long been planned to introduce the distributed timed task framework , Years ago austin Already connected , But the code hasn't been written for the Chinese New Year , The article has been delayed to today . Today I'm mainly talking with you Timing task This topic .

After reading this article, you will learn what is a scheduled task , And why austin The project will introduce a distributed timed task framework , You can download the code and see how I use it xxl-job Of .

01、 How to simply realize the timing function ？

I'm an introduction to watching videos Java Of , At that time Java Basics API When , The video you watch also has a timing function （JDK Native supports ）, I remember the video instructor wrote Timer To explain the timing task .

At that time, I didn't know the practical role of timed tasks , So at the beginning of my study , Never used Timer To realize the function of timing .

later , I learned Concurrent 了 . The lecturer at that time mentioned ScheduledExecutorService This interface , It is better than Timer More powerful , Usually we are in JDK It can be used to realize the function of timing

The strength lies in ScheduledExecutorService Inside is the thread pool ,Timer A single thread , It can make more rational use of resources .

When I learned concurrency , I don't pay much attention to it （ It's not the focus of concurrency ）, So I haven't used it ScheduledExecutorService To realize the function of timing .

Later , It's time to learn to do a project , At that time, the video had a Quartz Course . I remember understanding for a long time , Finally I realized , original Write so much code It is used to realize the function of timing .

As for the ratio ScheduledExecutorService and Timer What's good about it , The most intuitive is ： It supports cron expression .

Why do I understand for a long time , because Quartz Of api It's too complicated （ It also has its own terminology and conceptual things ）. But this kind of follow-up project , Follow my step .

and Quartz dependent API I can't remember , But then I understood ： So we can write code by 「 The component package 」 To do what you want , So this is cron expression .

When I was a Junior , I want to write a small project with the knowledge I have learned , It can also be regarded as combing through what they have learned . therefore , I remember. Quartz.

At that time, I had learned Spring/SpringBoot 了 . So when I search online Spring And Quartz When integrating , I learned that SpringTask, And then it was discovered that @Schedule annotation .

Just a simple annotation , Can realize the function of scheduled tasks , And support cron expression .

That, that, that, that , And a hammer Quartz ah ！

02、 Internship && Work Timing task

When I work , I learned a new term 「 Distributed timed task framework 」. When I enter the workplace , I just found that the original scheduled task works so well ！

List the common postures I use for timed tasks in real work ：

1、 Dynamically create scheduled tasks push Operation messages （ Timing push message ）

2、 Advertising settlement timing task Sweep the table Find the corresponding settleable record （ Periodic table scanning update status ）

3、 Update data records regularly every day （ Update data regularly ）

Many people ask me if I have ever used Distributed transactions , I tend to answer ： No! , We are all Sweep the table A shuttle ensures the final consistency of data, of course , If you are asked during the interview , Distributed transactions can blow . How do you actually scan your watch ？ Namely timing Sweep it .

in addition , I had a brief look at how the company's self-developed distributed timed task framework did , I remember it was based on Quartz extended , The extension has failover、 Fragmentation And so on. .

Generally speaking , Using timed tasks is in App launch perhaps Ahead of time Web page Configure scheduled tasks （ Timed task frameworks support cron Of expression , So it's a periodic or timed task ), This kind of scene is the most .

03、 Why distributed scheduled tasks

Mentioned earlier Timer/ScheduledExecutorService/SpringTask(@Schedule) It's all stand-alone , But once we're in the production environment , Application deployment Often It's all in cluster mode .

Under the cluster , We usually hope A scheduled task Execute only on one machine , At this time of the , The timing task implemented by a single machine is not easy to handle .

Quartz Yes, there is Cluster deployment Of the plan , So some people will take advantage of Database row lock Or use Redis Distributed lock To achieve their own scheduled tasks, run on a certain machine Application machine On ; It must be possible to do , Including some well-known distributed timed task frameworks , Can solve problems .

But these are not the only problems we have , For example, I want to support Fault tolerance function （ Failure to retry ）、 Fragmentation function 、 Manual trigger A mission 、 There is a better way to manage scheduled tasks Backend interface 、 route Load balancing, etc . These functions , It's doing 「 Distributed timed task framework 」 What we have .

Now that there are so many wheels , Then let's act as User / Demand side There's no need to re implement it yourself , Just use what you have , We can learn the implementation design idea of existing wheels .

04、 Distributed timed task foundation

Quartz Is an excellent open source component , It abstracts the timed task into three roles ： Scheduler 、 actuator and Mission , So that the distributed timed task frameworks on the market have similar role division .

For our users , Generally, it introduces a client package , Then according to its rules （ It may be using annotations to identify , Or implement an interface ）, Then customize our own scheduled task logic .

Look at the character abstraction and general posture corresponding to the above execution diagram , It should be easier to understand this process . We can think about two more questions ：

1、 Task information and scheduling information are needed Storage Of , Where is the store ？ The scheduler is required 「 notice 」 The actuator performs , that 「 notice 」 In what way ？

2、 How does the scheduler find the task to be executed ？

For the first question , Distributed timed task framework can be divided into Two schools ： Centralization and decentralization

• So-called 「 Centralization 」 refer to ： Scheduler and actuator Separate , The scheduler performs unified scheduling , Inform the actuator to perform scheduled tasks
• So-called 「 De centralization 」 refer to ： Scheduler and actuator coupling , Self scheduling and self execution

about 「 Centralization 」 In terms of genre , Storing relevant information is likely to be in database （DataBase）, And what we introduced client The bag is actually actuator Related code . Scheduler The task scheduling is realized The logic of , The remote invocation The actuator triggers the corresponding logic .

Scheduler 「 notice 」 When the actuator performs a task , It can be through 「RPC」 call , It can also write the task information to the message queue for consumption by the actuator to achieve the purpose .

about 「 De centralization 」 For genres, storing relevant information is likely to be in Registry Center （Zookeeper）, And what we introduced client The bag is actually actuator + Scheduler Related code .

Rely on the registry to complete Assignment of tasks ,「 Centralization 」 When scheduling, schools need to ensure that a task is consumed by only one machine , This requires writing distributed lock related logic in the code to ensure , and 「 De centralization 」 Relying on the registry eliminates this link .

For the second question , How does the scheduler find the task to be executed ？ Now the newer distributed timed task frameworks are generally used 「 Time wheel 」.

1、 If we want to find the task we are going to perform every day , May put these tasks in one List Then make a judgment , At this time, the query time complexity is O(n)

2、 A little bit better , We may put these tasks in a minimum pile （ Sort the time ）, The time complexity of addition, deletion and modification at this time is O(logn), And the query is O(1)

3、 Improve it , We put these tasks in one Ring array in , The time complexity of adding, deleting, modifying and checking at this time is O(1). But the size of the ring array at this time determines the size of the task we can store , Tasks beyond circular arrays need to be stored in another array structure .

4、 Finally, improve , We can have Multi-storey Ring array , Of ring arrays at different levels precision It's different , Using multi-layer ring array can greatly improve our accuracy .

05、 Distributed timed task framework

There are still many options for distributed timed task framework , Well known ones are ：XXL-JOB/Elastic-Job/LTS/SchedulerX/Saturn/PowerJob And so on. . Conditional companies may be based on Quartz Expand , Develop a set of distributed timed task framework in line with your own company .

I didn't come from this background , For me , my austin The technical selection of the project will mainly focus on two aspects （ In fact, it's about choice apollo The reason for being a distributed configuration center is the same ）： mature 、 Stable 、 Is the community active .

This time I chose xxl-job As austin Distributed task scheduling framework .xxl-job There are already many companies that have access to （ Explain his Open the box It's still in place ）. But the latest version is in 2021-02, There has been no major update in the past year .

06、 Why? austin Need a distributed timed task framework

go back to austin On the system architecture ,austin-admin The background management page has been created by me , This background management system will provide 「 The message template 」 Management function of .

Then sending a message is not just 「 Technology side 」 Call the interface to send , Many more are 「 Operation side 」 Push by setting the timing .

And this function , I need to use the distributed timed task framework as the middleware to support my business , And a very important point ： The distributed timed task framework needs to support dynamic The function of creating scheduled tasks .

When you click... On the page 「 start-up 」 When , You need to create a scheduled task , When you click... On the page 「 Pause 」 When , You need to stop the scheduled task , When you click... On the page 「 Delete 」 The template , If you've ever had a scheduled task , You need to delete it together . When you click... On the page 「 edit 」 And when it's saved , You also need to stop scheduled tasks .

Um. , That's all the process needed

07、austin Access xxl-job

Access xxl-job The steps of distributed timed task framework are still quite simple （ Look at the document and you'll basically ）, Let me put it simply . Access specific code, you can pull ausitn Come down and have a look , I'll focus on how I feel when I connect .

Official document ：https://www.xuxueli.com/xxl-job/#%E4%BA%8C%E3%80%81%E5%BF%AB%E9%80%9F%E5%85%A5%E9%97%A8

1、 Introduce... Into your own project xxl-job-core Of maven rely on

2、 stay MySQL In the implementation of /xxl-job/doc/db/tables_xxl_job.sql Of SQL Script

3、 from Gitee or GitHub download xxl-job Source code , modify xxl-job-admin Of the dispatch center database To configure , start-up xxl-job-admin project .

4、 Add... To your project xxl-job Related configuration information

5、 Use @XxlJob The annotation modification method is used to write the related logic of scheduled tasks

It should be easy to find from the small partners who access or have seen the documents ,xxl-job It belongs to 「 Centralization 」 Genre of distributed timed task framework , Scheduler and actuator are separate .

I mentioned earlier austin need dynamic Add, delete and modify scheduled tasks , and xxl-job Is to support the , But I don't think it's packaged well enough , Only on the scheduler http Interface . And call http Interfaces are relatively cumbersome , Many related JavaBean None of them are in core Package definition , I can only write it again .

therefore , It took me a long time and a lot of code to complete dynamic Adding, deleting and modifying scheduled tasks .

The scheduler and the actuator are deployed separately , signify , The network of scheduler and actuator is a must Accessible Of ： Originally, I didn't install any environment here , Include MySQL I am connected to ECs , But now I have to debug in a network accessible environment , So I have to start locally xxl-job-admin Dispatch center to debug .

When starting the actuator , Will open a new port for xxl-job-admin Dispatch center calls instead of reusing SpringBoot The default port is also strange ？

08、 summary

This article mainly talks about what is timed task 、 Why use timed tasks 、 stay Java If there are requirements related to scheduled tasks in the field, what can be used to realize 、 Basic knowledge of distributed timing tasks and how to access them XXL-JOB

I believe you have a basic understanding of the distributed timed task framework , If you are interested, you can choose an open source framework to learn , If you want to know the access code, you can put my austin Pull down the project and have a look .

The main code is austin-cron Of xxl It's a bag , The codes of distributed applications are mainly austin-web Of MessageTemplateController Coupled with the addition, deletion, modification and query of the template .

Next, I want to talk about when a timed task is triggered , Got a crowd file , How do I design to call messages to push and send .

You can see it here , It's not too much to praise ？ I am a 3y, See you next time .

austin project Source code Gitee link ：gitee.com/austin

austin project Source code GitHub link ：github.com/austin