How does schedulerx help users solve the problem of distributed task scheduling?

author ： Qianxi

Preface

In various business system scenarios , There are a lot of timing triggers 、 The cycle triggers the demand scenario of running the specified business task , The significance of the distributed task scheduling middleware platform is to manage and support the above scenarios . stay Linux Medium crontab、Java Medium Timer And so on, which involve periodically scheduling running tasks to complete a series of business scenarios of automatic processing .

Periodic and regular running is the most basic feature of task scheduling , However, with the expansion and development of business, higher requirements will be put forward for the ability of task scheduling middleware platform . In traditional business systems Quartz as well as Spring scheduling The package provides a lot of convenience for business development timing tasks in the way of framework integration , However, with the distributed and micro service deployment of various business applications , With a large number of scattered scheduled tasks, it is difficult to uniformly and visually monitor, operate and manage all global tasks in various business application systems , The distributed task scheduling middleware platform will effectively and visually control all scheduled tasks .

The distributed task scheduling platform is based on highly reliable scheduled task scheduling , Take visual management and control as the core value to reflect , Combined with the business development trend, expand the platform capability around these two basic elements .

SchedulerX overview

Platform resource management

From the perspective of unified operation and maintenance management and control for all business applications , Abstracting the team and deployment environment “ Space ” Concept for resource management isolation , Business applications and machine clusters are abstracted in a layer called “ Application grouping ”. Each business application can create a corresponding application group on the scheduling platform to connect with its actual business application , Thus, each business team can control the tasks of each business team without interference on a unified platform . And based on... On Alibaba cloud RAM Authority strategy , Reasonable resource authority control and isolation can be carried out uniformly .

Through the above resource model, business platform architecture managers can according to the organizational characteristics of their own team , Reasonable planning of space and application and configuration of permission policy , Clearly realize the access control isolation and global control of task scheduling resources .

Platform visual control

Task is the basic unit that the task scheduling platform controls scheduling operations , The visualization of task operation enables the tasks that were hidden in various business applications to run silently to be seen again , Let the operation status and execution results of each task be displayed and reminded . Without visualization , The running status and results of the task will be unknown or difficult to detect , Even after a lot of business iterative development , How many scheduled tasks exist in the application system cannot be managed in a standardized way .

Distributed fragment batch processing

With the development of business volume , In some timed scheduling task scenarios, there will be a large amount of data distributed batch processing requirements . Coordinating multiple machines to complete mass data processing regularly has also become an important functional feature of the task scheduling platform , Through the distributed batch processing model , Users can simply improve the efficiency of mass data processing .

Simple use of scenarios

The following will be based on the task scheduling platform , Specifically list several simple business use scenarios , To get a preliminary understanding of what can be done on the task scheduling platform .

Based on the operation and maintenance cluster broadcast scenario

In broadcast mode , The created scheduling task will issue the instruction of task operation to the business application cluster or install the... At a given cycle frequency Agent Of ECS colony , When the newly added resources are expanded, they will be dynamically broadcast to . Based on this feature , Users can construct many customized usage scenarios , for example ：

• journal / Temporary data shall be cleaned regularly ： Regularly clean up logs or temporary files generated by business .

• Server detection ： Users can broadcast shell Scripts quickly build simple server disks / Memory /CPU Wait for the health detection scene .

• Business cache refresh ： For business scenarios with local cache , The cache can be refreshed regularly and the specified business application service can be preheated .

• Business service detection ： Users can customize the collection and judgment of indicators of various business types , Flexible construction of lightweight business monitoring system based on scheduling platform .

Timed business scenarios

In the business scenarios of various industries , Perform scheduled task processing on the task scheduling platform , Is the most extensive form of business , for example ：

• Send message reminders regularly ： Payment reminder 、 Point expiration reminder 、 Customer employee birthday wishes reminder 、 Transaction order processing notification, etc .

• Regular data synchronization ： Employee organization structure information synchronization 、 Basic business information synchronization 、 Regular daily business data batch clearing 、 Transaction order timeout processing, etc .

• Regular data generation and push ： month / season / Annual report generation and push 、 Activity announcements are pushed regularly 、 Regular generation and push of consumption bills, etc .

When the volume of data to be processed by the above businesses gradually expands , In the original stand-alone processing mode, the processing efficiency bottleneck will slowly appear , At this time, the distributed cluster batch processing ability of task scheduling can give full play to the cluster cooperation ability to carry out mass data parallel processing .

Distributed batch processing scenario

Based on business batch processing capability , When the real-time business volume is large, users can cooperate with the service degradation strategy to build business periodic batch compensation tasks , To improve peak business processing capacity , To ensure the throughput of core business during business peak period . As the architecture designer of business system platform , If the timed distributed batch processing capability can be effectively combined with specific business, the overall business processing capability and stability of the system will be brought into full play . The following is an abstract summary of relevant business usage scenarios for reference .

In the normal scenario, the core trading service may pass through RPC/MQ Complete the call processing of downstream Services . But in this mode, when the business volume increases, some problems will arise ,RPC The downstream service capacity will affect the core service throughput capacity , Yes MQ When relying on, you need to ensure that the message delivery is normal and the consumer handles it correctly , These will become the bottleneck of improving the processing capacity of core services . Therefore, within the scope of business acceptance , Adopting the method of periodic batch processing compensation with complete decoupling between business services will greatly improve the processing efficiency , Moreover, the reliability guarantee of downstream services will be improved through idempotent and repeatable execution , At the same time, combined with the distributed batch processing model, the data can be loaded in batch 、 Batch processing to reduce DB Interactive pressure . At the same time, in terms of architecture design, the compensation mode can be integrated with micro service degradation .

Fast access SchedulerX

Create application groups

Enter the task scheduling console and select “ Public network ”region, First, in the “ Application management ”-> Create an , This information will become the next step in the business application 、Agent The core element that directly establishes the docking relationship with the task scheduling platform .

SpringBoot Project introduction  SchedulerX

Built locally SpringBoot Add the following dependencies to the project , And in Properties Add the application group configuration information created by the corresponding console , Start the application to establish the relationship between the business application and the task scheduling platform .

Log in and copy

      
      

       
       <dependencies>
       
       
  <dependency>
       
       
    <groupId>com.aliyun.schedulerx</groupId>
       
       
    <artifactId>schedulerx2-spring-boot-starter</artifactId>
       
       
    <version>1.3.2</version>
       
       
  </dependency>
       
       
</dependencies>
      
      
      
      

       
       
1.
       
       
2.
       
       
3.
       
       
4.
       
       
5.
       
       
6.
       
       
7.
      
      

Log in and copy

      
      

       
       #  Public cloud public network environment 
       
       
spring.schedulerx2.namespace=aad167f6-8bee-41a7-ba41-*********
       
       
spring.schedulerx2.endpoint=acm.aliyun.com
       
       
spring.schedulerx2.groupId=qianxi.text
       
       
spring.schedulerx2.appKey=lYgR6qq**********
      
      
      
      

       
       
1.
       
       
2.
       
       
3.
       
       
4.
       
       
5.
      
      

Other instructions

After completing the above steps , You can create and develop specific business scheduled tasks under subsequent applications , See the official manual for subsequent use , Click on ​ ​ here ​​ You can go there ！

summary

This paper defines the resources of the task scheduling platform 、 Visual control capability 、 The distributed batch processing capability is briefly described , And based on SchedulerX It provides some basic reference cases in combination with the actual business scenarios . It is hoped that through the above contents, we can easily get familiar with the access and use of the task scheduling platform , For existing users, we can also control and isolate platform resources in combination with the characteristics of our own team , And improve the processing efficiency through distributed batch processing after the growth of product business .

follow-up SchedulerX We will continue to improve the ability of visual control to serve the needs of enterprise level scheduled task management , Moreover, the platform management and control and timing scheduling service capabilities will be gradually compatible with the common open source component clients on the market . We will also combine with the actual scene , Continue to analyze and explain one by one , Coming soon .