August 14, 2020: what are the execution engines for data tasks?

5 Line code , Can let wechat small program on the shelf to their own APP in | Register and send it to Dajiang 、 Huawei 、 Cherry keyboard ！>>>

Fogo's answer 2020-08-14：

Know the answer

The most basic thing is crontab Timing task .
Considering the distributed partition timing task and high availability , It can be used elastic-job Such a framework . But such a framework cannot handle complex task dependencies .
Big data processing frameworks generally use DAG（ Directed acyclic graph ） Workflow scheduling system , adopt DAG The graph describes the complex dependencies between tasks . stay Spark In job scheduling system , Is based on DAG describe rdd Dependencies between .