当前位置:网站首页>2020-08-14:数据任务的执行引擎用的哪些?
2020-08-14:数据任务的执行引擎用的哪些?
2020-11-06 21:55:00 【福大大架构师每日一题】
福哥答案2020-08-14:
最基础的是 crontab 定时任务。
考虑到分布式分片定时任务和高可用,可以用elastic-job这样的框架。但是这样的框架无法处理复杂的任务依赖关系。
大数据处理框架一般用的是DAG(有向无环图)工作流类调度系统,通过DAG图描述任务之间的复杂的依赖关系。在Spark作业调度系统中,就是基于DAG描述rdd之间的依赖关系。
版权声明
本文为[福大大架构师每日一题]所创,转载请带上原文链接,感谢
https://my.oschina.net/u/4553401/blog/4495972
边栏推荐
- 华为Mate 40 系列搭载HMS有什么亮点?
- Take you to learn the new methods in Es5
- Using iceberg on kubernetes to create a new generation of cloud original data Lake
- How to turn data into assets? Attracting data scientists
- ORA-02292: 违反完整约束条件 (MIDBJDEV2.SYS_C0020757) - 已找到子记录
- Those who have worked in China for six years and a million annual salary want to share these four points with you
- 2020年数据库技术大会助力技术提升
- From overseas to China, rancher wants to do research on container cloud market
- ES中删除索引的mapping字段时应该考虑的点
- 面试官: ShardingSphere 学一下吧
猜你喜欢
Look! Internet, e-commerce offline big data analysis best practice! (Internet disk link attached)
常用SQL语句总结
What are Devops
With this artifact, quickly say goodbye to spam messages
Kubernetes and OAM to build a unified, standardized application management platform knowledge! (Internet disk link attached)
This project allows you to quickly learn about a programming language in a few minutes
What course of artificial intelligence? Will it replace human work?
Staying up late summarizes the key points of report automation, data visualization and mining, which is different from what you think
Zero basis to build a web search engine of its own
ES6 learning notes (2): teach you to play with class inheritance and class objects
随机推荐
Zero basis to build a web search engine of its own
Building a new generation cloud native data lake with iceberg on kubernetes
list转换map(根据key来拆分list,相同key的value为一个list)
StickEngine-架构12-通信协议
What are manufacturing and new automation technologies?
2020-09-04:函数调用约定了解么?
How to play sortable JS vuedraggable to realize nested drag function of forms
Behind the first lane level navigation in the industry
Live broadcast preview | micro service architecture Learning Series live broadcast phase 3
Why is the LS command stuck when there are too many files?
代码重构之法——方法重构分析
嘉宾专访|2020 PostgreSQL亚洲大会阿里云数据库专场:曾文旌
Using an example to understand the underlying processing mechanism of JS function
To Lianyun analysis: why is IPFs / filecoin mining so difficult?
What the hell is fastthreadlocal? The existence of ThreadLocal!!
What are Devops
Try to build my mall from scratch (2): use JWT to protect our information security and perfect swagger configuration
2020-09-09:裸写算法:两个线程轮流打印数字1-100。
window系统 本机查找端口号占用方法
嘉宾专访|2020 PostgreSQL亚洲大会阿里云数据库专场:王涛