Flink on paasta: yelp's new stream processing platform running on kubernetes

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" This article was originally published in yelp Engineering Blog , from InfoQ Translate and share ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" stay Yelp, We use... Every day "},{"type":"link","attrs":{"href":"https:\/\/flink.apache.org\/","title":"","type":null},"content":[{"type":"text","text":"Apache Flink"}]},{"type":"text","text":" Handle TB Level of stream data , Support for a variety of applications ：ETL The Conduit 、 Push notification 、 Robot filtering 、Session And so on . We run hundreds of Flink Homework , therefore , Without the right level of Automation , Like deployment 、 Restart and "},{"type":"link","attrs":{"href":"https:\/\/ci.apache.org\/projects\/flink\/flink-docs-release-1.11\/ops\/state\/savepoints.html","title":"","type":null},"content":[{"type":"text","text":" Save it "}]},{"type":"text","text":" Such routine operations can take thousands of hours for developers . lately , A new streaming platform has been added to our toolroom , It's based on Yelp Of PaaS service "},{"type":"link","attrs":{"href":"https:\/\/engineeringblog.yelp.com\/2015\/11\/introducing-paasta-an-open-platform-as-a-service.html","title":"","type":null},"content":[{"type":"text","text":"PaaSTA"}]},{"type":"text","text":". At its core is a "},{"type":"link","attrs":{"href":"https:\/\/kubernetes.io\/","title":"","type":null},"content":[{"type":"text","text":"Kubernetes"}]},{"type":"link","attrs":{"href":"https:\/\/kubernetes.io\/docs\/concepts\/extend-kubernetes\/operator\/","title":"","type":null},"content":[{"type":"text","text":"Operator"}]},{"type":"text","text":", It automatically monitors our Flink Clustered fleet Deployment and lifecycle ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/resource\/image\/94\/1e\/94e5ab8a26535ffba733bf8fe61b441e.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"Flink on PaaSTA on Kubernetes"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":" introduce Kubernetes Before "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" stay Yelp introduce Kubernetes Before ,Yelp Of Flink The workload runs on dedicated AWS"},{"type":"link","attrs":{"href":"https:\/\/aws.amazon.com\/emr\/","title":"","type":null},"content":[{"type":"text","text":"ElasticMapReduce"}]},{"type":"text","text":" On the cluster , These clusters are pre installed with Flink and "},{"type":"link","attrs":{"href":"https:\/\/hadoop.apache.org\/docs\/current\/hadoop-yarn\/hadoop-yarn-site\/YARN.html","title":"","type":null},"content":[{"type":"text","text":"YARN"}]},{"type":"text","text":". In order to achieve EMR Examples and Yelp Good synergy of the rest of the ecosystem , Our previous streaming platform Cascade In a "},{"type":"link","attrs":{"href":"https:\/\/www.docker.com\/","title":"","type":null},"content":[{"type":"text","text":"Docker"}]},{"type":"text","text":" A large number of "},{"type":"link","attrs":{"href":"https:\/\/puppet.com\/docs\/pe\/2019.8\/peuser<\/i>guide.html","title":"","type":null},"content":[{"type":"text","text":"Puppet"}]},{"type":"text","text":" Monomers , To apply configuration and start a set of common daemons （ stay Yelp Almost all hosts run ）."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/resource\/image\/ec\/44\/ec553349f86e55448f70c38bd7553544.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"Cascade The architecture of "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}