当前位置:网站首页>Didi elasticsearch cluster cross version upgrade and platform reconfiguration
Didi elasticsearch cluster cross version upgrade and platform reconfiguration
2020-11-06 01:15:00 【InfoQ】
Reading guide : Not long ago , sound of dripping water ES The team will maintain 30 Multiple ES colony ,3500 Multiple ES node ,8PB The data of , from 2.3.3 Seamless upgrade to 6.6.1. Under the premise of basically zero impact and change on user query writing , It's solved ES Cross major protocol incompatibility 、 Incompatible file format 、mapping Incompatibility, etc , The whole process is completely transparent to most users . At the same time, it's done Arius Architecture upgrade , The single machine query performance has been improved 40%, The whole cluster cpu falling 10%, write in tps promote 30%, The utilization rate of cluster resources has been improved 20%、0 fault 、 The cost of operation and maintenance has decreased 60% The achievement of .
This paper will systematically introduce didi from 2.3.3 Upgrade to across large versions 6.6.1 Problems encountered in the process and solutions , And in the search platform construction process of systematic thinking .
01 Background introduction
1. The cluster size
At present, Didi uses ES The version is 2.3.3, The number of clusters is 40 Multiple , Node scale has 3500+, The total capacity of the cluster is 8PB.
2. Business scale
1200 Multiple platform applications are using ES,30 Multiple core applications are using ES, Written in TPS Yes 1500W, Of the query QPS Yes 25W.
02 Problem analysis
For the above scale ES colony , from 2.3.3 Upgrade to 6.X edition , The small version will be determined based on the results of the final analysis , Potential problems need to be analyzed and distinguished .
1. Problem analysis
First of all, it analyzes the four problem areas :
- Engine side : As a result of 2.3.3 Upgrade to 6.X edition , The version gap is too big , Not compatible in file format or protocol , So you can't roll in place and upgrade directly , Need to double write relocation and upgrade , It costs a lot of machines to get involved
- The user side :6.X The version gradually does not support TCP Interface , Therefore, users need to adapt and upgrade ; There are also some differences between query and return value , If the user side adapts , Will greatly affect the progress of the upgrade
- Resource side : Because you can't scroll directly in place, upgrade directly , Need to double write using a lot of machines , But we can't provide the machines needed to upgrade , If resources cannot be guaranteed during the upgrade process , That will also greatly affect the progress of the upgrade
- Operation side : How to control the operation and maintenance of the new version of multi cluster ? How to verify the results of the upgrade ? How to guarantee the efficiency and quality of query ? All these problems need to be considered
Link to the original text :【https://www.infoq.cn/article/J6BFzWfJA0cbmgzN7R1w】. Without the permission of the author , Prohibited reproduced .
版权声明
本文为[InfoQ]所创,转载请带上原文链接,感谢
边栏推荐
- Ubuntu18.04上安裝NS-3
- Listening to silent words: hand in hand teaching you sign language recognition with modelarts
- JVM内存区域与垃圾回收
- [译] 5个Vuex插件,给你的下个VueJS项目
- 业务策略、业务规则、业务流程和业务主数据之间关系 - modernanalyst
- 基础知识点整理
- 7.2.2 compressing static resources through gzipresourceresolver
- DevOps是什么
- Jmeter——ForEach Controller&Loop Controller
- TF flags的简介
猜你喜欢
随机推荐
c++学习之路:从入门到精通
6.9.2 session flashmapmanager redirection management
python 下载模块加速实现记录
谁说Cat不能做链路跟踪的,给我站出来
高级 Vue 组件模式 (3)
一时技痒,撸了个动态线程池,源码放Github了
直播预告 | 微服务架构学习系列直播第三期
WeihanLi.Npoi 1.11.0/1.12.0 Release Notes
安装Anaconda3 后,怎样使用 Python 2.7?
Chainlink将美国选举结果带入区块链 - Everipedia
Network programming NiO: Bio and NiO
Asp.Net Core學習筆記:入門篇
經典動態規劃:完全揹包問題
神经网络简史
After brushing leetcode's linked list topic, I found a secret!
Python machine learning algorithm: linear regression
技術總監7年經驗,告訴大家,【拒絕】才是專業
7.3.2 File Download & big file download
技術總監,送給剛畢業的程式設計師們一句話——做好小事,才能成就大事
7.3.1 file upload and zero XML registration interceptor