当前位置:网站首页>Es log error appreciation -- allow delete
Es log error appreciation -- allow delete
2022-07-07 14:33:00 【Es assistant】
Error reporting
[o.e.x.w.e.ExecutionService] [16142xxxxxx20932] could not store triggered watch with id [xxxxx4-2022-07-07T04:04:33.872Z]: [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]
Error reporting analysis
When disk utilization exceeds 85%, Or achieve 100%, It can lead to Elasticsearch Cluster or Kibana Unable to provide services properly , The following problem scenarios may occur :
- When making an index request , Return similar
{[FORBIDDEN/12/index read-only/allow delete(api)];","type":"cluster_block_exception"}
The error of . - When operating on a cluster , Return similar
[FORBIDDEN/13/cluster read-only / allow delete (api)]
The error of . - Cluster in the Red state , In severe cases, nodes do not join the cluster ( It can be done by
GET _cat/allocation?v
Command view ), And there are unallocated fragments ( It can be done byGET _cat/allocation?v
Command view ). - adopt Elasticsearch The node monitoring page of the console , The disk utilization of cluster nodes has reached or approached 100%.
The above problems are caused by high disk utilization . There are three watermarks for the disk utilization of data nodes , Exceeding the water level may affect Elasticsearch or Kibana service .
- When the cluster disk utilization exceeds 85%: Will result in new tiles not being allocated .
- When the cluster disk utilization exceeds 90%:Elasticsearch It will try to migrate the shards in the corresponding nodes to other data nodes with low disk utilization .
- When the cluster disk utilization exceeds 95%: The system will be right Elasticsearch Each index in the corresponding node in the cluster is set forcibly read_only_allow_delete attribute , At this time, all indexes on the node will not be able to write data , Only the corresponding index can be read and deleted .
Solution
Clean up the expired data of the cluster
Users can access 【Kibana】>【Dev Tools】 Delete expired indexes to free up disk space . Steps are as follows : Warning : After the data is deleted, it cannot be recovered , Please operate carefully . You can also choose to keep the data , However, disk expansion is required .
First step : Enable cluster index batch operation permission .
PUT _cluster/settings { "persistent": { "action.destructive_requires_name": "false" } }
The second step : Delete data , for example DELETE NginxLog-12*.
DELETE index-name-*
- After performing the above steps , If the user Tencent cloud Elasticsearch The version is 7.5.1 Previous versions , Still need to be in Kibana Interface 【Dev Tools】 Execute the following command :
Turn off index read-only status , Execute the following command :
PUT _all/_settings { "index.blocks.read_only_allow_delete": null }
Turn off the read-only status of the cluster , Execute the following command :
PUT _cluster/settings { "persistent": { "cluster.blocks.read_only_allow_delete": null } }
- Check whether the cluster index is still
read_only
state , Whether the index write returns to normal . - If the cluster is still Red state , Execute the following command , Check whether there are unassigned shards in the cluster . GET /_cluster/allocation/explain
- Wait for the partition distribution to complete , View the cluster status . If the cluster status is still Red, Please pass After sales support Contact Tencent cloud technical support .
- To avoid the impact of high disk utilization Elasticsearch service , It is recommended to turn on the disk usage monitoring alarm , Check and receive the alarm messages in time , Take defensive measures in advance , For details, please refer to Suggestions on monitoring alarm configuration .
Cloud disk expansion reference :
https://cloud.tencent.com/document/product/845/56276
边栏推荐
猜你喜欢
一个简单LEGv8处理器的Verilog实现【四】【单周期实现基础知识及模块设计讲解】
Base64 encoding
AWS学习笔记(三)
多商戶商城系統功能拆解01講-產品架構
Démontage de la fonction du système multi - Merchant Mall 01 - architecture du produit
GVIM [III] [u vimrc configuration]
什么是云原生?这回终于能搞明白了!
Pytorch model trains practical skills and breaks through the bottleneck of speed
UML sequence diagram (sequence diagram)
LeetCode 648. 单词替换
随机推荐
电脑Win7系统桌面图标太大怎么调小
bashrc与profile
【立体匹配论文阅读】【三】INTS
Mmkv use and principle
Leetcode one question per day (636. exclusive time of functions)
Simple use of websocket
PAG体验:十分钟完成AE动效部署上线各平台!
Demis hassabis talks about alphafold's future goals
Docker deploy Oracle
Démontage de la fonction du système multi - Merchant Mall 01 - architecture du produit
EMQX 5.0 发布:单集群支持 1 亿 MQTT 连接的开源物联网消息服务器
Half an hour of hands-on practice of "live broadcast Lianmai construction", college students' resume of technical posts plus points get!
Reading and understanding of eventbus source code
Leetcode - Sword finger offer 05 Replace spaces
OAuth 2.0 + JWT 保护API安全
Attribute keywords ondelete, private, readonly, required
OAuth 2.0 + JWT protect API security
Demis Hassabis谈AlphaFold未来目标
electron remote 报错
Small game design framework