Es log error appreciation -- allow delete

Error reporting

[o.e.x.w.e.ExecutionService] [16142xxxxxx20932] could not store triggered watch with id [xxxxx4-2022-07-07T04:04:33.872Z]: [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]

Copy

Error reporting analysis

When disk utilization exceeds 85%, Or achieve 100%, It can lead to Elasticsearch Cluster or Kibana Unable to provide services properly , The following problem scenarios may occur ：

• When making an index request , Return similar {[FORBIDDEN/12/index read-only/allow delete(api)];","type":"cluster_block_exception"} The error of .
• When operating on a cluster , Return similar [FORBIDDEN/13/cluster read-only / allow delete (api)] The error of .
• Cluster in the Red state , In severe cases, nodes do not join the cluster （ It can be done by GET _cat/allocation?v Command view ）, And there are unallocated fragments （ It can be done by GET _cat/allocation?v Command view ）.
• adopt Elasticsearch The node monitoring page of the console , The disk utilization of cluster nodes has reached or approached 100%.

The above problems are caused by high disk utilization . There are three watermarks for the disk utilization of data nodes , Exceeding the water level may affect Elasticsearch or Kibana service .

• When the cluster disk utilization exceeds 85%： Will result in new tiles not being allocated .
• When the cluster disk utilization exceeds 90%：Elasticsearch It will try to migrate the shards in the corresponding nodes to other data nodes with low disk utilization .
• When the cluster disk utilization exceeds 95%： The system will be right Elasticsearch Each index in the corresponding node in the cluster is set forcibly read_only_allow_delete attribute , At this time, all indexes on the node will not be able to write data , Only the corresponding index can be read and deleted .

Solution

Clean up the expired data of the cluster

Users can access 【Kibana】>【Dev Tools】 Delete expired indexes to free up disk space . Steps are as follows ： Warning ： After the data is deleted, it cannot be recovered , Please operate carefully . You can also choose to keep the data , However, disk expansion is required .

First step ： Enable cluster index batch operation permission .

PUT _cluster/settings  {             "persistent": {              "action.destructive_requires_name": "false"             } }
 The second step ： Delete data , for example  DELETE NginxLog-12*.

DELETE index-name-*

Copy

1. After performing the above steps , If the user Tencent cloud Elasticsearch The version is 7.5.1 Previous versions , Still need to be in Kibana Interface 【Dev Tools】 Execute the following command ：

Turn off index read-only status , Execute the following command ：

PUT _all/_settings {             "index.blocks.read_only_allow_delete": null }

Copy

Turn off the read-only status of the cluster , Execute the following command ：

PUT _cluster/settings {             "persistent": {                 "cluster.blocks.read_only_allow_delete": null             } }

Copy

1. Check whether the cluster index is still read_only state , Whether the index write returns to normal .
2. If the cluster is still Red state , Execute the following command , Check whether there are unassigned shards in the cluster . GET /_cluster/allocation/explain
3. Wait for the partition distribution to complete , View the cluster status . If the cluster status is still Red, Please pass After sales support Contact Tencent cloud technical support .
4. To avoid the impact of high disk utilization Elasticsearch service , It is recommended to turn on the disk usage monitoring alarm , Check and receive the alarm messages in time , Take defensive measures in advance , For details, please refer to Suggestions on monitoring alarm configuration .

Cloud disk expansion reference ：

https://cloud.tencent.com/document/product/845/56276