Kibana report generation failed due to custom template

explain

The problems and solutions described in this paper are also applicable to Tencent cloud Elasticsearch Service（ES）.

background

Kibana Medium Dashboard Give us an intuitive data display . In real work , Can be used for reporting . stay Kibana in , We can generate what we want Report.

problem

Failed to generate report , Report errors ：

Can't reach the server. Please try agin.

open F12, An internal error is returned .

The question is rather strange , It needs to be analyzed in depth .

Question why

One 、 analysis kibana Abnormal log

open kibana Log , Through the analysis of , We found something unusual ：

"message":"[illegal_argument_exception] Rejecting mapping update to [.reporting-2021.10.24] as the final mapping would have more than 1 type: [esqueue, doc]"}
{"type":"response","@timestamp":"2021-10-27T05:53:12Z","tags":["api"],"pid":14595,"method":"post","statusCode":500,"req":{"url":"/api/reporting/generate/csv?jobParams=(conflictedTypesFields%3A!(kfext%2Ckfuin%2CrequestId)%2Cfields%3A!('%40timestamp'%2Ctext)%2CindexPatternId%3A'21fe4820-8916-11ea-8b39-a19e11c4dfcb'%2CmetaFields%3A!(_source%2C_id%2C_type%2C_index%2C_score)%2CsearchRequest%3A(body%3A(_source%3A(excludes%3A!()%2Cincludes%3A!('%40timestamp'%2Ctext))%2Cdocvalue_fields%3A!()%2Cquery%3A(bool%3A(filter%3A!()%2Cmust%3A!((query_string%3A(analyze_wildcard%3A!t%2Cdefault_field%3A'*'%2Cquery%3A'%22high%20risky%20with%20req%22'))%2C(range%3A('%40timestamp'%3A(format%3Aepoch_millis%2Cgte%3A1635310298396%2Clte%3A1635313898396))))%2Cmust_not%3A!()%2Cshould%3A!()))%2Cscript_fields%3A()%2Csort%3A!(('%40timestamp'%3A(order%3Adesc%2Cunmapped_type%3Aboolean)))%2Cstored_fields%3A!('%40timestamp'%2Ctext)%2Cversion%3A!t)%2Cindex%3A'account-admin-ol-*')%2Ctitle%3A'high%20risky%20with%20req'%2Ctype%3Asearch)","method":"post","headers":{"host":"kibana","connection":"close","content-length":"0","x-stgw-time":"1635313992.732","x-client-proto":"https","x-forwarded-proto":"https","x-client-proto-ver":"HTTP/2.0","x-real-ip":"116.233.19.162","x-forwarded-for":"116.233.19.162","sec-ch-ua":"\"Chromium\";v=\"92\", \" Not A;Brand\";v=\"99\", \"Google Chrome\";v=\"92\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36","kbn-version":"6.8.2","content-type":"application/json","accept":"*/*","origin":"https://es-3ktojklt.kibana.tencentelasticsearch.com:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://es-3ktojklt.kibana.tencentelasticsearch.com:5601/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"zh-CN,zh;q=0.9,en;q=0.8"},"remoteAddress":"10.0.130.254","userAgent":"10.0.130.254","referer":"https://es-3ktojklt.kibana.tencentelasticsearch.com:5601/app/kibana"},"res":{"statusCode":500,"responseTime":4695,"contentLength":9},"message":"POST /api/reporting/generate/csv?jobParams=(conflictedTypesFields%3A!(kfext%2Ckfuin%2CrequestId)%2Cfields%3A!('%40timestamp'%2Ctext)%2CindexPatternId%3A'21fe4820-8916-11ea-8b39-a19e11c4dfcb'%2CmetaFields%3A!(_source%2C_id%2C_type%2C_index%2C_score)%2CsearchRequest%3A(body%3A(_source%3A(excludes%3A!()%2Cincludes%3A!('%40timestamp'%2Ctext))%2Cdocvalue_fields%3A!()%2Cquery%3A(bool%3A(filter%3A!()%2Cmust%3A!((query_string%3A(analyze_wildcard%3A!t%2Cdefault_field%3A'*'%2Cquery%3A'%22high%20risky%20with%20req%22'))%2C(range%3A('%40timestamp'%3A(format%3Aepoch_millis%2Cgte%3A1635310298396%2Clte%3A1635313898396))))%2Cmust_not%3A!()%2Cshould%3A!()))%2Cscript_fields%3A()%2Csort%3A!(('%40timestamp'%3A(order%3Adesc%2Cunmapped_type%3Aboolean)))%2Cstored_fields%3A!('%40timestamp'%2Ctext)%2Cversion%3A!t)%2Cindex%3A'account-admin-ol-*')%2Ctitle%3A'high%20risky%20with%20req'%2Ctype%3Asearch) 500 4695ms - 9.0B"}

The core error is ：

[.reporting-2021.10.24] as the final mapping would have more than 1 type: [esqueue, doc]

Version of the problem ？

Why is there such a problem , This failure of the system index is nothing more than kibana And es Caused by inconsistent versions of ,check For a moment ：

[[email protected]_130_254_centos /usr/local/service/kibana]# more version.md 
6.8.2.2019121001
[[email protected]_130_254_centos /usr/local/service/kibana]# cur localhost:9200
{
  "name" : "1620648141000429932",
  "cluster_name" : "es-3ktojklt",
  "cluster_uuid" : "zH1tb_eUS5uHJf5edamMAg",
  "version" : {
    "number" : "6.8.2",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "f1ae577",
    "build_date" : "2019-11-25T13:31:48.079152Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

The versions are exactly the same , It's not about the version , Rule out the possibility .

Two 、 analysis Elasticsearch journal

Here we are. , The big probability is mapping The problem. , But generally who will change the system index mapping Well , Here I suspect there is a custom template mapping Interfering with the system index .

Search the system index in the log ：

Sure enough, I found something unusual ：

[.reporting-2021.10.24] creating index, cause [auto(bulk api)], templates [qidian_default, [email protected], qd-template, outerBoss-template, hand-nginx-template, hand-template, *, beeflow-java-template, zhiku-template, beeflow-template, test-template, $zhiku-template]

The creation of a system index , There are so many custom templates matching , There must be a problem .

Solution

These user-defined templates that affect the system index are temporarily voided , From the original ：

  "index_patterns": [
    "*"
  ]

Change it to ：

  "index_patterns": [
    "xxx*"
  ]

Then delete the system report index , Generate the report again , Then it can be executed normally ：

Problem solving .

Summary

The business is in normal use , You can customize the template to match the actual business index , There's nothing wrong with this in itself . But remember, you can't do it for convenience , All match * , This operation is very dangerous , There will be hidden dangers .