Troubleshoot Splunk kvstore "starting"

lately kvstore It's very unstable , Three of the same group at a time searh head appear "starting" The state of , When I use splunk show kvstore-status During the inspection .

Check log:

1: /opt/spunk/var/log/splunkd.log

at that time ： https://abc.com:8089 yes captain (splunk show shcluster-status)

05-25-2022 18:31:36.907 +0800 INFO  SHCRaftConsensus [845326 TcpChannelThread] - Rejecting AppendEntries RPC from https://abc.com:8089

 

05-25-2022 18:31:41.241 +0800 ERROR KVStorageProvider [848995 TcpChannelThread] - Failed to create client connection (Local)

05-25-2022 18:31:41.241 +0800 ERROR KVStoreIntrospection [848995 TcpChannelThread] - failed to get introspection data

05-25-2022 18:31:41.909 +0800 INFO  SHCRaftConsensus [845326 TcpChannelThread] - Rejecting AppendEntries RPC from https://abc.com:8089 would leave gap our Last logindex = 0 request 3213

2: /opt/splunkog/splunk/mongo.log/var/mongo.log

Be careful ：mongoDB Inside is GMT： Note that the time is xxxZ In the form of , This Beijing time is ：= This log GMT Time +12

2022-05-25T13:45:19.645Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to https://abc.com:8191 - HostUnreachable: Connection refused

3: kvstore Any change of state will restart , Restart ： There are : kv version The statement of , therefore ： Sure vi mongo.log Search for ： keyword ： “kv version ”. Search for .

4: To view the document ：

KV store troubleshooting tools - Splunk Documentation

 

starting The state of is to captain There was a problem connecting , So switch captain , Try to find that the problem has been solved , The specific reasons have not yet been explained log Find out ,FW It's all open , It may be the original caption Insufficient resources for , As a result of .

In the face of node Of kvstore When operating ：

step1: stop splunk

step2: splunk clean kvstore --local

step3: splunk start

step4: splunk show kvstore-status.

When performing the above operations , You can open a session: tail -f /opt/splunk/var/log/splunk/mongo.log see kv-store The connection of .

Reference resources ：

KV store troubleshooting tools - Splunk Documentation