Splunk manually synchronize search head

background ： One search head cluster down 了 , I'm going to move him out first , Then add it ：

Be careful ： On the other side shcluster node On the implementation ：

splunk remove shcluster-member -mgmt_uri https://abc:8089

splunk add shcluster-member -new_member_uri https://abc:8089

1: Check this node Of state :

Splunk show shcluster-status

last_conf_replication : Pending

2: check the log: /opt/splunk/var/log/splunk/splunkd.log

Error pulling configurations from the search head cluster captain; consider performing a destructive configuration resync on this search head cluster member.

According to the document , Execute manual synchronization command ：

Fix: Configuration updates that the cluster replicates - Splunk Documentation

splunk resync shcluster-replicated-config

3: Check node status:

The search head cluster node status:  up

Check kvstore log: normal ：

/opt/splunk/var/log/splunk/mong.log:

 2022-05-25T04:51:18.369Z I NETWORK  [conn2457] end connection 11.2.33.44:59596 (35 connections now open)

 2022-05-25T04:52:04.781Z I STORAGE  [repl writer worker 15] createCollection: [email protected][email protected]@4KTP3A5u36YQ2FgIR6 with provided UUID: 7dd9566e-2879-4c25-a06c-a7a3ffe4dad6

 2022-05-25T04:52:04.921Z I INDEX    [repl writer worker 3] build index on: [email protected][email protected]@4KTP3A5u36YQ2FgIR6 properties: { v: 2, unique: true, key: { _user: 1, _key: 1 }, name: "_UserAndKeyUniqueIndex", ns: "[email protected][email protected]@4KTP3A5u36YQ2FgIR6" }

 2022-05-25T04:52:04.921Z I INDEX    [repl writer worker 3]          building index using bulk method; build may temporarily use up to 500 megabytes of RAM

 2022-05-25T04:52:04.921Z I INDEX    [repl writer worker 3] build index done.  scanned 0 total records. 0 secs

 2022-05-25T04:52:06.377Z I ASIO     [NetworkInterfaceASIO-RS-0] Connecting to abc.cn:8191

 2022-05-25T04:52:06.390Z I ASIO     [NetworkInterfaceASIO-RS-0] Successfully connected to abc.cn:8191,