Accidentally deleted the data file of Clickhouse, can it be restored?

I believe that for those who want to use it in the production environment clickhouse For children's shoes , We will definitely pay attention to the reliability of data storage , There are three common ways to achieve this ：

1. Bottom disk do RAID : This method is similar to CH It doesn't matter , It belongs to the protection mechanism at the hardware level
2. utilize CH Provided ReplicatedMergeTree The engine stores multiple copies , This is the focus of this article , Let's talk about it in detail later
3. Backup data regularly , When you need to restore, manually execute the command to import , This method is detailed in above

What will be described below is on the premise of making copies , Same old thing , Let's start the journey of exploration by asking and answering ~~

Q1: Go straight ahead , Let's say you accidentally deleted the data file , Can you recover ？

The answer must be yes , But the premise is that what I said above ReplicatedMergeTree The engine made a copy , And the copy data is not damaged .

Q2: Will it automatically recover ？ Still need to execute any command ？

forehead , I can only say that it is semi-automatic , Because if you accidentally delete one parts In the catalog bin file , The system will not detect it automatically , Unless you restart the service （ I think in most cases we are definitely unwilling to do so ..）. When will it be tested ？ When you again select When querying this table , The detection will be triggered immediately , for the first time select The query will throw you an exception , Prompt that the file corresponding to this block cannot be found , Here's the picture ：

If you query again, you won't report an error , Of course, you can't find the missing data in this local table , Isn't that a lie ？ I thought it could be restored automatically ？ Don't worry. , Because it's no use rushing , because CH Data loss has been detected in the background , It will throw the task of data recovery into a queue , But it will not immediately perform the tasks in the queue , But will wait for a period of time , I'm not sure how long it will take , My observation here is about ten minutes , When I checked the lost data again, I found that the data had been recovered . Do I have to wait so long ？ Can you trigger automatic recovery immediately ？ unfortunately , I haven't found the corresponding method , If any friend knows , Welcome to leave a message ~~

Q3: Then how do you know that there are tasks waiting in line behind the scenes ？

The two methods ：

1. One is to read the log /var/log/clickhouse-server/clickhouse-server.log, You can see it when you query again ReplicatedMergeTreePartCheckThread Log
2. Look at the system table system.replication_queue

Q4: Go a little deeper , How does it judge that my file is damaged ？

It must be inseparable from you zookeeper Brother ,zookeeper Metadata information of each fragment and replica is recorded in , The checking thread compares the local file with zk Whether the metadata on is consistent , If it's not consistent , Will put the previous parts Directory move to detached Under the table of contents , And will parts The directory name is prefixed broken, And write a background task to the queue , Tell it to get from the replica node . When this task is actually called , It will send a request to the replica node , Copy the lost data back from the remote .

There is zk Under the circumstances , In case zk I don't work anymore ？ Will still check , But this time it parts Directory move to detached After the catalog , The prefix added becomes unexpected, At this time, no background tasks will be submitted , There is no way to recover automatically .

summary

Let's summarize briefly , Three conditions for lost data recovery ：

1. This table uses ReplicatedMergeTree engine
2. zookeeper Normal work
3. The copy file was not damaged during this period

Meet the above three conditions , Your data will be more secure ~~