How does Filebeat maintain file state?

Author: rab

---

1. Question

There is such a problem in actual testing/production. If the output target Kafka, ES, Kibana cannot be connected (accessed), how does Filebeat keep the file state?Is the file content successfully sent to the output?

2. Answer ️

The answer is yes, Filebeat will save the state of each file, it will flush the state in the registry file to disk, which records the harvester reads the last offset of the file, if output like Elasticsearch or Logstash is not accessible at this point, Filebeat keeps track of the last few lines sent, and continues reading the file when the output is available again, thus ensuring that all log lines are sent.

While Filebeat is running, state information for each input is also kept in memory.When Filebeat restarts, the data from the registry file is used to rebuild the state, and Filebeat continues each harvester at the last known location (ie, continues to collect the current latest data and output to the output destination).

That is, only Filebeat successfully sends the log content to the output destination (Kafka, ES, Kibana) from harvester until the output confirms that it has received the event, the next round of harvester will be opened.If you close Filebeat in the middle of sending events, it will not wait for the output to confirm that all events have been received before closing, any events sent to the output but not acknowledged before Filebeat shuts down, inThe is sent again when Filebeat restarts.This ensures that each event is sent at least once, but may end up sending duplicate events to the output.We can do this by setting shutdown_timeoutThe option configures Filebeat to wait a specific amount of time before shutting down.

For more questions, please refer to Official Documentation

---