Logstash - logstash pushes data to redis

redis

This article mainly introduces the use of redis As output The goal of

Configurable parameters

Field	Parameter type	explain
batch	boolean	When it comes to true When Redis Batch values and send 1 individual RPUSH Command instead of sending one command per value to push to the list , requirement data_type=“list”
batch_events	number	When batch by true When , This parameter limits queued RPUSH Number of events
batch_timeout	number	When batch by true When , This parameter limits the timeout
congestion_interval	number	Time interval of blocking check
congestion_threshold	number	When data_type yes list When , And have more than congestion_threshold term , Is blocked , Until someone consumes it . If there are no consumers ,Redis Will run out of memory
data_type	string	Optional parameters [“list”, “channel”], In fact, that is Redis Queue type
db	number	Redis Database number
host	array	Host name , If the host list is an array ,Logstash A random host will be selected for connection , If the host is disconnected , Then another host will be selected .
key	string	Redis The name of the list or channel
password	password	Password used for authentication
port	number	Port number
reconnect_interval	number	Reconnection interval
shuffle_hosts	boolean
timeout	number	Timeout time

Queue type selection

data_type To make sure redis Queue type . The optional parameter is list and channel. The specific differences between the two can be seen in redis Do the introduction in the data source article (Logstash from Redis Collect data in and output )[https://blog.csdn.net/qq330983778/article/details/105757604]

Turn on batch processing

Currently only if data_type by list You can use batch processing . Batch related parameters batch、batch_events、batch_timeout.

batch

Whether to enable batch processing , Off by default

batch_events

How many pieces of data are pushed by a single command , The default is 50

batch_timeout

How long to wait for a single command , The default is 5s

By default , Once at a time rpush The command will send a piece of data . After starting batch processing , The amount of data sent by each command depends on batch_events, The sending time depends on batch_timeout. It depends on which of the two conditions is satisfied first .

logstash The protection mechanism of

Currently only if data_type by list You can use

congestion_threshold

logstash A parameter is provided congestion_threshold, Mainly limited redis The maximum number of elements that can exist in a data item in , When list After the element in is greater than the value set by this parameter , Will block until other consumers consume list Data in . This operation is mainly to prevent when there are no consumers ,list Too much data leads to Redis Run out of memory . This parameter defaults to 0, Indicates that blocking detection is disabled

congestion_interval

This parameter is used to set the frequency of blocking detection . The default is 1s. When set to 0 When , Indicates every execution rpush Then conduct a test .

Push data to redis Configuration of

Here's how to use channel perhaps list Configuration of

Use channel

input {
    
	redis {
    
		key => "logstash-redis"
		host => "localhost"
		password => "dailearn"
		port => 6379
		db => "0"
		data_type => "list"
		type  => "redis"
		codec => plain{
    
            	charset=>"UTF-8"
       	}
	}
}

filter {
    
	json {
    
		source => "message"
	}
}

output{
    
		redis {
    
            host => "localhost" # This is marked redis Address of service 
            port => 6379
            codec => plain
            db => 0 #redis Database in ,select The object of 
            key => "logstash-message"#redis Key value in 
            data_type => channel # It is commonly list and channel
            password => "dailearn"
        }
}

Use list Do batch processing

input {
    
	redis {
    
		key => "logstash-redis"
		host => "localhost"
		password => "dailearn"
		port => 6379
		db => "0"
		data_type => "list"
		type  => "redis"
		codec => plain{
    
            	charset=>"UTF-8"
       	}
	}
}

filter {
    
	json {
    
		source => "message"
	}
}

output{
    
		redis {
    
			batch => true
			batch_events => 50
			batch_timeout => 5
            host => "localhost" # This is marked redis Address of service 
            port => 6379
            codec => plain
            db => 0 #redis Database in ,select The object of 
            key => "logstash-message"#redis Key value in 
            data_type => list # It is commonly list and channel
            password => "dailearn"
        }
}

Use of blocking checks

Here's how to use list Configuration for batch processing and blocking check

input {
    
	redis {
    
		key => "logstash-redis"
		host => "localhost"
		password => "dailearn"
		port => 6379
		db => "0"
		data_type => "list"
		type  => "redis"
		codec => plain{
    
            	charset=>"UTF-8"
       	}
	}
}

filter {
    
	json {
    
		source => "message"
	}
}

output{
    
		redis {
    
			batch => true
			batch_events => 50
			batch_timeout => 5
            host => "localhost" # This is marked redis Address of service 
            port => 6379
            codec => plain
			congestion_threshold => 5
			congestion_interval => 1
            db => 0 #redis Database in ,select The object of 
            key => "logstash-message"#redis Key value in 
            data_type => list # It is commonly list and channel
            password => "dailearn"
        }
		
		stdout {
    codec => rubydebug {
    }}
}

Test blockage check

In the above configuration, the blocking number is set to 5(congestion_threshold => 5), So now to logstash Send data in sequence

set one test message.time：1590462390513
set one test message.time：1590462391833
set one test message.time：1590462393645
set one test message.time：1590462394727
set one test message.time：1590462396589
...

Although it is set in the configuration 5 Data is blocked , actually redis Six pieces of data were inserted .

When inserted 5 When you have data , Console output is normal , But when redis When there are six pieces of data in the console, the console will frequently refresh the exception information , And you will not be able to insert data again . Of course, we need to pay attention to , Redundant data is simply blocked and not discarded , Trying to consume redis After the data of , Blocked data will still be inserted into redis in

About the setting of inspection frequency

When put congestion_interval When the value of is set to a large value , It may cause the actual inserted data to exceed the limit value . For example, in the above example, data transmission in sequence is modified to one-time transmission , Would make redis The actual data in is much larger than the required data

Conflict between batch and inspection frequency

In the above example, multiple pieces of data are inserted in one inspection cycle , At this time, according to the document, we can set congestion_interval => 0 To check every event once , But in fact, if you simulate the above scenario again, you will still find redis Is inserted with data far beyond the setting . This is because in the previous configuration batch => true, In batch configuration , An event contains multiple pieces of data , So although each event is detected once at this time , But it will still insert more data than the limit .

---

Limited personal level , The above content may not be clearly described or wrong , If development students find , Please let me know in time , I will revise the relevant contents as soon as possible . If my article is of any help to you , Please give it to me Like it . Your praise is my driving force .