Elk note 24 -- replace logstash consumption log with gohangout

Introduce

gohangout yes childe Big man use golang Imitative Logstash Open source project . For consumption Kafka data , Write after processing ES、Clickhouse etc. . be relative to elk Provided by the Chinese government logstash, It can be used more efficiently cpu resources 、 Reduce memory , Improve IO throughput .

I recently used logstash Process multiple in the project k8s Cluster logs , It is found that as the business volume increases , Needed logstash It takes up more resources , To reduce resources and improve log writing es The efficiency of , It was decided that logstash Replace with hangout. Effects to be used 、 Implementation method 、 Notes are recorded here , For those who need to learn .

Use gohangout

Use here gohangout consumption kafka journal , Then write it to es in ,gohangout Deployed in k8s colony .

Dockerfile

FROM alpine:3.15.4
RUN mkdir -p /home/
COPY . /home/
WORKDIR /home
RUN wget https://github.com/childe/gohangout/releases/download/v1.8.1/gohangout-linux-x64-91feead -O /home/gohangout && chmod u+x /home/gohangout
CMD sleep infinity

adopt docker build -t hangout:v1 . Packaging mirroring , And push it to the mirror warehouse ( The author tests for local k8s, Package the image directly on the node ).

Startup profile k8s-log.yaml

inputs:
  - Kafka:
      topic:
        k8s-log: 1
      codec: 'json'
      consumer_settings:
        bootstrap.servers: "192.168.2.11:9092"
        group.id: 'logstash-k8s-log'
        from.beginning: 'false'
        auto.commit.interval.ms: '5000'
        #auto.commit: 'false'

filters:
  - Date:
      src: time
      target: "@timestamp"
      formats:
        - 'ISO8601'
        - 'RFC3339'
      remove_fields: ["time"]

outputs:
  - Elasticsearch:
      es_version: 7
      hosts:
        - 'http://es_user:[email protected]_ip:9200'
      index: 'k8s_log_write' #golang The rendering method inside is to use numbers ,  Rather than using YYMM.
      index_type: 'logs'
      concurrent: 2
      bulk_actions: 36000
      bulk_size: 30
      flush_interval: 30

k8s deployment hangout-k8s-log.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: hangout-k8s-log
  name: hangout-k8s-log
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: hangout-k8s-log
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: hangout-k8s-log
    spec:
      containers:
        - command:
            - /home/gohangout
            - '-logtostderr'
            - '-v'
            - '5'
            - '--config'
            - k8s-log.yaml
          image: hangout:v1
          imagePullPolicy: IfNotPresent
          name: hangout-k8s-log
          resources:
            limits:
              cpu: '1'
              memory: 2000Mi
            requests:
              cpu: 100m
              memory: 200Mi

adopt kubectl apply -f hangout-k8s-log.yaml hold hangout Forces pull up , as follows ：

Use gohangout before and after comparison

In the second quarter Use gohangout It is a case simplified by the author , In order to further highlight the use effect , Here is a part of the actual project data , Convenient for comparison .

hangout and logstash contrast , Obviously hangout In a steady state 400MB The left and right memory can run stably , and logstash At least 1GB start ; CPU aspect ,hangout Utilization ratio is also higher than logstash high .

hangout	logstash

ES colony QPS： Adjusted for hangout after , colony QPS from 95K Rise rapidly to 160K.

kafka topic：topic Consumption flows from 6M Upgrade to 12.5M about , The consumption capacity has doubled .

Sum up CPU、 Memory 、ES QPS、kafka Consumption rate , Obviously hangout Performance ratio logstash superior .
Of course hangout There are also shortcomings , That is, it supports There are few plug-ins ,logstash The more complicated grok Matching is not necessarily easy to quickly convert to hangout Supported Syntax .
therefore , In the actual project, it is suggested that some large quantities , The single index of the parsing rule is adjusted to hangout( For example, pure json Class ).

matters needing attention

1. There will be multiple in the actual production environment topic and Multiple index, So you need to prepare multiple configurations , When packaging, try to create an image , And then in deployment Of command Pass through --config To specify different instances .

   /home/gohangout -logtostderr -v 5 --config k8s-log.yaml`
   

2. Deploy hangout perhaps logstash You can consider using deployment To deploy , Combined with HPA Dynamic scaling enables more efficient consumption kafka The log in .

explain

Software environment ：
kafka_2.12-3.1.1
k8s 1.23.6
gohangout v1.8.1
elk 7.15
Reference documents ：
github gohangout