Kubernetes cluster builds efk log collection platform

One 、efk Introduce

1.efk brief introduction

Kubernetes Developed a Elasticsearch Add on components to achieve cluster log management . This is a Elasticsearch、Filebeat（ perhaps Fluentd） and Kibana The combination of .

2.Elasticsearch Introduce

①Elasticsearch brief introduction

Elasticsearch It's based on Apache Lucene Open source search and data analysis engine ,Elasticsearch Use Java Development , And use Lucene As its core, it implements all the functions of index and search .

②Elasticsearch Characteristics

1.Elasticsearch It's a real-time , A distributed , Scalable search engine .
2.Elasticsearch Allow full-text and structured search and analysis of logs .
3.Elasticsearch It's a search engine , Responsible for storing logs and providing query interface .
4.Elasticsearch It is usually used to index and search a large amount of log data , It can also be used to search many different kinds of documents .

3.Filebeat Introduce

①Filebeat brief introduction

Filebeat Is a lightweight delivery tool for forwarding and centralizing log data .Filebeat Monitor the log file or location you specify , Collect log events , And forward them to Elasticsearch or Logstash Index .

②Fluentd brief introduction

Fluentd It's an open source data collector , Through it, data can be collected and consumed in a unified way , Be able to use and understand data better .

③Fluentd effect

1. stay kubernetes Each node in the cluster is installed Fluentd.
2. By getting the container log file 、 Filter and transform log data
3. Pass data to Elasticsearch colony , Index and store it in the cluster

4. Kibana Introduce

Kibana Is an open source analysis and visualization platform , Designed for use with Elasticsearch Used together . adopt kibana You can search 、 View and interaction are stored in Elasticsearch Data in , Use a variety of charts 、 Tables and maps, etc ,Kibana Be able to analyze and visualize data .

5、efk The architecture of the figure

Two 、 Check local kubernetes State of the cluster

[[email protected]-master ~]# kubectl get nodes -owide
NAME         STATUS   ROLES                  AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION          CONTAINER-RUNTIME
k8s-master   Ready    control-plane,master   10d   v1.23.1   192.168.3.201   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   containerd://1.6.6
k8s-node01   Ready    <none>                 10d   v1.23.1   192.168.3.202   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   containerd://1.6.6
k8s-node02   Ready    <none>                 10d   v1.23.1   192.168.3.203   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   containerd://1.6.6

3、 ... and 、 Configure default storage

1. Check nfs

[[email protected]-master efk]# showmount -e 192.168.3.201
Export list for 192.168.3.201:
/nfs/data *

2. edit sc.yaml file

[[email protected]-master efk]# cat sc.yaml 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: nfs-storage
 annotations:
   storageclass.kubernetes.io/is-default-class: "true"
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
parameters:
 archiveOnDelete: "true"  ##  Delete pv When ,pv Do you want to back up your content 

---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: nfs-client-provisioner
 labels:
   app: nfs-client-provisioner
 # replace with namespace where provisioner is deployed
 namespace: default
spec:
 replicas: 1
 strategy:
   type: Recreate
 selector:
   matchLabels:
     app: nfs-client-provisioner
 template:
   metadata:
     labels:
       app: nfs-client-provisioner
   spec:
     serviceAccountName: nfs-client-provisioner
     containers:
       - name: nfs-client-provisioner
         image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/nfs-subdir-external-provisioner:v4.0.2
         # resources:
         # limits:
         # cpu: 10m
         # requests:
         # cpu: 10m
         volumeMounts:
           - name: nfs-client-root
             mountPath: /persistentvolumes
         env:
           - name: PROVISIONER_NAME
             value: k8s-sigs.io/nfs-subdir-external-provisioner
           - name: NFS_SERVER
             value: 192.168.3.201 ##  Specify yourself nfs Server address 
           - name: NFS_PATH  
             value: /nfs/data  ## nfs Server shared directory 
     volumes:
       - name: nfs-client-root
         nfs:
           server: 192.168.3.201
           path: /nfs/data
---
apiVersion: v1
kind: ServiceAccount
metadata:
 name: nfs-client-provisioner
 # replace with namespace where provisioner is deployed
 namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: nfs-client-provisioner-runner
rules:
 - apiGroups: [""]
   resources: ["nodes"]
   verbs: ["get", "list", "watch"]
 - apiGroups: [""]
   resources: ["persistentvolumes"]
   verbs: ["get", "list", "watch", "create", "delete"]
 - apiGroups: [""]
   resources: ["persistentvolumeclaims"]
   verbs: ["get", "list", "watch", "update"]
 - apiGroups: ["storage.k8s.io"]
   resources: ["storageclasses"]
   verbs: ["get", "list", "watch"]
 - apiGroups: [""]
   resources: ["events"]
   verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: run-nfs-client-provisioner
subjects:
 - kind: ServiceAccount
   name: nfs-client-provisioner
   # replace with namespace where provisioner is deployed
   namespace: default
roleRef:
 kind: ClusterRole
 name: nfs-client-provisioner-runner
 apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: leader-locking-nfs-client-provisioner
 # replace with namespace where provisioner is deployed
 namespace: default
rules:
 - apiGroups: [""]
   resources: ["endpoints"]
   verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: leader-locking-nfs-client-provisioner
 # replace with namespace where provisioner is deployed
 namespace: default
subjects:
 - kind: ServiceAccount
   name: nfs-client-provisioner
   # replace with namespace where provisioner is deployed
   namespace: default
roleRef:
 kind: Role
 name: leader-locking-nfs-client-provisioner
 apiGroup: rbac.authorization.k8s.io

3. application sc.ymal file

[[email protected]-master efk]# kubectl apply -f sc.yaml 

4. Check sc relevant pod

[[email protected]-master efk]# kubectl get pods
NAME                                      READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-779b7f4dfd-zpqmt   1/1     Running   0          8s

5. test pv

① To write pv.yaml

[[email protected]-master efk]# cat pv.yaml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
 name: nginx-pvc
spec:
 accessModes:
   - ReadWriteMany
 resources:
   requests:
     storage: 200Mi

② function pv

kubectl apply -f pv.yaml

③ Check pv and pvc state

[[email protected]-master efk]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
pvc-939faa36-9c19-4fd9-adc9-cb30b270de75   200Mi      RWX            Delete           Bound    default/nginx-pvc   nfs-storage             40s
[[email protected]-master efk]# kubectl get pvc
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
nginx-pvc   Bound    pvc-939faa36-9c19-4fd9-adc9-cb30b270de75   200Mi      RWX            nfs-storage    44

Four 、 install helm Tools

1. download helm Binary package

wget https://get.helm.sh/helm-v3.9.0-linux-amd64.tar.gz

2. Unzip downloaded helm Compressed package

tar -xzf helm-v3.9.0-linux-amd64.tar.gz

3. Copy helm file

cp -a linux-amd64/helm /usr/bin/helm

4. see helm edition

[[email protected]-master addons]# helm version
version.BuildInfo{
    Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"clean", GoVersion:"go1.17.5"}

5、 ... and 、 To configure helm Warehouse

1. add to efk Of related components helm Source

[[email protected]-master ~]# helm repo add stable https://apphub.aliyuncs.com
"stable" has been added to your repositories
[[email protected]-master ~]# helm repo add elastic https://helm.elastic.co
"elastic" has been added to your repositories
[[email protected]-master ~]# helm repo add azure http://mirror.azure.cn/kubernetes/charts/
"azure" has been added to your repositories
[[email protected]-master ~]# 

2. see helm Warehouse

[[email protected]-master ~]# helm repo list 
NAME   	URL                                      
stable 	https://apphub.aliyuncs.com 
elastic	https://helm.elastic.co 
azure  	http://mirror.azure.cn/kubernetes/charts/

6、 ... and 、 install Elasticsearch

1. download Elasticsearch Of chart package

[[email protected]-master efk]# helm pull elastic/elasticsearch

2. decompression tar package

[[email protected]-master efk]#  tar -xzf elasticsearch-7.17.3.tgz

3. modify yaml file

① modify replicas

vim elasticsearch/values.yaml

replicas: 2
minimumMasterNodes: 1

esMajorVersion: ""

② Turn off persistent storage （ Optional ）

   ##
  persistence:
  enabled: false
  labels:
    # Add default labels for the volumeClaimTemplate of the StatefulSet
    enabled: false
  annotations: {
    }

4. install Elasticsearch application

 helm install elastic  elasticsearch

5. View run pod

[[email protected]-master efk]#  kubectl get pods
NAME                                      READY   STATUS    RESTARTS   AGE
cirror-28253                              1/1     Running   0          135m
elasticsearch-master-0                    1/1     Running   0          2m11s
elasticsearch-master-1                    1/1     Running   0          2m11s
nfs-client-provisioner-779b7f4dfd-p7xsz   1/1     Running   0          3h31m

7、 ... and 、 install filebeat

1. download filebeat

[[email protected]-master efk]# helm pull elastic/filebeat

2. decompression tar package

[[email protected]-master efk]# tar -xzf filebeat-7.17.3.tgz

3 see values.yaml file

[[email protected]-master filebeat]# cat values.yaml  -n
     1	---
     2	daemonset:
     3	  # Annotations to apply to the daemonset
     4	  annotations: {
    }
     5	  # additionals labels
     6	  labels: {
    }
     7	  affinity: {
    }
     8	  # Include the daemonset
     9	  enabled: true
    10	  # Extra environment variables for Filebeat container.
    11	  envFrom: []
    12	  # - configMapRef:
    13	  #     name: config-secret
    14	  extraEnvs: []
    15	  #  - name: MY_ENVIRONMENT_VAR
    16	  #    value: the_value_goes_here
    17	  extraVolumes:
    18	    []
    19	    # - name: extras
    20	    #   emptyDir: {
    }
    21	  extraVolumeMounts:
    22	    []
    23	    # - name: extras
    24	    #   mountPath: /usr/share/extras
    25	    #   readOnly: true
    26	  hostNetworking: false
    27	  # Allows you to add any config files in /usr/share/filebeat
    28	  # such as filebeat.yml for daemonset
    29	  filebeatConfig:
    30	    filebeat.yml: |
    31	      filebeat.inputs:
    32	      - type: container
    33	        paths:
    34	          - /var/log/containers/*.log 35 processors: 36 - add_kubernetes_metadata: 37 host: ${NODE_NAME} 38 matchers: 39 - logs_path: 40 logs_path: "/var/log/containers/" 41 42 output.elasticsearch: 43 host: '${NODE_NAME}' 44 hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}' 45 # Only used when updateStrategy is set to "RollingUpdate" 46 maxUnavailable: 1 47 nodeSelector: {} 48 # A list of secrets and their paths to mount inside the pod 49 # This is useful for mounting certificates for security other sensitive values 50 secretMounts: [] 51 # - name: filebeat-certificates 52 # secretName: filebeat-certificates 53 # path: /usr/share/filebeat/certs 54 # Various pod security context settings. Bear in mind that many of these have an impact on Filebeat functioning properly. 55 # 56 # - User that the container will execute as. Typically necessary to run as root (0) in order to properly collect host container logs. 57 # - Whether to execute the Filebeat containers as privileged containers. Typically not necessarily unless running within environments such as OpenShift. 58 securityContext: 59 runAsUser: 0 60 privileged: false 61 resources: 62 requests: 63 cpu: "100m" 64 memory: "100Mi" 65 limits: 66 cpu: "1000m" 67 memory: "200Mi" 68 tolerations: [] 69 70 deployment: 71 # Annotations to apply to the deployment 72 annotations: {} 73 # additionals labels 74 labels: {} 75 affinity: {} 76 # Include the deployment 77 enabled: false 78 # Extra environment variables for Filebeat container. 79 envFrom: [] 80 # - configMapRef: 81 # name: config-secret 82 extraEnvs: [] 83 # - name: MY_ENVIRONMENT_VAR 84 # value: the_value_goes_here 85 # Allows you to add any config files in /usr/share/filebeat 86 extraVolumes: [] 87 # - name: extras 88 # emptyDir: {} 89 extraVolumeMounts: [] 90 # - name: extras 91 # mountPath: /usr/share/extras 92 # readOnly: true 93 # such as filebeat.yml for deployment 94 filebeatConfig: 95 filebeat.yml: | 96 filebeat.inputs: 97 - type: tcp 98 max_message_size: 10MiB 99 host: "localhost:9000" 100 101 output.elasticsearch: 102 host: '${NODE_NAME}' 103 hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}' 104 nodeSelector: {} 105 # A list of secrets and their paths to mount inside the pod 106 # This is useful for mounting certificates for security other sensitive values 107 secretMounts: [] 108 # - name: filebeat-certificates 109 # secretName: filebeat-certificates 110 # path: /usr/share/filebeat/certs 111 # 112 # - User that the container will execute as. 113 # Not necessary to run as root (0) as the Filebeat Deployment use cases do not need access to Kubernetes Node internals 114 # - Typically not necessarily unless running within environments such as OpenShift. 115 securityContext: 116 runAsUser: 0 117 privileged: false 118 resources: 119 requests: 120 cpu: "100m" 121 memory: "100Mi" 122 limits: 123 cpu: "1000m" 124 memory: "200Mi" 125 tolerations: [] 126 127 # Replicas being used for the filebeat deployment 128 replicas: 1 129 130 extraContainers: "" 131 # - name: dummy-init 132 # image: busybox 133 # command: ['echo', 'hey'] 134 135 extraInitContainers: [] 136 # - name: dummy-init 137 138 # Root directory where Filebeat will write data to in order to persist registry data across pod restarts (file position and other metadata). 139 hostPathRoot: /var/lib 140 141 dnsConfig: {} 142 # options: 143 # - name: ndots 144 # value: "2" 145 hostAliases: [] 146 #- ip: "127.0.0.1" 147 # hostnames: 148 # - "foo.local" 149 # - "bar.local" 150 image: "docker.elastic.co/beats/filebeat" 151 imageTag: "7.17.3" 152 imagePullPolicy: "IfNotPresent" 153 imagePullSecrets: [] 154 155 livenessProbe: 156 exec: 157 command: 158 - sh 159 - -c 160 - | 161 #!/usr/bin/env bash -e 162 curl --fail 127.0.0.1:5066 163 failureThreshold: 3 164 initialDelaySeconds: 10 165 periodSeconds: 10 166 timeoutSeconds: 5 167 168 readinessProbe: 169 exec: 170 command: 171 - sh 172 - -c 173 - | 174 #!/usr/bin/env bash -e 175 filebeat test output 176 failureThreshold: 3 177 initialDelaySeconds: 10 178 periodSeconds: 10 179 timeoutSeconds: 5 180 181 # Whether this chart should self-manage its service account, role, and associated role binding. 182 managedServiceAccount: true 183 184 clusterRoleRules: 185 - apiGroups: 186 - "" 187 resources: 188 - namespaces 189 - nodes 190 - pods 191 verbs: 192 - get 193 - list 194 - watch 195 - apiGroups: 196 - "apps" 197 resources: 198 - replicasets 199 verbs: 200 - get 201 - list 202 - watch 203 204 podAnnotations: 205 {} 206 # iam.amazonaws.com/role: es-cluster 207 208 # Custom service account override that the pod will use 209 serviceAccount: "" 210 211 # Annotations to add to the ServiceAccount that is created if the serviceAccount value isn't set. 212 serviceAccountAnnotations: 213 {} 214 # eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/k8s.clustername.namespace.serviceaccount 215 216 # How long to wait for Filebeat pods to stop gracefully 217 terminationGracePeriod: 30 218 # This is the PriorityClass settings as defined in 219 # https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass 220 priorityClassName: "" 221 222 updateStrategy: RollingUpdate 223 224 # Override various naming aspects of this chart 225 # Only edit these if you know what you're doing 226 nameOverride: "" 227 fullnameOverride: "" 228 229 # DEPRECATED 230 affinity: {} 231 envFrom: [] 232 extraEnvs: [] 233 extraVolumes: [] 234 extraVolumeMounts: [] 235 # Allows you to add any config files in /usr/share/filebeat 236 # such as filebeat.yml for both daemonset and deployment 237 filebeatConfig: {} 238 nodeSelector: {} 239 podSecurityContext: {} 240 resources: {} 241 secretMounts: [] 242 tolerations: [] 243 labels: {} 

4. install filebeat

[[email protected]-master efk]# helm install fb filebeat
NAME: fb
LAST DEPLOYED: Sun Jul  3 13:03:21 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Watch all containers come up.
  $ kubectl get pods --namespace=default -l app=fb-filebeat -w

5. see filebeat relevant pod

[[email protected]-master efk]# kubectl get pods
NAME                                      READY   STATUS    RESTARTS   AGE
cirror-28253                              1/1     Running   0          151m
elasticsearch-master-0                    1/1     Running   0          18m
elasticsearch-master-1                    1/1     Running   0          18m
fb-filebeat-8fhg7                         1/1     Running   0          5m17s
fb-filebeat-lj5p7                         1/1     Running   0          5m17s
nfs-client-provisioner-779b7f4dfd-p7xsz   1/1     Running   0          3h47m

8、 ... and 、 install metricbeat

1. download metricbeat

helm pull stable/metricbeat

2. decompression tar package

[[email protected]-master efk]# tar -xzf  metricbeat-1.7.1.tgz

3. install metricbeat

[[email protected]-master efk]# helm install metric metricbeat 

4. see metricbeat relevant pod

[[email protected]-master efk]# kubectl get pods 
NAME                                      READY   STATUS    RESTARTS   AGE
cirror-28253                              1/1     Running   0          3h26m
elasticsearch-master-0                    1/1     Running   0          73m
elasticsearch-master-1                    1/1     Running   0          73m
fb-filebeat-8fhg7                         1/1     Running   0          60m
fb-filebeat-lj5p7                         1/1     Running   0          60m
metric-metricbeat-4jbkk                   1/1     Running   0          22s
metric-metricbeat-5h5g5                   1/1     Running   0          22s
metric-metricbeat-758c5c674-ldgg4         1/1     Running   0          22s
metric-metricbeat-bdth2                   1/1     Running   0          22s
nfs-client-provisioner-779b7f4dfd-p7xsz   1/1     Running   0          4h42m

Nine 、 install kibana

1. Download and install kibana

helm pull elastic/kibana

2. decompression kibana Of tar package

tar -xzf kibana-7.17.3.tgz 

3. Modify the service type

[[email protected] kibana]# vim values.yaml

##
service:
  port: 80
  type: NodePort
  ## Specify the nodePort value for the LoadBalancer and NodePort service types.
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport

4. To configure Elasticsearch Address

## Properties for Elasticsearch
##
elasticsearch:
  hosts:
    -   elastic-elasticsearch-coordinating-only.default.svc.cluster.local
  # - elasticsearch-1
  # - elasticsearch-2
  port: 9200

5. install kibana

[[email protected]-master stable]# helm install kb kibana 

5. Check pod

[[email protected]-master efk]# kubectl get pods
NAME                                      READY   STATUS    RESTARTS        AGE
cirror-28253                              1/1     Running   1 (6m28s ago)   5h50m
elasticsearch-master-0                    1/1     Running   1 (6m24s ago)   3h37m
elasticsearch-master-1                    1/1     Running   1 (6m27s ago)   3h37m
fb-filebeat-8fhg7                         1/1     Running   1 (6m28s ago)   3h24m
fb-filebeat-lj5p7                         1/1     Running   1 (6m24s ago)   3h24m
kb-kibana-5c46dbc5dd-htw7n                1/1     Running   0               2m23s
metric-metricbeat-4jbkk                   1/1     Running   1 (6m41s ago)   145m
metric-metricbeat-5h5g5                   1/1     Running   1 (6m24s ago)   145m
metric-metricbeat-758c5c674-ldgg4         1/1     Running   1 (6m24s ago)   145m
metric-metricbeat-bdth2                   1/1     Running   1 (6m27s ago)   145m
nfs-client-provisioner-779b7f4dfd-p7xsz   1/1     Running   2 (4m40s ago)   7h7m

Ten 、 visit kibana Of web

1. see svc

[[email protected]-master efk]# kubectl get svc
NAME                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
elasticsearch-master            ClusterIP   10.96.73.127   <none>        9200/TCP,9300/TCP   3h38m
elasticsearch-master-headless   ClusterIP   None           <none>        9200/TCP,9300/TCP   3h38m
kb-kibana                       NodePort    10.102.85.68   <none>        5601:31372/TCP      3m4s
kubernetes                      ClusterIP   10.96.0.1      <none>        443/TCP             15

2. Sign in kibanna

Open the browser , visit ——http://192.168.3.202:31372/