Loki learning summary (1) -- the best choice of Loki small and medium-sized project log system

Preface

The project has been completed formally , The log system is indispensable , At present, most log platforms are recommended to be based on ELK structure , however ELK It's quite heavy , Schema too large , Small and medium-sized projects are not very good Hold live , I hope to find a simpler one , If you can't find it again ELK, At this time, a product called  Loki  The log system of was born .Loki yes Grafana Labs The team's latest open source project , It's a level of scalability , High availability , Multi tenant log aggregation system . It's designed to be very cost-effective and easy to operate , Because it does not index log content , Instead, label each log stream . Project subject Prometheus inspire , The official introduction is ：Like Prometheus, but for logs, Be similar to Prometheus The log system of .

Loki Log system

One 、Loki Quick start

Loki As a rising star of the log system , The design is excellent , The design idea is to make log aggregation simpler , It is designed to be very cost-effective and easy to operate . It doesn't index the contents of the log , Instead, set a set of labels for each log stream . It mainly consists of three parts ：

• Promtail  Is the log collector , Collect application logs and send them to  Loki.

• Loki  Used for log storage and parsing , And provide query API Show downstream .

• Grafana  Responsible for Loki Log visualization of .

Loki flow chart

Compared with other log aggregation systems ,Loki It has the following characteristics ：

• Do not full-text index logs . Compress unstructured logs and index only metadata through storage ,Loki It will be easier to operate , More cost saving .
• By using and Prometheus The same label record stream indexes and groups logs , This makes log expansion and operation more efficient .
• Especially suitable for storage Kubernetes Pod journal ; Such as Pod Metadata such as tags are automatically deleted and indexed .

Simple to fit

Loki install

The first is installation , Run now demo I prefer fast and convenient Docker. Here's what I modified Docker Compose Script , According to your own needs, you can start with one click Loki.

version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:2.2.1
    container_name: loki-service
    volumes:
#     take loki The configuration file of is mounted locally  c:/docker/loki  Catalog 
      - c:/docker/loki:/etc/loki/
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/loki.yml
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.2.1
    container_name: promtail-service
    volumes:  
    #   To read the local log directory , This is a default configuration for running , Production is definitely not like this .   
      - c:/docker/log:/var/log/
    #  promtail   The configuration file of is also mounted locally  c:/docker/promtail Catalog   
      - c:/docker/promtail:/etc/promtail/
    command: -config.file=/etc/promtail/promtail.yml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    container_name: grafana-service
    ports:
      - "3000:3000"
    networks:
      - loki

Mount directory above  c:/docker/loki  and  c:/docker/promtail Sure Modify according to your own situation .

Loki To configure

In the file above  -config.file=/etc/loki/loki.yml  yes  Loki  Configuration file for , We need to put the configuration file  loki.yml  Put it in advance  c:/docker/loki  Next , I use the default configuration ：

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h       # Any chunk not receiving new logs in this time will be flushed
  max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h
  chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
  chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
  max_transfer_retries: 0     # Chunk transfers disabled

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

compactor:
  working_directory: /loki/boltdb-shipper-compactor
  shared_store: filesystem

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

ruler:
  storage:
    type: local
    local:
      directory: /loki/rules
  rule_path: /loki/rules-temp
  alertmanager_url: http://localhost:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true

Promtail Configuration of

and  Loki  similar ,Promtail  It should also be mounted locally  c:/docker/promtail Directory configuration  promtail.yml, The default configuration is also used here ：

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          #  This has something to do with the mounting location , You can guess 
          __path__: /var/log/*log

I guess  /var/log/*log  Is the location where the log is read , So I attached it locally  c:/docker/log, Wait, get some logs to the local directory , See if you can read it out .

start-up Loki

Execute after configuration  docker-compose -f <docker-compose.yml route > up  command , First download the image and then start three Docker Containers . Open after success  http://localhost:3000/  Sign in  Grafana, The default account password is  admin/admin. Then add the data source in the sidebar as  Loki. 

  And then click Log labels You can display the log labels collected by the current system , You can filter and query logs according to these labels

Then configure Loki Of URL by  http://loki:3100, Then click OK and test , A green prompt means success .

  We use  Docker Compose, therefore  hostname  It's the service name  loki. Then click on a compass shaped icon in the sidebar  Explore, Just enter the log UI 了 , There is nothing at this time .

  I have to keep a log , To get a Spring Boot application , And then in  application.yml  Configure log options in , Then start the application to generate some logs .

logging:
  file:
  #  Get suspected Promtail Try the log read path of 
    path: c:/docker/log
  level:
    org: debug

Then I input an expression to query the log found in the document （Loki query）{filename="/var/log/spring.log"}, File name goes to  c:/docker/log  See if there is a log .

Loki Query and display in Spring Boot journal

Promtail Logging agent

at present Promtail Logs can be traced from two sources ： Local log files and systemd journal , What we demonstrated above is the loading of local log files , This is the only way I can use it right now , Another way is through K8S Service discovery capability of .

Loki flow chart

If you deploy multiple applications on multiple application servers Promtail The daemons can monitor the static log files of multiple applications , And pass Loki API Push logs to Loki Aggregate management in .

Spring Boot Deployment topology of logs

Promtail Dynamic configuration

We just need to Loki Application deployment related Promtail The daemons can . Here I still use Docker Yes Promtail Deployment , But I can't use the default configuration anymore , At this moment ​​ config.yml​​ Should be ：

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/log/positions.yaml

client:
  url: http://${LOKI_HOST}:${LOKI_PORT}/loki/api/v1/push

scrape_configs:
  - job_name: system
    pipeline_stages:
    static_configs:
      - labels:
          app: ${APP_NAME}
          job: varlogs
          host: ${LOG_HOST}
          __path__: /var/log/*log

To build a common configuration , I've made some parameters dynamic . This is a ​​Loki2.1+​​​ Features provided by version , have access to ​​ ${} To reference environment variables , You can even specify a default value for it ​​ ${VAR:default_value}​​. But you have to know that in order to turn on this feature, you need to Promtail Add options to the startup command ​​ -config.expand-env​​.

Promtail Docker Mirror transformation

Based on this, I'm right Promtail Of Docker The mirror image has been transformed , Concrete ​​Dockerfile​​ by ：

FROM grafana/promtail:2.2.1
LABEL AUTHOR = felord.cn
VOLUME ["/var/log/"]
EXPOSE 9080
ENV LOKI_HOST="localhost"
ENV LOKI_PORT=3100
ENV APP_NAME="APP"
ENV LOG_HOST="localhost"
COPY config.yml /etc/promtail/
CMD ["-config.file=/etc/promtail/config.yml", "-config.expand-env"]

You can go through ​​ docker build -t loki-promtail:1.0 .​​ Command to build this custom Promtail Mirror image . Basic startup commands ：

docker run -d  --name promtail-service --network loki -v c:/docker/log:/var/log/  -e LOKI_HOST=loki -e APP_NAME=SpringBoot  loki-promtail:1.0

The directory attached to it ​​ c:/docker/log​​​ It's still the log directory of the application ,​​LOKI_HOST ​​ Be sure to be able to work with Loki Server communication , Whether you're through direct link or Docker The Internet ( This is used here. Docker bridge ). You can use Docker Compose Will apply and Promtail To bind , be-all Promtail The corresponding log will be sent to Loki Centralized management . In addition, through custom ​​Label​​ We can search the log by application name .

Two 、Loki grammar

Selectors

For the label part of a query expression , Wrap it in curly brackets {}, Then use the syntax of key value pairs to select tags , Multiple label expressions separated by commas , such as ：

|=： The log line contains a string
!=： The log line does not contain a string
|~： Log lines match regular expressions
!~： The log line does not match the regular expression

1 #  Exactly match ：|="2020-11-16 "
2 {app_kubernetes_io_instance="admin-service-test2-container-provider"}|="2020-11-16 "

1 #  Fuzzy matching ：|~"2020-11-16 "
2 {app_kubernetes_io_instance="admin-service-test2-container-provider"}|~"2020-11-16 "

1 #  Exclude filtering ：!=/!~ " Data Center "
2 {app_kubernetes_io_instance="admin-service-master-container-provider"}!=" Data Center "
3 {app_kubernetes_io_instance="admin-service-master-container-provider"}!~" Data Center "

1 #  Regular matching ： |~ "()"
2 {app_kubernetes_io_instance="admin-service-master-container-provider"}!~"(admin|web)"
3 {app_kubernetes_io_instance="admin-service-master-container-provider"}|~"ERROR|error"

3、 ... and 、Loki Error log view

First, query the error log through the expression

Then view the exception stack information according to the context

The exception information is as follows

  If there are not enough rows , You can click on the Load 10 more, One click will add 10 That's ok , The left side will show Found 20 rows, Historical query uses , Select history query record , Inquire about , Default hold 7 Day query record

  Use the split screen function , Query different logs according to different tag selectors  

According to the label selector , Automatically refresh logs

Four 、 Range queries

• rate： Count the number of entries per second
• count_over_time： Calculates the entries for each log stream within a given range

1 #  Thirty minute log line records 
2 count_over_time({app_kubernetes_io_instance="admin-service-master-container-web"}[30m]) 
3
4 # 12h The rate of errors in hours 
5 rate({app_kubernetes_io_instance=~".*master-container.*"} |~ "ERROR|error" [12h])

5、 ... and 、 Set operations

And vPromQL equally ,LogQL Support a subset of built-in aggregation operators , Can be used to aggregate elements of a single vector , This produces a new vector with fewer elements but with set values ：

• sum： Calculate the sum on the label
• min： Choose the least number of labels
• max： Select the maximum value above the label
• avg： Calculate the average value on the label
• stddev： Calculate the overall standard deviation on the label
• stdvar： Calculate the population standard deviation on the label
• count： Calculate the number of elements in the vector
• bottomk： Select the smallest by sample value k Elements
• topk： Select the largest by sample value k Elements

#  Statistics 1 The largest number of logs per hour 10 A service  
topk(10,sum(rate({app_kubernetes_io_instance=~".*master-container.*"}[60m])) by(container))

#  Count the recent 6 Error log count in hours 
sum(count_over_time({app_kubernetes_io_instance=~".*master-container.*"}|~"ERROR"[6h])) by (container)

6、 ... and 、Loki Url expression

URL as follows ：

 https://grafana-liwenliang.com/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Loki%22,%7B%22expr%22:%22%7Bapp_kubernetes_io_instance%3D~%5C%22user-service-test2-container-provider%5C%22%7D%7C~%5C%222020-11-05%5C%22%7C~%5C%22ERROR%5C%22%7C~%5C%22.com.test.scrm.%5C%22%22,%22maxLines%22:5000%7D%5D

analysis ：

• %7C Express |
• %5C%22 Express "
• Time ：now-1h alternative now-1min perhaps now-5min
• Project name ：user-service-test2-container-provider Can be replaced by .test2-container. perhaps event-service-test2-container-provider
• Query log ：2020-11-05 Can be replaced by 2020-11-04
• Delete a pipe %7C%5C%22ERROR%5C%22%7C%5C%22.com.dadi01.scrm.%5C%22 Delete this paragraph
• Finally, the generated link is pasted into the browser to access

URL encryption ：

The above can be done by url Encryption and decryption generate the final query url link , Enter the designated website ：https://www.sojson.com/encodeurl.html.

  In the above example, the decryption is as follows ：

 https://grafana-liwenliang.com/explore?orgId=1&left=["now-1h","now","Loki",{"expr":"{app_kubernetes_io_instance%3D~\"user-service-test2-container-provider\"}|~\"2020-11-05\"|~\"ERROR\"|~\".com.test.scrm.\"","maxLines":5000}]

According to the custom query statement ：

#  Query by date 
https://grafana-liwenliang.com/explore?orgId=1&left=["now-1h","now","Loki",{"expr":"{app_kubernetes_io_instance=~\"user-service-test2-container-provider\\"}|~\"2020-11-18\\"","maxLines":5000}]
#  encryption 
https://grafana-liwenliang.com/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Loki%22,%7B%22expr%22:%22%7Bapp_kubernetes_io_instance=~%5C%22user-service-test2-container-provider%5C%22%7D%7C~%5C%222020-11-18%5C%22%22,%22maxLines%22:5000%7D%5D

#  Query by service name 
https://grafana-liwenliang.com/explore?orgId=1&left=["now-1h","now","Loki",{"expr":"{app_kubernetes_io_instance=~\"data-service-test2-container-provider\\"}|~\"2020-11-18\\"","maxLines":5000}]
#  encryption 
https://grafana-liwenliang.com/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Loki%22,%7B%22expr%22:%22%7Bapp_kubernetes_io_instance=~%5C%22data-service-test2-container-provider%5C%22%7D%7C~%5C%222020-11-18%5C%22%22,%22maxLines%22:5000%7D%5D

#  Query according to the corresponding database 
https://grafana-liwenliang.com/explore?orgId=1&left=["now-1h","now","Loki",{"expr":"{app_kubernetes_io_instance=~\"data-service-test2-container-provider\\"}|~\"2020-11-18\\"|~\"databaseName:scrm_test\\"","maxLines":5000}]
##  encryption 
https://grafana-liwenliang.com/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Loki%22,%7B%22expr%22:%22%7Bapp_kubernetes_io_instance=~%5C%22data-service-test2-container-provider%5C%22%7D%7C~%5C%222020-11-18%5C%22%7C~%5C%22databaseName:scrm_test%5C%22%22,%22maxLines%22:5000%7D%5D

Encrypted url Paste directly into the browser to query

7、 ... and 、Loki api 

#  Search tags 
curl -G -s "http://localhost:3100/loki/api/v1/labels" | jq

curl -G -s  "http://localhost:3100/loki/api/v1/labels" | jq .data[]
"__name__"
"app"
"app_kubernetes_io_component"
"app_kubernetes_io_instance"
"app_kubernetes_io_managed_by"
"app_kubernetes_io_name"
"app_kubernetes_io_version"
"chart"
"component"
"container"
"controller_revision_hash"
"filename"
"helm_sh_chart"
"heritage"
"job"
"k8s_app"
"name"
"namespace"
"pod"
"pod_template_generation"
"pod_template_hash"
"release"
"releaseRevision"
"statefulset_kubernetes_io_pod_name"
"stream"
"task" 


#  Query the corresponding tag value according to the tag 
curl -G -s http://localhost:3100/loki/api/v1/label/<name>/values | jq

curl -G -s  "http://localhost:3100/loki/api/v1/label/app_kubernetes_io_instance/values" | jq .data[]
"admin-service-test2-container-provider"
"admin-service-test2-container-web"
"admin-service-uat-container-provider"
"admin-service-uat-container-web"
"data-service-test2-container-provider"
"data-service-uat-container-provider"
"domain-service-test2-container-provider"
"domain-service-uat-container-provider"
"equity-service-test2-container-provider"
"equity-service-uat-container-provider"
"event-service-test2-container-provider"
"event-service-uat-container-provider"
"gateway-service-test2-container-soul-bootstrap"
"gateway-service-uat-container-soul-bootstrap"
"job-admin-service-test2-container-executor"
"job-admin-service-test2-container-web"
"job-admin-service-uat-container-executor"
"job-admin-service-uat-container-web"
"kubernetes-dashboard"


#  Query the corresponding log according to the tag 
curl -G -s http://localhost:3100/loki/api/v1/query_range | jq

curl -G -s  "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query={app_kubernetes_io_instance=~".*test2-container.*"}|~"ERROR|error"' | jq .data.result | jq .[].values[0][1]
"2020-11-18 18:08:17.149 DEBUG org.apache.coyote.http11.Http11NioProtocol - Processing socket [[email protected]:java.nio.channels.SocketChannel[connected local=admin-service-test2-container-web-686d7c459d-fzc7d/ remote=/]] with status [ERROR]\n"
"2020-11-18 18:10:18.876 DEBUG io.lettuce.core.protocol.RedisStateMachine - Decoded LatencyMeteredCommand [type=GET, output=ValueOutput [output=[[email protected], error='null'], commandType=io.lettuce.core.protocol.AsyncCommand], empty stack: true\n"
"2020-11-18 18:07:03.967 DEBUG org.springframework.jdbc.datasource.DataSourceTransactionManager - Creating new transaction with n

Query log stream

Query the log at a certain point in time ：GET /loki/api/v1/query, Example ：

http://192.168.15.139:30100/loki/api/v1/query?direction=BACKWARD&limit=1000&query=sum(rate({job="anychatlog"}[2d]))

Parameter description ：

• query： Executes LogQL Inquire about
• limit: Maximum number of entries to return
• time: The evaluation time of the query is taken as a nanosecond Unix An era . The default is now .
• direction： Determine the sort order of logs . The value of support is forward or backward. The default is backward

Query the log of a certain time period ：GET /loki/api/v1/query_range, Example ：

http://192.168.15.139:30100/loki/api/v1/query_range?direction=BACKWARD&limit=1000&query={job="anychatlog"} &start=1641280408415000000&end=1641453208415000000&step=120

Parameter description ：

• query： Executes LogQL Inquire about
• limit: Maximum number of entries to return
• start: Start time of query , In nanoseconds Unix Era representation . The default is one hour ago .
• end: End time of query , In nanoseconds Unix Era representation . The default is now .
• step: With duration Format or floating-point seconds query resolution step .duration Means in the form of Of Prometheus Duration string [0-9]+[smhdwy]. for example ,5m Indicates that the duration is 5 minute . Default is based on start Dynamic values of and end. Only applicable to query types that generate matrix responses .
• interval： This parameter is experimental ; Refer to the instructions under steps and intervals . Only return （ Or greater than ） Specify the entries for the interval , It can be duration Format or floating point number . Only applicable to queries that generate stream responses .
• direction： Determine the sort order of logs . The value of support is forward or backward. The default is backward.

Search tags

Query tag list ：GET /loki/api/v1/labels

Parameters ：

• start: Start time of query , In nanoseconds Unix Era representation . The default is 6 Hours before .
• end: End time of query , In nanoseconds Unix Era representation . The default is now .

Example ： http://192.168.15.139:30100/loki/api/v1/labels

Query tag value ： Query the list of known values of a given tag within a given time span  GET /api/prom/label/<name>/values

Parameters ：

• name: GET /loki/api/v1/label The label in
• start: Start time of query , In nanoseconds Unix Era representation . The default is 6 Hours before .
• end: End time of query , In nanoseconds Unix Era representation . The default is now .

Example ：http://192.168.15.139:30100/loki/api/v1/label/job/values

Matches a list of specific tag sets

Returns a list of time series matching a specific tag set ：GET /loki/api/v1/series;POST /loki/api/v1/series

Parameters ：

• match[]=<series_selector>： Select the label of the log stream to return .match[] At least one parameter must be provided .
• start=: Start timestamp .
• end=: End timestamp .

Example ：http://192.168.15.139:30100/loki/api/v1/series?start=1640922291907000000&end=1641527091908000000&match[]={host="192.168.11.123"}

Delete log stream

explain ： need 2.3.0 Previous versions of Loki, And configure it according to the official documents

Delete log stream ：POST loki_addr/loki/api/admin/delete?match[]

Parameters ：

• match[]: Tag matcher , Used to identify the stream to be deleted from , At least one parameter must be provided
• <series_selector>: Query parameters
• start: Start time stamp
• end: End timestamp
• 204 The response indicates success .

Example ：http://192.168.15.139:30100/loki/api/admin/delete?match[]={job="anychatlog"}

List delete requests ：GET /loki/api/admin/delete

Example ：http://192.168.15.139:30100/loki/api/admin/delete

Cancel delete request ：POST /loki/api/admin/cancel_delete_request

Example ：http://192.168.15.139:30100/loki/api/admin/cancel_delete_request?request_id=dad569a8

Other commonly used API

GET /ready： When Loki When the ingestor is ready to receive flow , return HTTP 200. If in Kubernetes Up operation Loki/ready It can be used as a ready probe .

POST /flush： Flush all memory blocks held by the ingestor to the backup storage . Mainly used for local testing .

POST /ingester/flush_shutdown： Closing of the ingestor , In particular, it will always flush any memory blocks it holds . This helps to narrow down and enable WAL Your ingestor , We want to make sure the old WAL Directories are not isolated , Instead, refresh to the back end of our block .

GET /metrics： Open Prometheus indicators . of Exported indicator list , see also loki

GET /config：/config Expose the current configuration . Optional mode Query parameters can be used to modify the output . If it has this value , be diff Only the difference between the default configuration and the current configuration is returned . value defaults Return to the default configuration

GET /loki/api/v1/status/buildinfo： stay JSON The object... Exposes information in the build . Fields including version,revision,branch,buildDate,buildUser, and goVersion.

Be careful ：Loki http Api Official statement https://grafana.com/docs/loki/latest/api/.