Introduction to victoriametrics

VictoriaMetrics(VM)  It supports high availability 、 Cost effective and scalable monitoring solutions and time series databases , Can be used for Prometheus Long term remote storage of monitoring data .

In the previous chapter, we introduced Thanos Solutions can also be used to solve Prometheus The problem of high availability and remote storage , So why do we use it VictoriaMetrics Well ？ be relative to Thanos,VictoriaMetrics It is mainly a horizontally expandable Local full persistent storage scheme ( Performance ratio thanos Better performance ,thanos Not local full , Many of its historical data are stored in object storage , If you want to query historical data, you should pull it from the object storage , This is definitely slower than getting data locally ),VictoriaMetrics Not just time series databases , Its advantages are mainly reflected in the following points .

• External support Prometheus dependent API, Can be used directly Grafana As Prometheus Data source use
• Index data ingestion and query have high performance and good scalability , Performance ratio InfluxDB and TimescaleDB Higher than 20 times
• When dealing with high cardinality time series , Memory is also optimized , Than InfluxDB Less 10x times , Than Prometheus、Thanos or Cortex Less 7 times
• High performance data compression , And TimescaleDB comparison , Up to 70 Times of data points are stored in limited storage space , And Prometheus、Thanos or Cortex comparison , Less storage space required 7 times
• It is aimed at high latency IO And low IOPS Optimized storage for
• Provide a global query view , Multiple Prometheus Instance or any other data source may ingest data into VictoriaMetrics

• It's easy to operate

• VictoriaMetrics It consists of a small executable file without external dependencies

• All configurations are completed through clear command line flags and reasonable default values
• All data is stored in - storageDataPath In the directory pointed to by the command line parameter

• have access to  vmbackup/vmrestore  Tools can easily and quickly backup from real-time snapshots to S3 or GCS Object store

• Support obtaining data sources from third-party time series databases

• Due to the storage architecture , It can protect the storage from abnormal shutdown （ namely OOM、 Hardware reset or kill -9） Free from data corruption
• Also supporting indicators relabel operation

 

 

 

framework

---

VM It is divided into two schemes: single node and cluster , Select according to business requirements . The single node version directly runs a binary file , Officials suggest collecting data points (data points) lower than 100w/s, recommend VM Single node version , Simple and easy to maintain , But it does not support alarms . The cluster version supports data horizontal splitting . The picture below is  VictoriaMetrics  The official architecture diagram of the cluster version .

Here is the client , such as grafana, It queries our data through load balancing ,vmselect Is a stateless , If there is pressure, you can expand at will , It gets data from storage , And then go back .

vmstorge It's stateful , It is the only stateful component in the whole cluster , All time series data are stored in this vmstorge Inside .

vminsert It is also a stateless , So you can also expand horizontally , It inserts data into vmstorge Inside , Can pass Prometheus The remote write interface writes data to vminsert.

So it is divided into two ends , One end is vminsert Writing data , One end is vmselect Look up the data .

The source of writing data is from Prometheus, Other databases are also supported .

It mainly includes the following components ：

• vmstorage： Data storage and query result return , The default port is 8482
• vminsert： Data entry , Similar fragmentation can be achieved 、 Copy function , Default port 8480（ You can insert data slices into multiple storage Among the examples, go to , There is a set of algorithm for how to insert ）
• vmselect： Data query , Summary and data De duplication , Default port 8481
• vmagent： Data index capture , Support for a variety of back-end storage , It will occupy the local disk cache , Default port 8429
• vmalert： Alarm related components , No, if you don't need the alarm function, you can not use this component , The default port is 8880

The cluster scheme divides the functions into vmstorage、 vminsert、vmselect Components , If you want to replace Prometheus, You also need to use vmagent、vmalert（ If you don't want to use it at all Prometheus, Then you can use vmagent,

）. As can be seen from the figure above vminsert as well as vmselect（ almost ） It's all stateless , So the extension is very simple , Only vmstorage It's stateful .

vmagent characteristic

vmagent Compared with Prometheus It has more flexibility in capturing indicators , For example, in addition to pulling （pull） Indicators can also be pushed （push） indicators , There are many other features ：

• Can replace prometheus Of scraping target
• Support from the Kafka Read and write data
• Support based on prometheus relabeling Mode add 、 remove 、 modify labels, You can filter data before sending it to remote storage
• Support multiple data protocols ,influx line agreement ,graphite Text protocol ,opentsdb agreement ,prometheus remote write agreement ,json lines agreement ,csv Data etc.
• Support data collection at the same time , And replicate to a variety of remote storage systems
• Support unreliable remote storage , If remote storage is not available , The indicators collected will be in  -remoteWrite.tmpDataPath  buffer , Once the connection to remote storage is repaired , Buffered indicators will be sent to remote storage , The maximum disk usage of the buffer can be  -remoteWrite.maxDiskUsagePerURL  To limit .
• comparison prometheus Use less memory 、cpu、 disk io And network bandwidth
• When you need to grab a large number of targets , The capture target can be dispersed to multiple vmagent In the example
• High cardinality and high churn rates can be addressed by limiting the number of unique time series before grabbing time and sending it to remote storage systems
• You can load from multiple files scrape To configure

Supported functions and Prometheus almost , Can read Prometheus Configuration of , Also support rebleing, Support adding global labels , Persistent data to data disk .

It also supports protocols written remotely , writes vm among .

Outside, we can take the initiative to capture our indicators . 

If you are building your monitoring system from scratch , In fact, you can not use it from the beginning Prometheus, You can use it directly vm, But the alarm function is a little weak . If vmalter Finish this , No at all Prometheus What's the matter , Because in memory and storage , It is better than Prometheus.