How to use etcd to realize distributed /etc directory

etcd yes Kubernetes The preferred data storage system in , The name comes from “/etc” and ”distributed“,“/etc” yes Linux Directory where configuration files are stored , but etcd It's a Key-Value Key value database , Cannot store file .

background

etcd It is a key value database with both consistency and high availability , Simple 、 Security 、 Fast 、 trusted , At present, it is Kubernetes Primary data storage . Let's start with a paragraph etcd Official explanation of the name .

The name “etcd” originated from two ideas, the unix “/etc” folder and “d"istributed systems. The “/etc” folder is a place to store configuration data for a single system whereas etcd stores configuration information for large scale distributed systems. Hence, a “d"istributed “/etc” is “etcd”.

The above paragraph comes from etcd Official website ,etcd Imaginatively put etc (Linux Where the system is usually used to store configuration files ) And distributed ( Distributed ) The two concepts are combined , However , because etcd adopt HTTP API Provide services , therefore ” regret ” We can't realize a real distributed /etc Catalog . Next we will introduce , How to use JuiceFS, help etcd Achieve a truly distributed /etc Catalog .

Then you can use it etcd Achieve true distribution /etc Well ？ The following methods can be tested .

We use an open source distributed file system JuiceFS for /etc Provide POSIX Access capability of file interface , and JuiceFS have access to etcd As Metadata engine , Store the directory tree in the file system 、 Metadata such as file name , coordination JuiceFS CSI Driver, It can be used as Persistent Volume, stay Kubernetes Multiple application instances share configuration files in the platform , This is exactly distributed /etc 了 .

The following will be from what is JuiceFS、 Why? JuiceFS It can be distributed /etc And how to realize distributed /etc And so on , about etcd How to use JuiceFS Realize the sharing of configuration files in multiple application instances .

What is? JuiceFS

JuiceFS Is an open source cloud native distributed file system , Designing for cloud environments , Provide complete POSIX、HDFS and S3 API Compatibility .JuiceFS The architecture of adopts a layered plug-in design , Metadata engines have been opened to Redis、MySQL、PostgreSQL、TiKV Such as database support , And in v1.0-beta3 Officially supported in version etcd As a metadata engine . The data storage engine interfaces with any object storage system , stay v1.0-rc1 China also supports etcd As a storage engine , It is suitable for configuration files with small storage capacity .

Why? JuiceFS It can be distributed /etc

According to the layered architecture design mentioned above, we can see ,JuiceFS Store the metadata of the file in the database , File data is stored in object storage , Thus, users can access the same tree file system structure on different nodes , And achieve the purpose of sharing files . With JuiceFS Blessing of distributed file system , We can put the configuration file in the file system , Then in each application JuiceFS Mount into it /etc Catalog , And then realize real distributed "/etc". The whole process is shown in the following figure .

How to achieve distributed /etc

Next, let's say nginx Application, for example , about etcd How to use JuiceFS, Make more than one nginx Instances share the same configuration , Implement distributed /etc.

Deploy etcd

stay Kubernetes Environment , It is suggested to build an independent etcd Service supply JuiceFS Use , Instead of using the default in the cluster etcd service , Avoid affecting when file system access pressure is high Kubernetes The stability of clusters .

install etcd You can refer to the official documents , Build multi node etcd colony ; You can also use Bitnami Provided etcd Of chart Installation package .

In case of data sensitivity , Can be opened etcd Encrypted communication function , Encrypted transmission of data . Reference resources etcd Provided by the project Sample script .

stay JuiceFS Prepare the configuration file in

Install well etcd After cluster , Two lines of command initialize a JuiceFS file system .

$ juicefs format etcd://$IP1:2379,$IP2:2379,$IP3:2379/jfs --storage etcd --bucket etcd://$IP1:2379,$IP2:2379,$IP3:2379/data pics$ juicefs mount etcd://$IP1:2379,$IP2:2379,$IP3:2379/jfs /mnt/jfs

take JuiceFS volume Mount to /mnt/jfs After the directory , You can directly place nginx.conf file .

stay Kubernetes Use in JuiceFS

First create a Secret, Configure... In it etcd Connection information ：

apiVersion: v1kind: Secretmetadata:  name: juicefs-secret  namespace: kube-systemtype: OpaquestringData:  name: test  metaurl: etcd://$IP1:2379,$IP2:2379,$IP3:2379/jfs  storage: etcd  bucket: etcd://$IP1:2379,$IP2:2379,$IP3:2379/data

Both metadata engine and object storage use etcd, among $IP1、$IP2、$IP3 by etcd The client of IP. Then create a PV and PVC（ You can refer to file ）. And then you can do it in Nginx Application , take PVC Mount to /etc, as follows ：

apiVersion: apps/v1kind: Deploymentmetadata:  name: nginx-appspec:    …    volumes:      - name: config        persistentVolumeClaim:          claimName: etcd    containers:      - image: nginx        volumeMounts:        - mountPath: /etc/nginx          name: config         …

Nginx The application will read when it starts /etc/nginx Under the nginx.conf The configuration file , That is, read what we put in JuiceFS Medium nginx.conf Configuration file , and nginx.conf The document is passed JuiceFS volume Shared .

summary

etcd As a cloud native key value database , Used to store Kubernetes Service configuration information in the platform , It has become the standard of fact . But the configuration file management of many upper applications is still inconvenient , This article shares JuiceFS How to make etcd Become distributed "/etc" Methods , help etcd Completed the original dream .

Activity Notice

It will be held this Saturday Meetup “ A hybrid cloud / Practice and Prospect of multi cloud storage architecture ”, welcome Click here to sign up .