Dragon lizard community open source coolbpf, BPF program development efficiency increased 100 times

writing / The system operational  SIG（Special Interest Group）

introduction

BPF  Is a new dynamic tracking technology , At present, this technology is profoundly affecting our production and life .BPF  It plays a great role in the four application scenarios ：

• System fault diagnosis ： It can dynamically insert piles to perspective the core .

• Network performance optimization ： It can modify and forward the received and sent network packets .

• System security ： It can monitor the opening and closing of files to make security decisions .

• Performance monitoring ： It can see how long the function takes to know the performance bottleneck .

BPF  Technology also follows  Linux  With the development of the kernel ,Linux  The kernel version has experienced  3.x  towards  4.x  To  5.x  evolution ,eBPF  Technical support is also from  4.x  Start to improve , especially  5.x  The kernel also adds many advanced features . But there are a lot of cloud servers  3.10  The kernel version does not support  eBPF  Of , In order to make our existing  eBPF  Tools can run in these stock machines , We transplanted  BPF  To the lower version kernel , At the same time, it is based on  libbpf  Of  CO-RE  Ability , Ensure that a tool can run in  3.x/4.x/5.x  It's low 、 in 、 High kernel version .

BPF  There are many ways to develop , At present, the popular ones are ：

1） pure  libbpf  application development ： With the help of  libbpf  Library loading  BPF  Program to kernel ： This development method is not only inefficient , There is no basic library encapsulation , All necessary steps and basic functions need to be explored by yourself .

2） With the help of  BCC And other open source projects ： High development efficiency 、 Good portability , And support dynamic modification of kernel code , Very flexible . But there are deployment dependencies  Clang/LLVM  Such as the library ;  Run every time  Clang/LLVM  compile , Heavy consumption  CPU、 Memory and other resources , Easy to compete with other services .

coolbpf  project , With  CO-RE（Compile Once-Run Everywhere） Based on , Low resource consumption is reserved 、 Strong portability and other advantages , It also merges  BCC  Dynamic compilation features , It is suitable for batch deployment of developed applications in production environment .coolbpf  Created a new idea , Using the idea of remote compilation , Put the user's BPF The program is pushed to the remote server and returned to the user .o or .so, Provide high-level languages such as  Python/Rust/Go/C  Wait for loading , Then run safely in the full kernel version . Users only need to focus on their own function development , Don't care about the underlying Library （ Such as  LLVM、python  etc. ） install 、 Environment building , To the general  BPF  Lovers provide a new exploration and practice .

One 、BPF  Comparison of development methods

BPF  Experienced the traditional  setsockopt  The way of  sock filter  Packet filtering , Up to now  libbpf CO-RE  Way to develop monitoring and diagnostic functions , Is and  eBPF  Excellent instruction set capability closely combined with hardware and  libbpf  The open source and openness of the general library are inseparable , Let's review  BPF  Development mode , And on this basis, based on the idea of remote compilation as the core  coolbpf, It stood on the shoulders of giants , Optimized resources 、 Simple programming and efficiency improvement .

1、 Original stage

stay  BPF  Also called Berkeley message filtering (cBPF) When , It passes through  sock filter  Will be original  BPF  Instruction code , utilize  setsockopt  Load to kernel , adopt  setsockopt  Load to kernel , By means of  packet_rcv  call  runfilter  Run this program to filter messages . This way, ,BPF  The generation of bytecode is very primitive , It is similar to writing assembler by hand , The process is very painful .

static struct sock_filter filter[6] = {
 { OP_LDH, 0, 0, 12 }, // ldh [12]
 { OP_JEQ, 0, 2, ETH_P_IP }, // jeq #0x800, L2, L5
 { OP_LDB, 0, 0, 23 }, // ldb [23]
 { OP_JEQ, 0, 1, IPPROTO_TCP }, // jeq #0x6, L4, L5
 { OP_RET, 0, 0, 0 }, // ret #0x0
 { OP_RET, 0, 0, -1, }, // ret #0xffffffff
};
int main(int argc, char **argv)
{
…
 struct sock_fprog prog = { 6, filter };
 …
 sock = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL));
 …
 if (setsockopt(sock, SOL_SOCKET, SO_ATTACH_FILTER, &prog, sizeof(prog))) {
 return 1;
 }
…
 }


2、 Conservative stage

Examples are  samples/bpf  Below  sockex1_kern.c  and  sockex1_user.c, The code is divided into two parts , Usually named  xxx_kern.c  and  xxx_user.c, The former is loaded into kernel space and executed , The latter is performed in user space .BPF  After the program is written, it passes  Clang/LLVM  Compile ,xxx_user.c  Explicitly load the generated  xxx_kernel.o  file . Although this method uses the compiler to support automatic generation  BPF  Bytecode , But code organization and  BPF  The loading method is conservative , Users need to write a lot of repetitive code .

struct {
 __uint(type, BPF_MAP_TYPE_ARRAY);
 __type(key, u32);
 __type(value, long);
 __uint(max_entries, 256);
} my_map SEC(".maps");

SEC("socket1")
int bpf_prog1(struct __sk_buff *skb)
{
 int index = load_byte(skb, ETH_HLEN + offsetof(struct iphdr, protocol));
 long *value;

 if (skb->pkt_type != PACKET_OUTGOING)
 return 0;

 value = bpf_map_lookup_elem(&my_map, &index);
 if (value)
 __sync_fetch_and_add(value, skb->len);

 return 0;
}
char _license[] SEC("license") = "GPL";

int main(int ac, char **argv)
{
 struct bpf_object *obj;
 struct bpf_program *prog;
 int map_fd, prog_fd;
 char filename[256];
 int i, sock, err;
 FILE *f;

 snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);

 obj = bpf_object__open_file(filename, NULL);
 if (libbpf_get_error(obj))
 return 1;

 prog = bpf_object__next_program(obj, NULL);
 bpf_program__set_type(prog, BPF_PROG_TYPE_SOCKET_FILTER);

 err = bpf_object__load(obj);
 if (err)
 return 1;

 prog_fd = bpf_program__fd(prog);
 map_fd = bpf_object__find_map_fd_by_name(obj, "my_map");
 ...
 }


3、BCC  Initial stage

BCC  The emergence of broke the conservative development method , Excellent runtime compilation and basic library encapsulation , It greatly reduces the difficulty of development , There are many fans , Then start to attack cities and land , Similar to the rapid expansion of capital . Users only need to be in  Python  In the program  attach  a section  prog , Then analyze and process the data , The disadvantage is that it must be installed in the production environment  Clang  and  python  library , There are  CPU  Resources rush up instantaneously , Cause loading  BPF  The possibility of non recurrence of problems after the program .

int trace_connect_v4_entry(struct pt_regs *ctx, struct sock *sk)
{
 if (container_should_be_filtered()) {
 return 0;
 }

 u64 pid = bpf_get_current_pid_tgid();
 ##FILTER_PID##
 u16 family = sk->__sk_common.skc_family;
 ##FILTER_FAMILY##

 // stash the sock ptr for lookup on return
 connectsock.update(&pid, &sk);

 return 0;
}

# initialize BPF
b = BPF(text=bpf_text)
if args.ipv4:
 b.attach_kprobe(event="tcp_v4_connect", fn_name="trace_connect_v4_entry")
 b.attach_kretprobe(event="tcp_v4_connect", fn_name="trace_connect_v4_return")
b.attach_kprobe(event="tcp_close", fn_name="trace_close_entry")
b.attach_kretprobe(event="inet_csk_accept", fn_name="trace_accept_return")


4、BCC  Advanced stage

BCC  Become fashionable for a time , Captured many developers . Because of the progress of the times , Demand is changing .libbpf  Born and  CO-RE  Ideas prevail ,BCC  I am also changing , Start with  BTF  The way supports relocation , I hope the same program can be used in any  Linux  The system can run smoothly . However , Some structures are on different kernel versions , Or the name of the member changes 、 Or the meaning of members has changed （ From microseconds to milliseconds ）, This way requires program processing . stay  4.x  Wait for the medium version of the kernel , It also needs to go through  debuginfo  Generative independent  BTF  file , The process is still quite complicated .

SEC("kprobe/inet_listen")
int BPF_KPROBE(inet_listen_entry, struct socket *sock, int backlog)
{
 __u64 pid_tgid = bpf_get_current_pid_tgid();
 __u32 pid = pid_tgid >> 32;
 __u32 tid = (__u32)pid_tgid;
 struct event event = {};

 if (target_pid && target_pid != pid)
 return 0;

 fill_event(&event, sock);
 event.pid = pid;
 event.backlog = backlog;
 bpf_map_update_elem(&values, &tid, &event, BPF_ANY);
 return 0;
}

#include "solisten.skel.h"
...
int main(int argc, char **argv)
{
 ...
 libbpf_set_strict_mode(LIBBPF_STRICT_ALL);
 libbpf_set_print(libbpf_print_fn);

 obj = solisten_bpf__open();
 obj->rodata->target_pid = target_pid;
 err = solisten_bpf__load(obj);
 err = solisten_bpf__attach(obj);
 pb = perf_buffer__new(bpf_map__fd(obj->maps.events), PERF_BUFFER_PAGES,
 handle_event, handle_lost_events, NULL, NULL);
 ...
}


5、 Resource sharing stage

BCC  Although it also supports  CO-RE, But there are still relatively fixed codes , Problems that cannot be dynamically configured , At the same time, you also need to build a compilation project .coolbpf  Put the compilation resources on a server , Provide remote compilation capability , We share remote server resources , Only need to  bpf.c  Push to remote server , This server will start the motor , Accelerate output  .o  and  .so. No matter what users use  Python  still  Go  Language 、Rust  or  C  Language , Just in the program  ini t Load these when  .o  or  .so  You can put the  BPF  Program  attach  To the kernel hook  spot , Then focus on processing from  BPF  Program output information , Function development .

coolbpf  hold  BTF  Make 、 The code to compile 、 Data processing 、 Functional testing is integrated , Production efficiency has been greatly improved , send BPF  Development has entered a more elegant realm ：

• Open the box ： The kernel side only provides  bpf.c  that will do , Completely peel off the kernel compilation project .

• Reuse compilation results ： There is no compilation process on the local side , There are no library dependencies and  CPU、 Memory and other resource consumption problems .

• Adapt to different versions ： It is more suitable for the scenario where multiple different kernel versions coexist in the cluster .

Install locally first coolbpf, The command inside will put xx.bpf.c Send to the compilation server to compile .
pip install coolbpf

...
import time
from pylcc.lbcBase import ClbcBase

bpfPog = r"""
#include "lbc.h"

SEC("kprobe/wake_up_new_task")
int j_wake_up_new_task(struct pt_regs *ctx)
{
struct task_struct* parent = (struct task_struct *)PT_REGS_PARM1(ctx);

bpf_printk("hello lcc, parent: %d\n", _(parent->tgid));
return 0;
}

char _license[] SEC("license") = "GPL";
"""

class Chello(ClbcBase):
 def __init__(self):
 super(Chello, self).__init__("hello", bpf_str=bpfPog)
 while True:
 time.sleep(1)
 
 if __name__ == "__main__":
 hello = Chello()
 pass


Two 、coolbpf  Function and architecture

​ I analyzed it earlier  BPF  Development mode ,coolbpf  With the help of remote compilation, the process of development and compilation is further optimized , Summarize what it currently contains  6  Big function ：

1） Local compilation service , Basic library encapsulation ： The client uses the local container image compiler , Call the encapsulated general function library to simplify programming and data processing .

Local compilation service , Put the same libraries and common tools in the container image , Compile directly into the container when compiling . We use the following image to compile , Users can also use  docker  Build your own container image .

Container mirror ：registry.cn-hangzhou.aliyuncs.com/alinux/coolbpf:latest

Users can  pull  This image is compiled locally , Some common libraries and tools , The image provided by us has been included , It saves the complexity of building the environment .

2） Remote compilation service ： receive  bpf.c, Generate  bpf.so  or  bpf.o, It is provided for high-level languages to load , Users only focus on their own function development , Don't worry about the installation of the underlying Library 、 Environment building .

Remote compilation service , At present, users only need  pip install coolbpf, The program will automatically go to our compilation server to compile . You can also refer to it  compile/remote-compile/lbc/  Build your own compilation server （ We will open the source code of this compilation server later ）, The process may be complicated . Such a built server , You can use it personally or provide it to everyone in the company .

3） The high version feature passed  kernel module  The method is supplemented to the lower version , Such as  ring buffer  characteristic ,backport BPF  Feature to  3.10  kernel .

Due to stock  3.10  There are still many servers in the kernel , To make the same  BPF  The program can also run in the lower version kernel , For the convenience of maintenance and without modifying the program code , It only needs  install  One  ko, You can support  BPF, Let the lower version also enjoy  BPF  Dividends .

4）BTF  Automatic generation of and the latest kernel version crawler of the whole network . Automatically discover the latest  CentOS、ubuntu、Anolis  Wait for the kernel version , Automatically generate corresponding  BTF.

You need to compile multiple runs at once  CO-RE  Ability , No,  BTF  It doesn't work .coolbpf  Not only provide a production  BTF  Tools for , It will also automatically discover and make the latest kernel version  BTF, For everyone to download and use .

5） Function test automation of each kernel version , Automatic installation test after tool writing , Ensure that user functions are pre tested before running in the production environment .

Not running online BPF Procedures and tools , There are risks in a certain probability .coolbpf Provide a set of automated test process , In most kernel environments, basic functional tests are performed in advance , Ensure that there will be no major problems when the tool really runs in the production environment .

6）Python、Rust、Go、C  And other advanced language support .

at present  coolbpf  The project supports the use of  Python、Rust、Go  And  C  Language user program development , Developers of different languages can give full play to their greatest advantages in their best fields .

All in all ,coolbpf  bring  BPF  Program and application development are closed-loop on one platform , Effectively improve productivity , It covers the current mainstream development language , Suitable for more  BPF  Beginners learn , It is also suitable for system operation and maintenance personnel to efficiently develop monitoring and diagnostic programs .

The following figure for  coolbpf  Functions and tool support of , Welcome more excellent  BPF  Tools added ：

​

3、 ... and 、 Practice description

coolbpf  Currently includes  pylcc、rlcc、golcc  and  clcc, as well as  glcc  subdirectories , They are high-level languages Python、Rust  and  Go  The ability of the language to support remote and local compilation ,glcc（g  representative  generic) It is by integrating the higher version  BPF  Feature migration to lower versions , adopt  kernel module  To run on the lower version . Next, we will briefly introduce its use .

1、pylcc( be based on  Python  Of  LCC）

pylcc  stay  libbpf  Packaging on the basis of , Leave the complex compilation project to the container .

The code is very concise , It only takes three steps to complete ,pyLCC  Technical key points ：

1） perform  pip install coolbpf  install

2）xx.bpf.c  Compiling :

bpfPog = r""" 
#include "lbc.h"
LBC_PERF_OUTPUT(e_out, struct data_t, 128);
LBC_HASH(pid_cnt, u32, u32, 1024);
LBC_STACK(call_stack,32);


3）xx.py  To write , Just this step , The program can run . Users can analyze the data received from the kernel :

importtimefrompylcc.lbcBaseimportClbcBase
classPingtrace(ClbcBase):def__init__(self):super(Pingtrace, self).__init__("pingtrace")


bpf.c  It needs to actively include  lbc.h, It informs the remote server of its behavior , This file is not required locally . It reads as follows ：

#include "vmlinux.h"
#include <linux/types.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>
#include <bpf/bpf_tracing.h>


2、rlcc( be based on  Rust  Of  LCC）

Rust  The language supports remote compilation and local compilation . By means of  makefile  Use in  coolbpf  Your order  bpf.c  Send to server , Server return  .o, This with  Python  and  C  return  .so  Make a big difference ,Rust  Handle general  load、attach  The process of . Others are similar to  Python  Development of , I won't repeat .

compile example technological process :
SKEL_RS=1 cargo build --release  Generate  rust skel  file ;
SKEL_RS=0 cargo build --release  No need to generate  rust skel  file ;
Default  SKEL_RS  by  1.

compile rexample technological process :
rexample  The remote compilation function is used , The specific compilation process is as follows :
Run the command  mkdir build & cd build  Create build directory ;
Run the command  cmake ..  Generate  Makefile  file ;
Run the command  make rexample;
function  example  Program : ../lcc/rlcc/rexample/target/release/rexample.

fn main() -> Result<()>{
 let opts = Command::from_args();
 let mut skel_builder = ExampleSkelBuilder::default();
 if opts.verbose {
 skel_builder.obj_builder.debug(true);
 }
 
 bump_memlock_rlimit()?;
 let mut open_skel = skel_builder.open()?;
 
 let mut skel = open_skel.load()?;
 skel.attach()?;
 let perf = PerfBufferBuilder::new(skel.maps_mut().events())
 .sample_cb(handle_event)
 .lost_cb(handle_lost_events)
 .build()?;
 
 loop {
 perf.poll(Duration::from_millis(100))?;
 }
}


3、glcc(generic LCC, Migrate high version features to low version )

background ：

• Currently based  eBPF  The program can only be written in the high version kernel （ Support  eBPF  The kernel of ） Up operation , Can't support  eBPF  Function on the kernel .

• There are many online  Alios  perhaps  CentOS  The lower version kernel needs to be maintained .

• The stock of  BPF  Tool or project code , I hope I can run across the kernel without modification .

Therefore, we propose a method to run in the low version kernel  eBPF  Procedure method , So that binary programs can be used without any modification  BPF  Run on the kernel of .

Let's sort out the architecture , The lower version of the kernel runs  BPF  The possibility of .

Hook  It's a dynamic library , Because the lower version kernel does not support  bpf()  System call , Originally created in user mode  map、 establish  prog  And a lot of  helper  function （ Such as  bpf_update_elem  etc. ） Will not run ,Hook  Provide a dynamic mechanism , Turn these system calls into  ioctl  command , Set it to a place called  ebpfdriver  Of  kernel module, Create some data structure simulation through him  map  and  prog, Register at the same time  kprobe  and  tracepoint  Of  handler. So when data comes , Will run to register in  kprobe  and  tracepoint  The callback .

The operating mechanism is shown in the figure below ：

​

utilize  Hook  Program will  BPF  Of  syscall  convert to  ioctl  form , Pass system call parameters to  eBPF  drive , It includes the following functions ：

#define IOCTL_BPF_MAP_CREATE _IOW(';', 0, union bpf_attr *)
#define IOCTL_BPF_MAP_LOOKUP_ELEM _IOWR(';', 1, union bpf_attr *)
#define IOCTL_BPF_MAP_UPDATE_ELEM _IOW(';', 2, union bpf_attr *)
#define IOCTL_BPF_MAP_DELETE_ELEM _IOW(';', 3, union bpf_attr *)
#define IOCTL_BPF_MAP_GET_NEXT_KEY _IOW(';', 4, union bpf_attr *)
#define IOCTL_BPF_PROG_LOAD _IOW(';', 5, union bpf_attr *)
#define IOCTL_BPF_PROG_ATTACH _IOW(';', 6, __u32)
#define IOCTL_BPF_PROG_FUNCNAME _IOW(';', 7, char *)
#define IOCTL_BPF_OBJ_GET_INFO_BY_FD _IOWR(';', 8, union bpf_attr *)


eBPF  Drive received  Ioctl  request , Will be based on  cmd  To do the corresponding operation , Such as ：

A. IOCTL_BPF_MAP_CREATE： establish map.

B. IOCTL_BPF_PROG_LOAD： load  eBPF  Bytecode , Carry out security verification of bytecode and  jit  Generate machine code .

C. IOCTL_BPF_PROG_ATTACH： Will be eBPF Program attach To the specified kernel function , utilize register_kprobe  and  tracepoint_probe_register  Function complete  eBPF  programmatic  attach.

in addition , Some features of the higher version , such as  ringbuff, It can also be done through  ko  And other methods are used in lower versions . image  clcc  and  golcc  How to use , Please refer to  coolbpf  Of  github  link （ See the end of the article ）, I'm not going to repeat it here .

Four 、 summary

coolbpf  At present, it has the above  6  Big function , Its purpose is to simplify the development and compilation process , Let users focus on their own function development , Make the vast majority of  BPF  Enthusiasts get started quickly , Write your own functional programs quickly without worrying about environmental problems . Today we open source this system , Let it serve more people , To increase their productivity , Promoting social progress , Let more people participate in the construction of this project , Form a joint force , Break through a technology .

Our remote compilation service , The solution is the efficiency of productivity ; Low version of the  BPF  Support , The solution is the same problem that puzzles all developers  bin  The purpose of how files can run indiscriminately in multi kernel versions , At the same time, we also hope that more people will participate and improve together , Let the brothers and sisters of cloud computing industry and enterprise services fully enjoy  BPF  The dividend of Technology .

Dragon lizard community system operation and maintenance  SIG（Special Interest Group） Committed to creating a centralized host management system 、 Configuration deployment 、 Monitoring and alarming 、 Abnormal diagnosis 、 Automatic operation and maintenance platform with a series of functions such as security audit ,coolbpf  It is a sub project of the community , The goal is to provide a compilation and development platform , solve  BPF  Operation and productivity improvement on different system platforms .

Welcome more developers to join the system operation and maintenance  SIG:

website ：https://openanolis.cn/sig/sysom

Mailing list, ：[email protected]

coolbpf  link ：[email protected]:aliyun/coolbpf.git

——  End  ——