[translation] cloud like internal load balancer for kubernetes?

author ： Alex - saroyan 2022 year 2 month 21 Japan

Guest article , Originally published in Netris The blog of On , author Alex Saroyan

Everybody knows , Use Kubernetes Load balancer is a challenge . Back when I was in charge of network operation , The management of load balancer is also in the charge of my team . Every time Linux/Unix OPS To add or move application servers -- They all have to issue a ticket to the network team , Including the back-end address 、TCP List of ports , And the description of the health examination , To configure a large network load balancer . Network engineers and Linux The engineer , It's all human , Sometimes I make mistakes , Then they will cooperate to fix things , Keep production running .

Because of all the back and forth , It usually takes a few days to satisfy ordinary load balancer requests . at that time , This is the norm . Now? , In the era of the cloud , The specification is to provide load balancer services on demand , And with Kubernetes Seamless collaboration . Although it makes DevOps Engineers are easier , But it's not so easy to build and organize this cool load balancer as a service for the internal private cloud .

What do cloud computing practitioners expect from the user experience of the fourth layer load balancer ？

Public cloud providers automatically configure the services required by applications , Provides a lot of convenience .

All major public cloud providers provide on-demand load balancer Services , You use it to put your Kubernetes Applications are exposed on the public Internet . Or you are using one Ingress controller , Yours Ingress The controller is using an on-demand load balancer to make its public TCP Ports are exposed to the Internet .

How can we get this cloud like experience in the private cloud ？

A modern fourth layer load balancer （L4LB） Minimum expectation .

• And Kubernetes Native integration of
• instant 、 On demand configuration
• Can use management .K8s CRD、Terraform、RestAPI、 Intuitive GUI
• High availability
• Scalable
• TCP/HTTP health examination
• Easy to install and use （L4LB Not rocket science ）

A modern fourth layer load balancer （L4LB） Good expectations .

• Run on commodity hardware
• Support DPDK/SmartNIC HW Speed up
• Based on the well-known open source ecosystem and standard protocols （ There is no proprietary black box ）.

In my architecture , The first 4 Where is the most practical function of layer load balancer ？

Ethernet switches can load balance traffic . But this applies to Kubernetes Do you ？

The main job of Ethernet switch is to provide the highest performance communication between your private cloud application servers . Modern Ethernet switches can be used according to IP Address and TCP/UDP Port number hash algorithm to load balance network traffic . This load balancing technology is usually used in the second layer （ Such as LAG Interface balance ） And the third floor （ Such as cross switch fabric ECMP）. be based on ECMP The third layer of load balancing can also be used for server traffic load balancing , Usually used for ROH（ Route to the host ） High performance content delivery architecture . The advantage of this method is super high performance , But the disadvantage is that there is no way to distinguish between different TCP port . therefore , Every application and every TCP All ports need a dedicated public IP Address , This valuable resource is effectively wasted .

Summary ： Ethernet switches are suitable for layer 3 load balancing , But it is not suitable for the fourth floor , So it's not suitable for Kubernetes Load balancer （LoadBalancer Type of service ）.

Run the... Directly on the application server 4 Whether the layer load balancer is practical ？

stay Kubernetes Run on node Layer-4 Load balancing has many applications , Include Metallb、KubeVIP, Or use it directly kubeproxy or Cilium Of nodeport. These methods are in Kubernetes Aspect is quite simple , Easy to implement . The challenge lies in . How to make the flow From the network gateway route To these kubernetes Load Balancer . If we route traffic statically , This means that all incoming requests to any container on any node will be forced to use only one node to enter kubernetes colony . This may cause bottlenecks and high availability problems .

A better way is to use dynamic routing protocols （ Such as BGP And Metallb）, It solves the problem of high availability . under these circumstances , Front end public IP The address exists in every kubernetes Node （ Any broadcast ）, Physical networks （ Your switch or router ） For all runs Metallb Of kubernetes Nodes perform three-tier load balancing . Every kubernetes Nodes will need to perform a fourth level lookup , To check whether the target application is running locally on the current node . But in most cases , It must perform an additional forwarding operation , Will flow （ Through the physical network ） Sent to the run target container kubernetes node . This leads to sub optimal routing .

Besides , When you are in this situation, you will BGP When used for load balancer , Meeting and general BGP Used in conflict with others . A common example is with Calico CNI The conflict of .

reference .

https://metallb.universe.tf/configuration/calico/#workaround-peer-with-spine-routers

https://docs.projectcalico.org/networking/bgp

Summary ： Application server is not the best place for load balancing .

• Potential suboptimal routing
• stay kubernetes The node hosts a public IP Address （ Security ）.
• Add every time / Move / Delete kubernetes Node time , Need to change the router / On the switch BGP To configure .
• And Calico CNI Conflict

Why is the network gateway a good place to realize the load balancing function of the fourth layer ？

Every private network needs some kind of network gateway . It can be a Linux Router , A cheap device , Or an expensive dedicated high-performance router . Every packet entering or leaving the private cloud from the outside must go through the routing decision steps of the network gateway . In any case, the network gateway will rewrite the packet header for routing . Adding additional functions of layer 4 load balancing here does not change your network structure , And the overhead imposed on the gateway is also very small . therefore , The network gateway is almost the organic place of the fourth layer load balancing function .

What are the options （ Apply to Kubernetes）？

Use special load balancer equipment （ Such as F5） To do it Layer-4 Of k8s Load balancer , Whether too much is better than less ？

Some modern load balancer devices provide Kubernetes Integration of . Specialized load balancers provide many different types of complex and expensive load balancing services , and Layer-4 Just a small part of it . If your goal is only exposure Kubernetes The application and / Or route traffic to Kubernetes, Then a special load balancer is redundant . in addition , Most dedicated load balancers are not designed for other basic gateway functions , Such as routing or NAT, So they need a special router as a separate device . The opposite is true . Most routers do not provide layer 4 load balancing .

Economic 、 Cloud like network gateways and native Kubernetes Integrate

Linux machine + Optional DPDK/SmartNIC Acceleration can be a great network gateway , It has a fourth layer load balancer and Kubernetes Native integration of . Almost all ordinary Linux Distributions have the most basic building blocks , The second floor 、 The third level 、 The fourth layer network function .

• FRR（Free Range Routing） Is an open source routing protocol suite , Have the most advanced BGP Protocol implementation , It is equivalent to the traditional network router .
• Nftables（ It's like iptables, But better ） It has packet filtering function , You can see IP Data packet head , Such as TCP/UDP Port and perform basic operations . for example , allow / Reject some traffic , Or for SNAT/DNAT Traffic rewrite IP baotou .SNAT（ Source network address translation ） It is a basic function of network gateway , So that our private network can access resources on the public Internet .DNAT（ Destination network address translation ） It is the basic packet filtering function necessary for the fourth layer load balancer .

Netris SoftGate It's software , It can be configured automatically FRR、nftables、Linux Bridge and Linux Other aspects of the operating system , by Your private cloud Provide Basic network services . There are options based on DPDK Network acceleration , You can use SmartNIC Hardware to provide up to per node 100Gbps Traffic routing and layer 4 load balancing . You can automatically HA Use two nodes in mode , You can even expand them horizontally .

Netris Operator, Provide basic network services Kubernetes CRD（ Custom resource definition ）. Here is Layer-4 k8s Load balancer service YAML Example .

 specifications .
  backend:
  - 192.168.110.68:30590
  - 192.168.110.69:30590
   Check .
     Overtime ：2000
     type : tcp
   front end .
    IP：50.117.59.138
     port .80
  ownerTenant: Administrators 
   agreement : tcp
   place . The United States / New York City 
   state :  Activities 
...

Netris Operator Will put YAML Data push to Netris The controller , then Netris SoftGate（ Your idea is based on Linux Network gateway ） Will automatically configure itself , Provide any network services you require .

Netris The operator will continuously monitor kube-api, If you create a LoadBalancer Type of service , proper Netris CRD Will be created automatically . then Netris The controller will be marked for k8s Load balanced IP Allocate a public in the pool IP Address , And then through Netris SoftGate Rotate the load balancer out .

 type :  service 
 Metadata .
   name : podinfo
spec:
  type: Load balancer 
   Selectors .
     Applications : podinfo
   port .
    - name: http
       port .80
       agreement .TCP
       Target port ：http
...

I learned a lot from the perspective of being a network operator . I have never used Netris The experience of customers running large and medium-sized private cloud infrastructure has learned a lot .

My purpose here is to share my learning achievements with you , So when you need to design and build your next infrastructure , Even a small family / Office laboratory , It may help you .

meanwhile , Suppose someone is right Netris Interested in . under these circumstances , You can use our sandbox environment to play , Or install the free version , It is suitable for use anywhere , From small families / Office labs to small 1-2 Cabinet production .

We are very grateful to you for Netris And the feedback of this article 、 Comments and questions .

Thank you for reading here !