Openshift 4 - use verticalpodautoscaler to optimize application resource request and limit

《OpenShift 4.x HOL Tutorial summary 》
explain ： This article has been OpenShift 4.10 Verification in environment

understand VPA and HPA

HPA - Horizontal Pod Autoscaler and VPA - Vertical Pod Autoscaler There are two ways to extend the processing power of container applications ,HPA By expanding Pod Achieved by the quantity of , and VPA By adding a single Pod Of available resources .

Usually HPA It can be used when horizontal expansion is easy , for example Serverless、FaaS、 Stateless microservices . and VPA It is suitable for complex horizontal expansion , For example, message sequence processing 、 File read and write 、 Database operation, etc . It is generally not recommended to apply to the same resource at the same time HPA and VPA.

VPA Will automatically check Pod The health status of the container in and the current CPU And memory resources , And update the resource limit and request according to the consumption value it knows .VPA Use separate custom resources （CR） To update all... Associated with the workload object Pod.VPA It can automatically calculate these Pod The current CPU And memory usage , And use this data to determine optimized resource constraints and requests , To make sure that Pod Always operate efficiently . for example ,VPA It will reduce the number of resources requested over resources used Pod Resources for , Not for those who do not request sufficient resources Pod Increase resources . For example, one Pod Currently used CPU Of 50%, But I only asked 10%.VPA Will determine that Pod The consumption of CPU More than requested CPU, And delete Pod. Workload object （ Like a copy set ） Will restart Pod,VPA Update with recommended resources Pod.

VPA You can automatically delete those that do not meet the recommendations Pod（ It is recommended that the workload object run at least two copies , This will not affect continuous operation ）. Running new Pod When ,VPA Use a variant access webhook To ensure that it is running Pod Before adjusting its resources to optimized restrictions and requests . In addition to automatic updates Pod Outside , You can also delete it manually Pod,VPA Will use its suggestions to create new Pod.

have access to VPA To make better use of cluster resources , For example, to prevent Pod Keep more than you need CPU Resources more resources .VPA Monitor the workload actually used , And adjust resources , To ensure that the needs of other workloads can be met .

If you stop running in the cluster VPA Or delete specific VPA CR, By VPA Modified pod The resource request will not change . Any new pod Will get resources according to the definition in the workload object , Instead of before by VPA Advice provided .

VPA Need to associate a workload object ,VPA Support the policy update workload ：

• Auto and Recreate The mode will be Pod Automatically apply during the life cycle VPA Yes CPU And memory recommendations .VPA Will delete any items in the project that are incompatible with the suggestions Pod. When redeployed by workload objects ,VPA Will be updated in its recommendations Pod.
• Initial Patterns are only created Pod Automatically apply when VPA Suggest .
• Off The mode only provides recommended resource restrictions and request information , Users can manually apply the suggestions .off Mode does not update Pod.

Installation configuration VPA

1. Use the default configuration in OpenShift Install in Vertical Pod Autoscaler Operator.
   
2. It will be installed in openshift-vertical-pod-autoscaler Deploy the following resources in the project ：
   

When the application appears OOMKilled when , Automatic adjustment requests and limits To configure

The application runs in none VPA Under the circumstances

1. Create a new project .

$ oc new-project app-novpa

2. Deploy test applications . Be careful ： Although the maximum memory allocated for the container is 200Mi, But the app will apply 250M Of memory .

$ cat << EOF | oc -n app-novpa apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: stress-novpa
spec:
  selector:
    matchLabels:
      app: stress
  replicas: 1
  template:
    metadata:
      labels:
        app: stress
    spec:
      containers:
      - name: stress
        image: polinux/stress
        resources:
          requests:
            memory: "100Mi"
          limits:
            memory: "200Mi"
        command: ["stress"]
        args: ["--vm", "1", "--vm-bytes", "250M"]
EOF

3. View after deploying the application Pod Running state , confirm Pod appear OOMKilled The situation makes it unable to operate normally .

$ oc get pod -n app-novpa -w
NAME                            READY   STATUS              RESTARTS      AGE
stress-novpa-5f8cf46f67-cscjh   0/1     Pending             0             0s
stress-novpa-5f8cf46f67-cscjh   0/1     Pending             0             0s
stress-novpa-5f8cf46f67-cscjh   0/1     ContainerCreating   0             0s
stress-novpa-5f8cf46f67-cscjh   0/1     ContainerCreating   0             2s
stress-novpa-5f8cf46f67-cscjh   0/1     OOMKilled           0             3s
stress-novpa-5f8cf46f67-cscjh   1/1     Running             1 (1s ago)    4s
stress-novpa-5f8cf46f67-cscjh   0/1     OOMKilled           1 (2s ago)    5s
stress-novpa-5f8cf46f67-cscjh   0/1     CrashLoopBackOff    1 (2s ago)    6s
stress-novpa-5f8cf46f67-cscjh   0/1     OOMKilled           2 (18s ago)   22s

The application runs on VPA Under the circumstances

1. Create a new project .

$ oc new-project app-vpa

2. Deploy test applications , The maximum memory allocated for the container is 200Mi, Applications only apply 150M Memory .

$ cat << EOF | oc -n app-vpa apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: stress
spec:
  selector:
    matchLabels:
      app: stress
  replicas: 1
  template:
    metadata:
      labels:
        app: stress
    spec:
      containers:
      - name: stress
        image: polinux/stress
        resources:
          requests:
            memory: "100Mi"
          limits:
            memory: "200Mi"
        command: ["stress"]
        args: ["--vm", "1", "--vm-bytes", "150M"]
EOF

3. Confirm that the application can run normally at this time , Then check it out Pod The status of and the current impact on memory Request and Limit.

$ oc get deployment -n app-vpa
NAME     READY   UP-TO-DATE   AVAILABLE   AGE
stress   1/1     1            1           21m
 
$ oc get pod -n app-vpa -w
NAME                    READY   STATUS    RESTARTS   AGE
stress-589cd958-7dlhc   1/1     Running   0          14s
 
$ oc get pod -n app-vpa -l app=stress -o yaml | grep requests -A1
        requests:
          memory: 100Mi
 
$ oc get pod -n app-vpa -l app=stress -o yaml | grep limits -A1
        limits:
          memory: 200Mi

4. establish VerticalPodAutoscaler object , Name its association stress Of Deployment, And all containers under it ( containerName: ‘*’ ) It works . among minAllowed and maxAllowed Is aimed at Request The upper and lower effective limits of .

$ cat << EOF | oc -n app-vpa apply -f -
apiVersion: "autoscaling.k8s.io/v1"
kind: VerticalPodAutoscaler
metadata:
  name: stress-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: stress
  resourcePolicy:
    containerPolicies:
      - containerName: '*'
        minAllowed:
          cpu: 100m
          memory: 50Mi
        maxAllowed:
          cpu: 1000m
          memory: 1024Mi
        controlledResources: ["cpu", "memory"]
EOF

explain ：VPA Support blacklist mechanism , That is, containers in the blacklist will not be applied VPA Strategy .

5. View current period VPA object .

$ oc get vpa -n app-vpa
NAME         MODE   CPU   MEM       PROVIDED   AGE
stress-vpa   Auto   1     262144k   True       90s

6. see VPA Suggested allocation CPU And memory . among lowerBound The minimum amount of resources allocated to the proposal ,target The amount of resources allocated to the proposal ,upperBound The maximum amount of resources allocated to the proposal ,uncappedTarget Allocate resources for the latest recommendations .VPA Use lessBound and upperBound Value to determine a Pod Need to update . If Pod The resource request of is lower than lowerBound value , Or above upperBound value , be VPA Will be terminated Pod , And use target Value recreated Pod .

$ oc get vpa stress-vpa -n app-vpa -oyaml
...
  recommendation:
    containerRecommendations:
    - containerName: stress
      lowerBound:
        cpu: "1"
        memory: 262144k
      target:
        cpu: "1"
        memory: 262144k
      uncappedTarget:
        cpu: 1643m
        memory: 262144k
      upperBound:
        cpu: "1"
        memory: 1Gi

7. Change the name to stress Of Deployment, Change its used memory to “250M". Be careful ： At this time, the applied “250M" The memory has exceeded that of Deployment Medium limits Specify the allocation to Pod The memory limit of .

$ oc patch deployment stress -n app-vpa --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/args/3", "value": "250M" }]'

8. After modification, you can view Pod The change of , The original Pod Will be deleted , new Pod Will be created .

$ oc get pod -n app-vpa -w
NAME                      READY   STATUS        RESTARTS   AGE
stress-7b9459559c-ntnrv   1/1     Running       0          5s
stress-7d48fdb6fb-7dlhc   1/1     Terminating   0          22m

9. Confirm the new Pod The upper limit of available memory is adjusted to 500Mi.

$ oc get pod -n app-vpa -l app=stress -o yaml | grep limits -A1
        limits:
          memory: 500Mi

10. Change the name again stress Of Deployment, Change its used memory to “1500M". because Pod The requested memory has exceeded VPA in upperBound Defined “1Gi" go online , therefore Pod Again OOMKilled.

$ oc patch deployment stress -n app-vpa --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/args/3", "value": "1500M" }]'

$ oc get pod -n app-vpa -w
NAME                      READY   STATUS             RESTARTS     AGE
stress-5f8cf46f67-ntnrv   1/1     Running            0            5m54s
stress-947fdb66f-rfq5t    0/1     CrashLoopBackOff   1 (8s ago)   14s
stress-947fdb66f-rfq5t    0/1     OOMKilled          2 (17s ago)   23s

Reference resources

https://access.redhat.com/documentation/zh-cn/openshift_container_platform/4.10/html/nodes/nodes-pods-vertical-autoscaler-using-about_nodes-pods-vertical-autoscaler
https://rcarrata.com/kubernetes/predictive-autoscaling-patterns-with-vpa/
https://cloud.redhat.com/blog/how-full-is-my-cluster-part-4-right-sizing-pods-with-vertical-pod-autoscaler
https://medium.com/infrastructure-adventures/vertical-pod-autoscaler-deep-dive-limitations-and-real-world-examples-9195f8422724