Kubesphere multi node installation

Host list

Serial number	IP	Host name	To configure
1	192.168.85.161	k8s-master	8VCPU 32GB
2	192.168.85.162	k8s-node1	8VCPU 32GB
3	192.168.85.163	k8s-node2	8VCPU 32GB
4	192.168.85.164	nfs-server	8VCPU 32GB

Basic environment preparation

nfs Server installation deployment ( A little )

1. install docker

curl -sSL https://get.daocloud.io/docker | sh
#  Speed up 
curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io

2. Installation dependency

yum -y install socat conntrack ebtables ipset net-tools

3. To configure ssh Mutual visits

#  Execute on all nodes 
ssh-keygen
ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]
ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]
ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]

4. download KubeKey

#  Only in master You can execute it on 
export KKZONE=cn
curl -sfL https://get-kk.kubesphere.io | VERSION=v2.0.0 sh -

Create clusters

1. Create a sample configuration file

./kk create config --with-kubesphere

2. Edit profile

The plug-in depends on the usage and the size of server resources , Its installation .

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sample
spec:
  hosts:   #  Configure host list 
  - {
    name: k8s-master, address: 192.168.85.161, internalAddress: 192.168.85.161, privateKeyPath: "~/.ssh/id_rsa"}
  - {
    name: k8s-node1, address: 192.168.85.162, internalAddress: 192.168.85.162, privateKeyPath: "~/.ssh/id_rsa"}
  - {
    name: k8s-node2, address: 192.168.85.163, internalAddress: 192.168.85.163, privateKeyPath: "~/.ssh/id_rsa"}
  roleGroups:
    etcd:    # etcd  The node list 
    - k8s-master  
    control-plane:    #  Host node list 
    - k8s-master  
    worker:    #  List of work nodes 
    - k8s-node1
    - k8s-node2
  controlPlaneEndpoint:
    ## Internal loadbalancer for apiservers 
    # internalLoadbalancer: haproxy

    domain: lb.kubesphere.local
    address: ""
    port: 6443
  kubernetes:
    version: v1.21.5
    clusterName: cluster.local
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
    multusCNI:
      enabled: false
  registry:
    plainHTTP: false
    privateRegistry: ""
    namespaceOverride: ""
    registryMirrors: []
    insecureRegistries: []
  addons: 
  - name: nfs-client    #  Use nfs As persistent storage 
    namespace: kube-system
    sources: 
      chart: 
        name: nfs-client-provisioner
        repo: https://charts.kubesphere.io/main
        values:
        - storageClass.defaultClass=true
        - nfs.server=192.168.85.164
        - nfs.path=/data

---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
  name: ks-installer
  namespace: kubesphere-system
  labels:
    version: v3.2.1
spec:
  persistence:
    storageClass: ""
  authentication:
    jwtSecret: ""
  local_registry: ""
  namespace_override: ""
  # dev_tag: ""
  etcd:
    monitoring: true  #  Decide whether to open... According to the situation 
    endpointIps: localhost
    port: 2379
    tlsEnable: true
  common:
    core:
      console:
        enableMultiLogin: true
        port: 30880
        type: NodePort
    # apiserver:
    # resources: {}
    # controllerManager:
    # resources: {}
    redis:
      enabled: true  #  Decide whether to open... According to the situation 
      volumeSize: 2Gi
    openldap:
      enabled: true  #  Decide whether to open... According to the situation 
      volumeSize: 2Gi
    minio:
      volumeSize: 20Gi
    monitoring:
      # type: external
      endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
      GPUMonitoring:
        enabled: true  #  Decide whether to open... According to the situation 
    gpu:
      kinds:         
      - resourceName: "nvidia.com/gpu"
        resourceType: "GPU"
        default: true
    es:
      # master:
      # volumeSize: 4Gi
      # replicas: 1
      # resources: {}
      # data:
      # volumeSize: 20Gi
      # replicas: 1
      # resources: {}
      logMaxAge: 7
      elkPrefix: logstash
      basicAuth:
        enabled: false  #  Whether to use authentication 
        username: ""
        password: ""
      externalElasticsearchHost: ""
      externalElasticsearchPort: ""
  alerting:
    enabled: true  #  Decide whether to open... According to the situation 
    # thanosruler:
    # replicas: 1
    # resources: {}
  auditing:
    enabled: true  #  Decide whether to open... According to the situation 
    # operator:
    # resources: {}
    # webhook:
    # resources: {}
  devops:
    enabled: true  #  Decide whether to open... According to the situation 
    jenkinsMemoryLim: 2Gi
    jenkinsMemoryReq: 1500Mi
    jenkinsVolumeSize: 8Gi
    jenkinsJavaOpts_Xms: 512m
    jenkinsJavaOpts_Xmx: 512m
    jenkinsJavaOpts_MaxRAM: 2g
  events:
    enabled: true
    # operator:
    # resources: {}
    # exporter:
    # resources: {}
    # ruler:
    # enabled: true
    # replicas: 2
    # resources: {}
  logging:
    enabled: true
    containerruntime: docker
    logsidecar:
      enabled: true
      replicas: 2
      # resources: {}
  metrics_server:
    enabled: true
  monitoring:
    storageClass: ""
    # kube_rbac_proxy:
    # resources: {}
    # kube_state_metrics:
    # resources: {}
    # prometheus:
    # replicas: 1
    # volumeSize: 20Gi
    # resources: {}
    # operator:
    # resources: {}
    # adapter:
    # resources: {}
    # node_exporter:
    # resources: {}
    # alertmanager:
    # replicas: 1
    # resources: {}
    # notification_manager:
    # resources: {}
    # operator:
    # resources: {}
    # proxy:
    # resources: {}
    gpu:
      nvidia_dcgm_exporter:
        enabled: true
        # resources: {}
  multicluster:
    clusterRole: none 
  network:
    networkpolicy:
      enabled: true
    ippool:
      type: none
    topology:
      type: none
  openpitrix:
    store:
      enabled: false
  servicemesh:
    enabled: true
  kubeedge:
    enabled: false   
    cloudCore:
      nodeSelector: {
    "node-role.kubernetes.io/worker": ""}
      tolerations: []
      cloudhubPort: "10000"
      cloudhubQuicPort: "10001"
      cloudhubHttpsPort: "10002"
      cloudstreamPort: "10003"
      tunnelPort: "10004"
      cloudHub:
        advertiseAddress:
          - ""
        nodeLimit: "100"
      service:
        cloudhubNodePort: "30000"
        cloudhubQuicNodePort: "30001"
        cloudhubHttpsNodePort: "30002"
        cloudstreamNodePort: "30003"
        tunnelNodePort: "30004"
    edgeWatcher:
      nodeSelector: {
    "node-role.kubernetes.io/worker": ""}
      tolerations: []
      edgeWatcherAgent:
        nodeSelector: {
    "node-role.kubernetes.io/worker": ""}
        tolerations: []

3. Create a cluster using a configuration file

./kk create cluster -f config-sample.yaml

4. Verify the installation

After installation , You will see the following ：

#####################################################
### Welcome to KubeSphere! ###
#####################################################

Console: http://192.168.85.161:30880
Account: admin
Password: [email protected]

NOTES：
  1. After you log into the console, please check the
     monitoring status of service components in
     "Cluster Management". If any service is not
     ready, please wait patiently until all components
     are up and running.
  2. Please change the default password after login.

#####################################################
https://kubesphere.io             2022-07-28 03:10:29
#####################################################

install openelb

1. install

kubectl apply -f https://raw.githubusercontent.com/openelb/openelb/master/deploy/openelb.yaml

2. Enable strictARP

Need to be for kube-proxy Enable strictARP, In order to Kubernetes All network cards in the cluster stop responding to requests from other network cards ARP request , And by the OpenELB Handle ARP request .

kubectl edit configmap kube-proxy -n kube-system
...
ipvs:
  strictARP: true
...

Then execute the following command to restart kube-proxy Components ：

kubectl rollout restart daemonset kube-proxy -n kube-system

3. To configure EIP

apiVersion: network.kubesphere.io/v1alpha2
kind: Eip
metadata:
    name: eip-pool
    annotations:
      eip.openelb.kubesphere.io/is-default-eip: "true"
spec:
    address: 192.168.85.91-192.168.85.100  #  Address pool 
    protocol: layer2  #  agreement ：BGP, Layer 2, or VIP
    interface: eth0  #  network card 
    disable: false

4. How to use

1. establish deployment

1. establish yaml file

vi layer2-openelb.yaml

2. Write the following

apiVersion: apps/v1
kind: Deployment
metadata:
 name: layer2-openelb
spec:
 replicas: 2
 selector:
   matchLabels:
     app: layer2-openelb
 template:
   metadata:
     labels:
       app: layer2-openelb
   spec:
     containers:
       - image: luksa/kubia
         name: kubia
         ports:
           - containerPort: 8080

3. application yaml File creation deployment

kubectl apply -f layer2-openelb.yaml

2. establish service

1. establish yaml file

vi layer2-svc.yaml

2. Write the following

kind: Service
apiVersion: v1
metadata:
 name: layer2-svc
 annotations:  #  You need to add the following 3 A note 
   lb.kubesphere.io/v1alpha1: openelb
   protocol.openelb.kubesphere.io/v1alpha1: layer2
   eip.openelb.kubesphere.io/v1alpha2: eip-pool  # eip Address pool name 
spec:
 selector:
   app: layer2-openelb
 type: LoadBalancer
 ports:
   - name: http
     port: 80
     targetPort: 8080
 externalTrafficPolicy: Cluster

3. application yaml File creation service

kubectl apply -f layer2-svc.yaml

3. verification

[[email protected] ~]# kubectl get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
kubernetes   ClusterIP      10.233.0.1      <none>          443/TCP        34m
layer2-svc   LoadBalancer   10.233.49.106   192.168.85.91   80:31929/TCP   4s

[[email protected] ~]# curl 192.168.85.91
You've hit layer2-openelb-7b4fdf6f85-nvsws