[azure microservice service fabric] how to transfer seed nodes in the service fabric cluster

Be careful ： In the face of Service Fabric Before the operation of the node , Be sure to confirm if it is a seed node (Seed Node) And whether the number of current nodes is equal to SF The number of persistence layers required for is the same .

The reliability level is Service Fabric A property of cluster resources . For each node type , The configuration of this property must be the same . This attribute controls the replication factor of the cluster system service , It is the setting of cluster resource level .
The reliability level determines the lower limit of the number of nodes that the primary node type must have . The following values can be used for the reliability layer ：

• Platinum ： Run contains 7 Target replica sets and 9 System services of seed nodes .
• gold ： Run contains 7 Target replica sets and 7 System services of seed nodes .
• silver ： Run contains 5 Target replica sets and 5 System services of seed nodes .
• copper ： Run contains 3 Target replica sets and 3 System services of seed nodes .

The recommended minimum reliability level is “ silver ” level .

 

In the use of Service Fabric In the process of , Sometimes there is a problem with a node , In general ,SF The cluster is designed as a highly available cluster , When a single node is down, other nodes will take over . But sometimes we need to transfer the seed nodes , When there is such a need , You can refer to the steps ：

Preparation conditions ：

• Use Powershell/sfctl Already connected to the current SF colony （​ ​Connect-ServiceFabricCluster​​​ &​ ​sfctl cluster​​）
• Use Service Fabric Explorer Connect to SF colony

Execution steps ：

One ： Make sure node type The number of virtual machines meets reliability tier The minimum demand for , And meet at least one none-seed node

Two ： Use powershell perhaps

Log in and copy

      
      

       
       # power shell
       
       
Disable-ServiceFabricNode -NodeName _nt1vm_5 -Intent RemoveNode -Force
       
       

       
       
# sfctl
       
       
sfctl node disable --node-name _nt1vm_5  --deactivation-intent RemoveNode
      
      
      
      

       
       
1.
       
       
2.
       
       
3.
       
       
4.
       
       
5.
      
      

3、 ... and ： wait for seed node State by ‘disabling’ Convert to ‘disabled’

Waiting for the operation to trigger cluster upgrade complete , as well as _nt1vm_5 Convert to none-seed node

Four ： stay VMSS The interface directly removes the corresponding virtual machine （ Running on the Linux Upper Service Fabric Cluster I won't support it sfctl node transition command ）

5、 ... and ： Use powershell perhaps sfctl remove node state

Log in and copy

      
      

       
       #power shell
       
       
Remove-ServiceFabricNodeState -NodeName "_nt1vm_5"
      
      
      
      

       
       
1.
       
       
2.
      
      

Log in and copy

      
      

       
       #sfctl
       
       
sfctl node remove-state --node-name _nt1vm_5
      
      
      
      

       
       
1.
       
       
2.
      
      

 

 

Reference material ：

Random failures that cause cluster failures ：​ ​https://docs.azure.cn/zh-cn/service-fabric/service-fabric-disaster-recovery#random-failures-that-lead-to-cluster-failures​​

Service Fabric With the concept of seed nodes .  The seed node can maintain the availability of the basic cluster .

The seed node can sign a lease with other nodes , Act as a circuit breaker in the event of certain types of faults , This ensures that the cluster remains started .  If a random failure deletes most of the seed nodes in the cluster and cannot be recovered quickly , Then the group assembly will be automatically closed .  The cluster will then fail .

stay Azure in ,Service Fabric Resource provider management Service Fabric Cluster configuration .  By default , The resource provider allocates seed nodes between the fault domain and the promotion domain of the master node type .  If the primary node type is marked “ silver ” or “ gold ” Class durability , When deleting the seed node by reducing the type of the master node or manually deleting the seed node , The cluster tries to promote another non seed node from the available capacity of the primary node type .  If your available capacity is less than the primary node type required by the cluster reliability level , Then this attempt will fail .

In independence Service Fabric Cluster sum Azure in , The primary node type is the node type where the seed runs .  When defining the master node type ,Service Fabric It will automatically utilize the number of nodes provided when creating up to nine seed nodes and seven replicas for each system service .  If a set of random failures causes most of the copies to shut down at the same time , Then the system service will enter the arbitration loss state .  If most of the seed nodes are lost , The group meeting closed soon .

 

​ ​https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-best-practices-capacity-scaling#scaling-in​​

 

​ ​https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-cluster-capacity​​

 

​ ​https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-cluster-scale-in-out#manually-remove-vms-from-a-node-typevirtual-machine-scale-set​​

 

When facing problems in a complex environment , The way to check things needs ： The turbid and quiet Xu Qing , An Yidong's Xu Sheng . In the clouds , Just so !