当前位置:网站首页>Distributed (consistency protocol) leader election (dotnext.net.cluster implements raft election)
Distributed (consistency protocol) leader election (dotnext.net.cluster implements raft election)
2022-07-06 17:36:00 【Dotnet cross platform】
Distributed ( Agreement of conformity ) The election of leaders ( DotNext.Net.Cluster Realization Raft The election )
After the distributed lock, another highly available technology, shuangwen's distributed leadership election Or say Implementation of distributed consistency protocol
Distributed election is a necessary technology to achieve high availability , Want to achieve master-slave , There must be an election strategy , Only with master-slave can there be a real management side to coordinate and allocate resources .
First of all, we need to clarify what the goal of the consistency algorithm is , The main problem is that when using only a single server, data loss and other things happen due to errors . The way to solve this problem is very simple , It's backup , colony , Multiple servers , If you repeat the operation on multiple machines, you won't be afraid that a single machine will make mistakes . But what follows is , Data inconsistency 、 Disorder and so on , What the consistency algorithm wants to do is even if there is a node error , Externally, it is still a complete and working whole .
Election algorithm
Implement a consistent protocol ( The election ) There are two main algorithms
1. Raft
2. Paxos
amount to Paxos Speaking of , Raft The agreement is relatively simple . however ,Raft It's not easy to implement , If a friend tries to realize it, you can refer to , This place
Address :https://zinglix.xyz/2020/06/25/raft/
So am I A simple understanding .
Raft It is a non Byzantine consistency algorithm , That is, all communications are correct, not forged .N In the case of nodes (N It's odd ) Can tolerate at most (N−1)/2 Node failure
Why do you need a separate election algorithm
I once tried to achieve a WEB service function , I can't guarantee the high availability of this service , I don't want to use other existing services , I just want the service itself to support high availability .
at that time , Because of my cognitive problems , There is no available solution in a short time , But now there is .
If there were , Maybe it's a different scenery , To say .
General algorithm of distributed election
Simply put, it means finding a leader , Let's say I have a leader key,redis in , Who stole , Who is leader, It can also be realized . The leadership election implemented by this distributed lock can also be applied to simple projects , also , Support a single service host .
Want to use distributed Election Algorithm , The machine must at least 2 Table above , Or the conceptual two .
Raft There are three main characters in Leader( leaders )、Follower ( follower ) and Candidate( The candidate ), When a machine becomes a leader , Will become the main external counterpart , Then synchronize the docking to the following followers or candidates to synchronize information .
Because there can only be one main service , Play the role of coordination and management .
such , Will send the corresponding instructions ( journal ) Assign each client , Play the role of data consistency , such , When leaders are abolished , The next person will take over , It can continue to work .
Raft Examples of elections
I found a lot .Net Realized Raft, Many can only be said to be toys , It can't be used in production .
But fortunately, , There is really a production level .
That's it DotNext.Net.Cluster and DotNext.AspNetCore.Cluster ( Support http ) .
Mainly based on DotNext Components in .
DotNext.Net.Cluster and DotNext.AspNetCore.Cluster
1. DotNext.Net.Cluster Contains the cluster programming model 、Raft Transmission independent implementation of the algorithm 、Raft Of TCP and UDP Transport binding 、HyParView membersip Transport independent implementation of the Protocol , Used based on Gossip Messaging for
2. DotNext.AspNetCore.Cluster Is based on DotNext.Net.Cluster Library Raft and HyParView The specific implementation of the algorithm , Used to build ASP.NET Core Applications
List of supported features
List of supported features :
1. Network transmission :TCP、UDP、HTTP 1.1、HTTP/2、HTTP/3
2. TLS Support :TCP、HTTP 1.1、HTTP/2、HTTP/3
3. High performance supporting log compression 、 Universal Persistent Write-Ahead Log
4. Copy log entries across cluster nodes
5. And ASP.NET Core The framework is tightly integrated
6. Yes Docker/LXC/Windows Container friendly
7. Everything is extensible
• 7.1 Customize the pre write log
• 7.2 Customize network transmission
• 7.3 Cluster member discovery
be based on DotNext.Net.Cluster Of TCP Examples of elections
In fact, he also supports http Of , Of course , More postures , The boss has to dig himself .
General structure of the project
Careful friends will find , This is a .Net 6 Project , Because of its nuget package , Only support .Net 6 Of .
If you need it, you can change it yourself .
Projects are examples of reference sources , A change , Friends in need go directly to see official cases
Project focus
Install-Package DotNext.Net.Cluster -Version 4.6.0
DataModifier.cs
internal sealed class DataModifier : BackgroundService
{
private readonly IRaftCluster cluster;
private readonly ISupplier<long> valueProvider;
public DataModifier(IRaftCluster cluster, ISupplier<long> provider)
{
this.cluster = cluster;
valueProvider = provider;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
await Task.Delay(1000, stoppingToken).ConfigureAwait(false);
var leadershipToken = cluster.LeadershipToken;
TitleInfo.Show(!leadershipToken.IsCancellationRequested);
if (!leadershipToken.IsCancellationRequested)
{
var newValue = valueProvider.Invoke() + 500L;
Console.WriteLine(" Save the value generated by the leader node {0}", newValue);
var source = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken, leadershipToken);
try
{
var entry = new Int64LogEntry { Content = newValue, Term = cluster.Term };
await cluster.ReplicateAsync(entry, source.Token);
}
catch (Exception e)
{
Console.WriteLine(" Unknown exception {0}", e);
}
finally
{
source?.Dispose();
}
}
}
}
}
This should be the core service , It will communicate with other clients and conduct specific elections , And log transmission
Program.cs
class Program
{
static async Task Main(string[] args)
{
await UseTcpTransport(Path.Combine(AppContext.BaseDirectory, "raftConfig"));
}
static Task UseTcpTransport(string path)
{
// Get all configurations
var jsonConfiguration = new ConfigurationBuilder().SetBasePath(Environment.CurrentDirectory).AddJsonFile("appsettings.json", optional: true, reloadOnChange: true).Build();
var NodeInfo = new NodeInfo();
jsonConfiguration.Bind("NodeInfo", NodeInfo);
Console.WriteLine($"MainNode:{NodeInfo.MainNode}");
TitleInfo.Node = NodeInfo.MainNode;
var configuration = new RaftCluster.TcpConfiguration(IPEndPoint.Parse(NodeInfo.MainNode))
{
RequestTimeout = TimeSpan.FromMilliseconds(140),
LowerElectionTimeout = 150,
UpperElectionTimeout = 300,
TransmissionBlockSize = 4096,
ColdStart = false,
};
// Load all addresses
// The online environment rewrites the service itself
var builder = configuration.UseInMemoryConfigurationStorage().CreateActiveConfigurationBuilder();
foreach (var item in NodeInfo.Nodes)
{
var address = IPEndPoint.Parse(item);
builder.Add(ClusterMemberId.FromEndPoint(address), address);
}
builder.Build();
TitleInfo.Show();
return UseConfiguration(configuration, path);
}
static async Task UseConfiguration(RaftCluster.NodeConfiguration config, string? persistentStorage)
{
var loggerFactory = new LoggerFactory();
var loggerOptions = new ConsoleLoggerOptions
{
LogToStandardErrorThreshold = LogLevel.Warning
};
loggerFactory.AddProvider(new ConsoleLoggerProvider(new FakeOptionsMonitor<ConsoleLoggerOptions>(loggerOptions)));
config.LoggerFactory = loggerFactory;
using var cluster = new RaftCluster(config);
cluster.LeaderChanged += ClusterConfigurator.LeaderChanged;
var modifier = default(DataModifier?);
if (!string.IsNullOrEmpty(persistentStorage))
{
var state = new SimplePersistentState(persistentStorage, new AppEventSource());
cluster.AuditTrail = state;
modifier = new DataModifier(cluster, state);
}
await cluster.StartAsync(CancellationToken.None);
await (modifier?.StartAsync(CancellationToken.None) ?? Task.CompletedTask);
// The console waits for cancellation
using var handler = new CancelKeyPressHandler();
Console.CancelKeyPress += handler.Handler;
await handler.WaitAsync();
Console.CancelKeyPress -= handler.Handler;
// Out of Service
await (modifier?.StopAsync(CancellationToken.None) ?? Task.CompletedTask);
await cluster.StopAsync(CancellationToken.None);
}
}
On the whole , The project is still very simple .
I configured the address of the client
appsettings.json
This structure should be easy to understand , One is the address of the current end , One is the address of all nodes , Of course, it also includes the current address .
{
"NodeInfo": {
"MainNode": "127.0.0.1:6001" ,
"Nodes": [
"127.0.0.1:6001",
"127.0.0.1:6002",
"127.0.0.1:6003"
]
}
}
Operation mode
I am a Bin Three copies of the directory , Per appsettings.json Modify the , then , double-click RaftDemo.exe It's running .
Be careful
If you use a node, there is no egg , At least two nodes .
Running effect
summary
This library can be used in production environment , therefore , It's still worth studying .
Code address
https://github.com/kesshei/RaftDemo.git
https://gitee.com/kesshei/RaftDemo.git
Reference documents
https://zinglix.xyz/2020/06/25/raft/
https://github.com/dotnet/dotNext/tree/master/src/cluster
reading
One button three times !, Thank you for your support , Your support is my motivation !
another
For about a month , There are also a few iron powders , It will last longer , But it's more tiring in a row ( It's more unbearable by day ).
Thank you for your support .
边栏推荐
- 【逆向】脱壳后修复IAT并关闭ASLR
- ByteDance overseas technical team won the championship again: HD video coding has won the first place in 17 items
- 灵活报表v1.0(简单版)
- mysql的合计/统计函数
- [VNCTF 2022]ezmath wp
- DataGridView scroll bar positioning in C WinForm
- Serial serialold parnew of JVM garbage collector
- [reverse intermediate] eager to try
- 04个人研发的产品及推广-数据推送工具
- mysql高級(索引,視圖,存儲過程,函數,修改密碼)
猜你喜欢
Flink analysis (I): basic concept analysis
Development and practice of lightweight planning service tools
Concept and basic knowledge of network layering
03 products and promotion developed by individuals - plan service configurator v3.0
Connect to LAN MySQL
EasyRE WriteUp
Pyspark operator processing spatial data full parsing (4): let's talk about spatial operations first
Serial serialold parnew of JVM garbage collector
自动化运维利器ansible基础
Yarn: unable to load file d:\programfiles\nodejs\yarn PS1, because running scripts is prohibited on this system
随机推荐
Xin'an Second Edition: Chapter 23 cloud computing security requirements analysis and security protection engineering learning notes
07个人研发的产品及推广-人力资源信息管理系统
Flink源码解读(二):JobGraph源码解读
04个人研发的产品及推广-数据推送工具
Xin'an Second Edition: Chapter 26 big data security demand analysis and security protection engineering learning notes
06 products and promotion developed by individuals - code statistical tools
沉淀下来的数据库操作类-C#版(SQL Server)
JUnit unit test
华为认证云计算HICA
信息与网络安全期末复习(完整版)
How does wechat prevent withdrawal come true?
Final review of information and network security (based on the key points given by the teacher)
Huawei certified cloud computing hica
TCP连接不止用TCP协议沟通
Automatic operation and maintenance sharp weapon ansible Foundation
PySpark算子处理空间数据全解析(5): 如何在PySpark里面使用空间运算接口
Redis quick start
06个人研发的产品及推广-代码统计工具
Redis installation on centos7
自动化运维利器-Ansible-Playbook