Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.

Describe the bug while running hivemind albert experiment, we have one monitor peer and two worker peers. One of the nodes is working fine

But the other peer is stack at downloading parameters from peer. The reason I guess the is the speed of training. If first node train too fast then the other node cannot join, stuck in the download parameter. Can we limit the training speed or force the first node to wait others to join?

To Reproduce If applicable, please create a minimal script that reproduces the problem for you. It would be great to include script outputs as well.

If we change the albert to bert in the example, the speed for each iteration would be faster, then the new worker cannot join the training.

Environment Please list:

• python version (e.g. 3.8.1); 3.8
• hivemind.version; 1.1.0.dev0

#242 In this PR we are trying to get rid of GRPC in MoE module of hivemind. I do this PR draft now to be able to start review process and do it piece by piece, because some core thing are done and some results are achieved (I believe). There is still work to be done (it will be mentioned at the end of this message).

What is currently done:

• RemoteExperts are able to communicate through libp2p
• Some throughput performance optimizations were done: we achieved ~2GiB/sec at ffn_forward benchmark using GTX3060. It is done by enabling balancing multiple handlers of the same protocol in p2pd. Thus we can do packing/unpacking of messages on forward/backward in parallel. Results are at the end of the message
• Examples from hivemind tutorial are working
• tests/test_dht_experts.py are passing

What are topics to discuss:

• [x] Current implementation of RemoteExpert is quite heavy compared to GRPC version. GRPC version was in fact a few string fields containing endpoints, nothing more. Current version contains heavy object: connection to p2p-daemon, which is not serializable. Probably this is not the best decision.
• [x] Current moe/server/Server will not probably work if it has no DHT value inside, however old api documentation says that DHT is not mandatory for Server instance. There are two ways I see: make DHT mandatory or create a P2P instance inside a Server. Doing second option DHT inside Server probably might not be useful anymore

What is yet to be done:

• [x] Separate RemoteExpertInfo with endpoints from RemoteExpert and make function to make expert from its info
• [x] Separate thread and queue for async actions from _RemoteModuleCall
• [x] tests/test_moe.py and tests/test_training.py. And probably some other tests.
• [x] Rewrite RemoteMixtureOfExpert to the P2P (as RemoteExpert)
• [x] Refactoring of Server instance after discussion mentioned above
• [x] Wildly benchmark every possible scenario
• [x] FIx documentation for tutorials
• [x] Add test on scenarios, which might be not covered ~~After this PR is done we have to discuss dht.replicate_p2p() because it is not fork-safe and not in an obvious way~~

Current benchmarks results:

Current benchmarks were performed with GTX3060 and --preset ffn_forward. Each experiment was performed at least 5 times and results bellow are average. On demand I can provide more detailed data. Also worth mentioning that this results can change during review process.

| Branch | Batch size | Number of handlers | Throughput | | ---------------- | -------------- | -------------------------- | -------------------- | | server-p2p | 1024 | 1 | ~838 MiB/sec | | server-p2p | 1024 | 5 | ~2085 MiB/sec | | server-p2p | 1024 | 10 | ~2055 MiB/sec | | master | 1024 | *default | ~1526 MiB/sec | | master | 2048 | *default | ~2248 MiB/sec |

*default in column Number of handlers: TLDR it is 64. It means formula max(1, num_handlers or num_clients // 2) where num_handlers = None and num_clients = 128`.

What can be done after merging this

Thing discovered during review. They are not blocking this PR, but it is better to do them.

• [ ] Get rid of multiaddrs everywhere. P2P daemon should be able to communicate using PeerID only
• [ ] In some places there are CPU-bound thing happening inside async task. It is better to move them into thread executors. For example hivemind/moe/client/expert.py forward/Backward
• [ ] currently hivemind.Server does not check that inputs are correct. If user sends malformed inputs, it may OOM the server. We should check for that in some future PR. See #3
• [ ] if clients sends tensor of shape [0, 123], it will be split into zero messages and uid will not be passed. Server will receive uid=None and fail with cryptic KeyError(None). We should either forbid this on client side or ensure that zero-element tensors are serialized into a stream with first emty message.
• [ ] Test load balancing for unary handlers on python side

Error log from client-mode peer:

[2021/08/16 22:54:50.049][INFO][optim.collaborative.step:229] Beginning global optimizer step 0
[2021/08/16 22:54:50.253][INFO][optim.collaborative.fetch_collaboration_state:444] Collaboration accumulated 3696 samples from 2 peers; ETA 0.00 seconds (refresh in 0.50s.)
[2021/08/16 22:54:50.445][INFO][optim.collaborative.fetch_collaboration_state:444] Collaboration accumulated 3696 samples from 2 peers; ETA 0.00 seconds (refresh in 0.50s.)
/usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:87: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
[2021/08/16 22:56:57.350][ERROR][averaging.averager._run_allreduce:426] 
Traceback (most recent call last):
  File "/content/hivemind/hivemind/averaging/averager.py", line 416, in _run_allreduce
    averaging_outputs = [output async for output in allreduce]
  File "/content/hivemind/hivemind/averaging/averager.py", line 416, in <listcomp>
    averaging_outputs = [output async for output in allreduce]
  File "/content/hivemind/hivemind/averaging/allreduce.py", line 132, in run
    async for averaged_tensor_delta in self.tensor_part_container.iterate_output_tensors():
  File "/content/hivemind/hivemind/averaging/partition.py", line 134, in iterate_output_tensors
    await self._output_part_available[peer_index].wait()
  File "/usr/lib/python3.7/asyncio/locks.py", line 293, in wait
    await fut
concurrent.futures._base.CancelledError
[2021/08/16 22:56:57.352][INFO][optim.collaborative.step:250] Skipped averaging: averaging round failed with TimeoutError().
[2021/08/16 22:56:57.368][INFO][optim.collaborative.step:266] Optimizer step: done!

Error log from regular peer:

[2021/08/16 22:55:12.202][INFO][optim.collaborative.step:229] Beginning global optimizer step 0
[2021/08/16 22:55:12.212][INFO][optim.collaborative.fetch_collaboration_state:442] Collaboration accumulated 3856 samples from 2 peers; ETA 0.00 seconds (refresh in 0.50s.)
[2021/08/16 22:56:57.266][ERROR][averaging.averager._run_allreduce:426]
Traceback (most recent call last):
  File "/storage/hdd1/jheuristic/exp/testTPU/hivemind/hivemind/averaging/averager.py", line 416, in _run_allreduce
    averaging_outputs = [output async for output in allreduce]
  File "/storage/hdd1/jheuristic/exp/testTPU/hivemind/hivemind/averaging/averager.py", line 416, in <listcomp>
    averaging_outputs = [output async for output in allreduce]
  File "/storage/hdd1/jheuristic/exp/testTPU/hivemind/hivemind/averaging/allreduce.py", line 132, in run
    async for averaged_tensor_delta in self.tensor_part_container.iterate_output_tensors():
  File "/storage/hdd1/jheuristic/exp/testTPU/hivemind/hivemind/averaging/partition.py", line 134, in iterate_output_tensors
    await self._output_part_available[peer_index].wait()
  File "/home/jheuristic/anaconda3/envs/TPU/lib/python3.9/asyncio/locks.py", line 226, in wait
    await fut
asyncio.exceptions.CancelledError
[2021/08/16 22:56:57.268][ERROR][averaging.averager._step:365] Averager caught MatchmakingException('Unable to run All-Reduce: ')
Traceback (most recent call last):
  File "/storage/hdd1/jheuristic/exp/testTPU/hivemind/hivemind/averaging/averager.py", line 416, in _run_allreduce
    averaging_outputs = [output async for output in allreduce]
  File "/storage/hdd1/jheuristic/exp/testTPU/hivemind/hivemind/averaging/averager.py", line 416, in <listcomp>
    averaging_outputs = [output async for output in allreduce]
  File "/storage/hdd1/jheuristic/exp/testTPU/hivemind/hivemind/averaging/allreduce.py", line 132, in run
    async for averaged_tensor_delta in self.tensor_part_container.iterate_output_tensors():
  File "/storage/hdd1/jheuristic/exp/testTPU/hivemind/hivemind/averaging/partition.py", line 134, in iterate_output_tensors
    await self._output_part_available[peer_index].wait()
  File "/home/jheuristic/anaconda3/envs/TPU/lib/python3.9/asyncio/locks.py", line 226, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/storage/hdd1/jheuristic/exp/testTPU/hivemind/hivemind/averaging/averager.py", line 348, in _step
    await asyncio.wait_for(
  File "/home/jheuristic/anaconda3/envs/TPU/lib/python3.9/asyncio/tasks.py", line 492, in wait_for
    fut.result()
  File "/storage/hdd1/jheuristic/exp/testTPU/hivemind/hivemind/averaging/averager.py", line 427, in _run_allreduce
    raise MatchmakingException(f"Unable to run All-Reduce: {e}")
hivemind.averaging.matchmaking.MatchmakingException: Unable to run All-Reduce:
[2021/08/16 22:56:57.270][INFO][optim.collaborative.step:250] Skipped averaging: averaging round failed with MatchmakingException('Unable to run All-Reduce: ').
[2021/08/16 22:56:57.301][INFO][optim.collaborative.step:266] Optimizer step: done!

Describe the bug while running hivemind albert experiment, we have one monitor peer and two worker peers. One of the nodes is working fine

But the other peer is stack at downloading parameters from peer peer log is:

[2021/11/01 07:21:50.962][INFO][averaging.averager._load_state_from_peers:577] Downloading parameters from peer QmYQsw4kqPujvWv52sFsCorZs69LNhkxAsgBhBwGbaFfez
[2021/11/01 07:28:17.871][INFO][averaging.averager._load_state_from_peers:597] Finished downloading state from QmYQsw4kqPujvWv52sFsCorZs69LNhkxAsgBhBwGbaFfez


/opt/conda/lib/python3.9/site-packages/transformers/trainer.py:1347: FutureWarning: Non-finite norm encountered in torch.nn.utils.clip_grad_norm_; continuing anyway. Note that the default behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.

  nn.utils.clip_grad_norm_(

[2021/11/01 07:28:18.759][INFO][__main__.on_step_end:153] Step 0

[2021/11/01 07:28:18.760][INFO][__main__.on_step_end:154] Your current contribution: 0 samples

[2021/11/01 07:28:18.760][INFO][__main__.on_step_end:155] Performance: 0.002546124167199564 samples per second.

[2021/11/01 07:28:18.760][INFO][__main__.on_step_end:157] Local loss: 11.4107

[2021/11/01 07:28:18.986][INFO][optim.collaborative.fetch_collaboration_state:442] Collaboration accumulated 81 samples from 1 peers; ETA 36.99 seconds (refresh in 9.25s.)

[2021/11/01 07:28:19.004][INFO][optim.collaborative.step:208] Peer is out of sync.

[2021/11/01 07:28:20.243][INFO][averaging.averager._load_state_from_peers:577] Downloading parameters from peer QmXpVXnAY6L7WqeW4pzstGK18S1LySDonPmrxQka3GztJa

To Reproduce the monitor running script: python run_training_monitor.py --host_maddrs '/ip4/0.0.0.0/tcp/38888' --experiment_prefix albert --wandb_project albert

the worker peer script: python run_trainer.py --experiment_prefix albert --host_maddrs '/ip4/0.0.0.0/tcp/39997' --initial_peers [INITIAL_PEERS_FROM_MONITOR] --seed 42 --logging_first_step --logging_steps 100 --output_dir /train --overwrite_output_dir --logging_dir /train --target_batch_size 1024 --averaging_expiration 10 --per_device_train_batch_size 1 --gradient_accumulation_steps 1

Environment I was running this experiment in a docker container Please list:

• python version 3.9.7
• hivemind.version; 0.10.0
• Please copy and paste the output from pytorch [environment collection script]

Collecting environment information...
PyTorch version: 1.10.0
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.11.0-37-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 3090
Nvidia driver version: 460.91.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.3
[pip3] pytorch-ranger==0.1.1
[pip3] torch==1.10.0
[pip3] torch-optimizer==0.3.0
[pip3] torchaudio==0.10.0
[pip3] torchvision==0.11.1
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.3.0           h06a4308_520  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] mypy-extensions           0.4.3                    pypi_0    pypi
[conda] numpy                     1.21.3                   pypi_0    pypi
[conda] numpy-base                1.21.2           py39h79a1101_0  
[conda] pytorch                   1.10.0          py3.9_cuda11.1_cudnn8.0.5_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] pytorch-ranger            0.1.1                    pypi_0    pypi
[conda] torch                     1.10.0                   pypi_0    pypi
[conda] torch-optimizer           0.3.0                    pypi_0    pypi
[conda] torchaudio                0.10.0                   pypi_0    pypi
[conda] torchvision               0.11.1                   pypi_0    pypi

Considering the file transfer speed, I tested the bandwidth with iperf:

------------------------------------------------------------
Client connecting to 10.8.0.4, TCP port 5001
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[  3] local 10.8.0.5 port 39674 connected with 10.8.0.4 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.4 sec  21.0 MBytes  16.9 Mbits/sec

In our preliminary experiments, all peers have independent gating functions and we can only synchronize them manually. It would be great to implement some sort of builtin averaging mechanism.

For instance, every T seconds, assemble peers into groups at random, then perform all-reduce within each group. In case of failure, rollback and repeat T seconds later.

Hi there,

In some experiments, I face the situation where one gpu is lost during the training. And I have to restart the work again. Have ever encountered that issue? Thank you.

[depends on #238 to be merged ] After we've implemented P2P transport with nat traversal, we should switch the main components to libp2p backend to take advantage of this new transport.

One of three main components is hivemind.server.Server and its counterpart hivemind.client.RemoteExpert

On a client side, hivemind creates a RemoteExpert pytorch module that calls experts via _RemoteModuleCall (and _RemoteCallMany for DMoE)

A server receives incoming connections with several ConnectionHandler processes running in parallel. These processes run gRPC servers and hence should be switched to libp2p.

• Checklist
  • [x] find some way to attach several processes to one RPC (as in server/connection_handler.py)
  • [ ] make sure it passes tests/test_moe.py
  • [ ] make sure it passes tests/test_training.py
  • [x] tune performance in tests/benchmark_througphput.py

Describe the bug I run the albert example with wikitext data. I use one peer, default settings (target_batch_size=4096, train_batch_size=4, max_step=125000, lr=0.00176), but the loss did not decrease after training, it start as 11 and finish as 11.

Jan 15 10:30:14.734 [INFO] Step #1 loss = 11.04938 Jan 15 10:32:14.842 [INFO] Step #2 loss = 11.05589 Jan 15 10:34:14.975 [INFO] Step #3 loss = 11.06803 Jan 15 10:36:15.093 [INFO] Step #4 loss = 11.06271 Jan 15 10:38:15.228 [INFO] Step #5 loss = 11.06433 Jan 15 10:40:15.337 [INFO] Step #6 loss = 11.05447 Jan 15 10:41:45.401 [INFO] Step #7 loss = 11.06115 Jan 15 10:43:45.541 [INFO] Step #8 loss = 11.06025 .......... Jan 15 18:09:13.117 [INFO] Step #238 loss = 11.05597 Jan 15 18:11:13.233 [INFO] Step #239 loss = 11.06724 Jan 15 18:13:13.369 [INFO] Step #240 loss = 11.06289 Jan 15 18:15:13.494 [INFO] Step #241 loss = 11.05922 Jan 15 18:16:43.577 [INFO] Step #242 loss = 11.05226 Jan 15 18:18:43.691 [INFO] Step #243 loss = 11.05418 Jan 15 18:20:43.843 [INFO] Step #244 loss = 11.05638

To Reproduce Run the script in albert example. For monitor, I run:

python run_training_monitor.py
--experiment_prefix albert_experiment
--wandb_project albert_wandb

For trainer, I run:

IP=/ip4/192.168.0.188/tcp/45731/p2p/QmSRerwCPUSreHhwMuTLHoVHqTfWuT8J57w3sXFZtU8ECo

WANDB_DISABLED=true CUDA_VISIBLE_DEVICES=0 python run_trainer.py
--experiment_prefix albert_experiment
--initial_peers $IP
--logging_first_step
--output_dir ./outputs
--overwrite_output_dir
--logging_dir ./logs
--dataset_path="/home/protago/Xiangpeng/hivemind/examples/albert/data/albert_tokenized_wikitext"
--per_device_train_batch_size 4
--learning_rate 0.00176
--num_train_epochs=5
--save_steps=60000

Environment Please list:

• python version (e.g. 3.8.1); 3.8
• hivemind.version; 1.1.0
• Please copy and paste the output from pytorch environment collection script

If the script doesn't work, please report pytorch and numpy versions manually. We also encourage you to include any additional information that you believe can help us solve the issue.

Is your feature request related to a problem? Please describe.

Eh, this could be a question. I'm trying to use TrainingAverager with step(wait=False). That requires data_lock and use_old_local_tensor=True follows.

When use_old_local_tensor=True, is it correct to simply add the weight difference between local model and all-reduced model to the new model parameters? The gradients calculated from the old model parameter is being added to the new model parameters. That doesn't seem quite right.

Describe the solution you'd like

https://arxiv.org/abs/2101.06840 proposes Delayed Parameter Update. Parameter update is delayed by one step. Apparently, it makes little difference in the training curve if DPU is applied after 40 iterations in BERT-large training.

I think to implement DPU, you simply have to copy back the averaged tensor back to the model in the beginning of step().

Describe alternatives you've considered

I understand that if the weight difference is not added back, the local steps taken before the asynchronous all-reduce completes are being wasted. Not only it defeats purpose of asynchronous all-reduce(if local updates are going to be wasted until async completes, why not just go sync) but it also skips over input data which could trouble training.

This change seems to speed up (a) DHT get requests by 3.6x and (b) DHT creation by 1.2x (probably due to speeding up the communication with initial nodes).

benchmark_dht.py

nora, this PR, max_workers = 8:

nora, master (fb4813347a18a01d2c780232a5f86266bbd49d26, see #318), max_workers = 1:

Let's implement a basic example for collaborative training with ALBERT

• core training code ( @leshanbog )

  • [x] implement basic training scripts (run_first_peer/run_trainer) based on mryab/collaborative-training
  • [x] achieve exact match with old training code
  • [x] test fault tolerance against common network failures

• update metric logging code (@yhn112 )

  • [x] tune first peer's wandb to avoid crashes (or restart on crashes)
  • [x] use the same DHT key prefix for metrics and averaging (aka self.prefix)
  • [x] make the prototype pep8-compliant

• add basic security layer: (@borzunov )

  • [x] protect value types in:
    • [x] progress
    • [x] averaging
    • [x] metrics
  • [x] ensure that the averager validate reasonable min/max values (i.e. for batches_processed)
  • [x] make sure it supports DataParallel on peers
  • [x] make sure it will save/load scheduler state dict correctly with optimizer

• add full description in README.md

Describe the bug

Jan 04 22:30:14.302 [INFO] test-run-1112b accumulated 10 samples for epoch #0 from 2 peers. ETA 0.00 sec (refresh in 0.50 sec)
Jan 04 22:30:14.476 [INFO] Beginning optimizer step #0
Jan 04 22:31:26.924 [ERROR] [hivemind.optim.power_sgd_averager._aggregate_with_group:187] Expected out tensor to have dtype c10::BFloat16, but got float instead
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/hivemind/optim/power_sgd_averager.py", line 159, in _aggregate_with_group
    torch.matmul(m.reshape(-1, q.size(0)), q, out=p)
RuntimeError: Expected out tensor to have dtype c10::BFloat16, but got float instead
Jan 04 22:31:26.925 [WARN] [hivemind.optim.power_sgd_averager._register_allreduce_group:129] All-reduce group b'\xc4\xaf\xafv&\xcd\xd6q\xa3\xea\x9d-\x13\x0f\xa4hNQ\xf6>PHASE_P' did not finish.
Jan 04 22:31:26.925 [WARN] [hivemind.optim.power_sgd_averager._register_allreduce_group:129] All-reduce group b'\xc4\xaf\xafv&\xcd\xd6q\xa3\xea\x9d-\x13\x0f\xa4hNQ\xf6>PHASE_Q' did not finish.
Jan 04 22:31:26.925 [WARN] [hivemind.averaging.averager._step:482] PowerSGDGradientAverager caught MatchmakingException('Unable to run All-Reduce: Expected out tensor to have dtype c10::BFloat16, but got float instead'), retrying
Jan 04 22:35:47.094 [ERROR] [hivemind.optim.power_sgd_averager._aggregate_with_group:187] Expected out tensor to have dtype c10::BFloat16, but got float instead
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/hivemind/optim/power_sgd_averager.py", line 159, in _aggregate_with_group
    torch.matmul(m.reshape(-1, q.size(0)), q, out=p)
RuntimeError: Expected out tensor to have dtype c10::BFloat16, but got float instead
Jan 04 22:35:47.094 [WARN] [hivemind.optim.power_sgd_averager._register_allreduce_group:129] All-reduce group b'Q\x84\x8d\xa9\xf3\x90\xd4\xdf\xcc]\x153\x0c+\x9e\x90|\xed|\x8ePHASE_P' did not finish.
Jan 04 22:35:47.094 [WARN] [hivemind.optim.power_sgd_averager._register_allreduce_group:129] All-reduce group b'Q\x84\x8d\xa9\xf3\x90\xd4\xdf\xcc]\x153\x0c+\x9e\x90|\xed|\x8ePHASE_Q' did not finish.
Jan 04 22:35:47.095 [WARN] [hivemind.averaging.averager._step:482] PowerSGDGradientAverager caught MatchmakingException('Unable to run All-Reduce: Expected out tensor to have dtype c10::BFloat16, but got float instead'), retrying
Jan 04 22:40:07.221 [ERROR] [hivemind.optim.power_sgd_averager._aggregate_with_group:187] Expected out tensor to have dtype c10::BFloat16, but got float instead

To Reproduce

git clone https://github.com/the-beee/naifu-diffusion
cd naifu-diffusion
pip install -r requirements.txt
python trainer.py

Please update config/distributed.yaml to include the peers address in the hivemind section, before starting the second peer.

Environment

Collecting environment information...
PyTorch version: 1.13.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] (64-bit runtime)
Python platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] pytorch-lightning==1.8.6
[pip3] torch==1.13.1
[pip3] torch-ema==0.3
[pip3] torchmetrics==0.11.0
[pip3] torchvision==0.14.1
[pip3] hivemind==1.1.4
[conda] Could not collect

This PR applies Codespell to the repo and attempts to fix most of the typos found by this tool; the rest are debatable. Also, it adds Codespell to CI to prevent (or at least highlight) future typos, you can see that it works by navigating the PR diff or the diff for this commit.

When installing hivemind (as a dependency of petals) using pipenv, pipenv failed to resolve a valid version for protobuf. Could not find a version that matches protobuf<4.0.0,<4.0dev,<5.0dev,>=3.12.2,>=3.20.3,>=4.21.6

Here's the trimmed dependency graph for hivemind to show the conflicts:

- hivemind [required: ==1.1.3, installed: 1.1.3]
  - grpcio-tools [required: >=1.33.2, installed: 1.51.1]
    - protobuf [required: <5.0dev,>=4.21.6, installed: 3.20.3]
  - protobuf [required: <4.0.0,>=3.12.2, installed: 3.20.3]

I haven't tested if this causes any actual issues, but it looks risky.

Found in https://github.com/bigscience-workshop/petals/pull/150/files by @borzunov

TL;DR ModuleBackend's contain TaskPools as properties, but TaskPools refer to ModuleBackend's instance methods (e.g. self.forward)

This is harmless for run_server, but will potentially cause memory leaks if server is deleted and recreated.

Hi,

I can't seem to be able to read the training information (like here) out of the DHT that was created by hivemind.

I can connect to the DHT and run the following:

> dht.store("key", "value", expiration=get_dht_time() + 600)
> dht.get("key")
ValueWithExpiration(value='value', expiration_time=1670845892.2483625)

However, when training with hivemind, I can't seem to be able to get the data with two different behaviors after calling the get function after each other.

Only the second call shows some actual training progress data, but not complete (1 out of 4 peers) and not in a way that allows me to access it compared to the documentation.

It seems that there is some issue with the get call being run asynchronously and not being able to decode the returning LocalTrainingProgress.

How does the tutorial data get/store differ from what hivemind does with the LocalTrainingProgress?

First call to get

>>> dht.get("hivemind-123_progress")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/miniconda3/envs/conda-hivemind/lib/python3.9/site-packages/hivemind/dht/dht.py", line 173, in get
    return future if return_future else future.result()
  File "/home/ubuntu/miniconda3/envs/conda-hivemind/lib/python3.9/site-packages/hivemind/utils/mpfuture.py", line 257, in result
    return super().result(timeout)
  File "/home/ubuntu/miniconda3/envs/conda-hivemind/lib/python3.9/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/home/ubuntu/miniconda3/envs/conda-hivemind/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
msgpack.exceptions.ExtraData: unpack(b) received extra data.

Second call to get

>>> dht.get("hivemind-123_progress")
Dec 12 12:43:20.841 [ERROR] [asyncio._run:129] Task exception was never retrieved
future: <Task finished name='Task-13381' coro=<DHT._get() done, defined at /home/ubuntu/miniconda3/envs/conda-hivemind/lib/python3.9/site-packages/hivemind/dht/dht.py:175> exception=ExtraData({'peer_id': b"\x12 W\xb23\xa4\x85\xd0\xfa\xad\n[t\xec\xc7\xfe'\xed\x1d\x94\x03\n\xf6\x11e\xf4\xe3j,\xf7\xae\xd5h\xca", 'epoch': 24, 'samples_accumulated': 0, 'samples_per_second': 10.078083213276257, 'time': 1670842945.1815588, 'client_mode': False}, b'[signature:P3NGbBDc4ujJwy2afKJSEXD/lsM1s7icix+h5LoxGk1K6ZFvq5vaf7vs4mokUm0TmYbeGMq85DV1M3nr/+lrVg/WGAtC3moq9iiigaKiNnhszcZPx1ls+UOoIbZXGh35kdIzCIr2qsV9GxheuPaohErMoEzxN+kAytZ+wEtxoxEgOCAXEdOGVmee0Dx6eIQVzs96d7aIEpucNLGRu8ylOvgjcZNOu+MMyqVTom3R6yvl8RRTh3Dj/0cS7a0ajo+osIx7ENIadL8Zh8Vqmw+evLR2dZhAULYhN/wq1C/8dNYZzM1C2spbjG9hMYlD33RUhmD0gE+rWP0OKHA7vUPtSA==]')>
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/conda-hivemind/lib/python3.9/site-packages/hivemind/dht/dht.py", line 177, in _get
    result = await self._node.get(key, latest=latest, **kwargs)
  File "/home/ubuntu/miniconda3/envs/conda-hivemind/lib/python3.9/site-packages/hivemind/dht/node.py", line 543, in get
    result = await self.get_many([key], **kwargs)
  File "/home/ubuntu/miniconda3/envs/conda-hivemind/lib/python3.9/site-packages/hivemind/dht/node.py", line 565, in get_many
    results_by_id = await self.get_many_by_id(key_ids, sufficient_expiration_time, **kwargs)
  File "/home/ubuntu/miniconda3/envs/conda-hivemind/lib/python3.9/site-packages/hivemind/dht/node.py", line 620, in get_many_by_id
    search_results[key_id].add_candidate(self.protocol.storage.get(key_id), source_node_id=self.node_id)
  File "/home/ubuntu/miniconda3/envs/conda-hivemind/lib/python3.9/site-packages/hivemind/dht/node.py", line 844, in add_candidate
    self.finish_search()
  File "/home/ubuntu/miniconda3/envs/conda-hivemind/lib/python3.9/site-packages/hivemind/dht/node.py", line 873, in finish_search
    self.serializer.loads(value_bytes), item_expiration_time
  File "/home/ubuntu/miniconda3/envs/conda-hivemind/lib/python3.9/site-packages/hivemind/utils/serializer.py", line 72, in loads
    return msgpack.loads(buf, ext_hook=cls._decode_ext_types, raw=False)
  File "msgpack/_unpacker.pyx", line 201, in msgpack._cmsgpack.unpackb
msgpack.exceptions.ExtraData: unpack(b) received extra data.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/miniconda3/envs/conda-hivemind/lib/python3.9/site-packages/hivemind/dht/dht.py", line 173, in get
    return future if return_future else future.result()
  File "/home/ubuntu/miniconda3/envs/conda-hivemind/lib/python3.9/site-packages/hivemind/utils/mpfuture.py", line 257, in result
    return super().result(timeout)
  File "/home/ubuntu/miniconda3/envs/conda-hivemind/lib/python3.9/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/home/ubuntu/miniconda3/envs/conda-hivemind/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
msgpack.exceptions.ExtraData: unpack(b) received extra data.