When farming for a pool using the new pool protocol, chiadog triggers hundreds of "found proof" notifications per day, because your farmer finds proofs for partials (low difficulty solutions for a pool) all the time. This gets you a lot of messages by a telegram bot for example. ;)

Suggestion: Teach chiadog to differentiate "normal" proofs from partial proofs and disable partial proof notifications. Daily notifications about partial proofs should be sufficient. Or just make proof notification deactivatale.

I added the feature #65 Let me know what you think. I'll upload this to my farmer for long term testing

It parses the log for farmer_server entries that indicate incoming data from harvesters. It saves the last timestamp, the peer hash and the ip address in a list. If one harvester doesnt send any data for more than 300s, it sends a notification and removes the harvester from the list.

Describe the bug

I'm running the Windows version of chiadog. Every now and then I'll get the "Your harvester appears offline! No events for the past xxxx seconds" notification, and I will continue getting this notification until I restart chiadog. When I check my farm (using chia farm summary) everything is fine. I actually won a block during one of these fake outages.

Looking at timestamps, these outages seem to always coincide with a log rotation, but not every rotation. My log rotates every 40-50 minutes, but this only happens around once a day or so - so it's definitely not with every rotation.

It was suggested I turn off any other processes tailing (using get-content) debug.log, which I've done, but it didn't help.

Environment:

• OS: Windows 10
• Python version: 3.9.4
• PIP version: 21.1
• Chia version: 1.1.3

I personally don't have Windows to test and am not familiar with how the chia environment there is setup but can provide guidance on the steps necessary to add such support.

Here are some jumping off points if someone wants to take over:

• We need a WIndowsLogConsumer that is analogous to the UNIX implementation in FileLogConsumer. It needs to replace the tail -F command with the analogous command for PowerShell.
  • Possibly something like Get-Content <log-file> -Tail <#-of-rows> -Wait
• Add windows_log_consumer config. See here for an example. It needs to point to default path on Windows where debug.log is stored.
• The project is using pathlib for handling the file paths. I believe this is compatible with Windows paths but needs to be tested.

I suggest doing a quick test by directly modifying the FileLogConsumer and changing the path in the config to see if there are any other showstoppers to make this work on Windows. That will already be very valuable feedback.

I got this telegram alerts : ⚠️ Chia FULL_NODE Experiencing networking issues? Skipped some signage points! Last 20/64, current 22/64.

1. What is the actual log for above WARNING? I want to trace it.
2. How to solve it? Is it normal to get 2-3 warning like above every 1 hour?
3. Will is affect my chia WIN rate?

Thanks.

Describe the bug If signage points are logged out of order, skip alerts are triggered.

Chia debug log:

2021-05-19T13:14:51.592 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 9/64: 99dc2878cd36e748f35caf8b48fc3e048dc1d2306c6e5675a51f5a6beaaf8826
2021-05-19T13:14:52.165 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 10/64: fbd8a6cbfd9ceb973842b998f9e221791495bb0b95bb594078ff9022dd0c64c4
2021-05-19T13:15:24.908 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 12/64: 3e0f1d609eef221f21a5bb8f5c5290380563d265da219f49fca47c4cb1b8392d
2021-05-19T13:15:24.912 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 11/64: 2855217036e581b98e22e8776b41e61495751a7b2a9da27976120fef39c65e5a
2021-05-19T13:15:24.914 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 13/64: a5b4b7c0456eea6711776dd7781a828bac626e49f8f5f3083e031e456ffea0c7
2021-05-19T13:15:25.875 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 14/64: 2971d057ac883958a8e873d06d507ff66c13b58b30bc7a20e08014e2fb4dce10
2021-05-19T13:15:34.107 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 15/64: 520b8c1cc85449fc82351528e63e232f8d6bd85dc58a6a1abb11d486ea3b3f8f
2021-05-19T13:15:43.902 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 16/64: a0aa4580f66ee139c222297acd04f618bc1f5cfaa7aab75d76f63237a2ecdb7f

Chiadog log from same time:

[2021-05-19 13:15:24] [    INFO] --- Detected 1 skipped signage point. (non_skipped_signage_points.py:46)
[2021-05-19 13:15:24] [ WARNING] --- Experiencing networking issues? Skipped 2+ signage points in the last hour (non_skipped_signage_points.py:53)
[2021-05-19 13:15:25] [    INFO] --- Last farming event was 32 seconds ago. This is unusual (time_since_last_farm_event.py:47)

Notice signage point 12 that is logged before 11 completes.

Environment:

• OS: Ubuntu 20.04
• Python version: 3.8.5
• PIP version: 20.0.2 for Chia, latest for chiadog (installed via install.sh)
• Chia version: 1.1.5
• Chiadog revision: 67328d7960

On Linux it looks like tail -F may not reliably follow new log files. When this happens, chiadog will report "Your harvester appears to be offline", and inspecting the /proc file system shows that the tail process is still opening the old log file, eg. .chia/mainnet/log/debug.log.11.

Environment:

• OS: Ubuntu 20.04
• Python version: 3.8.5

Fresh install of all software (OS, Chia and all requirements for this program). After about 21 hours of it running, it suddenly started to report "Your harvester appears to be offline! No events for the past 417 seconds. (keep_alive_monitor.py:83)". It kept sending these reports every 5 minutes with the seconds reported climbing accordingly. The issue is that the harvester is not offline. The Chia software shows it is still farming and if I check the logs everything looks fine. I stopped chiadog and started it again and it is all fine now.

Environment:

• OS: Windows 10 Pro
• Python version: 3.9.5 (64-bit)
• PIP version: 21.1.1
• Chia version: 1.1.5

Describe the bug

There are still cases when shuffled signage points during forks are not correctly handled.

The following sequence of events generates 61 skipped SP event.

2021-05-13T22:12:25.148 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 60/64: 
2021-05-13T22:12:35.081 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 61/64: 
2021-05-13T22:12:41.911 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 62/64: 
2021-05-13T22:12:49.784 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 63/64: 
2021-05-13T22:12:58.453 full_node chia.full_node.full_node: INFO     ⏲️  Finished sub slot, SP 64/64, 
2021-05-13T22:13:07.659 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 1/64: 
2021-05-13T22:13:14.053 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 2/64: 
2021-05-13T22:13:42.341 full_node chia.full_node.full_node: INFO     ⏲️  Finished sub slot, SP 64/64, 
2021-05-13T22:13:50.884 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 1/64: 
2021-05-13T22:13:58.247 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 2/64: 
2021-05-13T22:14:06.288 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 3/64: 
2021-05-13T22:14:14.181 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 4/64: 
2021-05-13T22:14:22.166 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 5/64: 
2021-05-13T22:14:30.104 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 6/64: 
2021-05-13T22:14:38.398 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 7/64: 

One way to fix would be comparing expected time for 61 SP (61 times 8-9 seconds) with actual time difference to previous SP.

Environment:

• Chia version: 1.1.5

I was trying to run this with WSL to see if that would work. It seems like there is an issue where you can't follow a file in WSL without the ---disable-inotify flag on tail. I tested this and added that option in the log consumer and it does work after that. I noticed there have been issues with Windows as well during log rotations. I'm not sure if switching over to a pythonic way of reading the debug file will solve this, but it seems like it would be better since you don't need differentiate between OSes either.

Not sure if remote would work with this either, so it would require a bit of testing.

https://github.com/martomi/chiadog/blob/da1a904466e6df25b930d58cefb70eca6dce4a3a/src/chia_log/log_consumer.py#L72-L80

Maybe something like: https://stackoverflow.com/questions/12523044/how-can-i-tail-a-log-file-in-python/53121178#53121178

Edit:

Initially something like this appears to work with little resources:

    def _consume_loop(self):
        expanded_user_log_path = self._log_path.expanduser()
        logging.info(f"Consuming log file from {expanded_user_log_path}")

        try:
            fp = open(expanded_user_log_path, "r")
            st_results = os.stat(expanded_user_log_path)
            st_size = st_results[6]
            fp.seek(st_size)

            while self._is_running:
                where = fp.tell()
                line = fp.readline()
                if not line:
                    time.sleep(1)
                    fp.seek(where)
                elif fp.tell() > os.path.getsize(expanded_user_log_path):
                    logging.info("Log rotated, opening latest log file")
                    fp.close()
                    fp = open(expanded_user_log_path, "r")
                    fp.seek(0,2)
                else:
                    self._notify_subscribers(line)
        except Exception as e:
            fp.close()
            logging.critical(f"Error consuming log file: {e}")

I can run it for a bit and see what happens during rotations as well. This seems to work the same for windows.

Hi,

thansk for proviging this nice tool. Really like it.

Just need some hints how to improve my setup -

• average: 1.12s over 2999 searches
• over 5s: 63 occasions (2.1%)
• over 15s: 43 occasions (1.4%) Plots 🌱: 1781, new: 16 Eligible plots 🥇: 3.47 average Skipped SPs ⚠️: 1 (0.03%)

I get lots of those alerts >5 sec, normally in the range of 20 - 30 sec., how can I see which plot is responsible for that?

Also - those numbers above, are they in general ok? Have not got any chia so far, just wondering if this is bad luck or the setup is weak.

Any feedback?

Regards Joerg

I'm trying to spin up a new instance via docker-compose. There doesn't seem to be much documentation.

Have tried the following:

version: "3.7"

services:

  chiadog:
    image: "ghcr.io/martomi/chiadog:latest"
    container_name: "chiadog"
    volumes:
      - ~/chiadog/config:/root/.chiadog
    environment:
      - TZ=$TZ

I've put a config.yaml into the config folder but I am getting this error:

/chiadog/venv/lib/python3.10/site-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
[2022-08-02 19:24:25] [    INFO] --- Starting Chiadog (unknown) (main.py:54)
[2022-08-02 19:24:25] [    INFO] --- Connected (version 2.0, client OpenSSH_8.9p1) (transport.py:1873)
Traceback (most recent call last):
  File "/chiadog/main.py", line 113, in <module>
    init(conf)
  File "/chiadog/main.py", line 58, in init
    log_consumer = create_log_consumer_from_config(chia_logs_config)
  File "/chiadog/src/chia_log/log_consumer.py", line 234, in create_log_consumer_from_config
    platform, path = get_host_info(
  File "/chiadog/src/chia_log/log_consumer.py", line 187, in get_host_info
    client.connect(hostname=host, username=user, port=port)
  File "/chiadog/venv/lib/python3.10/site-packages/paramiko/client.py", line 415, in connect
    self._policy.missing_host_key(
  File "/chiadog/venv/lib/python3.10/site-packages/paramiko/client.py", line 825, in missing_host_key
    raise SSHException(
paramiko.ssh_exception.SSHException: Server 'x.x.xx.xx' not found in known_hosts

Which suggests I need to feed the docker some SSH keys?

Chiadog was working fine for days, then it started reporting that my harvester was offline, but that was not the case. This is what I found in the chiadog logs:

...
[2022-07-14 04:26:00] [    INFO] --- Detected 1 skipped signage point.This is expected to happen occasionally and not a reason for concern. (non_skipped_signage_points.py:39)
Exception in thread Thread-1 (_consume_loop):
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/chiadog/venv/lib/python3.10/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "/chiadog/venv/lib/python3.10/site-packages/retry/api.py", line 73, in retry_decorator
    return __retry_internal(partial(f, *args, **kwargs), exceptions, tries, delay, max_delay, backoff, jitter,
  File "/chiadog/venv/lib/python3.10/site-packages/retry/api.py", line 33, in __retry_internal
    return f()
  File "/chiadog/src/chia_log/log_consumer.py", line 83, in _consume_loop
    for log_line in Pygtail(self._expanded_log_path, read_from_end=True, offset_file=self._offset_path):
  File "/chiadog/venv/lib/python3.10/site-packages/pygtail/core.py", line 136, in __next__
    return self.next()
  File "/chiadog/venv/lib/python3.10/site-packages/pygtail/core.py", line 108, in next
    line = self._get_next_line()
  File "/chiadog/venv/lib/python3.10/site-packages/pygtail/core.py", line 280, in _get_next_line
    line = self._filehandle().readline()
  File "/usr/local/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 0-1: unexpected end of data
[2022-07-14 06:16:37] [ WARNING] --- Your harvester appears to be offline! No events for the past 511 seconds. (keep_alive_monitor.py:83)
[2022-07-14 06:21:38] [ WARNING] --- Your harvester appears to be offline! No events for the past 812 seconds. (keep_alive_monitor.py:83)
[2022-07-14 06:26:39] [ WARNING] --- Your harvester appears to be offline! No events for the past 1112 seconds. (keep_alive_monitor.py:83)
[2022-07-14 06:31:39] [ WARNING] --- Your harvester appears to be offline! No events for the past 1413 seconds. (keep_alive_monitor.py:83)
[2022-07-14 06:36:40] [ WARNING] --- Your harvester appears to be offline! No events for the past 1714 seconds. (keep_alive_monitor.py:83)
[2022-07-14 06:41:40] [ WARNING] --- Your harvester appears to be offline! No events for the past 2014 seconds. (keep_alive_monitor.py:83)
...

I restarted chiadog's container and it started functioning properly again.

• OS: Linux Manjaro, kernel 5.19.0-1-MANJARO #1 SMP PREEMPT_DYNAMIC Mon Jun 6 15:12:01 CEST 2022 x86_64 GNU/Linux
• Python version: 3.10
• PIP version: 21.0
• Chia version: 1.4.0
• Chiadog version: 0.7.2, docker container
• Harvester: local

Hello,

I have just upgraded to the 0.7.2 (Ubuntu 20.04.4 LTS + Chia 1.4). I am using it to monitor the local node / harvester + a remote harvester.

I have edited the config-example.yaml file with the few parameters I needed then renamed it as config.yaml.

in the output04.log I get the following [2022-07-14 10:19:05] [ INFO] --- Starting Chiadog (v0.7.2) (main.py:54) [2022-07-14 10:19:05] [ INFO] --- Connected (version 2.0, client OpenSSH_8.2p1) (transport.py:1873) [2022-07-14 10:19:05] [ INFO] --- Authentication (publickey) successful! (transport.py:1873) [2022-07-14 10:19:05] [ INFO] --- Enabled Posix network log consumer. (log_consumer.py:128) [2022-07-14 10:19:05] [ INFO] --- Connected (version 2.0, client OpenSSH_8.2p1) (transport.py:1873) [2022-07-14 10:19:05] [ INFO] --- Authentication (publickey) successful! (transport.py:1873) [2022-07-14 10:19:05] [ INFO] --- Consuming remote log file ~/.chia/mainnet/log/debug.log from 192.xxx.xxx.xxx:22 (OS.LINUX) (log_consumer.py:116) [2022-07-14 10:19:05] [ INFO] --- Keep-alive check period: 300 seconds (keep_alive_monitor.py:45) [2022-07-14 10:19:05] [ INFO] --- Initializing Telegram notifier. (telegram_notifier.py:13) [2022-07-14 10:19:05] [ INFO] --- Enabled stats for daily notifications (stats_manager.py:48) [2022-07-14 10:19:05] [ INFO] --- Summary notifications will be sent out every 24 hours starting from 21:00 (stats_manager.py:61) [2022-07-14 10:19:05] [ ERROR] --- Invalid config.yaml. Missing key: 'harvester_activity_handler' (log_handler.py:22) [2022-07-14 10:19:05] [ INFO] --- Initializing handler: harvester_activity_handler (init.py:28) [2022-07-14 10:19:05] [ INFO] --- Enabled parser for harvester activity - eligible plot events. (harvester_activity_parser.py:32) [2022-07-14 10:19:05] [ INFO] --- Enabled check for farming events. (time_since_last_farm_event.py:22) [2022-07-14 10:19:05] [ INFO] --- Enabled check for non-decreasing total plot count. (non_decreasing_plots.py:18) [2022-07-14 10:19:05] [ INFO] --- Enabled check for time taken to respond to challenges. (quick_plot_search_time.py:18) [2022-07-14 10:19:05] [ ERROR] --- Invalid config.yaml. Missing key: 'partial_handler' (log_handler.py:22) [2022-07-14 10:19:05] [ INFO] --- Initializing handler: partial_handler (init.py:28) [2022-07-14 10:19:05] [ INFO] --- Enabled parser for partial submitting stats. (partial_parser.py:28) [2022-07-14 10:19:05] [ ERROR] --- Invalid config.yaml. Missing key: 'block_handler' (log_handler.py:22) [2022-07-14 10:19:05] [ INFO] --- Initializing handler: block_handler (init.py:28) [2022-07-14 10:19:05] [ INFO] --- Enabled parser for block found stats. (block_parser.py:28) [2022-07-14 10:19:05] [ INFO] --- Enabled check for found blocks. (found_blocks.py:15) [2022-07-14 10:19:05] [ ERROR] --- Invalid config.yaml. Missing key: 'finished_signage_point_handler' (log_handler.py:22) [2022-07-14 10:19:05] [ INFO] --- Initializing handler: finished_signage_point_handler (init.py:28) [2022-07-14 10:19:05] [ INFO] --- Enabled parser for finished signage points. (finished_signage_point_parser.py:28) [2022-07-14 10:19:05] [ INFO] --- Enabled check for finished signage points. (non_skipped_signage_points.py:19) [2022-07-14 10:19:05] [ ERROR] --- Invalid config.yaml. Missing key: 'wallet_added_coin_handler' (log_handler.py:22) [2022-07-14 10:19:05] [ INFO] --- Initializing handler: wallet_added_coin_handler (init.py:28) [2022-07-14 10:19:05] [ INFO] --- Enabled parser for wallet activity - added coins. (wallet_added_coin_parser.py:26) [2022-07-14 10:19:05] [ INFO] --- Filtering transaction with mojos less than 0 (wallet_added_coin_handler.py:26) [2022-07-14 10:19:06] [ INFO] --- Detected new plots. Farming with 969 plots. (non_decreasing_plots.py:28) [2022-07-14 10:19:06] [ INFO] --- Connected HDD? The total plot count increased from 0 to 969. (non_decreasing_plots.py:33)

Did I miss something ?

Best regards,

In https://github.com/martomi/chiadog/commit/99cf0bb73ab3d944105f0c83509c9713b5ba07d4, The Docker image should be built and available in the "Packages" section, however there is no such section on this repo. Does this Action need to be enabled somehow, or is it waiting for a trigger like a commit to be made?

The previous maintainer has archived their repo so this official repo is now the only way to get up-to-date docker builds.

Thanks

https://github.com/martomi/chiadog/blob/99cf0bb73ab3d944105f0c83509c9713b5ba07d4/src/chia_log/log_consumer.py#L75

If there is some non-utf8(like GBK) character, here will throw exception, then make thread exit. I want to make a pr, but I am not good at pyhon, can't fix it. Thx

I just saw a nice feature idea on Reddit and wanted to add it here.

It'd be nice if Chiadog could send an alert when a new Chia version became available.

https://reddit.com/r/chia/comments/rn86j8/_/hpsast1/?context=1

It would especially be helpful for people whose farms run passively in the background and don't pay attention unless the Chiadog barks 🐶