py-image-dedup is a tool to sort out or remove duplicates within a photo library

Last update: Jan 02, 2023

Overview

py-image-dedup

py-image-dedup is a tool to sort out or remove duplicates within a photo library. Unlike most other solutions, py-image-dedup intentionally uses an approximate image comparison to also detect duplicates of images that slightly differ in resolution, color or other minor details.

It is build upon Image-Match a very popular library to compute a pHash for an image and store the result in an ElasticSearch backend for very high scalability.

How it works

Phase 1 - Database cleanup

In the first phase the elasticsearch backend is checked against the current filesystem state, cleaning up database entries of files that no longer exist. This will speed up queries made later on.

Phase 2 - Counting files

Although not necessary for the deduplication process it is very convenient to have some kind of progress indication while the deduplication process is at work. To be able to provide that, available files must be counted beforehand.

Phase 3 - Analysing files

In this phase every image file is analysed. This means generating a signature (pHash) to quickly compare it to other images and adding other metadata of the image to the elasticsearch backend that is used in the next phase.

This phase is quite CPU intensive and the first run take take quite some time. Using as much threads as feasible (using the -t parameter) is advised to get the best performance.

Since we might already have a previous version of this file in the database before analysing a given file the file modification time is compared to the given one. If the database content seems to be still correct the signature for this file will not be recalculated. Because of this, subsequent runs will be much faster. There still has to happen some file access though, so it is probably limited by that.

Phase 4 - Finding duplicates

Every file is now processed again - but only by means of querying the database backend for similar images (within the given max_dist). If there are images found that match the similarity criteria they are considered duplicate candidates. All candidates are then ordered by the following criteria (in this exact order):

pixel count (more is better)
EXIF data (more exif data is better)
file size (bigger is better)
file modification time (newer is better)
distance (lower is better)
filename contains "copy" (False is better)
filename length (longer is better) - (for "edited" versions)
parent folder path length (shorter is better)
score (higher is better)

The first candidate in the resulting list is considered to be the best available version of all candidates.

Phase 5 - Moving/Deleting duplicates

All but the best version of duplicate candidates identified in the previous phase are now deleted from the file system (if you didn't specify --dry-run of course).

If duplicates_target_directory is set, the specified folder will be used as a root directory to move duplicates to, instead of deleting them, replicating their original folder structure.

Phase 6 - Removing empty folders (Optional)

In the last phase, folders that are empty due to the deduplication process are deleted, cleaning up the directory structure (if turned on in configuration).

How to use

Install

Install py-image-dedup using pip:

pip3 install py-image-dedup

Configuration

py-image-dedup uses container-app-conf to provide configuration via a YAML file as well as ENV variables which generates a reference config on startup. Have a look at the documentation about it

See py_image_dedup_reference.yaml for an example in this repo.

Setup elasticsearch backend

Since this library is based on Image-Match you need a running elasticsearch instance for efficient storing and querying of image signatures.

Elasticsearch version

This library requires elasticsearch version 5 or later. Sadly the Image-Match library still specifies version 2, so a fork of the original project is used instead. This fork is maintained by me, and any contributions are very much appreciated.

Set up the index

py-image-dedup uses a single index (called images by default). When configured, this index will be created automatically for you.

Command line usage

py-image-dedup can be used from the command line like this:

py-image-dedup deduplicate --help

Have a look at the help output to see how you can customize it.

Daemon

CAUTION! This feature is still very much a work in progress. Always have a backup of your data!

py-image-dedup has a built in daemon that allows you to continuously monitor your source directories and deduplicate them on the fly.

When running the daemon (and enabled in configuration) a prometheus reporter is used to allow you to gather some statistical insights.

py-image-dedup daemon

Dry run

To analyze images and get an overview of what images would be deleted be sure to make a dry run first.

py-image-dedup deduplicate --dry-run

FreeBSD

If you want to run this on a FreeBSD host make sure you have an up to date release that is able to install ports.

Since Image-Match does a lot of math it relies on numpy and scipy. To get those working on FreeBSD you have to install them as a port:

pkg install pkgconf
pkg install py38-numpy
pkg install py27-scipy

For .png support you also need to install

pkg install png

I still ran into issues after installing all these and just threw those two in the mix and it finally worked:

pkg install freetype
pkg install py27-matplotlib  # this has a LOT of dependencies

Encoding issues

When using the python library click on FreeBSD you might run into encoding issues. To mitigate this change your locale from ANSII to UTF-8 if possible.

This can be achieved f.ex. by creating a file ~/.login_conf with the following content:

me:\
	:charset=ISO-8859-1:\
	:lang=de_DE.UTF-8:

Docker

To run py-image-dedup using docker you can use the markusressel/py-image-dedup image from DockerHub:

sudo docker run -t \
    -p 8000:8000 \
    -v /where/the/original/photolibrary/is/located:/data/in \
    -v /where/duplicates/should/be/moved/to:/data/out \
    -e PY_IMAGE_DEDUP_DRY_RUN=False \
    -e PY_IMAGE_DEDUP_ANALYSIS_SOURCE_DIRECTORIES=/data/in/ \
    -e PY_IMAGE_DEDUP_ANALYSIS_RECURSIVE=True \
    -e PY_IMAGE_DEDUP_ANALYSIS_ACROSS_DIRS=True \
    -e PY_IMAGE_DEDUP_ANALYSIS_FILE_EXTENSIONS=.png,.jpg,.jpeg \
    -e PY_IMAGE_DEDUP_ANALYSIS_THREADS=8 \
    -e PY_IMAGE_DEDUP_ANALYSIS_USE_EXIF_DATA=True \
    -e PY_IMAGE_DEDUP_DEDUPLICATION_DUPLICATES_TARGET_DIRECTORY=/data/out/ \
    -e PY_IMAGE_DEDUP_ELASTICSEARCH_AUTO_CREATE_INDEX=True \
    -e PY_IMAGE_DEDUP_ELASTICSEARCH_HOST=elasticsearch \
    -e PY_IMAGE_DEDUP_ELASTICSEARCH_PORT=9200 \
    -e PY_IMAGE_DEDUP_ELASTICSEARCH_INDEX=images \
    -e PY_IMAGE_DEDUP_ELASTICSEARCH_AUTO_CREATE_INDEX=True \
    -e PY_IMAGE_DEDUP_ELASTICSEARCH_MAX_DISTANCE=0.1 \
    -e PY_IMAGE_DEDUP_REMOVE_EMPTY_FOLDERS=False \
    -e PY_IMAGE_DEDUP_STATS_ENABLED=True \
    -e PY_IMAGE_DEDUP_STATS_PORT=8000 \
    markusressel/py-image-dedup:latest

Since an elasticsearch instance is required too, you can also use the docker-compose.yml file included in this repo which will set up a single-node elasticsearch cluster too:

sudo docker-compose up

UID and GID

To run py-image-dedup inside the container using a specific user id and group id you can use the env variables PUID=1000 and PGID=1000.

Contributing

GitHub is for social coding: if you want to write code, I encourage contributions through pull requests from forks of this repository. Create GitHub tickets for bugs and new features and comment on the ones that you are interested in.

License

py-image-dedup by Markus Ressel
Copyright (C) 2018  Markus Ressel

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see 
   .

Comments

v2.0.0 not published on PyPi

Hi Markus,

unfortunately I have some trouble with ElasticSearch and can't get the code running properly. I receive this TypeError: init() missing 1 required positional argument: 'el_version'

(virtual_env) M-Air:Documents michael$ py-image-dedup deduplicate -d "/Users/michael/Desktop/bilder_test/input" -dr -esh "127.0.0.1"
Traceback (most recent call last):
  File "/Users/michael/Documents/python_workspace/virtual_env/bin/py-image-dedup", line 8, in <module>
    sys.exit(cli())
  File "/Users/michael/Documents/Python_Workspace/virtual_env/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/michael/Documents/Python_Workspace/virtual_env/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/michael/Documents/Python_Workspace/virtual_env/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/michael/Documents/Python_Workspace/virtual_env/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/michael/Documents/Python_Workspace/virtual_env/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/michael/Documents/Python_Workspace/virtual_env/lib/python3.8/site-packages/py_image_dedup/cli/Main.py", line 91, in c_deduplicate
    image_signature_store=ElasticSearchStoreBackend(
  File "/Users/michael/Documents/Python_Workspace/virtual_env/lib/python3.8/site-packages/py_image_dedup/persistence/ElasticSearchStoreBackend.py", line 41, in __init__
    self._store = SignatureES(
TypeError: __init__() missing 1 required positional argument: 'el_version'

What can I do to solve this issue?

Thanks in advance, Michael

bug

opened by stormriderDE 8

README overhaul
Some things about how py-image-dedup works changed since v1.0.0 and the README needs some guidance on how to use the docker-compose file. A big overhaul of the README is necessary.

Specify

since a fork of image-match supporting elasticsearch v6 as well as v7 is now used the cumbersome package dependency section for it can be removed

how the daemon works

how and what statistics are exposed

how to use with docker-compose

enhancement
opened by markusressel 7
Simplified usage with Docker

Hi there.

This is more kind of a feature request then it is an issue. It would be great to have that entire solution in a self-contained Docker image. Both DB and application. This would make the usage far easier.

For instance one could directly use it on Synology-NASes.

Thanks for considering. If I have time, I might try to setup something myself.
enhancement

opened by qdrop17 7

TypeError: Unable to serialize xxx (type: )

Using the Docker image markusressel/py-image-dedup:latest, I modified the two mountpoint paths in docker-compose.yml and ran docker-compose up. I got a bunch of errors TypeError: Unable to serialize xxx (type: <class 'PIL.TiffImagePlugin.IFDRational'>) like this:

py-image-dedup_1  | ERROR:root:({'path': '/mnt/source/0.jpg', 'signature': [0, 0, 0, 0, 1, 0, 0, -2, 0, 0, 0, -1, 1, -1, -2, -2, 0, 0, 0, -1, -2, -2, -2, -2, 0, 0, 0, 2, -1, -1, -2, -2, 0, 0, 0, 1, -1, -1, -2, -1, 0, 0, 0, 1, 1, -1, -1, -1, 0, 0, 0, -1, 0, -1, -1, -1, 0, 0, 0, 0, 1, -1, -1, -2, 0, 0, 0, -1, 0, -2, -2, 0, 0, 0, 1, 0, -2, 0, -2, -2, 2, 2, 2, 2, -1, -2, -2, 2, 2, 2, 1, 1, -1, -1, 2, 2, 2, 2, 1, 1, -1, 2, 2, 2, 2, 2, 1, 1, 1, 2, 2, 1, 1, 1, 1, -1, -1, 2, -1, 1, 1, 1, 1, 1, 0, 1, 2, 2, 1, 1, 2, 0, -1, 2, 2, 1, 2, 2, 0, 1, 0, 2, 1, 0, 0, 2, 2, 0, -1, 0, -1, 2, 2, 2, 1, 1, 2, 1, 2, 2, -2, -2, -2, -2, -1, 1, 1, 1, -2, -2, -2, 1, -1, 1, 2, -2, -2, -2, -2, 1, -2, 2, -2, -2, -1, 1, -1, 2, 2, 1, 0, 1, -1, -2, -2, -2, 1, -2, -1, -1, -2, -2, -2, -1, -2, -2, -1, -1, -1, -1, 0, 2, 0, 1, 1, 0, 0, 1, -1, 0, 2, 0, 1, 2, -2, -2, -1, -2, 1, -2, 2, -1, -2, -1, -1, -1, 1, 1, -1, -2, -1, -2, -2, -1, -2, -1, -2, -2, 2, 2, -1, 2, -1, 2, 2, 1, 2, 0, 2, 1, 1, 2, 1, 1, -1, 1, 2, -1, 1, 1, 1, 1, 1, 1, -1, -1, 0, -1, 1, 2, 1, -1, 0, 0, 0, 1, 2, 0, 0, -1, 2, 0, 2, 0, 1, 2, -2, -2, -1, -2, -2, -2, -2, -1, 1, 1, 1, 2, -2, -1, 1, -2, 2, 2, -2, 2, -1, 2, 0, 2, 2, -2, -2, 1, -1, 1, 2, 2, -1, -1, -1, 1, -1, 2, 2, 1, -1, -1, 1, 1, 1, 2, 1, 2, -1, -1, -1, -1, 1, 1, 2, 2, -2, -2, 0, -1, 0, 1, 1, 0, 0, -1, 2, 0, 2, 0, -2, 2, -2, 2, 1, -2, 2, -2, 2, -1, 1, -1, -2, -2, -2, 1, -2, -1, 2, 0, -1, 2, 2, 1, 2, 2, -2, -2, -2, -2, -2, -2, -1, -1, -2, -2, -2, 2, -2, 1, 1, 1, -1, -1, -1, 2, 1, 2, 2, 1, -2, -2, -1, -1, 1, 2, -1, 2, -2, -1, 0, -1, 0, -1, 1, 0, 0, 2, 2, 0, 2, 0, 0, 2, -2, -2, -1, -2, -2, -2, -2, -2, 1, 2, -1, 2, 1, -1, -1, 2, 1, -2, 2, -1, 1, -2, 1, 1, -2, 1, -1, -1, -1, -1, 0, 1, 1, -1, -2, 1, -1, 1, 1, 1, -1, -2, -2, 1, -2, 1, 1, -2, -1, 1, 1, 2, 2, 2, -2, -2, -2, -1, 0, -2, 0, -2, -2, 0, 0, 0, 2, 0, 2, 0, 2, 1, -2, 2, 1, -2, 1, 2, -2, 2, 2, 1, 2, -1, 2, -2, 2, 2, -2, -1, 1, -2, 1, 2, 1, -1, -1, 0, -1, -1, 1, 1, -1, 1, -1, -1, -1, -1, -1, -2, 1, -1, -1, -1, -2, 1, -2, 1, 1, -2, 2, 2, 2, 2, -1, 2, 2, 2, 2, 2, 0, 1, 0, 2, 2, 0, 0, -2, -2, 0, -2, 0, 0, 0, -1, 2, 2, 2, 2, 0, 0, 0, -2, -2, -2, -2, -1, 0, 0, 0, -2, -1, -1, 1, -1, 0, 0, 0, 1, 1, 2, 1, 2, 0, 0, 0, -1, -1, -1, -2, -1, 0, 0, 0, 1, -1, -2, 1, -2, 0, 0, 0, 2, -2, -2, 2, 1, 0, 0, 0, -2, -2, 0, -1, 0, 0, 0, 0], 'metadata': {'path': '/mnt/source/0.jpg', 'py-image-dedup_datamodel-version': 5, 'filesize': 120968, 'file_modification_date': 1148934648.0, 'pixelcount': 272640, 'exif_data': {'ExifVersion': "b'0220'", 'ShutterSpeedValue': 5.64385986328125, 'ApertureValue': 4.33984375, 'DateTimeOriginal': '2006:05:17 19:53:58', 'DateTimeDigitized': '2006:05:17 19:53:58', 'ExposureBiasValue': 0.0, 'MaxApertureValue': 4.33984375, 'MeteringMode': 5, 'ColorSpace': 1, 'Flash': 0, 'FocalLength': 105.0, 'Make': 'Canon', 'Model': 'Canon EOS DIGITAL REBEL', 'Orientation': 1, 'YCbCrPositioning': 1, 'SensingMethod': 2, 'ExposureTime': 0.02, 'XResolution': 72.0, 'YResolution': 72.0, 'FNumber': 4.5, 'ISOSpeedRatings': 400, 'ResolutionUnit': 2, 'ExifOffset': 254, 'Software': 'QuickTime 7.1', 'DateTime': '2006:05:26 22:11:28', 'HostComputer': 'Mac OS X 10.4.6'}}, 'simple_word_0': 1149349, 'simple_word_1': 19141372, 'simple_word_2': 35611974, 'simple_word_3': 13522824, 'simple_word_4': 14554561, 'simple_word_5': 20775288, 'simple_word_6': 8741790, 'simple_word_7': 29222986, 'simple_word_8': 42574094, 'simple_word_9': 43033590, 'simple_word_10': 30292136, 'simple_word_11': 42869015, 'simple_word_12': 28651883, 'simple_word_13': 9327311, 'simple_word_14': 43044829, 'simple_word_15': 4606550, 'simple_word_16': 28859730, 'simple_word_17': 41811554, 'simple_word_18': 356231, 'simple_word_19': 14348913, 'simple_word_20': 34836237, 'simple_word_21': 10635091, 'simple_word_22': 472572, 'simple_word_23': 32949350, 'simple_word_24': 42867945, 'simple_word_25': 42924247, 'simple_word_26': 18976596, 'simple_word_27': 36961350, 'simple_word_28': 6283, 'simple_word_29': 39384954, 'simple_word_30': 30218445, 'simple_word_31': 31943833, 'simple_word_32': 14331348, 'simple_word_33': 15923789, 'simple_word_34': 36960569, 'simple_word_35': 10791109, 'simple_word_36': 24270779, 'simple_word_37': 529665, 'simple_word_38': 31886468, 'simple_word_39': 14291982, 'simple_word_40': 15569495, 'simple_word_41': 38010323, 'simple_word_42': 5560, 'simple_word_43': 12917880, 'simple_word_44': 3675836, 'simple_word_45': 32050547, 'simple_word_46': 4371750, 'simple_word_47': 14525399, 'simple_word_48': 37439498, 'simple_word_49': 33113092, 'simple_word_50': 4389140, 'simple_word_51': 1766441, 'simple_word_52': 45227, 'simple_word_53': 33067926, 'simple_word_54': 28658279, 'simple_word_55': 5075783, 'simple_word_56': 22304026, 'simple_word_57': 2303288, 'simple_word_58': 42775572, 'simple_word_59': 14379520, 'simple_word_60': 35730072, 'simple_word_61': 15169637, 'simple_word_62': 21522523, 'timestamp': datetime.datetime(2020, 11, 7, 11, 42, 43, 149502)}, TypeError("Unable to serialize 5.64385986328125 (type: <class 'PIL.TiffImagePlugin.IFDRational'>)"))
py-image-dedup_1  | Traceback (most recent call last):
py-image-dedup_1  |   File "/usr/local/lib/python3.8/site-packages/elasticsearch/serializer.py", line 120, in dumps
py-image-dedup_1  |     return json.dumps(
py-image-dedup_1  |   File "/usr/local/lib/python3.8/json/__init__.py", line 234, in dumps
py-image-dedup_1  |     return cls(
py-image-dedup_1  |   File "/usr/local/lib/python3.8/json/encoder.py", line 199, in encode
py-image-dedup_1  |     chunks = self.iterencode(o, _one_shot=True)
py-image-dedup_1  |   File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode
py-image-dedup_1  |     return _iterencode(o, 0)
py-image-dedup_1  |   File "/usr/local/lib/python3.8/site-packages/elasticsearch/serializer.py", line 106, in default
py-image-dedup_1  |     raise TypeError("Unable to serialize %r (type: %s)" % (data, type(data)))
py-image-dedup_1  | TypeError: Unable to serialize 5.64385986328125 (type: <class 'PIL.TiffImagePlugin.IFDRational'>)
py-image-dedup_1  |

% docker images
REPOSITORY                                      TAG                 IMAGE ID            CREATED             SIZE
markusressel/py-image-dedup                     latest              b5a69df15036        44 hours ago        1.3GB
docker.elastic.co/elasticsearch/elasticsearch   7.9.2               caa7a21ca06e        6 weeks ago         763MB

Dupe of #21?

bug

opened by jrc 6

Issue with connecting to Elasticsearch

Describe the bug I'm having issue with connections to my Elasticsearch instance.

The following is an example of the behaviour I'm experiences. I'm not sure how to handle it or if it is an issue with Elasticsearch or py-image-dedup.

issue

(venv) [email protected]:~/py-image-dedup$ docker-compose up
Creating network "py-image-dedup_docker-elk" with driver "bridge"
Creating py-image-dedup_elasticsearch_1 ... done
Creating py-image-dedup_py-image-dedup_1 ... done
Attaching to py-image-dedup_elasticsearch_1, py-image-dedup_py-image-dedup_1
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:26:45,833Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "unable to install syscall filter: ",
elasticsearch_1   | "stacktrace": ["java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed",
elasticsearch_1   | "at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:342) ~[elasticsearch-7.9.2.jar:7.9.2]",
elasticsearch_1   | "at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:617) ~[elasticsearch-7.9.2.jar:7.9.2]",
elasticsearch_1   | "at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:260) [elasticsearch-7.9.2.jar:7.9.2]",
elasticsearch_1   | "at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:113) [elasticsearch-7.9.2.jar:7.9.2]",
elasticsearch_1   | "at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:116) [elasticsearch-7.9.2.jar:7.9.2]",
elasticsearch_1   | "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) [elasticsearch-7.9.2.jar:7.9.2]",
elasticsearch_1   | "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.9.2.jar:7.9.2]",
elasticsearch_1   | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.9.2.jar:7.9.2]",
elasticsearch_1   | "at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.9.2.jar:7.9.2]",
elasticsearch_1   | "at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.9.2.jar:7.9.2]",
elasticsearch_1   | "at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.9.2.jar:7.9.2]",
elasticsearch_1   | "at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.9.2.jar:7.9.2]",
elasticsearch_1   | "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.9.2.jar:7.9.2]",
elasticsearch_1   | "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.9.2.jar:7.9.2]"] }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:26:47,794Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "version[7.9.2], pid[8], build[default/docker/d34da0ea4a966c4e49417f2da2f244e3e97b4e6e/2020-09-23T00:45:33.626720Z], OS[Linux/3.10.108/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15/15+36]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:26:47,824Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "JVM home [/usr/share/elasticsearch/jdk]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:26:47,827Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-16856439618775791360, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1g, -Xmx1g, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,767Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [aggs-matrix-stats]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,771Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [analysis-common]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,786Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [constant-keyword]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,790Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [flattened]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,791Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [frozen-indices]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,799Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [ingest-common]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,803Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [ingest-geoip]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,805Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [ingest-user-agent]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,826Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [kibana]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,836Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [lang-expression]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,837Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [lang-mustache]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,838Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [lang-painless]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,839Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [mapper-extras]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,840Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [parent-join]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,841Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [percolator]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,851Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [rank-eval]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,865Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [reindex]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,868Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [repository-url]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,869Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [search-business-rules]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,872Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [searchable-snapshots]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,874Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [spatial]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,877Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [tasks]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,885Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [transform]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,894Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [transport-netty4]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,895Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [vectors]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,896Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [wildcard]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,897Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-analytics]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,897Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-async]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,898Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-async-search]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,899Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-autoscaling]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,899Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-ccr]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,900Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-core]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,901Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-data-streams]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,902Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-deprecation]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,903Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-enrich]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,903Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-eql]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,904Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-graph]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,905Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-identity-provider]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,906Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-ilm]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,906Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-logstash]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,907Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-ml]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,908Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-monitoring]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,909Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-ql]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,910Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-rollup]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,910Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-security]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,912Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-sql]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,913Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-stack]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,913Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-voting-only-node]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,914Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-watcher]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,916Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "no plugins loaded" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:05,408Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/cachedev_1)]], net usable_space [54.8gb], net total_space [888.9gb], types [btrfs]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:05,417Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "heap size [1gb], compressed ordinary object pointers [true]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:06,025Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "node name [elasticsearch], node ID [Rnp6dtg9QsmDG0INGrREvg], cluster name [docker-cluster]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:35,273Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "[controller/221] [[email protected]] controller (64 bit): Version 7.9.2 (Build 6a60f0cf2dd5a5) Copyright (c) 2020 Elasticsearch BV" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:38,174Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:45,386Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=false, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:46,063Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:50,394Z", "level": "WARN", "component": "o.e.g.DanglingIndicesState", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:53,830Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "initialized" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:53,834Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "starting ..." }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:54,729Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "publish_address {192.168.16.2:9300}, bound_addresses {0.0.0.0:9300}" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:56,713Z", "level": "WARN", "component": "o.e.b.BootstrapChecks", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:56,715Z", "level": "WARN", "component": "o.e.b.BootstrapChecks", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:56,721Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "cluster UUID [VrXadE3iRXC2MvquiVfAtQ]" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:57,084Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "elected-as-master ([1] nodes joined)[{elasticsearch}{Rnp6dtg9QsmDG0INGrREvg}{j5fgjBsMS7aLU13iOZiOOw}{192.168.16.2}{192.168.16.2:9300}{dilmrt}{ml.machine_memory=8287412224, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 42, version: 210, delta: master node changed {previous [], current [{elasticsearch}{Rnp6dtg9QsmDG0INGrREvg}{j5fgjBsMS7aLU13iOZiOOw}{192.168.16.2}{192.168.16.2:9300}{dilmrt}{ml.machine_memory=8287412224, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:57,475Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "master node changed {previous [], current [{elasticsearch}{Rnp6dtg9QsmDG0INGrREvg}{j5fgjBsMS7aLU13iOZiOOw}{192.168.16.2}{192.168.16.2:9300}{dilmrt}{ml.machine_memory=8287412224, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 42, version: 210, reason: Publication{term=42, version=210}" }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:57,728Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "publish_address {192.168.16.2:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "VrXadE3iRXC2MvquiVfAtQ", "node.id": "Rnp6dtg9QsmDG0INGrREvg"  }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:57,743Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "started", "cluster.uuid": "VrXadE3iRXC2MvquiVfAtQ", "node.id": "Rnp6dtg9QsmDG0INGrREvg"  }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:59,605Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "license [8a370969-b86b-4e90-8722-29ba38b2975b] mode [basic] - valid", "cluster.uuid": "VrXadE3iRXC2MvquiVfAtQ", "node.id": "Rnp6dtg9QsmDG0INGrREvg"  }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:59,612Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "Active license is now [BASIC]; Security is disabled", "cluster.uuid": "VrXadE3iRXC2MvquiVfAtQ", "node.id": "Rnp6dtg9QsmDG0INGrREvg"  }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:59,689Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "recovered [1] indices into cluster_state", "cluster.uuid": "VrXadE3iRXC2MvquiVfAtQ", "node.id": "Rnp6dtg9QsmDG0INGrREvg"  }
elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:28:03,268Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[images][0]]]).", "cluster.uuid": "VrXadE3iRXC2MvquiVfAtQ", "node.id": "Rnp6dtg9QsmDG0INGrREvg"  }
py-image-dedup_1  | DEBUG:py_image_dedup.util:Starting daemon...
py-image-dedup_1  | Starting daemon...
py-image-dedup_1  | DEBUG:py_image_dedup.util:Starting prometheus reporter...
py-image-dedup_1  | Starting prometheus reporter...
py-image-dedup_1  | DEBUG:py_image_dedup.util:==> DRY RUN! No files or folders will actually be deleted! <==
py-image-dedup_1  | ==> DRY RUN! No files or folders will actually be deleted! <==
py-image-dedup_1  | DEBUG:py_image_dedup.util:Phase 1/6: Cleaning up database ...
py-image-dedup_1  | Phase 1/6: Cleaning up database ...
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 0% (1/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 1% (23/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 2% (46/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 3% (69/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 4% (92/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 5% (115/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 6% (138/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 7% (161/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 8% (184/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 9% (207/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 10% (229/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 11% (252/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 12% (275/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 13% (298/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 14% (321/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 15% (344/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 16% (367/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 17% (390/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 18% (413/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 19% (435/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 20% (458/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 21% (481/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 22% (504/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 23% (527/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 24% (550/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 25% (573/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 26% (596/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 27% (619/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 28% (641/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 29% (664/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 30% (687/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 31% (710/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 32% (733/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 33% (756/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 34% (779/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 35% (802/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 36% (825/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 37% (847/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 38% (870/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 39% (893/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 40% (916/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 41% (939/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 42% (962/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 43% (985/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 44% (1008/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 45% (1031/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 46% (1053/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 47% (1076/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 48% (1099/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 49% (1122/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 50% (1145/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 51% (1168/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 52% (1191/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 53% (1214/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 54% (1237/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 55% (1259/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 56% (1282/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 57% (1305/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 58% (1328/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 59% (1351/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 60% (1374/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 61% (1397/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 62% (1420/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 63% (1443/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 64% (1465/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 65% (1488/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 66% (1511/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 67% (1534/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 68% (1557/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 69% (1580/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 70% (1603/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 71% (1626/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 72% (1649/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 73% (1671/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 74% (1694/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 75% (1717/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 76% (1740/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 77% (1763/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 78% (1786/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 79% (1809/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 80% (1832/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 81% (1855/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 82% (1877/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 83% (1900/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 84% (1923/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 85% (1946/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 86% (1969/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 87% (1992/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 88% (2015/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 89% (2038/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 90% (2061/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 91% (2083/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 92% (2106/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 93% (2129/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 94% (2152/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 95% (2175/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 96% (2198/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 97% (2221/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 98% (2244/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 99% (2267/2289)
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 100% (2289/2289)
py-image-dedup_1  | DEBUG:py_image_dedup.util:Phase 2/6: Counting files ...
py-image-dedup_1  | Phase 2/6: Counting files ...
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Counting files: 100% (1/1)
py-image-dedup_1  | DEBUG:py_image_dedup.util:Phase 3/6: Analyzing files
py-image-dedup_1  | Phase 3/6: Analyzing files
py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Analyzing files in '/mnt/source': 0% (1/179325)
py-image-dedup_1  | WARNING:elasticsearch:POST http://elasticsearch:9200/images/_doc?refresh=false [status:N/A request:0.021s]
py-image-dedup_1  | Traceback (most recent call last):
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
py-image-dedup_1  |     conn = connection.create_connection(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
py-image-dedup_1  |     raise err
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
py-image-dedup_1  |     sock.connect(sa)
py-image-dedup_1  | ConnectionRefusedError: [Errno 111] Connection refused
py-image-dedup_1  |
py-image-dedup_1  | During handling of the above exception, another exception occurred:
py-image-dedup_1  |
py-image-dedup_1  | Traceback (most recent call last):
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/elasticsearch/connection/http_urllib3.py", line 232, in perform_request
py-image-dedup_1  |     response = self.pool.urlopen(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
py-image-dedup_1  |     retries = retries.increment(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 525, in increment
py-image-dedup_1  |     raise six.reraise(type(error), error, _stacktrace)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
py-image-dedup_1  |     raise value
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
py-image-dedup_1  |     httplib_response = self._make_request(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
py-image-dedup_1  |     conn.request(method, url, **httplib_request_kw)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
py-image-dedup_1  |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1282, in request
py-image-dedup_1  |     self._send_request(method, url, body, headers, encode_chunked)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1328, in _send_request
py-image-dedup_1  |     self.endheaders(body, encode_chunked=encode_chunked)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1277, in endheaders
py-image-dedup_1  |     self._send_output(message_body, encode_chunked=encode_chunked)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1037, in _send_output
py-image-dedup_1  |     self.send(msg)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 975, in send
py-image-dedup_1  |     self.connect()
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
py-image-dedup_1  |     conn = self._new_conn()
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
py-image-dedup_1  |     raise NewConnectionError(
py-image-dedup_1  | urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused
py-image-dedup_1  | WARNING:elasticsearch:POST http://elasticsearch:9200/images/_doc?refresh=false [status:N/A request:0.005s]
py-image-dedup_1  | Traceback (most recent call last):
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
py-image-dedup_1  |     conn = connection.create_connection(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
py-image-dedup_1  |     raise err
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
py-image-dedup_1  |     sock.connect(sa)
py-image-dedup_1  | ConnectionRefusedError: [Errno 111] Connection refused
py-image-dedup_1  |
py-image-dedup_1  | During handling of the above exception, another exception occurred:
py-image-dedup_1  |
py-image-dedup_1  | Traceback (most recent call last):
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/elasticsearch/connection/http_urllib3.py", line 232, in perform_request
py-image-dedup_1  |     response = self.pool.urlopen(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
py-image-dedup_1  |     retries = retries.increment(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 525, in increment
py-image-dedup_1  |     raise six.reraise(type(error), error, _stacktrace)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
py-image-dedup_1  |     raise value
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
py-image-dedup_1  |     httplib_response = self._make_request(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
py-image-dedup_1  |     conn.request(method, url, **httplib_request_kw)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
py-image-dedup_1  |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1282, in request
py-image-dedup_1  |     self._send_request(method, url, body, headers, encode_chunked)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1328, in _send_request
py-image-dedup_1  |     self.endheaders(body, encode_chunked=encode_chunked)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1277, in endheaders
py-image-dedup_1  |     self._send_output(message_body, encode_chunked=encode_chunked)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1037, in _send_output
py-image-dedup_1  |     self.send(msg)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 975, in send
py-image-dedup_1  |     self.connect()
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
py-image-dedup_1  |     conn = self._new_conn()
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
py-image-dedup_1  |     raise NewConnectionError(
py-image-dedup_1  | urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused
py-image-dedup_1  | WARNING:elasticsearch:POST http://elasticsearch:9200/images/_doc?refresh=false [status:N/A request:0.015s]
py-image-dedup_1  | Traceback (most recent call last):
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
py-image-dedup_1  |     conn = connection.create_connection(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
py-image-dedup_1  |     raise err
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
py-image-dedup_1  |     sock.connect(sa)
py-image-dedup_1  | ConnectionRefusedError: [Errno 111] Connection refused
py-image-dedup_1  |
py-image-dedup_1  | During handling of the above exception, another exception occurred:
py-image-dedup_1  |
py-image-dedup_1  | Traceback (most recent call last):
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/elasticsearch/connection/http_urllib3.py", line 232, in perform_request
py-image-dedup_1  |     response = self.pool.urlopen(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
py-image-dedup_1  |     retries = retries.increment(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 525, in increment
py-image-dedup_1  |     raise six.reraise(type(error), error, _stacktrace)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
py-image-dedup_1  |     raise value
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
py-image-dedup_1  |     httplib_response = self._make_request(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
py-image-dedup_1  |     conn.request(method, url, **httplib_request_kw)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
py-image-dedup_1  |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1282, in request
py-image-dedup_1  |     self._send_request(method, url, body, headers, encode_chunked)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1328, in _send_request
py-image-dedup_1  |     self.endheaders(body, encode_chunked=encode_chunked)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1277, in endheaders
py-image-dedup_1  |     self._send_output(message_body, encode_chunked=encode_chunked)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1037, in _send_output
py-image-dedup_1  |     self.send(msg)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 975, in send
py-image-dedup_1  |     self.connect()
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
py-image-dedup_1  |     conn = self._new_conn()
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
py-image-dedup_1  |     raise NewConnectionError(
py-image-dedup_1  | urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused
py-image-dedup_1  | WARNING:elasticsearch:POST http://elasticsearch:9200/images/_doc?refresh=false [status:N/A request:0.006s]
py-image-dedup_1  | Traceback (most recent call last):
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
py-image-dedup_1  |     conn = connection.create_connection(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
py-image-dedup_1  |     raise err
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
py-image-dedup_1  |     sock.connect(sa)
py-image-dedup_1  | ConnectionRefusedError: [Errno 111] Connection refused
py-image-dedup_1  |
py-image-dedup_1  | During handling of the above exception, another exception occurred:
py-image-dedup_1  |
py-image-dedup_1  | Traceback (most recent call last):
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/elasticsearch/connection/http_urllib3.py", line 232, in perform_request
py-image-dedup_1  |     response = self.pool.urlopen(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
py-image-dedup_1  |     retries = retries.increment(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 525, in increment
py-image-dedup_1  |     raise six.reraise(type(error), error, _stacktrace)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
py-image-dedup_1  |     raise value
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
py-image-dedup_1  |     httplib_response = self._make_request(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
py-image-dedup_1  |     conn.request(method, url, **httplib_request_kw)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
py-image-dedup_1  |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1282, in request
py-image-dedup_1  |     self._send_request(method, url, body, headers, encode_chunked)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1328, in _send_request
py-image-dedup_1  |     self.endheaders(body, encode_chunked=encode_chunked)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1277, in endheaders
py-image-dedup_1  |     self._send_output(message_body, encode_chunked=encode_chunked)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1037, in _send_output
py-image-dedup_1  |     self.send(msg)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 975, in send
py-image-dedup_1  |     self.connect()
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
py-image-dedup_1  |     conn = self._new_conn()
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
py-image-dedup_1  |     raise NewConnectionError(
py-image-dedup_1  | urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused
py-image-dedup_1  | WARNING:elasticsearch:POST http://elasticsearch:9200/images/_doc?refresh=false [status:N/A request:0.004s]
py-image-dedup_1  | Traceback (most recent call last):
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
py-image-dedup_1  |     conn = connection.create_connection(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
py-image-dedup_1  |     raise err
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
py-image-dedup_1  |     sock.connect(sa)
py-image-dedup_1  | ConnectionRefusedError: [Errno 111] Connection refused
py-image-dedup_1  |
py-image-dedup_1  | During handling of the above exception, another exception occurred:
py-image-dedup_1  |
py-image-dedup_1  | Traceback (most recent call last):
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/elasticsearch/connection/http_urllib3.py", line 232, in perform_request
py-image-dedup_1  |     response = self.pool.urlopen(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
py-image-dedup_1  |     retries = retries.increment(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 525, in increment
py-image-dedup_1  |     raise six.reraise(type(error), error, _stacktrace)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
py-image-dedup_1  |     raise value
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
py-image-dedup_1  |     httplib_response = self._make_request(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
py-image-dedup_1  |     conn.request(method, url, **httplib_request_kw)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
py-image-dedup_1  |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1282, in request
py-image-dedup_1  |     self._send_request(method, url, body, headers, encode_chunked)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1328, in _send_request
py-image-dedup_1  |     self.endheaders(body, encode_chunked=encode_chunked)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1277, in endheaders
py-image-dedup_1  |     self._send_output(message_body, encode_chunked=encode_chunked)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1037, in _send_output
py-image-dedup_1  |     self.send(msg)
py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 975, in send
py-image-dedup_1  |     self.connect()
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
py-image-dedup_1  |     conn = self._new_conn()
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
py-image-dedup_1  |     raise NewConnectionError(
py-image-dedup_1  | urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused
py-image-dedup_1  | WARNING:elasticsearch:POST http://elasticsearch:9200/images/_doc?refresh=false [status:N/A request:0.003s]
py-image-dedup_1  | Traceback (most recent call last):
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
py-image-dedup_1  |     conn = connection.create_connection(
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
py-image-dedup_1  |     raise err
py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
py-image-dedup_1  |     sock.connect(sa)
py-image-dedup_1  | ConnectionRefusedError: [Errno 111] Connection refused
py-image-dedup_1  |

To Reproduce Running with the following

docker-compose.yaml

version: '3.7'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      - discovery.type=single-node
      - cluster.routing.allocation.disk.threshold_enabled=true 
      - cluster.routing.allocation.disk.watermark.flood_stage=200mb
      - cluster.routing.allocation.disk.watermark.low=500mb 
      - cluster.routing.allocation.disk.watermark.high=300mb
      - node.name=elasticsearch
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    networks:
      - docker-elk
    restart: on-failure
    volumes:
      - ./esdata1:/usr/share/elasticsearch/data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9200"]
      interval: 30s
      timeout: 30s
      retries: 10
  py-image-dedup:
    image: markusressel/py-image-dedup:latest
    environment:
      - PUID=0
      - PGID=100
      # change configuration to your liking
      - PY_IMAGE_DEDUP_DRY_RUN=True
      - PY_IMAGE_DEDUP_ANALYSIS_SOURCE_DIRECTORIES=/mnt/source/
      - PY_IMAGE_DEDUP_ANALYSIS_RECURSIVE=True
      - PY_IMAGE_DEDUP_ANALYSIS_ACROSS_DIRS=True
      - PY_IMAGE_DEDUP_ANALYSIS_FILE_EXTENSIONS=.png,.jpg,.jpeg
      - PY_IMAGE_DEDUP_ANALYSIS_THREADS=4
      - PY_IMAGE_DEDUP_ANALYSIS_USE_EXIF_DATA=True
      - PY_IMAGE_DEDUP_DEDUPLICATION_DUPLICATES_TARGET_DIRECTORY=/mnt/duplicates/
      - PY_IMAGE_DEDUP_ELASTICSEARCH_HOST=elasticsearch
      - PY_IMAGE_DEDUP_ELASTICSEARCH_PORT=9200
      - PY_IMAGE_DEDUP_ELASTICSEARCH_INDEX=images
      - PY_IMAGE_DEDUP_ELASTICSEARCH_AUTO_CREATE_INDEX=True
      - PY_IMAGE_DEDUP_ELASTICSEARCH_MAX_DISTANCE=0.1
      - PY_IMAGE_DEDUP_REMOVE_EMPTY_FOLDERS=False
      - PY_IMAGE_DEDUP_STATS_ENABLED=True
      - PY_IMAGE_DEDUP_STATS_PORT=8000
    volumes:
      # optionally mount a YAML configuration file
      # into /app/py_image_dedup.yaml instead of using environment:
      # - /mnt/data3/py_image_dedup.yaml:/app/py_image_dedup.yaml
      # change this to your local source directory:
      - /volume2/pictures/gto:/mnt/source:rw
      # change this to your local duplicates directory:
      - /volume2/Media/photo-dupes:/mnt/duplicates/
    links:
      - elasticsearch
    networks:
     - docker-elk
    ports:
      - "8000:8000"
    depends_on:
      elasticsearch:
          condition: service_healthy
    restart: on-failure
networks:
  docker-elk:
    driver: bridge

question

opened by fredrike 5

random crash

Describe the bug Tried to run the program on my image folder, but at a certain point, it consistently crashes, without any usable error log.

To Reproduce Steps to reproduce the behavior:

Set up elastic, py-image-dedup using docker-compose
run on image folder
see it analyze files, and at a certain percentage, always crash

Expected behavior The program should finish analyzing my files, and report back the duplicates.

Screenshots

INFO:py_image_dedup.library.progress_manager:Analyzing files in '/mnt/source': 63% (2037/3233)
INFO:py_image_dedup.library.progress_manager:Analyzing files in '/mnt/source': 64% (2070/3233)
INFO:py_image_dedup.library.progress_manager:Analyzing files in '/mnt/source': 65% (2102/3233)
INFO:py_image_dedup.library.progress_manager:Analyzing files in '/mnt/source': 66% (2134/3233)
INFO:py_image_dedup.library.progress_manager:Analyzing files in '/mnt/source': 67% (2167/3233)
Killed
<docker automatically restarts>
DEBUG:py_image_dedup.util:Starting daemon...
Starting daemon...
DEBUG:py_image_dedup.util:Starting prometheus reporter...
Starting prometheus reporter...
DEBUG:py_image_dedup.util:==> DRY RUN! No files or folders will actually be deleted! <==
DEBUG:py_image_dedup.util:Phase 1/6: Cleaning up database ...
==> DRY RUN! No files or folders will actually be deleted! <==
Phase 1/6: Cleaning up database ...

Desktop (please complete the following information):

OS: Ubuntu 20.04, docker

bug

opened by mortee 5

Installing via setup.py does not use forked image-match version
Describe the bug Image-Match dependency on 'elasticsearch<2.4,>=2.3'

To Reproduce Steps to reproduce the behavior:

Clone Git

Setup virtual environment

Run python setup.py install

pkg_resources.ContextualVersionConflict: (elasticsearch 7.10.0 (/Users/michael/Documents/Python_Workspace/py-image-dedup/myenv/lib/python3.8/site-packages/elasticsearch-7.10.0-py3.8.egg), Requirement.parse('elasticsearch<2.4,>=2.3'), {'image-match'})

Expected behavior Conflict with Elasticsearch 7.10.0 on py-image-dedup-end and elasticsearch<2.4,>=2.3 on image-match-end

Screenshots

Desktop (please complete the following information):

OS: Mac OS

Additional context Add any other context about the problem here.
bug
opened by stormriderDE 4
Bump tqdm from 4.49.0 to 4.50.0
Bumps tqdm from 4.49.0 to 4.50.0.

Release notes

Sourced from tqdm's releases.

tqdm v4.50.0 stable

add bar colour option (#1040 <- #450)

for notebook as well as std/CLI

plain text as well as hex (e.g. 'green' or '#00ff00')

notebook improvements (#1032 <- #1031)

split bar description into separate widget

improve reset() (handle width and colour)

fix exceptions when file is closed (#1041, #1036 <- #1033)

add & update tests

minor documentation updates

move and configure issue/pr templates

Commits

00dabd6 bump version, merge branch 'devel'

53a84ea closed file: use miniters=inf rather than hacky disable=1

098745d test and fix disable when writing to closed files

ea2367e Merge branch 'colour' into devel

06fb6ec add tests

c89b0fe fix notebook

8baa057 support hex colours

afe9ad8 support CLI colours

dd5739a notebook: add colour

e835860 Merge branch 'notebook-nototal' into devel

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

@dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

Additionally, you can set the following in your Dependabot dashboard:

Update frequency (including time of day and day of week)

Pull request limits (per update run and/or open at any time)

Out-of-range updates (receive only lockfile updates, if desired)

Security updates (receive only security updates, if desired)

dependencies
opened by dependabot-preview[bot] 3
container_app_conf

Running the example.py results in the following error: ModuleNotFoundError: No module named 'container_app_conf

To replicate the error: create a python environment install py-image-dedup via pip install py-image-dedup python3 example.py

Any idea why this could happen? pip list shows py-imaged-dedup as installed but the installation gave some error messages. I can post them here if needed

opened by byagiz 2
Hitting lots of errors from lib folder

everytime I run the image dedup it hits an error but they all seems to becoming from... ..\python\python36\lib[some package] some examples are: python\python36\lib\site-packages\urllib3\util\retry.py python\python36\lib\http\client.py python\python36\lib\site-packages\urllib3\connectionpool.py ...

idk if you I need to copy and paste the massive errors that are occurring but idk what to do anymore. Either from deleting everything in the lib folder or what to do anymore.
help wanted

opened by HosAkh 2
Bump elasticsearch from 6.8.2 to 7.17.7
Bumps elasticsearch from 6.8.2 to 7.17.7.

Release notes

Sourced from elasticsearch's releases.

7.17.7

Client is compatible with Elasticsearch 7.17.7

7.17.6

Client is compatible with Elasticsearch 7.17.6

7.17.5

Client is compatible with Elasticsearch 7.17.5

7.17.4

Client

Client is compatible with Elasticsearch 7.17.4

7.17.3

Client

Client is compatible with Elasticsearch 7.17.3

7.17.2

Client

Client is compatible with Elasticsearch 7.17.2

7.17.1

Helpers

Fixed the from parameter to be rewritten to from_ when used with the scan and async_scan helpers (elastic/elasticsearch-py#1897)

Transport

Fixed AiohttpHttpConnection to not leak TLS connections when the socket isn't explicitly shutdown by the peer (elastic/elasticsearch-py#1919)

7.17.0

Machine Learning

Added the ml.get_model_snapshot_upgrade_stats API

Added the body parameter to the ml.forecast and ml.open_job APIs

Transform

Added the timeout parameter to the transform.delete_transform, transform.preview_transform, transform.put_transform, transform.update_transform, and transform.upgrade_transform APIs

7.16.3

Client

API is compatible with Elasticsearch 7.16.3

7.16.2

Client

... (truncated)

Commits

9221e2d Bumps to version 7.17.7

72f102d Update assemble task - SNAPSHOT

d411230 non-root user to build artifacts inside the container

576fae9 Bumps to version 7.17.6

7b5389d Add release notes for 7.173 and 7.17.4

a0ee925 Bumps to version 7.17.5

2f5e8e2 Bumps to version 7.17.4

12ab98a [7.17] Use major.minor-SNAPSHOT for STACK_VERSION

16d57d2 [7.17] Implement '.ci/make.sh bump' to set version

d2432ef Pin click to version compatible with black v21.12b0

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 1
Bump elasticsearch from 6.8.2 to 7.17.8
Bumps elasticsearch from 6.8.2 to 7.17.8.

Release notes

Sourced from elasticsearch's releases.

7.17.8

Removed the work-around for a bug in aiohttp when sending HEAD requests. This change was backported from v8.x of elastic-transport.

7.17.7

Client is compatible with Elasticsearch 7.17.7

7.17.6

Client is compatible with Elasticsearch 7.17.6

7.17.5

Client is compatible with Elasticsearch 7.17.5

7.17.4

Client

Client is compatible with Elasticsearch 7.17.4

7.17.3

Client

Client is compatible with Elasticsearch 7.17.3

7.17.2

Client

Client is compatible with Elasticsearch 7.17.2

7.17.1

Helpers

Fixed the from parameter to be rewritten to from_ when used with the scan and async_scan helpers (elastic/elasticsearch-py#1897)

Transport

Fixed AiohttpHttpConnection to not leak TLS connections when the socket isn't explicitly shutdown by the peer (elastic/elasticsearch-py#1919)

7.17.0

Machine Learning

Added the ml.get_model_snapshot_upgrade_stats API

Added the body parameter to the ml.forecast and ml.open_job APIs

Transform

Added the timeout parameter to the transform.delete_transform, transform.preview_transform, transform.put_transform, transform.update_transform, and transform.upgrade_transform APIs

7.16.3

Client

API is compatible with Elasticsearch 7.16.3

... (truncated)

Commits

9b0d43b Backport aiohttp conditional HEAD bug workaround

ebad38c Add release notes for 7.17.5, 7.17.6 and 7.17.7

673a720 Bumps 7.17 to 7.17.8

9221e2d Bumps to version 7.17.7

72f102d Update assemble task - SNAPSHOT

d411230 non-root user to build artifacts inside the container

576fae9 Bumps to version 7.17.6

7b5389d Add release notes for 7.173 and 7.17.4

a0ee925 Bumps to version 7.17.5

2f5e8e2 Bumps to version 7.17.4

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 0
ProcessPoolExecutor

Is your feature request related to a problem? Please describe. Due to the GIL, multithreading isn't really sufficient for fully utilizing a multicore CPU.

Describe the solution you'd like Integrate ProcessPoolExecutor to work around the GIL limitation.
enhancement

opened by markusressel 0
Watch file changes in newly created folders

Is your feature request related to a problem? Please describe. Currently only folders that exist at the time of FileWatcher setup, therefore folders that are created while the FileWatcher is already running are not watched.

Describe the solution you'd like Add new observers on the fly when a new folder has been created.
enhancement

opened by markusressel 0
Logging & Console output

To run py-image-dedup as a daemon, logging should be used more extensively. Almost all output is currently done through click.echo() and even with logging the cli should support displaying the progress in an interactive way.
enhancement

opened by markusressel 0

Releases(v2.0.1)

v2.0.1(Jul 10, 2022)
What’s Changed

convert IFDRational to float (#279) by @fredrike

Poetry (#234) by @markusressel

Dependency upgrades

Source code(tar.gz)
Source code(zip)
v2.0.0(Oct 7, 2020)
This is a breaking release, since most of the configuration has been moved to a config file.

added "analyse" cli command

markusressel/#2_move_instead_of_delete

added preference to enable/disable auto index creation

Feature/#5 file config

moved most of the configuration parameters to the config

added option to disable removal of empty folders

#1 remove folders only if empty due to deduplication

better error handling while threading

better summary output

automatically detect elasticsearch version

support both v6 and v7 (needs patches in image-match too)

elasticsearch fix for new hit count format

Feature/#3 run as service

Feature/#4 docker

feature/#12_statistics

Source code(tar.gz)
Source code(zip)
v1.0.0(Mar 8, 2019)
initial release

Source code(tar.gz)
Source code(zip)

py-image-dedup is a tool to sort out or remove duplicates within a photo library

Related tags

Overview

py-image-dedup

How it works

Phase 1 - Database cleanup

Phase 2 - Counting files

Phase 3 - Analysing files

Phase 4 - Finding duplicates

Phase 5 - Moving/Deleting duplicates

Phase 6 - Removing empty folders (Optional)

How to use

Install

Configuration

Setup elasticsearch backend

Elasticsearch version

Set up the index

Command line usage

Daemon

Dry run

FreeBSD

Encoding issues

Docker

UID and GID

Contributing

License

Comments

tqdm v4.50.0 stable

7.17.7

7.17.6

7.17.5

7.17.4

Client

7.17.3

Client

7.17.2

Client

7.17.1

Helpers

Transport

7.17.0

Machine Learning

Transform

7.16.3

Client

7.16.2

Client

7.17.8

7.17.7

7.17.6

7.17.5

7.17.4

Client

7.17.3

Client

7.17.2

Client

7.17.1

Helpers

Transport

7.17.0

Machine Learning

Transform

7.16.3

Client

Releases(v2.0.1)

v2.0.1(Jul 10, 2022)

What’s Changed

v2.0.0(Oct 7, 2020)

v1.0.0(Mar 8, 2019)

Owner

Markus Ressel

Python command line tool and python engine to label table fields and fields in data files.

cmdpxl: a totally practical command-line image editor

A simple python script to execute a command when a YubiKey is disconnected

Stream comments, submissions from subreddits and users across reddit right in your terminal

Ssl-tool - A simple interactive CLI wrapper around openssl to make creation and installation of self-signed certs easy

A collection of command-line interface games written in python

WA Terminal is a CLI application that allows us to login and send message with WhatsApp with a single command.

Detect secret in source code, scan your repo for leaks. Find secrets with GitGuardian and prevent leaked credentials. GitGuardian is an automated secrets detection & remediation service.