Hi,

This started as a fix for #19 , but has grown somewhat.

So far I have:

• Fixed memory consumption issues (#8, #19, and probably #3) by storing only mail file names (not whole Message objects) while detecting possible duplicate mails. The Messsage objects are loaded from disk while processing clusters of mail files that have the same hash.
• Fixed the cmp issue described in #34
• Attempted to fix several issues with encodings #32 . I have rudimentary fixes forcing decoding of the message lines, but there are still issues. Any help is welcome.
• Added a progress bar in place of printing a '.' every 100 messages

Cheers, Kevin

Since transitioning from a simple script to a package, it's not at all clear how to use this any more. README.rst is missing any installation instructions, and the usage text begins:

Usage: __init__.py [OPTIONS] [MAILDIR [MAILDIR ...]]

which is obviously wrong.

I am using

mdedup -C -1 -S -1 -s delete-smaller maildirname

But I still get "Check that mail differences are within the limits." messages, and nothing is deleted. All duplicates are either ignored or skipped.

│ Mails      │   Metric │
╞════════════╪══════════╡
│ Found      │      227 │
├────────────┼──────────┤
│ Rejected   │        0 │
├────────────┼──────────┤
│ Kept       │      227 │
├────────────┼──────────┤
│ Unique     │        7 │
├────────────┼──────────┤
│ Duplicates │      220 │
├────────────┼──────────┤
│ Deleted    │        0 │
╘════════════╧══════════╛
╒══════════════════════════════════════╤══════════╕
│ Duplicate sets                       │   Metric │
╞══════════════════════════════════════╪══════════╡
│ Total                                │      117 │
├──────────────────────────────────────┼──────────┤
│ Ignored                              │        7 │
├──────────────────────────────────────┼──────────┤
│ Skipped                              │      110 │
├──────────────────────────────────────┼──────────┤
│ Rejected (bad encoding)              │        0 │
├──────────────────────────────────────┼──────────┤
│ Rejected (too dissimilar in size)    │        0 │
├──────────────────────────────────────┼──────────┤
│ Rejected (too dissimilar in content) │        0 │
├──────────────────────────────────────┼──────────┤
│ Deduplicated                         │        0 │

What might be wrong?

Is there a way to rename the files rather than delete duplicates?

Thanks for help.

Vikas

Hi,

I've had a number of unhandled exceptions related to all the Unicode issues (#32, #33, #35). I accidentally originally installed the Python 2 version of maildir-deduplicate, and it is better on the Python 3 version, but several of my mails still manage to cause unhandled exceptions.

These seem to predominantly be mails which have been wrongly encoded - which should have been marked and encoded as UTF, but haven't been. Python does seem to recognize something is off and shift those characters to 0xFFFD, the Unicode Replacement Character, but nevertheless fails with UnicodeEncodeError: 'ascii' codec can't encode character '\ufffd' in position 13: ordinal not in range(128).

I understand that maildir-deduplicate can't magically know in what particular way a fucked up mail was fucked up and treat the wrong data correctly. That mail was encoded wrongly and that's my problem, not maildir-deduplicate's.

What's annoying me is the lack of handling on the exception. There are over 4000 mails in that maildir, the vast majority of which are perfectly RFC-compliant, and I can't parse that folder because a handful of mails are screwed and maildir-deduplicate doesn't properly handle it.

I don't expect the software to magically fix broken input. But if I have 1 broken e-mail out of a thousand, I do expect it to just skip the broken one and do the other 999.

Unicode-errors are the most common ones, but it's not limited to that: I've also had one run fail on me because of a missing header in a collection:

  File "/usr/local/lib/python3.4/dist-packages/maildir_deduplicate/deduplicate.py", line 355, in get_lines_from_message_body
    header_text, sep, body = message.as_string().partition("\n\n")
  File "/usr/lib/python3.4/email/message.py", line 159, in as_string
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib/python3.4/email/generator.py", line 112, in flatten
    self._write(msg)
  File "/usr/lib/python3.4/email/generator.py", line 178, in _write
    self._dispatch(msg)
  File "/usr/lib/python3.4/email/generator.py", line 211, in _dispatch
    meth(msg)
  File "/usr/lib/python3.4/email/generator.py", line 269, in _handle_multipart
    g.flatten(part, unixfrom=False, linesep=self._NL)
  File "/usr/lib/python3.4/email/generator.py", line 112, in flatten
    self._write(msg)
  File "/usr/lib/python3.4/email/generator.py", line 178, in _write
    self._dispatch(msg)
  File "/usr/lib/python3.4/email/generator.py", line 211, in _dispatch
    meth(msg)
  File "/usr/lib/python3.4/email/generator.py", line 269, in _handle_multipart
    g.flatten(part, unixfrom=False, linesep=self._NL)
  File "/usr/lib/python3.4/email/generator.py", line 112, in flatten
    self._write(msg)
  File "/usr/lib/python3.4/email/generator.py", line 186, in _write
    msg.replace_header('content-transfer-encoding', munge_cte[0])
  File "/usr/lib/python3.4/email/message.py", line 559, in replace_header
    raise KeyError(_name)
KeyError: 'content-transfer-encoding'

The fact that you're using exceptions at all is good. But not handling exceptions is bad, and not handling exceptions in a program designed for batch processing is just wrong.

I will try to hack something up for my local installation and I will submit a patch if I succeed, but this is ultimately a question of design mentality: You are currently placing the burden of dealing with problematic input on the user. You're essentially saying "this program will work fine...if you made sure those 10000 mails you want to scan are all RFC-compliant in advance!".

I do believe it would greatly increase the usefulness of this tool if you expected it to fail on some messages and dealt with that gracefully, instead of just crashing back into the terminal in the middle of processing.

Thank you for your efforts. I haven't actually gotten this tool to work yet, but thanks to your work, I at least have a shot at dealing with these mails. I do appreciate the time you're investing.

Preliminary checks

• [x] I am running the latest version
• [x] I have read the Code of Conduct
• [x] I have checked there is not other Issues or Pull Requests covering the same topic to open

Describe the bug

When running mdedup -a delete-discarded -s select-one ./.MyMailDirFolder I get the follwoing result:

NotImplementedError: delete-discarded action not implemented yet.

To reproduce

Steps to reproduce the behavior:

1. The full mdedup -a delete-discarded -s select-one ./.MyMailDirFolder CLI invocation you used.

   $ mdedup -a delete-discarded -s select-one ./.MyMailDirFolder
   

2. The data set leading to the bug. Try to produce here the minimal subset of mails leading to the bug, and add copies of those mails (eventually censored). This effort will help maintainers add this particular edge-case to the set of unittests to prevent future regressions. You can reduce down the issue to a particular deduplicate subset by using the --hash-only parameter.

Expected behavior

Duplicated emails being deleted

Is this an expected behaviour?

# pip3 install --user maildir-deduplicate
[...]
Successfully installed boltons-18.0.0 click-6.7 click-log-0.2.1 maildir-deduplicate-2.1.0 progressbar2-3.37.1 python-utils-2.3.0 tabulate-0.8.2

# ~/.local/bin/mdedup --help
Traceback (most recent call last):
  File "/root/.local/bin/mdedup", line 7, in <module>
    from maildir_deduplicate.cli import cli
  File "/root/.local/lib64/python3.4/site-packages/maildir_deduplicate/cli.py", line 51, in <module>
    @click_log.init(logger)
AttributeError: 'module' object has no attribute 'init'

# python3 --version
Python 3.4.6
# pip3 --version
pip 9.0.1 from /usr/lib64/python3.4/site-packages (python 3.4)

This is on a funtoo system. Used --user so that dependencies wouldn't collide with package manager.

Not having much luck getting this to work:

Traceback (most recent call last):
  File "/tmp/python/bin/mdedup", line 9, in <module>
    load_entry_point('maildir-deduplicate==1.0.1.dev0', 'console_scripts', 'mdedup')()
  File "/tmp/python/local/lib/python2.7/site-packages/click/core.py", line 700, in __call__
    return self.main(*args, **kwargs)
  File "/tmp/python/local/lib/python2.7/site-packages/click/core.py", line 680, in main
    rv = self.invoke(ctx)
  File "/tmp/python/local/lib/python2.7/site-packages/click/core.py", line 1027, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/tmp/python/local/lib/python2.7/site-packages/click/core.py", line 873, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmp/python/local/lib/python2.7/site-packages/click/core.py", line 508, in invoke
    return callback(*args, **kwargs)
  File "/tmp/python/local/lib/python2.7/site-packages/click/decorators.py", line 16, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/tmp/maildir-deduplicate/maildir_deduplicate/cli.py", line 137, in deduplicate
    dedup.add_maildir(maildir)
  File "/tmp/maildir-deduplicate/maildir_deduplicate/deduplicate.py", line 72, in add_maildir
    mail_file, message, self.use_message_id)
  File "/tmp/maildir-deduplicate/maildir_deduplicate/deduplicate.py", line 95, in compute_hash
    canonical_headers_text = cls.canonical_headers(mail_file, message)
  File "/tmp/maildir-deduplicate/maildir_deduplicate/deduplicate.py", line 115, in canonical_headers
    for value in mail.get_all(header):
AttributeError: Message instance has no attribute 'get_all'

Hi @kdeldycke

Since the changes we made yesterday I have been able to analyse a bigger set of my maildir. It appears that we missed a few encoding issues. The attached seems to fix them, but checking this on another test-set (specifically a non-English maildir) would be wise.

Cheers, K

I tried to use mdedup on my maildir with 8276 and got the following error:

Traceback (most recent call last):
  File "/home/user/bin/mdedup", line 11, in <module>
    sys.exit(cli())
  File "/home/user/.local/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/user/.local/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/user/.local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/user/.local/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user/.local/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/user/.local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/user/.local/lib/python2.7/site-packages/maildir_deduplicate/cli.py", line 139, in deduplicate
    dedup.add_maildir(maildir)
  File "/home/user/.local/lib/python2.7/site-packages/maildir_deduplicate/deduplicate.py", line 80, in add_maildir
    mail_file, message, self.use_message_id)
  File "/home/user/.local/lib/python2.7/site-packages/maildir_deduplicate/deduplicate.py", line 103, in compute_hash
    canonical_headers_text = cls.canonical_headers(mail_file, message)
  File "/home/user/.local/lib/python2.7/site-packages/maildir_deduplicate/deduplicate.py", line 125, in canonical_headers
    canonical_value = cls.canonical_header_value(header, value)
  File "/home/user/.local/lib/python2.7/site-packages/maildir_deduplicate/deduplicate.py", line 148, in canonical_header_value
    value = re.sub('\s+', ' ', value).strip()
  File "/usr/lib/python2.7/re.py", line 155, in sub
    return _compile(pattern, flags).sub(repl, string, count)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xdc in position 4: ordinal not in range(128)

Unfortunately I can't identify the message, which causes this error. mdedup -v doesn't show more information. How can I find the problematic message?

I am getting a reproducible error:

I use this command python init.py -o /path-to-my-dir/getmaildir/

This is the whole output (yes, there are a lot of mails in that dir)

Processing 460581 mails in /mnt/rraid/COMPRESSED/Backup/Mails/getmaildir .........................Traceback (most recent call last):
  File "__init__.py", line 572, in <module>
    main()
  File "__init__.py", line 516, in main
    duplicates_run(opts, maildir_paths)
  File "__init__.py", line 534, in duplicates_run
    mail_count += collate_folder_by_hash(mails_by_hash, maildir, opts.message_id)
  File "__init__.py", line 287, in collate_folder_by_hash
    mail_hash, header_text = compute_hash_key(mail_file, message, use_message_id)
  File "__init__.py", line 271, in compute_hash_key
    canonical_headers_text = get_canonical_headers(mail_file, message)
  File "__init__.py", line 182, in get_canonical_headers
    canonical_value = get_canonical_header_value(header, value)
  File "__init__.py", line 241, in get_canonical_header_value
    utc_timestamp = email.utils.mktime_tz(parsed)
  File "/usr/lib/python2.7/email/_parseaddr.py", line 154, in mktime_tz
    if data[9] is None:
TypeError: 'NoneType' object has no attribute '__getitem__'

any ideas?

(update: the error message)

The way I understand it, mail-deduplicate now requires python 3.7 or later. Bionic is on 3.6. Given that it is an LTS release of Ubuntu would it be possible to support python 3.6 as well?

Bumps click-extra from 3.5.0 to 3.6.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps coverage from 7.0.0 to 7.0.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Description

This PR is ready to be merged. The merge event will trigger[^1] the:

1. creation of a v7.1.1 tag on main branch
2. build if project based on Poetry, then release to PyPi
3. publication of a GitHub release

[^1]: as defined by release.yaml.

How-to release v7.1.1

1. click Re-run all jobs from the workflow run, to refresh the release date to today
2. wait for the re-run to complete and check the result in diff view
3. click Ready for review button below, to get this PR out of Draft mode
4. click Rebase and merge button below (‼️ do not ~Squash and merge~: the auto-tagging job needs the 2 distinct commits in this PR)

Description

Ready to be merged into main branch, at the discretion of the maintainers, to bump the minor part of the version number.

To bump version to v7.2.0

1. click Ready for review button below, to get this PR out of Draft mode
2. click Rebase and merge button below

Description

Ready to be merged into main branch, at the discretion of the maintainers, to bump the major part of the version number.

To bump version to v8.0.0

1. click Ready for review button below, to get this PR out of Draft mode
2. click Rebase and merge button below