Hi guys,

here's a first proposition of integration of Let's Encrypt (and more generally certificate management).

It adds the following features :

Check certificate status

usage: yunohost domain cert-status [-h] [--full] [domainList [domainList ...]]

positional arguments:
  domainList  Domains to check

optional arguments:
  -h, --help  show this help message and exit
  --full      Show more details

Install Let's Encrypt certificates

usage: yunohost domain cert-install [-h] [--force] [--no-checks]
                                    [--self-signed]
                                    [domainList [domainList ...]]

positional arguments:
  domainList     Domains for which to install the certificates

optional arguments:
  -h, --help     show this help message and exit
  --force        Install even if current certificate is not self-signed
  --no-checks    Does not perform any check that your domain seems correcly
                 configured (DNS, reachability) before attempting to install.
                 (Not recommended)
  --self-signed  Install self-signed certificate instead of Let's Encrypt

Renew Let's Encrypt certificate

usage: yunohost domain cert-renew [-h] [--force] [--email] [--no-checks]
                                  [domainList [domainList ...]]

positional arguments:
  domainList   Domains for which to renew the certificates

optional arguments:
  -h, --help   show this help message and exit
  --force      Ignore the validity treshold (30 days)
  --email      Send an email to root with logs if some renewing fails
  --no-checks  Does not perform any check that your domain seems correcly
               configured (DNS, reachability) before attempting to renew. (Not
               recommended)

Some more details / explanations

• Let's Encrypt certificates are managed using acme-tiny. The choice was driven by the fact that it is a lightweight module, somewhat easy to use, and can be installed through pip install acme-tiny ;
• I edited the ssowatconf function to automatically authorize the ACME challenge uri /.well-known/blahif it is present in the nginx conf ;
• cert-install automatically adds a small cron-job (2 lines of code), calling yunohost domain cert-renew --email every week, which will renew Let's Encrypt certificates which are about to expire, and send an email to root if some renewing fails ;
• By default, cert-install and cert-renew include some checks that the domain seems to have a reasonable setup to attempt Let's Encrypt certificate installation (DNS poiting to global IP, + being able to GET a page) ;
• I moved the generation of self-signed certificate into a new file, and cert-install can be used to regenerate a self-signed certificate (in case you don't want Let's Encrypt anymore for some reason). The short/mid-term goal is also to refactorize this part of code to get rid of os.system() calls and use the OpenSSL.crypto module instead.

There are more features we can think about including in the future, such as supporting other ACME authorities, supporting DNS-based challenge, set level of security (e.g. deactivate TLS v1.0 and 1.1), import certificates from non-ACME authorities, provide a diagnosis for domain (e.g. inform user a Let's Encrypt cert can be installed, display current SSL Labs rating) etc... But for now, let's focus on the basic stuff :)

Remaining work

• How do we manage acme-tiny (and python-tabulate) as dependencies. At the moment this code does not work if you didn't do a pip install acme-tiny and pip install tabulate for example.
• How do we make sure the transition from letsencrypt_ynh is smooth
• ???

Cheers !

The problem

• There are no way to protect a permission.
• There are no way to set multiple uris for the same permission.
• There are no clean way to manage multiple uris for the same application.
• https://github.com/YunoHost/issues/issues/1486
• https://forum.yunohost.org/t/yunohost-3-7-spooky-testing-call-for-feedback/9385/31?u=josue
• https://github.com/YunoHost/issues/issues/1420
• https://github.com/YunoHost/issues/issues/1534

Solution

• Add an attribute in LDAP to know if this permission is protected (solve ttps://github.com/YunoHost/issues/issues/1486)
• Add an attribute in LDAP to know if we need to set the auth HEADER or not to the application (solve https://github.com/YunoHost/issues/issues/1420)
• Add the possibility to have multiple uris for the same permission. Note that we will have one main uris for each permission an optionally we will have a list of additional uris.
• Add an attribute in LDAP to have the label for each permissions and add on other attribute to know if we will show a tile in the SSO for this permission (solve https://github.com/YunoHost/issues/issues/1534)
• Rework the ssowatconf.json to pass all this informations to SSOwat.

Here is an example of how will look like the new ssowatconf.json.

{
    "additional_headers": {
        "Auth-User": "uid", 
        "Email": "mail", 
        "Name": "cn", 
        "Remote-User": "uid"
    }, 
    "domains": [
        "domainA.tld", 
        "domainB.tld"
    ], 
    "permissions": {
        "sogo.main": { // Equals to actual protected_uris
            "users": [
                "alice",
                "bob"
            ],
            "label": "SOGo", // Mandatory
            "show_tile": true,
            "auth_header": true,
            "protected": true,
            "uris": [   // Not mandatory, note that the first entry will be the main uris, the others will be the additional uris.
                "domainA.tld/SOGo",
                "domainB.tld/something",
                "domainA.tld/someotherthing"
            ]
        },
        "gitea.main": { // Equals to actual skipped_uris
            "users": [
                "alice",
                "jack"
            ],
            "label": "Gitea",
            "show_tile": true,
            "auth_header": false,
            "protected": false,
            "uris": [
                "domainB.tld/gitea"
            ]
        },
        "gitea.admin": { // Equals to actual unprotected_uris
            "users": [
                "alice"
            ],
            "label": "Gitea admin",
            "show_tile": false,
            "auth_header": true,
            "protected": false,
            "uris": [
                "domainB.tld/gitea/admin"
            ]
        },
        "regex-example.main": { // Equals to actual unprotected_regex
            "users": [
                "alice"
            ],
            "label": "Regex example",
            "show_tile": true,
            "auth_header": true,
            "protected": false,
            "uris": [
                "re:domainB.tld/test[a-z]+/example"
            ]
        },
        "other-example.main": { // Equals to actual protected but with no auth header
            "users": [
                "alice"
            ],
            "label": "Other example B",
            "show_tile": false,
            "auth_header": false,
            "protected": true,
            "uris": [
                "re:domainB.tld/test[1-9]+/other-example"
            ]
        },
        "core": { // Core permission
            "users": [], // Have no sens here, so just put nothing...
            "label": "Core permissions",
            "show_tile": false,
            "auth_header": false,
            "protected": false,
            "uris": [
                "^[^/]*/%.well%-known/acme%-challenge/.*$", 
                "^[^/]*/%.well%-known/autoconfig/mail/config%-v1%.1%.xml.*$"
            ]
        }
    }, 
    "portal_domain": "domain.tld", 
    "portal_path": "/yunohost/sso/", 
    "redirected_regex": {
        "domain.tld/yunohost[\\/]?$": "https://domain.tld/yunohost/sso/"
    }, 
    "redirected_urls": {}, 
}

PR Status

• [x] Implement a migration
• [x] Manage backup/restore
• [x] Implement the unit test
• [x] Test the code (done by the unit test)
• [x] Try to remove the big legacy code in app_ssowatconf. https://github.com/YunoHost/yunohost/pull/935
• [x] Update webadmin with new feature (attribute : label, show_tile). https://github.com/YunoHost/yunohost-admin/pull/297
• [x] Update documentation https://github.com/YunoHost/doc/pull/1263
• [x] Test everything :smile: Done with theses apps (maybe we need to test with some other tricky app, but I don't know which one):
  • wordpress
  • nextcloud
  • gitea
  • etherpad_mypads
  • leed
• [ ] Fix the test (maybe we need to run the migration before to launch the test). Locally all test pass.

How to test

• Create a protected permission (in a app) or just use the mail permission (which should be protected).
• Try to add or remove the visitors group in this permission. ...

Note that to test this with the unit test you need this branch : https://github.com/YunoHost/test_apps/pull/5

Validation

• [ ] Principle agreement 0/2 :
• [ ] Quick review 0/1 :
• [ ] Simple test 0/1 :
• [ ] Deep review 0/1 :

Problems

To enhance applications protection against hackers/spammers, etc., we can propose helpers to ease the creating of fail2ban jails.

Solution

Add ynh_add_fail2ban_config and ynh_remove_fail2ban_config helpers. A successful implementation example is the piwigo app.

PR Status

Working, but opinion welcome! And should be implemented in other applications to validate its principle.

Validation

• [ ] Principle agreement 0/2 :
• [ ] Quick review 0/1 :
• [ ] Simple test 0/1 :
• [ ] Deep review 0/1 :

The problem

Sharing files with XMPP's http upload mechanism is currently impossible. This PR is an attempt to address issue #1278

Metronome's configuration is ready for http upload but port 5290 is not reachable.

Solution

http upload requires a dedicated subdomain (I have chosen jabber.thedomain.net instead of upload.thedomain.net to avoid possible conflicts with possible other "upload" things).

This subdomain should be reachable via HTTPS so we need:

• [X] add a DNS entry for that subdomain
• [X] transparently include jabber.thedomain.net as a SAN in the same certificate for thedomain.net
• [X] explicitly define the storage path in metronome's config (the same will be configured in nginx)
• [X] create an nginx config for that subdomain
• [x] silently recreate existing Letsencrypt certificates

PR Status

The PR ready for review.

How to test

Create 2 accounts : alice and bob Install Dino and configure those accounts. Happily share pictures between alice and bob.

Validation

• [ ] Principle agreement 0/2 :
• [ ] Quick review 0/1 :
• [ ] Simple test 0/1 :
• [ ] Deep review 0/1 :

I pull request just for memo and to signal this work (it was in my _common.sh file for packaging...).

The work is unfinished. The api is similar to ynh_mysql helpers.

I purpose this change to improve the helper 'ynh_install_app_dependencies'. Before this change if the dependences are not installable the install didn't fail. By these change the helper generate an error and the install stop.

Problem

issue: https://forum.yunohost.org/t/probleme-mail-dns-2d-domaine/3460

Solution

• Change the header
• add anonymisation
• En fait, cette correction est le fruit d'une recherche de près de 10h sur postfix. Finalement, j'ai trouvé la solution dans la conf d'un autre logiciel, par hasard. Par soucis d'honnêteté, le site dont j'ai puisé cette conf est mailinabox : https://github.com/mail-in-a-box/mailinabox/blob/master/conf/postfix_outgoing_mail_header_filters

PR Status

REOPENED TEST/ REVIEW Highly Needed (RHN)

EXAMPLE (for JimBoJoe and Reviewers)

before with postsrsd :

Return-Path: <[email protected]> # ou 'Return-Path: <[email protected]>' sans Postsrsd 

Received: from mwinf5c54 (mwinf5c54.ANONYME.net [10.23.111.104])
by mwinb1c03 with LMTPA;
Wed, 13 Sep 2017 15:28:52 +0200

X-Sieve: CMU Sieve 2.3

Received: from domainprincipal.tld ([45.065.99.90])
by mwinf5c54 with ME
id 91Ur1w00Q0RFXA5011UrSl; Wed, 13 Sep 2017 15:28:52 +0200

X-bcc: [email protected]

X-ME-bounce-domain: DomainExterne.tld

X-ME-engine: default

X-me-spamcause: (0)(0000)gggruggvucftvghtrhhoucdtuddrfeelledrgeeggdeijecutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfogfdpggftiffpkfenuceurghilhhou
hhtmecugedttdenucenucfjughrpefhvffukffftgggsehttdertddttdejnecuhfhrohhmpefluhhlihgvnhcuifhomhgvshcuffhirghsuceojhhulhhivghnsegrshg
tphgrphhhihhlrghtvghlihgvrdgvuheqnecukfhppedugedurddvheehrddufedtrdduleenucfrrghrrghmpehhvghlohepsheggedrvghupdhinhgvthepudeguddrv
dehhedrudeftddrudelpdhmrghilhhfrhhomhepshhrshdtpehofigufeeprghopegrshgtphgrphhhihhlrghtvghlihgvrdgvuhepjhhulhhivghnsehsgeegrdgvuhd
prhgtphhtthhopegtlhhjuhegjeeisehorhgrnhhgvgdrfhhr

X-me-spamlevel: not-spam

X-ME-Helo: domainprincipal.tld

X-ME-IP: 45.065.99.90  # MY IP adress

X-ME-Entity: ofr

Received: from domainprincipal.tld (localhost [IPv6:::1])
by domainprincipal.tld (Postfix) with ESMTPSA id C22136B5
for <[email protected]>; Wed, 13 Sep 2017 15:28:50 +0200 (CEST)

From: Name of the user <[email protected]>

To: [email protected]

Subject: ceci est un message de test

Message-ID: <20170913132850.Horde.JcbFlT1kgDPDbzDz7ZcP2Ca@mail.server-d-envoi-test-domaine.tld>

Date: Wed, 13 Sep 2017 13:28:50 +0000

...

DKIM-Signature: ANONYME-LK2R

After :

Return-Path: <[email protected]>

Received: from mwinf5c56 (mwinf5c56.ANONYME.net [10.23.111.106])
by mwinb1c03 with LMTPA;
Tue, 26 Sep 2017 10:36:54 +0200

X-Sieve: CMU Sieve 2.3

Received: from domainprincipal.tld ([45.065.99.90])
by mwinf5c56 with ME
id E8ct1w0030RFXA5018ct2f; Tue, 26 Sep 2017 10:36:54 +0200

X-bcc: [email protected]

X-ME-bounce-domain: DomainExterne.tld

X-ME-engine: default

X-me-spamcause: (0)(0000)gggruggvucftvghtrhhoucdtuddrfeelledrjedvgddtkecutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfogfdpggftiffpkfenuceurghilhhou
hhtmecugedttdenucenucfjughrpefhvffukffftgggsehttdertddttdejnecuhfhrohhmpefluhhlihgvnhcuifhomhgvshcuffhirghsuceojhhulhhivghnsegrshg
tphgrphhhihhlrghtvghlihgvrdgvuheqnecukfhppedugedurddvheehrddufedtrdduleenucfrrghrrghmpehhvghlohepsheggedrvghupdhinhgvthepudeguddrv
dehhedrudeftddrudelpdhmrghilhhfrhhomhepjhhulhhivghnsegrshgtphgrphhhihhlrghtvghlihgvrdgvuhdprhgtphhtthhopegtlhhjuhegjeeisehorhgrnhh
gvgdrfhhr

X-me-spamlevel: not-spam

X-ME-Helo: domainprincipal.tld

X-ME-IP: 45.065.99.90

X-ME-Entity: ofr

Received: from authenticated-user (PRIMARY_HOSTNAME [PUBLIC_IP])nge.fr>; Tue, 26 Sep 2017 10:36:52 +0200 (CEST)

From: Name Of my User <[email protected]>

To: [email protected]

Subject: ceci est un message de test

Message-ID: <20170926083652.Horde.yPJwWVW9iq7zfLHZDP99fPB@mail.server-d-envoi-test-domaine.tld>

Date: Tue, 26 Sep 2017 08:36:52 +0000
...
DKIM-Signature: ANONYME-LK2R

Remerciement

Donc, un grand merci à eux (sans le vouloir).

Hello,

As reported here or here, YunoHost post install fails on slow hardware/vps because slapd is to slow to restart itself after its regen-conf.

This patch is an attempt to fix this but I don't have a good testing environment (my vagrant is too fast for that). Maybe testing that it's possible to run something using the admin user could be a better test but I don't see how to do it easily.

A workarround would be to use my patch to runs this kind of operation using root instead of admin but this is a workaround, not a real fix (and this bug could still generate other problems).

Cheers,

Based on the work from @mbugeia in #58 , rebased onto current unstable.

List of modifications applied after this initial work :

• ou=aliases (instead of ou=alias) to follow ou=users, ou=domains, ...
• Removing alias_update for the moment (one can delete and re-create)
• Main modification : do not make alias entries in LDAP inherit from inetOrgPerson. inetOrgPerson are for people (with first name, surname, ...). Instead, create a specific dedicated mailAlias objectclass. Alias objects in ldap inherits from both mailAlias and mailAccount. mailAlias is the "structural" class for holding the aliases. It enforces that the maildrop be non empty. mailAccount holds the mail address/mail forward addresses. Using mailAccount (as it is done for users) makes it transparent for most downstream applications (postfix, all apps searching for mail accounts in ldap) so that no modification is required for them.
• add support for mail-forward with mail addresses containing a '+' separator
• remove the need to call alias_init : the creation of the ou=aliases in the ldap db is done automatically at first use.

The problem

Discussion about using a dedicated php service for each app https://github.com/YunoHost/issues/issues/1536

Solution

It works perfectly

PR Status

We should probably reload the main service during an upgrade though.

How to test

Use this config file as an extra helper to source into the scripts.

Validation

• [ ] Principle agreement 0/2 :
• [ ] Quick review 0/1 :
• [ ] Simple test 0/1 :
• [ ] Deep review 0/1 :

I get lots of suggestion of indexes by slapd in my logs:

<= mdb_equality_candidates: (cn) not indexed <= mdb_equality_candidates: (gidNumber) not indexed <= mdb_equality_candidates: (mail) not indexed <= mdb_equality_candidates: (member) not indexed <= mdb_equality_candidates: (memberUid) not indexed <= mdb_equality_candidates: (sudoUser) not indexed <= mdb_equality_candidates: (uidNumber) not indexed <= mdb_equality_candidates: (uniqueMember) not indexed <= mdb_equality_candidates: (virtualdomain) not indexed <= mdb_substring_candidates: (sudoUser) not indexed

Since Yunohost makes it hard to edit LDAP server configuration (see https://github.com/YunoHost/issues/issues/1350), the default configuration should contain indexes for the fields used by Yunohost a lot.

The problem

...

Solution

...

PR Status

...

How to test

...

Validation

• [ ] Principle agreement 0/2 :
• [ ] Quick review 0/1 :
• [ ] Simple test 0/1 :
• [ ] Deep review 0/1 :

The problem

When we enable dovecot pop3 on sans-nuage.fr, the dovecot fail with error:

Fatal: service(pop3) access(/usr/lib/dovecot/pop3) failed: No such file or directory

Solution

Install dovecot-pop3d package

PR Status

Ready

How to test

...

The problem

• Fix and improve handling of PasswordAuthentication option
• Add a setting for PermitRootLogin usage on local network

Solution

• Directly use the value of ssh_password_authentication
• Add ssh_allow_root_on_localnet to manage the last part of the SSH template

PR Status

First release and review

How to test

Play with the ssh settings.

The problem

We can't use another shell like sh or zsh.

Solution

Add the possibility to change it

PR Status

Yolotested in prod.

Maybe we should manage the moment when the shell is removed from the system, but meh, it's not simple.

How to test

...

The problem

OCSP stapling is in fact broken because nginx complains in the log file :

r3.o.lencr.org could not be resolved (110: Operation timed out) while requesting certificate status, responder: r3.o.lencr.org, certificate: "/etc/yunohost/certs/domain.tld/crt.pem"

Also related to : https://github.com/YunoHost/issues/issues/1099

Solution

I don't know why, but Nginx doesnt like 127.0.0.1 as a resolver, despite the fact that dnsmasq listens on 0.0.0.0:53 and a dig request on 127.0.0.1 does work ...

Using Google resolver works, but this is probably not what we want, so opening as draft PR only

PR Status

Using Google resolver works, but this is probably not what we want, so opening as draft PR only

How to test

openssl s_client -connect domain.tld:443 -status | grep OCSP

should display:

OCSP response: 
OCSP Response Data:
    OCSP Response Status: successful (0x0)