Hi,

I am trying to use django-configurations with a Python3/Django-1.8.8project: everything works fine when run from the command line -- e.g., "python manage.py runserver" - yet fails when from inside PyCharm (env variables are set in the run settings pane) as follows:

DJANGO_CONFIGURATION=Dev;DJANGO_SETTINGS_MODULE=myApp.settings;PYTHONUNBUFFERED=1

The frameworks and interpreter are set up using a virtual environment; again, when run from the command line everything is fine.

django-configuration is set up as follows:

wsgi.py:

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myApp.settings')
os.environ.setdefault('DJANGO_CONFIGURATION', 'Dev')

from configurations.wsgi import get_wsgi_application

application = get_wsgi_application()

manage.py:

import os
import sys

if __name__ == "__main__":
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myApp.settings')
    os.environ.setdefault('DJANGO_CONFIGURATION', 'Dev')

    from configurations.management import execute_from_command_line
    execute_from_command_line(sys.argv)

From settings:

import os
from configurations import Configuration, values

# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/

class Common(Configuration):
#  ...lots of common settings....


class Prod(Common):
    # SECURITY WARNING: don't run with debug turned on in production!
    DEBUG = False
    # AWS/S3 - PRODUCTION
    AWS_S3_SECURE_URLS = False       # use http instead of https
    AWS_QUERYSTRING_AUTH = False     # don't add complex authentication-related query parameters for requests
# ... even more PROD settings... 



class Dev(Common):
    DEBUG = True
    TEMPLATE_DEBUG = DEBUG
    #AWS/S3 - DEVELOPMENT
    AWS_S3_SECURE_URLS = False       # use http instead of https
    AWS_QUERYSTRING_AUTH = False     # don't add complex authentication-related query parameters for requests
# ... development settings... 

Any suggestions would be appreciated...

Console output

/Users/spector/Work/myApp/BackEnd/myAppSensorBackend/env/bin/python /Applications/PyCharm.app/Contents/helpers/pydev/pydevconsole.py 55302 55303
PyDev console: starting.

import sys; print('Python %s on %s' % (sys.version, sys.platform))
import django; print('Django %s' % django.get_version())
sys.path.extend(['/Users/spector/Work/myApp/BackEnd/myAppSensorBackend', '/Users/spector/Work/myApp/BackEnd/myAppSensorBackend/src', '/Applications/PyCharm.app/Contents/helpers/pycharm', '/Applications/PyCharm.app/Contents/helpers/pydev'])
if 'setup' in dir(django): django.setup()
import django_manage_shell; django_manage_shell.run("/Users/spector/Work/myApp/BackEnd/myAppSensorBackend/src")
Python 3.5.1 (default, Jan 22 2016, 08:54:32) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Django 1.8.8
Traceback (most recent call last):
  File "<input>", line 5, in <module>
  File "/Users/spector/Work/myApp/BackEnd/myAppSensorBackend/env/lib/python3.5/site-packages/django/__init__.py", line 17, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/Users/spector/Work/myApp/BackEnd/myAppSensorBackend/env/lib/python3.5/site-packages/django/conf/__init__.py", line 48, in __getattr__
    self._setup(name)
  File "/Users/spector/Work/myApp/BackEnd/myAppSensorBackend/env/lib/python3.5/site-packages/django/conf/__init__.py", line 44, in _setup
    self._wrapped = Settings(settings_module)
  File "/Users/spector/Work/myApp/BackEnd/myAppSensorBackend/env/lib/python3.5/site-packages/django/conf/__init__.py", line 92, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 662, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/Users/spector/Work/myApp/BackEnd/myAppSensorBackend/src/myApp/settings.py", line 132, in <module>
    class Prod(Common):
  File "/Users/spector/Work/myApp/BackEnd/myAppSensorBackend/env/lib/python3.5/site-packages/configurations/base.py", line 32, in __new__
    raise ImproperlyConfigured(install_failure)
django.core.exceptions.ImproperlyConfigured: django-configurations settings importer wasn't correctly installed. Please use one of the starter functions to install it as mentioned in the docs: http://django-configurations.readthedocs.org/

This incorporates both @jrdietrick and @blueyed open PRs, bumps version to 0.9 and updates the changelog. I think give it a week for review and comment and then release a new version for christmas :)

Using the cookbook [1] to setup celery and django-configuration makes django freezing.

Ctrl+C will not help but kill <PID>

This is also described in #145

The workaround described there is to use

from configurations import importer
importer.install()

Using this settings make celery works again. The cookbook should point out the correct celery settings!

[1] http://django-configurations.readthedocs.io/en/stable/cookbook/#calling-a-django-management-command

I think its time to release a new version with support for Django 3+ and Python 3.6+

• [x] #315
• [x] #309
• [x] #310
• [x] #319
• [x] #320
• [x] Publish release: https://github.com/jazzband/django-configurations/releases/tag/untagged-6ba2bc9f41d4148913fc
• [x] Get Project Lead to push to PyPI: https://github.com/jazzband/help/issues/254
• [x] Update docs on RTD

Disclaimer: I have read the discussions in https://github.com/jezdez/django-configurations/issues/105#issuecomment-88895512 and https://github.com/jezdez/django-configurations/commit/0ff4c7612d28036449edd4b2cf3b58ca1eded0f2, and I understand the situation this package is in. I pondered long and hard as to whether I should even open this pr.

I completely get it; by the time I am done: root-causing this issue, learning how configurations works, learning what the Django guys were trying to do in 1.8, learning how to run tox, writing tests, and finally authoring this pr, I will have spent many hours on these three lines of diff. Hours I could have spent on "real work". I'm just fortunate -- and love (and need) this package enough! -- that my current workload happens to be light enough, and I happen to have enough caffeine, to make it happen. I fully understand that it could take hours more of time that someone doesn't have to confidently sign off on it.

tl;dr: If there's no time to even read this pull req, let alone merge it, no hard feelings!

OK, so here's the deal:

Django switched from optparse to argparse in Django 1.8 for extending management commands. To ease deprecation, until Django 1.10, there's some fun shim logic in Django. It says, "If the command I'm running appends anything to its option_list (the old way), we'll still use optparse. Otherwise, we'll use argparse."

Note that, until Django 1.10, the old optparse behavior gets checked first. In other words, if option_list is manipulated and add_arguments (the new way) is implemented, we still use optparse, and anything in add_arguments is ignored.

That's important for configurations, because it means that we can't just, for instance, add our --configuration option to option_list up front and deal with the problem when it really gets deprecated in Django 1.10. In order to correctly wrap management commands from arbitrary packages, which may or may not have shifted over to add_arguments yet, we have to wait and see what the command does (and thus what parser we get back), and then play our hand.

https://github.com/jezdez/django-configurations/commit/6ce3740365d92c329c1e20036266cd090d57b7c4 made an attempt at doing this, but the base.BaseCommand.option_list += configuration_options causes two problems, actually:

1. It doesn't actually add the configuration option to the command we are running right now. We just got handed the parser, which already looked at BaseCommand.option_list. So this line only adds the option for any commands run after this. This is the subject of the test in the second commit of this pr, which fails on Django 1.8 if run without the patch in the other commit.
2. We're touching base.BaseCommand.option_list, which is a class variable. If we now run another management command in the same Django instance (say, via call_command), we will add the --configuration option a second time. Do this enough times (three, exactly), and you'll get an OptionConflictError. I encountered this while using django-nose, which evidently runs some subcommands under its test command. This is the cause of #109, and pr #110 implements a slightly hacky but working fix, but doesn't fix problem 1 above.

The proposed patch fixes both issues by:

• maintaining the existing "wait and see" behavior that accommodates both mgmt commands which have and have not upgraded their argument extension
• handling the optparse fallback case by one-off adding our argument(s) to the parser in our wrapped routine, leaving the global BaseCommand.option_list untouched

P.S. - Using the three management commands below and then running manage.py command1 on Django 1.8 is enough to minimally repro the OptionConflictError problem. It would not be difficult to write a test to validate this doesn't happen any longer, but seemed like overkill.

# command1.py
class Command(BaseCommand):

    option_list = BaseCommand.option_list + (
        make_option('--arg1', action='store_true'),
    )

    def handle(self, *args, **options):
        call_command('command2')

# command2.py
class Command(BaseCommand):

    option_list = BaseCommand.option_list + (
        make_option('--arg2', action='store_true'),
    )

    def handle(self, *args, **options):
        call_command('command3')

# command3.py
class Command(BaseCommand):

    option_list = BaseCommand.option_list + (
        make_option('--arg3', action='store_true'),
    )

    def handle(self, *args, **options):
        print 'hi'

Django 1.5.1 django-configurations 0.2.1

When I run any command it says:

ImproperlyConfigured: The SECRET_KEY setting must not be empty.

But SECRET_KEY is set in Base configuration.

Looks like somehow it does not pick up class-based configs: when I run ./manage.py, only django commands are listed.

These changes add a few additional badges to the README. Thus should enhance transparency and make the project on PyPI more easily acessible.

Note that this now shows that building the docs failed. Not sure why, because building them locally doesn't fail asking for a tests/requirements.txt file. :confused:

@jezdez I added a django version check to use CommandParser instead of LaxOptionParser if django>=1.8. I've tested it locally, this commit closes #65. django-configurations and pytest-django tests succeed.

Thank you for the update. I've been looking forward to this for a while. It fixes somethings that didn't work (for me), but I figured out some work-arounds so I'm ok using the last version for now.

So after the new update:

% ./manage.py shell_plus --ipython
Usage: manage.py [options]

manage.py error: no such option: --ipython

This is valid and works in the previous version. Basically any option passed doesn't work.

% ./manage.py syncdb --noinput
Usage: manage.py [options]

manage.py error: no such option: --noinput

Did I need to alter manage.py or anything else as I upgraded from the old version?

Thanks, Shige

Noticed this in recent HEAD version (5ece10704) in comparison to what is on PyPI.

I have a configuration setup with a Common class (which inherits from Configuration) and then Local and Production subclasses. The Production subclass has a SecretValue like so:

EMAIL_HOST_PASSWORD = values.SecretValue(environ_prefix=None, environ_name='SENDGRID_PASSWORD')

Even when DJANGO_CONFIGURATION is set to "Local", a ValueError is being thrown that the secret value must be set. Previous versions would only throw this error if the configuration was set to "Production". It seems that when environ_name is manually set, validation occurs when it shouldn't be.

Either the Value.__call__ code should be fixed or SecretValue should have late_binding = True by default.

Celery 2.5.x used to work with django-configurations.

3.0 do not.

it seem it's an issue with djcelery.loaders.

here is the issue on their side:

https://github.com/celery/django-celery/issues/186

is there anyway to run configurations.importer. install() for those applications that rely on django internal importlib.import_module(self.SETTINGS_MODULE)?

If a project wants to use django-configurations then doing so precludes the use of --config (or any other shortening of --configuration) in all the management commands in the project, including those inherited from other packages.

This appears to be due to the default behaviour of argparse.ArgumentParser, specifically the allow_abbrev argument, in combination with they way that ConfigurationImporter checks for the configuration to use from the command line -- namely by running Django's argument parser before any other arguments have been added.

I'm hitting this with https://github.com/thread/django-lightweight-queue/, whose queue_runner command has a --config which is being hidden by this behaviour. The result of which is that queue_runner --config=./special-config.py results in an AttributeError: "Couldn't find configuration './special-config.py' in module '.settings'".

While in some cases it may be possible to change to just not using conflicting argument names, this won't always be possible if, for example, the name comes from a separate library.

I suspect it may be possible to fix this in django-configurations by changing ConfigurationImporter to set allow_abbrev=False in the parser it creates in check_options, though I realise this might be considered a breaking change.

Fixes #267. Related issue in the django-stubs repo - https://github.com/typeddjango/django-stubs/issues/417

I've also added a section to the docs describing how to use this.

# …/settings.py
from configurations import Configuration

class MyConfiguration(Configuration):
    DEBUG = False
    # …

# …/test_thing.py
from django.test import modify_settings

@modify_settings(DEBUG=True)
def test_settings():
    from django.conf import settings

    assert hasattr(settings, "DEBUG")
    assert settings.DEBUG

Running the above test will blow up here with the error TypeError: 'bool' object is not iterable. This makes sense, looking at the code:

            try:
                # When called from SimpleTestCase.setUpClass, values may be
                # overridden several times; cumulate changes.
                value = self.options[name]
            except KeyError:
                value = list(getattr(settings, name, []))  # <-- Boom!

In that context, getattr(settings, name, …) gets the value of MyConfiguration.DEBUG, which is False.

In my opinion, an end-user who uses an application that was built using django-configurations should in most cases not see an unwieldy traceback, but instead a nicely formatted error detailing what went wrong and what they should do to rectify it. In this regard, the PR is not aimed at application developers, but instead aims to improve the user experience of those end-users.

As an example, instead of a traceback, the user now receives something like this:

Couldn't setup values of configuration Debug
    * DEBUG was given an invalid value
        - DEBUG is taken from the environment variable DJANGO_DEBUG as a BooleanValue
        - '5' was received but that is invalid
    * Value of DEBUG2 could not be retrieved from environment
        - DEBUG2 is taken from the environment variable DJANGO_DEBUG2 as a BooleanValue

Additionally, this PR closes #332 because I was getting annoyed at the failing tests

This feature is implemented by introducing our own custom exception type that is raised during Value setup and processing and caught by our importer. When caught, an error handler pretty-prints the information that is included in the exception. This information is, in turn, automatically retrieved from the relevant Value.

THIS IS STILL A WORK IN PROGRESS because I still want to include more actionable information in the error message. Nonetheless, the PR gives others the opportunity to chime in about the feature.

TODOs:

• [ ] Add more information to error messages (e.g. which possible values are accepted in the environment or help texts)
• [ ] Add automatic generation of example values where that makes sense (e.g. for django secret keys)
• [ ] Update documentation to explain how application developers can raise their own ConfigurationErrors to display well-formatted errors to end-users, how help_text, help_reference and example_generator can be used
• [ ] Get Code Quality up to standards (flake8 + test coverage)

django-cache-url and dj-email-url recently partially changed their behavior. We should update our tests to reflect the changes.

In detail, the following changes have been made:

• django-cache-url now uses djangos native redis backend for django>4
• dj-email-url added support for the EMAIL_TIMEOUT setting which defaults to None

This avoids breaking the normal MRO for attributes, which searches the class hierarchy for attribute definitions. If attributes are copied onto every class in the hierarchy, the C3 MRO algorithm finds the attribute in a child class even though that class didn't actually define it, which results in surprising attribute values when classes in the hierarchy set the same attribute to different values.

Fixes #329