Collectfast
A faster collectstatic command.
Features
- Efficiently decide what files to upload using cached checksums
- Parallel file uploads
Supported Storage Backends
storages.backends.s3boto3.S3Boto3Storagestorages.backends.gcloud.GoogleCloudStoragedjango.core.files.storage.FileSystemStorage
Running Django's collectstatic command can become painfully slow as more and more files are added to a project, especially when heavy libraries such as jQuery UI are included in source code. Collectfast customizes the builtin collectstatic command, adding different optimizations to make uploading large amounts of files much faster.
Installation
Install the app using pip:
$ python3 -m pip install Collectfast
Make sure you have this in your settings file and add 'collectfast' to your INSTALLED_APPS, before 'django.contrib.staticfiles':
STATICFILES_STORAGE = "storages.backends.s3boto3.S3Boto3Storage"
COLLECTFAST_STRATEGY = "collectfast.strategies.boto3.Boto3Strategy"
INSTALLED_APPS = (
# ...
'collectfast',
)
Note: 'collectfast' must come before 'django.contrib.staticfiles' in INSTALLED_APPS.
Note: The boto strategy will set preload_metadata on the remote storage to True, see #30.
Upload Strategies
| Collectfast Strategy | Storage Backend |
|---|---|
| collectfast.strategies.boto3.Boto3Strategy | storages.backends.s3boto3.S3Boto3Storage |
| collectfast.strategies.gcloud.GoogleCloudStrategy | storages.backends.gcloud.GoogleCloudStorage |
| collectfast.strategies.filesystem.FileSystemStrategy | django.core.files.storage.FileSystemStorage |
Custom strategies can also be made for backends not listed above by implementing the collectfast.strategies.Strategy ABC.
Usage
Collectfast overrides Django's builtin collectstatic command so just run python manage.py collectstatic as normal.
You can disable Collectfast by using the --disable-collectfast option or by setting COLLECTFAST_ENABLED = False in your settings file.
Setting Up a Dedicated Cache Backend
It's recommended to setup a dedicated cache backend for Collectfast. Every time Collectfast does not find a lookup for a file in the cache it will trigger a lookup to the storage backend, so it's recommended to have a fairly high TIMEOUT setting.
Configure your dedicated cache with the COLLECTFAST_CACHE setting:
CACHES = {
'default': {
# Your default cache
},
'collectfast': {
# Your dedicated Collectfast cache
},
}
COLLECTFAST_CACHE = 'collectfast'
If COLLECTFAST_CACHE isn't set, the default cache will be used.
Note: Collectfast will never clean the cache of obsolete files. To clean out the entire cache, use cache.clear(). See docs for Django's cache framework.
Note: We recommend you to set the MAX_ENTRIES setting if you have more than 300 static files, see #47.
Enable Parallel Uploads
The parallelization feature enables parallel file uploads using Python's concurrent.futures module. Enable it by setting the COLLECTFAST_THREADS setting.
To enable parallel uploads, a dedicated cache backend must be setup and it must use a backend that is thread-safe, i.e. something other than Django's default LocMemCache.
COLLECTFAST_THREADS = 20
Debugging
By default, Collectfast will suppress any exceptions that happens when copying and let Django's collectstatic handle it. To debug those suppressed errors you can set COLLECTFAST_DEBUG = True in your Django settings file.
Contribution
Please feel free to contribute by using issues and pull requests. Discussion is open and welcome.
Testing
The test suite is built to run against live S3 and GCloud buckets. You can disable live tests by setting SKIP_LIVE_TESTS=true or running make test-skip-live. To run live tests locally you need to provide API credentials to test against. Add the credentials to a file named storage-credentials in the root of the project directory:
export AWS_ACCESS_KEY_ID='...'
export AWS_SECRET_ACCESS_KEY='...'
export GCLOUD_CREDENTIALS='{...}' # Google Cloud credentials as JSON
Install test dependencies and target Django version:
python3 -m pip install -r test-requirements.txt
python3 -m pip install django==2.2
Run test suite:
make test
Code quality tools are broken out from test requirements because some of them only install on Python >= 3.7.
python3 -m pip install -r lint-requirements.txt
Run linters and static type check:
make lint
License
Collectfast is licensed under the MIT License, see LICENSE file for more information.
2.4k Jan 06, 2023
1.5k Jan 01, 2023
62 Apr 15, 2022
3 Feb 24, 2022
2.6k Jan 03, 2023
42 Jan 11, 2022
2.3k Jan 05, 2023
1.4k Dec 26, 2022
72 Aug 18, 2022
10 Sep 11, 2022
536 Dec 25, 2022
912 Dec 25, 2022