This change adds S3PrefixedStorage class which extends S3Storage with "prefix" required argument. I thought it would be nice to be ale to store all files within one folder in S3. So, I implemented it after reviewing this discussion: https://github.com/amol-/depot/issues/13#issuecomment-142432557 As suggested, prefix here is set in storage, not in the file field.

Can you please review and provide feedback? One thing I am not sure about in this implementation is that prefix ends up in UploadedFile['file_id']. Perhaps it would be more clear to not leak prefix outside of the storage class.

How do I configure s3 settings to use with depot? I'm looking for an example. What should the Depot manager syntax look like?

So far, I've been trying to look at your code and documentation to figure this out but nothing comes up.

from depot.manager import DepotManager
from depot.io.awss3 import S3Storage

S3_LOCATION = 'test.s3-website-us-west-2.amazonaws.com'
S3_KEY = 'XXXX'
S3_SECRET = 'XXXX'
S3_UPLOAD_DIRECTORY = 'media'
S3_BUCKET = 'test'
#
s3 = S3Storage(S3_KEY,S3_SECRET,bucket=S3_BUCKET)

DepotManager.configure('media',{
    'depot.backend': depot.io.awss3.S3Storage????
})

# DepotManager.configure('media', {'depot.storage_path': 'media/'})

app.wsgi_app = DepotManager.make_middleware(app.wsgi_app)

Thanks!

I have observed this behavior:

1. Create an SQLAlchemy model with a Depot file field, stored using the LocalFileStorage
2. Add the model to a session
3. Rollback the session

Here I would not expect the file from step 1 to have been stored to disk, but it seems to me that it is. Is this intentional?

Performing session.flush() before the rollback will however result in the file being removed during the rollback.

@amol-

I'm doing multiple image uploads per POST request via wtforms. I'm trying to enable async background upload. Is there a way to integrate celery async uploads with DEPOT? If so could you please provide us with a solution for this? I'm currently using Celery with RabbitMQ.

Suggestions would be good too.

Thanks!

Currently the package has definition of integration tests by means of .travis.yml. This allows for automated testing after changes are committed, however, a developer cannot run such tests locally without travis service.

Proposed solution (keep it simple)

• create tox.ini with definition of environments for all target python versions. Such environments would be run by default (would be listed in top section).
• each environment would run all the tests, which are easy to handle. Candidates for tests to skip are all tests requiring access to Internet or other installed services (like Mongo).
• .travis.yml would be kept as it is now as it runs more extensive set of tests.

Possible extensions of proposed solution

• drive even extended tests (requiring access to Internet or using external services like Mongo) by tox.ini. Possibly allow skiping such tests, e.g. by means of some env. variables or similar methods.
• modify .travis.yml to use what extended tox.ini is providing (I have seen couple of such examples). This way automated and local tests could be the same (or very similar).

If such idea sounds acceptable, I could provide PR with initial Proposed solution, which would conclude this issue.

Extended solution may follow, but it could include more work and extend the basic functionality too far blocking possible simplification of locally run tests for developer.

Hi,

First, thanks for this awesome library, it fits almost everything I need except one use case.

I have a mixed tree of static and dynamic files:

http://ex.com/x/y/z1 can be a dynamic object (my logic here)
http://ex.com/x/y/z2 can be a static picture (static serving using filedepot lookup)

Therefore, I need to lookup the static files by their request path (eg. /x/y/z2 --their actual URI ID). When using create(), file_id is generated by an arbitrary uuid, there I can't use get() to look up by their URI ID. Would it be possible to allow overriding the way file_id is generated so i could compute it using the URI ?

Thanks,

Guillaume

Fixes #64 by using URL encoding when unidecode isn't installed. It also removes the dependency on unidocde so that it is only installed when users want it/can use it.

For our development environment, I setup minio to simulate S3 storage. But communicating with it using existing depot and boto code was impossible (had to be SSL, needed patch to use custom port, custom s3 authentication mechanism, install CA to verify certs,...)

So I gave up and wrote a layer to work with boto3, I figured I'll share it with you guys. There is one thing missing - encrypt_key support. I don't understand what that's supposed to do in the original code, therefore I have no clue how to migrate that to boto3.

Find the boto3 wrapper attached to this issue:

depot_minio.py.zip

Here is how I call it

DepotManager.configure('s3devel', {'depot.backend': 'depot_minio.MinioStorage',
            'depot.access_key_id': '...',
            'depot.secret_access_key': '....',
            'depot.bucket': 'my-bucket',
            'depot.endpoint': 'http://localhost:9000',
            'depot.prefix': 'my-prefix',
        })

I am willing to help integrate this into your project, if you are interested to support boto3 and minio. Just let me know how I can help br mike

I miss a way of listing all objects stored in a depot, which would be really useful. Was this left out of the API intentionally, or could this be added?

How do I use depot to specify which bucket to update/save an image to?

Currently I'm using sqlalchemy to as the orm and following your details my folder looks like

This is what my folder looks like currently

/media
   /8404b4c7-2106-11e5-827f-685b358e848d
     /metadata.json
     /file
``

I want to separate out say for example, profile avatar and user images


Also, is there a way to get the size of the image that is stored within the folder/bucket?

I use the SqlAlchemy ORM system. When I get an object and I delete it, the associated file is deleted to.

d = DBSession.query(Document).filter_by(name=u_('Foo2')).first()
DBSession.delete(d)

But when I create query to delete multiple row, row are deleted in the db but the associated files are not deleted

DBSession.query(Document).filter_by(name=u_('Foo2')).delete()
DBSession.commit()

Thanks for the development & maintenance of the filedepot package!

I recently started seeing this deprecation warning when using the WithThumbnailFilter class with pillow v9.2.0 (I'm running on Python 3.10.7):

lib/python3.10/site-packages/depot/fields/filters/thumbnails.py:37: DeprecationWarning: 
BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
    thumbnail.thumbnail(self.thumbnail_size, Image.BILINEAR)

It looks like Image.BILINEAR was deprecated in pillow 9.1.0 in favour of Image.Resampling.BILINEAR.

Hello!

I need ssl verification and option to set path to my selfsigned certificate when work with S3.

There's no support of parametres verify and use_ssl для boto3.resource (boto3 docs).

Can you add this support to your depot.io.boto3.S3Storage?

Something like this

class S3Storage:
     ...

     def __init__(self, access_key_id, secret_access_key, bucket=None, region_name=None,
                 policy=None, storage_class=None, endpoint_url=None, prefix='',
                 use_ssl=None, verify=None):
         ...
         kw = {}
         ...
         if use_ssl is not None:
             kw['use_ssl'] = use_ssl
         if verify is not None:
             kw['verify'] = verify
         self._s3 = self._conn.resource('s3', **kw)
         ...

Or even this (support of all S3 configuration parametres)

class S3Storage:
     ...

     def __init__(
        self,
        access_key_id,
        secret_access_key,
        bucket=None,
        policy=None,
        storage_class=None,
        prefix='',
        s3_params=None,
    ):
         ...
         s3_params = s3_params or {}
         self._s3 = self._conn.resource('s3', **s3_params)
         ...

so I can configure depot like this

depot.manager.DepotManager.configure(
    'default',
    {
        'depot.backend': 'depot.io.boto3.S3Storage',
        'depot.access_key_id': settings.depot_access_key_id,
        'depot.secret_access_key': settings.depot_secret_access_key,
        'depot.bucket': 'bububucket',
        'depot.policy': settings.depot_policy,

        'depot.s3_params': {
            'endpoint_url': settings.depot_endpoint_url,
            'use_ssl': True,
            'verify': 'certs.pem',
            'config': botocore.config.Config(
                retries={
                    "max_attempts": MAX_RETRIES,
                    "mode": "standard",
                },
                s3={
                    "addressing_style": 'virtual',
                },
            ),
        },
        'depot.prefix': 'default_prefix'
    },
)

Not really an issue, more of a question 😅

Is there a way to reprocess all filters without re-uploading the main file? I guess the file will have to be downloaded anyway to be processed, but I'd like to avoid the extra upload 👀

I poked around a bit and the only thing that came to mind was to download/replace/upload every file.

Use with care, especially with MongoDB. This is probably going to be a controversial PR, but I wanted to put it out there. It should be 100% backwards compatible.

If a depot is being stored locally, obviously the backup and restore can just use local archiving tool. But if they're remote and someone wants to either export the depot to another installation or do some other sort of extraction, we need to be able to replicate depots exactly, including ID. One easy way to do this is to allow fileids to optionally be set on create.

This is needed because if the database is storing file ids, then it would break everything to have the new store use a new ID.