Microsoft Azure Storage Library for Python

Hi,

I have azure-storage-blob version 0.37.1 and still i get an error when i import BlockBlobService.

from azure.storage.blob import BlockBlobService # import azure sdk packages

Error message: ImportError: cannot import name 'BlockBlobService'

Here is a list of existing azure storage packages in my current virtual environment:

azure-storage-blob==0.37.1 azure-storage-common==0.37.1 azure-storage-nspkg==2.0.0

How can i fix this?

Thanks.

Hi guys,

I am seeing random authentication issues when using account SAS to upload large files ( >20 GB) to blob storage.

azure.common.AzureHttpError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:aa7f238b-0001-0006-3d71-cf3f37000000
Time:2017-05-18T00:54:58.0436290Z</Message><AuthenticationErrorDetail>sr is mandatory. Cannot be empty</AuthenticationErrorDetail></Error>

The authentication error detail showed sr is mandatory. Cannot be empty. Per Account SAS documentation, account SAS doesn't have the sr parameter while service SAS has it. Using the same token, I was able to successfully upload 2 large files at different time, but it also failed many times due to authentication issue. The SAS token is given to me, and I do not have direct access to the account, so I may not be able to change or generate a service SAS. Could you please take a look if this is a bug or simply I just miss something?

Thanks

Which service(blob, file, queue) does this issue concern?

queue (possibly others?)

What problem was encountered?

Installing azure-storage-queue (in a clean virtualenv) frequently (so not always) leads to a module installation which cannot be imported. The installation steps followed are the ones documented in https://docs.microsoft.com/en-us/azure/storage/queues/storage-python-how-to-use-queue-storage

In a fresh Python-3.6.3 virtualenv

$ pip install azure-storage-queue
... installation happens successfully, no errors ...
$  python -c 'from azure.storage.queue import QueueService'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'azure.storage'

Sometimes the installation does result into a module which can be imported without errors but most often, ... it does not.

Have you found a mitigation/solution?

A mitigation yes

$ pip install azure-storage-queue --upgrade --force-reinstall Fixes the problem. Keep in mind that the installation is done on a clean virtualenv.

I'd expect the import to succeed without any further ado after installing azure-storage-queue. If specific details are required, pls let me know.

Tnx,

Jelle

from azure.storage.blob import BlockBlobService

block_blob_service = BlockBlobService(account_name='myaccount', account_key='mykey')

block_blob_service.create_container('backups')

Part of my pip freeze ::

pip freeze appdirs==1.4.0 azure==1.0.3 azure-common==1.1.4 azure-mgmt==0.20.2 azure-mgmt-common==0.20.0 azure-mgmt-compute==0.20.1 azure-mgmt-network==0.20.1 azure-mgmt-nspkg==1.0.0 azure-mgmt-resource==0.20.1 azure-mgmt-storage==0.20.0 azure-nspkg==1.0.0 azure-servicebus==0.20.1 azure-servicemanagement-legacy==0.20.2 azure-storage==0.33.0

The 0.33.0 version is clearly there, why does it keep saying no module named storage.blob?

Which service(blob, file, queue) does this issue concern?

azure.storage.blob.baseblobservice.BaseBlobService.exists

What problem was encountered?

When checking if a blob exists in container with azure.storage.blob. baseblobservice.exists, an error is logged when the blob does not exist in container. The expected behavior would be that no error is logged when the blob does not exist.

Have you found a mitigation/solution?

Hint: azure.storage.blob.baseblobservice.BaseBlobService.get_blob_properties sends a request via _perform_request that raises an exception which is logged due to https://github.com/Azure/azure-storage-python/blob/master/azure-storage-common/azure/storage/common/storageclient.py#L348

blob_service.create_blob_from_path(container, blob_path, file, content_settings=content_settings, timeout=300)

$ time make upload-big-files
./src/7.upload-2-azure.sh    16041-big-files1   az--repoX
Init of upload_pack.py
Latest local pack found: 1.5.3.0203.161011-112303Z
Adding additional mime content types
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): repoX.blob.core.windows.net
DEBUG:requests.packages.urllib3.connectionpool:"GET /repoX-pub?restype=container HTTP/1.1" 200 None
DEBUG:requests.packages.urllib3.connectionpool:"HEAD /repoX-pub/repoX/games/big-files/files-v1/latest-build.cfg HTTP/1.1" 200 0
blob: repoX/games/big-files/files-v1/latest-build.cfg EXISTS
DEBUG:requests.packages.urllib3.connectionpool:"GET /repoX-pub/repoX/games/big-files/files-v1/latest-build.cfg HTTP/1.1" 206 73
DEBUG:requests.packages.urllib3.connectionpool:"HEAD /repoX-pub/repoX/games/big-files/files-v1/1.5.3.0203-161010-165722/.ok HTTP/1.1" 404 0
Local version (1.5.3.0203.161011-112303Z) is equal or newer than Azure version (1.5.3.0203-161010-165722)
uploading webgl.memgz
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (2): repoX.blob.core.windows.net
DEBUG:requests.packages.urllib3.connectionpool:"PUT /repoX-pub/repoX/games/big-files/files-v1/1.5.3.0203.161011-112303Z/bin.webgl/webgl.mem?timeout=300 HTTP/1.1" 201 None
uploading webgl.datagz
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (3): repoX.blob.core.windows.net
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (4): repoX.blob.core.windows.net
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (5): repoX.blob.core.windows.net
########################################################
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 349, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 936, in _send_output
    self.send(message_body)
  File "/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 908, in send
    self.sock.sendall(data)
  File "/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 891, in sendall
    v = self.send(data[count:])
  File "/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 861, in send
    return self._sslobj.write(data)
  File "/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 586, in write
    return self._sslobj.write(data)
socket.timeout: The write operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 597, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/util/retry.py", line 245, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/packages/six.py", line 309, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 349, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 936, in _send_output
    self.send(message_body)
  File "/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 908, in send
    self.sock.sendall(data)
  File "/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 891, in sendall
    v = self.send(data[count:])
  File "/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 861, in send
    return self._sslobj.write(data)
  File "/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 586, in write
    return self._sslobj.write(data)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/azure/storage/storageclient.py", line 212, in _perform_request
    response = self._httpclient.perform_request(request)
  File "/usr/local/lib/python3.5/site-packages/azure/storage/_http/httpclient.py", line 114, in perform_request
    proxies=self.proxies)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', timeout('The write operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/myroot/src/azure_module.py", line 92, in upload_azure_blob
    blob_service.create_blob_from_path(container, blob_path, file, content_settings=content_settings, timeout=300)
  File "/usr/local/lib/python3.5/site-packages/azure/storage/blob/blockblobservice.py", line 393, in create_blob_from_path
    timeout=timeout)
  File "/usr/local/lib/python3.5/site-packages/azure/storage/blob/blockblobservice.py", line 490, in create_blob_from_stream
    timeout=timeout)
  File "/usr/local/lib/python3.5/site-packages/azure/storage/blob/blockblobservice.py", line 814, in _put_blob
    return self._perform_request(request, _parse_base_properties)
  File "/usr/local/lib/python3.5/site-packages/azure/storage/storageclient.py", line 266, in _perform_request
    raise ex
  File "/usr/local/lib/python3.5/site-packages/azure/storage/storageclient.py", line 242, in _perform_request
    raise AzureException(ex.args[0])
azure.common.AzureException: ('Connection aborted.', timeout('The write operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/myroot/src/upload_pack.py", line 158, in <module>
    handle_deploy_target(deploy_yaml, uploader_yaml)
  File "/myroot/src/upload_pack.py", line 130, in handle_deploy_target
    upload_pack_files_to_azure(pack_upload_patterns, container, game_azure_base_dir, blob_service)
  File "/myroot/src/upload_pack.py", line 63, in upload_pack_files_to_azure
    azure_module.upload_azure_blob(file, container, dest_path, blob_service, content_encoding, cache_control)
  File "/myroot/src/azure_module.py", line 95, in upload_azure_blob
    logger.exception(error)
NameError: name 'error' is not defined
make: *** [upload-big-files] Error 1

real  2m10.694s
user  0m1.964s
sys 0m1.381s

Hello, issue is easily reproduceable: Create a flask web app add azure-storage to requirements.txt and try to push it to the Web App... It will fail with 2.7 and 3.4. See attached log file.

Running setup.py install for cryptography
BLA-BLA-BLA
remote: distutils.errors.DistutilsError: Setup script exited with error: Unable to find vcvarsall.bat
remote:
remote: ----------------------------------------
remote: Cleaning up...
remote: .........................
remote: Command D:\home\site\wwwroot\env\Scripts\python.exe -c "import setuptools, tokenize;__file__='D:\\home\\site\\wwwroot\\env\\build\\cryptography\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record D:\local\Temp\pip-1tk19zep-record\install-record.txt --single-version-externally-managed --compile --install-headers D:\home\site\wwwroot\env\include\site\python3.4 failed with error code 1 in D:\home\site\wwwroot\env\build\cryptography
remote: Storing debug log for failure in D:\home\pip\pip.log
remote: An error has occurred during web site deployment.

You owe me a couple of days of my life :( So it works for any version without that stupid broken cryptography.

fail.txt

This can currently also be achieved via [b.name for b in bbs.list_blobs()] but that parses the full XML and discards most of the parsed information again. With this change listing the blob names is not anymore CPU-bound for us.

How can I list files under a specific regex expression? If not possible how can I at least get the names of just one level of name (e.g. if path is CONTAINER/top1/bottom, CONTAINER/top2/bottom I would like to get only top1 and top2 rather than listing all the blobs under the container). I know I can give a prefix to list_blobs but than won't do for what I need above, wouldn't it?!

Currently if you want to know the public access level of a container, you have to use the get_container_acl call. One would reasonably expect that get_container_acl would return the ACL only, not the ACL and maybe some other properties too. Public access seems like it would logically be returned by the get_container_properties call since it is a property of the container itself, whereas the ACL is a list of stored settings for use in generating SAS tokens.

Another scenario in which this conflation of the two concepts is painful is when you just want to update the public access policy for a container. This can easily be done in the portal, but if you try:

set_container_acl(container, access_policy='blob')

This will result in wiping out your container's ACL, which almost certainly wasn't your intent. Instead, you have to do a call to get_container_acl (even though you don't want to change anything) and then pass the signed identifiers back in to the call to set_container_acl to perserve them.

There isn't a call for set_container_properties but it might be good from a usability standpoint to consider creating one and splitting out that functionality.

Which service(blob, file, queue) does this issue concern?

azure-storage-blob v1.2.0rc1

What problem was encountered?

I have a web server, which I want to give it read access to my Azure blob storage. Additionally, I want the web server to have independent permissions from mine (i.e., my user account). I think using the new features added to Azure (e.g., Added support for OAuth authentication for HTTPS requests) I can leverage OAuth2.0 access tokens and Azure AD roles for this purpose. Therefore, I followed this documentation, and created a service principal and assigned Azure Storage permission to it. Then in the Azure portal, under Storage -> Access Control (IAM), I defined Storage Blob Data Reader (Preview) role and assigned it to my service principal.

So, I implemented a logic similar to this to acquire access token, then I use the token to read the blob. However, I get an error saying that I am not allowed to access the blob. Then I assign myself the Storage Blob Data Reader (Preview) role, and then I'm able to read the blob. So, the service principal's role is not effective, i.e., I cannot assign myself a Contributor (read/write) and give the service principal the Reader role.

Am I missing some important setting here?

Have you found a mitigation/solution?

No.

Update 1:

Following is a summary of the different combinations I tried and their resulted outcome:

| [email protected] Role | Service Principal Role** | Read/Write Result | Expected Outcome | | :--- | :--- | :--- | :--- | | Contributor* | Contributor* | Successful (i.e., can read and write) | Successful | | Contributor* | None | Successful (i.e., can read and write) | Fail | | None | Contributor* | AuthorizationFailure This request is not authorized to perform this operation. RequestId:509428ef-901e-00f3-4b93-051b26000000 | Successful |

*Contributor = STORAGE BLOB DATA CONTRIBUTOR (PREVIEW) **Service Principal ID: 58ad99f1-19e9-4b08-8121-c372e1f14653

Update 2:

In the request for the an OIDC/OAuth2.0 access token, I set the client_id attribute is to the service principal ID.

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Please accept this contribution adding the standard Microsoft SECURITY.MD :lock: file to help the community understand the security policy and how to safely report security issues. GitHub uses the presence of this file to light-up security reminders and a link to the file. This pull request commits the latest official SECURITY.MD file from https://github.com/microsoft/repo-templates/blob/main/shared/SECURITY.md.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

https://github.com/Azure/azure-storage-python/blob/4306898850dd21617644fc537a57d025e833db74/azure-storage-common/azure/storage/common/_serialization.py#L46

Milliseconds are being stripped from datetime values when serializing:

return value.strftime('%Y-%m-%dT%H:%M:%SZ') --> return value.strftime('%Y-%m-%dT%H:%M:%S.%fZ')

Hi not sure if its even possible, but can I stream for example video file via flask ?

If yes is it there some example because I cant find anything on internet. (I cant create SAS redirection in browser because three js has problem with it via CORS,... )

something like this ?? :

@app.get("/myvideo")
def video():
      return myBlob.stream()

Hi, before 2 days there was released cryptography==38.x.x

which makes azure python web app crash (because cryptography fails to load), it could nice to freeze cryptography>=2.x.x as it is now (for storage and tables), because it will cause a lot of problems :)

Downgrading cryptography to cryptography==37.0.4 can solve this problem.

New version of library is probably written for new 22 ubuntu and azure runs on 20 version.

2022_09_08_ln0sdlwk00013K_default_docker.log

Just adding here log keys for people also searching this problem:

ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /home/site/wwwroot/antenv/lib/python3.7/site-packages/cryptography/hazmat/bindings/_rust.abi3.so)

Which service(blob, file, queue) does this issue concern?

Note: for package version >= 12.0.0 please post the issue here instead: https://github.com/Azure/azure-sdk-for-python/issues
      for table service, please post the issue here instead: https://github.com/Azure/azure-cosmosdb-python.

azure.mgmt.recoveryservicesbackup

Which version of the SDK was used? Please provide the output of pip freeze.

(venv) [email protected]:~$ pip freeze
azure-common==1.1.28
azure-core==1.24.1
azure-mgmt-core==1.3.1
azure-mgmt-recoveryservicesbackup==4.0.0
certifi==2022.6.15
charset-normalizer==2.1.0
idna==3.3
isodate==0.6.1
msrest==0.7.1
oauthlib==3.2.0
requests==2.28.1
requests-oauthlib==1.3.1
six==1.16.0
typing_extensions==4.2.0
urllib3==1.26.9

What problem was encountered?

(venv) [email protected]:~$ python
Python 3.7.3 (default, Jan 22 2021, 20:04:44) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from azure.mgmt.recoveryservicesbackup import RecoveryServicesBackupClient
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'RecoveryServicesBackupClient' from 'azure.mgmt.recoveryservicesbackup' (/home/zachgleason/venv/lib/python3.7/site-packages/azure/mgmt/recoveryservicesbackup/__init__.py)

Have you found a mitigation/solution?

No