• safety version: 1.7.0
• Python version: Python 3.6.1
• Operating System: Windows-10-10.0.16299-SP0, AMD64

Description

• Trying to use safety check
• Same error always results: UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-79: character maps to <undefined>

What I Did

safety check -r simple-requirements.txt

Contents of simple-requirements.txt

safety

• There are absolutely no unicode characters in this file

Traceback

$ safety check -r simple-requirements.txt
Warning: unpinned requirement 'safety' found in simple-requirements.txt, unable to check.
Traceback (most recent call last):
  File "c:\users\nicholas\appdata\local\programs\python\python36\Lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\nicholas\appdata\local\programs\python\python36\Lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\nicholas\.virtualenvs\pybotics-d30fj9Hx\Scripts\safety.exe\__main__.py", line 9, in <module>
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\site-packages\click\core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\site-packages\click\core.py", line 697, in main
    rv = self.invoke(ctx)
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\site-packages\click\core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\site-packages\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\site-packages\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\site-packages\safety\cli.py", line 66, in check
    key=key
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\site-packages\click\termui.py", line 420, in secho
    return echo(style(text, **styles), file=file, nl=nl, err=err, color=color)
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\site-packages\click\utils.py", line 259, in echo
    file.write(message)
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-79: character maps to <undefined>

Similar Issues

• https://github.com/pyupio/safety/issues/22

Hi,

This is a feature request.

Lets start by stating that, following the spec, URLs to tarballs are part of the requirements file format https://pip.readthedocs.io/en/1.1/requirements.html#requirements-file-format

Now, the pkg_resources.parse_requirements function used by safety does not support them: https://github.com/pypa/setuptools/blob/master/pkg_resources/init.py#L2850 It raises a RequirementParseError: Invalid requirement, parse error.

I had a look at how they handle this in pip, and it's ugly: https://github.com/pypa/pip/blob/master/pip/req/req_set.py#L690

pip-tools does not support them. It actually crashes in a bad way if you try so: https://github.com/nvie/pip-tools/issues/416

By the way, URLs to tarball specified as editable requirements (with -e) work fine: curiously pkg_resources.parse_requirements handle them perfectly well.

What do you think ? Should safety handle them ?

• safety version: GitHub integration
• Python version: 2.7.x
• Operating System:

Description

We have an status integrated for varryfying each PR going into the master branch.

I now encounter that this status is Pending for over 12 hours.

Is there currently a server issue or am I doing something wrong with the integration?

• safety version: 1.10.3
• Python version: 3.8.12
• Operating System: Ubuntu 20.04.3 LTS

Description

Ran safety against the latest update and got a report of a failure on numpy, despite being on 1.22.1.

In the free safety DB, the values for numpy are expressed as:

"numpy": [
        "<1.13.2",
        "<1.16.3",
        "<1.21.0",
        "<1.22.0",
        "<1.8.1",
        ">0"
    ],

I'm not sure why >0 was added in the February release, but it seems to be causing this problem

What I Did

2022-02-01T15:10:46.7671452Z +==============================================================================+
2022-02-01T15:10:46.7677519Z |                                                                              |
2022-02-01T15:10:46.7726508Z |                               /$$$$$$            /$$                         |
2022-02-01T15:10:46.7726760Z |                              /$$__  $$          | $$                         |
2022-02-01T15:10:46.7727062Z |           /$$$$$$$  /$$$$$$ | $$  \__//$$$$$$  /$$$$$$   /$$   /$$           |
2022-02-01T15:10:46.7727279Z |          /$$_____/ |____  $$| $$$$   /$$__  $$|_  $$_/  | $$  | $$           |
2022-02-01T15:10:46.7727504Z |         |  $$$$$$   /$$$$$$$| $$_/  | $$$$$$$$  | $$    | $$  | $$           |
2022-02-01T15:10:46.7727745Z |          \____  $$ /$$__  $$| $$    | $$_____/  | $$ /$$| $$  | $$           |
2022-02-01T15:10:46.7727979Z |          /$$$$$$$/|  $$$$$$$| $$    |  $$$$$$$  |  $$$$/|  $$$$$$$           |
2022-02-01T15:10:46.7728200Z |         |_______/  \_______/|__/     \_______/   \___/   \____  $$           |
2022-02-01T15:10:46.7728430Z |                                                          /$$  | $$           |
2022-02-01T15:10:46.7728651Z |                                                         |  $$$$$$/           |
2022-02-01T15:10:46.7728876Z |  by pyup.io                                              \______/            |
2022-02-01T15:10:46.7729111Z |                                                                              |
2022-02-01T15:10:46.7729338Z +==============================================================================+
2022-02-01T15:10:46.7729569Z | REPORT                                                                       |
2022-02-01T15:10:46.7729828Z | checked 147 packages, using free DB (updated once a month)                   |
2022-02-01T15:10:46.7730086Z +============================+===========+==========================+==========+
2022-02-01T15:10:46.7730332Z | package                    | installed | affected                 | ID       |
2022-02-01T15:10:46.7730578Z +============================+===========+==========================+==========+
2022-02-01T15:10:46.7730786Z | numpy                      | 1.22.1    | >0                       | 44715    |
2022-02-01T15:10:46.7731010Z +==============================================================================+

• safety version: 1.8.5
• Python version: 3.6.7
• Operating System: Ubuntu 18.04.1 LTS

Description

Run the following command from a terminal: echo "Jinja==1.0.0" | safety check --stdin --full-report --json The result that will be returned looks like this:

[
    [
        "jinja",
        "<2.7.2",
        "1.0.0",
        "jinja 2.7.2 fixes a security issue: Changed the default folder for the filesystem cache to be user specific and read and write protected on UNIX systems.  See http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=734747 for more information.",
        "25863"
    ],
    [
        "jinja",
        "<2.7.3",
        "1.0.0",
        "jinja 2.7.3 fixes a security issue: Corrected the security fix for the cache folder.",
        "25864"
    ]
]

As you can see it returns a list of all CVE's that where found along with information about the CVE. There is a big downside about the current structure though (or atleast I think so). As this is a list with values inside it means I have to do extra checks or dangerous operations to get values out. Now imagine that the script is run from Python within a os.popen or equivalent way like this:

command = (
    "echo \"Jinja==1.0.0\"" | safety check --stdin --full-report --json"
)
cve_result_details = json.loads(os.popen(command).read())

I'll have a JSON dict just like it was sent. Now how can I safely get out the upper version of the CVE? I'd have to do something like:

if cve_result_details:
    upper_version = cve_result_details[0][1]

This feels pretty dangerous & risky. I'd propose another JSON structure that looks like this:

{
    "cve_reports": [
      {
         "package_name": "jinja",
        "upper_version": "<2.7.2",
        "installed_version": "1.0.0",
        "package_description": "jinja 2.7.2 fixes a security issue: Changed the default folder for the filesystem cache to be user specific and read and write protected on UNIX systems.  See http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=734747 for more information.",
        "25863"
        }
    ]
}

This would allow to do cleaner & safer operations. To get out the upper version I could now do:

upper_version = cve_result.get('cve_reports').get('upper_version')

The benefits:

• Will not crash if no result
• Cleaner to write and easier to understand

The code was correctly trapping FileNotFoundError for the case where stty is not available on Windows. However, on Python 2.7, the exception raised by subprocess.check_output() is WindowsError, a subclass of OSError.

Fixes #65.

Hi,

This is just to let you know that I developped a pre-commit hook based on your lib: https://github.com/Lucas-C/pre-commit-hooks-safety

It is not yet fully functional : a small limitation in pre-commit is a blocker, but I intend to fix it in this issue: https://github.com/pre-commit/pre-commit/issues/426

• Use 'io' package as a more reliable way of opening files on Windows in Python 2.7
• Add python_requires, which helps pip determine if the package is compatible with the version of Python.
• Change development status to "Production/Stable" in PyPI classifiers
• Cleanup and formatting of setup.py

Description

Dockerise the safety command/tool to make it easier to be used in local development and within CI/CD pipelines.

What I Did

Here's a copy of my Dockerfile that I used to do this locally, it may be valuable to others (or if someone wants to PR it back into master):

FROM ubuntu:18.04

ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8

RUN apt-get -qy update && \
    apt-get -qy install python3-pip python-dev build-essential && \
    pip3 install safety && \
    apt-get -qy clean && \
    rm -rf /var/lib/apt/lists/* && \
    rm -rf /tmp/*

ENTRYPOINT ["/usr/local/bin/safety"]

Then you can simply run a command as per the README.md, like so:

echo "insecure-package==0.1" | docker run --rm docker-pyup-safety check --stdin

Quoting distutils-sig:

We're in the process of starting to plan for a release of pip (the long-awaited pip 10). We're likely still a month or two away from a release, but now is the time for people to start ensuring that everything works for them. One key change in the new version will be that all of the internal APIs of pip will no longer be available, so any code that currently calls functions in the "pip" namespace will break. Calling pip's internal APIs has never been supported, and always carried a risk of such breakage, so projects doing so should, in theory, be prepared for such things. However, reality is not always that simple, and we are aware that people will need time to deal with the implications.

Just in case it's not clear, simply finding where the internal APIs have moved to and calling them under the new names is not what people should do. We can't stop people calling the internal APIs, obviously, but the idea of this change is to give people the incentive to find a supported approach, not just to annoy people who are doing things we don't want them to ;-)

So please - if you're calling pip's internals in your code, take the opportunity now to check out the in-development version of pip, and ensure your project will still work when pip 10 is released.

And many thanks to anyone else who helps by testing out the new version, as well :-)

Thanks, Paul

Safety uses pip.get_installed_distributions which has moved to https://github.com/pypa/pip/blob/master/src/pip/_internal/utils/misc.py#L333

• safety version: 1.10.3
• Python version: 3.10.2
• Operating System: MacOS 11.7

Description

Hello 👋

safety currently requires cryptography>=39.0.0 for its checks to pass (see screenshot), citing the below reason, with ID 51159:

Cryptography 39.0.0 drops support for C library "LibreSSL" < 3.4, as these versions are not receiving security support anymore.

However, it looks like version 39.0.0 of cryptography is under development and hasn't been released yet.

As far as I can see, the only way around this for now is to ask safety to ignore its check on cryptography.

There's a comment on commit https://github.com/pyupio/safety-db/commit/e582a03665fde14b58dd247b5d33aa4c0f0832e5 about it here.

Thanks 🙏

What I Did

Ran the following:

safety check --full-report --file=requirements.txt

• safety version: 2.3.5
• Python version: All
• Operating System: All

Description

We are attempting to create virtual environments for our developers that include all production dependencies for their package plus all of our static analyzers and security checks, so they can reproduce our CI system locally. It has recently come up that the latest version of safety pins to packaging<22.0 which ~is incompatible~ requires some extra work to operate with some of our existing package dependencies.

Can you shed some more light on the comment in this commit: https://github.com/pyupio/safety/commit/aa1b1532818a1b2cb229b14907cad64c927fa8c6? What exactly is meant by "preventing issues?" Is this something that you could use help on or is on your roadmap to support?

What I Did

First look at my input file (reduced to show minimal reproducible error):

> cat test.in
pandera

Next, compile that file to a lockfile:

> pip-compile --no-emit-index-url --no-header --quiet --pip-args --no-input --resolver=legacy --output-file=test.txt test.in        
> cat test.txt                                                                                                                      
mypy-extensions==0.4.3
    # via typing-inspect
numpy==1.23.5
    # via
    #   pandas
    #   pandera
packaging==22.0
    # via pandera
pandas==1.5.2
    # via pandera
pandera==0.13.4
    # via -r test.in
pydantic==1.10.2
    # via pandera
python-dateutil==2.8.2
    # via pandas
pytz==2022.6
    # via pandas
six==1.16.0
    # via python-dateutil
typing-extensions==4.4.0
    # via
    #   pydantic
    #   typing-inspect
typing-inspect==0.8.0
    # via pandera
wrapt==1.14.1
    # via pandera

Now to show the dev requirements:

> cat test-dev.in                                                                                                                   
-c test.txt

safety

And finally, attempt to lock that file:

> pip-compile --no-emit-index-url --no-header --quiet --pip-args --no-input --resolver=legacy --output-file=test-dev.txt test-dev.in
Could not find a version that matches packaging<22.0,==22.0,>=21.0 (from -c test.txt (line 7))
Tried: 14.0, 14.0, 14.1, 14.1, 14.2, 14.2, 14.3, 14.3, 14.4, 14.4, 14.5, 14.5, 15.0, 15.0, 15.1, 15.1, 15.2, 15.2, 15.3, 15.3, 16.0, 16.0, 16.1, 16.1, 16.2, 16.2, 16.3, 16.3, 16.4, 16.4, 16.5, 16.5, 16.6, 16.6, 16.7, 16.7, 16.8, 16.8, 17.0, 17.0, 17.1, 17.1, 18.0, 18.0, 19.0, 19.0, 19.1, 19.1, 19.2, 19.2, 20.0, 20.0, 20.1, 20.1, 20.2, 20.2, 20.3, 20.3, 20.4, 20.4, 20.5, 20.5, 20.6, 20.6, 20.7, 20.7, 20.8, 20.8, 20.9, 20.9, 21.0, 21.0, 21.1, 21.1, 21.2, 21.2, 21.3, 21.3, 22.0, 22.0
There are incompatible versions in the resolved dependencies:
  packaging==22.0 (from -c test.txt (line 7))

I realize there are some pretty simple workarounds in this case, and we are investigating tools like pip-compile-multi to solve this problem more robustly with our internal tooling, but I just wanted to flag in case others do not have the luxury of dodging a specific version of packaging. It could be someone is reliant on a new feature. I also acknowledge that this new version of packaging is hot off the press, and will take time to onboard. Let me know if I can help, I really like this tool!

% flake8 . --count --select=E9,F63,F7,F82,Y --show-source --statistics

./safety/util.py:557:20: F821 undefined name 'Context'
        self, ctx: "Context", param: "Parameter", incomplete: str
                   ^
./safety/util.py:557:38: F821 undefined name 'Parameter'
        self, ctx: "Context", param: "Parameter", incomplete: str
                                     ^
2     F821 undefined name 'Context'
2

https://click.palletsprojects.com/en/8.1.x/api/?highlight=parameter#click.Context https://click.palletsprojects.com/en/8.1.x/api/?highlight=parameter#click.Parameter

• safety version: 2.0.0
• Python version: 3.6
• Operating System: ubuntu 18

Description

Safety --disable-telemetry option does not appear to disable telemetry

What I Did

We can see below safety is attempting to reach out to https://pyup.io/aws/safety/free/insecure.json?telemetry=%7B%22safety_version%22%3A+%222.0.0%22%7D despite --disable-telemetry flag passed in.

(safety) [email protected]:~/workspace/Application_Testing/DataApi/PR-554$ safety --version
safety, version 2.0.0
(safety) [email protected]:~/workspace/Application_Testing/DataApi/PR-554$ safety --debug --disable-telemetry check -r requirements.txt
2022-07-01 19:21:59,689 safety.cli => Telemetry enabled: False
2022-07-01 19:21:59,690 safety.cli => Running check command
2022-07-01 19:21:59,715 safety.cli => Not local DB used, Getting announcements
2022-07-01 19:21:59,715 safety.safety => Getting announcements
2022-07-01 19:21:59,715 safety.util => Telemetry body built: {'safety_version': '2.0.0'}
2022-07-01 19:21:59,715 safety.safety => Telemetry body sent: {'safety_version': '2.0.0'}
2022-07-01 19:21:59,717 urllib3.connectionpool => Starting new HTTPS connection (1): pyup.io:443
2022-07-01 19:22:00,097 safety.safety => Unexpected but HANDLED Exception happened getting the announcements: HTTPSConnectionPool(host='pyup.io', port=443): Max retries exceeded with url: /api/v1/safety/announcements/ (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))
2022-07-01 19:22:00,098 safety.cli => Calling the check function
2022-07-01 19:22:00,098 safety.util => Telemetry body built: {'safety_version': '2.0.0'}
2022-07-01 19:22:00,099 urllib3.connectionpool => Starting new HTTPS connection (2): pyup.io:443
2022-07-01 19:22:00,405 safety.cli => Expected SafetyError happened: Check your network connection, unable to reach the server
Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/urllib3/connectionpool.py", line 710, in urlopen
    chunked=chunked,
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/urllib3/connection.py", line 424, in connect
    tls_in_tls=tls_in_tls,
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 450, in ssl_wrap_socket
    sock, context, tls_in_tls, server_hostname=server_hostname
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/usr/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/requests/adapters.py", line 450, in send
    timeout=timeout
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/urllib3/connectionpool.py", line 786, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pyup.io', port=443): Max retries exceeded with url: /aws/safety/free/insecure.json?telemetry=%7B%22safety_version%22%3A+%222.0.0%22%7D (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/safety/safety.py", line 117, in fetch_database_url
    r = session.get(url=url, timeout=REQUEST_TIMEOUT, headers=headers, proxies=proxy, params=telemetry_data)
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/requests/sessions.py", line 542, in get
    return self.request('GET', url, **kwargs)
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='pyup.io', port=443): Max retries exceeded with url: /aws/safety/free/insecure.json?telemetry=%7B%22safety_version%22%3A+%222.0.0%22%7D (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/safety/cli.py", line 114, in check
    params=params)
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/safety/util.py", line 639, in new_func
    return f(*args, **kwargs)
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/safety/safety.py", line 258, in check
    db = fetch_database(key=key, db=db_mirror, cached=cached, proxy=proxy, telemetry=telemetry)
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/safety/safety.py", line 164, in fetch_database
    data = fetch_database_url(mirror, db_name=db_name, key=key, cached=cached, proxy=proxy, telemetry=telemetry)
  File "/var/lib/jenkins/workspace/Application_Testing/DataApi/PR-554/.tox/safety/lib/python3.6/site-packages/safety/safety.py", line 119, in fetch_database_url
    raise NetworkConnectionError()
safety.errors.NetworkConnectionError: Check your network connection, unable to reach the server
Check your network connection, unable to reach the server

• safety version: latest
• Python version: 3.7
• Operating System: alpine docker

Description

I have a local pypi repository, can I use it offline when working with safety?

What I Did

offline runner: `Connection to pypi.org timed out. (connect timeout=15)')': /simple/safety/`

• safety version: 1.10.3
• Python version: Any
• Operating System: Mac and Linux

Description

I would like to get a native build of safety for the arm64/aarch64 architecture and the Mac and Linux OSs.

But the current releases are only for the amd64/x86_64 architectures.

hadolint/hadolint#411 has some possibly relevant discussions.