Simple library for exploring/scraping the web or testing a website you’re developing

Related tags

Web Crawlingrobox
Overview

codecov Language grade: Python Run tests view examples PyPI version

Overview

Robox is a simple library with a clean interface for exploring/scraping the web or testing a website you’re developing. Robox can fetch a page, click on links and buttons, and fill out and submit forms. Robox is built on top of two excelent libraries: httpx and beautifulsoup4.


Robox has all the standard features of httpx, including async, plus:

  • clean api
  • caching
  • downloading files
  • history
  • understands robots.txt

Examples

from robox import Robox


with Robox() as robox:
    page = robox.open("https://httpbin.org/forms/post")
    form = page.get_form()
    form.fill_in("custname", value="foo")
    form.check("topping", values=["Onion"])
    form.choose("size", option="Medium")
    form.fill_in("comments", value="all good in the hood")
    form.fill_in("delivery", value="13:37")
    page = page.submit_form(form)
    assert page.url == "https://httpbin.org/post"

or use async version:

import asyncio

from pprint import pprint
from robox import AsyncRobox


async def main():
    async with AsyncRobox(follow_redirects=True) as robox:
        page = await robox.open("https://www.google.com")
        form = page.get_form()
        form.fill_in("q", value="python")
        consent_page = await page.submit_form(form)
        form = consent_page.get_form()
        page = await consent_page.submit_form(form)
        pprint(list(page.get_links_by_text("python")))


asyncio.run(main())

Caching can be easily configured via httpx-cache

from robox import Robox, DictCache, FileCache


with Robox(cache=DictCache()) as robox:
    p1 = robox.open("https://httpbin.org/get")
    assert not p1.from_cache
    p2 = robox.open("https://httpbin.org/get")
    assert p2.from_cache

See examples folder for more detailed examples.

Installation

Using pip:

pip install robox

Robox requires Python 3.8+. See Changelog for changes.

Comments
  • Bump pytest-asyncio from 0.17.2 to 0.20.1

    Bump pytest-asyncio from 0.17.2 to 0.20.1

    Bumps pytest-asyncio from 0.17.2 to 0.20.1.

    Release notes

    Sourced from pytest-asyncio's releases.

    pytest-asyncio 0.20.1


    title: 'pytest-asyncio: pytest support for asyncio'

    image

    image

    image

    Supported Python versions

    image

    pytest-asyncio is an Apache2 licensed library, written in Python, for testing asyncio code with pytest.

    asyncio code is usually written in the form of coroutines, which makes it slightly more difficult to test using normal testing tools. pytest-asyncio provides useful fixtures and markers to make testing easier.

    @pytest.mark.asyncio
    async def test_some_asyncio_code():
        res = await library.do_something()
        assert b"expected result" == res
    

    pytest-asyncio has been strongly influenced by pytest-tornado.

    Features

    • fixtures for creating and injecting versions of the asyncio event loop
    • fixtures for injecting unused tcp/udp ports
    • pytest markers for treating tests as asyncio coroutines
    • easy testing with non-default event loops
    • support for [async def]{.title-ref} fixtures and async generator fixtures
    • support auto mode to handle all async fixtures and tests automatically by asyncio; provide strict mode if a test suite should work with different async frameworks simultaneously, e.g. asyncio and trio.

    Installation

    ... (truncated)

    Changelog

    Sourced from pytest-asyncio's changelog.

    0.20.1 (22-10-21)

    • Fixes an issue that warned about using an old version of pytest, even though the most recent version was installed. [#430](https://github.com/pytest-dev/pytest-asyncio/issues/430) <https://github.com/pytest-dev/pytest-asyncio/issues/430>_

    0.20.0 (22-10-21)

    • BREAKING: Removed legacy mode. If you're upgrading from v0.19 and you haven't configured asyncio_mode = legacy, you can upgrade without taking any additional action. If you're upgrading from an earlier version or you have explicitly enabled legacy mode, you need to switch to auto or strict mode before upgrading to this version.
    • Deprecate use of pytest v6.
    • Fixed an issue which prevented fixture setup from being cached. [#404](https://github.com/pytest-dev/pytest-asyncio/issues/404) <https://github.com/pytest-dev/pytest-asyncio/pull/404>_

    0.19.0 (22-07-13)

    • BREAKING: The default asyncio_mode is now strict. [#293](https://github.com/pytest-dev/pytest-asyncio/issues/293) <https://github.com/pytest-dev/pytest-asyncio/issues/293>_
    • Removes setup.py since all relevant configuration is present setup.cfg. Users requiring an editable installation of pytest-asyncio need to use pip v21.1 or newer. [#283](https://github.com/pytest-dev/pytest-asyncio/issues/283) <https://github.com/pytest-dev/pytest-asyncio/issues/283>_
    • Declare support for Python 3.11.

    0.18.3 (22-03-25)

    • Adds pytest-trio <https://pypi.org/project/pytest-trio/>_ to the test dependencies
    • Fixes a bug that caused pytest-asyncio to try to set up async pytest_trio fixtures in strict mode. [#298](https://github.com/pytest-dev/pytest-asyncio/issues/298) <https://github.com/pytest-dev/pytest-asyncio/issues/298>_

    0.18.2 (22-03-03)

    • Fix asyncio auto mode not marking static methods. [#295](https://github.com/pytest-dev/pytest-asyncio/issues/295) <https://github.com/pytest-dev/pytest-asyncio/issues/295>_
    • Fix a compatibility issue with Hypothesis 6.39.0. [#302](https://github.com/pytest-dev/pytest-asyncio/issues/302) <https://github.com/pytest-dev/pytest-asyncio/issues/302>_

    0.18.1 (22-02-10)

    • Fixes a regression that prevented async fixtures from working in synchronous tests. [#286](https://github.com/pytest-dev/pytest-asyncio/issues/286) <https://github.com/pytest-dev/pytest-asyncio/issues/286>_

    0.18.0 (22-02-07)

    • Raise a warning if @​pytest.mark.asyncio is applied to non-async function. [#275](https://github.com/pytest-dev/pytest-asyncio/issues/275) <https://github.com/pytest-dev/pytest-asyncio/issues/275>_
    • Support parametrized event_loop fixture. [#278](https://github.com/pytest-dev/pytest-asyncio/issues/278) <https://github.com/pytest-dev/pytest-asyncio/issues/278>_
    Commits
    • c8d0174 fix: Do not warn about outdated pytest version when pytest>=7 is installed. (...
    • 6450ddb Prepare release of v0.20.0. (#428)
    • 150f29c Build(deps): Bump hypothesis in /dependencies/default (#427)
    • adc8809 Build(deps): Bump typing-extensions in /dependencies/default (#425)
    • 4abf9d1 Build(deps): Bump zipp from 3.8.1 to 3.9.0 in /dependencies/default (#424)
    • eb487bc Build(deps): Bump hypothesis in /dependencies/default (#423)
    • 907c461 Refactor pytest_pycollect_makeitems (#421)
    • d45ab21 feat: Add deprecation warning for pytest < 7. (#420)
    • cab20f4 Build(deps): Bump importlib-metadata in /dependencies/default (#415)
    • d9f7756 Build(deps): Bump hypothesis in /dependencies/default (#416)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 2
  • Bump pytest from 7.0.1 to 7.1.3

    Bump pytest from 7.0.1 to 7.1.3

    Bumps pytest from 7.0.1 to 7.1.3.

    Release notes

    Sourced from pytest's releases.

    7.1.3

    pytest 7.1.3 (2022-08-31)

    Bug Fixes

    • #10060: When running with --pdb, TestCase.tearDown is no longer called for tests when the class has been skipped via unittest.skip or pytest.mark.skip.
    • #10190: Invalid XML characters in setup or teardown error messages are now properly escaped for JUnit XML reports.
    • #10230: Ignore .py files created by pyproject.toml-based editable builds introduced in pip 21.3.
    • #3396: Doctests now respect the --import-mode flag.
    • #9514: Type-annotate FixtureRequest.param as Any as a stop gap measure until 8073{.interpreted-text role="issue"} is fixed.
    • #9791: Fixed a path handling code in rewrite.py that seems to work fine, but was incorrect and fails in some systems.
    • #9917: Fixed string representation for pytest.approx{.interpreted-text role="func"} when used to compare tuples.

    Improved Documentation

    • #9937: Explicit note that tmpdir{.interpreted-text role="fixture"} fixture is discouraged in favour of tmp_path{.interpreted-text role="fixture"}.

    Trivial/Internal Changes

    7.1.2

    pytest 7.1.2 (2022-04-23)

    Bug Fixes

    • #9726: An unnecessary numpy import inside pytest.approx{.interpreted-text role="func"} was removed.
    • #9820: Fix comparison of dataclasses with InitVar.
    • #9869: Increase stacklevel for the NODE_CTOR_FSPATH_ARG deprecation to point to the user's code, not pytest.
    • #9871: Fix a bizarre (and fortunately rare) bug where the [temp_path]{.title-ref} fixture could raise an internal error while attempting to get the current user's username.

    7.1.1

    pytest 7.1.1 (2022-03-17)

    Bug Fixes

    • #9767: Fixed a regression in pytest 7.1.0 where some conftest.py files outside of the source tree (e.g. in the [site-packages]{.title-ref} directory) were not picked up.

    7.1.0

    pytest 7.1.0 (2022-03-13)

    ... (truncated)

    Commits
    • 4645bcd Remove incorrect output in how-to/fixtures.rst
    • fadfb4f Prepare release version 7.1.3
    • ab96ea8 Merge pull request #10258 from pytest-dev/backport-10252-to-7.1.x
    • fc0e024 [7.1.x] Fix regendoc
    • 8f5088f Merge pull request #10249 from pytest-dev/backport-10231-to-7.1.x
    • aae93d6 Ignore type-errors related to attr.asdict
    • 71b79fc [7.1.x] Ignore editable installation modules
    • 89f7518 Merge pull request #10222 from pytest-dev/backport-10171-to-7.1.x
    • 88fc45b [7.1.x] Update fixtures.rst w/ finalizer order
    • d0b53d6 Merge pull request #10221 from pytest-dev/backport-10217-to-7.1.x
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 2
  • Bump black from 22.1.0 to 22.8.0

    Bump black from 22.1.0 to 22.8.0

    Bumps black from 22.1.0 to 22.8.0.

    Release notes

    Sourced from black's releases.

    22.8.0

    Highlights

    • Python 3.11 is now supported, except for blackd as aiohttp does not support 3.11 as of publishing (#3234)
    • This is the last release that supports running Black on Python 3.6 (formatting 3.6 code will continue to be supported until further notice)
    • Reword the stability policy to say that we may, in rare cases, make changes that affect code that was not previously formatted by Black (#3155)

    Stable style

    • Fix an infinite loop when using # fmt: on/off in the middle of an expression or code block (#3158)
    • Fix incorrect handling of # fmt: skip on colon (:) lines (#3148)
    • Comments are no longer deleted when a line had spaces removed around power operators (#2874)

    Preview style

    • Single-character closing docstring quotes are no longer moved to their own line as this is invalid. This was a bug introduced in version 22.6.0. (#3166)
    • --skip-string-normalization / -S now prevents docstring prefixes from being normalized as expected (#3168)
    • When using --skip-magic-trailing-comma or -C, trailing commas are stripped from subscript expressions with more than 1 element (#3209)
    • Implicitly concatenated strings inside a list, set, or tuple are now wrapped inside parentheses (#3162)
    • Fix a string merging/split issue when a comment is present in the middle of implicitly concatenated strings on its own line (#3227)

    Blackd

    • blackd now supports enabling the preview style via the X-Preview header (#3217)

    Configuration

    • Black now uses the presence of debug f-strings to detect target version (#3215)
    • Fix misdetection of project root and verbose logging of sources in cases involving --stdin-filename (#3216)
    • Immediate .gitignore files in source directories given on the command line are now also respected, previously only .gitignore files in the project root and automatically discovered directories were respected (#3237)

    Documentation

    • Recommend using BlackConnect in IntelliJ IDEs (#3150)

    Integrations

    • Vim plugin: prefix messages with Black: so it's clear they come from Black (#3194)
    • Docker: changed to a /opt/venv installation + added to PATH to be available to non-root users (#3202)

    Output

    • Change from deprecated asyncio.get_event_loop() to create our event loop which removes DeprecationWarning (#3164)
    • Remove logging from internal blib2to3 library since it regularly emits error logs about failed caching that can and should be ignored (#3193)

    Parser

    • Type comments are now included in the AST equivalence check consistently so accidental deletion raises an error. Though type comments can't be tracked when running on PyPy 3.7 due to standard library limitations. (#2874)

    Performance

    ... (truncated)

    Changelog

    Sourced from black's changelog.

    22.8.0

    Highlights

    • Python 3.11 is now supported, except for blackd as aiohttp does not support 3.11 as of publishing (#3234)
    • This is the last release that supports running Black on Python 3.6 (formatting 3.6 code will continue to be supported until further notice)
    • Reword the stability policy to say that we may, in rare cases, make changes that affect code that was not previously formatted by Black (#3155)

    Stable style

    • Fix an infinite loop when using # fmt: on/off in the middle of an expression or code block (#3158)
    • Fix incorrect handling of # fmt: skip on colon (:) lines (#3148)
    • Comments are no longer deleted when a line had spaces removed around power operators (#2874)

    Preview style

    • Single-character closing docstring quotes are no longer moved to their own line as this is invalid. This was a bug introduced in version 22.6.0. (#3166)
    • --skip-string-normalization / -S now prevents docstring prefixes from being normalized as expected (#3168)
    • When using --skip-magic-trailing-comma or -C, trailing commas are stripped from subscript expressions with more than 1 element (#3209)
    • Implicitly concatenated strings inside a list, set, or tuple are now wrapped inside parentheses (#3162)
    • Fix a string merging/split issue when a comment is present in the middle of implicitly concatenated strings on its own line (#3227)

    Blackd

    • blackd now supports enabling the preview style via the X-Preview header (#3217)

    Configuration

    • Black now uses the presence of debug f-strings to detect target version (#3215)
    • Fix misdetection of project root and verbose logging of sources in cases involving --stdin-filename (#3216)
    • Immediate .gitignore files in source directories given on the command line are now also respected, previously only .gitignore files in the project root and automatically discovered directories were respected (#3237)

    Documentation

    • Recommend using BlackConnect in IntelliJ IDEs (#3150)

    Integrations

    ... (truncated)

    Commits
    • 2018e66 Prepare docs for release 22.8.0 (#3248)
    • 0019261 Update stable branch after publishing to PyPI (#3223)
    • 7757078 Improve & update release process to reflect recent changes (#3242)
    • 767604e Use .gitignore files in the initial source directories (#3237)
    • 2c90480 Use strict mypy checking (#3222)
    • ba618a3 Add parens around implicit string concatenations where it increases readabili...
    • c0cc19b Delay worker count determination
    • afed2c0 Load .gitignore and exclude regex at time of use
    • e269f44 Lazily import parallelized format modules
    • c47b91f Fix misdetection of project root with --stdin-filename (#3216)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 2
  • Bump pytest-asyncio from 0.17.2 to 0.19.0

    Bump pytest-asyncio from 0.17.2 to 0.19.0

    Bumps pytest-asyncio from 0.17.2 to 0.19.0.

    Release notes

    Sourced from pytest-asyncio's releases.

    pytest-asyncio 0.19.0


    title: 'pytest-asyncio: pytest support for asyncio'

    image

    image

    image

    Supported Python versions

    image

    pytest-asyncio is an Apache2 licensed library, written in Python, for testing asyncio code with pytest.

    asyncio code is usually written in the form of coroutines, which makes it slightly more difficult to test using normal testing tools. pytest-asyncio provides useful fixtures and markers to make testing easier.

    @pytest.mark.asyncio
    async def test_some_asyncio_code():
        res = await library.do_something()
        assert b"expected result" == res
    

    pytest-asyncio has been strongly influenced by pytest-tornado.

    Features

    • fixtures for creating and injecting versions of the asyncio event loop
    • fixtures for injecting unused tcp/udp ports
    • pytest markers for treating tests as asyncio coroutines
    • easy testing with non-default event loops
    • support for [async def]{.title-ref} fixtures and async generator fixtures
    • support auto mode to handle all async fixtures and tests automatically by asyncio; provide strict mode if a test suite should work with different async frameworks simultaneously, e.g. asyncio and trio.

    Installation

    ... (truncated)

    Changelog

    Sourced from pytest-asyncio's changelog.

    0.19.0 (22-07-13)

    • BREAKING: The default asyncio_mode is now strict. [#293](https://github.com/pytest-dev/pytest-asyncio/issues/293) <https://github.com/pytest-dev/pytest-asyncio/issues/293>_
    • Removes setup.py since all relevant configuration is present setup.cfg. Users requiring an editable installation of pytest-asyncio need to use pip v21.1 or newer. [#283](https://github.com/pytest-dev/pytest-asyncio/issues/283) <https://github.com/pytest-dev/pytest-asyncio/issues/283>_
    • Declare support for Python 3.11.

    0.18.3 (22-03-25)

    • Adds pytest-trio <https://pypi.org/project/pytest-trio/>_ to the test dependencies
    • Fixes a bug that caused pytest-asyncio to try to set up async pytest_trio fixtures in strict mode. [#298](https://github.com/pytest-dev/pytest-asyncio/issues/298) <https://github.com/pytest-dev/pytest-asyncio/issues/298>_

    0.18.2 (22-03-03)

    • Fix asyncio auto mode not marking static methods. [#295](https://github.com/pytest-dev/pytest-asyncio/issues/295) <https://github.com/pytest-dev/pytest-asyncio/issues/295>_
    • Fix a compatibility issue with Hypothesis 6.39.0. [#302](https://github.com/pytest-dev/pytest-asyncio/issues/302) <https://github.com/pytest-dev/pytest-asyncio/issues/302>_

    0.18.1 (22-02-10)

    • Fixes a regression that prevented async fixtures from working in synchronous tests. [#286](https://github.com/pytest-dev/pytest-asyncio/issues/286) <https://github.com/pytest-dev/pytest-asyncio/issues/286>_

    0.18.0 (22-02-07)

    • Raise a warning if @​pytest.mark.asyncio is applied to non-async function. [#275](https://github.com/pytest-dev/pytest-asyncio/issues/275) <https://github.com/pytest-dev/pytest-asyncio/issues/275>_
    • Support parametrized event_loop fixture. [#278](https://github.com/pytest-dev/pytest-asyncio/issues/278) <https://github.com/pytest-dev/pytest-asyncio/issues/278>_
    Commits
    • 2da33c4 docs: Prepare v0.19.0 release. (#385)
    • 07beb80 opt into strict mode by default (#380)
    • 25c54a5 Clarify documentation of event_loop fixture (#375)
    • 49f07a4 Bump typing-extensions from 4.2.0 to 4.3.0 in /dependencies/default (#382)
    • 739198b Bump hypothesis from 6.48.0 to 6.48.3 in /dependencies/default (#381)
    • db72f25 Bump importlib-metadata from 4.11.4 to 4.12.0 in /dependencies/default (#378)
    • 4cf16cf Bump hypothesis from 6.47.3 to 6.48.0 in /dependencies/default (#377)
    • f13c85f docs: Fix typo in README.
    • b463f72 Python 3.11 support (#370)
    • 860ff51 Bump hypothesis from 6.47.2 to 6.47.3 in /dependencies/default (#373)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 2
  • Bump black from 22.1.0 to 22.6.0

    Bump black from 22.1.0 to 22.6.0

    Bumps black from 22.1.0 to 22.6.0.

    Release notes

    Sourced from black's releases.

    22.6.0

    Style

    • Fix unstable formatting involving #fmt: skip and # fmt:skip comments (notice the lack of spaces) (#2970)

    Preview style

    • Docstring quotes are no longer moved if it would violate the line length limit (#3044)
    • Parentheses around return annotations are now managed (#2990)
    • Remove unnecessary parentheses around awaited objects (#2991)
    • Remove unnecessary parentheses in with statements (#2926)
    • Remove trailing newlines after code block open (#3035)

    Integrations

    • Add scripts/migrate-black.py script to ease introduction of Black to a Git project (#3038)

    Output

    • Output Python version and implementation as part of --version flag (#2997)

    Packaging

    • Use tomli instead of tomllib on Python 3.11 builds where tomllib is not available (#2987)

    Parser

    • PEP 654 syntax (for example, except *ExceptionGroup:) is now supported (#3016)
    • PEP 646 syntax (for example, Array[Batch, *Shape] or def fn(*args: *T) -> None) is now supported (#3071)

    Vim Plugin

    • Fix strtobool function. It didn't parse true/on/false/off. (#3025)

    Full Changelog: https://github.com/psf/black/compare/22.3.0...22.6.0


    Thank you!

    • @​jpy-git for improving our parentheses formatting significantly
    • @​siuryan for fixing a fmt: skip bug, making it a little less annoying to use :)
    • @​isidentical for implementing support for PEP 654 and 646 syntax
    • @​defntvdm for fixing our vim plugin, especially as we (the maintainers) don't really know vim script sadly
    • @​idorrington92 for fixing the docstring bug where Black would move the closing quotes causing it to violate the line length limit (whoops!)
    • @​hbrunn for contributing the migrate-black script
    • @​saroad2 for improving newline handling after code blocks and test infrastructure improvements

    ... and everyone else who contributed documentation, tests, or other improvements to the Black project!

    ... (truncated)

    Changelog

    Sourced from black's changelog.

    22.6.0

    Style

    • Fix unstable formatting involving #fmt: skip and # fmt:skip comments (notice the lack of spaces) (#2970)

    Preview style

    • Docstring quotes are no longer moved if it would violate the line length limit (#3044)
    • Parentheses around return annotations are now managed (#2990)
    • Remove unnecessary parentheses around awaited objects (#2991)
    • Remove unnecessary parentheses in with statements (#2926)
    • Remove trailing newlines after code block open (#3035)

    Integrations

    • Add scripts/migrate-black.py script to ease introduction of Black to a Git project (#3038)

    Output

    • Output Python version and implementation as part of --version flag (#2997)

    Packaging

    • Use tomli instead of tomllib on Python 3.11 builds where tomllib is not available (#2987)

    Parser

    • PEP 654 syntax (for example, except *ExceptionGroup:) is now supported (#3016)
    • PEP 646 syntax (for example, Array[Batch, *Shape] or def fn(*args: *T) -> None) is now supported (#3071)

    Vim Plugin

    • Fix strtobool function. It didn't parse true/on/false/off. (#3025)

    22.3.0

    Preview style

    • Code cell separators #%% are now standardised to # %% (#2919)
    • Remove unnecessary parentheses from except statements (#2939)
    • Remove unnecessary parentheses from tuple unpacking in for loops (#2945)
    • Avoid magic-trailing-comma in single-element subscripts (#2942)

    Configuration

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 2
  • Bump pre-commit from 2.17.0 to 2.19.0

    Bump pre-commit from 2.17.0 to 2.19.0

    Bumps pre-commit from 2.17.0 to 2.19.0.

    Release notes

    Sourced from pre-commit's releases.

    pre-commit v2.19.0

    Features

    Fixes

    Updating

    • Change pre-commit-validate-config / pre-commit-validate-manifest to pre-commit validate-config / pre-commit validate-manifest.

    pre-commit v2.18.1

    Fixes

    pre-commit v2.18.0

    Features

    Fixes

    ... (truncated)

    Changelog

    Sourced from pre-commit's changelog.

    2.19.0 - 2022-05-05

    Features

    Fixes

    Updating

    • Change pre-commit-validate-config / pre-commit-validate-manifest to pre-commit validate-config / pre-commit validate-manifest.

    2.18.1 - 2022-04-02

    Fixes

    2.18.0 - 2022-04-02

    Features

    ... (truncated)

    Commits
    • cc9d950 v2.19.0
    • 96bf685 fix non-unique id
    • 1b86655 Merge pull request #2375 from pre-commit/search-the-tracker
    • af46701 add search term required input to issue template
    • bec1728 Merge pull request #2370 from pre-commit/pre-commit-ci-update-config
    • 81129ce [pre-commit.ci] pre-commit autoupdate
    • 7f18926 Merge pull request #2362 from pre-commit/deprecate-separate-entry-points
    • 3929fe4 upgrade CI to ubuntu-latest / windows-latest
    • 777ffdd deprecate pre-commit-validate-{config,manifest}
    • e1ce4c0 Merge pull request #2349 from pre-commit/pre-commit-ci-update-config
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 2
  • Bump pytest from 7.0.1 to 7.1.2

    Bump pytest from 7.0.1 to 7.1.2

    Bumps pytest from 7.0.1 to 7.1.2.

    Release notes

    Sourced from pytest's releases.

    7.1.2

    pytest 7.1.2 (2022-04-23)

    Bug Fixes

    • #9726: An unnecessary numpy import inside pytest.approx{.interpreted-text role="func"} was removed.
    • #9820: Fix comparison of dataclasses with InitVar.
    • #9869: Increase stacklevel for the NODE_CTOR_FSPATH_ARG deprecation to point to the user's code, not pytest.
    • #9871: Fix a bizarre (and fortunately rare) bug where the [temp_path]{.title-ref} fixture could raise an internal error while attempting to get the current user's username.

    7.1.1

    pytest 7.1.1 (2022-03-17)

    Bug Fixes

    • #9767: Fixed a regression in pytest 7.1.0 where some conftest.py files outside of the source tree (e.g. in the [site-packages]{.title-ref} directory) were not picked up.

    7.1.0

    pytest 7.1.0 (2022-03-13)

    Breaking Changes

    • #8838: As per our policy, the following features have been deprecated in the 6.X series and are now removed:

      • pytest._fillfuncargs function.
      • pytest_warning_captured hook - use pytest_warning_recorded instead.
      • -k -foobar syntax - use -k 'not foobar' instead.
      • -k foobar: syntax.
      • pytest.collect module - import from pytest directly.

      For more information consult Deprecations and Removals in the docs.

    • #9437: Dropped support for Python 3.6, which reached end-of-life at 2021-12-23.

    Improvements

    • #5192: Fixed test output for some data types where -v would show less information.

      Also, when showing diffs for sequences, -q would produce full diffs instead of the expected diff.

    ... (truncated)

    Commits
    • 2f2f1a6 Prepare release version 7.1.2
    • 5c04f3a [7.1.x] Fix wrong log_file docs (#9879)
    • 078733c Merge pull request #9872 from pytest-dev/backport-9871-to-7.1.x
    • 3a7ead6 [7.1.x] fix: move 'import getpass' statement to try-clause
    • 6d75333 [7.1.x] Increase stacklevel to point at user's code (#9870)
    • ddbb998 [7.1.x] Increase stacklevel to point at user's code
    • 0ec5886 Merge pull request #9855 from pytest-dev/backport-9854-to-7.1.x
    • f2469fc [7.1.x] Docs: link to easy issues in contributing guide
    • 94ec0f8 Merge pull request #9846 from pytest-dev/backport-9842-to-7.1.x
    • 5ef96fd [7.1.x] fix comparison of dataclasses with InitVar
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 2
  • Bump pre-commit from 2.17.0 to 2.18.1

    Bump pre-commit from 2.17.0 to 2.18.1

    Bumps pre-commit from 2.17.0 to 2.18.1.

    Release notes

    Sourced from pre-commit's releases.

    pre-commit v2.18.1

    Fixes

    pre-commit v2.18.0

    Features

    Fixes

    Updating

    • Remove python3.6 support. Note that pre-commit still supports running hooks written in older versions, but pre-commit itself requires python 3.7+.
    • pre-commit has migrated from the master branch to main.
    Changelog

    Sourced from pre-commit's changelog.

    2.18.1 - 2022-04-02

    Fixes

    2.18.0 - 2022-04-02

    Features

    Fixes

    Updating

    • Remove python3.6 support. Note that pre-commit still supports running hooks written in older versions, but pre-commit itself requires python 3.7+.
    • pre-commit has migrated from the master branch to main.
    Commits
    • 0276e25 v2.18.1
    • f5af0a9 Merge pull request #2324 from pre-commit/local-hooks-py27
    • 1722448 fix python 2.7 repo: local hooks
    • c5a39ae v2.18.0
    • 3aa2ce8 Merge pull request #2323 from pre-commit/move-try-slightly
    • a138c85 move patch discarding inside try for staged_files_only
    • 7602abc Merge pull request #2322 from pre-commit/default-install-hook-types
    • e11163d Merge pull request #2301 from jeff-m-sullivan/rscript-path
    • d650160 Merge pull request #2252 from daschuer/worktree_fix
    • fd0177a implement default_install_hook_types
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 2
  • Bump black from 22.1.0 to 22.3.0

    Bump black from 22.1.0 to 22.3.0

    Bumps black from 22.1.0 to 22.3.0.

    Release notes

    Sourced from black's releases.

    22.3.0

    Preview style

    • Code cell separators #%% are now standardised to # %% (#2919)
    • Remove unnecessary parentheses from except statements (#2939)
    • Remove unnecessary parentheses from tuple unpacking in for loops (#2945)
    • Avoid magic-trailing-comma in single-element subscripts (#2942)

    Configuration

    • Do not format __pypackages__ directories by default (#2836)
    • Add support for specifying stable version with --required-version (#2832).
    • Avoid crashing when the user has no homedir (#2814)
    • Avoid crashing when md5 is not available (#2905)
    • Fix handling of directory junctions on Windows (#2904)

    Documentation

    • Update pylint config documentation (#2931)

    Integrations

    • Move test to disable plugin in Vim/Neovim, which speeds up loading (#2896)

    Output

    • In verbose, mode, log when Black is using user-level config (#2861)

    Packaging

    • Fix Black to work with Click 8.1.0 (#2966)
    • On Python 3.11 and newer, use the standard library's tomllib instead of tomli (#2903)
    • black-primer, the deprecated internal devtool, has been removed and copied to a separate repository (#2924)

    Parser

    • Black can now parse starred expressions in the target of for and async for statements, e.g for item in *items_1, *items_2: pass (#2879).
    Changelog

    Sourced from black's changelog.

    22.3.0

    Preview style

    • Code cell separators #%% are now standardised to # %% (#2919)
    • Remove unnecessary parentheses from except statements (#2939)
    • Remove unnecessary parentheses from tuple unpacking in for loops (#2945)
    • Avoid magic-trailing-comma in single-element subscripts (#2942)

    Configuration

    • Do not format __pypackages__ directories by default (#2836)
    • Add support for specifying stable version with --required-version (#2832).
    • Avoid crashing when the user has no homedir (#2814)
    • Avoid crashing when md5 is not available (#2905)
    • Fix handling of directory junctions on Windows (#2904)

    Documentation

    • Update pylint config documentation (#2931)

    Integrations

    • Move test to disable plugin in Vim/Neovim, which speeds up loading (#2896)

    Output

    • In verbose, mode, log when Black is using user-level config (#2861)

    Packaging

    • Fix Black to work with Click 8.1.0 (#2966)
    • On Python 3.11 and newer, use the standard library's tomllib instead of tomli (#2903)
    • black-primer, the deprecated internal devtool, has been removed and copied to a separate repository (#2924)

    Parser

    • Black can now parse starred expressions in the target of for and async for statements, e.g for item in *items_1, *items_2: pass (#2879).
    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 2
  • Bump pytest-asyncio from 0.17.2 to 0.18.3

    Bump pytest-asyncio from 0.17.2 to 0.18.3

    Bumps pytest-asyncio from 0.17.2 to 0.18.3.

    Release notes

    Sourced from pytest-asyncio's releases.

    pytest-asyncio 0.18.3


    title: 'pytest-asyncio: pytest support for asyncio'

    image

    image

    image

    Supported Python versions

    image

    pytest-asyncio is an Apache2 licensed library, written in Python, for testing asyncio code with pytest.

    asyncio code is usually written in the form of coroutines, which makes it slightly more difficult to test using normal testing tools. pytest-asyncio provides useful fixtures and markers to make testing easier.

    @pytest.mark.asyncio
    async def test_some_asyncio_code():
        res = await library.do_something()
        assert b"expected result" == res
    

    pytest-asyncio has been strongly influenced by pytest-tornado.

    Features

    • fixtures for creating and injecting versions of the asyncio event loop
    • fixtures for injecting unused tcp/udp ports
    • pytest markers for treating tests as asyncio coroutines
    • easy testing with non-default event loops
    • support for [async def]{.title-ref} fixtures and async generator fixtures
    • support auto mode to handle all async fixtures and tests automatically by asyncio; provide strict mode if a test suite should work with different async frameworks simultaneously, e.g. asyncio and trio.

    Installation

    ... (truncated)

    Changelog

    Sourced from pytest-asyncio's changelog.

    0.18.3 (22-03-25)

    • Adds pytest-trio <https://pypi.org/project/pytest-trio/>_ to the test dependencies
    • Fixes a bug that caused pytest-asyncio to try to set up async pytest_trio fixtures in strict mode. [#298](https://github.com/pytest-dev/pytest-asyncio/issues/298) <https://github.com/pytest-dev/pytest-asyncio/issues/298>_

    0.18.2 (22-03-03)

    • Fix asyncio auto mode not marking static methods. [#295](https://github.com/pytest-dev/pytest-asyncio/issues/295) <https://github.com/pytest-dev/pytest-asyncio/issues/295>_
    • Fix a compatibility issue with Hypothesis 6.39.0. [#302](https://github.com/pytest-dev/pytest-asyncio/issues/302) <https://github.com/pytest-dev/pytest-asyncio/issues/302>_

    0.18.1 (22-02-10)

    • Fixes a regression that prevented async fixtures from working in synchronous tests. [#286](https://github.com/pytest-dev/pytest-asyncio/issues/286) <https://github.com/pytest-dev/pytest-asyncio/issues/286>_

    0.18.0 (22-02-07)

    • Raise a warning if @​pytest.mark.asyncio is applied to non-async function. [#275](https://github.com/pytest-dev/pytest-asyncio/issues/275) <https://github.com/pytest-dev/pytest-asyncio/issues/275>_
    • Support parametrized event_loop fixture. [#278](https://github.com/pytest-dev/pytest-asyncio/issues/278) <https://github.com/pytest-dev/pytest-asyncio/issues/278>_
    Commits
    • bcdc049 docs: Prepare release of v0.18.3. (#313)
    • f979af9 Do not try to initialize async fixtures without explicit asyncio mark in stri...
    • 133d8a8 Fix CI linting workflow (#308)
    • 82d212b Extract changelog to separate file (#306)
    • 929608e docs: Prepare release 0.18.2. (#304)
    • 2359807 Bump actions/checkout from 2 to 3 (#301)
    • cad1b94 fix: Add positional arguments to the synchronous test function wrapper. (#303)
    • 357cddb Bump actions/setup-python from 2 to 3 (#300)
    • beee1f6 Update changelog
    • 9246f58 Fix asyncio auto mode not marking static methods (closes pytest-dev/pytest-as...
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 2
  • Bump pytest from 7.0.1 to 7.1.1

    Bump pytest from 7.0.1 to 7.1.1

    Bumps pytest from 7.0.1 to 7.1.1.

    Release notes

    Sourced from pytest's releases.

    7.1.1

    pytest 7.1.1 (2022-03-17)

    Bug Fixes

    • #9767: Fixed a regression in pytest 7.1.0 where some conftest.py files outside of the source tree (e.g. in the [site-packages]{.title-ref} directory) were not picked up.

    7.1.0

    pytest 7.1.0 (2022-03-13)

    Breaking Changes

    • #8838: As per our policy, the following features have been deprecated in the 6.X series and are now removed:

      • pytest._fillfuncargs function.
      • pytest_warning_captured hook - use pytest_warning_recorded instead.
      • -k -foobar syntax - use -k 'not foobar' instead.
      • -k foobar: syntax.
      • pytest.collect module - import from pytest directly.

      For more information consult Deprecations and Removals in the docs.

    • #9437: Dropped support for Python 3.6, which reached end-of-life at 2021-12-23.

    Improvements

    • #5192: Fixed test output for some data types where -v would show less information.

      Also, when showing diffs for sequences, -q would produce full diffs instead of the expected diff.

    • #9362: pytest now avoids specialized assert formatting when it is detected that the default __eq__ is overridden in attrs or dataclasses.

    • #9536: When -vv is given on command line, show skipping and xfail reasons in full instead of truncating them to fit the terminal width.

    • #9644: More information about the location of resources that led Python to raise ResourceWarning{.interpreted-text role="class"} can now be obtained by enabling tracemalloc{.interpreted-text role="mod"}.

      See resource-warnings{.interpreted-text role="ref"} for more information.

    • #9678: More types are now accepted in the ids argument to @pytest.mark.parametrize. Previously only [str]{.title-ref}, [float]{.title-ref}, [int]{.title-ref} and [bool]{.title-ref} were accepted; now [bytes]{.title-ref}, [complex]{.title-ref}, [re.Pattern]{.title-ref}, [Enum]{.title-ref} and anything with a [__name__]{.title-ref} are also accepted.

    • #9692: pytest.approx{.interpreted-text role="func"} now raises a TypeError{.interpreted-text role="class"} when given an unordered sequence (such as set{.interpreted-text role="class"}).

      Note that this implies that custom classes which only implement __iter__ and __len__ are no longer supported as they don't guarantee order.

    ... (truncated)

    Commits
    • 0ffe9e0 Prepare release version 7.1.1
    • 6f2c1ec Merge pull request #9784 from pytest-dev/backport-9768-to-7.1.x
    • a65c47a Merge pull request #9783 from pytest-dev/backport-9780-to-7.1.x
    • 30d995e [pre-commit.ci] auto fixes from pre-commit.com hooks
    • 10a14d1 [7.1.x] testing: fix tests when run under -v or -vv
    • f4cfc59 [pre-commit.ci] auto fixes from pre-commit.com hooks
    • f1df807 [7.1.x] config: restore pre-pytest 7.1.0 confcutdir exclusion behavior
    • 7d4d1ec Merge pull request #9758 from pytest-dev/release-7.1.0
    • 1dbffcc [pre-commit.ci] auto fixes from pre-commit.com hooks
    • d53a5fb Prepare release version 7.1.0
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 2
  • Bump certifi from 2021.10.8 to 2022.12.7

    Bump certifi from 2021.10.8 to 2022.12.7

    Bumps certifi from 2021.10.8 to 2022.12.7.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the Security Alerts page.
    dependencies 
    opened by dependabot[bot] 1
  • Bump pytest-asyncio from 0.17.2 to 0.20.2

    Bump pytest-asyncio from 0.17.2 to 0.20.2

    Bumps pytest-asyncio from 0.17.2 to 0.20.2.

    Release notes

    Sourced from pytest-asyncio's releases.

    pytest-asyncio 0.20.2


    title: 'pytest-asyncio: pytest support for asyncio'

    image

    image

    image

    Supported Python versions

    image

    pytest-asyncio is an Apache2 licensed library, written in Python, for testing asyncio code with pytest.

    asyncio code is usually written in the form of coroutines, which makes it slightly more difficult to test using normal testing tools. pytest-asyncio provides useful fixtures and markers to make testing easier.

    @pytest.mark.asyncio
    async def test_some_asyncio_code():
        res = await library.do_something()
        assert b"expected result" == res
    

    pytest-asyncio has been strongly influenced by pytest-tornado.

    Features

    • fixtures for creating and injecting versions of the asyncio event loop
    • fixtures for injecting unused tcp/udp ports
    • pytest markers for treating tests as asyncio coroutines
    • easy testing with non-default event loops
    • support for [async def]{.title-ref} fixtures and async generator fixtures
    • support auto mode to handle all async fixtures and tests automatically by asyncio; provide strict mode if a test suite should work with different async frameworks simultaneously, e.g. asyncio and trio.

    Installation

    ... (truncated)

    Changelog

    Sourced from pytest-asyncio's changelog.

    0.20.2 (22-11-11)

    • Fixes an issue with async fixtures that are defined as methods on a test class not being rebound to the actual test instance. [#197](https://github.com/pytest-dev/pytest-asyncio/issues/197) <https://github.com/pytest-dev/pytest-asyncio/issues/197>_
    • Replaced usage of deprecated @pytest.mark.tryfirst with @pytest.hookimpl(tryfirst=True) [#438](https://github.com/pytest-dev/pytest-asyncio/issues/438) <https://github.com/pytest-dev/pytest-asyncio/pull/438>_

    0.20.1 (22-10-21)

    • Fixes an issue that warned about using an old version of pytest, even though the most recent version was installed. [#430](https://github.com/pytest-dev/pytest-asyncio/issues/430) <https://github.com/pytest-dev/pytest-asyncio/issues/430>_

    0.20.0 (22-10-21)

    • BREAKING: Removed legacy mode. If you're upgrading from v0.19 and you haven't configured asyncio_mode = legacy, you can upgrade without taking any additional action. If you're upgrading from an earlier version or you have explicitly enabled legacy mode, you need to switch to auto or strict mode before upgrading to this version.
    • Deprecate use of pytest v6.
    • Fixed an issue which prevented fixture setup from being cached. [#404](https://github.com/pytest-dev/pytest-asyncio/issues/404) <https://github.com/pytest-dev/pytest-asyncio/pull/404>_

    0.19.0 (22-07-13)

    • BREAKING: The default asyncio_mode is now strict. [#293](https://github.com/pytest-dev/pytest-asyncio/issues/293) <https://github.com/pytest-dev/pytest-asyncio/issues/293>_
    • Removes setup.py since all relevant configuration is present setup.cfg. Users requiring an editable installation of pytest-asyncio need to use pip v21.1 or newer. [#283](https://github.com/pytest-dev/pytest-asyncio/issues/283) <https://github.com/pytest-dev/pytest-asyncio/issues/283>_
    • Declare support for Python 3.11.

    0.18.3 (22-03-25)

    • Adds pytest-trio <https://pypi.org/project/pytest-trio/>_ to the test dependencies
    • Fixes a bug that caused pytest-asyncio to try to set up async pytest_trio fixtures in strict mode. [#298](https://github.com/pytest-dev/pytest-asyncio/issues/298) <https://github.com/pytest-dev/pytest-asyncio/issues/298>_

    0.18.2 (22-03-03)

    • Fix asyncio auto mode not marking static methods. [#295](https://github.com/pytest-dev/pytest-asyncio/issues/295) <https://github.com/pytest-dev/pytest-asyncio/issues/295>_
    • Fix a compatibility issue with Hypothesis 6.39.0. [#302](https://github.com/pytest-dev/pytest-asyncio/issues/302) <https://github.com/pytest-dev/pytest-asyncio/issues/302>_

    0.18.1 (22-02-10)

    • Fixes a regression that prevented async fixtures from working in synchronous tests. [#286](https://github.com/pytest-dev/pytest-asyncio/issues/286) <https://github.com/pytest-dev/pytest-asyncio/issues/286>_

    0.18.0 (22-02-07)

    • Raise a warning if @​pytest.mark.asyncio is applied to non-async function. [#275](https://github.com/pytest-dev/pytest-asyncio/issues/275) <https://github.com/pytest-dev/pytest-asyncio/issues/275>_
    • Support parametrized event_loop fixture. [#278](https://github.com/pytest-dev/pytest-asyncio/issues/278) <https://github.com/pytest-dev/pytest-asyncio/issues/278>_
    Commits
    • 07a1416 Prepare release of v0.20.2.
    • dc3ad21 Build(deps): Bump pytest-trio in /dependencies/default (#441)
    • d9faba8 Build(deps): Bump mypy from 0.982 to 0.990 in /dependencies/default (#440)
    • fe63e34 Handle bound fixture methods correctly (#439)
    • 38fc032 Bump to pytest 7.2.0 (#438)
    • 28ba705 Build(deps): Bump hypothesis in /dependencies/default (#437)
    • 91e723a Build(deps): Bump zipp from 3.9.0 to 3.10.0 in /dependencies/default (#434)
    • 0ca201b Fix setuptools deprecation warning for license_file (#432)
    • c8d0174 fix: Do not warn about outdated pytest version when pytest>=7 is installed. (...
    • 6450ddb Prepare release of v0.20.0. (#428)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • Bump pytest from 7.0.1 to 7.2.0

    Bump pytest from 7.0.1 to 7.2.0

    Bumps pytest from 7.0.1 to 7.2.0.

    Release notes

    Sourced from pytest's releases.

    7.2.0

    pytest 7.2.0 (2022-10-23)

    Deprecations

    • #10012: Update pytest.PytestUnhandledCoroutineWarning{.interpreted-text role="class"} to a deprecation; it will raise an error in pytest 8.

    • #10396: pytest no longer depends on the py library. pytest provides a vendored copy of py.error and py.path modules but will use the py library if it is installed. If you need other py.* modules, continue to install the deprecated py library separately, otherwise it can usually be removed as a dependency.

    • #4562: Deprecate configuring hook specs/impls using attributes/marks.

      Instead use :pypytest.hookimpl{.interpreted-text role="func"} and :pypytest.hookspec{.interpreted-text role="func"}. For more details, see the docs <legacy-path-hooks-deprecated>{.interpreted-text role="ref"}.

    • #9886: The functionality for running tests written for nose has been officially deprecated.

      This includes:

      • Plain setup and teardown functions and methods: this might catch users by surprise, as setup() and teardown() are not pytest idioms, but part of the nose support.
      • Setup/teardown using the @​with_setup decorator.

      For more details, consult the deprecation docs <nose-deprecation>{.interpreted-text role="ref"}.

    Features

    • #9897: Added shell-style wildcard support to testpaths.

    Improvements

    • #10218: @pytest.mark.parametrize() (and similar functions) now accepts any Sequence[str] for the argument names, instead of just list[str] and tuple[str, ...].

      (Note that str, which is itself a Sequence[str], is still treated as a comma-delimited name list, as before).

    • #10381: The --no-showlocals flag has been added. This can be passed directly to tests to override --showlocals declared through addopts.

    • #3426: Assertion failures with strings in NFC and NFD forms that normalize to the same string now have a dedicated error message detailing the issue, and their utf-8 representation is expresed instead.

    • #7337: A warning is now emitted if a test function returns something other than [None]{.title-ref}. This prevents a common mistake among beginners that expect that returning a [bool]{.title-ref} (for example [return foo(a, b) == result]{.title-ref}) would cause a test to pass or fail, instead of using [assert]{.title-ref}.

    • #8508: Introduce multiline display for warning matching via :pypytest.warns{.interpreted-text role="func"} and enhance match comparison for :py_pytest._code.ExceptionInfo.match{.interpreted-text role="func"} as returned by :pypytest.raises{.interpreted-text role="func"}.

    • #8646: Improve :pypytest.raises{.interpreted-text role="func"}. Previously passing an empty tuple would give a confusing error. We now raise immediately with a more helpful message.

    • #9741: On Python 3.11, use the standard library's tomllib{.interpreted-text role="mod"} to parse TOML.

      tomli{.interpreted-text role="mod"}` is no longer a dependency on Python 3.11.

    • #9742: Display assertion message without escaped newline characters with -vv.

    • #9823: Improved error message that is shown when no collector is found for a given file.

    ... (truncated)

    Commits
    • 3af3f56 Prepare release version 7.2.0
    • bc2c3b6 Merge pull request #10408 from NateMeyvis/patch-2
    • d84ed48 Merge pull request #10409 from pytest-dev/asottile-patch-1
    • ffe49ac Merge pull request #10396 from pytest-dev/pylib-hax
    • d352098 allow jobs to pass if codecov.io fails
    • c5c562b Fix typos in CONTRIBUTING.rst
    • d543a45 add deprecation changelog for py library vendoring
    • f341a5c Merge pull request #10407 from NateMeyvis/patch-1
    • 1027dc8 [pre-commit.ci] auto fixes from pre-commit.com hooks
    • 6b905ee Add note on tags to CONTRIBUTING.rst
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • Bump black from 22.1.0 to 22.10.0

    Bump black from 22.1.0 to 22.10.0

    Bumps black from 22.1.0 to 22.10.0.

    Release notes

    Sourced from black's releases.

    22.10.0

    Highlights

    • Runtime support for Python 3.6 has been removed. Formatting 3.6 code will still be supported until further notice.

    Stable style

    • Fix a crash when # fmt: on is used on a different block level than # fmt: off (#3281)

    Preview style

    • Fix a crash when formatting some dicts with parenthesis-wrapped long string keys (#3262)

    Configuration

    • .ipynb_checkpoints directories are now excluded by default (#3293)
    • Add --skip-source-first-line / -x option to ignore the first line of source code while formatting (#3299)

    Packaging

    • Executables made with PyInstaller will no longer crash when formatting several files at once on macOS. Native x86-64 executables for macOS are available once again. (#3275)
    • Hatchling is now used as the build backend. This will not have any effect for users who install Black with its wheels from PyPI. (#3233)
    • Faster compiled wheels are now available for CPython 3.11 (#3276)

    Blackd

    • Windows style (CRLF) newlines will be preserved (#3257).

    Integrations

    • Vim plugin: add flag (g:black_preview) to enable/disable the preview style (#3246)
    • Update GitHub Action to support formatting of Jupyter Notebook files via a jupyter option (#3282)
    • Update GitHub Action to support use of version specifiers (e.g. <23) for Black version (#3265)

    22.8.0

    Highlights

    • Python 3.11 is now supported, except for blackd as aiohttp does not support 3.11 as of publishing (#3234)
    • This is the last release that supports running Black on Python 3.6 (formatting 3.6 code will continue to be supported until further notice)
    • Reword the stability policy to say that we may, in rare cases, make changes that affect code that was not previously formatted by Black (#3155)

    ... (truncated)

    Changelog

    Sourced from black's changelog.

    22.10.0

    Highlights

    • Runtime support for Python 3.6 has been removed. Formatting 3.6 code will still be supported until further notice.

    Stable style

    • Fix a crash when # fmt: on is used on a different block level than # fmt: off (#3281)

    Preview style

    • Fix a crash when formatting some dicts with parenthesis-wrapped long string keys (#3262)

    Configuration

    • .ipynb_checkpoints directories are now excluded by default (#3293)
    • Add --skip-source-first-line / -x option to ignore the first line of source code while formatting (#3299)

    Packaging

    • Executables made with PyInstaller will no longer crash when formatting several files at once on macOS. Native x86-64 executables for macOS are available once again. (#3275)
    • Hatchling is now used as the build backend. This will not have any effect for users who install Black with its wheels from PyPI. (#3233)
    • Faster compiled wheels are now available for CPython 3.11 (#3276)

    Blackd

    • Windows style (CRLF) newlines will be preserved (#3257).

    Integrations

    • Vim plugin: add flag (g:black_preview) to enable/disable the preview style (#3246)
    • Update GitHub Action to support formatting of Jupyter Notebook files via a jupyter option (#3282)
    • Update GitHub Action to support use of version specifiers (e.g. <23) for Black version (#3265)

    22.8.0

    Highlights

    • Python 3.11 is now supported, except for blackd as aiohttp does not support 3.11 as of publishing (#3234)

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • Bump pre-commit from 2.17.0 to 2.20.0

    Bump pre-commit from 2.17.0 to 2.20.0

    Bumps pre-commit from 2.17.0 to 2.20.0.

    Release notes

    Sourced from pre-commit's releases.

    pre-commit v2.20.0

    Features

    • Expose source and object-name (positional args) of prepare-commit-msg hook as PRE_COMMIT_COMIT_MSG_SOURCE and PRE_COMMIT_COMMIT_OBJECT_NAME.

    Fixes

    pre-commit v2.19.0

    Features

    Fixes

    Updating

    • Change pre-commit-validate-config / pre-commit-validate-manifest to pre-commit validate-config / pre-commit validate-manifest.

    pre-commit v2.18.1

    Fixes

    pre-commit v2.18.0

    Features

    ... (truncated)

    Changelog

    Sourced from pre-commit's changelog.

    2.20.0 - 2022-07-10

    Features

    • Expose source and object-name (positional args) of prepare-commit-msg hook as PRE_COMMIT_COMIT_MSG_SOURCE and PRE_COMMIT_COMMIT_OBJECT_NAME.

    Fixes

    2.19.0 - 2022-05-05

    Features

    Fixes

    Updating

    • Change pre-commit-validate-config / pre-commit-validate-manifest to pre-commit validate-config / pre-commit validate-manifest.

    2.18.1 - 2022-04-02

    Fixes

    ... (truncated)

    Commits
    • 78a2d86 v2.20.0
    • e3dc5b7 Merge pull request #2454 from pre-commit/asottile-patch-1
    • ebce88c remove warnings checks
    • d6cc8a1 Merge pull request #2453 from hroncok/python3.11
    • 901e831 Tests: Adjust traceback regexes to allow Python 3.11+ ^^^^^^^
    • 98bb7e6 Merge pull request #2440 from pre-commit/pre-commit-ci-update-config
    • 706d1e9 Merge pull request #2439 from pre-commit/all-repos_autofix_type-checking
    • 3ebd101 [pre-commit.ci] pre-commit autoupdate
    • d8b5930 remove imports from TYPE_CHECKING (py37+)
    • 170335c Merge pull request #2429 from pre-commit/remove-config-option-when-unused
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • Bump beautifulsoup4 from 4.10.0 to 4.11.1

    Bump beautifulsoup4 from 4.10.0 to 4.11.1

    Bumps beautifulsoup4 from 4.10.0 to 4.11.1.

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
Releases(v0.2.3)
Owner
Dan Claudiu Pop
Dan Claudiu Pop
Crawler in Python 3.7, 3.8. 3.9. Pypy3

Description Python Crawler written Python 3. (Supports major Python releases Python3.6, Python3.7 and Python 3.8) Installation and Use Setup VirtualEn

Vinit Kumar 2 Mar 12, 2022
Scrapy uses Request and Response objects for crawling web sites.

Requests and Responses¶ Scrapy uses Request and Response objects for crawling web sites. Typically, Request objects are generated in the spiders and p

Md Rashidul Islam 1 Nov 03, 2021
一款利用Python来自动获取QQ音乐上某个歌手所有歌曲歌词的爬虫软件

QQ音乐歌词爬虫 一款利用Python来自动获取QQ音乐上某个歌手所有歌曲歌词的爬虫软件,默认去除了所有演唱会(Live)版本的歌曲。 使用方法 直接运行python run.py即可,然后输入你想获取的歌手名字,然后静静等待片刻。 output目录下保存生成的歌词和歌名文件。以周杰伦为例,会生成两

Yang Wei 11 Jul 27, 2022
Telegram Group Scrapper

this programe is make your work so much easy on telegrame. do you want to send messages on everyone to your group or others group. use this script it will do your work automatically with one click. a

HackArrOw 3 Dec 03, 2022
Scrapes the Sun Life of Canada Philippines web site for historical prices of their investment funds and then saves them as CSV files.

slocpi-scraper Sun Life of Canada Philippines Inc Investment Funds Scraper Install dependencies pip install -r requirements.txt Usage General format:

Daryl Yu 2 Jan 07, 2022
CRI Scrape is a tool for get general info about Italian Red Cross in GAIA Platform

CRI Scrape CRI Scrape is a tool for get general info about Italian Red Cross in GAIA Platform Disclaimer This code is only for educational purpose. So

Vincenzo Cardone 0 Jul 23, 2022
Web Scraping Practica With Python

Web-Scraping-Practica Integrants: Guillem Vidal Pallarols. Lídia Bandrés Solé Fitxers: Aquest document és el primer que trobem. A continuació trobem u

2 Nov 08, 2021
Web scraper build using python.

Web Scraper This project is made in pyhthon. It took some info. from website list then add them into data.json file. The dependencies used are: reques

Shashwat Harsh 2 Jul 22, 2022
Instagram_scrapper - This project allow you to scrape the list of followers, following or both from a public Instagram account, and create a csv or excel file easily.

Instagram_scrapper This project allow you to scrape the list of followers, following or both from a public Instagram account, and create a csv or exce

Lakhdar Belkharroubi 5 Oct 17, 2022
Creating Scrapy scrapers via the Django admin interface

django-dynamic-scraper Django Dynamic Scraper (DDS) is an app for Django which builds on top of the scraping framework Scrapy and lets you create and

Holger Drewes 1.1k Dec 17, 2022
京东茅台抢购

截止 2021/2/1 日,该项目已无法使用! 京东:约满即止,仅限京东实名认证用户APP端抢购,2月1日10:00开始预约,2月1日12:00开始抢购(京东APP需升级至8.5.6版本及以上) 写在前面 本项目来自 huanghyw - jd_seckill,作者的项目地址我找不到了,找到了再贴上

abee 73 Dec 03, 2022
robobrowser - A simple, Pythonic library for browsing the web without a standalone web browser.

RoboBrowser: Your friendly neighborhood web scraper Homepage: http://robobrowser.readthedocs.org/ RoboBrowser is a simple, Pythonic library for browsi

Joshua Carp 3.7k Dec 27, 2022
Python framework to scrape Pastebin pastes and analyze them

pastepwn - Paste-Scraping Python Framework Pastebin is a very helpful tool to store or rather share ascii encoded data online. In the world of OSINT,

Rico 105 Dec 29, 2022
Explore scraping with BeautifulSoup!

beautifulsoup-scrape Explore scraping with BeautifulSoup! Part One: Start from Shakespeare As my professor is a poet (yes, and he teaches me data and

Chuqin 2 Oct 05, 2022
This code will be able to scrape movies from a movie website and also provide download links to newly uploaded movies.

Movies-Scraper You are probably tired of navigating through a movie website to get the right movie you'd want to watch during the weekend. There may e

1 Jan 31, 2022
SkyScrapers: A collection of variety of Scraping Apps

SkyScrapers Collection of variety of Web Scraping Apps The web-scrapers involved

Biplov Pokhrel 3 Feb 17, 2022
Using Selenium with Python to Web Scrap Popular Youtube Tech Channels.

Web Scrapping Popular Youtube Tech Channels with Selenium Data Mining, Data Wrangling, and Exploratory Data Analysis About the Data Web scrapi

David Rusho 0 Aug 18, 2021
Newsscraper - A simple Python 3 module to get crypto or news articles and their content from various RSS feeds.

NewsScraper A simple Python 3 module to get crypto or news articles and their content from various RSS feeds. 🔧 Installation Clone the repo locally.

Rokas 3 Jan 02, 2022
ChromiumJniGenerator - Jni Generator module extracted from Chromium project

ChromiumJniGenerator - Jni Generator module extracted from Chromium project

allenxuan 4 Jun 12, 2022
for those who dont want to pay $10/month for high school game footage with ads

nfhs-scraper Disclaimer: I am in no way responsible for what you choose to do with this script and guide. I do not endorse avoiding paywalls or any il

Conrad Crawford 5 Apr 12, 2022