A modern API testing tool for web applications built with Open API and GraphQL specifications.

Last update: Dec 30, 2022

Overview

Schemathesis

Schemathesis is a modern API testing tool for web applications built with Open API and GraphQL specifications.

It reads the application schema and generates test cases, which will ensure that your application is compliant with its schema.

The application under test could be written in any language; the only thing you need is a valid API schema in a supported format.

Simple to use and yet powerful to uncover hard-to-find errors thanks to the property-based testing approach backed by state-of-the-art Hypothesis library.

📣 Please fill out our quick survey so that we can learn how satisfied you are with Schemathesis, and what improvements we should make. Thank you!

Features

Content-Type, schema, and status code conformance checks for Open API;
Testing of explicit examples from the input schema;
Stateful testing via Open API links;
Concurrent test execution;
Targeted testing;
Storing and replaying network requests;
Built-in ASGI / WSGI application support;
Ready-to-go Docker image;
Configurable with user-defined checks, string formats, hooks, and targets.

Installation

To install Schemathesis via pip run the following command:

pip install schemathesis

Usage

You can use Schemathesis in the command line:

schemathesis run --stateful=links --checks all http://0.0.0.0:8081/schema.yaml

Or in your Python tests:

import schemathesis

schema = schemathesis.from_uri("http://0.0.0.0:8081/schema.yaml")


@schema.parametrize()
def test_api(case):
    case.call_and_validate()

CLI is simple to use and requires no coding; the in-code approach gives more flexibility.

Both examples above will run hundreds of requests against the API under test and report all found failures and inconsistencies along with instructions to reproduce them.

💡 See a complete working example project in the /example directory. 💡

Contributing

Any contribution to development, testing, or any other area is highly appreciated and useful to the project. For guidance on how to contribute to Schemathesis, see the contributing guidelines.

Support this project

Hi, my name is Dmitry! I started this project during my work at Kiwi.com. I am grateful to them for all the support they provided to this project during its early days and for the opportunity to evolve Schemathesis independently.

In order to grow the community of contributors and users, and allow me to devote more time to this project, please donate today.

Also, I occasionally write posts about Schemathesis in my blog and offer consulting services for businesses.

Commercial support

If you are interested in the effective integration of Schemathesis to your private project, you can schedule an appointment and I will help you do that.

License

The code in this project is licensed under MIT license. By contributing to Schemathesis, you agree that your contributions will be licensed under its MIT license.

Comments

Store requests & responses
Closes #379

[x] Add command to cassette

[x] Documentation

[x] Reuse prepared request to avoid double computation

[x] CLI output about saved cassette file
opened by Stranger6667 26
Add hooks to modify request before request is sent
Description

Schemathesis currently has hooks that allow users to extend the set of Schemathesis validation functions. I would like to be able to write hooks that allow me to alter the request before it is sent to the API. This will allow me to test specific behavior of the API.

Example of one extended validation rule:

Accept header MUST be used to indicate allowable request methods in a 405 response.

To exercise this validation rule, we need to send a request with an operation that we know is not in the API definition for that endpoint. This idea is mentioned in #108, so this behavior may be included as part of the broader goal to generate invalid data in #65.

However, some of the rules are likely too specific to be part of the default request generation. For example:

Requests including Accept header with only formats not supported by the server MUST be rejected with a 406 status code and appropriate error response model.

Design

The validation hooks are simple and easy to use, and I think the request hooks should use the same design with different parameters. I think the should request hooks should take a request object and a case object (or any object that contains the API description for the endpoint associated with the request).

Questions to Consider

Does mutating an initially valid request cause problems?

Do the extended validation rules still run when the status code is not in the API definition? 415 response, for example, may not be documented in the OpenAPI description.

Which requests do we modify? We only want to modify a subset of the requests. Maybe we generate a valid request for each hook in addition to the default requests. Each additional valid request will be modified by a single hook.

Priority: Medium Type: Feature
opened by barrett-schonefeld 21
[FEATURE] Proxy traffic + disable certificate verification

1: Add --proxy function to proxy traffic thru other tools.

2: add --no-verify function to allow self signed certificate to proxy https thru tools.

(Error example: Error: requests.exceptions.SSLError: HTTPSConnectionPool(host='', port=443): Max retries exceeded with url: /v0/docs/openapi.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))
Priority: Medium Type: Documentation

opened by wellenc-lex 20
validation of response schema not working for recursive schemas

In response_schema_conformance (in checks.py) schemathesis is validating the response schema after it has tried to resolve it. If the response schema is recursive this doesnt work. It would be better to directly pass the response schema to jsonschema.validate() instead of the resolved one.

if i replace schema = case.endpoint.schema._get_response_schema(definition) with schema = responses[str(response.status_code)]['content'][content_type] (will fail if response has no content type) the validation works fine.

since jsonschema.validate() can handle recursive schemas, this might also be a better fix for https://github.com/kiwicom/schemathesis/issues/435, validate the raw schema and then resolve it as much as needed to safe time wasted in unnecessary recursions.

PS: just wanted to say thank you for schemathesis and not just keep opening new issues. I t has really helped us ;)
Priority: Medium Type: Bug

opened by jeremyschiemann 15

Support multiple examples specified in Open API schema

As mentioned in #582 we can generate multiple examples via the examples keyword:

parameters:
  - in: query
    name: limit
    schema:
      type: integer
      maximum: 50
    examples:       # Multiple examples
      zero:         # Distinct name
        value: 0    # Example value
        summary: A sample limit value  # Optional description
      max: # Distinct name
        value: 50   # Example value
        summary: A sample limit value

Priority: Medium Type: Enhancement

opened by Stranger6667 14

Validate error for nullable properties since v1.3.0

Hi, I have schemathesis==1.3.3 and I have the same issue as #335:

Details: 

None is not of type 'string'

Failed validating 'type' in schema['items']['properties']['family_name']:
    {'maxLength': 50,
     'nullable': True,
     'title': 'Family Name',
     'type': 'string'}

On instance[0]['family_name']:
    None

I tested older versions, This is working in v1.2.0, but regressed since v1.3.0

Priority: High Type: Bug

opened by grigi 14

Falsifying examples are not shown in case of hypothesis errors

If the test run discovers multiple errors and hypothesis.errors.MultipleFailures: Hypothesis found 2 distinct failures. is raised, then all the other output is missing

Also, it goes for timeout errors, we can display it as "E" in the CLI output and not stop the whole test suite
Priority: High Type: Bug

opened by Stranger6667 14
[BUG] startup failure caused by dependencies, requires httpx
Checklist

[x] I checked the FAQ section of the documentation

[x] I looked for similar issues in the issue tracker

Describe the bug Because the upstream starlette updated the dependencies,the following error occurs ModuleNotFoundError: No module named 'httpx'

To Reproduce

https://github.com/HXSecurity/DongTai/actions/runs/3132544194/jobs/5084991907 File "/home/runner/.local/bin/schemathesis", line 5, in from schemathesis.cli import schemathesis File "/home/runner/.local/lib/python3.8/site-packages/schemathesis/init.py", line 7, in from . import auth, fixups, graphql, hooks, runner, serializers, targets File "/home/runner/.local/lib/python3.8/site-packages/schemathesis/graphql.py", line 3, in from .specs.graphql import nodes File "/home/runner/.local/lib/python3.8/site-packages/schemathesis/specs/graphql/init.py", line 1, in from .loaders import from_asgi, from_dict, from_file, from_path, from_url, from_wsgi File "/home/runner/.local/lib/python3.8/site-packages/schemathesis/specs/graphql/loaders.py", line 9, in from starlette.testclient import TestClient as ASGIClient File "/home/runner/.local/lib/python3.8/site-packages/starlette/testclient.py", line 15, in import httpx ModuleNotFoundError: No module named 'httpx'

Expected behavior It seems that dependencies that are still in version 0.x.y should not be used openly, which means their api is still changing frequently.

Environment (please complete the following information):

OS:Linux

Python version: 3.8

Schemathesis version: 3.17.2

Additional context looks related to this https://github.com/encode/starlette/pull/1376

It looks like this issue is not fully resolved. https://github.com/schemathesis/schemathesis/issues/1417
Type: Bug Status: Review Needed
opened by Bidaya0 13
[FEATURE] A way to refresh auth
Is your feature request related to a problem? Please describe. Currently, if the auth token expires during the test run, there is no built-in way to refresh it. It will be nice to have some way to register a function that will refresh auth. And maybe some refresh interval

Describe the solution you'd like A decorator that will register such a function that will return headers required for authorization

CLI implementation notes

Here are my thoughts about implementing this feature.

It might look like this for the end-user - similar to other hooks:

import schemathesis @schemathesis.register_auth_provider(refresh_interval=600) # in seconds def auth_provider(auth: AuthStorage): response = requests.post(...) token = response.json()["access_token"] ... # Set the auth

This decorator may store the function & refresh period in some global variable (it is global for a run anyway) that will be available for the Schemathesis internals.

Then whenever Schemathesis makes a network call (or WSGI / ASGI calls) we check whether we need to call this function (depending on the time since the last auth function call + some leeway) and call it on some longer-lived storage for auth. E.g. we might need to put the auth into query params or into headers (usually it is the case). This might require some small object that will allow the user to set the auth according to their needs. Then we can properly set the auth in those network calls. It might be something like:

# Schemathesis internals class AuthStorage: ... # attrs for storing query / header values def set_query(self, name: str, value: str) -> None self.query[name] = value ... # similarly for other auth types ... # End-user code @schemathesis.register_auth_provider(...) def set_auth(auth: AuthStorage): ... # get the token auth.set_query("api_token", token) # Adjust network calls def _network_test(case, ..., auth: Optional[AuthStorage]): if auth is not None: set_auth(case, auth) # Calls the auth provider function if needed ... # make the call

This storage lifetime is bound to the runner, so it could be initialized somewhere here. Also, I think some parts of the code above surely could be factored out into something more isolated.

Keep in mind, that it requires synchronization as it could be accessed from multiple threads (e.g. lock over accessing the auth)

Python test implementation notes

It might be similar in Python tests. E.g., the end-user registers some auth provider on the schema object or globally:

schema = ... # Creates a pytest fixture that provides a single `set_auth` method. @schema.register_auth_provider(refresh_interval=600) def auth(auth_storage): ... @schema.parametrize() def test_api(case, auth): auth.set_auth(case) # Does all the stuff under the hood
Priority: High Type: Feature Hacktoberfest Difficulty: Medium
opened by Stranger6667 13
Response code range not supported
Thanks for adding check of response code!

Response codes may contain a range of code in the format XXX https://swagger.io/specification/#responsesObject

My output

if errors: > raise AssertionError(*errors) E AssertionError: Received a response with a status code, which is not defined in the schema: 404 E E Declared status codes: 4XX, 200
Priority: High Type: Feature
opened by MissiaL 13
[Questions] Access to data created by the schemathesis

Hi, I would like to access the data created by schemathesis. Currently, I run this command to test the API. I see the data logged by the API.

schemathesis run -D negative http://localhost:9990/openapi.json -E "/cumulative_proba"

I would love to see the data generated and use it to make my own list of specific cases, where I know the responses. I see some information in the example and Unicode folder under .hypothesis but can't decode them to the raw input values that is sent to API.

Please let me know, Is there any way to get the data or generate like 100 examples, that I can see the exact values of the data generated.

thank you so much

opened by pdubey2018 12
[BUG] st --report, generated report on app.schemathesis.io doesn't show errors
Checklist

[x] I checked the FAQ section of the documentation [x] I looked for similar issues in the issue tracker

Describe the bug I run st run -E /api/invalid https://example.schemathesis.io/openapi.json --report

It generated report at https://app.schemathesis.io/r/SPJFZ/, but the report doesn't show any errors, despite that this run generated one error:

____________________________________________________________ POST /api/invalid _______________ InvalidSchema: Invalid type name

To Reproduce See Describe the bug

Expected behavior Report at https://app.schemathesis.io/r/SPJFZ/ should diplay the above error.

Environment (please complete the following information):

OS: Windows 10

Python version: 3.10.5

Schemathesis version: 3.17.5

Spec version: Open API 3.0.2

Type: Bug Status: Review Needed
opened by robert-dzikowski 1
[FEATURE] CLI option for queries

Describe the solution you'd like I would like to have the option of providing a query value in the CLI to make sure all the tests I perform use this specific value. This query option should function similarly to the --header option that Schemathesis currently has.

Describe alternatives you've considered I have the option of enforcing the values by using an enum instead of an example, and by setting the query to REQUIRED. This however would tell readers of the schema that this value is always used, which is not how the API actually works. I do not want a schema that is just for testing, I want to be able to provide the schema to consumers of the API with the correct context.
Status: Review Needed Type: Feature

opened by hoog1511 1

[BUG] starlette 0.21.0 uses httpx testclient

Checklist

[x] I checked the FAQ section of the documentation
[x] I looked for similar issues in the issue tracker

Describe the bug starlette==0.21.0 updated to use httpx for the testclient instead of requests.

In particular starlette.testclient.TestClient now inherits httpx.Client.

To Reproduce Use Case.call_asgi(...) in a unit test.

Error:

  File "/mnt/win/work/nightcrawler/tests/schema/lookup/test_txt.py", line 22, in test_txt
    response = case.call_asgi(headers={'Authorization': 'Bearer supersecret'})
  File "/mnt/win/work/nightcrawler/.pyenv/lib/python3.10/site-packages/schemathesis/models.py", line 397, in call_asgi
    return self.call(base_url=base_url, session=client, headers=headers, **kwargs)
  File "/mnt/win/work/nightcrawler/.pyenv/lib/python3.10/site-packages/schemathesis/models.py", line 327, in call
    response = session.request(**data)  # type: ignore
  File "/mnt/win/work/nightcrawler/.pyenv/lib/python3.10/site-packages/starlette/testclient.py", line 448, in request
    return super().request(
  File "/mnt/win/work/nightcrawler/.pyenv/lib/python3.10/site-packages/httpx/_client.py", line 803, in request
    request = self.build_request(
  File "/mnt/win/work/nightcrawler/.pyenv/lib/python3.10/site-packages/httpx/_client.py", line 346, in build_request
    headers = self._merge_headers(headers)
  File "/mnt/win/work/nightcrawler/.pyenv/lib/python3.10/site-packages/httpx/_client.py", line 413, in _merge_headers
    merged_headers.update(headers)
  File "/mnt/win/work/nightcrawler/.pyenv/lib/python3.10/site-packages/httpx/_models.py", line 199, in update
    headers = Headers(headers)
  File "/mnt/win/work/nightcrawler/.pyenv/lib/python3.10/site-packages/httpx/_models.py", line 79, in __init__
    self._list = [
  File "/mnt/win/work/nightcrawler/.pyenv/lib/python3.10/site-packages/httpx/_models.py", line 85, in <listcomp>
    for k, v in headers
ValueError: too many values to unpack (expected 2)
Falsifying example: test_txt(
    case=Case(path_parameters={'domain': 'A.ac'}, headers={'Authorization': 'Bearer '}, query={}),
    self=<tests.schema.lookup.test_txt.TestTXTSchema testMethod=test_txt>,
)

Additional context I forced the headers to be a dict instead of a requests.structures.CaseInsensitiveDict and that furthered the state, but it next up failed with:

  File "/mnt/win/work/nightcrawler/tests/schema/lookup/test_txt.py", line 23, in test_txt
    case.validate_response(response)
  File "/mnt/win/work/nightcrawler/.pyenv/lib/python3.10/site-packages/schemathesis/models.py", line 424, in validate_response
    copied_response = copy_response(response)
  File "/mnt/win/work/nightcrawler/.pyenv/lib/python3.10/site-packages/schemathesis/utils.py", line 279, in copy_response
    response.freeze()
AttributeError: 'Response' object has no attribute 'freeze'
Falsifying example: test_txt(
    case=Case(path_parameters={'domain': 'A.ac'}, headers={'Authorization': 'Bearer '}, query={}),
    self=<tests.schema.lookup.test_txt.TestTXTSchema testMethod=test_txt>,
)

This seems to be caused by now getting a httpx.Response back instead of a requests.Response, and these objects are very different.

Type: Bug Status: Review Needed

opened by grigi 6

[BUG] Stateful testing restarts too soon and API is not explored
Checklist

[x] I checked the FAQ section of the documentation

[x] I looked for similar issues in the issue tracker

Describe the bug Stateful testing seems a really powerful for automating the API exploration, but it's difficult to make it work. It just stops too soon and filters lot of data. I'll attacch a simple case where i get a subset of the schema, configured with links, and run it.

I run the test with pytest -s --log-cli-level=INFO --junitxml=report.xml -v test.py so the output is shown as the tests goes, as you can see the machine is restarted a lot of times, and the actual examples wich are tried are just a few between each run.

Test Output
INFO root:state-machine-ad-hoc-service.py:55 RESTART
INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:63 PUT /api-portale/api/v1/organizations/{organization_id} INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:63 POST /api-portale/api/v1/contacts INFO root:state-machine-ad-hoc-service.py:63 POST /api-portale/api/v1/contacts INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:63 GET /api-portale/api/v1/contacts/{contact_id} INFO root:state-machine-ad-hoc-service.py:63 POST /api-portale/api/v1/organizations INFO root:state-machine-ad-hoc-service.py:63 POST /api-portale/api/v1/contacts INFO root:state-machine-ad-hoc-service.py:63 PUT /api-portale/api/v1/organizations/{organization_id} INFO root:state-machine-ad-hoc-service.py:63 POST /api-portale/api/v1/organizations INFO root:state-machine-ad-hoc-service.py:55 RESTART

To Reproduce

I've attacched the openapi yaml, this is the test code, note that rising up the max_examples and stateful_step_count doesn't improve the behaviour.

import schemathesis import pytest from hypothesis.stateful import initialize, run_state_machine_as_test from hypothesis import settings, reproduce_failure, Phase, Verbosity, HealthCheck from schemathesis.models import Case from schemathesis.utils import NOT_SET import logging logger = logging.getLogger(__file__) logger.setLevel(logging.INFO) path = "schema-path.yaml" base_url = "http://localhost:8080/api-portale/api/v1/" schema = schemathesis.from_path( path, base_url = base_url ) auth_headers = { 'X-Cart-username': 'gestore' } OrganizationWorkflow = schema.clone(operation_id=['addContact', 'getContact', 'createOrganization', 'updateOrganization']).as_state_machine() class SimpleWorkflow(OrganizationWorkflow): def setup(self): logging.info("RESTART\n\n") #end def get_call_kwargs(self, case): return {"headers": auth_headers } #end def before_call(self, case : Case): logging.info(case.operation.verbose_name) #end def validate_response(self, response, case): case.validate_response(response, checks=(schemathesis.checks.status_code_conformance,)) #end #end test_case_1 = SimpleWorkflow.TestCase

Expected behavior The full machine gets explored by schemathesis. This is a simple case, but rarely it can follow the link createOrganization -> updateOrganization because with these few tries it cannot build randomly a valid Organization.

Furthermore, when I raise the max_examples or the stateful_step_count, the tests stops with: hypothesis.errors.FailedHealthCheck: It looks like your strategy is filtering out a lot of data.

Nothing really changes if I suprress this healthCheck, the state machine still stops too soon.

openapi.yaml.txt
Type: Bug Status: Review Needed
opened by fscarlato 1
[BUG] Schemathesis Can't handle a simple recursive reference:
Checklist

[x] I checked the FAQ section of the documentation

[x] I looked for similar issues in the issue tracker

I'm still getting: hypothesis.errors.Unsatisfiable: Currently, Schemathesis can't generate data for this operation due to recursive references in the operation definition. See more information in this issue - https://github.com/schemathesis/schemathesis/issues/947

And the schema is a simple Parent-Child relationship:

CategoryChildCreate: required: - category_id - category_label type: "object" properties: category_id: $ref: "#/components/schemas/CategoryId" category_label: type: "string" description: type: string maxLength: 255 image: type: "string" format: "base64" children: type: array items: $ref: "#/components/schemas/CategoryChildCreate"

If this is a recursion level problem, why it isn't solved by defining a maximum depth? Can this maximum depth be specified from the command line?
Type: Bug Status: Review Needed
opened by fscarlato 0
[QUESTION] How to authenticate with http-only cookie ?

Hello,

I hope you don't mind this question.

I am using FastAPI, and my authentification consist in a http-only cookie being set. Usually sharing a common TestClient is enough to keep the cookie set between requests. But I can't find a way to share a TestClient for the auth request and the case call request.

I need a context for the TestClient too (with TestClient(my_app) as session:), as I need to fire the startup and shutdown event for the database connection.

Is there a way to share a common session between the auth and the case.call ?

Thanks for your time
Type: Documentation

opened by triplem161 1

Releases(v3.17.5)

v3.17.5(Nov 8, 2022)
Added

Python 3.11 support. #1632

Fixed

Allow Werkzeug<=2.2.2. #1631

Source code(tar.gz)
Source code(zip)
v3.17.4(Oct 19, 2022)
Fixed

Appending an extra slash to the / path. #1625

Source code(tar.gz)
Source code(zip)
v3.17.3(Oct 10, 2022)
Fixed

Missing httpx dependency. #1614

Source code(tar.gz)
Source code(zip)
v3.17.2(Aug 27, 2022)
:bug: Fixed

Insufficient timeout for report uploads.

Source code(tar.gz)
Source code(zip)
v3.17.1(Aug 19, 2022)

This release adds support for newer requests versions.
Source code(tar.gz)
Source code(zip)
v3.17.0(Aug 17, 2022)
From this release, you can simultaneously run positive & negative tests via CLI by passing -D all

:rocket: Added

Support for exception groups in newer Hypothesis versions. #1592

A way to generate negative and positive test cases within the same CLI run via -D all.

:bug: Fixed

Allow creating APIs in Schemathesis.io by name when the schema is passed as a file.

Properly trim tracebacks on Hypothesis>=6.54.0.

Skipping negative tests when they should not be skipped.

:wrench: Changed

pytest: Generate positive & negative within the same test node.

CLI: Warning if there are too many HTTP 403 API responses.

Runner: BeforeExecution.data_generation_method and AfterExecution.data_generation_method changed to lists of DataGenerationMethod as the same test may contain data coming from different data generation methods.

Source code(tar.gz)
Source code(zip)
v3.16.5(Aug 11, 2022)
:bug: Fixed

CLI: Hanging on CTRL-C when --report is enabled.

Internal error when GraphQL schema has its root types renamed. #1591

Source code(tar.gz)
Source code(zip)
v3.16.4(Aug 8, 2022)
:wrench: Changed

Suggest using --wait-for-schema if API schema is not available.

Source code(tar.gz)
Source code(zip)
v3.16.3(Aug 8, 2022)
A handful of new usability improvements!

:rocket: New

CLI: --max-failures=N option to exit after first N failures or errors. #1580

CLI: --wait-for-schema=N option to automatically retry schema loading for N seconds. #1582

CLI: Display old and new payloads in st replay when the -v option is passed. #1584

:bug: Fixed

Internal error on generating negative tests for query parameters with explode: true.

Source code(tar.gz)
Source code(zip)
v3.16.2(Aug 5, 2022)
More improvements on user experience!

From now Schemathesis will warn you if you specify an invalid API location and all responses are 404

WARNING: All API responses have a 404 status code. Did you specify the proper API location?

:rocket: New

after_load_schema hook, which is designed for modifying the loaded API schema before running tests. For example, you can use it to add Open API links to your schema via schema.add_link

New utf8_bom fixup. It helps to mitigate JSON decoding errors inside the response_schema_conformance check when the payload contains BOM. #1563

CLI: Warning if all responses are 404

:bug: Fixed

docs: Description of -v or --verbosity option for CLI.

:wrench: Changed

Execute before_call / after_call hooks inside the call_* methods. It makes them available for the pytest integration.

Source code(tar.gz)
Source code(zip)
v3.16.1(Jul 29, 2022)
A little patch release that improves user experience!

From now Schemathesis will warn you if you forgot to specify proper API credentials

WARNING: Most of the responses from `GET /api/secret/` have a 401 status code. Did you specify proper API credentials?

Also, we're extending SaaS test reports a bit! Schemathesis will send us information about what CLI options you use, so we can provide you with suggestions on how you can do more with CLI! Don't worry, all the data is anonymized and there is no sensitive info.

What's Changed

Add SCHEMATHESIS_BASE_URL environment variable for specifying --base-url in CLI

Collect anonymized CLI usage data in reports

Warning if the API returns too many HTTP 401

Source code(tar.gz)
Source code(zip)
v3.16.0(Jul 22, 2022)
Hello friends!

Schemathesis 3.16.0 brings a brand new GitHub integration and many readability improvements to the test results. You will have much less visual clutter reading Schemathesis reports, and the failures will be much more concise from now.

:octocat: GitHub Integration

Now you can use a native GitHub application to collect & display API fuzzing results directly in your PR! It makes Schemathesis much simpler to use in your GitHub Actions workflows and brings more Schemathesis.io features closer to you:

API Issues Dashboard: Debug issues faster with all the report details in a UI;

Additional checks: Get your API checked more thoroughly by extra conformance checks;

Grab your installation now!

After installation, you'll be redirected to SaaS to grab your access token, which you need to add to your workflow.

Depending on your application deployment, you'll need to add a new step to your GitHub Actions workflow. You can use this code sample to test your API in a pull request or run tests against a publicly resolvable API:

jobs: api-tests: runs-on: ubuntu-20.04 steps: - uses: schemathesis/[email protected] with: # API schema location schema: 'http://127.0.0.1:5001/openapi.json' # Set your token from secrets token: ${{ secrets.SCHEMATHESIS_TOKEN }}

You can find other deployment scenarios in our documentation

Then you'll get your report:

Read the full integration guide in our docs.

A few notes:

Report upload size has some limitations now. My guess is that 250kb should be enough for most cases - let me know if I am wrong;

Using reporting with public repositories is free. Private ones are available on the paid plan;

What is next?

This release doesn't support GitHub organizations. The support will be added soon as well as more features like:

Precise API Coverage: Per-keyword coverage achieved by the test suite;

Stateful Flows: See how stateful tests are exercising your API endpoints in flow diagrams;

Schema fixes: Fix your API schema immediately by applying an autogenerated patch.

Your feedback is important :heart:

Please, let us know what you think about the integration in this discussion.

:bug: Fixed

Compatibility with hypothesis >= 6.49 in #1538.

Support for Hypothesis' skipping tests when the generate phase is disabled.

Generating headers with schemas of array or object types.

Do not report optional headers missing.

:wrench: Changed

Pytest: Turn off schema validation by default. Pass validate_schema=True to the schema loader of your choice to keep the old behavior.

Code samples: Do not display irrelevant HTTP headers. It affects ones automatically set by requests when test cases go to the application under test.

CLI: Display the test environment metadata only if the -v option is present.

:fire: Removed

SaaS: Implicit reporting when the proper credentials were specified. From now on, every report upload requires the explicit --report CLI option.

CLI: Text representation of HTTP requests in the CLI output.

Source code(tar.gz)
Source code(zip)
v3.15.6(Jun 23, 2022)
:bug: Fixed

Do not discard dots (.) in OpenAPI expressions during parsing.

Source code(tar.gz)
Source code(zip)
v3.15.5(Jun 21, 2022)
:bug: Fixed

TypeError when using --auth-type=digest in CLI.

Source code(tar.gz)
Source code(zip)
v3.15.4(Jun 6, 2022)
This release makes Schemathesis work on schemas that do not comply with the Open API spec due to YAML specifics (missing quotes around keys that can be parsed as integers, booleans, etc)

Added

Support generating data for Open API request payloads with wildcard media types. #1526

Changed

Mark tests as skipped if there are no explicit examples and --hypothesis-phases=explicit is used. #1323

Parse all YAML mapping keys as strings, ignoring the YAML grammar rules. For example, on: true will be parsed as {"on": True} instead of {True: True}. Even though YAML does not restrict keys to strings, in the Open API and JSON Schema context, this restriction is implied because the underlying data model comes from JSON.

INTERNAL: Improve flexibility of event serialization.

INTERNAL: Store request / response history in SerializedCheck.

Source code(tar.gz)
Source code(zip)
v3.15.3(May 28, 2022)
Fixed

Deduplication of failures caused by malformed JSON payload. #1518

Do not re-raise InvalidArgument exception as InvalidSchema in non-Schemathesis tests. #1514

Source code(tar.gz)
Source code(zip)
v3.15.2(May 9, 2022)
This is a small bugfix release :)

:bug: Fixed

Avoid generating negative query samples that result in an empty query.

Editable installation via pip.

Source code(tar.gz)
Source code(zip)
v3.15.1(May 3, 2022)
This release adds a few improvements that simplify customization and fixes a few minor issues that appear when running in Docker.

:rocket: Added

OpenAPI: Expose APIOperation.get_security_requirements that returns a list of security requirements applied to the API operation

Attach originally failed checks to "grouped" exceptions.

:bug: Fixed

Internal error when Schemathesis doesn't have permission to create its hosts.toml file.

Do not show an internal Hypothesis warning multiple times when the Hypothesis database directory is not usable.

Do not print not relevant Hypothesis reports when run in CI.

Invalid verbose_name value in SerializedCase for GraphQL tests.

Source code(tar.gz)
Source code(zip)
v3.15.0(May 1, 2022)
Major improvements for GraphQL

This release introduces a bunch of nice improvements for GraphQL testing - now you can test mutations, extend Schemathesis with custom scalars, and run tests at least twice as fast as before!

Other things include cleaner pytest output, human-readable cassette payloads, reporting flaky failures, better negative testing, and much more! See below

:rocket: Added

GraphQL: Mutations supports. Schemathesis will generate random mutations by default from now on.

GraphQL: Support for registering strategies to generate custom scalars.

Custom auth support for schemas created via from_pytest_fixture.

:wrench: Changed

Do not encode payloads in cassettes as base64 by default. This change makes Schemathesis match the default Ruby's VCR behavior and leads to more human-readable cassettes. Use --cassette-preserve-exact-body-bytes to restore the old behavior. #1413

Bump hypothesis-graphql to 0.9.0.

Avoid simultaneous authentication requests inside auth providers when caching is enabled.

Reduce the verbosity of the pytest output. A few internal frames and the "Falsifying example" block are removed from the output.

Skip negative tests on API operations that are not possible to negate. #1463

Make it possible to generate negative tests if at least one parameter can be negated.

Treat flaky errors as failures and display a full report about the failure. #1081

Do not duplicate failing explicit examples in the HYPOTHESIS OUTPUT CLI output section. #881

:bug: Fixed

GraphQL: Semantically invalid queries without aliases.

GraphQL: Rare crashes on invalid schemas.

Internal error inside BaseOpenAPISchema.validate_response on requests>=2.27 when response body contains malformed JSON. #1485

schemathesis.from_pytest_fixture: Display each failure if Hypothesis found multiple of them.

🏎️ Performance

GraphQL: Over 2x improvement from internal optimizations.

Source code(tar.gz)
Source code(zip)
v3.14.2(Apr 21, 2022)
🎉 Schemathesis.io: schema-based API testing as a service. 🎉

This release adds support for Schemathesis.io - you can upload your test results and visualize them. Or, you can signup to a Pro account and use our servers to run enhanced tests against your API.

Feel free to join our Discord - we'd love to hear your feedback!

:rocket: CLI features

Support for auth customization & automatic refreshing. #966

Open API link name customization via the name argument to schema.add_link.

st as an alias to the schemathesis command line entrypoint.

st auth login / st auth logout to authenticate with Schemathesis.io.

X-Schemathesis-TestCaseId header to help to distinguish test cases on the application side. #1303

Support for comma separated lists in the --checks CLI option. #1373

Hypothesis Database configuration for CLI via the --hypothesis-database option. #1326

Make the SCHEMA CLI argument accept API slugs from Schemathesis.io.

:bug: Bug fixes

Using @schema.parametrize with test methods on pytest>=7.0.

Show the proper Hypothesis configuration in the CLI output. #1445

Missing source attribute in the Case.partial_deepcopy implementation. #1429

Duplicated failure message from content_type_conformance and response_schema_conformance checks when the checked response has no Content-Type header. #1394

Not copied case & response inside Case.validate_response.

Ignored pytest.mark decorators when they are applied before schema.parametrize if the schema is created via from_pytest_fixture. #1378

:wrench: Chores and Improvements

Enable Open API links traversal by default. To disable it, use --stateful=none.

Do not validate API schema by default. To enable it back, use --validate-schema=true.

Add the api_slug CLI argument to upload data to Schemathesis.io.

Show response status code on failing checks output in CLI.

Improve error message on malformed Open API path templates (like /foo}/). #1372

Improve error message on malformed media types that appear in the schema or in response headers. #1382

Relax dependencies on pyyaml and click.

Add --cassette-path that is going to replace --store-network-log. The old option is deprecated and will be removed in Schemathesis 4.0

P.S. These release notes contain entries from multiple releases, as they were not properly displayed on GitHub before
Source code(tar.gz)
Source code(zip)
v3.14.1(Apr 18, 2022)
What's Changed

fix: Compatibility with pytest>=7.0 by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1460

Full Changelog: https://github.com/schemathesis/schemathesis/compare/v3.14.0...v3.14.1
Source code(tar.gz)
Source code(zip)
v3.14.0(Apr 17, 2022)
What's Changed

chore: Validate requests kwargs to catch cases when the ASGI integration is used, but the proper ASGI client is not supplied by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1399

chore: Add st as an alias to schemathesis by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1400

chore: Enable Open API links traversal by default by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1401

chore: Do not validate API schema by default by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1402

feat: Add a command to authenticate with Schemathesis.io by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1403

fix(docs): Bump Sphinx version by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1406

chore: api_id CLI argument to upload data to Schemathesis.io by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1404

fix: Create auth file even if some parent dir does not exist by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1408

chore: Update pre-commit by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1416

chore: Deep-clone Case instances before passing to check functions by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1419

chore: Deep-clone Response instances before passing to check functions by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1420

chore: Pass custom formats to strategy factories by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1421

chore: Support for Hypothesis>=6.41.0 by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1426

chore: Ping SaaS on login by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1424

fix: Missing media_type in the Case.partial_deepcopy implementation. by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1428

refactor: Move data generation method choices into a separate variable by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1431

chore: Show response status code on failing checks output in CLI by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1433

fix: Missing source attribute in the Case.partial_deepcopy implementation by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1432

chore: Update CLI auth by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1430

feat: Open API link name customization by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1435

feat: X-Schemathesis-TestCaseId header by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1434

Update builds by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1436

fix: Duplicated failure message from content_type_conformance and response_schema_conformance checks by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1437

fix: Not copied case & response inside Case.validate_response by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1438

chore: Improve error message on malformed Open API path templates by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1440

fix: Ignored pytest.mark decorators when they are applied before schema.parametrize if the schema is created via from_pytest_fixture by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1441

feat: Support for comma separated lists in the --checks CLI option by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1442

Configure Hypothesis db by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1444

chore: Improve error messages for malformed media types by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1446

chore: Fail fast if Schemathesis.io responds with an error by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1447

chore: Add --cassette-path that is going to replace --store-network-log by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1448

fix: Compatibility with pytest-asyncio>=0.17.1 by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1453

test: Cleanup test code by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1455

feat: Make the SCHEMA CLI argument accept API slug from Schemathesis.io by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1456

test: Update Flask dependency by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1458

chore: Relax dependencies on pyyaml and click by @Stranger6667 in https://github.com/schemathesis/schemathesis/pull/1459

Full Changelog: https://github.com/schemathesis/schemathesis/compare/v3.13.3...v3.14.0
Source code(tar.gz)
Source code(zip)
v3.13.9(Apr 14, 2022)
:bug: Bug fixes

436100c Compatibility with pytest-asyncio>=0.17.1

:wrench: Chores and Improvements

bae14f2 Release 3.13.9

Source code(tar.gz)
Source code(zip)
v3.13.8(Apr 5, 2022)
:bug: Bug fixes

72698c0 Missing media_type in the Case.partial_deepcopy implementation.

:wrench: Chores and Improvements

f2be90d Release 3.13.8

Source code(tar.gz)
Source code(zip)
v3.13.7(Apr 2, 2022)
:wrench: Chores and Improvements

2b151c6 Release 3.13.7

a530b9e Support for Hypothesis>=6.41.0

Source code(tar.gz)
Source code(zip)
v3.13.6(Mar 31, 2022)
:wrench: Chores and Improvements

2848a2a Release 3.13.6

724b98b Deep-clone Response instances before passing to check functions

Source code(tar.gz)
Source code(zip)
v3.13.5(Mar 31, 2022)
:rocket: Features

4c645b5 Extra requests related options for the replay command

cf03191 APIOperation.iter_parameters helper to iterate over all parameters.

ebf9061 Support for readOnly and writeOnly Open API keywords

a3392a7 Support client-side certificates in CLI

2903305 New before_init_operation hook

f136a16 (internal) description attribute for all parsed parameters inside APIOperation.

f4576bd Schemathesis.io integration

3a730d0 New process_call_kwargs CLI hook

94ed201 New before_call and after_call CLI hooks.

36829e2 Add ExecutionEvent.is_terminal attribute that indicates whether an event is the last one in the stream

b8d4484 A way to stop the Schemathesis runner's event stream manually via events.stop()

c76285c GraphQL support in CLI

e922209 Case.data_generation_method attribute that provides the information of the underlying data generation method

9217358 Negative testing

1cb291b Additional context for each failure coming from the runner.

d3e1766 (runner) Add BeforeExecution.verbose_name & SerializedCase.verbose_name that reflect specification-specific API operation name

b78a164 Support for disabling ANSI color escape codes via the NO_COLOR environment variable or the --no-color CLI option

9ac1b75 More GraphQL loaders

36c0c87 Support for data_generation_methods and code_sample_style in all GraphQL loaders

22354a9 ASGI support for GraphQL schemas

0bf11c7 Support for app & base_url arguments for the from_pytest_fixture runner

fa0820f from_wsgi loader for GraphQL apps

517d5f2 before_generate_case hook

d0d743e Add event_type field to the debug output

c61dcee --debug-output-file CLI option to enable storing the underlying runner events in the JSON Lines format in a separate file for debugging purposes

006b8a4 Display failing response payload in the CLI output

4f94584 Add count_operations flag to runner.prepare

23665e2 Support for external examples

e291644 Support YAML serialization

ea64c29 --dry-run CLI option. When applied, Schemathesis won't send any data to the server and won't perform any response checks

9c13450 A helpful error message when an operation is not found during the direct schema access

7c528f4 Support data generation for text/plain media types with charset

e8bd16c Base URL for GraphQL schemas

f724df9 Run response_schema_conformance check on media types that are encoded with JSON (#927)

:bug: Bug fixes

639b7ab (docs) Bump Sphinx version

375f4f6 Use full_path in error messages in recoverable schema-level errors

f9ac0e2 Use the same correlation_id in BeforeExecution and AfterExecution events if the API schema contains an error that causes an InvalidSchema exception during test execution.

fe23f2e Properly handle error if Open API parameter doesn't have content or schema keywords

5370c80 (example) asyncpg==0.23.0 installs from wheels

3b518b9 (example) fix postgres version to current latest one

4fd42cb (example) mount main.py into container

92805d5 (example) recreate web app on failure

3208712 (example) bind web app ports to 127.0.0.1

89cda52 Generating illegal Unicode surrogates in queries

fe51d57 Not-escaped single quotes in generated Python code samples

ccb39b7 Improper handling of base_url in call_asgi, when the base URL has a non-empty base path

45a0eb1 Generating values not compliant with the ECMAScript regex syntax

380bcf4 Not raising an error when tests generated by schemas loaded with from_pytest_fixture match no API operations

b820385 Silently failing to detect numeric status codes when the schema contains a shared parameters key

24155c9 Ignored hooks defined on a schema instance when it is loaded via from_pytest_fixture

5ffba33 Respect the data_generation_methods config option defined on a schema instance when it is loaded via from_pytest_fixture

589be99 Handle KeyboardInterrupt that happens outside of the main test loop inside the runner

0ad3437 Unescaped quotes in generated Python code samples on some schemas

700af30 Generation of invalid headers in some cases

9577f0f (build) Unpinned Python version specifier in the pytest-legacy job

b765e8f Pass data_generation_method to GraphQLCase

120cbcf Unresolvable dependency due to incompatible changes in the new hypothesis-jsonschema release

cf63173 (example) Docker setup for example project

a80424b (example) Typo in example project's schema

df14316 Properly report all failures when custom checks are passed to case.validate_response

54cce04 Stop worker threads on failures with exit_first enabled

b1172a1 Rewrite not resolved remote references to local ones

167b8a9 Re-used referenced objects during inlining

471cebd Stop workers on interrupted event (#1239)

c8179b9 TypeError on case.call with bytes data on GraphQL schemas

6168c22 Internal error in make_case calls for GraphQL schemas.

0bec4b3 Preserve non-body parameter types in requests during Open API runtime expression evaluation

9395a74 TypeError during negative testing on Open API schemas with parameters that have non-default style value

424dfa2 KeyError when the response_schema_conformance check is executed against responses without schema definition

703430c When EventStream.stop is called, the next event always is the last one

4dbf750 Incorrect deduplication applied to response schema conformance failures that happen to have the same failing validator but different input values

cd91522 Compatibility with newer Hypothesis versions

f9fa796 Open API style & explode for parameters derived from security definitions

6e72c3f Apply the Open API's style & explode keywords to explicit examples

586ba77 Disable filtering optimization for headers when there are keywords other than type

31d322f Too much filtering in headers that have schemas with the pattern keyword

288bf17 Invalid multipart payload generated for unusual schemas for the multipart/form-data media type

1b8da90 Ignored explicit Content-Type override in Case.as_requests_kwargs

bb6fe04 Internal error on unusual schemas for the multipart/form-data media type

acd2ac4 Ignored Open API specific keywords & types in schemas with deeply nested references

dbd4cfd Ignored $ref keyword in schemas with deeply nested references.

46ec17d Missing support for date string format

5841dda UnicodeEncodeError when sending application/octet-stream payloads that has no format: binary in their schemas

5098a1d Bump minimum hypothesis-graphql version to 0.4.1

f8f571b (windows) UnicodeDecodeError during schema loading via the from_path loader if it contains certain Unicode symbols

f8d90b9 Using parametrized pytest fixtures with the from_pytest_fixture loader

d558063 Return type of make_case for GraphQL schemas

dbd6720 Silently ignoring some incorrect usages of @schema.given`

90198ec Missing @schema.given implementation for schemas created via the from_pytest_fixture loader

c7cf6d8 (deps) Do not use importlib-metadata==3.8 in dependencies

8d665f7 Encoding for response payloads displayed in the CLI output

5d25d55 Output types for evaluation results of $response.body and $request.body runtime expressions

d6eb7ac Missing body parameters during Open API links processing in CLI

a13ffce Displaying wrong headers in the FAILURES block of the CLI output

d77c9a3 UnicodeDecodeError on cURL command generation

1043c40 Wrong test results in some cases when the tested schema contains a media type that Schemathesis doesn't know how to work with

4df34f9 Percent-encode . and .. strings in path parameters

5355c13 loosen importlib-metadata version constraint

d97019d Prevent a small terminal width causing a crash when printing percentage

3224601 Return a default terminal size to prevent crashes on systems with zero-width terminals (some CI/CD servers)

11cb8b9 Docker tags for Buster-based images

079f645 Prevent a TypeError when additional_checks is a list

07c52a5 Programmatic addition of Open API links via add_link

09fa0c2 Generating stateful tests, with common parameters behind a reference

c3a787b Flaky Hypothesis error during explicit examples generation

63f4dea Processing parameters common for multiple API operations if they are behind a reference

3c0883c Import in tests

9e74ec6 Generating incomplete explicit examples

e1a54c0 CLI execution stops on errors during example generation

b8f42df Handling of API operations that contain reserved characters in their paths

f1566f7 CLI crash on schemas with operation names longer than the current terminal width

7ab5c86 Invalid types in x-examples

5ca14bf TypeError during nullable array parameter serialization

4774398 Collecting nullable parameters that are behind references

27bce30 Explicit examples for multipart forms

d578519 A possibility to override APIStateMachine.step

d25efba Unsatisfiable error in stateful testing

afa3daa Skip explicit examples generation if this phase is disabled via config

a5d3c65 Do not suggest to disable schema validation if it is already disabled

9b93056 Apply body-level hooks

fad2c19 Internal error if filling missing explicit examples led to Unsatisfiable errors

bce08c4 --exitfirst CLI option trims the progress bar output when a failure occurs

b63a578 Use typing instead of typing_extensions

152ccb8 Excessive reference inlining in large schemas

44c6549 CLI failure if the tested operation is GET and has payload examples (#930)

530d5b6 Distinguishing between no payload requests & null payload (#929)

6f6cff0 Allow sending null as request payload if the schema expects it (#922)

:wrench: Chores and Improvements

fbe5e04 Release 3.13.5

542e7a2 Deep-clone Case instances before passing to check functions

f4ead9e Release 3.13.4

b552584 Support Werkzeug>=2.1.0

7a5a8fe Pin aevea/action-kaniko

7ca318e Validate requests kwargs to catch cases when the ASGI integration is used, but the proper ASGI client is not supplied

2fee032 Release 3.13.3

c810f3e Update pre-commit

38a582e Release 3.13.2

e933553 Use Schemathesis default User-Agent when communicating with SaaS

a84fe70 Release 3.13.1

24442ba Release 3.13.0

aebe61a Always show traceback for errors in Schemathesis.io integration

2815f96 Update Schemathesis.io integration

2eaea40 Release 3.12.3

a395fcd Disable the duplicate-code pylint lint

57d9c74 Release 3.12.2

d146ea6 Release 3.12.1

4e2d446 Release 3.12.0

c337784 Update typing-extensions requirement to >=3.7,<5

26d7905 Update pre-commit

23dd6c8 Drop Python 3.6 support

e927c51 Release 3.11.7

7a10fbf Support Python 3.10

48ee34c Update dev dependencies

02935ff Upgrade pre-commit

ac453f2 Release 3.11.6

accb4da Release 3.11.5

dded3b1 Generate tests for API operations with the HTTP TRACE method on Open API 2.0

d56afa2 Send response.elapsed to Schemathesis.io

0e6f204 Release 3.11.4

7621f40 Minor changes to the Schemathesis.io integration

bee03a4 Add AfterExecution.data_generation_method

d97a102 Release 3.11.3

19ee5f9 Release 3.11.2

8026971 Use str for subtest context

5250ea4 Raise an error if a test function wrapped with schema.parametrize matches no API operations

8fd0f41 Use verbose_name & data_generation_method parameters to subtest context instead of path & method

69c9e96 (test) Disable deadline for some flaky tests

87ac234 update release notary

9a7c239 Release 3.11.1

b980cfa Update click and pyyaml versions, update tests to reflect that click

5a09a03 Release 3.11

bd4b9c5 Use case-insensitive dictionary for Case.headers

e966976 Bound validators caching during negative testing

94b642d Bump default_language_version for pre-commit

b3d0faa Minor changes for Schemathesis.io integration

d31b017 GraphQL schema loaders now accept single DataGenerationMethod instances for the data_generation_methods argument

f485bcf Add AfterExecution.verbose_name

94ee105 Update the bug report template

56bc46b Fix typing for BeforeExecution.from_operation

11471d8 Store data_generation_method in BeforeExecution

78dd617 Improve reporting of jsonschema errors which are caused by non-string object keys

7e71b00 Update pre-commit

b1fc60e Show cURL code samples by default instead of Python

6987f7b Move data generation method to strategy factory mapping to the module level

8d49a88 DataGenerationMethod.all shortcut to get all possible enum variants.

c7527af Release 3.10.0

3f04815 Improve error messages when the loaded API schema is not in JSON or YAML

576dba4 Improve error messages for Schemathesis.io

84473eb Add an extra message in case of errors with Schemathesis.io

669755c Show Schemathesis.io test report URL

5780fd2 Open API schema loaders now accept single DataGenerationMethod instances for the data_generation_methods argument

608216f Relax colorama and typing-extensions

e14df88 Add tool.poetry.urls section

829fcb9 Reduce Docker images size

b6d75cd Add timeouts when loading external schema components or external examples

2418188 Do not use header filtration when not necessary

affc9a8 Minor refactoring of reference inlining

e75345c Change OpenAPI20CompositeBody.definition type to List[OpenAPI20Parameter]

fd86bad Pin werkzeug to >=0.16.0

f196825 Remove unused parameter_cls attributes from BaseOpenAPISchema

72a946c Remove duplicated keyword

3b8ee7f Release 3.9.7

7c50df3 Check non-string response status codes when Open API links are collected

dd247d2 Release 3.9.6

d4560b4 Release 3.9.5

4700a6c Release 3.9.4

e16d42a Remove unused imports

5e18c2b Release 3.9.3

bbcbd92 Release 3.9.2

9a6cc1d Return response from case.call_and_validate

be2f772 Release 3.9.1

7ed2306 Convert all FailureContext class attributes to instance attributes

b6b3c59 Add SerializedCase.cookies

01064a7 Add API schema to the Initialized event

6fad51c ExecutionEvent.asdict adds the event_type field which is the event class name

6256be0 Release 3.9.0

a9fbccd Add EventStream.finish method

250c20e Avoid ``pytest`warnings when internal Schemathesis classes are in the test module scope

7d8c0be Release 3.8.0

622a1c9 Improve APIOperation.make_case behavior

4d08fd2 Bump hypothesis-jsonschema version

37b1000 Python values of True, False and None are converted to their JSON equivalents when generated for path parameters or query

0b90566 Raise UsageError if schema.parametrize or schema.given are applied to the same function more than once

d30e9c0 Release 3.7.8

a6112dc (deps) Update dependencies in /example

8411718 Release 3.7.7

0daefff Release 3.7.6

909205d Release 3.7.5

996bc8e Release 3.7.4

bf894c8 Make SerializedCase.path_template return path templates as they are in the schema, without base path

6dcce2e Release 3.7.3

04b24b4 Release 3.7.2

bd066f0 Add SerializedCase.media_type attribute

02df4dc Release 3.7.1

5593a81 Various internal changes

bc3c08b Release 3.7.0

06ecf49 Extend the default test duration deadline to 15 seconds

6dd1586 Set default timeout for network requests to 10 seconds.

7f26e1f Use different exception classes for not_a_server_error and status_code_conformance checks.

af7cde3 Release 3.6.11

c3339d7 Release 3.6.10

d7a0e9e Bump hypothesis-jsonschema version

844655a Explicitly add colorama to dependencies

b1d45b8 Fix hypothesis-jsonschema version constraint

86b396b Release 3.6.9

ce92e34 Release 3.6.8

0779b97 Relax dependency on starlette to >=0.13,<1

9297654 Release 3.6.7

8ca014c Update pre-commit hooks

59840e1 Call shutdown on all handlers before CLI exit

a7d2bf1 Release 3.6.6

667d8b7 Improve error message for failing Hypothesis deadline healthcheck in CLI

b8bed68 Release 3.6.5

e362292 Generate valid header values for Bearer auth by construction rather than by filtering

d414477 (deps) bump urllib3 from 1.25.6 to 1.25.8 in /example

9d9255b Release 3.6.4

e85a13d Bump minimum hypothesis-graphql version to 0.5.0

aa632df Remove obsolete pylint and mypy tox envs

1050b81 Release 3.6.3

95ec3ef (deps) bump asyncpg from 0.19.0 to 0.21.0 in /example

d786d33 Release 3.6.2

6d7ddad Remove unused imports

24212a9 Release 3.6.1

3b10a2b (deps) Bump Sphinx version

60ed847 Release 3.6.0

301dfae Fix tox warning

1f69add Deprecate schemathesis.runner.prepare

04f22e8 Initial support for GraphQL schemas in Schemathesis runner

a2d9beb Loaders unification

7586416 Improve error handling on from_path incorrect usage

ced53fc Validate schema paths in ASGI / WSGI loaders

08ba562 Validate GraphQL loaders in Schemathesis runner

a25b91c Fix invalid import

70754a5 Add blank impl for GraphQLSchema.get_stateful_tests

5307af4 Unify test IDs

3355ce9 Use different DB entries for each API operation in CLI

d689d77 Update GraphQL loader

e69a2ea Do not show overly verbose raw schemas in Hypothesis output for failed GraphQL tests

c46d5b8 Generate separate tests for each field for GraphQL schemas

7d87261 Release 3.5.3

2825d3e (deps) bump pyyaml from 5.1.2 to 5.4 in /example

43419ba Release 3.5.2

93b6800 Prefix worker thread names with schemathesis_

89f059d Release 3.5.1

99013f9 Release 3.5.0

6e17291 Release 3.4.1

2206999 Release 3.4.0

f32c681 Do not store absent request/response payloads as empty strings in VCR cassettes

9524c08 (deps) bump jinja2 from 2.10.3 to 2.11.3 in /example

66eb76e Release 3.3.1

2356141 (internal) Make SerializedCheck.example and Check.example not optional

3afd29c Release 3.3.0

46c1162 A way to control which code sample style to use

89384ca Add extra information to events, emitted by the Schemathesis runner.

fdeb037 Release 3.2.2

f0e0b0a Support Hypothesis 6

caa2c9f Release 3.2.1

6db2490 Release 3.2.0

7bae578 Sort keys when serializing to JSON in hash calculation

13810ff Release 3.1.3

069f5cd Release 3.1.2

d97ddbe Release 3.1.1

73b51d8 (deps) bump aiohttp from 3.6.2 to 3.7.4 in /example

3c07cac Release 3.1.0

9f3cd95 Support the latest cryptography version in Docker images

e54b764 Release 3.0.9

59091d1 Release 3.0.8

d6e7c2f Release 3.0.7

c32eda0 Release 3.0.6

29ea09c Release 3.0.5

8cee5de Bump mypy to 0.800

6514888 Release 3.0.4

53ab237 Use the same ref resolving approach in get_operation_by_id

f436963 Release 3.0.3

67c1a0f Release 3.0.2

dcab4bc Release 3.0.1

16daedb Release 3.0.0

1d793e3 Continue testing after schema parsing errors if possible

418b56e Update pre-commit hooks

882d12f Export from_aiohttp in the public API

21072ae Do not export Stateful in the public API

de8fca0 Do not export init_default_strategies in the public API

40b1860 Catch extra Hypothesis output

82b3094 A better error message for invalid regex syntax

8401998 Improve error messages for serialization errors (#998)

11b28c8 Fix a typo

a61453e Allow the user in docker image to write to /app

079da46 Do not require --base-url for schemas loaded via a file when dry run is enabled

5d9e82e Deduplicate collected errors that don't come from failed checks

441f825 Display all non-check failures in CLI tests

a543240 Shorter error messages on invalid schemas in some cases

1c2c91f Allow running before_add_examples hooks even if default examples generation failed

ffdcee6 Fix passing the all option to the test server

e85d4e0 Use Python 3.9 for Schemathesis's Docker image

000c9c7 Remove deprecated seed-isort-config pre-commit hook

bec459e Update pre-commit to v3.4.0

f4a7fd7 Disallow using --auth together with --header that sets the Authorization header.

7b7e6ea Enforce path parameters to be required if schema validation is disabled

4f5ecef Use keyword arguments for get_case_strategy call

7266446 Unify the test server

:package: Other

ea4855b Fix grammar

6603099 Update UTM labels in links

909a661 Update image links

21622f5 Add a link to SaaS

b0ec4de Update SaaS tutorial

b378064 Disable deadline for test_read_only

bb03393 Update README

bdfb0f1 Update Changelog

536fb28 Update README

8a556ee Increase the value of max_examples to reduce test flakiness

8e2eb7c Install setuptools explicitly

ad9a502 Clarify readme

18da638 Fix RST syntax

279e32b Increase max_examples in test_hidden_failure to decrease the number of flaky test runs

169c5b4 Add more Docker usage examples

70e3489 Use the most recent poetry version

3316723 Add suppress_health_check to test_pet to avoid flakiness

2482fda Remove test that relies on old Hypothesis behavior

9adce81 Add links to Deriving Semantics-Aware Fuzzers from Web API Schemas

67577ff Update LICENSE

562d3fa Use longer timeouts for test_app on Windows

d9300c0 Update changelog

85e3932 Update changelog

ed3762b Add a note about filters being treated as regular expressions

5630bd5 Add missing section to changelog.rst

aa6ded7 Update FAQ

d377303 Update changelog

239315a Generate tests on demand when multiple workers are used during CLI runs

b804378 Add a recipe for per-route timeouts

eea6e61 Remove MutationResult.is_success and MutationResult.is_failure helpers

e8823d8 Cache split_schema results for Open API schemas during negative testing

63f8323 Cache get_full_path for Open API schemas

905a113 Properly cache rewritten_components

c34b332 (oas3) Inline only components/schemas before passing schemas to hypothesis-jsonschema

0f4fff6 Use sets during negative testing data generation

b39cc91 Fix RST syntax

722ed8a Avoid copy.deepcopy calls in some cases

d8bbc71 Dramatically improve CLI startup performance

e0403bd Cache rewritten schema components for Open API schemas

1a07106 Avoid extra resolve_all call in some cases

ebaa7e8 Avoid calling deepcopy in some cases

3019796 Fix flaky test

5684c82 Remove obsolete hook mention

4ae7f01 Clarify FAQ entry

c274ade Add a new FAQ entry

c55fde2 Update Docker docs

60cb07e Add missing changelog entry

b0d5177 Adjust changelog

ec91777 Document CLI options & add a recipe for HTTP(S) proxy

8c50c3b Clarify communication language

583d344 Disable deadline on flaky test

5ce74cc Use poetry<1.1.9

7530ff0 Minor changelog correction

212ef23 Update Service docs

e2c7a5f Add a link to a walkthrough

97b47b5 Add tests for schema mutations

822b084 Add extra tests for is_valid_header

5980df1 Add missing changelog entry

e5178f3 Disable some health checks for slow tests

d7e90c0 Add more tests

3080028 Fix typo in help message for --header

975fb44 Fix Service tests on Windows

4a217d8 Use result.verbose_name unconditionally in display_subsection

b0ca226 Use cls in class methods instead of concrete classes

47813ef Improve the way internal events are handled in CLI

3d5d093 Update GraphQL docs

9fcad07 Fix RST syntax

12b0049 Disable deadline for occasionally slow tests

5e46a7d Simplify examples serialization

00d3796 Improve schemas to avoid unneeded filtering

bd0cc79 Reduce the amount of filtering needed to generate valid headers and cookies

c5e5770 Clarify scope for the add_case hook

bb7a8a3 Remove unused fixtures

efd1969 Fix corpus tests

42a2113 Fix insufficient validation in test apps

a371c3b Update notes on stateful testing in CLI

88818f6 Disable deadline in flaky tests

4e2114c Avoid flakiness in test_path_parameters_encoding

9382d71 Pin the sphinx_rtd_theme version

a2b0de3 Fix argument order in hook examples

9afd280 Fix example for APIStateMachine.get_call_kwargs

0f50b02 (runner) Move fixups installation out of Schemathesis runner

5b9d757 (runner) Convert Hypothesis configuration options earlier

4652ec5 (test) Schema loading tests

6d77911 (test) Restructure test app

beca586 (test) Group tests for code samples generation

2281c04 (test) Remove unused imports

c607b1b Remove low-level data generation tests

2c5437e Add Effective API schemas testing to the list of additional content

309001b Move schemathesis.loaders to schemathesis.specs.openapi.loaders

ec54781 (test) Reuse fixtures in tests for pytest_subtests integration

bf60043 use functioning fixup name in example

f293c32 Update GraphQL schema in CONTRIBUTING

86830b1 Add the from_wsgi loader to the public API docs

9010191 Minor corrections

92a4ab4 Add comments

64f6985 Update docs

9d695ae Add an internal caching layer for data generation strategies

c173aa5 Fix typos & include available fixups

c8de514 Add recipes section

bb5bfbe Remove unnecessary new lines in changelog

e08f625 Remove obsolete mentions of form_data

a2f164a Build docker images with buster as base image

7311d1f Document how to run custom checks via the Python API

faa5120 Do not use f-strings without interpolation

067d82c Add docstrings

8cea838 Remove not used HookLocation

8459dd9 Add register_check to the API reference

0fbf8e1 Add basic documentation for the public API

08018ef Disable the filter_too_much health check in test_global_body_hook

a863007 Disable Hypothesis health checks for test_openapi_links_multiple_threads

63622e7 Use --hypothesis-max-examples in a slow test

711fb32 Split corpus tests into more jobs

cf8c4d7 Decrease the maximum number of examples in the CI Hypothesis profile

bf063c0 Add manual triggers

2f08618 Ignore thread exceptions in test for keyboard interrupt

6738258 Add missing imports to code examples

b0b6ea6 Use blacken-docs

a9cb16e Add a CI job to check docs

3394c55 Rename endpoint to operation where necessary

a0ec86a Improve processing of API operation definitions

881c35e Add tests on schemas from Open API catalog

5f14f45 Document corner cases for stateful tests

b337551 Decrease memory usage

103186d Use compiled versions of Open API spec validators

74ddafe Add more tests on nullable parameters serialization

4258045 Suppress Hypothesis health checks in test_optional_form_parameters

4f0c8e7 Disable derandomization in stateful tests

b6cbbc8 Fix the schema_url fixture

3358c83 Fix Hypothesis warnings

0e5705d Rewrite the tests for stateful feature

93c7c95 Fix scheduled jobs

25bb3c4 Fix mutmut test error

8ce5d35 Update FAQ

40858b5 (changelog) Link additional issue

70d48c4 Do not use conditionals for detecting the content of Hypothesis enums

a261895 (cli) Show option default values in the CLI help output

fbe6e23 (cli) Group CLI parameters help output

7cb6392 Provide additional info in the test server output

71fb6ec Move exit code decision out from the output handler

57a702f Clarify low variance in generated data

d9ed57e Fix test_global_body_hook test

036fc49 Add example project

6d60cdc Remove deprecated git checkout HEAD^2 step from the CodeQL job

2e9794d Disable the too_slow health check for some tests

2ca9f50 Add a rare possibility into an assertion inside test_hypothesis_failed_event

c37304c Additional input validation for test server

7646027 Improve reliability of the test server

2e28b7e Add a test for interactions recording in ASGI apps

7298a0a Remove not needed condition

f510a39 Remove not needed code for GraphQL schemas

7bb50bc Remove deprecated stateful testing integration from the pytest plugin. (#933)

c6a9129 Minor refactorings & additional tests

86c52cb Add more tests (#931)

0a9e271 Fix displaying GIF on the PyPI page

d2e5168 Add more tests (#926)

edf9352 Use assume in some pytester-style tests (#924)

faabe7e Rename get_hypothesis_conversions -> get_parameter_serializer

2dcadf3 Open API parameters handling

Source code(tar.gz)
Source code(zip)
v3.13.4(Mar 29, 2022)
:bug: Bug fixes

639b7ab (docs) Bump Sphinx version

:wrench: Chores and Improvements

f4ead9e Release 3.13.4

b552584 Support Werkzeug>=2.1.0

7a5a8fe Pin aevea/action-kaniko

7ca318e Validate requests kwargs to catch cases when the ASGI integration is used, but the proper ASGI client is not supplied

:wrench: Chores and Improvements

41ab56e Release 2.8.6

fd54e0d Bump hypothesis-jsonschema version

baa97df Support for Werkzeug>=2.1.0

9e1fb3d Update graphql-server

b1e77fd Update pre-commit

:package: Other

c90a7b8 Fix failing tests

Source code(tar.gz)
Source code(zip)
v2.8.6(Mar 29, 2022)

Source code(tar.gz)
Source code(zip)
v3.13.3(Feb 20, 2022)
:rocket: Features

4c645b5 Extra requests related options for the replay command

:wrench: Chores and Improvements

2fee032 Release 3.13.3

c810f3e Update pre-commit

:package: Other

ea4855b Fix grammar

6603099 Update UTM labels in links

909a661 Update image links

Source code(tar.gz)
Source code(zip)

A modern API testing tool for web applications built with Open API and GraphQL specifications.

Related tags

Overview

Schemathesis

Features

Installation

Usage

Contributing

Support this project

Commercial support

Links

License

Comments

Description

Design

Questions to Consider

Releases(v3.17.5)

v3.17.5(Nov 8, 2022)

Added

Fixed

v3.17.4(Oct 19, 2022)

Fixed

v3.17.3(Oct 10, 2022)

Fixed

v3.17.2(Aug 27, 2022)

:bug: Fixed

v3.17.1(Aug 19, 2022)

v3.17.0(Aug 17, 2022)

:rocket: Added

:bug: Fixed

:wrench: Changed

v3.16.5(Aug 11, 2022)

:bug: Fixed

v3.16.4(Aug 8, 2022)

:wrench: Changed

v3.16.3(Aug 8, 2022)

:rocket: New

:bug: Fixed

v3.16.2(Aug 5, 2022)

:rocket: New

:bug: Fixed

:wrench: Changed

v3.16.1(Jul 29, 2022)

What's Changed

v3.16.0(Jul 22, 2022)

:octocat: GitHub Integration

What is next?

Your feedback is important :heart:

:bug: Fixed

:wrench: Changed

:fire: Removed

v3.15.6(Jun 23, 2022)

:bug: Fixed

v3.15.5(Jun 21, 2022)

:bug: Fixed

v3.15.4(Jun 6, 2022)

Added

Changed

v3.15.3(May 28, 2022)

Fixed

v3.15.2(May 9, 2022)

:bug: Fixed

v3.15.1(May 3, 2022)

:rocket: Added

:bug: Fixed

v3.15.0(May 1, 2022)

Major improvements for GraphQL

:rocket: Added

:wrench: Changed

:bug: Fixed

🏎️ Performance

v3.14.2(Apr 21, 2022)

🎉 Schemathesis.io: schema-based API testing as a service. 🎉

:rocket: CLI features

:bug: Bug fixes

:wrench: Chores and Improvements

v3.14.1(Apr 18, 2022)

What's Changed

v3.14.0(Apr 17, 2022)

What's Changed