This strategy allows to produce single unicode characters by specified rules: you may have exact characters or characters that belongs some Unicode Category. Additionally, you may reverse your rule to specify which characters you don't want to get.

characters strategy preserved existed OneCharStringStrategy behaviour when with no arguments it produces all the characters except surrogates.

characters with exact only argument converted into sampled_from strategy implicitly.

Generally tests passed, but travis fails due to irrelevant reasons. What I can fix will send with additional PRs.

Feedback is welcome (:

Allows users to register settings profiles and load them in as needed. Allowing for different defaults for different environments

Is this kind of what you were thinking for the profile setup?

This code should enable the following pattern

Settings.register_profile('dev', max_examples=5, max_shrinks=10, max_iterations=100)
Settings.register_profile('ci', max_examples=1000, max_shrinks=1000, max_iterations=2000)

Settings.load_profile(os.getenv('HYPOTHESIS_PROFILE', 'default'))

If you think im close ill iterate with you as needed. If we get to something you like let me know and ill write up the docs.

One of my least favourite things about Hypothesis is that it uses the word 'strategy' to describe its data generators. This was an internal name that I unthinkingly propagated into the public API, and was at best vaguely descriptive of what it did back in Hypothesis < 3.0 and is completely irrelevant to its function now if not actively misleading.

Additionally, it's looking increasingly likely that Hypothesis for other languages is going to start being a thing. If that's the case, I will not be propagating this mistake to them, and it would be good to have Hypothesis-for-Python use the common terminology.

This feels like a huge big scary change, and it is but that's mostly updating the documentation, articles, and user perception. In terms of updating the code:

1. One of the oddities around how Hypothesis is structured is that the type SearchStrategy is not actually part of the public API. There's nowhere public that users can actually import it from, and users are not allowed to subclass it - all creation of strategies goes through hypothesis.strategies. So nothing really needs to be done there.
2. The hypothesis.strategies module can easily be renamed and have a module that just imports and re-exports everything in its __all__.

Set against that relative ease is that we're renaming what is effectively the main entry point to the Hypothesis API. There's a lot of documentation to be updated and we'll be tripping over the terminology for a long time. So if we do this it comes with a significant cognitive burden and a moderate amount of user ill-will.

So, two questions:

1. Should we do this? My vote is yes, but I'm sufficiently nervous about the idea that I could easily be talked down.
2. If so, what should we call it?

Possible alternative names

I'm not currently flush with good names for this. I'd like generator or something along those lines, but that would be horrendously confusing in a Python context. Alternatives:

• Recipe
• Schema
• Kitten (we probably shouldn't go with this one)
• Provider

Suggestions very much welcome.

This implements an extras module providing pandas core data types.

It is currently a very provisional pull request. It's probably buggy, probably slow, and definitely incomplete. I'm putting it out early to solicit feedback from the interested, as I don't really use pandas myself so would like some commentary from people who do! Feedback on the API would definitely be particularly appreciated - the end result will probably look more or less like what's currently there, but it's more of a sketch than anything else.

TODO

• [x] ~~Merge #826~~ I think we can release without. It will make datetime data types slightly problematic initially, but that will get fixed whenever #826 is released.
• [x] More documentation, with usage examples
• [x] Update RELEASE.rst
• [x] Final sign-off on the API from @sritchie
• [x] Review and anything that comes up in it

What this introduces

This implements strategies for Array API implementations inside hypothesis.extra.array_api (closes #3037). As the Array API is largely based on NumPy behaviour, I imitated hypothesis.extra.numpy where appropriate and so these strategies will hopefully feel familiar to the extra's contributors and users.

Strategies in array_api do not import array modules, instead taking them as the xp argument. For the most part they assume that the user has indeed provided an Array API-compliant module.

The strategies would be used by Array API implementers (e.g. @asmeurer's compliance suite) and array-consuming libraries (examples).

Many tests are based on the test/numpy ones. There is a mock array-API implementation in xputils.py. Tests will try to import NumPy's Array API implementation (numpy/numpy#18585 was merged just today) and will fall back to the mocked one. I couldn't easily mock a "compliant-looking" array object so a particular non-compliance warning is suppressed and some assertions are skipped when necessary.

cc @MattiP

Specific requests for feedback

An immediate concern I have is with how I form pretty ("reflective") reprs for xp consuming strategies. Default use of @defines_strategy (and thus LazyStrategy) is prone to produce some rather noisy repr strings, so I ended up wrapping these strategies with a custom decorator @pretty_xp_repr... it seems to be a hacky solution, especially since it gets called multiple times.

I'm also wondering if you're happy with the get_strategies_namespace() method that Zac suggested. It could be nice to not even have the top-level strategies (i.e. array module needs to be passed) to require users to use it, which could mitigate confusion. There could also be some magic where a user could import say extra.array_api.pytorch and have Hypothesis auto import a (future) Array API implementation in PyTorch.

I see the NumPy extra (and thus this PR) violates the house API style. Let me know if for array_api I should take the oppurtunity to drop potentially undesirable features in arrays() such as inferring strategies via dtype and passing kwargs via elements.

The shape/axis/index strategies were implemented namely to avoid importing NumPy by using extra.numpy, which also allows use of Array API naming conventions and removes some NumPy-specific limitations. They near-verbatim emulate those in the NumPy extra, so a future PR could have extra.numpy wrap them to deal with small differences.

The Array API is not finalised yet. Some tests may need slight modification in the future if the consortium decide in data-apis/array-api#212 that they don't want polymorphic return values in xp.unique().

Hello to everyone at the sprints! This issue is the place to get started, comment to claim an issue (please talk to me first), and so on. Thanks for helping out!

The "what is Hypothesis" pack: my talk if you like videos, the docs as a general overview, and these quick excercises to try it out.

All of the following are valued contributions: reading docs or trying to use hypothesis and telling me what you found confusing; adding Hypothesis tests to other open-source projects (e.g. Pandas, dateutil, Xarray, etc - ask me!); new documentation or blog posts or artwork; and of course traditional code whether bugfixes or new features!

General checklist:

1. Talk to Zac about what you want to do - I can help you find the right issue (start here) or other way to contribute :smile:
2. (optional): read CONTRIBUTING.rst and check what's in the guides/ directory for tips.
3. Comment below so people don't work on the same issue by accident!
4. Do the thing :snake:
5. Open a PR :tada:

Closes #293. This pull:

• adds a new function from_type to look up a strategy that can generate instances of the given type
• upgrades builds() to infer missing arguments based on type hints
• upgrades @given to infer missing arguments from type hints
• adds a new function register_type_strategy to register custom types that can't be automatically derived (based on a known child class or type hints, using builds)

It's been a long time coming, but I think I'm done. (again 😉)

The django integration currently relies on the simple email strategy in provisional.py. It would be nice to have a native email address type which was a bit better at hitting edge cases.

When this issue was first opened, we relied on the fake-factory package for Django email fields, which was substantially slower and less devious than Hypothesis strategies. That has been fixed, but we'd still like to improve the email strategy before making it part of the public API.

Anyone working on this issue should start with the provisional strategy, move it to strategies.py, and gradually expand it to generate unusual things allowed by the relevant RFCs (see below). Very obscure features - such as >255 character addresses, or using an IP address instead of a domain name - should be avoided, or at least gated behind an optional argument (eg emails(allow_obscure=False)).

The idea of tests/cover is that it's supposed to be a relatively small fast set of tests that gets 100% coverage and anything not required for that should go in tests/nocover. This idea has been seen more than in breach than observance (including from me), and as a result the coverage check is one of our slowest build jobs. See e.g. this build where it took 13 minutes.

It would be nice to get the coverage check under 5 minutes on Travis. I propose many small applications of the following algorithm:

1. Run "PYTHONPATH=src python -m pytest tests/cover --durations=10"
2. Pick the slowest test
3. Move it into nocover.
4. Run "make check-coverage" (or "tox -e coverage")
5. If this causes us to have less than 100% coverage, come up with a faster test to cover that line to put in tests/cover.
6. Open a pull request with that one test move.

This probably won't be sufficient to get the time under control on its own, but if we get it to the point where no individual test takes > a second we can start thinking about other measures.

The Hypothesis settings system is currently a confusing mess of confusingness. It asks users to make decisions they can't possibly have sufficient information to make, and ties those decisions very closely to Hypothesis internals.

I think we should give serious consideration to deprecating most of them. This ties in to #534

Here is a rough list of what I think about specific settings:

• buffer_size - very bad. Totally about Hypothesis internals, totally undocumented as to what it really means. Hypothesis should figure out something sensible to do here and then do it.
• ~~database_file - not intrinsically bad, but redundant with database. May be worth considering making the database API more public and just using the database setting.~~ deprecated in #1196
• database - good. Sensible operational decision that the user can reasonably have strong opinions about and Hypothesis shouldn't.
• derandomize - Good though arguably redundant with seed.
• max_examples - mostly reasonable but confusingly named (conflicts with example decorator while hilariously ignoring it entirely). I feel like having it be a max might also be a bad idea.
• max_iterations - bad, often trips people up when they forget to set it, there's no real way people could have enough information to set it. Hypothesis should grow a better heuristic here.
• max_shrinks - tuning heuristic. Only really useful to make testing strategies faster, but crucial there (~10x slowdown in our tests without it).
• min_satisfying_examples - Same. See also #534 and comments on #518. This is very tied to Hypothesis's internal notion of what an example is.
• perform_health_check - sensible operational decision in which the user deliberately decides to ignore warnings, but I feel like it might be better dropped and asking people to explicitly suppress the health check they don't want using perform_health_check.
• phases - this is a useful feature that I think supplants a lot of the use cases for things like max_shrinks, max_examples, etc and apparently is completely undocumented! That should be fixed.
• stateful_step_count - totally bad. Why is this even here? Hypothesis should just do something sensible automatically and provide a function argument to override that.
• ~~strict - good. Sensible operational decision as to how the user wants to respond to Hypothesis deprecations~~ bad - deprecated in favour of normal warning control.
• suppress_health_check - good. Explicit request to suppress health checks. Totally reasonable to want.
• ~~timeout - not intrinsically bad from a user point of view but should probably go anyway due to #534~~ now deprecated
• verbosity - Good. Useful debugging tool, totally sensible thing to want to tune.

All of this should be done with a very careful deprecation process which preserves the current default behaviour, warns when you execute a behaviour that will change in future, and gives people an option to opt in to the future behaviour.

Pending Deprecation Plans

The following are suggested routes for some deprecations:

• Deprecate perform_health_check=False and suggest suppress_health_check=list(HealthCheck)
• Deprecate max_iterations, and default to not_set - meaning five times max_examples (the longstanding factor before the default for that was reduced to speed up under coverage).
• Deprecate stateful_step_count and set it to not_set by default. Add a max_steps property on GenericStateMachine which also defaults to not_set (or maybe None?). Determine actual max_steps in order of "Look on object, look on settings, use a default". When the deprecation plan moves to dropping it,

I really hate it when people submit a monster pull request to my projects. So... I decided to submit a monster pull request to yours.

This improves hypothesis.extra.django in the following ways:

• Allows developers to override the default field mappings using add_default_field_mapping (previously impossible).
• Performs a full_clean() on all models before saving, ensuring they're valid and allowing the model to perform it's full lifecycle.
• Provides public strategies for generating data for many django model field types. For example, you can get data for a SlugField using slug_field_values(MyModel._meta.get_field("some_field"))
• Provides a public strategy for generating data for any registered field field_values(MyModel._meta.get_field("some_field"))
• Provides a defines_field_strategy decorator for easily creating new field strategies.
• Minimizes the size of generated data by biasing towards model default fields, configurable via the __default_bias argument to models(). Non-trivial models previously overflowed the data buffer frequently, triggering health checks.
• Added built-in support for UrlField, DateField, TimeField and SlugField mapping.
• Multi-db support for models() strategy via __db parameter.
• Ability to use add_default_field_mapping as a decorator to a strategy factory.

Fixes bugs:

• Uncaught DataErrors no longer blowing up tests.
• Fields no longer silently truncated by unicode null.
• EmailField strategy no longer generates values that don't pass EmailValidator.

Potential "breaking" changes:

• The default_value strategy used to omit values from the model data dict. Now it includes the field's default value. This is now in line with what the public docs actually say, but it's still different from current behaviour. I'd class this as a bugfix.

To do:

• Improve performance of UrlField and EmailField implementations.
• Update documentation.
• Fix formatting errors.

I'll happily tackle these todos if I get some sort of positive feedback on these changes. :)

Closes #3530, as @rsokl and I discussed a few weeks ago.

Plus a small fix so that KeyboardInterrupt (and other exceptions not treated as test failure) interrupt immediately as they do for generated examples.

I noticed that e.g. the Getting started with Hypothesis article demonstrates the use of hypothesis.reject(). However, I wasn’t able to find more documentation on that function, likely because there is none:

https://github.com/HypothesisWorks/hypothesis/blob/226028dcf62b7c7a239574af6f5d69559be0febc/hypothesis-python/src/hypothesis/control.py#L23-L24

Would it make sense/be possible to elaborate?

Thank you!

A classic error when testing is to write a test function that can never fail, even on inputs that aren't allowed or manually provided. @rsokl mentioned a nice design pattern for @pytest.mark.parametrize() of including an xfailed param to check that the test can fail; and after some discussion we agreed that this would be nice to have in Hypothesis too.

So: following #3516, I'd like to add an @example(...).xfail(condition: bool = True, *, reason: str = "", raises: type[BaseException] | tuple[...] = BaseException) - matching the interface of pytest.mark.xfail() (omitting the subset that doesn't make sense here). Implementation-wise, this will return self if condition is False, otherwise return an instance of a new XfailExample subclass which is known and handled by the execution logic in core.py. If the exception raised is not an instance of raises and our failure_exceptions_to_catch() it propogates as usual; if nothing is raised then we raise an error (using pytest.xfail() if available). Naturally we'll also support .via().xfail() and .xfail().via() 😁

It's been a while since we revisited the way that Hypothesis reports failing examples, but I think it's time:

• We'll need to revisit reporting in order to support https://github.com/HypothesisWorks/hypothesis/issues/3411, and it'd be nice to get that ready before the engine changes both to break the work into smaller chunks, and to get improvements into users' hands as soon as possible.
• I'd like to encourage more use of the @example() decorator, perhaps with .via() on Python 3.9+, and reporting failures in that format seems like an effective way to do so - and makes reproducing a mere matter of copy-pasting when the database doesn't make it fully automatic.

For both of these reasons, using the __repr__ of custom objects is unsatisfying, because they often - and by default - don't consist of executable code which would return an equivalent object. Happily, our friends over at CrossHair recently solved this problem in a way I think we can imitate: represent custom objects by (recursively) representing the call that we executed in order to create them!

You can see their implementation here; in short we can store a tree of fragments (via get_pretty_function_description() etc.) in a dict keyed off the ID of the object, and spanning the lifetime of that example. For anything not in the dict, i.e. where the function call was not executed by Hypothesis builds() or .map() (or flatmap, or recursive, or etc.), we'll fall back to our existing pretty-printing code. One notable divergence: I'll want to store the start and end span of the underlying buffer that generated each fragment, since that's a key component of #3411.