Add format-based negative test generation - #33
Merged
Conversation
When an OpenAPI spec documents a 400 response for a POST or PUT
endpoint, the plugin now generates one extra request body per
required string property annotated with a recognised format
(email, uri, ipv4, ipv6, hostname, idn-hostname, uuid). Each
generated body replaces only that field with a deliberately
invalid value while keeping all other fields valid.
The server must respond 400 or 422; additionally the response body
must contain at least one non-whitespace string. An empty {} body
is treated as a contract violation — error responses need to be
human-readable.
Format-invalid objects are only generated for required properties
and only when 400 / 422 is in the spec's documented responses, so
servers that do not validate format (and do not document 400) are
unaffected.
Eight new mock servers cover email, URI, UUID, IPv4, IPv6,
hostname, a no-message 400 negative case, and URL port-range
edge cases (port 0 and > 65535 invalid; 1–65535 valid).
Corresponding integration tests and four unit tests were added.
README updated with a format-validation table and an OpenAPI
3.0 vs 3.1 compatibility table. docs/how-it-works/test-cases.md
updated with format-validation details. CHANGELOG entry added
under v0.3.1.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Closes #— (no linked issue)
When an OpenAPI spec documents a
400response for a POST or PUT endpoint, theplugin now automatically generates one extra request body per required
string property annotated with a recognised format (
email,uri,ipv4,ipv6,hostname,idn-hostname,uuid). Each generated body replaces onlythat one field with a deliberately invalid value while keeping the rest valid.
The server must respond 400 or 422. The plugin additionally requires the
response body to contain at least one non-whitespace string — an empty
{}body is treated as a contract violation.
Format-invalid objects are only generated for required properties and only
when
400/422is documented in the spec's responses, so servers that donot validate format are unaffected.
Changes
src/pytest_openapi/case_generator.py—_INVALID_FORMAT_VALUESdict;generate_object_test_casesinjects one format-invalid object per requiredformat field;
generate_string_test_casesappends invalid values whenvalid_only=False.src/pytest_openapi/contract.py—contains_invalid_format_value()and_has_nonempty_string_value()helpers;is_negative_testdetection updatedin
test_post_endpoint_single,test_post_endpoint, andtest_put_endpoint; 400/422 responses now require a non-empty body.src/pytest_openapi/__init__.py— bumped to v0.3.1.tests/test_servers/covering email, URI, UUID,IPv4, IPv6, hostname, a no-message-400 negative case, and URL port-range edge
cases.
tests/test_integration.py— 25 new integration tests.tests/test_unit.py— 4 new unit tests for the case generator.README.md— format-validation table, OpenAPI 3.0 vs 3.1 compatibilitytable.
docs/how-it-works/test-cases.md— format-validation section.CHANGELOG.md— v0.3.1 entry.Test plan
docker compose -f tests/docker-compose.yaml …)schema_based_api(no 400 documented, email field not required) unaffected — count stays at 13 passed🤖 Generated with Claude Code