Skip to content

Add format-based negative test generation - #33

Merged
sinan-ozel merged 3 commits into
mainfrom
email-format-validation-tests
Apr 20, 2026
Merged

Add format-based negative test generation#33
sinan-ozel merged 3 commits into
mainfrom
email-format-validation-tests

Conversation

@sinan-ozel

Copy link
Copy Markdown
Owner

What this PR does

Closes #— (no linked issue)

When an OpenAPI spec documents a 400 response for a POST or PUT endpoint, the
plugin now automatically generates one extra request body per required
string property annotated with a recognised format (email, uri, ipv4,
ipv6, hostname, idn-hostname, uuid). Each generated body replaces only
that one field with a deliberately invalid value while keeping the rest valid.

The server must respond 400 or 422. The plugin additionally requires the
response body to contain at least one non-whitespace string — an empty {}
body is treated as a contract violation.

Format-invalid objects are only generated for required properties and only
when 400 / 422 is documented in the spec's responses, so servers that do
not validate format are unaffected.

Changes

  • src/pytest_openapi/case_generator.py_INVALID_FORMAT_VALUES dict;
    generate_object_test_cases injects one format-invalid object per required
    format field; generate_string_test_cases appends invalid values when
    valid_only=False.
  • src/pytest_openapi/contract.pycontains_invalid_format_value() and
    _has_nonempty_string_value() helpers; is_negative_test detection updated
    in test_post_endpoint_single, test_post_endpoint, and
    test_put_endpoint; 400/422 responses now require a non-empty body.
  • src/pytest_openapi/__init__.py — bumped to v0.3.1.
  • 8 new mock servers under tests/test_servers/ covering email, URI, UUID,
    IPv4, IPv6, hostname, a no-message-400 negative case, and URL port-range edge
    cases.
  • tests/test_integration.py — 25 new integration tests.
  • tests/test_unit.py — 4 new unit tests for the case generator.
  • README.md — format-validation table, OpenAPI 3.0 vs 3.1 compatibility
    table.
  • docs/how-it-works/test-cases.md — format-validation section.
  • CHANGELOG.md — v0.3.1 entry.

Test plan

  • All existing tests pass (docker compose -f tests/docker-compose.yaml …)
  • New email format validation tests pass (valid address → 200, invalid → 400 with message)
  • No-message 400 test correctly fails when server returns empty body
  • URI, UUID, IPv4, IPv6, hostname format tests pass
  • URL port edge cases: port 0 → 400, port > 65535 → 400, port 1–65535 → 200
  • schema_based_api (no 400 documented, email field not required) unaffected — count stays at 13 passed

🤖 Generated with Claude Code

sinan-ozel and others added 3 commits April 20, 2026 10:01
When an OpenAPI spec documents a 400 response for a POST or PUT
endpoint, the plugin now generates one extra request body per
required string property annotated with a recognised format
(email, uri, ipv4, ipv6, hostname, idn-hostname, uuid). Each
generated body replaces only that field with a deliberately
invalid value while keeping all other fields valid.

The server must respond 400 or 422; additionally the response body
must contain at least one non-whitespace string. An empty {} body
is treated as a contract violation — error responses need to be
human-readable.

Format-invalid objects are only generated for required properties
and only when 400 / 422 is in the spec's documented responses, so
servers that do not validate format (and do not document 400) are
unaffected.

Eight new mock servers cover email, URI, UUID, IPv4, IPv6,
hostname, a no-message 400 negative case, and URL port-range
edge cases (port 0 and > 65535 invalid; 1–65535 valid).
Corresponding integration tests and four unit tests were added.

README updated with a format-validation table and an OpenAPI
3.0 vs 3.1 compatibility table. docs/how-it-works/test-cases.md
updated with format-validation details. CHANGELOG entry added
under v0.3.1.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@sinan-ozel
sinan-ozel merged commit deb5ac0 into main Apr 20, 2026
7 checks passed
@sinan-ozel
sinan-ozel deleted the email-format-validation-tests branch April 20, 2026 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant