diff --git a/behat_and_ai.rst b/behat_and_ai.rst new file mode 100644 index 0000000..59cbb43 --- /dev/null +++ b/behat_and_ai.rst @@ -0,0 +1,84 @@ +Behat and AI +============ + +AI coding agents can generate and modify code based on behavioural descriptions. +Behat provides a reliable way to define, verify, and preserve behaviour in these workflows. + +By expressing behaviour as executable scenarios, Behat ensures that AI-generated code +satisfies a clear and objective contract. + +Using Behat as a Behavioural Contract +------------------------------------- + +Behat scenarios define observable behaviour in a precise and executable form: + +.. code-block:: gherkin + + Feature: Refund processing + + Scenario: Refunding a paid order + Given there is a paid order + When I refund the order + Then the order should be marked as refunded + + Scenario: Prevent duplicate refunds + Given an order has already been refunded + When I refund the order + Then the refund should be rejected + +AI agents can implement features and use Behat to verify that the expected behaviour is satisfied. +When scenarios pass, the behavioural contract is fulfilled. + +Executable and Verifiable Behaviour +----------------------------------- + +Behat scenarios serve as executable specifications. They can be used by AI agents to: + +* verify correctness after implementing a feature +* detect behavioural regressions +* ensure behaviour remains consistent during refactoring + +Because scenarios are executable, they remain accurate as the system evolves. + +Generating Scenarios with AI +---------------------------- + +AI agents can generate Behat scenarios from behavioural descriptions. +These scenarios can then be reviewed, refined, and executed. + +This allows behaviour to be defined before or alongside implementation. + +Behat provides a structured and executable representation of behaviour +that can be used directly by AI tooling. + +Human Review and Iteration +-------------------------- + +As with any specification or testing tool, generated scenarios and assertions should be reviewed carefully. +It is critical to ensure that both the human-readable scenarios and the executable step definitions +accurately represent the intended behaviour. They also need to be robust enough to prove that +the actual behaviour matches the expectations, even if the implementation changes. + +Once validated, scenarios serve as the authoritative definition of the feature. AI agents can implement +or modify the system until all scenarios pass. + +This creates a reliable feedback loop where behaviour is defined, verified, and preserved. + +Maintaining Behaviour Over Time +------------------------------- + +With well-designed scenarios and step definitions, Behat ensures that behaviour remains stable as code changes. +AI agents can safely modify or refactor code, and Behat will detect any unintended behavioural changes. + +As long as scenarios pass, the defined behaviour is preserved. + +Behat MCP Server +---------------- + +Behat provides an `MCP (Model Context Protocol) server`_ that allows AI agents to interact +directly with Behat projects. + +The MCP server enables agents to discover scenarios, execute Behat, +and integrate behavioural verification into automated workflows. + +.. _`MCP (Model Context Protocol) server`: https://github.com/Behat/mcp-server diff --git a/conf.py b/conf.py index 5a7a7ac..be0fc71 100644 --- a/conf.py +++ b/conf.py @@ -61,7 +61,7 @@ logo_height = 50, # Base path for "Edit on GitHub" links - github_url="https://github.com/Behat/docs/blob/v3.0/", + github_url="https://github.com/Behat/docs/blob/v3.x/", # Global header / footer links as text|target header_links=', '.join([ diff --git a/index.rst b/index.rst index 0d6c844..c5af569 100644 --- a/index.rst +++ b/index.rst @@ -96,6 +96,7 @@ and many more. cookbooks releases useful_resources + behat_and_ai community .. _`Behaviour Driven Development`: https://en.wikipedia.org/wiki/Behavior-driven_development diff --git a/quick_start.rst b/quick_start.rst index 2cb8090..aad9bb7 100644 --- a/quick_start.rst +++ b/quick_start.rst @@ -455,21 +455,32 @@ code we could come up with to fulfil our scenario. Something like this: #[Then('I should have :arg1 product(s) in the basket')] public function iShouldHaveProductInTheBasket($count) { - // Normally you would import this class - we are using the fully qualified name - // to highlight that Behat does not come with an assertion tool (see note below). - \PHPUnit\Framework\Assert::assertCount( - intval($count), - $this->basket - ); + if (count($this->basket) !== intval($count)) { + throw new \Exception( + sprintf( + 'The basket should have %d item(s), but it has %d.', + intval($count), + count($this->basket) + ) + ); + } } #[Then('the overall basket price should be £:arg1')] public function theOverallBasketPriceShouldBePs($price) { - \PHPUnit\Framework\Assert::assertSame( - floatval($price), - $this->basket->getTotalPrice() - ); + $expectedPrice = floatval($price); + $actualPrice = $this->basket->getTotalPrice(); + + if ($expectedPrice !== $actualPrice) { + throw new \Exception( + sprintf( + 'Expected basket total price to be %s, but got %s.', + $expectedPrice, + $actualPrice + ) + ); + } } } @@ -477,27 +488,13 @@ As you can see, in order to test and implement our application, we introduced 2 ``Shelf`` and ``Basket``. The first is responsible for storing products and their prices, the second is responsible for the representation of our customer basket. Through appropriate step definitions we declare products' prices and add products to the basket. We then compare the -state of our ``Basket`` object with our expectations using PHPUnit assertions. +state of our ``Basket`` object with our expectations and throw exception if the expectations aren't met. .. note:: Behat doesn't come with its own assertion tool, but you can use any proper assertion - tool out there. A proper assertion tool is a library whose assertions throw - exceptions on failure. For example, if you're familiar with PHPUnit you can use - its assertions in Behat by installing it via composer: - - .. code-block:: bash - - $ php composer.phar require --dev phpunit/phpunit - - and then by simply using assertions in your steps: - - .. code-block:: php - - \PHPUnit\Framework\Assert::assertCount( - intval($count), - $this->basket - ); + tool out there. + Learn more about :ref:`assertion-tools`. Now try to execute your feature tests: diff --git a/requirements.txt b/requirements.txt index c5c756e..926b4d4 100644 --- a/requirements.txt +++ b/requirements.txt @@ -7,13 +7,13 @@ click==8.1.8 colorama==0.4.6 docutils==0.21.2 h11==0.16.0 -idna==3.10 +idna==3.15 imagesize==1.4.1 Jinja2==3.1.6 MarkupSafe==3.0.2 packaging==24.1 -Pygments==2.18.0 -requests==2.32.4 +Pygments==2.20.0 +requests==2.33.0 setuptools==78.1.1 sniffio==1.3.1 snowballstemmer==2.2.0 @@ -28,10 +28,10 @@ sphinxcontrib-htmlhelp==2.1.0 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==2.0.0 sphinxcontrib-serializinghtml==2.0.0 -starlette==0.49.1 +starlette==1.3.1 typing_extensions==4.13.1 -urllib3==2.5.0 +urllib3==2.7.0 uvicorn==0.34.0 watchfiles==1.0.4 websockets==15.0.1 -wheel==0.44.0 +wheel==0.46.2 diff --git a/useful_resources.rst b/useful_resources.rst index 4fe2abd..8f9cbc2 100644 --- a/useful_resources.rst +++ b/useful_resources.rst @@ -18,6 +18,60 @@ Integrating Behat with PHPStorm More information on integrating Behat with PHPStorm can be found in this `blog post`_. +.. _assertion-tools: + +Assertion tools +--------------- + +A proper assertion tool is a library whose assertions throw exceptions on failure. + +For example a list of the most known: + +- https://github.com/webmozarts/assert +- https://github.com/beberlei/assert +- https://github.com/zenstruck/assert + +.. admonition:: Caution with PHPUnit + :class: caution + + If you are familiar with PHPUnit, you can use its assertion library + + .. code-block:: bash + + $ php composer.phar require --dev phpunit/phpunit + + and then by simply using assertions in your steps: + + .. code-block:: php + + \PHPUnit\Framework\Assert::assertCount( + intval($count), + $this->basket + ); + + **WARNING: using PHPUnit for assertions no longer works with PHP 11.3.0 and later out-of-the-box**. + + This is due to a change in how PHPUnit's internal components are initialized. The recommended workaround + to use the PHPUnit assertions is to bootstrap PHPUnit during Behat execution from a ``BeforeSuite`` hook: + + .. code-block:: php + + use Behat\Hook\BeforeSuite; + + class FeatureContext { + + #[BeforeSuite] + public static function initPhpunit() { + (new \PHPUnit\TextUI\Configuration\Builder())->build([]); + } + } + + If you have multiple suites, you may want to use a static variable in the hook to ensure the initialization only + runs once. + + Learn more at https://github.com/Behat/Behat/issues/1618. + + Behat cheat sheet ----------------- diff --git a/user_guide/command_line_tool/formatting.rst b/user_guide/command_line_tool/formatting.rst index a9cb3d4..f57b32e 100644 --- a/user_guide/command_line_tool/formatting.rst +++ b/user_guide/command_line_tool/formatting.rst @@ -124,6 +124,8 @@ The following options are specific to the Pretty formatter: ``ShowOutputOption`` enum values, defaults to ``ShowOutputOption::Yes``. * ``shortSummary`` show just a list of failing scenarios at the end of the output. If false, a full summary (which also includes a list of failing steps) will be printed. Defaults to true +* ``printSkippedSteps`` If the output should include any steps which are skipped by the runner. Useful when you + don't want to see all the skipped steps after a failed step. Defaults to true Progress formatter ^^^^^^^^^^^^^^^^^^ diff --git a/user_guide/context/hooks.rst b/user_guide/context/hooks.rst index 066ff40..b7fd31a 100644 --- a/user_guide/context/hooks.rst +++ b/user_guide/context/hooks.rst @@ -267,7 +267,9 @@ Tagged Hooks Sometimes you may want a certain hook to run only for certain scenarios, features or steps. This can be achieved by associating a ``BeforeFeature``, ``AfterFeature``, ``BeforeScenario`` or ``AfterScenario`` hook with one -or more tags. You can also use ``OR`` (``||``) and ``AND`` (``&&``) tags: +or more tags. You can also use ``OR`` (``,``) and ``AND`` (``&&``) tags: + +Use the ``,`` tag to execute a hook when it has *any* of the provided tags: .. code-block:: php diff --git a/user_guide/gherkin.rst b/user_guide/gherkin.rst index 11589fb..57c4bcb 100644 --- a/user_guide/gherkin.rst +++ b/user_guide/gherkin.rst @@ -16,6 +16,12 @@ real, human language telling you what code you should write. If you're still new to Behat, jump into the :doc:`/quick_start` first, then return here to learn more about Gherkin. +.. note:: + + You can configure whether Behat's Gherkin parsing is compatible with + previous Behat versions, or with the official ``cucumber/gherkin`` + parsers. See :doc:`gherkin/parser_mode` for more details. + Gherkin Syntax -------------- @@ -103,3 +109,9 @@ run: Behat the ability to have multilanguage features in one suite. .. _`Business Readable, Domain Specific Language`: http://martinfowler.com/bliki/BusinessReadableDSL.html + +.. toctree:: + :maxdepth: 2 + :hidden: + + gherkin/parser_mode diff --git a/user_guide/gherkin/parser_mode.rst b/user_guide/gherkin/parser_mode.rst new file mode 100644 index 0000000..23885bb --- /dev/null +++ b/user_guide/gherkin/parser_mode.rst @@ -0,0 +1,164 @@ +Gherkin Compatibility Mode +========================== + +Behat uses the `behat/gherkin`_ library to parse your feature files into the data structures that +Behat will use to execute them. + +In most cases, this parses identically to `the official parsers provided by the Cucumber project`_. +However, there are some small differences in how our parser has traditionally treated some specific +syntax compared to the official parsers. + +To resolve this, we have added a ``GherkinCompatibilityMode`` setting to the parser. This setting +has two possible options: + +* ``GherkinCompatibilityMode::LEGACY`` - match our previous behaviour. This is the default in Behat 3.x. +* ``GherkinCompatibilityMode::GHERKIN_32`` - match the official parsers. This will become the default in Behat 4.0. + +.. caution:: + ``GherkinCompatibilityMode::GHERKIN_32`` is currently considered experimental. We expect that + there will be more changes to how the parser behaves in this mode before we mark it as stable. + +Configuring the parser mode +--------------------------- + +In Behat >= 3.30, you can specify the parser compatibility mode for your project in +your :doc:`/user_guide/configuration`: + +.. code-block:: php + + withProfile(new Profile('default') + ->withGherkinOptions(new GherkinOptions() + ->withCompatibilityMode(GherkinCompatibilityMode::GHERKIN_32) + ) + ) + ; + +Differences between parser modes +-------------------------------- + +Tables containing whitespace or escaped newlines +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In ``GHERKIN_32`` mode, table cells can include newlines, which will be unescaped during parsing. Note that +newlines are unescaped **after** we remove the cell padding. + +For example, with the following table: + +.. code-block:: gherkin + + Given 3 lines of poetry on 5 lines: + | \nraindrops--\nher last kiss\ngoodbye.\n | + +In ``GHERKIN_32`` mode, the table will parse as: + +.. code-block:: php + + [ + [ + <<getTags()`` will **include** the ``@`` prefix. In legacy mode, + this was removed. This may affect custom hooks / event listeners that inspect the tag values at + runtime. + + +File language +~~~~~~~~~~~~~ + +In ``GHERKIN_32`` mode, if a file includes a ``#language`` annotation: + +* Any whitespace in / around the tag will be ignored - so ``# language : fr`` will be + recognised as a valid language tag. In legacy mode, this would have been treated as a comment. +* Parsing fails if the language is not recognised - so ``#language: no-such`` will cause an error. + In legacy mode, this would have been ignored and parsing would continue in the default language. + +Whitespace following step keywords +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In ``GHERKIN_32`` mode, a space between a step keyword and the rest of the text is treated as part of the keyword. This +is because in a small number of languages there is no space after the keyword. + +With a step in English like ``Then something should happen``, if you call ``StepNode::getKeyword()`` then: + +* In ``GHERKIN_32`` mode the return value will be ``'Then '`` +* In ``LEGACY`` mode the return value will be ``'Then'`` + +In a language that does not place spaces after the keyword (e.g. Japanese), the return value will be the same in both +modes. + +Elements with descriptions +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Gherkin syntax allows multi-line descriptions on ``Feature:``, ``Background:``, ``Scenario:``, ``Scenario Outline:``, +and ``Examples:`` elements. + +Historically, we only parsed the description separately for a ``Feature`` node. For other nodes, we parsed the full +text as a multi-line title. + +In ``GHERKIN_32`` mode, if one of the elements listed above has multi-line text, then: + +* The first line (containing the keyword) will be parsed as the title. +* Following lines will be parsed as the description. +* Any blank lines between the title & description will be ignored (in legacy mode, these were included at the start of + the description). +* Any left padding will be removed from the first line of the description, but subsequent lines will have the same + left padding / indentation as the feature file. In legacy mode, we attempted to left-trim all lines to match the + indentation of the keyword. + + +.. _`behat/gherkin`: http://martinfowler.com/bliki/BusinessReadableDSL.html +.. _`the official parsers provided by the Cucumber project`: https://github.com/cucumber/gherkin