tests: add a unit test suite and measure coverage - #22
Draft
insatomcat wants to merge 6 commits into
Draft
Conversation
The project had no test and no CI, so the two OpenSSF gold coverage criteria (test_statement_coverage90 and test_branch_coverage80) could not be evaluated at all. Add a pytest suite covering the five modules. Everything OVS touches is mocked (subprocess, sysfs, /proc, the network stack), so the suite needs neither root nor a cluster and runs anywhere. The measured result is 98.75 % of statements and 98.23 % of branches. Also: - declare PyYAML in the project dependencies. setup_ovs.py imports yaml to read .yaml/.yml configurations but nothing required it, so an installed package crashed on those files. - add a test extra, and enable branch coverage in pyproject.toml. - add a CI workflow running the suite on python 3.9 to 3.13, checking that the wheel builds reproducibly, and feeding the coverage report to SonarCloud. - add sonar-project.properties so the coverage report is imported. This only takes effect once Automatic Analysis is turned off on the SonarCloud project, since it never runs the tests. Five tests are marked xfail(strict=True). Each pins a bug found while writing the suite rather than encoding it as expected behaviour, so the suite fails again once the bug is fixed and the marker has to go. Signed-off-by: Florent Carli <florent.carli@rte-france.com>
python -m build creates its output directory before invoking the backend. With --outdir first, setuptools flat-layout auto-discovery then sees 'first' as a second top-level package next to setup_ovs and refuses to build. Build into RUNNER_TEMP instead. Signed-off-by: Florent Carli <florent.carli@rte-france.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
SonarCloud flagged four vulnerabilities, all in the workflow this branch adds, and around thirty maintainability issues in the new tests. Workflow: - pin every action to a full commit SHA instead of a moving major tag, which is what the OpenSSF supply chain guidance asks for. The major version is kept in a trailing comment. - add requirements-ci.txt and install through it, so a CI run does not silently pick up a new release of pytest or build between two runs. Every pin supports python 3.9 to 3.13. Tests: - add an assert_rejects helper in test_check.py so the assertion block holds a single call, instead of building the configuration inside the pytest.raises block. Shorter and unambiguous about what is expected to raise. - set helpers.dry_run through monkeypatch rather than by hand, and let monkeypatch restore it. The autouse fixture now captures the real find_command before the test body runs, so its teardown still clears the memoization when a test replaces that function. No behaviour change, same 217 tests and same coverage. Signed-off-by: Florent Carli <florent.carli@rte-france.com>
The OpenSSF statement and branch coverage criteria are self-asserted and only require a FLOSS tool able to measure them. coverage.py qualifies, so the evidence does not have to come from a third party. Print the coverage table into the job summary, which gives a readable and linkable figure on every run, independently of whether SonarCloud has been switched to CI analysis yet. Also drop the SonarCloud coverage badge from the README. The coverage metric does not exist on the project yet, so that badge rendered as an error. Point at the CI workflow instead, which works today. Signed-off-by: Florent Carli <florent.carli@rte-france.com>
insatomcat
marked this pull request as draft
August 2, 2026 18:01
clear_ovs walks ignored_bridges and removes each entry from the list of live bridges. The case where an ignored bridge is not up, so the removal is skipped and the loop just continues, had no test. It was the last partial branch in ovs.py, now at 100 percent on both statements and branches. Also add a test for a blank line in the ovs-vsctl list-br output, which the empty-name guard exists for. Signed-off-by: Florent Carli <florent.carli@rte-france.com>
vm_manager pull request 94 pins SonarSource/sonarqube-scan-action to v8.2.1 and sets SONAR_HOST_URL, which v8 needs to target SonarQube Cloud rather than a server. Use the same version and the same environment so both repositories stay on one convention. Signed-off-by: Florent Carli <florent.carli@rte-france.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The repository had no test and no CI. The two OpenSSF gold coverage criteria (
test_statement_coverage90andtest_branch_coverage80) could therefore not be evaluated at all: they ask for a measured figure, not for a test suite that merely exists.What this brings
A pytest suite of 219 tests over the five modules, running in about 0.2 s. Everything that touches the system is mocked (
subprocess, sysfs,/proc, the network stack), so the suite needs neither root, nor OVS, nor a cluster, and runs on any machine.Measured result:
test_statement_coverage90test_branch_coverage80build_reproducibleSOURCE_DATE_EPOCHContents
tests/, atestextra, and branch coverage enabled inpyproject.tomlsonar-project.propertiesso the report is importedrequirements-ci.txt, a pinned toolchain used as a pip constraints file, so a CI run does not silently pick up a new release of pytest or build between two runs. Every pin supports Python 3.9 to 3.13. Actions are pinned to full commit SHAs for the same reason, with the major version kept in a trailing comment.setup_ovs.pyimportsyamlto read.yamland.ymlconfigurations, but nothing required it, so an installed package crashed on those files.Reproducible build
Measured both ways rather than assumed. Two builds with
SOURCE_DATE_EPOCHpinned produce byte-for-byte identical wheels. Without it they differ: setuptools stamps the archive with the source file mtimes, which change on every checkout. Thereproducible-buildjob therefore builds the wheel twice and compares the SHA-256 of the two archives.Bugs found while writing the tests
None of them is encoded as expected behaviour. Each is marked
xfail(strict=True)with its reason: the day the bug is fixed the suite goes red again and forces the marker to be removed. These five points are still open, they are not fixed here.helpers.run_commanddoes nothing when the caller passescheck=(helpers.py:60). Thereturn subprocess.run(...)sits inside theif "check" not in kargsbranch. No caller passeschecktoday, so it is latent, but it is a trap for the next one.tagrange is never validated (check.py:283). The guard isif "vlan" in portwhile the attribute actually consumed byovs._create_bridgesistag.check.py:270): it sits insideif attribute in portand then testsattribute not in port. A vxlan port with neitherkeynorremote_ippasses the check, then raisesKeyErrorfurther down inovs.py:239.check.py:111)._attribute_is_a_portis documented as a TCP/UDP port check but enforces the VLAN tag range, 0 to 4095. A TCP/UDP port goes up to 65535.check.py:165).dpdk_interfacesandsystem_interfacesare locals of_check_port_configuration, which runs once per port, so they are always empty.Two pending items, outside this PR
Neither blocks the badge. The OpenSSF coverage criteria are self-asserted and only require a FLOSS tool able to measure them, which
coverage.pyis, so the figures published in the CI run summary are sufficient evidence on their own."Code coverage information is not supported"in the Sonar documentation). Switching to CI analysis would add thecoveragemetric and its history to the dashboard, nothing more.SONAR_TOKENsecret is not configured yet. The scanner step is conditioned on its presence, so CI stays green in the meantime. It must only be added once Automatic Analysis is off, since a CI analysis started while it is active fails the build.