feat(itf): add dual-QEMU ivshmem integration-test environment by Rahul-Sutariya · Pull Request #617 · eclipse-score/communication

Rahul-Sutariya · 2026-06-30T07:01:34Z

Add a dual_qemu ITF/pytest plugin that boots two QNX VMs sharing one QEMU ivshmem-plain region, extending the upstream single-VM qemu plugin.

plugin: config.py, ivshmem_qemu.py, dual_qemu_process.py, init.py (target_a/target_b fixtures; sequential boot to avoid KVM race), BUILD, example config, README, example/run_example_manually.sh.
integration_testing.bzl: dual_qemu_integration_test macro (QNX-only, flaky).

castler

Let's first align on the test infrastructure - then we can check out the example :)

castler · 2026-07-13T12:16:03Z

+def dual_qemu_integration_test(
+        name,
+        srcs,
+        filesystem,


Do we not need two filesystems here?

We can have multiple binaries in one filesystem Ex.

pkg_application( name = "gateway_test_apps", app_name = "gateway_test", bin = [":app1", ":app2"], # both binaries in one package )

If we want separate, then i can extend the macro to accept two filesystem targets and build two IFS images

Yes - I think we should have two filesystems to be more variable (e.g. imagine having different IP configurations one day or different boot ups or what ever)

Adapted the changes for second filesystem, THanks

castler · 2026-07-13T12:17:18Z

+    # Driving two real QNX guests under KVM has rare, environment-induced boot
+    # nondeterminism (e.g. a guest occasionally wedging during device bring-up).
+    # The fixtures already stagger boots and wait for stable SSH; mark the test
+    # flaky so bazel transparently retries such infrastructure hiccups.


Ouch - is this possible to be fixed? Should we add a ticket with a todo?

It's a KVM + QNX -kernel boot path timing issue — the second guest occasionally wedges during SMP AP startup when both initialize under the same KVM instance. The plugin already mitigates by booting VMs sequentially and using single-core guests, which makes it very rare but not fully eliminable from our side.

A proper fix would require either:

QNX fixing the boot-path race in their -kernel loader (upstream QNX issue), or

Using the -bios/UEFI boot path instead of -kernel (different QNX IFS packaging, not trivial).

castler · 2026-07-13T14:16:37Z

+logger = logging.getLogger(__name__)
+
+
+class IvshmemQemu:


We now copied here a lot from: https://github.com/eclipse-score/itf/blob/main/score/itf/plugins/qemu/qemu.py can we not use some software mechanisms to reduce code duplication? Inheritance? Decorators?

The Qemu class uses double-underscore name mangling (__build_qemu_command, __find_available_kvm_support, etc.), which Python mangles to _Qemu__build_qemu_command — making inheritance/override impossible from a subclass. There's no extra_args hook or any other extension point to inject additional -object/-device flags.

we can rename __build_qemu_command to _build_qemu_command (drop the name mangling) and add an overridable _extra_qemu_args() hook that the base command assembly calls. (itf repo changes) then we can avoid duplication, want me creat pr for ITF repo?

S-Core/ITF pr: eclipse-score/itf#122

castler · 2026-07-13T14:22:43Z

+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.stop()
+
+    def start(self, subprocess_params=None):


should this not only start one instance? and then when we have another instance of this class, we will start the otherone?

Yes, that's exactly how it works. Each IvshmemQemu instance starts one QEMU process. In the plugin's _targets fixture, the loop creates a separate instance per VM

castler · 2026-07-13T14:23:29Z

+logger = logging.getLogger(__name__)
+
+
+class DualQemuProcess:


Should we not inherit from here: https://github.com/eclipse-score/itf/blob/main/score/itf/plugins/qemu/qemu_process.py#L22?

Where is our equivalent to https://github.com/eclipse-score/itf/blob/main/score/itf/plugins/qemu/qemu_target.py ?

QemuProcess: Can't inherit — it hardcodes Qemu(...) in init with no way to inject a different launcher. If upstream accepted the Qemu instance as a constructor parameter (DI), DualQemuProcess could be eliminated entirely.

QemuTarget: We already reuse it directly (from score.itf.plugins.qemu.qemu_target import QemuTarget). No custom equivalent needed — DualQemuProcess exposes the same .stop()/.restart()/.console interface that QemuTarget expects.

castler · 2026-07-13T14:24:28Z

+logger = logging.getLogger(__name__)
+
+
+def _wait_for_ssh(target, total_timeout: int = 180, interval: int = 3, stable_successes: int = 3):


These helper functions must be used already somewhere else in the other Qemu plugin, no?

No — the single-VM plugin only does pre_tests_phase (5 SSH retries with fixed timeouts). It doesn't need stable-success counting or boot-restart logic because a single QNX guest under KVM boots reliably. These helpers address problems specific to the dual-VM scenario (sshd instability during concurrent boots, occasional total boot wedges, and the first VM going quiet while the second initializes). They're new logic rather than duplication from upstream.

castler · 2026-07-15T08:09:29Z

+        makers = []
+        for index, vm in enumerate(dual_config.vms):
+            def make_process(index=index, vm=vm):
+                return DualQemuProcess(


Why are we not using

with DualQemuProcess(...) as target_a: with DualQemuProcess(...) as target_b: yield target_a, tarbet_b

This would make things IMHO way more clear, would avoid the complicated loop and would enable to give each QemuProcess also its custom configuration.

Adapted the changes, Thanks

Add a `dual_qemu` ITF/pytest plugin that boots two QNX VMs sharing one QEMU ivshmem-plain region, extending the upstream single-VM `qemu` plugin. - plugin: config.py, ivshmem_qemu.py, dual_qemu_process.py, __init__.py (target_a/target_b fixtures; sequential boot to avoid KVM race), BUILD, example config, README, example/run_example_manually.sh. - integration_testing.bzl: dual_qemu_integration_test macro (QNX-only, flaky). - example: offset_list.h + simple_ivshmem.cpp map the ivshmem BAR via pci-server and exchange a self-relative offset-pointer list, proving cross-VM pointer resolution; simple_ivshmem_test.py verifies both VMs.

note: Cross-VM shared memory forces the region onto the ivshmem PCI BAR (device/SHMCTL_PHYS memory), but the production lib-memory ControlBlock puts a process-shared pthread_mutex there and locks it during init. QNX refuses to operate a process-shared mutex on device-mapped BAR memory (returns EINVAL even with write-back caching), so the producer aborts — making the umbrella score::memory::shared unusable over ivshmem, unlike the mutex-free demo.

github-project-automation Bot added this to COM - Communication FT Jun 30, 2026

github-project-automation Bot moved this to Backlog in COM - Communication FT Jun 30, 2026

Rahul-Sutariya force-pushed the rasu_dual_qemu_intergation_test_plugin branch from f56837b to eedfda4 Compare June 30, 2026 11:14

Rahul-Sutariya force-pushed the rasu_dual_qemu_intergation_test_plugin branch 4 times, most recently from 5bd6beb to 38c934b Compare July 13, 2026 07:14

Rahul-Sutariya marked this pull request as ready for review July 13, 2026 10:00

Rahul-Sutariya requested review from LittleHuba, bemerybmw, castler, crimson11, hoe-jo and limdor as code owners July 13, 2026 10:00

castler reviewed Jul 13, 2026

View reviewed changes

Rahul-Sutariya force-pushed the rasu_dual_qemu_intergation_test_plugin branch 2 times, most recently from 6c98422 to 01bea3d Compare July 14, 2026 13:59

Rahul-Sutariya requested a review from castler July 14, 2026 13:59

castler reviewed Jul 15, 2026

View reviewed changes

Rahul-Sutariya added 2 commits July 15, 2026 17:57

Rahul-Sutariya force-pushed the rasu_dual_qemu_intergation_test_plugin branch from 01bea3d to b783e72 Compare July 15, 2026 12:28

		logger = logging.getLogger(__name__)


		def _wait_for_ssh(target, total_timeout: int = 180, interval: int = 3, stable_successes: int = 3):

Uh oh!

Conversation

Rahul-Sutariya commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

castler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rahul-Sutariya Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rahul-Sutariya Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rahul-Sutariya Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Rahul-Sutariya commented Jun 30, 2026 •

edited

Loading

Rahul-Sutariya Jul 14, 2026 •

edited

Loading

Rahul-Sutariya Jul 14, 2026 •

edited

Loading

Rahul-Sutariya Jul 14, 2026 •

edited

Loading