[benchmarker] Add benchmarker configuration and report definitions by BenjaminPelletier · Pull Request #1567 · interuss/monitoring

BenjaminPelletier · 2026-07-13T06:48:23Z

This PR adds the schemas for inputs (configuration) to a new benchmarker tool described in #1566 as well as outputs (report). The top level input is BenchmarkConfiguration: it describes a sequence of scenarios to run using loads composed of virtual users, some of which use uss_qualifier resources, bookended by setup and teardown actions, plus artifacts to generate from the data gathered. isas_uncontended.jsonnet provides a concrete example (verified with an initial implementation) of virtual flight planner users who create and delete ISAs at a particular location. It produces a series of graphs for different internode latencies between DSS instances, each showing the throughput vs load graph for each of the three DSS instances. The top level output is BenchmarkRunReport, similar to uss_qualifier's TestRunReport.

Example output from implementation sketch:

the-glu · 2026-07-13T06:54:48Z

+      },
+    }
+    for latency_ms in latencies_ms
+  ] + [


Not sure if it's already correct based on just configuration, but 'make clean-locally' should be run after each 'test' on each 'latencies', to avoid current state of crdb "leaking" between 'test'.

('test' = measure on one specific latency there).

Yes, make clean-locally is run after all measurements for a particular latency are complete

the-glu · 2026-07-13T07:00:20Z

+          } for latency_ms in latencies_ms
+        ],
+      },
+    },


Would it be possible to show with that:

average between dss ?

latency / load_factor swapped ?

(as examples in the file)

Yes, but adding features would presumably be in a different PR -- this one is already enormous :)

the-glu · 2026-07-13T07:05:53Z

+from monitoring.benchmarker.configurations.loads import BenchmarkLoadName
+
+
+class BenchmarkScenarioName(str):


nit/general meta question unrelated to the PR: why use a specific type instead of just str?

I think that's a valuable question. There are lots of ways to use strings to encode information and not all ways of encoding information in a str are valid in the places where this type is used. Making a class like this allows clear indication of exactly what kind of string (what it includes, what it encodes, what it does) is expected. Previously, this codebase used something like BenchmarkScenarioName = str, but straight equality like that made it more difficult for type checkers and IDEs to show the right hints and check the right things.

the-glu · 2026-07-13T07:08:08Z

+
+    Benchmark configuration must contain a `resources.astm.f3548.v21.DSSInstanceResource` resource with each of these names."""
+
+    dss_selection_strategy: Optional[ASTMDSSSelectionStrategy]


What is the default? Because I think we want random in most of the cased to avoid issue with CRDB placement that can be random.

My intent is for this field to be required if the dss_pool contains more than 1 instance (raise a ValueError if len(dss_pool) > 1 && dss_selection_strategy is missing or None). But, there's no reason to specify this field if dss_pool contains just one instance.

the-glu · 2026-07-13T07:19:36Z

+from monitoring.uss_qualifier.resources.definitions import ResourceCollection
+
+
+class BenchmarkConfiguration(ImplicitDict):


nit: Should we have a name there for identification of various config?

I'm not sure what we would do with that. Most other names are effectively handles/pointers to objects so they can be used multiple times within the configuration without duplication. I don't think we need that for the configuration itself. And, it will already have some kind of name by virtue of the path to the file it is defined in.

That was more to display it e.g. in the graph / dumped data if data, to be able to track origin of a specific graph, like identifiers in a report (eventually with the hash/dirty flag to keep track of version, should included test evolve with time)

the-glu · 2026-07-13T07:23:58Z

+      flux to remain in steady-state after `end_time` until completion of the last operation started before
+      `end_time` for the throughput calculation to be valid.  Instead, the partial work of operations in progress
+      at `end_time` effectively discarded by this approach should be (statistically) exactly balanced by the
+      partial work included "for free" of operations started before `start_time` that end within the time window.


That fine if the "load" is not cut at the end of the test (witch locust unfortunately do.), will it be correct there?

Benchmarker doesn't have a "test" entity currently. Each scenario winds down its users before exiting, so that means the entire benchmark run completes with users wound down. But, user_ramp doesn't wind down any users in between steps within a scenario.

I don't think load cutting at the end matters for this though; the question is whether load is sufficiently stable before beginning the sample period, and that's the purpose of the throughput validity criteria. Basically, a step starts which causes new users to be spun up (existing users continue doing what they were doing). Then, the engine waits until the throughput validity criteria are met and only starts the throughput measurement period at that point. The throughput validity criteria defined for this PR are just that every user has completed at least 1 flight. I think that should definitely get us into steady-state without waiting longer than necessary.

I'm not sure I understood the mechanism, but I will check in next PR with the code, that probably easier.

Co-authored-by: Maximilien Cuony <the-glu@users.noreply.github.com>

barroco

LGTM

barroco · 2026-07-13T19:23:25Z

+    """Shell command to run as a benchmark action."""
+
+    command: str
+    """Shell command to run."""


@BenjaminPelletier just a remark about security but I don't know how address it yet.
The fact that arbitrary command line could be run via configuration may be a risk for users.
We should think about sandboxing the execution. What do you think ?

If any of our users were going to provide execution of benchmarker as a service using configurations they didn't control or approve, I agree this would need to be mitigated as a security concern. However, I expect benchmarker to generally be run by a user that already has full shell access, so automating shell commands doesn't seem like an escalation in privilege. Certainly all users should be careful about running code they're unfamiliar with and a benchmarker configuration is fairly close to code (without the restriction of requiring a benchmark designer to commit code into this repo before enabling others to run their benchmark). Sandboxing is difficult as one of the key initial use cases is automating container orchestration for a benchmark sweep against internode latency. But, I think we can certainly add features that could constrain actions (limiting capabilities) to fit within a certain security boundary if that were useful.

I agree this should be noted in the README when written.

…1567) Co-authored-by: Maximilien Cuony <the-glu@users.noreply.github.com> ebf1e29

Add benchmarker configuration and report definitions

966026a

github-project-automation Bot added this to InterUSS Backlog Jul 13, 2026

BenjaminPelletier marked this pull request as ready for review July 13, 2026 07:02

the-glu reviewed Jul 13, 2026

View reviewed changes

BenjaminPelletier and others added 5 commits July 13, 2026 10:29

Update monitoring/benchmarker/configurations/artifacts/raw_report.py

5d785c1

Co-authored-by: Maximilien Cuony <the-glu@users.noreply.github.com>

Update monitoring/benchmarker/configurations/loads.py

a1ff4c5

Co-authored-by: Maximilien Cuony <the-glu@users.noreply.github.com>

Update monitoring/benchmarker/configurations/loads.py

f0ae25e

Co-authored-by: Maximilien Cuony <the-glu@users.noreply.github.com>

Update monitoring/benchmarker/configurations/loads.py

4a20965

Co-authored-by: Maximilien Cuony <the-glu@users.noreply.github.com>

Fix resource type comment

645d179

barroco approved these changes Jul 13, 2026

View reviewed changes

barroco reviewed Jul 13, 2026

View reviewed changes

BenjaminPelletier merged commit ebf1e29 into interuss:main Jul 13, 2026
23 checks passed

BenjaminPelletier deleted the benchmarker-config branch July 13, 2026 19:45

github-project-automation Bot moved this to Done in InterUSS Backlog Jul 13, 2026

github-actions Bot added a commit that referenced this pull request Jul 13, 2026

[benchmarker] Add benchmarker configuration and report definitions (#…

a36a056

…1567) Co-authored-by: Maximilien Cuony <the-glu@users.noreply.github.com> ebf1e29

		from monitoring.benchmarker.configurations.loads import BenchmarkLoadName


		class BenchmarkScenarioName(str):


		Benchmark configuration must contain a `resources.astm.f3548.v21.DSSInstanceResource` resource with each of these names."""

		dss_selection_strategy: Optional[ASTMDSSSelectionStrategy]

		from monitoring.uss_qualifier.resources.definitions import ResourceCollection


		class BenchmarkConfiguration(ImplicitDict):

Uh oh!

Conversation

BenjaminPelletier commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

barroco left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

BenjaminPelletier commented Jul 13, 2026 •

edited

Loading