Add adaptive compressed telemetry execution benchmark

[hyizhak]

Background

Production telemetry pipelines must trade storage footprint against ingestion, decoding, and query CPU. Existing analytical placement tasks cannot test whether a candidate implementation actually achieves those trade-offs. This contribution adds a CPU-executed benchmark in which candidates co-design a lossless telemetry block format and exact query operators.

What changed

• Add ComputerSystems/AdaptiveCompressedTelemetryExecution, an execution-backed C++20 task over five correlated telemetry columns.
• Compile and run candidate encode, decode, and compressed-query implementations on three deterministic workloads.
• Measure actual encoded bytes and pinned-CPU timings in separate processes, while excluding compilation and file I/O from timed regions.
• Verify the complete decoded stream byte-for-byte and compare every query result with an independent Python oracle.
• Convert measured storage and CPU work into a continuous cost ratio, so correct incomplete optimizations receive partial credit.
• Add resource limits, a raw starter and baseline log, unified Frontier metadata, and English/Chinese task-index entries.

The task uses only Python's standard library and g++; it requires no database, service, network access, accelerator, task-specific package, or Docker image.

Sources

• OpenTelemetry Logs Data Model
• FastLanes: A SIMD-Friendly Composable Compression Library
• MorphStore: Analytical Query Engine with a Holistic Compression-Enabled Processing Model

Verification

Direct evaluation from the task directory:

cd benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution
python verification/evaluator.py scripts/init.cpp

Unified evaluation from the repository root:

python -m frontier_eval \
  task=unified \
  task.benchmark=ComputerSystems/AdaptiveCompressedTelemetryExecution \
  algorithm=openevolve \
  algorithm.iterations=0

Validation performed before submission:

• Raw starter: valid=1, correctness=1, combined_score=1.0 across all three scenarios.
• Unified zero-iteration evaluation: passed with score 1.0.
• Alternate-seed evaluation: passed.
• Invalid-decode and timeout diagnostics: correctly rejected.
• A simple diagnostic codec achieved exact results, a 0.232 encoded/raw ratio, and repeat scores of 2.88x and 2.90x, demonstrating optimization headroom.
• JSON validation, staged-diff whitespace checks, and starter/baseline identity checks: passed.

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new engineering benchmark task titled AdaptiveCompressedTelemetryExecution under the ComputerSystems category. The task focuses on the co-design of telemetry data compression and query execution, simulating real-world cloud observability challenges.
• Modified File Structure & Modifications:
  • TASK_DETAILS.md & TASK_DETAILS_zh-CN.md: Updated the task table to include the new benchmark.
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/README.md: Provides high-level overview, requirements, and execution commands.
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/Task.md: Detailed technical specification, including schema, API, and the economic scoring model.
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/scripts/init.cpp: The editable starter code for agents, implementing a basic uncompressed baseline.
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/verification/: Contains the core evaluation logic, including codec_api.h (ABI), benchmark_driver.cpp (performance measurement), and evaluator.py (orchestration).
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/references/problem_config.json: Defines workload parameters, pricing coefficients, and resource limits.
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/frontier_eval/: Metadata files for integration with the unified evaluation framework.

2. AI Content Analysis

• Estimated AI Component: 15%
• Reasoning & Evidence: The high-level architectural design, the specific choice of the "cost-per-logical-TiB" metric, and the integration of CPU pinning/affinity in the driver suggest human expert design. However, the C++ starter code in scripts/init.cpp and baseline/solution.cpp exhibits highly standard, boilerplate patterns (e.g., the load_value helper and the switch statement for query kinds) that are typical of AI-assisted scaffolding. The documentation is exceptionally clean and follows a structured template seen in other tasks in this repo, likely refined by AI.

3. Engineering & Economic Assessment

• Engineering Reality Check: Production-grade. This task addresses a non-trivial problem in distributed systems: the trade-off between storage density and query latency. It moves beyond simple compression (like Zstd/Gzip) by requiring "query-aware" formats, similar to industry-leading research like FastLanes or Apache Parquet/ORC optimizations. It correctly handles edge cases such as memory limits (1 GiB) and CPU isolation.
• Economic Value: High. In modern cloud infrastructure, telemetry (logs/metrics) is one of the largest cost drivers. A solution that improves the compression ratio while maintaining high query throughput directly translates to significant reductions in cloud spend (S3 storage + EC2 compute).

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: AdaptiveCompressedTelemetryExecution
  • Execution & Dependencies: The README.md clearly documents the requirements (Python 3.10+, g++ C++20, Linux) and provides the exact commands for both direct and unified evaluation.
• Documentation Quality: Excellent. The Task.md is comprehensive, providing mathematical formulas for the economic objective and clear definitions of the logical schema. No redundant information or grammatical errors were detected in the provided diff.
• Organizational Structure: Logical and Scalable. The separation of verification (immutable), scripts (mutable), and references (config) follows best practices for benchmark suites.

5. Security & Privacy Check

• Sensitive Files: Clean. No .env, API keys, or temporary build artifacts (__pycache__, .o files) were included.
• Absolute Paths: None detected. The scripts and documentation use relative paths or environment-agnostic commands.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 在 ComputerSystems 类别下引入了一个名为 AdaptiveCompressedTelemetryExecution 的新工程基准测试任务。该任务侧重于遥测数据压缩与查询执行的协同设计，模拟了现实世界中云原生可观测性面临的挑战。
• 修改的文件结构与变更摘要:
  • TASK_DETAILS.md & TASK_DETAILS_zh-CN.md: 更新任务列表，加入新的基准测试。
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/README.md: 提供高层概述、环境要求和运行命令。
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/Task.md: 详细的技术规范，包括数据 Schema、API 定义和经济评分模型。
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/scripts/init.cpp: 供 Agent 编辑的初始代码，实现了一个基础的未压缩基准。
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/verification/: 包含核心评估逻辑，如 codec_api.h (ABI)、benchmark_driver.cpp (性能测量) 和 evaluator.py (自动化脚本)。
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/references/problem_config.json: 定义了工作负载参数、定价系数和资源限制。
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/frontier_eval/: 用于集成到统一评估框架的元数据文件。

2. AI 成分分析

• 预估 AI 含量: 15%
• 判断依据与证据: 高层架构设计、特定的“每逻辑 TiB 成本”指标以及驱动程序中 CPU 亲和性的设置表明了人类专家的设计。然而，scripts/init.cpp 和 baseline/solution.cpp 中的 C++ 初始代码表现出非常标准的模板化模式（例如 load_value 辅助函数和查询类型的 switch 语句），这是典型的 AI 辅助生成的代码。文档非常整洁，遵循了仓库中其他任务的结构化模板，可能经过 AI 润色。

3. 工程与经济评估

• 工程现实检验: 生产级。 该任务解决了分布式系统中一个非平庸的问题：存储密度与查询延迟之间的权衡。它超越了简单的压缩（如 Zstd/Gzip），要求实现“查询感知”的格式，类似于 FastLanes 或 Apache Parquet/ORC 优化等业界领先的研究。它正确处理了边缘情况，如内存限制 (1 GiB) 和 CPU 隔离。
• 经济价值: 高。 在现代云基础设施中，遥测数据（日志/指标）是最大的成本驱动因素之一。如果一个解决方案能在保持高查询吞吐量的同时提高压缩率，将直接转化为云支出的显著降低（S3 存储 + EC2 计算）。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是
  • task_name: AdaptiveCompressedTelemetryExecution
  • 运行与依赖: README.md 清晰地记录了环境要求（Python 3.10+、g++ C++20、Linux），并提供了直接评估和统一评估的确切命令。
• 文档质量: 优秀。 Task.md 内容详尽，提供了经济目标的数学公式和逻辑 Schema 的清晰定义。在提供的 Diff 中未检测到冗余信息或语法错误。
• 组织结构: 逻辑清晰且具备可扩展性。 将 verification（不可变）、scripts（可变）和 references（配置）分离，符合基准测试套件的最佳实践。

5. 安全与隐私检查

• 敏感文件: 未发现异常。 未包含 .env、API 密钥或临时构建产物（__pycache__、.o 文件）。
• 绝对路径: 未检测到。 脚本和文档均使用相对路径或与环境无关的命令。

[wrh-human]

Review — AdaptiveCompressedTelemetryExecution (PR #94)

Thank you for contributing this benchmark! Telemetry compression is a real engineering domain, and the overall direction is solid. Several issues need to be addressed before merging.

---

1. Domain, Economic Value, and Frontier-Eng Fit ✅

Telemetry compression in computer systems is a genuine engineering problem — the storage and query cost of telemetry data at scale is a real pain point in the industry. The task requires candidates to design block-level codecs and adaptive compression strategies, starting from a feasible baseline and iteratively improving via a verifier. This aligns with Frontier-Eng's positioning.

2. Not purely numerical ✅

The agent writes C++ encode/decode/query functions, requiring design of data layouts, compression algorithms, and query optimization strategies — not parameter tuning.

3. Search space ✅

Block-level columnar encoding, adaptive strategy selection, and SIMD vectorization provide a large enough design space that brute-force search is infeasible.

4. Evaluator and engineering verification ✅

The evaluator compiles and actually runs the candidate code, measuring wall-clock time on pinned CPUs (with warmup and measured rounds), rather than relying on analytical prediction. Verification includes:

• Byte-for-byte comparison of decoded output against the original input
• Raw input deleted before decoding, preventing the candidate from cheating via file references
• Query results compared against an independent Python oracle
• Economic cost model (storage price + CPU price)
• Deterministic seeds, fully reproducible

5. Constraints and verifier enforcement ✅

problem_config.json defines a complete set of constraints. _run_driver sets RLIMIT_AS, RLIMIT_CPU, and RLIMIT_FSIZE via preexec_fn. However, RLIMIT_NPROC is missing.

6. Baseline experiment ✅

result_log.txt is included, showing valid=1.0 and combined_score=1.0 for the baseline.

---

Issues to address before merging

Issue 1 (most critical): Evaluator lacks EVOLVE-BLOCK boundary validation

The evaluator passes the candidate source directly to the compiler (around line 634) without any check that the candidate has not modified code outside the EVOLVE-BLOCK markers. constraints.txt explicitly requires "Only modify code between the EVOLVE-BLOCK markers" (point 1), but the evaluator never enforces this.

As implemented, the evaluator allows the agent to:

• Modify the encode_block / decode_block / query_block function signatures (as long as compilation succeeds)
• Add global state outside the EVOLVE-BLOCK
• Add extra #include directives
• Modify or remove the EVOLVE-BLOCK markers themselves

Suggestion: add source validation before compilation to verify that code outside the EVOLVE-BLOCK matches the baseline.

Issue 2: No test file

This PR has zero tests — no test_evaluator.py exists. At minimum, the following should be added:

• An end-to-end test verifying the baseline returns valid=1.0 and combined_score=1.0
• Tests for encode/decode/query correctness
• A test rejecting EVOLVE-BLOCK boundary violations (after Issue 1 is fixed)

Issue 3: _preexec_limits lacks RLIMIT_NPROC

RLIMIT_AS, RLIMIT_CPU, and RLIMIT_FSIZE are set, but RLIMIT_NPROC is missing — a candidate could fork child processes without limit within the evaluation window. Please add:

try:
    resource.setrlimit(resource.RLIMIT_NPROC, (64, 64))
except (AttributeError, ValueError, OSError):
    pass

Issue 4: Workload loop lacks per-workload error isolation

The scenario loop (around line 650) has no try/except. If scenario 2 fails (candidate crash or decode mismatch), the entire evaluation terminates with valid=0 and no partial results retained.

Issue 5: result_log.txt lacks environment details and per-workload breakdown

The log only contains aggregate JSON metrics. It is missing:

• Environment information (CPU model, kernel version, etc.)
• Per-scenario throughput data (encode/decode/query GiB/s)
• A headroom check demonstrating optimization headroom beyond the baseline

---

Non-blocking suggestion

The task compiles with -O3 -march=native, which means baseline results are host-dependent (different CPU instruction sets produce different throughput). Please document this explicitly in the README to avoid ambiguity when comparing leaderboard results.

---

Summary

The core direction and engineering verification design are sound. Issue 1 (missing EVOLVE-BLOCK validation) is the most critical — it directly affects benchmark integrity. All five issues should be fixed before merging.

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new benchmark task titled AdaptiveCompressedTelemetryExecution under the ComputerSystems category. The task focuses on the co-design of telemetry data compression and query execution, challenging participants to optimize for both storage footprint and computational efficiency (CPU time) in an OLAP-style context.
• Modified File Structure & Modifications:
  • TASK_DETAILS.md & TASK_DETAILS_zh-CN.md: Updated the task registry table to include the new entry under the ComputerSystems section.
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/README.md: Provides a high-level overview, requirements (C++20, Python 3.10), and execution commands.
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/Task.md: Detailed technical specification including the logical schema (5 columns), API definitions, workload descriptions, and the economic scoring formula.
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/baseline/solution.cpp & scripts/init.cpp: Provides a starter/baseline implementation using raw memory copies (no compression).
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/verification/: Contains the core infrastructure, including codec_api.h (ABI), benchmark_driver.cpp (C++ timing/execution), and evaluator.py (scoring/validation).
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/frontier_eval/: Metadata files for integration with the frontier_eval framework.
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/references/problem_config.json: Configuration for pricing coefficients and scenario parameters.

2. AI Content Analysis

• Estimated AI Component: 15%
• Reasoning & Evidence: The overall structure follows a highly standardized template consistent with other tasks in this repository, which may have been scaffolded by AI. However, the domain-specific depth—referencing academic papers like FastLanes and MorphStore, and the specific implementation of the OpenTelemetry Logs Data Model—indicates significant human expert intervention. The C++ boilerplate in init.cpp (e.g., std::memcpy for raw data handling) is standard, but the complex orchestration in the evaluator.py and benchmark_driver.cpp (handling CPU pinning and resource limits) reflects high-quality engineering rather than generic AI output.

3. Engineering & Economic Assessment

• Engineering Reality Check: This is a production-grade engineering problem. It addresses a non-trivial challenge in modern observability systems: the trade-off between compression ratios and query latency. The inclusion of CPU pinning (sched_setaffinity), memory limits (RLIMIT_AS), and deterministic workload generation (steady, bursty, incident) demonstrates a sophisticated understanding of systems benchmarking.
• Economic Value: High. Telemetry data (logs/metrics) is one of the largest cost drivers in cloud infrastructure. By modeling a "monthly cost per logical TiB" that combines storage and CPU costs, this PR provides a direct metric for technical debt reduction and cost optimization in real-world data platforms.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: AdaptiveCompressedTelemetryExecution
  • Execution & Dependencies: The README.md clearly documents the environment requirements (C++20, Linux-like host for CPU affinity) and provides exact commands for both direct evaluation (python verification/evaluator.py scripts/init.cpp) and unified evaluation via frontier_eval.
• Documentation Quality: High. The Task.md is exceptionally detailed, providing clear semantics for query types (CountEqual, SumWhereRange, etc.) and a transparent breakdown of the scoring algorithm. No spelling or grammatical errors were detected in the provided diff.
• Organizational Structure: The structure is logical, modular, and follows the repository's established hierarchy. Separating the verification logic from the baseline and scripts ensures a clean boundary for participants.

5. Security & Privacy Check

• Sensitive Files: Clean. No .env, API keys, or IDE-specific configurations were found.
• Absolute Paths: None detected. The scripts and documentation use relative paths (e.g., verification/evaluator.py).

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 在 ComputerSystems 类别下引入了一个名为 AdaptiveCompressedTelemetryExecution 的新基准测试任务。该任务侧重于遥测数据压缩与查询执行的协同设计，挑战参与者在 OLAP 场景下同时优化存储占用和计算效率（CPU 时间）。
• 修改的文件结构与变更摘要:
  • TASK_DETAILS.md & TASK_DETAILS_zh-CN.md: 更新了任务注册表，在 ComputerSystems 章节中添加了新条目。
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/README.md: 提供了任务概览、环境要求（C++20, Python 3.10）和运行命令。
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/Task.md: 详细的技术规范，包括逻辑架构（5 列）、API 定义、工作负载描述和经济评分公式。
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/baseline/solution.cpp & scripts/init.cpp: 提供了基于原始内存拷贝（无压缩）的初始/基准实现。
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/verification/: 包含核心基础设施，包括 codec_api.h (ABI)、benchmark_driver.cpp (C++ 计时/执行) 和 evaluator.py (评分/验证)。
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/frontier_eval/: 用于集成到 frontier_eval 框架的元数据文件。
  • benchmarks/ComputerSystems/AdaptiveCompressedTelemetryExecution/references/problem_config.json: 价格系数和场景参数的配置文件。

2. AI 成分分析

• 预估 AI 含量: 15%
• 判断依据与证据: 整体结构遵循了该仓库高度标准化的模板，这部分可能由 AI 生成框架。然而，领域特定的深度——引用了 FastLanes 和 MorphStore 等学术论文，以及对 OpenTelemetry 日志数据模型的具体实现——表明了显著的人工专家干预。init.cpp 中的 C++ 模板代码（如用于原始数据处理的 std::memcpy）较为常规，但 evaluator.py 和 benchmark_driver.cpp 中复杂的编排逻辑（处理 CPU 亲和性绑定和资源限制）反映了高质量的工程实践，而非通用的 AI 输出。

3. 工程与经济评估

• 工程现实检验: 这是一个生产级别的工程问题。它解决了现代可观测性系统中一个非平庸的挑战：压缩率与查询延迟之间的权衡。引入 CPU 亲和性绑定 (sched_setaffinity)、内存限制 (RLIMIT_AS) 以及确定性工作负载生成（稳定、突发、事故场景）展示了对系统基准测试的深刻理解。
• 经济价值: 高。遥测数据（日志/指标）是云基础设施中最大的成本驱动因素之一。通过建模结合了存储和 CPU 成本的“每逻辑 TiB 每月成本”，此 PR 为现实世界数据平台的技术债务减少和成本优化提供了直接的衡量指标。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是
  • task_name: AdaptiveCompressedTelemetryExecution
  • 运行与依赖: README.md 清晰地记录了环境要求（C++20，类 Linux 主机以支持 CPU 亲和性），并提供了直接评估（python verification/evaluator.py scripts/init.cpp）和通过 frontier_eval 进行统一评估的确切命令。
• 文档质量: 高。Task.md 非常详尽，提供了清晰的查询类型语义（CountEqual, SumWhereRange 等）和透明的评分算法分解。在提供的 diff 中未检测到拼写或语法错误。
• 组织结构: 文件组织逻辑清晰、模块化，并遵循仓库既有的层级结构。将 verification 逻辑与 baseline 和 scripts 分离，确保了参与者的操作边界清晰。

5. 安全与隐私检查

• 敏感文件: 未发现异常。未发现 .env、API 密钥或特定于 IDE 的配置。
• 绝对路径: 未检测到。脚本和文档均使用相对路径（例如 verification/evaluator.py）。

[hyizhak]

Thank you for the detailed review. I addressed all five issues and the non-blocking documentation suggestion in commit 2125ccc.

1. EVOLVE-BLOCK enforcement

   • The evaluator now requires exactly one ordered marker pair, decodes the source as UTF-8, and byte-compares both immutable shell portions with the frozen baseline before compilation.
   • It compiles a temporary copy of those already-validated bytes, avoiding a validation/compile mismatch.
   • A regression test confirms that an edit outside the block is rejected before compilation.

2. Evaluator tests

   • Added verification/test_evaluator.py with four tests covering the full baseline encode/decode/query pipeline, independent query-oracle results, shell-boundary rejection, a deliberately failing middle scenario that does not stop the final scenario, and application of the process limit.
   • The baseline test calls the complete evaluator and asserts valid=1.0, combined_score=1.0, and all three phases for all scenarios.

3. Process limit

   • Added frozen max_processes=64 configuration and RLIMIT_NPROC enforcement with portability guards.

4. Per-scenario failure isolation

   • Scenario setup, baseline execution, candidate execution, correctness checks, and scoring now retain scenario-level errors and continue through later scenarios.
   • Successful ratios remain visible in partial_combined_score; failed_scenarios and failure_summary are preserved in artifacts.
   • The official valid and combined_score remain zero if any scenario fails, preventing a candidate from improving its official result by intentionally skipping a difficult scenario.

5. Baseline evidence

   • Expanded baseline/result_log.txt with OS/CPU/compiler details and per-scenario compression ratio, encode/decode/query throughput, modeled cost, and score.
   • Added a real headroom experiment: a temporary fixed-width delta codec passed exact decode and query validation, scored 2.556401106833809, reduced compression ratio from 1.0000755310058593 to 0.32512435913085935, and reduced total modeled monthly cost from 74.1947212474412 to 29.255996084436042. The candidate is not committed as the starter.

The README now explicitly documents that -march=native makes absolute measurements host- and compiler-dependent and that comparisons should use the co-measured baseline on the same evaluator host.

Validation completed:

• 4/4 tests pass under Python 3.10 and Python 3.12;
• direct baseline evaluation returns valid=1.0, combined_score=1.0;
• UnifiedTask process-isolation evaluation returns valid=1.0, combined_score=1.0 and loads the scenario artifacts;
• Python compilation, JSON validation, shell syntax, and git diff --check pass.