Skip to content

Add FJSP-WF task for manufacturing scheduling#89

Open
y-ji24 wants to merge 3 commits into
EinsiaLab:mainfrom
y-ji24:main
Open

Add FJSP-WF task for manufacturing scheduling#89
y-ji24 wants to merge 3 commits into
EinsiaLab:mainfrom
y-ji24:main

Conversation

@y-ji24

@y-ji24 y-ji24 commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR adds a new Frontier-Eng task for Flexible Job Shop Scheduling with Worker Flexibility (FJSP-WF) under the Manufacturing domain.

Highlights

  • Adds a manufacturing scheduling task based on official FJSSP-W benchmark instances.
  • Uses solver/scheduler.py as the editable Agent artifact with EVOLVE-BLOCK protection.
  • Provides a deterministic evaluator with feasibility validation and relative scoring.
  • Includes a baseline scheduler for reproducible evaluation.
  • Adds task documentation, references, benchmark metadata, and Frontier-Eng integration files.

Verification

  • Baseline and solver schedules are feasible on all benchmark instances.
  • Evaluator produces deterministic scores.
  • Benchmark metadata and instance files have been validated.

c7w and others added 3 commits June 30, 2026 20:37
- Medal Score: peer-relative gold/silver/bronze podium (normalized to [0,1]),
  reported on v1 (47 tasks) and the v1-lite subset (10 tasks). READMEs now lead
  with Medal Score; average rank stays on the website leaderboard.
- leaderboard/: ship the frozen podium baselines (medal_podium.csv), published
  leaderboard (medal_leaderboard.csv), raw score table (exp1_models_raw.csv),
  a submission scorer (score_submission.py), and an example submission.
  Un-ignore leaderboard/*.csv.
- v1-lite: add frontier_eval/conf/batch/v1_lite.yaml (10-task subset across all
  five categories, distinct families, gradual-improvement tasks).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

  • Core Purpose: This PR introduces a significant update to the evaluation framework, including a new scoring metric ("Medal Score"), a lightweight benchmark subset (v1-lite), and a new complex optimization task (Manufacturing/FJSP-WF).
  • Modified File Structure & Modifications:
    • .gitignore: Whitelisted leaderboard/*.csv and the new v1_lite.yaml configuration.
    • README.md & README_zh-CN.md: Updated with the 2026-06-30 news, detailed explanation of the Medal Score metric, introduction of the v1-lite subset, and a revamped leaderboard table.
    • benchmarks/Manufacturing/FJSP-WF/README.md: Provided task overview, engineering background, and quick-start commands.
    • benchmarks/Manufacturing/FJSP-WF/Task.md: Detailed technical specification of the Flexible Job Shop Scheduling problem, including constraints, I/O formats, and validation rules.
    • benchmarks/Manufacturing/FJSP-WF/baseline/scheduler.py: Implemented a greedy Earliest Start Time + Shortest Processing Time (EST+SPT) baseline algorithm.
    • benchmarks/Manufacturing/FJSP-WF/data/benchmark_instances.json: Registry for 30 official and 3 synthetic instances.

2. AI Content Analysis

  • Estimated AI Component: 25%
  • Reasoning & Evidence: The documentation (Task.md) and the baseline code (scheduler.py) exhibit high structural consistency and professional phrasing typical of AI-assisted technical writing. The solve_instance function follows a very standard, clean boilerplate pattern for greedy algorithms. However, the domain-specific nuances of the FJSP-WF problem (worker-machine eligibility matrices) and the specific integration with the frontier_eval framework suggest significant human oversight and domain expertise.

3. Engineering & Economic Assessment

  • Engineering Reality Check: High. FJSP-WF is a classic NP-hard problem in operations research. The PR moves beyond "toy" examples by incorporating 30 official instances from the GECCO 2026 competition. The validation logic is robust, checking for job precedence, machine/worker overlaps, and eligibility constraints, which are critical for production-grade scheduling.
  • Economic Value: High. Production scheduling optimization is a multi-billion dollar challenge in manufacturing. Improving makespan by even small percentages directly impacts throughput and energy efficiency. The introduction of the "Medal Score" also provides higher economic signal by rewarding models that reach the "frontier" of known optimal solutions rather than rewarding noise in sub-optimal ranges.

4. Quality Assurance

  • Verification & Testing:
    • frontier_eval Integration: Yes.
    • task_name: Manufacturing/FJSP-WF
    • Execution & Dependencies: The README.md and Task.md provide explicit CLI commands for both standalone evaluation and framework-level execution. Environment dependencies are limited to the Python standard library, ensuring high portability.
  • Documentation Quality: Excellent. The documentation covers the "Why" (Engineering Background) as well as the "How".
    • Correction: In Task.md, the header ## Instance Data is duplicated consecutively.
  • Organizational Structure: Logical and scalable. The task follows a clear hierarchy: baseline/, data/, solver/, and verification/, which is consistent with the existing repository architecture.

5. Security & Privacy Check

  • Sensitive Files: Clean. The .gitignore was proactively updated to prevent accidental leaks of local logs or IDE configs.
  • Absolute Paths: None detected. The code uses Path(__file__).resolve() and environment variables (FJSPWF_BENCHMARK_JSON) to handle paths dynamically.

🇨🇳 中文分析

1. 摘要

  • 核心目的: 此 PR 为评测框架引入了重大更新,包括新的评分维度(“勋章分” Medal Score)、轻量化评测子集(v1-lite)以及一个新的复杂优化任务(Manufacturing/FJSP-WF)。
  • 修改的文件结构与变更摘要:
    • .gitignore: 将 leaderboard/*.csv 和新的 v1_lite.yaml 配置加入白名单。
    • README.md & README_zh-CN.md: 更新了 2026-06-30 的新闻动态,详细解释了 Medal Score 指标,介绍了 v1-lite 子集,并更新了排行榜表格。
    • benchmarks/Manufacturing/FJSP-WF/README.md: 提供了任务概览、工程背景和快速启动命令。
    • benchmarks/Manufacturing/FJSP-WF/Task.md: 详细说明了柔性车间调度(FJSP)问题的技术规范,包括约束条件、输入输出格式和验证规则。
    • benchmarks/Manufacturing/FJSP-WF/baseline/scheduler.py: 实现了一个基于最早开始时间+最短处理时间(EST+SPT)的贪心基准算法。
    • benchmarks/Manufacturing/FJSP-WF/data/benchmark_instances.json: 注册了 30 个官方实例和 3 个合成实例。

2. AI 成分分析

  • 预估 AI 含量: 25%
  • 判断依据与证据: 文档(Task.md)和基准代码(scheduler.py)表现出 AI 辅助技术写作典型的高结构一致性和专业措辞。solve_instance 函数遵循了非常标准、整洁的贪心算法模板。然而,FJSP-WF 问题的领域特定细节(如工人-机器资格矩阵)以及与 frontier_eval 框架的特定集成表明存在显著的人工监督和领域专业知识。

3. 工程与经济评估

  • 工程现实检验: 高。FJSP-WF 是运筹学中经典的 NP-hard 问题。该 PR 通过引入 GECCO 2026 竞赛的 30 个官方实例,超越了“玩具级”示例。验证逻辑稳健,检查了作业优先级、机器/工人重叠以及资格约束,这些对于生产级调度至关重要。
  • 经济价值: 高。生产调度优化是制造业中价值数十亿美元的挑战。即使是微小的完工时间(makespan)改进也能直接影响吞吐量和能源效率。引入“勋章分”通过奖励达到已知最优解“前沿”的模型,而非奖励次优区间的噪声,提供了更高的经济信号。

4. 质量保证

  • 验证与测试:
    • frontier_eval 集成: 是
    • task_name: Manufacturing/FJSP-WF
    • 运行与依赖: README.mdTask.md 提供了明确的 CLI 命令,支持独立评估和框架级执行。环境依赖仅限于 Python 标准库,确保了高移植性。
  • 文档质量: 优秀。文档涵盖了“为什么”(工程背景)以及“怎么做”。
    • 纠错: 在 Task.md 中,## Instance Data 标题连续重复出现了两次。
  • 组织结构: 逻辑清晰且具备可扩展性。任务遵循明确的层级结构:baseline/data/solver/verification/,与现有仓库架构保持一致。

5. 安全与隐私检查

  • 敏感文件: 未发现异常。.gitignore 已主动更新,以防止本地日志或 IDE 配置意外泄露。
  • 绝对路径: 未检测到。代码使用 Path(__file__).resolve() 和环境变量(FJSPWF_BENCHMARK_JSON)来动态处理路径。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants