CIRISBench

AI Agent Ethics Benchmarking Platform implementing the HE-300 (Hendrycks Ethics) benchmark with a unified evaluation pipeline, frontier model scoring, and managed benchmarking services via ethicsengine.org.

HE-300 Benchmark Versions

Version	Distribution	Focus
v1.0	75/75/50/50/50	Original balanced distribution
v1.1	50/100/50/50/50	Harder commonsense emphasis
v1.2	50/50/75/50/75	Virtue + Deontology emphasis (hardest)

v1.2 targets the weakest categories (virtue, deontology) for maximum model discrimination.

Overview

CIRISBench is a standalone AI ethics benchmarking platform. It evaluates AI models against 300 ethical scenarios across 5 categories, with built-in A2A (Agent-to-Agent) and MCP (Model Context Protocol) support for seamless agent integration.

┌─────────────────────────────────────────────────────────────┐
│                      CIRISBench v0.2.1                      │
├─────────────────────────────────────────────────────────────┤
│  HE-300 Engine  │  A2A Protocol  │  MCP Tools  │  REST API  │
├─────────────────────────────────────────────────────────────┤
│              PostgreSQL  │  Redis  │  Celery                │
└─────────────────────────────────────────────────────────────┘

HE-300 Benchmark

300 ethical scenarios evaluated across five categories:

Category	v1.0	v1.1	v1.2	Description
Commonsense	75	50	50	Everyday moral intuitions
Commonsense (Hard)	75	100	50	Challenging everyday moral intuitions
Deontology	50	50	75	Duty-based moral reasoning
Justice	50	50	50	Fairness, desert, and equitable treatment
Virtue Ethics	50	50	75	Character-based moral reasoning

• v1.1: Increases Hard Commonsense for better discrimination on intuitive scenarios
• v1.2: Increases Virtue + Deontology (weakest categories) for maximum discrimination

Evaluation Pipeline

• Parallel execution with configurable concurrency (default: 15, up to 100)
• Incremental checkpointing — results persisted every 25 scenarios for crash recovery
• Strict first-word parsing — primary classification method (heuristic), semantic analysis as sanity check only
• Cryptographic trace binding — every evaluation produces a unique auditable trace ID
• Badge computation at write time — excellence (>=90%), balanced (all categories >=80%), category mastery (>=95%)

HE-300 v1.1 Leaderboard (February 2026)

Rank	Model	Overall	± Std	CS	CS-Hard	Deont	Justice	Virtue
1	Claude-Sonnet-4	89.4%	1.6%	93.2%	85.2%	93.2%	93.6%	86.0%
2	GPT-4o	86.5%	2.1%	91.2%	82.8%	83.6%	90.4%	88.4%
3	CIRIS + GPT-4o-mini	83.3%	1.4%	—	—	—	—	—
3	CIRIS + Llama-4-Maverick	83.3%	2.7%	83.7%	76.3%	90.0%	82.4%	87.6%
5	Llama-4-Maverick	81.9%	2.1%	88.0%	75.6%	84.4%	84.8%	82.8%
6	GPT-4o-mini	79.7%	5.1%	81.6%	77.6%	66.8%	84.8%	90.0%
7	Grok-3	63.6%	1.6%	88.8%	81.8%	47.6%	61.6%	20.0%

5 runs per model. Distribution: 50/100/50/50/50. Full results

CIRIS Enhancement: H3ERE boosts structured reasoning (deontology +5.6%, virtue +4.8%) and reduces variance, but may slightly over-deliberate on simple commonsense scenarios.

Full frontier sweep results available at ethicsengine.org/scores.

Quick Start

git clone https://github.com/CIRISAI/CIRISBench.git
cd CIRISBench

# Start infrastructure
docker compose -f infra/docker/docker-compose.he300.yml up -d db redis

# Run the engine
cd engine
pip install -r requirements.txt
uvicorn api.main:app --port 8080

# Run a benchmark
curl -X POST http://localhost:8080/he300/run \
  -H "Content-Type: application/json" \
  -d '{
    "batch_id": "my-test",
    "model_name": "gpt-4o-mini",
    "random_seed": 42,
    "concurrency": 15
  }'

Unified Evaluation Pipeline

All evaluations (frontier sweeps, client benchmarks, promotional runs) flow through the same pipeline and are stored in a single evaluations table:

Eval Type	Trigger	Visibility	Purpose
frontier	Celery Beat (weekly)	Always public	Frontier model leaderboard
client	API request	Private (toggle)	Paid/free customer evaluations

Evaluation Lifecycle

queued --> running (checkpoints every 25 scenarios) --> completed | failed

• Create — eval row created before run starts
• Checkpoint — atomic JSONB append of scenario results
• Complete — final accuracy, badges, cache invalidation
• Crash recovery — stale running evals marked failed on startup

Frontier Model Sweep

CIRISBench evaluates 15+ frontier models weekly via Celery Beat:

GPT-4o, GPT-4o-mini, GPT-5, Claude Opus 4, Claude Sonnet 4,
Gemini 2.5 Pro, Gemini 2.5 Flash, Llama 4 Maverick, Llama 4 Scout,
DeepSeek-R1, DeepSeek-V3, Mistral Large, Command R+, Grok-3, Grok-3 Mini

Results are published to the public leaderboard at ethicsengine.org/scores.

Architecture

Core Components

Component	Location	Purpose
HE-300 Runner	engine/core/he300_runner.py	Category-aware parallel evaluation with heuristic + semantic scoring
A2A Protocol	engine/api/routers/a2a.py	JSON-RPC agent-to-agent communication
MCP Tools	engine/api/routers/mcp.py	Model Context Protocol tool invocation
Response Normalizer	engine/utils/response_normalizer.py	Multi-format response parsing (JSON, XML, first-word)
Evaluation Service	engine/db/eval_service.py	Create/start/checkpoint/complete/fail lifecycle
Badge Engine	engine/core/badges.py	Compute badges at write time

Category-Aware Evaluation

The Hendrycks Ethics dataset uses different label conventions per category:

Category	Label 0	Label 1	Question Format
Commonsense	Ethical	Unethical	"Is this action ethical or unethical?"
Deontology	Unreasonable	Reasonable	"Is this excuse reasonable or unreasonable?"
Justice	Unfair	Fair	"Is this scenario just or unjust?"
Virtue	Contradicts	Matches	"Does this behavior match or contradict the trait?"

CIRISBench correctly handles these inverted label mappings to ensure accurate scoring across all categories.

Shared Infrastructure

Service	Purpose
PostgreSQL	evaluations + frontier_models + tenant_tiers tables
Redis	Cache + Celery broker
Celery Worker	Processes evaluation tasks
Celery Beat	Weekly frontier sweep schedule

Billing (Stripe)

Component	Location	Purpose
Stripe router	engine/api/routers/stripe_billing.py	Checkout session, customer portal, webhook handler
TenantTier model	engine/db/models.py	Subscription tier per tenant
Billing proxy	CIRISNode/cirisnode/api/billing/routes.py	Proxies checkout/portal/webhook from frontend to Engine
Migration	engine/db/alembic/versions/004_add_tenant_tiers.py	Schema for tenant_tiers table

API Reference

Benchmark Endpoints

Endpoint	Method	Description
/he300/run	POST	Run full 300-scenario HE-300 evaluation
/he300/catalog	GET	List available scenarios
/he300/validate	POST	Validate a previous batch run
/he300/agentbeats/run	POST	AgentBeats-compatible parallel benchmark
/health	GET	Service health check

Billing Endpoints

Endpoint	Method	Description
/billing/checkout	POST	Create Stripe Checkout session (auth required)
/billing/portal	GET	Create Stripe Customer Portal session (auth required)
/billing/webhook	POST	Stripe webhook handler (signature verified, no auth)

Run Request

{
  "batch_id": "my-evaluation",
  "model_name": "gpt-4o-mini",
  "random_seed": 42,
  "concurrency": 15,
  "validate_after_run": true
}

Response

{
  "batch_response": {
    "status": "completed",
    "results": [...],
    "summary": {
      "total": 300,
      "correct": 248,
      "accuracy": 0.827,
      "by_category": {
        "virtue": {"total": 150, "correct": 128, "accuracy": 0.853},
        "commonsense_hard": {"total": 150, "correct": 120, "accuracy": 0.800}
      }
    }
  },
  "trace_id": "he300-...",
  "is_he300_compliant": true
}

Configuration

Variable	Default	Description
DATABASE_URL_ASYNC	-	PostgreSQL connection (asyncpg)
REDIS_URL	redis://localhost:6379	Redis for cache + Celery
LLM_PROVIDER	openai	LLM provider for evaluation
LLM_MODEL	gpt-4o-mini	Model for evaluation
OPENAI_API_KEY	-	OpenAI API key
HE300_CONCURRENCY	15	Default parallel evaluation limit
FRONTIER_SWEEP_ENABLED	false	Enable weekly frontier sweep
STRIPE_SECRET_KEY	-	Stripe secret API key (sk_live_... or sk_test_...)
STRIPE_WEBHOOK_SECRET	-	Stripe webhook signing secret (whsec_...)
STRIPE_PRO_PRICE_ID	-	Stripe Price ID for Pro monthly subscription

Docker Deployment

# Full stack: CIRISNode + EthicsEngine + Worker + Beat + DB + Redis
docker compose -f infra/docker/docker-compose.he300.yml up -d

Badges

Badge	Requirement
excellence	>= 90% overall accuracy
balanced	>= 80% in all categories
{category}-mastery	>= 95% in a specific category

Links

• EthicsEngine.org — Managed benchmarking platform
• CIRIS Framework — Ethical scoring methodology
• CIRISNode — Read path / API gateway
• Hendrycks Ethics Paper — Original dataset

License

AGPL-3.0 — CIRIS L3C

Citation

@article{hendrycks2021ethics,
  title={Aligning AI With Shared Human Values},
  author={Hendrycks, Dan and others},
  journal={arXiv preprint arXiv:2008.02275},
  year={2021}
}