Skip to content

[Awareness] Idle Spacebot Consumed Entire Monthly Token Plan in ~2 Days — Cortex Timer-Driven Synthesis #593

Description

@DogaOztuzun

TL;DR

Spacebot was installed for practice/testing. With zero active usage (only 2 chat inputs over 11 days), the default cortex tick interval of 60s burned through a $100–$200 monthly token subscription in approximately a few days. The token waste came from timer-driven synthesis running on unchanged memories/conversations.


Environment

  • Spacebot version during incident: v0.4.x (previous version)
  • Spacebot version now: v0.5.0
  • Config: Mostly defaults, cortex tick_interval_secs = 60
  • Usage pattern: Installed for practice, not actively used

What Happened — Timeline

Date Event
Apr 27 Spacebot deployed
Apr 29 Brief usage (2 chat messages + 1 worker run)
Apr 30 – May 6 Zero user activity
May 7 API key returns 401 Unauthorized — limit exhausted
May 8 Investigation reveals ~2B tokens consumed while idle

Root Cause: Timer-Driven Synthesis

The default [defaults.cortex] configuration:

[defaults.cortex]
tick_interval_secs        = 60   # Every 60 seconds
worker_timeout_secs       = 1800
branch_timeout_secs       = 120
circuit_breaker_threshold = 3

On each tick, cortex attempts:

  1. generate_knowledge_synthesis
  2. generate_bulletin
  3. generate_profile
  4. Association backfill

Even with zero new activity, each tick re-processes the full agent context.

Result:

  • ~2B tokens consumed over ~11 days of idle operation
  • Provider's monthly subscription usage limit fully exhausted
  • Quick, simple setup — no active usage, no new memories or conversations

Impact

  • Financial: Consumed entire monthly token plan ($100–$200 estimated)
  • Operational: API key invalidated, Spacebot degraded to unusable state
  • Trust: User almost uninstalled after thinking it was a bug/misconfiguration

Suggestions (Non-Blaming)

1. Warning/Documentation

A prominent warning in docs would help:

⚠️ Default cortex interval is 60s. For metered/token-limited plans, increase to 86400 (daily) or disable until actively using Spacebot.

2. Configuration Guide

A dedicated doc for cortex configuration:

# Conservative (metered plans)
[defaults.cortex]
tick_interval_secs = 86400  # Daily

# Moderate (moderate usage)
[defaults.cortex]
tick_interval_secs = 3600   # Hourly

# Aggressive (unlimited plans)
[defaults.cortex]
tick_interval_secs = 300    # Every 5 min

3. Token Estimation

Could cortex estimate tokens per tick based on memory count and warn if it exceeds a threshold?

4. Idle Detection

Skip synthesis if no memories/conversations changed since last tick.


What's Already in v0.5.0 vs What's Planned

Note: The token consumption happened on v0.4.x (previous version). We updated to v0.5.0 after discovering the issue.

Shipped in v0.5.0 (great additions that helped us investigate):

  • token_usage table — tracks every API call with input/output tokens
  • cortex_events table — logs every synthesis attempt with errors
  • ✅ Config watcher — reloads config without restart
  • ✅ Working memory tables — foundation for the redesign

Still NOT in v0.5.0 (the root cause remains):

  • Change-driven synthesis — no version counter or skip logic found in DB schema or binary
  • ❌ Idle detection — cortex still runs on every tick regardless of changes

I see the working memory implementation plan (Phase 3) addresses this with change-driven synthesis:

let current_version = knowledge_synthesis_version.load();
let last_version = knowledge_synthesis_last_version.load();
if current_version == last_version {
    return;  // Skip - nothing changed!
}

Status check: The plan doc is ~3 weeks old. Is Phase 3 (change-driven synthesis) still in progress? Is there a timeline or branch we can follow?


Our Mitigation (Applied)

We changed the config to:

[defaults.cortex]
tick_interval_secs = 86400  # Once per day

This reduces consumption in my providers.

Also reconfigured routing to use a primary model for conversations and a cheaper model for background tasks.


Questions for the Team

  1. Is there an ETA for Phase 3 (change-driven synthesis)?
  2. Should the default tick_interval_secs be higher for new installations?
  3. Would a "first-run wizard" that asks about token plans be helpful?
  4. Can we help test or contribute to the working memory redesign?

Thank you for v0.5.0


Labels: awareness, documentation, cortex, token-usage, configuration

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions