Add cost-aware cache eviction policy#94
Conversation
Protects cached responses that were expensive to generate (e.g. slow LLM calls) from eviction, blending generation cost with staleness instead of relying on recency alone like LRU.
luis-gasparschroeder
left a comment
luis-gasparschroeder
left a comment
There was a problem hiding this comment.
Thank you for contributing to vCache! The cost-aware policy makes sense; before merging, please
- extract normalization and priority calculation into helper methods,
- use min-max normalization, and
- replace time.time() with time.perf_counter(). Reason being that perf_counter() is monotonic and designed for measuring elapsed durations, whereas system clock adjustments can make time.time() produce inaccurate or even negative latency values.
- Extract normalization and priority calculation into helper methods (_staleness_seconds, _min_max_normalize, _compute_priority). - Switch from max-only scaling to true min-max normalization, so the freshest/cheapest item in the cache maps to 0 instead of just the most stale/expensive item mapping to 1. - Measure inference latency with time.perf_counter() instead of time.time(), since perf_counter is monotonic and immune to system clock adjustments that could otherwise produce inaccurate or negative latency values.
Safaael25
left a comment
Safaael25
left a comment
There was a problem hiding this comment.
@luis-gasparschroeder
Done!! thanks for the review. Extracted _staleness_seconds, _min_max_normalize, and _compute_priority as helpers, switched to true min-max normalization (min→0, max→1) with the README updated to match, and switched to time.perf_counter() for the three latency measurements this PR added. All existing tests still pass unchanged.
There was a problem hiding this comment.
@Safaael25, looks good, and thank you for addressing the comments. Please run the ruff formatter and we are good to merge
c3e7bdb to
090e667
Compare
|
@luis-gasparschroeder Done! I ran the Ruff formatter. Thanks for the review! If there's anything else that needs to be changed or improved, please let me know and I'll be happy to work on it. |
luis-gasparschroeder
left a comment
luis-gasparschroeder
left a comment
There was a problem hiding this comment.
Looks good, thanks!
2 participants
Cost-Aware Eviction Policy
Adds a new CostAwareEvictionPolicy that factors in how expensive a cached response was to generate (e.g. LLM inference latency) when deciding what to evict — not just recency like LRU.
Why: Existing policies (FIFO/LRU/MRU) treat every cache miss as equally costly to replace. In practice, some cached responses come from a slow/expensive model call and others are cheap — evicting the expensive ones just because they're stale wastes the work that produced them. This policy protects expensive-to-regenerate entries while still evicting cheap, stale ones.
How it works: each item gets a priority score blending normalized staleness and normalized (inverted) cost, weighted by a configurable cost_weight (0 = pure LRU, 1 = cost-only). Items with unknown cost default to "free," so it degrades gracefully to LRU with no special-casing.
What changed:
EmbeddingMetadataObj gains a cost field
Cache.add() / EmbeddingStore.add_embedding() thread cost through
VerifiedDecisionPolicy now measures real LLM inference latency and passes it as the cost
New CostAwareEvictionPolicy strategy + tests + README docs