Feat/Add GPCA (Grace-Period Confidence-Cost Aware) NEW eviction policy - #100
Open
Safaael25 wants to merge 2 commits into
Open
Feat/Add GPCA (Grace-Period Confidence-Cost Aware) NEW eviction policy#100Safaael25 wants to merge 2 commits into
Safaael25 wants to merge 2 commits into
Conversation
SCUEvictionPolicy treats any item without a confirmed t_prime (fewer than 6 real observations) as maximally evictable, indistinguishable from a genuinely bad match. GPCA replaces that with a grace period: unconfirmed items are protected in proportion to generation cost and access frequency instead of being evicted first; confirmed items are protected by SCU-style confidence blended with frequency. Benchmarked against LRU/FIFO/SCU/CostAware/CostAwareSCU on real LmArena data: shows a substantial hit-rate and hit-precision improvement under tight-to-medium cache pressure (up to ~6x at 20-40MB on a real repeated- query workload), but a reproducible weakness protecting infrequently- reused expensive items across long reuse gaps, and no measurable edge once the cache is generous relative to the working set. Not a universal improvement -- a targeted one, documented as such.
- Unit tests covering cold-start cost/frequency protection, the confirmed- item confidence+frequency blend, and the specific SCU behavior GPCA fixes (an unconfirmed-but-expensive item is protected instead of always evicted first). - README section documenting the protection-score formula and disclosing the empirical results honestly: a real hit-rate/precision improvement under tight-to-medium cache pressure, and a real weakness protecting expensive items across long reuse gaps. - Export GPCAEvictionPolicy from the strategies package, matching the other eviction policies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
GPCA is a new eviction policy I designed and implemented myself. It fixes
a specific gap in the existing SCU policy: SCU treats any cached item that
hasn't yet accumulated enough observations to have a confirmed similarity
threshold as maximally evictable — indistinguishable from a proven bad
match, regardless of how expensive or how frequently accessed it is.
GPCA replaces that with a "grace period": while an item is still
unconfirmed, it's protected in proportion to its generation cost and how
often it's already been accessed, instead of being the first thing
evicted. Once confirmed, protection blends SCU-style confidence with
access frequency.
Motivation
I noticed SCU's blind spot specifically punishes expensive,
infrequently-accessed items the hardest — exactly the items a cost-aware
policy should be protecting. GPCA is my attempt to close that gap while
keeping the confidence-based reasoning SCU already does well.
Results
Benchmarked against LRU, FIFO, SCU, and CostAware on real conversational
data: GPCA shows a real, substantial hit-rate and hit-precision
improvement under tight-to-medium cache pressure on workloads with
genuine repeated queries. This is a targeted improvement, not a universal
one — the advantage fades once the cache is generous relative to the
working set, and GPCA has a known weakness protecting rarely-reused
expensive items over long gaps, which I've documented in the README
rather than hidden.
Testing
4 new unit tests covering cold-start cost/frequency protection, the
confirmed-item blend, and the specific SCU behavior this fixes.