Skip to content

Evaluation: Increase max unique rows #995

Description

@AkhileshNegi

Is your feature request related to a problem?
Fast evaluation is currently limited to small datasets, which restricts the ability to effectively utilize direct LLM calls for evaluations, causing inefficiencies.

Describe the solution you'd like

  • Increase EVAL_FAST_MAX_UNIQUE_ROWS from 10 to 100
  • Allow up to ~500 total items per fast run with a default duplication factor of 5
  • Maintain eligibility checks based on unique rows
  • Monitor API concurrency and memory usage during testing
  • Confirm failure threshold is appropriate at the new scale
  • Follow rollout plan to test and deploy the changes.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Fields

No fields configured for issues without a type.

Projects

Status
In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions