Evaluator

SchemaEvaluator

SchemaEvaluator(
    llm: BaseChatModel,
    evaluation_prompt: str,
    max_retries: int = 3,
)

Scores an extraction schema against a set of expert questions.

For each question the LLM judge checks whether the schema contains the fields needed to answer it, and reports a CoverageAssessment with a confidence level. The aggregate result is a SchemaEvaluation with confidence-weighted counts.

Parameters:

Name	Type	Description	Default
llm	BaseChatModel	Language model used as the judge.	required
evaluation_prompt	str	Prompt template with {questions} and {schema} placeholders.	required
max_retries	int	Retries for structured-output calls that fail with an openai.OpenAIError. Default: 3.	3

SchemaEvaluation

SchemaEvaluation(
    total_questions: int,
    covered_questions: int,
    high_confidence_coverage: int,
    medium_confidence_coverage: int,
    low_confidence_coverage: int,
)

Aggregate evaluation result for a schema against a question set.

Attributes:

Name	Type	Description
total_questions	int	Number of questions evaluated.
covered_questions	int	Questions answered by the schema (any confidence).
high_confidence_coverage	int	Covered questions judged with high confidence.
medium_confidence_coverage	int	Covered questions judged with medium confidence.
low_confidence_coverage	int	Covered questions judged with low confidence.

CoverageAssessment

Bases: BaseModel

Per-question verdict from the LLM judge.

Attributes:

Name	Type	Description
is_covered	bool	Whether the schema contains the fields needed to answer the question.
confidence	Literal['high', 'medium', 'low']	How certain the judge is — high (clearly covered or not), medium (partially covered or ambiguous), low (uncertain).