Convert the evaluation data into formats that can be used by the evaluator. This should most commonly be a string. Parameters are the raw input from the run, the raw output, raw reference output, and the raw run.
// Chain input: { input: "some string" }
// Chain output: { output: "some output" }
// Reference example output format: { output: "some reference output" }
const formatEvaluatorInputs = ({
rawInput,
rawPrediction,
rawReferenceOutput,
}) => {
return {
input: rawInput.input,
prediction: rawPrediction.output,
reference: rawReferenceOutput.output,
};
};
The prepared data.
Optional agentA list of tools available to the agent, for TrajectoryEvalChain.
Optional chainOptional criteriaThe criteria to use for the evaluator.
Optional distanceThe distance metric to use for comparing the embeddings.
Optional embeddingThe embedding objects to vectorize the outputs.
Optional feedbackThe feedback (or metric) name to use for the logged evaluation results. If none provided, we default to the evaluationName.
Optional llmGenerated using TypeDoc
The name of the evaluator to use. Example: labeled_criteria, criteria, etc.