Review and challenge model evaluation results, including assessments of performance (e.g. groundedness, factual accuracy, retrieval quality, relevance, and instruction adherence), and evaluate the reliability of LLM-as-a-Judge methodologies through calibration against human-labelled benchmarks.
You’ll assist in maintaining all company standard operating procedures to always ensure their currency and to liaise with all departments to ensure that new procedures are introduced, when necessary, in a timely manner.
Lead Rigorous Model Evaluations: Architect and execute comprehensive evaluation suites for LLMs and multimodal models, identifying edge cases in multi-step reasoning, factuality, adversarial robustness, safety, and alignment.
Develop and implement effective quality control systems, inspection methods and processes to ensure products consistently meet customer, regulatory and company requirements.