- Paya Lebar Road Geylang Southeast (Singapore) Singapore
全职 LLM Evaluation Engineer 工作, 薪水 up to SGD 12,000, KUAILU SOFTWARE (SINGAPORE) PTE. LTD. Southeast (Singapore) 公司招聘中 - Ricebowl
分享
保存
工作地点
职位描述
岗位职责
Job Description
Build and maintain automated LLM evaluation pipelines covering multiple dimensions, including general capabilities, Agent capabilities, and persona/role-playing capabilities. Support one-click evaluation, historical result comparison, and regression testing.
Conduct general capability evaluations using established benchmarks such as MMLU, C-Eval, HumanEval, GSM8K, MATH, and IFEval, including benchmark deployment, execution, and results analysis.
Conduct Agent capability evaluations, including setting up evaluation environments and tracking performance metrics for benchmarks such as BFCL, τ-bench, and GAIA.
Design and execute persona and role-playing evaluation frameworks, covering metrics such as identity recognition, role consistency, multi-turn stability, and style consistency.
Record and analyse evaluation results across model training runs, perform comparisons and anomaly detection, and prepare evaluation reports for training checkpoints.
Conduct regular intermediate evaluations during the pre-training stage to track the evolution and improvement of model capabilities.
Job Requirements
Bachelor's degree or above in Computer Science, Artificial Intelligence, or a related discipline.
Familiar with mainstream LLM evaluation benchmarks and frameworks, such as lm-eval-harness, OpenCompass, and EvalPlus.
Proficient in Python, with the ability to independently build evaluation pipelines covering model inference/deployment, batch evaluation, and results analysis.
Familiar with LLM inference frameworks such as vLLM and SGLang, with the ability to deploy models for batch inference and evaluation.
Experience in evaluation data analysis and visualisation.
Detail-oriented and rigorous, with a strong focus on ensuring the reproducibility and reliability of evaluation results.
Good to Have
Experience with Agent evaluation, particularly BFCL, τ-bench, GAIA, or SWE-bench.
Experience with persona or role-playing evaluation, such as CharacterBench or RMTBench.
Experience in evaluation automation and CI/CD integration.
Understanding of LLM training workflows, with the ability to interpret the relationship between training checkpoints and evaluation results.
Kuailu Technology is dedicated to building a one-stop AI-powered intelligent office system. With a low-code platform as its foundation and AI agents as the engine, it integrates a full-scenario office ecosystem encompassing AI-driven office work, collaboration, and meetings. The system covers all business scenarios, including HR, finance, procurement, administration, and CRM. By establishing a unified data foundation to replace multiple software systems, it breaks down internal information silos, enables seamless data flow across entire business processes, and activates core data assets related to people, capital, and materials. This transforms data into intelligent productivity, helping enterprises achieve low-cost, high-efficiency, and integrated smart office operations.
Build and maintain automated LLM evaluation pipelines covering multiple dimensions, including general capabilities, Agent capabilities, and persona/role-playing capabilities. Support one-click evaluation, historical result comparison, and regression testing.
Conduct general capability evaluations using established benchmarks such as MMLU, C-Eval, HumanEval, GSM8K, MATH, and IFEval, including benchmark deployment, execution, and results analysis.
Conduct Agent capability evaluations, including setting up evaluation environments and tracking performance metrics for benchmarks such as BFCL, τ-bench, and GAIA.
Design and execute persona and role-playing evaluation frameworks, covering metrics such as identity recognition, role consistency, multi-turn stability, and style consistency.
Record and analyse evaluation results across model training runs, perform comparisons and anomaly detection, and prepare evaluation reports for training checkpoints.
Conduct regular intermediate evaluations during the pre-training stage to track the evolution and improvement of model capabilities.
Job Requirements
Bachelor's degree or above in Computer Science, Artificial Intelligence, or a related discipline.
Familiar with mainstream LLM evaluation benchmarks and frameworks, such as lm-eval-harness, OpenCompass, and EvalPlus.
Proficient in Python, with the ability to independently build evaluation pipelines covering model inference/deployment, batch evaluation, and results analysis.
Familiar with LLM inference frameworks such as vLLM and SGLang, with the ability to deploy models for batch inference and evaluation.
Experience in evaluation data analysis and visualisation.
Detail-oriented and rigorous, with a strong focus on ensuring the reproducibility and reliability of evaluation results.
Good to Have
Experience with Agent evaluation, particularly BFCL, τ-bench, GAIA, or SWE-bench.
Experience with persona or role-playing evaluation, such as CharacterBench or RMTBench.
Experience in evaluation automation and CI/CD integration.
Understanding of LLM training workflows, with the ability to interpret the relationship between training checkpoints and evaluation results.
Kuailu Technology is dedicated to building a one-stop AI-powered intelligent office system. With a low-code platform as its foundation and AI agents as the engine, it integrates a full-scenario office ecosystem encompassing AI-driven office work, collaboration, and meetings. The system covers all business scenarios, including HR, finance, procurement, administration, and CRM. By establishing a unified data foundation to replace multiple software systems, it breaks down internal information silos, enables seamless data flow across entire business processes, and activates core data assets related to people, capital, and materials. This transforms data into intelligent productivity, helping enterprises achieve low-cost, high-efficiency, and integrated smart office operations.
重要安全守则
申请工作时，切勿提供您的银行或信用卡详细资料。不要转账或完成无关的在线调查问卷。如果您发现可疑内容，请举报此招聘广告。
了解更多