AI Evaluation Scientist

JAPAN AI, a Geniee group company expanding its enterprise AI agent products, is setting up a specialist organisation to evaluate the quality, safety and reliability of AI output, and is hiring an AI Evaluation Scientist to lead it. The person defines and measures what makes a good AI using machine learning, statistics and psychometrics, rather than simply testing.

The scientist designs, builds and runs the evaluation infrastructure: evaluation-metric research (LLM-as-judge calibration, reward modelling, benchmark design and validity), automated evaluation pipelines in CI/CD with quality gates, multi-turn and tool-use evaluation harnesses, automated red teaming and policy-compliance checks, hallucination detection, regression tests, and statistical experiment design with quality reports for development teams.

Applicants need a master's degree or higher, or equivalent experience, in computer science, AI, machine learning, mathematics, physics, psychometrics or a related field, at least three years in ML engineering, data science, research or ML evaluation, deep knowledge of LLM evaluation, practical statistics, and Python evaluation pipelines. Fluent Japanese or business-level English (either one) is required.