AI Evaluation Scientist

JAPAN AI, a Geniee group company expanding its enterprise AI agent suite, is hiring an AI Evaluation Scientist for a new organisation that scientifically evaluates the quality, safety and reliability of AI output. The person is not a tester but a scientist who defines and measures what makes a good AI, using machine learning, statistics and psychometrics.

The scientist leads the design, build and operation of the evaluation infrastructure: researching evaluation metrics (calibrating LLM-as-judge methods, reward modelling, benchmark design), building automated evaluation pipelines that act as quality gates in CI/CD, automated red teaming and safety and policy-compliance checks, hallucination detection, regression tests for prompts and tools, and statistical experiment design such as A/B tests, with findings fed back to research and development.

Applicants need a master's degree or higher, or equivalent experience, in computer science, machine learning, statistics, mathematics, physics, psychometrics or a related field, and at least three years as an ML engineer, data scientist, research engineer or in ML and AI evaluation.