Data Scientist - AI Evaluation

This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Data Scientist - AI Evaluation in United States.

This role is focused on ensuring the reliability, accuracy, and real-world performance of AI systems that power consumer-facing experiences. The Data Scientist will develop metrics, evaluation frameworks, and experiments to measure how AI models perform across retrieval, ranking, recommendations, and outcomes. Working closely with ML engineers and product teams, this role transforms ambiguous product questions into measurable hypotheses, identifies failure modes, and drives continuous improvement. Success requires strong analytical skills, a deep understanding of AI evaluation, and the ability to translate complex technical insights into actionable product outcomes. The position offers an opportunity to shape how AI effectiveness is measured and trusted across the organization while collaborating in a fast-paced, innovation-driven environment.

Accountabilities:

Define, implement, and maintain metrics and scoring frameworks to evaluate AI agent performance across the full shopping experience
Design and run experiments to measure model improvements, regressions, and user impact
Build and maintain evaluation datasets, benchmarks, and automated evaluation pipelines
Translate product and engineering questions into clear, structured hypotheses and measurable analyses
Identify edge cases, failure modes, and gaps in AI model performance, recommending actionable improvements
Create dashboards and reporting that make AI system performance visible, trusted, and actionable
Collaborate closely with ML engineers, product managers, and other stakeholders to guide iteration and validate model changes

Requirements:

4–6+ years of experience in data science, AI/ML evaluation, applied AI, or related roles
Deep expertise in evaluating AI/ML systems such as ranking, recommendation engines, or LLMs
Strong experience in experimentation methodologies (A/B testing, causal inference, offline and live evaluations)
Background in consumer products, user-facing systems, or e-commerce/marketplace platforms
Ability to translate ambiguous, complex problems into structured analyses and actionable metrics
Strong product mindset, with focus on real user outcomes and measurable impact
Excellent communication skills with the ability to influence across engineering and product teams
Proficiency with data analysis, statistical modeling, and evaluation frameworks

Benefits:

Competitive base salary range of $225,000–$280,000 USD, depending on experience and location
Equity through stock options
Comprehensive healthcare coverage (medical, dental, vision)
401(k) retirement plan
Flexible PTO and company holidays
Fully remote work within the United States
Periodic company offsites and team gatherings

Why Apply Through Jobgether?

We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.

We appreciate your interest and wish you the best!

Why Apply Through Jobgether?

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.

#LI-CL1

Data Scientist - AI Evaluation

USA Remote Jobs