Senior Software Engineer - AI Interaction Evaluator

This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Senior Software Engineer - AI Interaction Evaluator in United States.

This is a highly specialized evaluation role focused on assessing the quality, reasoning, and engineering judgment of modern AI coding agents. Rather than writing production software, you will analyze how advanced AI systems behave when solving real engineering problems and determine whether their outputs reflect strong developer intuition. The work sits at the intersection of software engineering expertise and AI evaluation, requiring sharp technical instincts and the ability to judge “engineering taste” beyond syntax correctness. You will help define what high-quality AI-assisted development looks like in practice by critically reviewing model interactions and identifying strengths, weaknesses, and failure patterns. Operating in a fast-moving, experimental environment, your feedback will directly influence how AI coding tools are refined and improved. This role is ideal for senior engineers who enjoy deep technical judgment work and shaping next-generation developer experiences.

Accountabilities

Evaluate AI-generated coding interactions end-to-end, assessing usefulness, correctness at a conceptual level, and alignment with strong engineering reasoning.
Analyze whether AI explanations, preambles, and reasoning reflect high-quality developer thinking or introduce confusion or weak logic.
Distinguish between varying levels of output quality and provide structured, opinionated assessments of model performance.
Identify what makes interactions effective or ineffective, including clarity, trustworthiness, and engineering coherence.
Provide detailed qualitative feedback on AI behavior, highlighting what worked, what failed, and where reasoning feels misleading or incomplete.
Contribute to defining evaluation standards for high-quality AI-assisted development workflows.
Help shape benchmarks for what “good engineering judgment” looks like in AI coding tools.

Requirements

You bring deep software engineering experience and strong technical intuition, enabling you to evaluate code quality and system behavior without relying solely on execution or line-by-line verification. You are comfortable making subjective but structured judgments and articulating why an engineering solution feels right or wrong. You have hands-on experience with modern programming workflows and AI-assisted development tools, and you understand what high-quality engineering output looks like in practice.

Staff-level or Principal-level software engineering experience (or equivalent senior expertise)
Strong proficiency in at least one core language: TypeScript/JavaScript or Python
Experience using modern AI coding tools such as Codex, Claude Code, or Cursor
Deep familiarity with AI-assisted development workflows and developer tooling
Ability to evaluate code and system behavior at a conceptual level without full execution
Strong engineering judgment and ability to define what “good” looks like in ambiguous scenarios
Clear, direct communication style with ability to provide structured, opinionated feedback
Experience mentoring engineers or defining engineering best practices (nice to have)
Exposure to prompt engineering or AI evaluation workflows (nice to have)

Benefits

Competitive hourly compensation: $100–$200/hour
Flexible part-time engagement (approximately 10–20 hours/week)
Remote work with high autonomy and async-friendly collaboration
Opportunity to influence the development of next-generation AI coding tools
Short-term contract with potential for extension
Work directly at the frontier of AI-assisted software engineering evaluation
Fast onboarding process with immediate start availability

How Jobgether works:

We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.

We appreciate your interest and wish you the best!

Why Apply Through Jobgether?

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.

#LI-CL1

Senior Software Engineer - AI Interaction Evaluator

Requirements

Benefits

USA Remote Jobs