Senior AI Quality Engineer (LLM Evaluation & Automation)

This is a remote position responsible for owning and maintaining the evaluation harness and quality gate for measurable agent quality, replacing the previous late-stage evaluation specialist model.

Location: Remote

Responsibilities

• Build and maintain the MVP eval harness: golden tasks, exception tasks, scorecard metrics, and regression packs.
• Wire evals into CI so quality regressions fail builds and releases.
• Define and maintain release-gate thresholds with Product and the Tech Lead.
• Lay the path for later adversarial and drift-testing expansion without overbuilding MVP scope.

Requirements

• Experience evaluating ML, LLM, or non-deterministic systems.
• Strong test and benchmark design capability.
• Comfort working with noisy metrics, thresholds, and probabilistic behavior.
• Good scripting and automation skills.

Additional Information

• Uses AI to generate candidate eval cases and failure hypotheses, but never confuses generated tests with validated quality.
• Approaches AI quality as an operating system, not a QA afterthought.
• The first reference agent has a published scorecard and gated eval path.
• Golden and exception tests run automatically.
• The team can explain what “good enough to ship” means in measurable terms.