Senior Software Engineer - AI Agent Evaluation

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

Location: Canada

Salary: Up to $50/hr depending on level and pace, estimated at ~20 hours per task

Responsibilities

• Create challenging tasks and evaluation criteria within realistic simulated environments
• Build realistic developer environments with codebase, infrastructure, and context
• Design tasks from intermediate states of environments, craft prompts, define success criteria, and ensure solvability by AI agents
• Write tests that verify agent solutions, accepting valid approaches and rejecting incorrect ones
• Iterate on tasks and tests based on QA feedback, review solutions, analyze failures, and refine tasks

Requirements

• 5+ years in software development
• Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
• Experience writing functional and integration tests
• English proficiency - B2+

Additional Information

• Participation is project-based, not permanent employment.
• Tasks are estimated to take 20 hours, with flexible scheduling.
• Compensation up to $50/hr, depending on level and pace.