Senior Software Engineer - Testing AI Coding Agents

Mindrift is seeking experienced software developers to create and evaluate AI coding tasks for testing AI systems' ability to handle real-world developer scenarios. The role involves designing realistic developer environments, crafting tasks, writing verification tests, and iterating based on feedback.

Location: Poland

Salary: Up to $50/hr

Responsibilities

• Build realistic developer environments with codebase, infrastructure, and context.
• Design tasks from intermediate states of these environments, including crafting prompts and defining success criteria.
• Write tests to verify agent solutions, accepting valid approaches and rejecting incorrect ones.
• Iterate on tasks and tests based on QA feedback, reviewing solutions, analyzing failures, and refining tasks.

Requirements

• 5+ years in software development.
• Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis.
• Experience writing functional and integration tests.
• English proficiency - B2+.

Benefits

• Flexible schedule, estimated 20 hours per task.
• Project-based participation.
• Opportunity to work on challenging AI evaluation tasks.

Additional Information

• Participation is project-based.
• Tasks are estimated to take around 20 hours, with flexible scheduling.
• Compensation up to $50/hr depending on level and pace.