Toloka operates a platform for developing and testing safe AI products, combining automated workflows with human experts across 50+ domains, virtual agent testing environments, and global annotator networks.
<p><strong>About Toloka</strong></p> <p>At <strong>Toloka AI</strong> we create data that powers leading GenAI models and innovations. We work with frontier labs, big tech, renowned AI startups, enterprises and non-profit research organizations worldwide. We use a combination of Experts + Crowd + Tech Platform to teach AI models to reason and evaluate their efficacy and safety. We have experts in more than 50 different domains—from doctors and lawyers to physicists and engineers—and boast one of the most diverse global crowds, <strong>representing over</strong> <strong>100 countries and speaking 40+ languages</strong>. We are a well-funded startup with an enviable portfolio of clients including <strong>Anthropic</strong>, <strong>Amazon</strong>, <strong>Microsoft</strong>, <strong>Poolside</strong>, <strong>Recraft</strong>, and <strong>Shopify</strong>.</p> <p>Recently, we secured strategic investment led by <strong>Bezos Expeditions </strong>and<strong> Nebius Group</strong> with participation from <strong>Mikhail Parakhin</strong>, <strong>CTO of Shopify</strong> and board advisor to leading GenAI companies, who now serves as our Chairman of the Board. Our remote-first team is globally distributed around the world: <strong>USA, UK, the Netherlands, Serbia</strong>, and more.</p> <p> </p> <p><strong>About the Position</strong></p> <ul> <li>Led applied ML initiatives within the Delivery division, integrating LLM and AI-agent technologies into active client engagements with a focus on Anti-Fraud.</li> <li>Optimize output quality and automate manual workflows to scale operational capacity and improve business margins.</li> </ul> <p> </p> <p><strong>Core Outcomes You’ll Drive</strong></p> <ul> <li><strong>Automation lift:</strong> Achieve double-digit percentage improvements per account by replacing manual steps with robust LLM/agent pipelines.</li> <li><strong>Quality & reliability:</strong> Raise client acceptance rates and reduce rework through automated LLM checks and evaluation harnesses.</li> <li><strong>Throughput & margin:</strong> Shorten cycle times and expand task capacity by productizing reusable components across projects.</li> <li><strong>Safety & governance:</strong> Enforce strict guardrails and solution auditability through systematic red-teaming.</li> </ul> <p> </p> <p><strong>What you’ll do</strong></p> <ul> <li><strong>Solution architecture:</strong> Design and deploy agentic workflows (tool use, planning, retrieval, critique loops) for data generation, evaluation, <strong>anti-fraud, and safety</strong> use cases.</li> <li><strong>Automated evaluation:</strong> Build judge models, auto-grading systems, and self-verification pipelines to codify acceptance criteria into repeatable checks.</li> <li><strong>Productization:</strong> Create reusable reference architectures, templates, and components to scale delivery across multiple client accounts.</li> <li><strong>AI-driven scaling:</strong> Integrate LLMs into contributor workflows (pre-label → verify → escalate) to eliminate operational toil and maximize expert leverage.</li> <li><strong>Observability:</strong> Implement production measurement frameworks (task KPIs, drift), online A/B tests, and cost/latency dashboards.</li> <li><strong>Customer collaboration:</strong> Partner with Delivery Leads, PMs, and client teams to translate business requirements into pragmatic ML designs.</li> <li><strong>Technical stewardship:</strong> Set engineering standards for prompts, agents, data pipelines, and CI/CD via active code and design reviews.</li> </ul> <p> </p> <p><strong>What we're looking for</strong></p> <ul> <li><strong>Production experience:</strong> Over 5–8+ years of expertise in applied ML and LLM development, with a documented history of launching agentic workflows into production.</li> <li><strong>Domain expertise:</strong> Strong background in building anti-fraud platforms, trust & safety systems, or real-time anomaly detection frameworks.</li> <li><strong>Delivery focus:</strong> A pragmatic approach, capable of transforming vague project needs into executable solutions defined by strict timelines and measurable KPIs.</li> <li><strong>AI quality assurance:</strong> Extensive experience in building automated evaluation pipelines and grading systems to maintain consistency at scale.</li> <li><strong>Automation mindset:</strong> A natural drive for automation, with a track record of identifying and optimizing manual operations to enhance throughput and margins.</li> <li><strong>System architecture:</strong> Skill in pragmatic engineering, prioritizing simple, observable, and efficient designs that align with latency and cost goals.</li> <li><strong>Hands-on engineering:</strong> A technical profile featuring advanced Python proficiency and the ability to tune agents and prompts within live systems personally.</li> </ul> <p> </p> <p><strong>What we can offer</strong></p> <ul> <li>You will be part of an international, dynamic environment that drives innovation and sets new standards in the AI and technology sector.</li> <li>Competitive compensation package including base salary, bonus, and ESOP</li> <li>Paid PTO and benefits will vary depending on location</li> <li>We offer a full remote or hybrid model (if you are based in NL or Serbia)</li> <li>IT setup and home office allowances.</li> </ul> <p> </p> <p><strong>Equal Opportunity Employer:</strong></p> <p>Toloka is committed to providing equal opportunity and fostering an inclusive environment. We welcome applications from all qualified individuals and do not discriminate on the basis of race, religion, color, national origin, sex, sexual orientation, gender identity, age, marital status, veteran status, disability, or any other characteristic pro