<div class="content-intro"><p>Zafin is an AI platform company helping regulated institutions modernize how critical work is designed, governed, and delivered. Our technology enables organizations to move faster while maintaining the governance, accountability, and control required in highly regulated environments.</p> <p>Our portfolio includes <strong>Zafin AIOS</strong>, an agent orchestration platform for governed AI work; the <strong>Zafin Banking Platform</strong>, which helps banks modernize product, pricing, offers, billing, loyalty, and relationship management; and <strong>Zafin IO</strong>, an integration platform that connects data, systems, and workflows across complex enterprise environments.</p> <p>Headquartered in Toronto, Canada, Zafin partners with leading financial institutions across North America, Europe, the Middle East, Africa, and Asia-Pacific. As AI transforms the future of financial services, we're building the platforms that help regulated organizations adopt AI responsibly and at scale.</p></div><p><strong>What’s the Opportunity?</strong><span data-ccp-props="{}"> </span></p> <p>The AI Evaluation Engineer ensures AI agent solutions are accurate, reliable, safe, and production-ready within regulated banking environments. The role provides objective, evidence-based evaluation of AI agent behaviour against defined business, quality, risk, performance, and regulatory criteria, helping ensure AI solutions deliver consistent, trusted outcomes in production.</p> <p>Working within the Reliability Testing phase of the AIOS (Zafin's AI Operating System) delivery lifecycle, the role designs, executes, and leads evaluation activities that validate AI agent behaviour across the full development lifecycle. This includes developing meaningful evaluation scenarios, identifying defects and failure modes, monitoring quality across releases, and providing actionable feedback that continuously improves AI agent reliability. The role helps operationalize AIOS's principle of reliability first, velocity second through disciplined evaluation and objective production-readiness decisions.</p> <p>The AI Evaluation Engineer works closely with Agent Engineers, Industry Consultants, Agent Architects, AI Knowledge & Governance, Product, and Delivery teams to ensure evaluation reflects intended business logic, regulatory requirements, technical standards, and real-world operating conditions. Depending on experience and level, the role may also lead evaluation activities, coach other evaluation engineers, improve evaluation practices, and drive quality improvements across multiple AI agent initiatives.</p> <p>Ultimately, the role helps ensure AI agent capabilities earn and maintain customer trust by delivering consistent, reliable, and explainable outcomes in production.</p> <p><strong>What Will You Do?</strong><span data-ccp-props="{}"> </span></p> <ul> <li>Design and execute structured evaluation scenarios that validate AI agent accuracy, reliability, safety, compliance, and business outcomes.</li> <li>Validate AI agent behaviour against approved business rules, policies, technical requirements, source knowledge, and expected outcomes.</li> <li>Conduct regression evaluation across releases and monitor behavioural drift, performance degradation, and newly introduced failure modes.</li> <li>Identify, document, prioritize, and track quality issues, defects, and production risks.</li> <li>Support production-readiness decisions through objective evaluation evidence and recommendations.</li> <li>Analyze evaluation results to identify root causes, recurring quality trends, and opportunities to improve prompts, workflows, knowledge, integrations, and engineering practices.</li> <li>Maintain reusable evaluation scenarios, benchmark datasets, expected outcomes, regression suites, and supporting evidence.</li> <li>Contribute to continuous improvement of evaluation methodologies, automation, tooling, and engineering feedback loops.</li> <li>Support investigation of production issues and validate corrective actions.</li> <li>Partner with Agent Engineering, Industry Consultants, Agent Architects, AI Knowledge & Governance, Product, and Delivery teams to ensure evaluation reflects business requirements and production expectations.</li> <li>Participate in production-readiness reviews, release planning, and AI agent optimization activities.</li> <li>Communicate evaluation findings, quality risks, and recommendations clearly to technical and business stakeholders.</li> <li>Depending on experience and level, you may also: <ul> <li>Lead evaluation activities across one or more AI agent initiatives or squads.</li> <li>Review evaluation approaches, production-readiness recommendations, and quality evidence produced by other Evaluation Engineers.</li> <li>Coach and mentor less experienced Evaluation Engineers, supporting technical growth and consistent evaluation practices.</li> <li>Coordinate evaluation priorities across multiple concurrent initiatives and support delivery planning.</li> <li>Drive improvements in evaluation tooling, automation, benchmark management, CI/CD quality integration, and operational effectiveness.</li> <li>Analyze systemic quality trends and lead continuous improvement initiatives across multiple AI agent capabilities.</li> <li>Partner with Delivery and Engineering leadership to improve quality outcomes, operational consistency, and AI agent reliability.</li> </ul> </li> </ul> <p><span data-contrast="auto"><strong>What Do You Need to Succeed?</strong></span><span data-ccp-props="{}"> </span></p> <p><span data-contrast="auto"><strong>Must Haves</strong></span><span data-ccp-props="{}"> </span></p> <ul> <li>Typically<strong> </strong>3–10+ years of relevant experience in software quality engineering, AI evaluation, AI quality engineering, machine learning evaluation, software testing, or related disciplines.</li> <li>Level and scope of responsibility will be deter