AI Product Manager, AI Platform - Quality and Evaluation

About this role

<p>Are you passionate about building AI products that customers can trust by making quality measurable rather than subjective? Do you enjoy partnering with engineers and researchers to evaluate large language models, improve AI system performance, and turn data into product decisions? Are you excited by the opportunity to define the future of AI quality, observability, and evaluation platforms that power intelligent applications? If so, we invite you to be a part of our innovative team.</p> <p>As a Product Manager on Ridgeline's AI Platform team, you'll define and execute the strategy for the systems that ensure our AI capabilities are accurate, reliable, and continuously improving. You'll lead initiatives spanning LLM evaluation frameworks, AI model benchmarking, evaluation harnesses, prompt and model experimentation, AI quality platforms, and observability capabilities that help teams understand, measure, and improve production AI performance. You'll work closely with engineering, AI researchers, design, and product teams to deliver platform capabilities that enable every AI-powered experience across Ridgeline. You'll leverage cutting-edge AI technologies and tools in a fast-moving, creative, progressive work environment while helping establish best practices for developing and operating enterprise AI products.</p> <p>At Ridgeline, how we work matters as much as what we build. Ridgeliners act like owners, choose growth over comfort, and communicate with transparency. We assume positive intent, bias toward action, and bring solutions—not just problems. We celebrate wins, learn from setbacks, and thrive in a resilient, collaborative, high-performing culture. If this excites you, we’d love to meet you!</p> <p><strong>You must be work authorized in the United States without the need for employer sponsorship.</strong></p> <p><strong>The impact you will have</strong></p> <ul> <li>Define and execute the product strategy and roadmap for Ridgeline's AI Quality Platform</li> <li>Build scalable LLM evaluation frameworks and automated evaluation harnesses that enable continuous model validation</li> <li>Establish standardized offline and online evaluation methodologies, golden datasets, and quality benchmarks across AI products</li> <li>Define the roadmap for AI observability, enabling teams to monitor quality, latency, cost, reliability, and user outcomes in production</li> <li>Partner with engineering and AI researchers to build continuous regression testing and experimentation capabilities for generative AI systems</li> <li>Drive product decisions using AI quality metrics, customer feedback, and quantitative evaluation data</li> <li>Collaborate with engineering teams to deliver platform capabilities that enable trusted, measurable, and continuously improving AI experiences across Ridgeline</li> <li>Translate advances in foundation models, evaluation techniques, and AI tooling into platform capabilities that create meaningful customer value</li> <li>Prioritize product investments using customer insights, product analytics, and measurable business outcomes</li> <li>Communicate product vision, roadmap, and priorities clearly across technical and executive stakeholders</li> <li>Foster a collaborative environment rooted in ownership, continuous learning, experimentation, and transparency</li> </ul> <p><strong>What we look for</strong></p> <ul> <li>5+ years of product management experience building highly technical platform, infrastructure, or developer-focused products</li> <li>Demonstrated experience shipping production AI products powered by large language models</li> <li>Experience defining products that incorporate LLM evaluation frameworks, AI quality measurement, and continuous model evaluation</li> <li>Deep understanding of offline and online evaluation methodologies, model benchmarking, regression testing, and AI quality metrics</li> <li>Experience with LLM observability platforms such as Datadog LLM Observability, Langfuse, LangSmith, Arize AI, Braintrust, Galileo, or similar AI quality and evaluation solutions</li> <li>Familiarity with prompt engineering, experimentation frameworks, evaluation datasets, and LLM-as-a-judge techniques for generative AI applications</li> <li>Experience partnering effectively with software engineers, ML engineers, AI researchers, and cross-functional stakeholders</li> <li>Strong analytical skills with the ability to translate AI performance signals into product decisions</li> <li>Excellent written and verbal communication skills</li> <li>Willingness to learn about cutting-edge technologies while cultivating expertise in investment management workflows and enterprise AI</li> <li>An aptitude for problem solving</li> <li>Ability to communicate effectively</li> <li>Serious interest in having fun at work</li> </ul> <p><strong>Bonus</strong></p> <ul> <li>Experience building products around retrieval-augmented generation (RAG), AI agents, MCP, or model orchestration frameworks</li> <li>Experience with synthetic data generation and automated evaluation pipelines</li> <li>Experience building internal developer platforms or ML infrastructure products</li> <li>Experience in enterprise SaaS, developer platforms, or financial technology</li> </ul> <p><strong>About Ridgeline</strong></p> <p>Ridgeline is headquartered in Lake Tahoe, with offices in New York, Reno, the Bay Area, Dublin Ireland. Ridgeline is recognized by Fast Company as a “Best Workplace for Innovators,” by Frost &amp; Sullivan as a “Technology Innovation Leader,” and by The Software Report as a “Top 100 Software Company.”</p> <p>Ridgeline is proud to be a community-minded, discrimination-free equal opportunity workplace.</p> <p>Ridgeline processes the information you submit in connection with your application in accordance with the&nbsp;<a href="https://www.ridgelineapps.com/legal/candidate-privacy-policy">Ridgeline Applicant Privacy Statement</a>. Please review the Ridgeline Applicant Privacy Statement in f