AI Agent Evaluator

About the company

Hippocratic AI develops clinically safe generative AI agents that perform non-diagnostic, patient-facing tasks for healthcare organizations.

About this role

AI AGENT DEVELOPMENT & EVALUATION Primary Care, Acute Symptom Triage & Travel Health | Contract, up to 10 hrs/week ABOUT THE ROLE We are seeking Nurses & Nurse Practitioners with clinical experience in one or more of the following areas: - acute symptom management - CHF and hypertension management - diabetes management - travel health to serve as a subject matter expert supporting the evaluation of AI-driven clinical agents. In this role, you will evaluate agent conversations and recommendations spanning acute symptom triage and escalation decisions, insulin/diabetes titration, and pre-travel health guidance, ensuring clinical accuracy, guideline adherence, and patient safety. This is a part-time, remote contract position (1099) requiring up to 10 hours per week, with flexible scheduling. KEY RESPONSIBILITIES - Review recorded and simulated AI agent interactions involving acute symptom triage and assess the appropriateness of escalation and de-escalation decisions. - Review agent conversations related to insulin and diabetes medication titration for clinical accuracy and guideline adherence. - Evaluate agent-delivered travel health guidance, including pre-travel counseling, vaccination recommendations, and destination-specific risk assessment. - Identify clinical safety concerns, edge cases, and gaps in the agent's triage logic or clinical reasoning. - Score and annotate agent performance using structured evaluation rubrics. - Provide clear, actionable written feedback to clinical and product teams to inform agent refinement. - Participate in periodic case review and calibration sessions with other clinical SMEs to ensure consistent evaluation standards. - Help develop and refine test scenarios used in ongoing agent quality assurance. QUALIFICATIONS - Registered Nurses: active, unrestricted RN License Nurse Practitioners: active, unrestricted Nurse Practitioner license in the United States, with current board certification (ANCC or AANP). - Minimum of 5+ years of direct clinical experience in primary care, urgent care, telephone/virtual triage, or a comparable acute-care setting. - Clinical experience managing patients with diabetes, including insulin titration. - Strong, current working knowledge of relevant triage protocols and clinical practice guidelines. - Ability to critically evaluate clinical reasoning and identify inaccuracies or safety risks in written or recorded clinical interactions. - Comfortable working independently in a remote, asynchronous contract role. - Strong written communication skills for delivering structured clinical feedback. RECOMMENDED It is not a requirement that candidates have all of the recommended criteria - Experience in telephone triage or nurse advice line settings. - Experience in travel medicine, occupational health, or pre-travel consultation. - Familiarity with CDC and other travel medicine guidelines. - Prior experience in clinical quality assurance, utilization review, or case review. - Familiarity with AI-assisted clinical tools or digital health platforms. ENGAGEMENT DETAILS - Contract (1099) position, up to 10 hours per week. - Fully remote; flexible scheduling. - Hourly compensation, commensurate with experience. - Engagement expected to run in support of ongoing agent development and FDA evaluation milestones. Please be aware of recruitment scams impersonating Hippocratic AI. All recruiting communication will come from @hippocraticai.com http://hippocraticai.com email addresses. We will never request payment or sensitive personal information during the hiring process.