Senior Machine Learning Engineer

Cover is a fast-growing Insurtech startup making insurance accessible, transparent, and easy — for individuals and businesses.

We are looking for a driven, highly skilled Senior Machine Learning Engineer (AI Evaluation & Reliability) to join our AI team. This is not a traditional model-training role — we are seeking an engineering and evaluation expert to build automated systems that inspect, benchmark, and ensure the reliability of complex AI and Agent-based workflows at scale. If you have a rigorous analytical mindset, strong statistical foundations, and a passion for making production AI systems bulletproof, this role is for you.

What You'll Do:

• Design, build, and scale automated evaluation pipelines combining deterministic tests, model-based grading (LLM-as-a-Judge), and Ground Truth validation
• Implement trace and behavior analysis to catch failures, validate tool-calling workflows, and ensure real-world task execution
• Build and maintain continuous evaluation frameworks, benchmark datasets, and automated regression suites
• Develop rigorous evaluation metrics for domain accuracy, data layer integrity, and quality of generated insights
• Establish production quality monitoring, real-time error classification, and statistical anomaly detection
• Build human-in-the-loop workflows to continuously benchmark, calibrate, and improve automated evaluation engines
• Partner closely with engineering teams to establish quality standards, prevent regressions, and govern production releases

What We're Looking For:

• 7+ years of software engineering and machine learning experience, including 3+ years hands-on with production LLM and AI Agent systems — Must
• Proven track record designing and deploying automated evaluation frameworks in production — Must
• Strong mathematical, statistical, and data validation background (hypothesis testing, error analysis, calibration, anomaly detection) — Must
• Advanced proficiency in Python, SQL, and core ML/data libraries
• Practical experience with cloud infrastructure (AWS/GCP), Docker, and CI/CD pipelines
• B.Sc. in Computer Science, Mathematics, Statistics, or a related quantitative field (M.Sc. — strong advantage)
• Experience in FinTech, InsurTech, or enterprise SaaS — strong advantage
• A proactive problem-solver with high attention to detail who thrives in a fast-paced, dynamic startup environment