Data Scientist – Evaluations & AI Agents

Job Title

Data Scientist – Evaluations & AI Agents

At Ballerine, we’re looking for a passionate Data Scientist to take our product to the next level:

• Define and implement evaluation & product quality methodologies
• Develop tools for testing, experiments, and quality metrics to ensure top-tier performance
• Dive into the world of AI Agents and lead innovation across research and product
• Apply your expertise in payments, fraud, risk, and compliance to design meaningful, business-driven evaluation frameworks

• Develop and track KPIs and metrics for product and model quality
• Build evaluation frameworks (quantitative & qualitative)
• Lead A/B testing, human-in-the-loop pipelines, and error analysis
• Partner closely with product and engineering to drive continuous quality improvements
• Research and implement cutting-edge approaches in LLMs and AI Agents
• Leverage domain knowledge in payments, fraud prevention, merchant risk, and regulatory compliance to ensure our solutions are industry-ready

• Experience as a Data Scientist / ML Engineer with a strong background in evaluation & experimentation
• Proficiency in Python, SQL, and ML frameworks
• Product-oriented mindset with the ability to connect technical quality with business impact
• Genuine passion for AI Agents and pushing the boundaries of innovation
• Domain knowledge in payments, fraud, merchant risk, or compliance — a strong plus
• Ability to own processes end-to-end — from analysis to production implementation

• Be at the forefront of AI-native merchant risk intelligence
• Lead innovation, not just “do data”
• Work in a sharp, fast-moving team already piloting with major global players