Senior AI/ML Engineer - AI Systems Evaluation

REAL is building an AI Execution Platform for real estate organizations.

Today, the data required to run real estate is scattered across fragmented systems, leading to missed insights and preventable financial leakage.

REAL transforms this complexity into connected intelligence and automated execution, enabling enterprises to operate with greater precision and confidence.

REAL Values

Ownership: We take responsibility and move decisively.
Clarity: We simplify complexity to deliver meaningful impact.
Accuracy: Precision matters in everything we build.
Velocity: We work with urgency and intent.
Partnership: We collaborate closely with customers and teammates.

Role Overview

Own the systems that define, measure, and enforce AI quality at REAL.
Translate ambiguous model behavior into measurable signals, automated tests, and release gates.
Operate across evaluation design, tooling, and production integration.

What You'll Do

Design evaluation architectures (benchmarks, regression suites, coverage)
Build automated pipelines to run and score evals across models and prompts
Implement scoring systems (LLM-as-judge, rubrics, hybrid approaches)
Create and maintain golden datasets + edge-case suites
Develop internal tools for prompt testing, dataset generation, experiment tracking
Instrument systems for traces, outputs, and debugging
Detect regressions and enforce quality gates in CI/CD
Monitor model performance in production
Close the loop between eval insights and product improvements

Requirements

What We're Looking For

3-6 years building production software, internal platforms, ML/data infrastructure, experimentation systems, or AI tooling
Strong backend and systems engineering fundamentals with hands-on applied AI experience
Strong Python, production-level systems experience
Built testing frameworks or validation systems end-to-end
Hands-on with LLMs / RAG / agent workflows
Understands eval methods (benchmarking, A/B, LLM-as-judge, HITL)
Experience with observability / logging / experiment tracking
Strong systems thinking (coverage, reliability, reproducibility)
Comfort with non-deterministic systems

Nice to Have

Experience with eval, tracing, observability, or experimentation tooling (one or more of the following: LangSmith, Braintrust, Phoenix, MLflow, OpenTelemetry, PostHog, custom eval stacks)
Familiarity with dataset/versioning workflows, HITL systems, and production AI observability systems
CI/CD integration for model evaluation
Background in search, retrieval, or document systems
Built internal platforms or developer tools
Experience working in startups and business driven environments