AI Quality Lead

AI Quality Lead

About hio

hio is an AI Business Partner for small and medium business owners- helping them grow their business through every customer conversation, acting as a proactive partner that surfaces opportunities and flags what actually needs the owner's attention. We've raised a $16M seed round backed by investors behind OpenAI and Anthropic, alongside Wix.

We're a high-talent, fast-moving team, and we're just getting started.

About the role

You'll be the guardian of hio's quality. The person who holds the line on what great looks like as we grow. As more of the business runs through hio, the trust owners place in it comes down to the quality of every reply, insight, and decision it makes on their behalf. You'll shape how we think about that quality, raise the bar over time, and make sure hio stays something owners are proud to put their name on.

What you'll own

• Define what quality means for hio's AI - across response accuracy, owner-voice and tone fidelity, cross-selling judgment, and the autonomy question of when hio should act on its own versus surface something for the owner.
• Build and run the eval system: offline test sets, LLM-as-judge graders validated against human review, regression suites, and production monitoring that flags drift before it reaches customers.
• Turn real business-owner replies into ground truth - mining how owners actually respond so hio learns their voice and instincts rather than defaulting to generic, process-heavy framing.
• Run the human-review loop: sample production output, label failure modes, and feed sharp, specific findings back into prompts, context, and product decisions.
• Partner with product and engineering on every model, prompt, and context change - owning the before/after measurement that says whether it shipped an improvement or a regression.
• Be the internal source of truth on quality: report clearly on where hio is strong, where it's weak, and what it'll take to close the gap.

What we're looking for

Required

• 2-3 years of experience in AI / LLM quality, evaluation, or closely related role.
• Fluent AI operator, not just an evaluator. Strong prompting craft and a natural habit of working in tight build-test-refine loops, using AI daily to move faster.
• Hands-on experience evaluating LLM or AI product output - you've designed eval sets, written and validated graders, and used the results to actually change a product.
• Fluency with modern eval practice: LLM-as-judge, human review and labeling, regression testing, and production monitoring - and a clear sense of when each one applies.
• Solid testing fundamentals. Comfortable with traditional functional QA across web and mobile, so quality holds up at the product level, not just the AI layer.
• Genuinely detail-obsessed. You notice the off-key word, the subtly wrong tone, the reply that's technically correct but lands badly - and you can articulate why.
• Strong written judgment about voice and tone. You can tell the difference between something that sounds like a real business owner and something that sounds like AI, and explain the gap.
• Comfortable being the rigor in the room - making quality measurable and defensible, and holding the bar when there's pressure to ship.

Preferred

• Experience with the SMB ecosystem. You understand how these owners talk to their customers.
• Experience evaluating multi-channel or agentic systems