LLM Testing & Evaluation Engineer

Job description

LLM Testing & Evaluation Engineer

About Insait:

Insait is an innovative SaaS company specializing in developing LLM-based solutions for the financial and insurance industries. We work with major clients globally, building B2B2C applications at the forefront of technology, deeply integrating AI, product, and data.About the Role:

We’re looking for a LLM Testing & Evaluation Engineer to ensure our clients go live quickly and with the highest product quality.

In this role, you'll be responsible for evaluating real-world use cases, improving the performance of our AI agents, and helping shape the quality of every customer experience.

This is a hands-on role that requires experience working with LLMs via API (e.g., OpenAI, Anthropic) and a strong foundation in prompt engineering and test thinking.

Responsibilities:

• Design and execute structured test plans for AI workflows
• Simulate real-world interactions to identify edge cases and inconsistencies
• Monitor and evaluate LLM responses using defined success metrics
• Optimize prompt design and ensure output quality
• Collaborate with engineering and product teams to drive improvements

Requirements:

• 1–3 years of experience working with LLMs or generative AI APIs
• Strong skills in prompt engineering and prompt evaluation
• Experience in QA or testing methodologies — automation experience is a plus
• Detail-oriented mindset with excellent analytical and communication skills
• High proficiency in English