Quality Assurance & Model Evaluator

BYO Inc – Quality Assurance & Model Evaluator

Job Description:

BYO Inc. is seeking a skilled Quality Assurance & Model Evaluator to ensure the reliability, accuracy, and overall performance of our product. This role involves both traditional software quality assurance and specialized evaluation of large language models. The ideal candidate will have a strong analytical mindset, a deep understanding of natural language processing, and a passion for maintaining the highest standards in AI-generated content.

Requirements & Qualifications

Education:

Bachelor’s degree in Computer Science, Engineering, Data Science, or a related field (or equivalent practical experience).

Experience:

2+ years in Quality Assurance, data analysis, or software testing, ideally within an AI/ML or NLP environment.

Proven experience designing and executing comprehensive test plans, including both manual and automated testing frameworks.

Technical Skills:

Proficiency in Python and SQL, with hands-on experience using data analysis libraries such as Pandas and visualization tools like Matplotlib.

Basic understanding of NLP frameworks (e.g., HuggingFace) and evaluation metrics for language models.

Expertise in Quality Assurance:

Strong foundation in software testing methodologies, including functional, regression, and performance testing.

Experience in developing automated test scripts and integrating them into testing pipelines.

Familiarity with bug tracking and test management tools.

Ability to define quality standards and establish metrics to evaluate software performance

Domain Expertise:

Solid grasp of natural language processing and the unique challenges associated with evaluating large language model outputs.

Demonstrated ability to identify and mitigate issues such as grammatical errors, hallucinations, and other quality concerns.

Analytical & Communication Skills:

Strong problem-solving mindset with exceptional attention to detail.

Excellent written and verbal communication skills, with a proven track record of collaborating effectively in cross-functional teams.

Bonus Qualifications:

Experience working with production-level AI systems or in an operational setting that emphasizes model reliability and safety.