Senior Machine Learning Engineer- LLMs & Self-Hosted AI

We are looking for a highly skilled Senior Machine Learning Engineer to lead our transition from on-demand, third-party LLM APIs to a fully self-hosted, scalable model ecosystem.

Our core product is an advanced, agentic support chatbot capable of complex reasoning, API tool calling, database lookups, and orchestrating specialized Small Language Models (SLMs) for targeted NLP tasks. As we scale, our current deployment infrastructure (AWS SageMaker) is becoming unsustainable. You will be responsible for architecting, deploying, and optimizing an infrastructure capable of supporting 50 to 100 distinct models ranging from 100M to 70B parameters.

What You’ll Do

• Inference Optimization: Deploy and manage large-scale models using high-performance inference engines (like vLLM) to ensure low latency and high throughput for our agentic chatbot.
• Agentic Workflows: Develop and refine the chatbot's agentic capabilities, ensuring reliable tool-use, routing, and interactions between massive LLMs and specialized SLMs.
• Model Fine-Tuning: Design and execute fine-tuning strategies to improve model accuracy on specific domain tasks and tool-calling execution.
• Rigorous Evaluation: Build comprehensive offline and online evaluation frameworks to constantly measure model performance and business impact through structured A/B testing.
  
  

• Strong proficiency in Python and Bash scripting.
• Deep experience with PyTorch and the Hugging Face ecosystem.
• Experience using AI coding assistants natively in the terminal, specifically Claude Code, to accelerate development workflows.
  
  

• Proven experience deploying models using vLLM, TGI, or similar high-performance inference servers.
• Strong fundamental understanding of LLM architectures, attention mechanisms, and generation parameters.
• Hands-on experience building Agentic systems (ReAct, function/tool calling, RAG).
• Expertise in fine-tuning strategies (e.g., SFT, RLHF, DPO) and parameter-efficient techniques (PEFT/LoRA).
  
  

• Offline Metrics: Deep understanding of classification/summarization metrics (Precision, Recall, F1, AUC) and retrieval metrics (MRR, NDCG, Precision/Recall @ k).
• Online Metrics & A/B Testing: Strong statistical foundation to design and analyze A/B tests safely, including the use of t-tests, Mann-Whitney U tests, and bootstrapping techniques.
  
  

• Containerization & Orchestration: Experience with Ray for orchestrating large-scale model deployments across multi-GPU clusters.
• Model Quantization: Experience with memory optimization techniques like AWQ, GPTQ, GGUF, or FlashAttention to fit 70B models efficiently onto hardware.
• API Development: Proficiency in building robust, asynchronous microservices using FastAPI to serve model requests.
• Knowledge of Data Engineering principles: dataset collection, cleaning, processing, and scalable storage.