Senior Software Engineer, AI Infrastructure & Benchmarking

The Senior Software Engineer is responsible for designing, developing, and maintaining the large-scale software systems that power Red Hat’s model benchmarking and validation workflows. Unlike a Machine Learning Engineer, whose primary focus is on model performance, accuracy evaluation, and ML experimentation, the Senior Software Engineer’s focus is on building robust, scalable, and maintainable infrastructure that enables those evaluations to run efficiently at scale.

This role combines deep expertise in distributed systems, automation, and cloud-native architecture with a strong command of modern engineering tools and frameworks used across the industry. The engineer is expected to design, implement, and contribute to open-source tools that integrate seamlessly with the AI benchmarking ecosystem, while staying at the forefront of emerging technologies and best practices.

A core aspect of this position is developing a generic, highly scalable benchmarking platform designed to set the industry standard for AI model benchmarking. This requires exceptional software engineering quality, architectural foresight, and the ability to build systems that are flexible enough to support diverse use cases while remaining highly reliable and efficient in production environments.

What you will do:

Platform Architecture & Development: Lead the design and implementation of a modular, extensible, and cloud-native benchmarking platform capable of supporting large-scale AI model evaluations across diverse workloads and hardware.

Scalable Infrastructure: Build distributed systems optimized for high-throughput and low-latency execution of benchmarks, ensuring the platform can scale seamlessly to meet growing demands.

Open-Source Leadership: Design and contribute to open-source benchmarking and validation tools, fostering community adoption and influencing industry benchmarking standards.

Integration with AI Tooling: Develop robust APIs, services, and orchestration layers to connect the benchmarking platform with inference engines and downstream consumers such as AI Hub and observability platforms.

Workflow Automation: Implement advanced automation for scheduling, executing, and monitoring benchmarks using Kubernetes, OpenShift, and Argo Workflows.

Data Systems & APIs: Build reliable data pipelines for benchmark result ingestion, storage, querying, and integration with decision-support tools.

Performance & Reliability Engineering: Proactively identify and resolve system bottlenecks, optimize for resource efficiency, and ensure high availability.

Engineering Excellence: Apply industry-leading development practices, including automated testing, CI/CD, and rigorous code review, to maintain exceptional software quality.

Ecosystem Awareness: Maintain deep knowledge of the AI infrastructure and benchmarking ecosystem, tracking emerging frameworks, standards, and best practices to guide platform evolution.

What you will bring:

Proficiency in Python for backend development, API integration, and data processing.

Hands-on experience with Kubernetes (Deployments, Services, Ingress).

Ability to create and maintain Helm charts.

Strong understanding of Docker for building and managing containers.

Experience with CI/CD pipelines (GitHub Actions or Jenkins).

Proficiency with Argo CD for workflow orchestration and GitOps.

Familiarity with model serving frameworks such as vLLM, TGI, or LMDeploy.

Experience with cloud platforms (AWS or GCP)

Knowledge of monitoring tools such as Prometheus, Grafana, or Streamlit.

Experience working with PostgreSQL and an ORM such as SQLAlchemy.

Nice to Have

Experience with Go for tooling and infrastructure development.

Familiarity with Argo Workflows for pipeline orchestration.

Experience with OpenShift or RHOAI.

Direct hands-on experience with cloud GPUs.

Knowledge of rollout event monitoring and advanced observability practices.

Familiarity with artifact and code security scanning tools (e.g., Trivy, Grype).

Experience with LLM benchmarking frameworks such as GuideLLM.

About Red Hat

Red Hat is the world’s leading provider of enterprise open source software solutions, using a community-powered approach to deliver high-performing Linux, cloud, container, and Kubernetes technologies. Spread across 40+ countries, our associates work flexibly across work environments, from in-office, to office-flex, to fully remote, depending on the requirements of their role. Red Hatters are encouraged to bring their best ideas, no matter their title or tenure. We're a leader in open source because of our open and inclusive environment. We hire creative, passionate people ready to contribute their ideas, help solve complex problems, and make an impact.

Inclusion at Red Hat
Red Hat’s culture is built on the open source principles of transparency, collaboration, and inclusion, where the best ideas can come from anywhere and anyone. When this is realized, it empowers people from different backgrounds, perspectives, and experiences to come together to share ideas, challenge the status quo, and drive innovation. Our aspiration is that everyone experiences this culture with equal opportunity and access, and that all voices are not only heard but also celebrated. We hope you will join our celebration, and we welcome and encourage applicants from all the beautiful dimensions that compose our global village.

Equal Opportunity Policy (EEO)
Red Hat is proud to be an equal opportunity workplace and an affirmative action employer. We review applications for employment without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, ancestry, citizenship, age, veteran status, genetic information, physical or mental disability, medical condition, marital status, or any other basis prohibited by law.

Red Hat does not seek or accept unsolicited resumes or CVs from recruitment agencies. We are not responsible for, and will not pay, any fees, commissions, or any other payment related to unsolicited resumes or CVs except as required in a written contract between Red Hat and the recruitment agency or party requesting payment of a fee.

Red Hat supports individuals with disabilities and provides reasonable accommodations to job applicants. If you need assistance completing our online job application, email application-assistance@redhat.com. General inquiries, such as those regarding the status of a job application, will not receive a reply.