GenAI Platform Engineer – LLM Gateway
India • Full-time
Posted 3d ago
Job Location
India
Tech Stack
Remote Work Policy
On-site
Employment Type
Full-time
Categories
LLM Engineer
About the job
Accountabilities:
- Design, develop, and operate enterprise LLM Gateway features, primarily using Python, enabling secure and scalable access to multiple large language model providers.
- Build and maintain multi-provider model integrations, intelligent routing, failover mechanisms, API key management, and access control capabilities.
- Develop platform features for usage metering, budget management, cost transparency, and chargeback reporting.
- Integrate agentic AI capabilities into the platform while ensuring appropriate security and governance controls.
- Develop and maintain Model Context Protocol (MCP) servers, cross-provider tool calling, and agent-to-agent connectivity.
- Maintain and enhance the platform's open-source foundations by assessing upstream releases, managing custom extensions, and delivering upgrades without introducing regressions.
- Contribute to administrative interfaces, self-service portals, developer experiences, and technical documentation.
- Build and maintain automated quality gates within CI/CD pipelines, including unit, integration, contract, and performance testing.
- Implement observability solutions using metrics, distributed tracing, and logging tools such as OpenTelemetry and Langfuse.
- Monitor platform performance and proactively identify and resolve latency, reliability, error-rate, and infrastructure cost issues.
- Participate in production support and on-call rotations, leading root-cause investigations and resolving complex incidents.
- Conduct blameless postmortems and translate incident findings into sustainable engineering improvements.
- Apply secure-by-default engineering practices, including Single Sign-On (SSO), OAuth 2.0/OIDC, Role-Based Access Control (RBAC), and secure secrets management.
- Maintain dependency security, version pinning, and software supply-chain hygiene.
- Evaluate emerging generative AI technologies, including model routing solutions, guardrails, agent registries, and agentic frameworks, and provide recommendations for adoption.
- Support internal engineering teams in onboarding and effectively using the platform through technical guides, runbooks, code samples, and reusable AI agent skills.
- Contribute to InnerSource initiatives, participate in code reviews, and collaborate with open-source communities and technology vendors.
- Identify opportunities to improve platform capabilities through upstream contributions, technical experimentation, and continuous innovation.
Requirements:
- Bachelor's or Master's degree in Computer Science, Computer Applications, Engineering, or a related discipline, or equivalent practical experience.
- At least 9 years of professional software engineering experience, including a minimum of 3 years developing and delivering production-grade GenAI or LLM-based applications and platforms.
- Strong proficiency in Python backend development, including asynchronous programming, FastAPI or similar frameworks, REST APIs, and streaming responses.
- Hands-on experience integrating LLM provider APIs, such as OpenAI, Azure OpenAI, and AWS Bedrock.
- Experience implementing tool calling, function calling, and integrations across multiple LLM providers.
- Proven experience deploying and operating services on AWS using containerization technologies and CI/CD pipelines.
- Practical knowledge of Docker, GitHub Actions, and automated testing integrated into software delivery workflows.
- Strong understanding of backend architecture, scalable services, API integrations, and production software engineering practices.
- Demonstrated ability to troubleshoot complex production issues, conduct root-cause analysis, and implement long-term solutions.
- Excellent written and verbal communication skills in English, with the ability to collaborate effectively across globally distributed teams.
- Ability to work independently, take ownership of technical deliverables, and contribute to cross-functional engineering initiatives.
Preferred qualifications and experience:
- Experience working with AI/LLM gateways or API gateways, particularly model routing, API key management, access controls, and cost tracking.
- Experience operating and extending open-source software in production environments, including custom patches, upgrades, regression testing, and upstream contributions.
- Knowledge of agentic AI architectures and tools, particularly Model Context Protocol (MCP).
- Experience with LangGraph, LangChain, or similar agent orchestration frameworks.
- Understanding of multi-agent orchestration, Retrieval-Augmented Generation (RAG), and vector search technologies.
- Experience with performance and load testing, as well as observability platforms such as OpenTelemetry, Langfuse, Prometheus, and Grafana.
- Familiarity with Kubernetes and Infrastructure as Code (IaC) tools such as Terraform.
- Frontend development experience with React and TypeScript for building internal applications, administrative tools, and developer portals.
- Regular use of AI-assisted development tools such as GitHub Copilot and Claude Code, with an interest in creating reusable AI agent skills.
- Experience working in pharmaceutical, healthcare, or other regulated industries.
- Previous experience mentoring engineers or delivering technical enablement through workshops, documentation, and onboarding programs.
- Relevant AWS certifications are considered an advantage.
Benefits:
- Full-time employment opportunity.
- Fully remote working arrangement from India.
- Opportunity to work on enterprise-scale generative AI infrastructure and LLM platform engineering.
- Exposure to leading AI technologies and multiple LLM providers, including OpenAI, Azure OpenAI, and AWS Bedrock.
- Hands-on experience with modern agentic AI frameworks, MCP integrations, and multi-agent architectures.
- Opportunity to contribute to open-source projects and influence the evolution of enterprise AI platforms.
- Professional development through exposure to cloud-native technologies, advanced automation, and emerging AI capabilities.
- Collaboration with globally distributed engineering teams and international technical specialists.
- Opportunity to take ownership of impactful engineering projects and influence platform architecture and technical direction.
- Regular collaboration with colleagues in European time zones.
- Participation in a structured production support and on-call rotation.
