New: upload your resume and get matched to AI jobs. Free to start →

GenAI Platform Engineer – LLM Gateway

India • Full-time

Posted 3d ago

Remote Work Policy

On-site

Employment Type

Full-time

Categories

LLM Engineer

About the job

Accountabilities:

  • Design, develop, and operate enterprise LLM Gateway features, primarily using Python, enabling secure and scalable access to multiple large language model providers.
  • Build and maintain multi-provider model integrations, intelligent routing, failover mechanisms, API key management, and access control capabilities.
  • Develop platform features for usage metering, budget management, cost transparency, and chargeback reporting.
  • Integrate agentic AI capabilities into the platform while ensuring appropriate security and governance controls.
  • Develop and maintain Model Context Protocol (MCP) servers, cross-provider tool calling, and agent-to-agent connectivity.
  • Maintain and enhance the platform's open-source foundations by assessing upstream releases, managing custom extensions, and delivering upgrades without introducing regressions.
  • Contribute to administrative interfaces, self-service portals, developer experiences, and technical documentation.
  • Build and maintain automated quality gates within CI/CD pipelines, including unit, integration, contract, and performance testing.
  • Implement observability solutions using metrics, distributed tracing, and logging tools such as OpenTelemetry and Langfuse.
  • Monitor platform performance and proactively identify and resolve latency, reliability, error-rate, and infrastructure cost issues.
  • Participate in production support and on-call rotations, leading root-cause investigations and resolving complex incidents.
  • Conduct blameless postmortems and translate incident findings into sustainable engineering improvements.
  • Apply secure-by-default engineering practices, including Single Sign-On (SSO), OAuth 2.0/OIDC, Role-Based Access Control (RBAC), and secure secrets management.
  • Maintain dependency security, version pinning, and software supply-chain hygiene.
  • Evaluate emerging generative AI technologies, including model routing solutions, guardrails, agent registries, and agentic frameworks, and provide recommendations for adoption.
  • Support internal engineering teams in onboarding and effectively using the platform through technical guides, runbooks, code samples, and reusable AI agent skills.
  • Contribute to InnerSource initiatives, participate in code reviews, and collaborate with open-source communities and technology vendors.
  • Identify opportunities to improve platform capabilities through upstream contributions, technical experimentation, and continuous innovation.

Requirements:

  • Bachelor's or Master's degree in Computer Science, Computer Applications, Engineering, or a related discipline, or equivalent practical experience.
  • At least 9 years of professional software engineering experience, including a minimum of 3 years developing and delivering production-grade GenAI or LLM-based applications and platforms.
  • Strong proficiency in Python backend development, including asynchronous programming, FastAPI or similar frameworks, REST APIs, and streaming responses.
  • Hands-on experience integrating LLM provider APIs, such as OpenAI, Azure OpenAI, and AWS Bedrock.
  • Experience implementing tool calling, function calling, and integrations across multiple LLM providers.
  • Proven experience deploying and operating services on AWS using containerization technologies and CI/CD pipelines.
  • Practical knowledge of Docker, GitHub Actions, and automated testing integrated into software delivery workflows.
  • Strong understanding of backend architecture, scalable services, API integrations, and production software engineering practices.
  • Demonstrated ability to troubleshoot complex production issues, conduct root-cause analysis, and implement long-term solutions.
  • Excellent written and verbal communication skills in English, with the ability to collaborate effectively across globally distributed teams.
  • Ability to work independently, take ownership of technical deliverables, and contribute to cross-functional engineering initiatives.

Preferred qualifications and experience:

  • Experience working with AI/LLM gateways or API gateways, particularly model routing, API key management, access controls, and cost tracking.
  • Experience operating and extending open-source software in production environments, including custom patches, upgrades, regression testing, and upstream contributions.
  • Knowledge of agentic AI architectures and tools, particularly Model Context Protocol (MCP).
  • Experience with LangGraph, LangChain, or similar agent orchestration frameworks.
  • Understanding of multi-agent orchestration, Retrieval-Augmented Generation (RAG), and vector search technologies.
  • Experience with performance and load testing, as well as observability platforms such as OpenTelemetry, Langfuse, Prometheus, and Grafana.
  • Familiarity with Kubernetes and Infrastructure as Code (IaC) tools such as Terraform.
  • Frontend development experience with React and TypeScript for building internal applications, administrative tools, and developer portals.
  • Regular use of AI-assisted development tools such as GitHub Copilot and Claude Code, with an interest in creating reusable AI agent skills.
  • Experience working in pharmaceutical, healthcare, or other regulated industries.
  • Previous experience mentoring engineers or delivering technical enablement through workshops, documentation, and onboarding programs.
  • Relevant AWS certifications are considered an advantage.

Benefits:

  • Full-time employment opportunity.
  • Fully remote working arrangement from India.
  • Opportunity to work on enterprise-scale generative AI infrastructure and LLM platform engineering.
  • Exposure to leading AI technologies and multiple LLM providers, including OpenAI, Azure OpenAI, and AWS Bedrock.
  • Hands-on experience with modern agentic AI frameworks, MCP integrations, and multi-agent architectures.
  • Opportunity to contribute to open-source projects and influence the evolution of enterprise AI platforms.
  • Professional development through exposure to cloud-native technologies, advanced automation, and emerging AI capabilities.
  • Collaboration with globally distributed engineering teams and international technical specialists.
  • Opportunity to take ownership of impactful engineering projects and influence platform architecture and technical direction.
  • Regular collaboration with colleagues in European time zones.
  • Participation in a structured production support and on-call rotation.