Software Engineer, Inference
$195k - $345k • Remote • Redwood City, CA • FullTime
Posted 21d ago
Job Location
Redwood City, CA
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
AI Infrastructure Engineer
About the job
Luma is seeking a Software Engineer to own the serving of their AI models. This role involves integrating new model architectures into the inference engine, scaling deployments across thousands of machines, and optimizing the utilization of expensive GPU fleets while meeting internal service level objectives (SLOs). The work focuses on large-scale inference systems, including scheduling, fleet management, deployment pipelines, and ensuring reliability across diverse clusters and hardware providers. This position is ideal for a strong systems engineer comfortable with model serving and Kubernetes at scale, rather than those seeking pure modeling work.
Responsibilities
- Integrate new model architectures into the inference engine.
- Collaborate with research, engineering, and infrastructure teams to optimize model efficiency and deployments.
- Develop internal tooling for measuring, profiling, and tracking inference jobs and workflows.
- Automate, test, and maintain inference services for maximum uptime and reliability.
- Manage and optimize inference workloads across clusters and hardware providers, scaling deployments.
- Build scheduling systems for optimal GPU resource utilization and SLO adherence.
- Maintain CI/CD pipelines for model checkpoints and SDKs.
Requirements
- Strong Python and system-architecture skills.
- Experience deploying models with PyTorch, Hugging Face, vLLM, SGLang, TensorRT-LLM, or similar.
- Experience with queues, scheduling, traffic control, and fleet management at scale.
- Experience with Linux, Docker, and Kubernetes, including orchestration, deployment, and scheduling.
- Familiarity with Redis and S3-compatible storage.