Staff Software Engineer, Inference API
Remote • Sunnyvale, CA • FullTime
Posted 24d ago
About the job
Cerebras is developing a new generation of disaggregated AI inference systems that combine GPU-accelerated prefill with ultra-fast decode on the Cerebras Wafer-Scale Engine. We are seeking a Software Engineer to build and enhance the ML API layer, making this heterogeneous serving system accessible, reliable, and user-friendly. You will operate across our inference APIs, model integration layer, request-routing services, and the Cerebras inference platform to ensure a consistent experience across various models and accelerator backends. This role is at the intersection of machine learning systems, API design, model serving, and distributed systems, enabling new model architectures and inference capabilities while defining stable user-facing behavior and ensuring features like streaming, sampling, tool use, structured outputs, and multimodal inputs function correctly in production.
Responsibilities
- Build and maintain production ML inference APIs for chat completions, text generation, streaming, model configuration, tool calling, structured outputs, and multimodal inputs.
- Create a unified serving experience with consistent request and response semantics across different inference backends.
- Integrate emerging foundation models, tokenizers, prompt formats, sampling methods, and other model-specific features into the serving platform.
- Maintain API compatibility with widely adopted interfaces while designing Cerebras-specific extensions, establishing clear versioning and deprecation practices.
- Extend and integrate custom inference services with vLLM, PyTorch, Hugging Face libraries, AMD ROCm, and Cerebras runtime components.
- Build control and data paths for coordinating GPU prefill with Cerebras decode, including request routing and error handling.
- Optimize streaming behavior, time to first token, request latency, throughput, batching, and communication between serving components.
- Develop validation systems for tokenization, sampling, logits, generated outputs, and compatibility across serving backends.
- Define service indicators and build structured logging, tracing, metrics, and diagnostic tooling for production inference traffic.
- Create conformance tests, workload-replay tools, model-validation suites, and performance benchmarks.
- Develop intuitive configuration, SDKs, documentation, examples, and debugging tools to improve developer experience.
- Collaborate with compiler, runtime, kernel, cloud, product, and solutions teams to translate requirements into scalable serving capabilities.
Requirements
- 5+ years of software engineering experience, with individual-contributor ownership of production software or distributed systems.
- Strong programming ability in Python and Go.
- Experience developing performance-sensitive or highly concurrent services in C++, Rust, or a similar systems language.
- Experience building stable APIs with clear validation, error handling, observability, compatibility, and versioning practices.
- Experience integrating software across service, framework, runtime, and infrastructure boundaries.
- Experience designing or maintaining OpenAI-compatible, gRPC, REST, or streaming inference APIs.
- Experience with Linux, containers, Kubernetes or comparable orchestration systems, CI/CD, and operating latency-sensitive services in production.
- Ability to diagnose correctness, reliability, and performance issues across multiple components of a distributed serving system.
- Strong communication and cross-functional execution skills.
- Bachelor's degree in computer science, Computer Engineering, Electrical Engineering, or a related discipline, or equivalent practical experience.