Senior Software Engineer, ML API – AI Inference
Salary not listed • Toronto, CAN • Full-time
Posted 3d ago
Job Location
Toronto, CAN
Tech Stack
Remote Work Policy
On-site
Employment Type
Full-time
Categories
LLM Engineer
About the job
Cerebras Systems is developing a new generation of disaggregated AI inference systems that combine GPU-accelerated prefill with ultra-fast decode on the Cerebras Wafer-Scale Engine. We are seeking a Senior Software Engineer to build and enhance the ML API layer, making this heterogeneous serving system accessible, reliable, and user-friendly. You will be responsible for significant features and components within our inference APIs, model integration layer, and request-routing services, ensuring a consistent experience across various models and accelerator backends. This role is at the intersection of machine learning systems, API design, model serving, and distributed systems, requiring you to translate evolving model capabilities and customer needs into clear designs, lead implementation, and manage features through validation, rollout, and production.
Responsibilities
- Own features end-to-end, including design, implementation, testing, and operation for chat completions, text generation, streaming, model configuration, tool calling, structured outputs, multimodal inputs, and emerging inference capabilities.
- Lead multi-component delivery by writing design documents, evaluating technical tradeoffs, breaking down projects into milestones, and coordinating implementation with engineers and partner teams.
- Ensure consistent API behavior across GPU prefill, Cerebras decode, and other inference backends, maintaining compatibility with adopted interfaces and evolving APIs through best practices.
- Integrate foundation models, tokenizers, chat templates, generation configuration, sampling methods, and model-specific features into the serving platform.
- Extend inference services and their integration with frameworks and runtimes like vLLM, PyTorch, Hugging Face libraries, LLGuidance, and Cerebras runtime components.
- Implement routing, state-transfer integration, error handling, retries, cancellation, and request lifecycle management for disaggregated inference.
- Deliver measurable performance improvements by diagnosing bottlenecks and enhancing time to first token, streaming latency, throughput, and resource efficiency.
- Ensure correctness and release quality through conformance tests, model-validation suites, numerical comparisons, integration tests, and performance benchmarks.
- Strengthen production reliability by improving service metrics, tracing, logging, alerts, and diagnostic tools, and investigating production incidents.
- Improve customer and developer experience by partnering with product, support, and solutions teams to understand user needs and enhance configuration, documentation, examples, and debugging workflows.
- Raise the engineering quality bar through thoughtful design and code reviews, mentoring engineers, sharing domain knowledge, and proactively removing roadblocks.
Requirements
- 5+ years of software engineering experience, including independent ownership of production services, distributed systems, or substantial software components.
- Strong programming ability in Python and proficiency in Go, C++, Rust, or another production service language.
- Experience designing and delivering features across multiple components, including documenting designs, evaluating tradeoffs, and coordinating implementation.
- Experience building and evolving production APIs with sound practices for validation, error handling, compatibility, and testing; familiarity with REST, gRPC, or streaming protocols.
- Experience integrating software across service, framework, runtime, or infrastructure boundaries.
- Experience operating services on Linux using containers, Kubernetes or comparable orchestration systems, and CI/CD.
- Ability to diagnose correctness, reliability, and performance issues across distributed components and deliver measurable improvements.
- Strong communication and collaboration skills, including experience leading technical delivery for a small project and supporting other engineers.
- Ability to translate ambiguous requirements into maintainable software while balancing customer needs, delivery timelines, and operational quality.
- Bachelor’s degree in Computer Science.