Distributed Software Engineer
Salary not listed • Toronto, CAN • Full-time
Posted 1mo ago
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Infrastructure Engineer
About the job
Cerebras Systems is seeking a Distributed Software Engineer to join their Cluster Engineering team. This team is responsible for the software that manages thousands of wafers, servers, and switches, transforming them into a reliable, efficient, and observable cloud for AI workloads. The role involves building and operating the software stack, which is primarily Go and Python on Kubernetes, for both on-premise deployments and Cerebras's own cloud. The goal is to ensure clusters are stable, workloads are scheduled effectively, and the system is observable to users, operators, and AI agents.
Responsibilities
- Develop declarative, CRD-driven automation for bare-metal networking, OS, and application software across large clusters.
- Implement push-button cluster installation, upgrade, and security patching processes with defined downtime budgets.
- Create Kubernetes operators for scheduling large inference workloads, including resource locking, priority queues, and health-aware placement.
- Build gRPC control-plane services for authorization, admission webhooks, and quota policy in a multi-tenant environment.
- Develop and maintain metrics and log pipelines with custom exporters for wafer-scale systems, servers, and network fabric.
- Implement failure detection, high-availability control planes, and automated recovery mechanisms.
- Expose fleet management capabilities through CLIs, APIs, and an MCP gateway for users, operators, and AI agents.
Requirements
- 5+ years of experience building and operating production distributed systems or infrastructure software.
- Proficiency in production-quality Go and Python.
- Deep Kubernetes expertise, including writing or debugging controllers and operators, and understanding CRDs, reconciliation semantics, and RBAC.
- Strong debugging skills across distributed systems, Linux, and networking.
- Practitioner-level experience with Prometheus and Grafana, including PromQL, exporter design, and alerting.
- Demonstrated ability to work independently, build context, make decisions, and drive work across teams.
- Experience adopting AI in engineering workflows, including active use of coding agents and rigorous verification of their output.
Benefits
- Build a breakthrough AI platform beyond the constraints of the GPU.
- Publish and open source cutting-edge AI research.
- Work on one of the fastest AI supercomputers in the world.
- Job stability with startup vitality.
- A simple, non-corporate work culture that respects individual beliefs.