Senior Software Development Engineer in Test (SDET) - AI Cluster
Sunnyvale, CA • FullTime
Posted 3mo ago
About the job
Cerebras Systems is seeking a Senior Software Development Engineer in Test (SDET) to join their AI Cluster team. This role is crucial for ensuring the reliability of large-scale AI infrastructure deployments, aiming for 99.999% uptime. The ideal candidate will innovate and execute tests on cutting-edge AI infrastructure, define optimized test strategies, and adapt quickly to new technologies within the rapidly evolving ML community and AI model landscape. A key focus will be on automating tests to drive efficiency and scalability across all cluster features, including high availability, failure scenarios, performance, stress, and security.
Responsibilities
- Innovate and execute tests on cutting-edge AI infrastructure.
- Define optimized test strategies and methodologies.
- Adapt to new technologies and bring expertise to a diverse team.
- Break down large distributed systems challenges into smaller, unit-testable components.
- Implement an 'automate first' approach for 100% automated tests covering cluster features.
- Champion cluster security, reliability (99.9999% uptime), and ease of use with observability.
- Test all components of the AI cluster, including cluster software (Kubernetes, Prometheus, Grafana) and hardware (ML wafer scale accelerators, CPU runtime nodes, interconnects).
Requirements
- Bachelor's or Master's degree in Computer Science, Electrical Engineering, AI, Data Science, or a related field.
- 5+ years of experience testing enterprise software, distributed systems, or datacenter hardware and software.
- Strong coding skills in Python, Golang, or C/C++.
- Strong debugging skills for large distributed systems, hardware, and software, with experience in tools like pdb, gdb, strace, and network monitors.
- Strong understanding of operating system internals (memory management, file systems, security, performance).
- Strong understanding of datacenter layout and device performance characteristics (Servers, Memory, BIOS, PCIe, networking, storage).
- Experience with cloud technologies like AWS, Kubernetes, and Docker.
- Experience with monitoring tools like Grafana and Prometheus is a plus.
- Understanding and experience of ML model training and inference is a plus.
- Understanding of ML hardware accelerators like GPUs and custom ASICs is a plus.
Benefits
- Build a breakthrough AI platform beyond the constraints of the GPU.
- Publish and open source cutting-edge AI research.
- Work on one of the fastest AI supercomputers in the world.
- Job stability with startup vitality.
- Simple, non-corporate work culture that respects individual beliefs.