Senior Systems Engineer, Workers AI
$194k - $266k • In-Office
Posted 5d ago
Job Location
In-Office
Tech Stack
Remote Work Policy
On-site
Categories
AI Infrastructure Engineer
About the job
You will design and build the core infrastructure that powers AI inference across Cloudflare's global network. This includes real-time voice, frontier open LLMs, and customer-deployed models running on a heterogeneous fleet of GPUs and next-generation accelerators. You will work alongside AI/ML engineers, hardware partners, and Cloudflare product teams to solve complex problems in distributed systems and high-performance computing. The role involves addressing challenges such as sub-second model cold starts, multi-accelerator workload scheduling, efficient KV cache management, and developing a model deployment platform for both internal use and customer models. This is an opportunity to be at the center of building a novel AI inference platform embedded in the fabric of the internet, defining how AI runs at the edge of the network.
Responsibilities
- Develop and maintain core components of the serverless inference platform for high availability and scalability.
- Optimize the model scheduling system for increased efficiency and resource utilization.
- Implement improvements to request routing logic to reduce end-user latency.
- Drive measurable improvements in platform reliability and resilience by mitigating systemic risks.
- Expand and refine the observability stack (metrics, logging, tracing) and tune alerts.
- Lead complex, cross-functional technical projects from concept to deployment.
- Mentor junior engineers and contribute to a collaborative engineering culture.
Requirements
- Proven experience in systems engineering, with a focus on distributed, high-performance systems.
- Expert proficiency in Rust programming, particularly in an asynchronous environment.
- Deep understanding and hands-on experience with networking and application protocols (e.g., TCP, HTTP, WebSocket).
- Solid experience with scaling and performance optimization techniques, including load balancing and caching in a distributed environment.