Senior / Staff Software Engineer — AI Platform
Remote • New York City, NY (Hybrid); San Francisco, CA (Hybrid)
Posted 4d ago
Job Location
New York City, NY (Hybrid); San Francisco, CA (Hybrid)
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Agent Engineer
About the job
Snorkel AI is seeking experienced Software Engineers at Senior/Staff levels to join their AI Platform team. This role involves building the core infrastructure and systems that power AI development at scale, focusing on synthetic data generation, evaluation, agentic workflows, simulation environments, and LLM infrastructure. You will work on systems designed to make inherently non-deterministic workloads observable, reproducible, measurable, and scalable, enabling rapid experimentation and reliable production systems for AI models and agents.
Responsibilities
- Design and build infrastructure for large-scale agentic workloads, including multi-step agents interacting with tools and environments.
- Build scalable synthetic data generation and automated labeling systems for training and evaluation datasets.
- Design evaluation infrastructure for measuring AI system behavior, including reproducible experiments and continuous evaluation.
- Build orchestration and distributed compute systems for running AI experiments and simulations reliably.
- Develop infrastructure for agent simulation environments, including provisioning, isolation, and scalable execution.
- Build and operate LLM infrastructure for routing, rate limiting, caching, and efficient execution across multiple model providers.
- Instrument agent and model workloads to make failures observable and debuggable, capturing key metrics and state.
- Design systems to make non-deterministic workloads reproducible and measurable for experiment comparison and diagnosis.
- Improve developer experience for AI experimentation through APIs, SDKs, and workflow abstractions.
- Collaborate with research, product, and engineering teams to translate experimental workflows into platform capabilities.
Requirements
- 5+ years building production software systems, with experience in AI/ML infrastructure, ML platforms, distributed systems, data platforms, or backend infrastructure.
- Experience operating non-deterministic AI or ML workloads in production or at significant scale.
- Experience building infrastructure for experimentation, evaluation, model development, synthetic data, agentic workflows, training, inference, or production ML systems.
- Strong proficiency in Python and experience building production-quality APIs, services, and developer tooling.
- Strong background in distributed systems and cloud platforms (AWS preferred), including compute orchestration, storage, networking, isolation, and failure handling.
- Experience with workflow or distributed execution frameworks (e.g., Prefect, Airflow, Dagster, Ray, Kubernetes).
- Strong understanding of production system fundamentals: observability, telemetry, reliability, performance, debugging, incident response, and cost management.
- Ability to reason about AI system quality beyond traditional metrics, including evaluation design and experiment reproducibility.
- Track record of leading complex engineering initiatives and influencing stakeholders.
- Ability to work in a fast-paced, research-driven environment.