Senior Software Engineer, Training & Experimentation
$180k - $250k • London, England, United Kingdom; New York, New York, United States; San Francisco, California, United States • Full-time
Posted 2mo ago
This job has been open for more than 60 days. It may no longer be actively hiring, so check with the company before putting a lot into an application.
Job Location
London, England, United Kingdom; New York, New York, United States; San Francisco, California, United States
Tech Stack
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Agent Engineer
About the job
Lightning AI is seeking a Senior Software Engineer to join the Training & Experimentation team. This team builds the platform that enables developers to experiment at scale and train their own intelligence, covering everything from foundation model training to fine-tuning open-source models and launching distributed training jobs. The role involves designing and scaling backend systems to make AI development faster, more reliable, and easier to use, working across distributed training infrastructure, workload orchestration, experiment management, developer tooling, and platform APIs. This is an opportunity to build systems that support real-world AI workloads at scale and shape the developer experience for a global user base.
Responsibilities
- Design and build backend services for AI agent orchestration and execution.
- Develop scalable APIs and platform capabilities for tool use, workflow orchestration, memory, and state management.
- Build reliable infrastructure for long-running, distributed AI workflows.
- Partner with product, research, and platform engineering teams to productionize new agent capabilities.
- Improve platform reliability, observability, and performance.
- Evaluate and strengthen technical architecture, engineering processes, and developer tooling.
- Maintain high software quality through testing, automation, and continuous delivery.
- Mentor engineers on system design, distributed systems, and backend engineering best practices.
Requirements
- Extensive experience building scalable backend systems using Go, Python, Rust, or similar languages.
- Strong understanding of distributed systems, asynchronous processing, APIs, and cloud-native application architecture.
- Experience designing and operating production services at scale.
- Hands-on experience building backend platforms, developer tools, or infrastructure products.
- Familiarity with cloud platforms (AWS, GCP, Azure) and container technologies (Docker, Kubernetes).
- Proven ability to own large technical initiatives from architecture through production.
- Commitment to software quality, automated testing, observability, and continuous delivery.
- Strong communication and collaboration skills.
- Ability to thrive in fast-moving, ambiguous environments.
Benefits
- Discretionary bonus
- Meaningful equity (RSUs)
- Comprehensive Health Coverage (Medical, dental, vision)
- Retirement Savings (401(k) matching in U.S., pension contributions in U.K.)
- Unlimited PTO
- Company holidays
- Floating holidays