Senior Research Engineer, LLM Training & Post-Training
$165k - $310k • Remote • New York, New York, United States; Remote; San Francisco, California, United States; Seattle, Washington, United States
Posted 1mo ago
Job Location
New York, New York, United States; Remote; San Francisco, California, United States; Seattle, Washington, United States
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Research Engineer
About the job
Lightning AI is seeking an experienced Senior Research Engineer to join their Research Engineering function. This role will focus on advancing the training, fine-tuning, evaluation, and deployment of large language models across Lightning AI's platform and customer workloads. The ideal candidate will have deep experience training and improving transformer-based language models, strong software engineering fundamentals, and a passion for solving complex problems in model training, evaluation, and AI systems. This position offers the opportunity to work on models that power the Lightning AI platform, customer-specific workloads, and research that translates into reusable training and platform capabilities.
Responsibilities
- Design, build, and optimize training and post-training pipelines for large language models.
- Improve model quality through supervised fine-tuning, continued pretraining, preference optimization, reinforcement learning, evaluation, and experimentation.
- Build and improve PyTorch-based training infrastructure, tooling, and developer workflows.
- Optimize distributed training across multi-GPU environments by improving throughput, memory efficiency, scalability, and GPU utilization.
- Investigate model training issues, including convergence, instability, communication overhead, and performance bottlenecks.
- Design evaluation methodologies, benchmark models, analyze failure modes, and achieve model improvements through experimentation.
- Collaborate directly with customers to understand real-world workloads and translate those learnings into improvements across Lightning AI's research platform.
- Partner closely with research, infrastructure, and platform engineering teams to build production-ready AI systems.
- Contribute to open-source projects through new features, tooling improvements, documentation, and community engagement.
Requirements
- Significant experience training, fine-tuning, evaluating, and/or optimizing transformer-based language models using PyTorch.
- Experience with modern LLM training techniques.