Senior Deep Learning Engineer, Cosmo 3D Spatial
Remote • US, CA, Santa Clara
Posted 1mo ago
Job Location
US, CA, Santa Clara
Tech Stack
Remote Work Policy
Fully remote
Categories
Machine Learning Engineer
About the job
NVIDIA is at the forefront of the AI revolution, exploring the next frontier of Physical AI, which involves machines that can perceive, reason about, and act within the three-dimensional world. The NVIDIA Cosmos platform provides open foundation models for Physical AI, designed to interpret images, video, and text to create a structured understanding of physical scenes, including motion, object interactions, geometry, and physical context. These models serve as the reasoning layer for robots, autonomous vehicles, and smart infrastructure. The Cosmos Engineering team is responsible for developing the core capabilities behind these models and is seeking a Senior Deep Learning Engineer to lead the data engine and end-to-end training verification for 3D spatial reasoning and perception. This role involves ensuring models accurately learn and represent 3D spatial concepts, working closely with research scientists to translate hypotheses into measurable capabilities in shipped models, and focusing on the critical role of data and evaluations in achieving frontier model quality.
Responsibilities
- Own the 3D data engine for Cosmos spatial reasoning, including sourcing, curating, filtering, and balancing large-scale real-world image and video corpora into vision-language training data.
- Build annotation and auto-labeling pipelines for large-scale 3D-grounded supervision, including various spatial reasoning tasks.
- Manage end-to-end data quality, including semantic deduplication, automated quality scoring, coverage analysis, and sampling strategies for training mixtures.
- Oversee and validate end-to-end model training pipelines (pre-training and supervised fine-tuning), ensuring reproducibility and diagnosing anomalies.
- Develop and operate the 3D and spatial evaluation suite, including public benchmarks and in-house benchmarks with full traceability.
- Collaborate with research scientists to translate 3D research hypotheses into dataset and ablation experiments.
- Optimize data throughput, sharding, and dataloader performance on large multi-node GPU clusters.
- Ship results into Cosmos releases and open-source contributions, enhancing data and evaluation rigor.
Requirements
- MS or PhD in Computer Science, Electrical/Computer Engineering, Robotics, or a related field, or equivalent experience.
- 12+ years of proven experience building deep learning systems in Python with PyTorch or JAX on Linux.
- Deep expertise in 3D computer vision, multi-view geometry, structure-from-motion or SLAM, depth and camera pose estimation, point cloud processing, or 3D reconstruction, with the ability to produce and validate 3D ground truth at scale.
- Hands-on experience with vision-language models, including building training data and evaluations.
- Demonstrated experience building large-scale multimodal data pipelines (distributed video/image processing, deduplication, annotation, quality metrics, dataset versioning).
- Experience running and validating large model training on multi-GPU, multi-node clusters, including distributed training and sharding strategies (data, tensor, pipeline parallelism, FSDP).
- Rigorous evaluation methodology, including designing robust benchmarks and conducting honest analysis.
- Excellent written and verbal communication skills, with a track record of effective collaboration with research scientists.
Benefits
- Competitive salaries
- Generous benefits package