Member of Technical Staff - Pretraining

About the job

Responsibilities

• Lead large-scale pretraining experiments for multimodal foundation models, including architecture, objective functions, and scaling strategies.
• Develop and evaluate novel ideas in architecture, optimizers, and training algorithms.
• Contribute to the full stack, from low-level GPU and systems optimizations to research code and high-level model design.
• Independently lead focused research projects and drive larger cross-team initiatives.

Requirements

• Led or co-owned pretraining for a foundation model (image, video, LLM, or multimodal) that shipped to production or a major release.
• Own architectural decisions impacting the model, such as attention patterns, modulation schemes, loss formulations, and tokenization strategies.
• Deep experience with large-scale distributed training, including FSDP/TP/PP, multi-node runs at 500+ GPUs, and debugging complex training issues.
• Strong intuition for architecture and objective design, with a history of making impactful design choices for real models.
• Track record of shipping, demonstrated by top-venue publications and/or significant production impact.
• Deep proficiency in Python and PyTorch, with the ability to read and modify low-level training code.
• Familiarity with visual generative models is required.

Benefits

• Equity
• Reasonable travel costs covered for in-person meetings