Sr. Software Engineer, Multimodal Understanding
$180k - $440k • Remote • Palo Alto, CA
Posted 21h ago
About the job
SpaceXSI is building Super Intelligence to help humanity understand the universe, and you will join the multimodal team to push toward superhuman multimodal intelligence. This role involves advancing understanding and generation across modalities—image, video, audio, and text—spanning the full stack from data curation to end-to-end product experiences. You will collaborate cross-functionally to deliver frontier capabilities in multimodal reasoning, world modeling, tool use, agentic behaviors, and interactive human-AI collaboration, contributing to models that can see, hear, reason about, and interact with the world in real time.
Responsibilities
- Design, build, and optimize large-scale distributed systems for multimodal pre-training, post-training, inference, data processing, and tokenization.
- Develop high-throughput pipelines for data acquisition, preprocessing, filtering, generation, decoding, loading, crawling, visualization, and management.
- Advance multimodal capabilities including spatial-temporal compression, cross-modal alignment, world modeling, reasoning, emergent abilities, audio/image/video understanding & generation, real-time video processing, and noisy data handling.
- Drive data quality and studies: curation (human/synthetic), filtering techniques, analysis, and scalable pipelines.
- Create evaluation frameworks, internal benchmarks, reward models, and metrics that capture real-world usage, failure modes, interactive dynamics, and human-AI synergy.
- Innovate on algorithms, modeling approaches, hardware/software/algorithm co-design, and scaling paradigms for state-of-the-art performance.
- Build research tooling, user-friendly interfaces, prototypes/demos, full-stack applications, and enable rapid iteration based on feedback.
- Work across the stack (pre-training → SFT/RL/post-training) to enable reasoning, tool calling, agentic behaviors, orchestration, and seamless real-time interactions.
Requirements
- Hands-on experience with multimodal pre-training, post-training, or fine-tuning (vision, audio, video, or cross-modal).
- Expert-level proficiency in Python, with strong experience in at least one of: JAX / PyTorch / XLA.
- Proven track record building or optimizing large-scale distributed ML systems (training/inference optimization, GPU utilization, multi-GPU/TPU setups, hardware co-design).
- Deep experience designing and running data pipelines at scale: curation, filtering, generation, quality studies, especially for noisy/real-world multimodal data.
- Strong fundamentals in evaluation design, benchmarks, reward modeling, or RL techniques (particularly for interactive/agentic behaviors).
- Proactive self-starter who thrives in high-intensity environments and is passionate about pushing multimodal AI frontiers.
- Willingness to own end-to-end initiatives and do whatever it takes to deliver breakthrough user experiences.
- Familiarity with state-of-the-art in multimodal LLMs, scaling laws, tokenizers, compression techniques, reasoning, or agentic systems.
- Proficiency in Rust and/or C++ for performance-critical components.
- Hands-on work with large-scale orchestration tools such as Spark, Ray, or Kubernetes.
- Background building full-stack tooling: performant interfaces, real-time research demos/apps, or end-to-end product ownership.
- Passion for end-to-end user experience in interactive, real-time multimodal AI systems.
Benefits
- Equity
- Comprehensive medical, vision, and dental coverage
- Access to a 401(k) retirement plan
- Short & long-term disability insurance
- Life insurance
- Various other discounts and perks