Senior Staff Applied AI Engineer - Context Retrieval
$229k - $343k • Remote • Mountain View, California; San Francisco, California
Posted 26d ago
Job Location
Mountain View, California; San Francisco, California
Tech Stack
Remote Work Policy
Fully remote
Categories
Applied AI Engineer
About the job
Databricks is seeking a Senior Staff Applied AI Engineer to lead the development of context retrieval systems for Databricks agents across various SaaS providers. This is a foundational role focused on building both the underlying retrieval stack and the intelligent search subagents that leverage it. The goal is to significantly enhance the ability of Databricks agents to access and understand relevant information from diverse data sources, thereby improving their performance in tasks ranging from answering complex queries to debugging and data analysis. This role requires deep expertise in information retrieval and a proven track record of shipping production-ready retrieval systems for RAG and agentic workloads.
Responsibilities
- Build the end-to-end retrieval stack, including query understanding, content indexing, hybrid retrieval, ranking, and evaluation.
- Develop retrieval mechanisms for heterogeneous data, encompassing both structured assets (tables, SQL, dashboards) and unstructured content (docs, wikis, tickets, media).
- Create connectors and retrieval adapters for enterprise knowledge systems within SaaS platforms.
- Optimize retrieval systems to serve both LLMs (for grounded, hallucination-resistant context) and human users (for intuitive discovery).
- Develop advanced query understanding capabilities for agentic workflows, including rewriting, decomposition, and intent classification.
- Build scalable content understanding pipelines to extract structure, entities, embeddings, and metadata from various asset types.
- Design and implement search subagents that reason about retrieval needs, plan multi-hop searches, and verify the sufficiency of retrieved context.
- Establish a comprehensive evaluation framework for both retrieval performance and subagent decision quality, incorporating offline metrics, LLM-as-judge, and online experimentation.
- Set technical direction, mentor engineers, and collaborate with cross-functional leaders to advance the AI capabilities of Databricks agents.
Requirements
- 10+ years of software engineering experience, with substantial experience in building production retrieval, search, or RAG systems at scale.
- Deep expertise in Information Retrieval (IR), including lexical retrieval (BM25, Lucene/Elasticsearch/OpenSearch), dense retrieval (embeddings, ANN indexes like FAISS, ScaNN, HNSW), hybrid retrieval, and learning-to-rank.
- Hands-on experience with modern LLM-era retrieval techniques such as RAG architectures, query rewriting, re-ranking with cross-encoders, long-context strategies, and grounding methods.
- Experience designing agentic systems that utilize retrieval, including search planners, iterative retrieval, self-reflection, and tool-using agents.
- Strong understanding of relevance evaluation metrics (nDCG, MRR, Precision@K, Recall@K) and experience with offline/online experimentation, LLM-as-judge frameworks, and human labeling pipelines.
- Experience working with both structured and unstructured data.