Sr. AI & Data Engineer - Databricks
$70k - $206k • Dallas, 5205 N OConnor Las Colinas, Corp
Posted 1d ago
Job Location
Dallas, 5205 N OConnor Las Colinas, Corp
Tech Stack
Remote Work Policy
On-site
Categories
AI Engineer
About the job
We are seeking an experienced, hands-on Data & AI Engineer with deep expertise in Databricks and Azure data services to design, build, and optimize scalable data pipelines. You will design and implement autonomous agents using Databricks Mosaic AI Agent Framework and other frameworks like LangChain or AutoGen. This role involves close collaboration with analytics, data science, and cloud teams to ensure reliable, high-performance data delivery for clients. You will have the opportunity to work on diverse client initiatives, leveraging the latest data and analytics technologies in a highly collaborative environment. The work covers various aspects of Data Engineering including Data Modelling, Data Integration, Data Analytics, Pipeline Development, Agentic AI, RAG, Cloud and Infrastructure integration, Data Quality, Observability & Governance.
Responsibilities
- Architect, develop, and maintain scalable, low-latency data pipelines using Azure Databricks workflows and Spark.
- Design and optimize Lakehouse architecture using Delta Lake tables (Bronze, Silver, Gold layers).
- Partner with data scientists and business analysts to deliver clean, validated datasets for analytics and AI.
- Design and implement autonomous agents using Databricks Mosaic AI Agent Framework, LangChain, or AutoGen.
- Build and optimize vector search indexes using Databricks Vector Search.
- Process large-scale distributed datasets on the Databricks Intelligence Platform with Apache Spark.
- Design and implement data models to meet business requirements.
- Continuously evaluate and improve data architecture processes for efficiency.
- Design AI agents with retrieval, orchestration, tool invocation, evaluation harnesses, and lifecycle observability.
- Monitor pipeline health and performance, implementing automated alerting and observability frameworks.
Requirements
- Minimum of three years of professional experience as a Data & AI Engineer or related role.
- Minimum of three years of experience with Microsoft Azure cloud data services.
- Minimum of two years of experience with Databricks (PySpark, SQL, Delta Lake, workflows).
- Bachelor's degree or equivalent work experience (minimum 12 years, or 6 years with an Associate Degree).
Benefits
- Medical, dental, vision, life, and long-term disability coverage
- 401(k) plan
- Bonus opportunities
- Paid holidays
- Paid time off