Software Engineer - Hosted Model Infrastructure
Palo Alto, CA • Full-time
Posted 4mo ago
Job Location
Palo Alto, CA
Tech Stack
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Infrastructure Engineer
About the job
Palantir is seeking a Software Engineer to join their Hosted Model Infrastructure team. This role focuses on enabling and deploying AI models in diverse and challenging production environments, including air-gapped networks, edge nodes, and enterprise systems with strict data sovereignty requirements. You will be responsible for the end-to-end ownership of services, working across the full stack from inference engines and GPU scheduling to deployment pipelines and observability. The primary goal is to deliver new models and capabilities quickly and continuously, ensuring frontier AI runs on customer-controlled hardware, often with constrained resources and limited access. This position is ideal for someone who wants to solve complex problems at the intersection of infrastructure and machine learning to support critical customers.
Responsibilities
- Build high-performance model serving infrastructure that integrates with security models, hardware constraints, and inference engines.
- Design intelligent request handling for multi-tenant model access, including authentication, rate limiting, concurrency control, and audit logging.
- Build and maintain packaging and deployment pipelines for fast, secure, and reliable model rollouts in on-premises and air-gapped environments.
- Develop observability for production AI systems to enable easy service monitoring and fast incident triage.
- Debug complex issues and performance problems across the stack, including inference engines, container runtimes, and GPU drivers, in restricted environments.
- Design and run testing and benchmarking infrastructure to validate model deployments across varying GPU hardware.
- Collaborate with product teams and customers to understand requirements, debug production issues, and deliver necessary models and capabilities.
- Integrate hosted model infrastructure with Palantir's deployment, configuration, and identity systems.
Requirements
- 4+ years of professional software engineering experience building and operating production systems.
- Engineering background in Computer Science, Mathematics, Software Engineering, Physics, or a similar field.
- Strong coding skills with demonstrated proficiency in programming languages such as Java, C++, Python, Rust, or similar.
- Familiarity with the Python ML ecosystem is valuable.
- Experience with containers, Kubernetes, and deploying backend services in production environments.
- Strong written and verbal communication skills.
- Ability to iterate quickly with teammates and maintain a high bar for quality.
- Ownership mindset and bias toward quality.
- High empathy for customer needs and drive to deliver reliable, easy-to-use models.
- Ability to work effectively across multiple languages and layers of the stack.
- Strong debugging skills and motivation to trace problems through the entire system.
- Curiosity about emerging AI capabilities and ability to quickly evaluate and integrate new technologies.
- Active US Security clearance, or eligibility and willingness to obtain one, is beneficial but not necessary.