ML Infrastructure Engineer
$160k - $250k • San Francisco • FullTime
Posted 15d ago
Job Location
San Francisco
Tech Stack
Remote Work Policy
On-site
Employment Type
FullTime
Categories
AI Infrastructure Engineer
About the job
At Mach9, ML infrastructure engineers build and maintain the systems that power production AI models for civil engineering and surveying. Our ML pipeline spans 10,000+ miles of labeled survey data, image segmentation networks, and 3D prediction models serving real-time inference to surveyors and engineers in the field. This role is ideal for mid-career ML infrastructure engineers with experience building for both training and inference. You'll build training pipelines that handle deep transformer models on hundreds of terabytes of 3D point cloud and image data. You'll also architect our inference infrastructure, delivering both heavy offline detection algorithms and real-time responsive inference that integrates directly with our CAD software.
Responsibilities
- Design and build a centralized system for versioning training data, generated datasets, and model artifacts, with full lineage tracking.
- Develop and maintain reliable, reproducible ML training and data generation pipelines.
- Refactor and harden existing training and data generation scripts into composable, testable, and maintainable components.
- Create CI/CD workflows for validating data pipelines and model training runs, including automated correctness checks and regression detection.
- Build tooling that enables ML engineers to launch, monitor, and debug training jobs with minimal friction.
- Optimize and scale real-time model inference services to meet latency and throughput requirements in production.
- Own the deployment path from trained model artifact to production endpoint, ensuring reliable rollouts, rollback, and monitoring.
Requirements
- 3+ years of work experience in relevant fields.
- Bachelor's or Master's degree in Computer Science, Engineering, or equivalent experience.
- Strong communication skills and the ability to work closely with ML researchers and engineers.
- Experience designing and building data versioning, artifact management, or dataset lineage systems (e.g., DVC, LakeFS, Weights & Biases, or custom solutions).
- Hands-on experience with ML pipeline orchestration tools (e.g., Airflow, Prefect, Metaflow, or similar).
- Experience with model serving and inference optimization — profiling latency, reducing memory footprint, or scaling serving infrastructure.
- Ability to read and refactor ML training code to ensure reliability.
- Proficient with Python and PyTorch.