Staff Test & Release Engineer, Inference Cloud
$155k - $235k • Remote • Santa Clara • FullTime
Posted 2d ago
About the job
d-Matrix is at the forefront of software and hardware innovation, aiming to unleash the potential of generative AI. We foster a culture of respect, collaboration, humility, and direct communication, valuing diverse perspectives to create better solutions. We are seeking passionate, execution-driven individuals ready to tackle challenges and shape the future of AI. The Inference Cloud Engineering team is responsible for the deployment and orchestration technology behind d-Matrix’s inference services, including an internal cloud for model testing and a customer-facing cloud with self-service inference endpoints. This role will own the end-to-end testing and release process for this critical software, ensuring features are proven, releases are predictable, and rollbacks are safe, even with scarce and shared hardware resources.
Responsibilities
- Design and build end-to-end test infrastructure for Inference Cloud deployments on demand.
- Automate user stories and acceptance criteria into end-to-end tests, covering failure cases.
- Integrate real hardware (GPU and Corsair nodes) into CI, managing shared resources and ensuring test signal quality.
- Own the release strategy for orchestration, Kubernetes tooling, and benchmarking tools, including versioning, cadence, and rollback.
- Lead the transition to independently released components, defining breakdown, cadence, versioning, and compatibility testing.
- Define release gates for functional tests, upgrade/compatibility tests, model accuracy, and performance baselines.
- Partner with QE, product, platform, and infrastructure teams on a shared test strategy.
- Enable feature engineers to write and own tests for their own work.
Requirements
- 6+ years in test infrastructure, software engineering in test, or release engineering, with experience owning end-to-end testing or releases for distributed systems.
- Proven track record of building test or release automation that improved reliability or speed.
- Experience running test and release infrastructure across on-premises clusters and public cloud (preferably Azure).
- Strong Kubernetes experience, including multi-service workloads, Helm, operators, custom resources, and automated test cluster setup/teardown.
- Experience designing CI/CD pipelines (e.g., GitLab CI, GitHub Actions, Buildkite, Argo) with artifact versioning, environment promotion, and rollback.
- Proficiency in Python and pytest (or comparable framework).
- Fluency in a second language sufficient to read and test code (e.g., Rust).
- Experience testing on shared, limited hardware (GPU/accelerator fleets), managing scheduling, contention, and faults.
- Sound judgment on test scope and ability to explain tradeoffs.
- Record of building consensus and driving strategy adoption.
- Fluency with AI and agentic tools for test writing, failure triage, and log/code searching, with verification discipline.