Senior Software Engineer - Model Evaluation & AI Systems
$180k - $240k • Remote • USA - Remote • FullTime
Posted 2mo ago
Job Location
USA - Remote
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
Applied AI Engineer
About the job
Deepgram is seeking a Senior Software Engineer - Model Evaluation & AI Systems to join a team focused on validating the quality of their speech, audio, and multilingual models before customer release. This role involves owning the evaluation and quality assurance processes for models across speech-to-text, text-to-speech, and LLM/multimodal systems. You will build the necessary pipelines, harnesses, canaries, and test frameworks to detect regressions and quality issues, working closely with Research to automate and enforce model expectations. The ideal candidate will be a strong engineer comfortable with building test infrastructure and analyzing model behavior, with a passion for quality, measurement, and automation.
Responsibilities
- Define and build evaluation methodologies for speech-to-text, text-to-speech, and emerging LLM, RAG, agent, and multimodal systems.
- Design, build, and maintain automated evaluation pipelines for batch and streaming data, focusing on correctness, reproducibility, and ease of adoption.
- Build scalable and reproducible evaluation infrastructure, including harnesses, orchestration, and result-aggregation pipelines.
- Translate research benchmarks and expected model metrics into automated, enforceable pass/fail gates.
- Build and operate canaries and continuous monitoring systems to detect production quality regressions.
- Partner with DevOps/Infra to set up ephemeral test environments and results-aggregation infrastructure.
- Collaborate with Research, model training, inference, and product teams to provide evaluation signals for release and optimization decisions.
- Integrate evaluation and quality gates into CI/CD pipelines for continuous quality verification.
- Contribute to raising engineering and QA standards through code reviews and technical discussions.
Requirements
- BS, MS, or PhD in Computer Science, AI, Applied Math, or a related field, or equivalent experience.
- 5+ years of professional software or QA engineering experience, with a proven track record in shipping test infrastructure or evaluation systems.
- Solid backend/scripting experience in languages like Python, Rust, or Go.
- Experience designing and building automated test pipelines, evaluation frameworks, or data-processing systems.
- Strong analytical skills with comfort in reasoning about metrics, thresholds, and statistical variation.
- Ability to tackle ambiguous technical challenges and communicate effectively across research, engineering, and product teams.