Sr. Software Development Engineer in Test, AI/ML
CA$115k - CA$172k • Canada, Waterloo, ON • Full-time
Posted 8d ago
Job Location
Canada, Waterloo, ON
Tech Stack
Remote Work Policy
On-site
Employment Type
Full-time
Categories
LLM Engineer
About the job
Sony Interactive Entertainment (SIE) is seeking a Sr. Software Development Engineer in Test specializing in ML/AI quality. This role focuses on developing automation for model evaluation, leveraging LLMs for test generation, and validating AI-powered workflows. You will guide quality strategy, develop automation frameworks, and manage the implementation of complex, cross-functional machine learning products and services. The position involves leading quality initiatives from inception to completion, ensuring team synchronization, managing dependencies, and confident release delivery. You will collaborate closely with ML, engineering, product, and infrastructure teams to shape how quality is built, monitored, and expanded, while leading embedded Quality Engineering (QE) efforts within projects.
Responsibilities
- Define and execute quality strategies, test plans, and automation coverage for ML-powered services and platform components.
- Utilize LLMs and AI-assisted techniques to generate, expand, and maintain test cases for ML-powered workflows.
- Design scenario-based test suites for AI features, covering adversarial prompts, edge cases, ambiguous inputs, and underrepresented user scenarios.
- Lead QE efforts for multi-functional projects, including risk assessment, dependency management, and release readiness.
- Design, develop, and maintain scalable automation frameworks for backend services, APIs, and ML inference systems using Python and/or Java.
- Build automated validation for ML and LLM outputs, including ranking behavior, score distributions, prompt/response quality, hallucination indicators, and probabilistic model evaluation.
- Debug test failures, service anomalies, model inconsistencies, and AI behavior regressions to identify root causes and drive resolution.
- Perform functional, integration, regression, API, end-to-end, performance, and reliability testing for distributed systems.
- Improve automation reliability, reduce flakiness, and optimize execution efficiency.
- Partner with engineering and ML teams to integrate automated testing into CI/CD pipelines and release workflows.
- Collaborate across teams to establish scalable quality standards, tooling, and guidelines.
Requirements
- Bachelor’s degree in Computer Science or equivalent.
- 5+ years of experience as an SDET or QE engineer focused on backend and distributed systems.
- Experience using LLMs to generate, transform, and prioritize test cases for AI-powered experiences.
- Experience with AI evaluation tooling, prompt evaluation frameworks, model monitoring, or human-in-the-loop review workflows.
- Strong experience testing RESTful APIs, microservices, and distributed architectures.
- Proficiency in Python, Java, JS, or similar languages for automation development.
- Hands-on experience with automation frameworks such as pytest, JUnit, Selenium, Playwright, Cypress, or Appium.
- Experience with CI/CD systems and test pipelines (Jenkins, GitHub Actions, etc.).
- Experience with cloud and container technologies (AWS, GCP, Kubernetes, Docker).
- Familiarity with databases, monitoring, and observability tools.
- Strong understanding of SDLC, Agile methodologies, and release processes.
- Excellent problem-solving, debugging, and communication skills.
- Practical understanding of how to test non-deterministic AI systems and separate model variance from quality regressions.
- Proven risk management and dependency coordination skills.
- Ability to influence engineering teams and promote quality guidelines.