Software Engineer, Machine Learning
Remote • Vancouver, BC • FullTime
Posted 16d ago
About the job
Ema builds AI Employees for HR, IT, and Finance, automating busy work across the employee experience from recruiting and onboarding to IT support and payroll. Our team, comprised of engineers from top tech companies and graduates from prestigious universities, is backed by leading investors and operates at the forefront of Agentic AI in production. We are developing systems where AI agents learn and improve from their real-world tasks, creating a continuous loop of production data, training, evaluation, and enhanced agent performance. The core challenge lies in determining the most effective interventions to improve agent behavior in live workflows.
Responsibilities
- Design context, tools, skills, and orchestration for multi-step agents, including cross-modal work (documents, slides, images, audio, video).
- Test and evaluate the impact of additional reasoning, search, or verification on latency and cost.
- Build self-improvement loops with explicit permissions, evaluation gates, and rollback mechanisms.
- Curate trajectories for Supervised Fine-Tuning (SFT), optimize preferences, and run Reinforcement Learning (RL) on real agent tasks.
- Investigate and compare methods like Direct Preference Optimization (DPO), GRPO, or DAPO, assessing process and outcome supervision, shaping rewards, and distilling frontier behavior into smaller models.
- Measure the transferability of performance gains beyond the training environment.
- Develop reproducible training and evaluation environments for enterprise workflows, including fixture tenants and simulated users.
- Identify and address potential shortcuts agents might exploit during training.
- Mine production agent steps for failures and build curated corpora with synthetic augmentation.
- Calibrate judges against human labels, construct behavior-level benchmarks, and quantify data quality, performance uplift, and reliability.
- Connect enterprise information with user- and tenant-level learnings through retrieval, memory, and context graphs.
- Separate retrieval failures from failures in using retrieved context.
- Test strategies for retaining, updating, and retrieving information to improve future decisions.
- Build and evaluate routing, ensembles, caching, and small-model specialization for quality per dollar.
- Measure downstream task success alongside latency and cost.
- Define problems and baselines for projects.
- Build necessary data or environments for testing.
- Run experiments and own serving and integration.
- Monitor systems post-deployment and perform failure analysis.
- Collaborate with researchers and engineers across different locations.
Requirements
- Substantial work in at least one of agent/tool-use systems, post-training, reward modeling or RL environments, retrieval and memory, or evaluation design.
- Ability to explain mechanisms, rejected alternatives, and identified failure modes.
- Demonstrated zero-to-one ownership of a system, model, or research project from an ambiguous problem to a working result.
- Understanding of contributions, tradeoffs made, and changes observed when work met real users or tasks.
- Proficiency in statistical judgment, including sizing experiments, choosing baselines, and accounting for variation.
- Ability to distinguish real improvements from biases, data leakage, or benchmark shortcuts.
- Production engineering judgment for debugging across model and system boundaries.
- Ability to isolate failures and translate findings into maintainable production code.
- Commitment to honest measurement and retiring approaches with clean negative results.
- A Master's or PhD in a relevant field, or equivalent work/research experience.
- Demonstrated depth through papers, open-source contributions, trained models, or well-documented experiments.