Data Scientist II, EngThrive Insights - CoreAI
$102k - $202k • United States, Washington, Redmond • Full-time
Posted 11d ago
Job Location
United States, Washington, Redmond
Tech Stack
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Engineer
About the job
The Engineering Thrive Insights team is seeking a Data Scientist II to enhance AI coding tools for thousands of Microsoft engineers. This role focuses on the AI Data Flywheel, a platform that gathers signals from engineers using AI coding tools and transforms them into actionable insights and experiments for continuous product improvement. You will define success metrics, design and analyze experiments, develop statistical and machine-learning models, and identify opportunities to advance AI-assisted software development. The position involves building rigorous measurement frameworks, evaluating product changes, and translating complex data into recommendations for engineering and product teams, while deepening expertise in experimentation, causal inference, developer productivity, and AI product evaluation at Microsoft scale.
Responsibilities
- Examine projects, considering influencing factors and evaluating project plans for resources, risks, and constraints.
- Define metrics and frameworks for evaluating AI/LLM systems and agentic product performance.
- Experiment with models and prompting approaches for product tasks.
- Diagnose issues and identify improvements within AI/LLM systems.
- Prototype and develop AI models using containerized environments for scalable tools and applications.
- Develop and deploy systems for retrieval-augmented generation workflows.
- Acquire and analyze data using querying, visualization, and reporting techniques.
- Explore data for key attributes and contribute to data quality reports.
- Perform data-science experiments using established methodologies, statistics, optimization, and probability theory.
- Select appropriate tools and techniques for data analysis and model development.
- Prepare data and collaborate with data engineers and architects.
- Apply modeling and machine learning solutions to identify optimal approaches.
- Understand modeling techniques, including validation, hypothesis testing, and data optimization.
- Assess risks such as data leakage and bias/variance tradeoff.
- Write scripts in T-SQL, U-SQL, KQL, Python, or R.
- Construct hypotheses, design controlled experiments, analyze results, and communicate findings.
- Understand operational considerations for model deployment, including performance, scalability, and monitoring.
- Develop operational models that run at scale in partnership with data engineering teams.
- Stay updated on embedding models and vector databases.
- Maintain knowledge of current industry trends, tools, and techniques.
- Contribute to thought leadership and intellectual property best practices.
Requirements
- Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field.
- OR Master's Degree in a related field AND 1+ year(s) data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) or consulting experience.
- OR Bachelor's Degree in a related field AND 2+ year(s) data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) or consulting experience.
