Software Engineer, Performance Instrumentation

About the job

Responsibilities

• Define and maintain performance data requirements from various sources like profiling libraries, annotation APIs, frameworks, runtimes, communication libraries, schedulers, configurations, and infrastructure telemetry.
• Identify and demonstrate visibility gaps, and collaborate with teams to build solutions.
• Inform the design of low-overhead instrumentation and annotation APIs from a consumer perspective.
• Ensure data quality at ingestion through validation, sanitization, checks, and alerting.
• Measure and manage the overhead and observer effects of instrumentation on real workloads.
• Collaborate with multiple partner teams, manage commitments, and ensure continuous progress.

Requirements

• Bachelor's degree in Computer Science, Computer Engineering, or related field, or equivalent experience.
• 8+ years of relevant software engineering experience, including hands-on experience with instrumentation or tracing in production machine learning or systems software.
• Proficiency in CUDA.
• Working knowledge of machine learning frameworks like PyTorch or JAX.
• Experience with annotation or tracing APIs such as NVTX.
• Demonstrated experience in maintaining data quality in a pipeline, including validation, checks, alerting, and provenance.
• Proven ability to measure instrumentation overhead and make trade-offs among overhead, fidelity, sampling, and aggregation.
• Proven history of implementing modifications in code managed by other groups by identifying requirements, reaching consensus on interfaces, and ensuring completion.
• Strong debugging skills across component boundaries and ability to validate changes through code review, tests, and independent measurements.
• Excellent communication skills, especially in conveying evidence, uncertainty, and next steps.
• Proficiency in Python and C++.

Benefits

• Equity
• Benefits