Senior Software Engineer, AIOps and Observability
$200k - $322k • Remote • US, CA, Santa Clara
Posted 2d ago
Job Location
US, CA, Santa Clara
Tech Stack
Remote Work Policy
Fully remote
Categories
AI Engineer
About the job
NVIDIA is seeking a Senior Software Engineer to design and develop AIOps and Observability platforms. These platforms are crucial for monitoring, diagnosing, and optimizing millions of internal assets and services across various environments including cloud, on-premise, data centers, supply chain, and edge. You will collaborate with a team of engineers, product managers, and partners to shape the observability strategy, roadmap, and best practices for NVIDIA. Additionally, you will mentor other engineers in observability, machine learning, and related tools and techniques.
Responsibilities
- Lead the design, development, and deployment of AIOps & Observability platforms, covering metrics, logs, traces, events, alerts, dashboards, and visualizations.
- Drive the technical vision and roadmap for AIOps and Observability initiatives, ensuring alignment with business goals and industry best practices.
- Collaborate with internal teams and customers to understand their observability needs and deliver effective solutions.
- Establish and implement observability standards, guidelines, and processes across NVIDIA.
- Research, evaluate, and adopt new observability technologies and frameworks to enhance user experience.
- Provide peer reviews to other engineers, offering feedback on performance, scalability, security, and correctness.
- Work with Data Scientists to implement machine learning models for anomaly detection, forecasting, and root cause analysis on logs, metrics, and events, ensuring data quality, security, and compliance.
- Develop and operate scalable, reliable, and distributed systems capable of handling high traffic and complex workloads.
- Develop AI agents and AI-native observability tools to accelerate the detection, understanding, and resolution of production issues.
- Build agentic workflows that reason across various data sources (logs, metrics, traces, events, alerts, topology, incident history) to support anomaly detection, forecasting, root cause analysis, automated debugging, and remediation recommendations.
- Establish scalable data pipelines and instrumentation for collecting, aggregating, and visualizing telemetry and operational metrics.
Requirements
- Bachelor’s degree in computer science and engineering, or related field, or equivalent experience.
- 12+ years of experience in product development and full stack engineering.
- 5+ years of experience in developing and operating observability platforms and solutions, preferably in a cloud-native environment.
- Strong knowledge and experience with observability tools such as Prometheus, Victoria Metrics, Vector, Loki, Grafana, Alert Manager, Clickhouse, OpenTelemetry.
- Hands-on knowledge in AIOps tools such as BigPanda, PagerDuty, Datadog.
- Experience with Kubernetes, Nomad, Docker, and microservices architectures.
- Experience with streaming services (NATS, Kafka) for ingesting billions of events.
- Proficiency in one or more programming languages such as Go, Python, Java, C#.
- Experience with developing Observability solutions for On-prem and Public Cloud environments.
- Experience with running large Observability platforms on BareMetal Infrastructure.
- Demonstrated experience and expertise in using machine learning and Generative AI for predictive monitoring, incident diagnosis, summarization, and correlation.
- Proficiency in AI/ML systems, generative AI, or agentic AI frameworks.
Benefits
- Equity
- Benefits