Senior Staff Software Engineer, Agentic Systems - Moveworks
Remote • Mountain View, CALIFORNIA, United States • Full-time
Posted 1mo ago
About the job
Moveworks is building the runtime infrastructure that powers its AI agents, which are responsible for orchestrating, executing, and delivering responses to millions of enterprise users in real time. This role focuses on distributed systems engineering at the forefront of agentic AI. The AI agents are designed to plan and execute multi-step workflows, interact with tools, and manage human input, all while ensuring correctness, observability, and low latency. The systems responsible for these capabilities are what you will build and own.
Responsibilities
- Develop and maintain the agent orchestration engine, a state machine managing long-running agent sessions and coordinating LLM calls and tool invocations.
- Implement distributed session management, including lease-based ownership, heartbeat protocols, and crash recovery mechanisms.
- Build and manage an event-driven message pipeline using SQS FIFO queues, Kafka consumers, and real-time streaming via gRPC and Socket.IO.
- Utilize structured concurrency with Python asyncio TaskGroups for concurrent task management within sessions.
- Develop observability infrastructure, including OpenTelemetry instrumentation and distributed trace context propagation.
- Design and implement caching and state layers using Redis and DynamoDB, incorporating features like per-organization scoping and hot-reload configuration.
Requirements
- Deep experience in distributed systems, including consistency models, idempotency, exactly-once delivery, and distributed locking/leasing.
- Proficiency in concurrent/async programming, such as Python asyncio or Go goroutines, with experience in structured concurrency and cancellation handling.
- Experience with event-driven architectures, message queues (SQS, Kafka), pub/sub patterns, backpressure, and delivery guarantees.
- Strong knowledge of database systems for infrastructure, including DynamoDB (conditional writes, transactions) and Redis (connection pooling, pub/sub).
- Experience with observability tools and concepts, such as OpenTelemetry, distributed tracing, span context propagation, and Prometheus metrics.
- Familiarity with gRPC/protobuf, including streaming RPCs, service interface design, and error handling patterns.
- 12+ years of experience building production backend/infrastructure systems.
- Strong proficiency in Python or Go, ideally both.
- Experience designing and operating systems that handle significant real-world traffic.
- Comfort with ambiguity and solving novel problems without established solutions.