Staff Software Engineer, Agentic Systems - Moveworks
Remote • Mountain View, CALIFORNIA, United States • Full-time
Posted 1mo ago
About the job
Moveworks is building the runtime infrastructure that powers its AI agents, enabling them to orchestrate, execute, and deliver responses to millions of enterprise users in real time. This role focuses on distributed systems engineering at the forefront of agentic AI. The AI agents are designed to plan, execute multi-step workflows, call tools, and interact with users, all while ensuring correctness, observability, and low latency. The successful candidate will build and own the systems that make these advanced agent capabilities possible.
Responsibilities
- Build and own the agent orchestration engine, a state machine managing long-running agent sessions across multiple LLM calls and tool invocations.
- Develop distributed session management systems, including lease-based ownership, heartbeat protocols, and crash recovery via checkpointing.
- Implement event-driven message pipelines using SQS FIFO queues, Kafka consumers, and real-time streaming via gRPC and Socket.IO.
- Utilize structured concurrency with Python asyncio TaskGroups for concurrent task management within sessions.
- Develop observability infrastructure, including OpenTelemetry instrumentation and distributed trace context propagation.
- Build caching and state layers using Redis and DynamoDB KV stores with per-organization/per-bot scoping.
Requirements
- Deep experience in distributed systems, including consistency models, idempotency, exactly-once delivery, and distributed locking/leasing.
- Extensive experience in concurrent/async programming, particularly Python asyncio and structured concurrency.
- Proven experience with event-driven architectures, message queues (SQS, Kafka), pub/sub, and backpressure.
- Strong background in database systems for infrastructure, such as DynamoDB (conditional writes, transactions) and Redis.
- Expertise in observability, including OpenTelemetry, distributed tracing, and Prometheus metrics.
- Experience with gRPC/protobuf, including streaming RPCs and service interface design.
- 7+ years of experience building production backend/infrastructure systems.
- Strong proficiency in Python or Go, ideally both.
- Experience designing and operating systems that handle real traffic at scale.
- Comfort with ambiguity and solving novel problems.