Senior Software Engineer, Agentic Systems - Moveworks
Remote • Mountain View, CALIFORNIA, United States • Full-time
Posted 1mo ago
About the job
Moveworks is building the runtime infrastructure that powers its AI agents, which deliver real-time responses to millions of enterprise users. This role focuses on distributed systems engineering at the core of agentic AI. The AI agents can plan, execute multi-step workflows, call tools, and interact with users while maintaining correctness, observability, and low latency. The systems responsible for these capabilities are what you will build and own.
Responsibilities
- Build and own the agent orchestration engine, a state machine managing long-running agent sessions, coordinating planning, execution, and user interaction across multiple LLM calls and tool invocations.
- Develop distributed session management using lease-based ownership, DynamoDB conditional writes, heartbeat protocols, and crash recovery via checkpointing.
- Implement an event-driven message pipeline using SQS FIFO queues, Kafka consumers, and real-time streaming via gRPC and Socket.IO.
- Utilize structured concurrency with Python asyncio TaskGroups for concurrent tasks per session, including message polling, lease heartbeats, output publishing, and orchestrator execution with fail-fast semantics and graceful cancellation.
- Develop observability infrastructure with OpenTelemetry instrumentation, distributed trace context propagation, and custom span lifecycle management.
- Build caching and state layers using Redis and DynamoDB KV stores with per-org/per-bot scoping, batch read optimization, and hot-reload configuration.
Requirements
- 5+ years building production backend/infrastructure systems.
- Strong proficiency in Python or Go (ideally both).
- Experience designing and operating systems that handle real traffic at scale.
- Deep experience in at least 3 of the following areas: distributed systems (consistency models, idempotency, exactly-once delivery, distributed locking/leasing), concurrent/async programming (Python asyncio, Go goroutines, structured concurrency, cancellation handling), event-driven architectures (message queues, pub/sub, backpressure, delivery guarantees), database systems for infrastructure (DynamoDB, Redis), observability (OpenTelemetry, distributed tracing, span context propagation, Prometheus metrics), gRPC/protobuf (streaming RPCs, service interface design, error handling patterns).
- Comfort with ambiguity and novel problems without textbook solutions.