Software Engineer, Agents (Internal Audit)
$170k - $300k • Remote • San Francisco, CA • FullTime
Posted 1mo ago
Job Location
San Francisco, CA
Tech Stack
Remote Work Policy
Fully remote
Employment Type
FullTime
Categories
AI Agent Engineer
About the job
Fieldguide is building a new state of trust for global commerce and capital markets by automating and streamlining the work of assurance and audit practitioners. This role is a product-focused, 0→1 opportunity to own agent quality and ship agents that perform real audit work. You will collaborate closely with practitioners, potentially owning major agent areas end-to-end or setting technical direction for agentic audit work. The position is open to all levels, with specific responsibilities and scope calibrated during the interview process based on experience.
Responsibilities
- Make agent judgment repeatable by running error analysis on testing data and implementing fixes.
- Manage tradeoffs between quality, latency, and cost across multi-phase agent runs.
- Build structured-output pipelines to convert model output into audit artifacts.
- Translate ambiguous problem statements into actionable plans and shipped features, clearly documenting decisions.
- Collaborate directly with subject matter experts and design-partner firms to iterate on agent changes rapidly.
- Expand agent coverage into new internal audit controls and areas.
- Own major agent areas end-to-end, from reasoning to final reviewer sign-off (Senior level).
- Define and lead the team's evaluation and error analysis practices, determining criteria for shipping or rolling back changes (Senior level).
- Shape roadmaps and define architectural tradeoffs with PMs and designers, clarifying agent vs. auditor roles (Senior level).
- Make complex model and orchestration decisions for multi-phase runs (Senior level).
- Mentor other engineers and improve 0→1 execution and applied evaluation rigor (Senior level).
- Drive agent initiatives beyond Internal Audit, influencing agent development across Fieldguide (Staff level).
- Establish and promote engineering standards for agent reliability, reproducibility, and defensibility (Staff level).
- Partner with leadership to define long-term technical strategy for agentic audit work (Staff level).
- Act as a trusted advisor to Engineering, Product, and Design leaders (Staff level).
- Represent Fieldguide externally through writing, speaking, and open-source contributions (Staff level).
Requirements
- Shipped LLM-backed product features to production with real users.
- Possess an Applied AI skillset including evals, error analysis, and model selection decisions.
- Be comfortable full-stack with sufficient backend depth for agent orchestration.
- Demonstrate autonomy working from ambiguous specifications.
- Exhibit a collaborative approach working across PM, design, and domain experts.
- Have strong opinions on model selection, prompting, and orchestration tradeoffs, supported by evidence.
- Be energized by 0→1 work, preferring to define problems over inheriting specs.
- Possess strong instincts for human-in-the-loop design.
- Be a genuine team player, collaborating effectively with PM, design, domain experts, and customer-facing teams.
- Ship code that is reviewable, tested, and instrumented.
- Be able to quickly internalize complex domains.
- Experience with Python, TypeScript, React, Postgres, Hasura, GraphQL (Nice-to-have).
- Experience with Temporal or comparable durable-execution/workflow orchestration (Nice-to-have).
- Hands-on evaluation experience with tools like Langfuse, Braintrust, LangSmith, Arize Phoenix, or comparable (Nice-to-have).
- Experience with structured output, schema contracts, and generating artifacts from model output (Nice-to-have).
- Startup experience, as a founder or early engineer (Nice-to-have).
- A 0→1 track record of starting initiatives without existing scaffolding (Nice-to-have).
- Experience working directly with customers and being present when they use built products (Nice-to-have).
- Background in internal audit, SOX, accounting, or another regulated domain (Nice-to-have).
- Experience with document processing, including PDF and Excel manipulation and annotation (Nice-to-have).