Patterns for Continuous Extraction, Part 1

A fun problem we have at Finch is continuous extraction.

Today, extraction (producing structured data from a bounded unstructured input) is a ubiquitous and well-benchmarked application of LLMs.

Continuous extraction, in mild contrast, can be differentiated as convergence on structured data from a stream of unstructured inputs. As in, “here’s a new photo, what data should change?”

In 2024, as we first examined automating the retrieval of information, we quickly identified our complications:

• Our inputs
  • Unbounded: Artifacts are repeatedly added to a case over months, with high variance on timing.
  • Multimodal: documents; a photo or video; a phone call or sms; and via email, fax, paper mail.
  • Concurrent: 100+ page PDFs in the first five minutes, or a single phone call in the first week.
  • Sparse: artifacts contain fragments of entities.
• Our outputs
  • Relational: We model the facts of a case as relations; Ex: People, Corporations, Vehicles, Insurance Policies, and Claims are a small subset of the durable entities of concern.
  • Collaborative: attorneys and paralegals are operating daily on these relations and freely editing.
  • Sensitive: our burden of accuracy (and burden of proof) is high. Actions downstream of extractions happen within the first 24 hours of a case.

Bounded extraction produces a snapshot. Continuous extraction is inferring how a new fragment alters existing state.

Early approach: naive pipelines

To help set the stage for the problem we were trying to solve, I’ll lay out our first cut at this: fashioning a pipeline per entity. For example:

1. On a new Artifact in a case:

   “List relevant Vehicles explicitly mentioned in this document.” => VehicleMention[]

2. On a new Mention saved, serially:

   “Given all mentions in the case, compare to the current state and suggest operations.” => VehicleSuggestion[]

(Where a Suggestion is an operation of add | update | no-op | remove and a payload.)

Benefits:

• Mentions were nice ‘compressed’ snapshots of Artifacts that could be performed concurrently, and acted as a provenance mechanism
• Straightforward evals given separation of concerns
• Serial reconciliation resolved concurrency issues

Issues:

• Reconciliation workload and context window increased monotonically
• Model reluctance to remove duplicative records during reconciliation
• No protections for human-written data; models consistently re-created removed entities.
• Idiosyncrasies: models strained to reproduce UUIDs faithfully or would sometimes use invalid UUIDs to express indecision.
• Brittle: the main lever for partial failures was to rerun the reconciliation.

Additional pipelines were heavy boilerplate, in particular given accompanying relationships (i.e VehicleOwner, VehicleOccupant).

Our code was now crowding with loosely coupled concerns:

• evolving code serving all the needs and side effects of a human-facing CRUD application.
• decoupled asynchronous pipelines for each entity of concern, with independent schemas for structured outputs at each stage.

Ideating around Agentic CRUD

As a team using coding agents on a daily basis we stay attuned to the evolving capabilities of LLMs and draw inspiration from what we see.

We repeatedly critiqued our extraction pipelines and tried to envision an agentic harness that would meet all our requirements:

1. Provenance: all mutations are attributed to either human or agent writers, and human writes should be protected
2. Citations: individual column mutations are cited to the subset of the artifact(s) that motivated the change.
3. Reversibility
4. Concurrency safety
5. Fault tolerance
6. Not a PITA to maintain
7. Django-friendly

Where we landed was a handful of contracts and features that more tightly coupled our relational data models, our human facing APIs, and our agent facing APIs.

What would it take to hand the API to an agent?

We started with a shared set of relations that would enforce structure and unlock more downstream opportunities:

(This may be vaguely recognizable as a frankenschema of event sourcing and common audit log designs.)

• Anchor: An opt-in facade that grants a semantic identifier, captures human review status status, stores denormalized snapshots on deletion. identity and lifecycle.
• Revision: a mutation envelope. The op (create | patch | delete), the actor (human/agent) and supporting context.
• FieldRevision: The contents of each Revision, exploded by individual column changes.
• Citation: the evidence. Artifact, page, bounding box, snippet. Attached per field, not per record; “this vehicle exists” and “this vehicle’s plate” have different evidence.

Together with thoughtful implementation around schema projection and Pydantic-AI, these relations were enough to unlock:

• shared write surfaces for humans and agents
• shared human facing API envelopes
• shared UI components for inspecting provenance and citations
• entity-specific read and write policies
• composable CRUD ToolSets and affordances to allow any agent to extract
• column level proposed changes
• concurrency safety

Over the next couple of posts I’ll walk through:

• Shared schema projections, dual read surfaces, and Pydantic AI Capabilities encapsulating read / write affordances
• Evaluations and observations adopting this framework internally and the effects it had on our case intelligence