Patterns for Continuous Extraction, Part 1

A fun problem we have at Finch is continuous extraction.

Today, extraction (producing structured data from a bounded unstructured input) is a ubiquitous and well-benchmarked application of LLMs.

Continuous extraction, in mild contrast, can be differentiated as convergence on structured data from a stream of unstructured inputs. As in, “here’s a new photo, what data should change?”

In 2024, as we first examined automating the retrieval of information, we quickly identified our complications:

• Our inputs
  • Unbounded: Artifacts are repeatedly added to a case over months, with high variance on timing.
  • Multimodal: documents; a photo or video; a phone call or sms; and via email, fax, paper mail.
  • Concurrent: 100+ page PDFs in the first five minutes, or a single phone call in the first week.
  • Sparse: artifacts contain fragments of entities.
• Our outputs
  • Relational: We model the facts of a case as relations; Ex: People, Corporations, Vehicles, Insurance Policies, and Claims are a small subset of the durable entities of concern.
  • Collaborative: attorneys and paralegals are operating daily on these relations and freely editing.
  • Sensitive: our burden of accuracy (and burden of proof) is high. Actions downstream of extractions happen within the first 24 hours of a case.

Bounded extraction produces a snapshot. Continuous extraction is inferring how a new fragment alters existing state.

Early approach: naive pipelines

To help set the stage for the problem we were trying to solve, I’ll lay out our first cut at this: fashioning a pipeline per entity. For example:

1. On a new Artifact in a case:

   “List relevant Vehicles explicitly mentioned in this document.” => VehicleMention[]

2. On a new Mention saved, serially:

   “Given all mentions in the case, compare to the current state and suggest operations.” => VehicleSuggestion[]

(Where a Suggestion is an operation of add | update | no-op | remove and a payload.)

Benefits:

• Mentions were nice ‘compressed’ snapshots of Artifacts that could be performed concurrently, and acted as a provenance mechanism
• Straightforward evals given separation of concerns
• Serial reconciliation resolved concurrency issues

Issues:

• Reconciliation workload and context window increased monotonically
• Model reluctance to remove duplicative records during reconciliation
• No protections for human-written data; models consistently re-created removed entities.
• Idiosyncrasies: models strained to reproduce UUIDs faithfully or would sometimes use invalid UUIDs to express indecision.
• Brittle: the main lever for partial failures was to rerun the reconciliation.

Additional pipelines were heavy boilerplate, in particular given accompanying relationships (i.e VehicleOwner, VehicleOccupant).

Our code was now crowding with loosely coupled concerns:

• evolving code serving all the needs and side effects of a human-facing CRUD application.
• decoupled asynchronous pipelines for each entity of concern, with independent schemas for structured outputs at each stage.

Agentic CRUD

As a team using coding agents on a daily basis we stay attuned to the evolving capabilities of LLMs and draw inspiration from what we see.

We repeatedly critiqued our extraction pipelines and tried to envision an agentic harness that would meet all our requirements:

1. Provenance: all mutations are attributed to either human or agent writers, and human writes should be protected
2. Citations: individual column mutations are cited to the subset of the artifact(s) that motivated the change.
3. Reversibility
4. Concurrency safety
5. Fault tolerance
6. Not a PITA to maintain

Where we landed was a handful of contracts and features that more tightly coupled our relational data models, our human facing APIs, and our agent facing APIs.

What would it take to hand the API to an agent?

We put the invariants in the schema and the smarts in one interceptor. Every enrolled entity type gets the same five tables:

• Anchor: identity and lifecycle. Matter, entity type, approval status, a generation counter, a short human-legible ref. Deletes snapshot into a graveyard here, and reads return the dead, so a later artifact can’t resurrect what a person removed.
• Revision: one write, one envelope. The op (create | patch | delete), the actor (a user, or an agent and its run id), and a prose support_statement.
• FieldRevision: the value itself, per field, with a status (pending | active | rejected | superseded). Values are superseded, never overwritten. One invariant: at most one active and one pending revision per field. A constraint, not a prompt.
• Citation: the evidence. Artifact, page, bounding box, snippet. Attached per field, not per record; “this vehicle exists” and “this vehicle’s plate” have different evidence.

The record you read is a projection of this log.

The CRUD over it is dumb on purpose: create_vehicle / read_vehicles / update_vehicle / delete_vehicle, generated per entity type. Agent writes are app writes, against the same rows and rules the application serves to humans. The smarts live in one interceptor on the write path. Every mutation needs evidence. Generation tokens catch races (a stale writer re-reads and reconciles instead of clobbering). Refusals come back as deterministic bounce-backs the model can act on. And the entire write policy is three lines:

def status(actor, anchor):
    if actor.is_human:   return ACTIVE
    if anchor.approved:  return PENDING
    return ACTIVE

Humans write directly. An agent writing to a record a human approved produces a pending diff; accept or reject is one click. Everything else lands active, because review is a budget and we spend it on records humans have vouched for.

Against the requirements: provenance is the Revision actor; citations are field-level rows; reversibility is supersession plus the graveyard; concurrency is the generation token; fault tolerance comes from writes being idempotent proposals; maintenance is one EnrollmentSpec that generates the repository, service, API, and toolset. A new entity type is an enrollment, not a project.

What’s next

This post is just the tables and the contract that make agent writes boring. Nothing here actually drives a write. Part 2 covers the agent side: the planner, scopes and capability manifests, the eval harness, and one matter walked through three artifacts, row by row.