Folder runs — bulk ingest of a folder

A folder run ingests every eligible descendant of one folder into a knowledge-base space, as a single named unit. It exists so an operator can say "add this folder" once and have the result be reviewable, revertible and auditable as a group, rather than as N unrelated ingests.

A run is asynchronous by construction: it is created, enqueued, and executed by a background worker. That is not a performance choice — a run at the 200-source cap does far more work than one request can hold, and its progress has to survive the request that started it.

Starting a run

POST /api/pages/:id/folder-runs starts a run over the folder at :id. It requires wiki:page:write — the same permission as the folder preview, because it acts on the plan that preview produced — and, because it approves every source of the run, a HUMAN caller under the same R7 gate as a single ingest (a service or agent principal gets 403; see Ingest loop). Its body carries two fields:

{ "idempotencyKey": "<a key you choose>", "targetSpaceId": "<the knowledge-base space>" }

The manifest is not something you send. The request re-runs the folder preview and takes its complete eligible set, so what the run covers is what the preview showed, and a caller cannot compose a run out of pages it chose itself.

Everything the request creates commits together: the ledger row, the queued job, and the binding between them. There is no window in which a run exists without the job that will execute it, or a job exists naming a run that does not.

The idempotency key

The key is yours to choose and yours to reuse, and reusing it is how a retry avoids starting a second run over the same folder. The endpoint's promise is precise: at most one run per key.

Outcome	Answer
The request created and enqueued the run	201 with the run and job ids
The key already named this same request	200 with the original run and job ids
The key names a different request	409 — never aliased onto the first run

"A different request" means a different folder, a different payload, or a different actor: the key alone does not identify a request, and one person's retry can never adopt another's run.

The response carries the run id and the job id, and nothing else. The run's status, its timestamps and its progress live on the run listing (GET /api/pages/:id/folder-runs), which is the surface to read them from — this endpoint is not a status poll.

A retry is judged against the world as it is now. If a source has been trashed, reclassified or moved out of the folder since the run was created, the retry is refused rather than handed the original ids. That is deliberate: the check that refuses it is the same one that makes a page you may not see indistinguishable from one that does not exist, and reversing the order to answer retries first would have to give that up. Nothing is lost when it happens — the run itself is not stranded, and the folder's run listing is where you recover its ids.

What is refused, and when

Refusals happen before anything is created, so a rejected request costs no run id and no queued job:

• a folder you cannot see — absent, deleted, in another tenant, or above your clearance — answers 404, identically in all four cases, and so do "that is not a descendant" and "that page is trashed";
• a folder with more than 200 descendants is refused rather than truncated: the preview an operator approves must describe the whole folder. The cap is on the descendants walked, not on the eligible ones — a folder of 300 pages that would ingest none of them is refused just the same, because the walk stops before eligibility is resolved at all. Those descendants are the ones your clearance can reach: the walk reads through the same page-visibility rules everything else does, so a branch hanging off an ancestor you cannot see is neither walked nor counted. Split the folder and start a run per sub-folder;
• a destination that is not a knowledge-base space is refused, because that is what the ingest itself requires — a run into an ordinary space could only fail on its first source.

The destination is judged first. A request that names both an ineligible destination and a folder it cannot see answers the destination's 409, not the folder's 404. The order is deliberate rather than incidental: reading the folder means walking its descendants and resolving each one's nomination policy through ancestors you may not be cleared for, and that walk is a privileged read the platform must audit. Judging the destination first means a request that was already refusable from its own body never causes it. Nothing is disclosed by the precedence — a folder you cannot see and one that does not exist stay indistinguishable from each other whichever destination you name.

A folder that previews nothing is not a refusal: it starts a legitimate run with an empty manifest, which completes having committed nothing.

The ledger

wiki.folder_runs records the run itself, and it is a table rather than a column on wiki.ingest_batches because a run exists before its first batch: it is created when the request is accepted, may sit at pending, and may fail without ever committing anything.

Status	Meaning
pending	Created and enqueued; no worker has taken it.
running	A worker has entered it. Resumable — see Crash recovery.
completed	Every source in the manifest was settled. The only status a group revert accepts.
failed	A source faulted. Retried as a new run, never resumed.
reverted	A completed run was undone as a group.

wiki.folder_run_sources holds the run's manifest — the sources the preview selected, fixed at creation and immutable afterwards. The run's identity, its target space and its declared size are immutable too, so a run cannot be quietly redefined once it exists.

Execution

The worker (job type wiki-folder-run) processes the manifest one source per transaction. Never one transaction for the run: a source that is refused or faults must not roll back the sources the run has already committed.

Each source is judged as it is reached, not as it was previewed — a governance change landing between the two is respected rather than bypassed. A source may therefore be skipped rather than ingested:

Disposition	Why
already_committed	A previous attempt of this run already ingested it.
source_unavailable	Deleted, or reclassified above the run's UNCLASSIFIED execution pin.
outside_run_root	Reparented out of the initiating folder since the run was enqueued.
excluded	Excluded by its own marker or by an ancestor's ingest policy.
already_covered	The knowledge base already covers it.

A skip is a normal outcome and the run continues. A fault — malformed input, an authorization failure, a database error — is different: the run stops there and settles failed, because a failed run can be neither resumed nor group-reverted, so continuing past a genuine fault would only enlarge the set of batches stranded under it.

Attribution

Runs are executed by a machine, and nothing a machine did is attributed to a person.

• Status transitions, and the audit entries that record them, name the system actor. The one entry a person is named on is ingest.folder_run.created, because a person is what made that request.
• Pages and revisions the run creates are owned by and authored by the tenant's seeded wiki agent user — a machine identity, not the person who approved the run. A tenant whose agent user is missing or suspended has its runs refused rather than executed under someone's name: page revisions are append-only, so an attribution recorded once cannot be corrected. The refusal hands the job back to the queue rather than consuming it, so seeding or reactivating the agent user lets the queued run proceed on its next attempt.
• The approving human appears only as approval provenance, never as authorship: on each source's approvedBy, on the queued job that carries it there, and on the ingest.folder_run.created entry for the request that started the run. Everything the run itself then does — the status transitions, the pages, the revisions — is attributed to the machine.

The ledger also binds a run to exactly one queue job, and a worker whose job differs is refused before it writes anything.

Audit

Every entry is written in the same transaction as the change it describes, so "the run moved" and "we recorded that it moved" are atomic:

• ingest.folder_run.created — the request that created the run and enqueued its job, attributed to the person who made it. A replayed idempotency key creates nothing and so records nothing.
• ingest.folder_run.started — pending → running, filed by the worker.
• ingest.folder_run.completed or ingest.folder_run.failed — the terminal transition, and a run receives exactly one of the two, never both.

Each carries the run's size and how its sources came out; none carries the run's classification ceiling, its manifest, or any page or batch id. A run's exact composition is withheld from readers who are not cleared for it, and the audit channel does not reopen that — the ceiling reaches the entry only as the entry's own classification. Every ingested source additionally files an ingest.batch.committed entry.

Crash recovery

Job delivery is at-least-once: a worker that dies mid-run is recovered by the queue's stale-claim lease and handed the same job again. The recovered run re-reads its manifest and re-walks it from the start; every source a previous attempt committed is recognised and skipped, so the run finishes without re-ingesting anything.

The result is that a crash costs time, never duplicated content and never a duplicated audit trail: at most one ingest batch and one commit entry per manifest source — exactly one for every source actually ingested, and none for one the run skipped — however many attempts it took. A replay of a run that already reached a terminal status writes nothing at all — no status, no audit, no batch — because an audit entry for a transition that did not happen would be a false record.

One attempt drives a run

The lease recovers a worker that died; it cannot tell a dead worker from a slow one. A handler still executing past the lease is not told that a new attempt now holds its job, so two attempts can be executing one run at the same time. The run ledger records which queue attempt currently owns the run, and every write is fenced on it:

• entering a pending run stamps the attempt; a newer attempt resuming a running run takes it over, and an older attempt arriving afterwards stands down without writing;
• every source's transaction verifies, under the run's row lock and immediately before it commits, that the run is still running and still owned by this attempt — otherwise the whole ingest is rolled back, so no batch, page or commit entry from a superseded attempt ever lands;
• the terminal transition is guarded on the same stamp, so a superseded attempt cannot file a terminal entry over the owner's run.

The terminal entry's committedCount is read from the ledger under that same lock rather than added up from the attempt's own walk, so it states what the run holds — the property a completed-only group revert relies on. Each transition entry also records the attempt that performed it, so a completed entry filed by attempt 2 after a started entry filed by attempt 1 reads as exactly what happened.

A rolled-back source keeps its read audit entries and loses its write entries. A source's transaction may record that its governance walk crossed a classification boundary the worker cannot see; if the transaction is then rolled back — by the fence, or by a fault — those read entries are filed again in a separate transaction, because the read happened whether or not the ingest survived, while ingest.batch.committed and the other write entries stay rolled back with the batch they described.

Reverting

A completed run can be reverted as a group, undoing every batch it committed in one audited transaction. See source retraction for the single-batch equivalent. A failed run is not revertible: it is retried as a new run instead.