Folder runs — bulk ingest of a folder

A folder run ingests every eligible descendant of one folder into a knowledge-base space, as a single named unit. It exists so an operator can say "add this folder" once and have the result be reviewable, revertible and auditable as a group, rather than as N unrelated ingests.

A run is asynchronous by construction: it is created, enqueued, and executed by a background worker. That is not a performance choice — a run at the 50-source cap does far more work than one request can hold, and its progress has to survive the request that started it.

Starting a run

POST /api/pages/:id/folder-runs starts a run over the folder at :id. It requires wiki:page:write — the same permission as the folder preview, because it acts on the plan that preview produced — and, because it approves every source of the run, a HUMAN caller under the same R7 gate as a single ingest (a service or agent principal gets 403; see Ingest loop). Its body carries two fields:

{ "idempotencyKey": "<a key you choose>", "targetSpaceId": "<the knowledge-base space>" }

The manifest is not something you send. The request re-runs the folder preview and takes its complete eligible set, so what the run covers is what the preview showed, and a caller cannot compose a run out of pages it chose itself.

Everything the request creates commits together: the ledger row, the queued job, and the binding between them. There is no window in which a run exists without the job that will execute it, or a job exists naming a run that does not.

The idempotency key

The key is yours to choose and yours to reuse, and reusing it is how a retry avoids starting a second run over the same folder. The endpoint's promise is precise: at most one run per key.

Outcome	Answer
The request created and enqueued the run	201 with the run and job ids
The key already named this same request	200 with the original run and job ids
The key names a different request	409 — never aliased onto the first run

"A different request" means a different folder, a different payload, or a different actor: the key alone does not identify a request, and one person's retry can never adopt another's run.

The response carries the run id and the job id, and nothing else. The run's status, its timestamps and its progress live on the run listing (GET /api/pages/:id/folder-runs), which is the surface to read them from — this endpoint is not a status poll.

A retry is judged against the world as it is now. If a source has been trashed, reclassified or moved out of the folder since the run was created, the retry is refused rather than handed the original ids. That is deliberate: the check that refuses it is the same one that makes a page you may not see indistinguishable from one that does not exist, and reversing the order to answer retries first would have to give that up. Nothing is lost when it happens — the run itself is not stranded, and the folder's run listing is where you recover its ids.

What is refused, and when

Refusals happen before anything is created, so a rejected request costs no run id and no queued job:

• a folder you cannot see — absent, deleted, in another tenant, or above your clearance — answers 404, identically in all four cases, and so do "that is not a descendant" and "that page is trashed";
• a folder with more than 50 descendants is refused rather than truncated: the preview an operator approves must describe the whole folder. The cap is on the descendants walked, not on the eligible ones — a folder of 80 pages that would ingest none of them is refused just the same, because the walk stops before eligibility is resolved at all. Those descendants are the ones your clearance can reach: the walk reads through the same page-visibility rules everything else does, so a branch hanging off an ancestor you cannot see is neither walked nor counted. Split the folder and start a run per sub-folder;
• a destination that is not a knowledge-base space is refused, because that is what the ingest itself requires — a run into an ordinary space could only fail on its first source.

The destination is judged first. A request that names both an ineligible destination and a folder it cannot see answers the destination's 409, not the folder's 404. The order is deliberate rather than incidental: reading the folder means walking its descendants and resolving each one's nomination policy through ancestors you may not be cleared for, and that walk is a privileged read the platform must audit. Judging the destination first means a request that was already refusable from its own body never causes it. Nothing is disclosed by the precedence — a folder you cannot see and one that does not exist stay indistinguishable from each other whichever destination you name.

A folder that previews nothing is not a refusal: it starts a legitimate run with an empty manifest, which completes having committed nothing.

The ledger

wiki.folder_runs records the run itself, and it is a table rather than a column on wiki.ingest_batches because a run exists before its first batch: it is created when the request is accepted, may sit at pending, and may fail without ever committing anything.

Status	Meaning
pending	Created and enqueued; no worker has taken it.
running	A worker has entered it. Resumable — see Crash recovery.
completed	Every source in the manifest was settled.
failed	A source faulted. Retried as a new run, never resumed.
reverted	The run was undone as a group, and stopped if it was still in progress.

wiki.folder_run_sources holds the run's manifest — the sources the preview selected, fixed at creation and immutable afterwards. The run's identity, its target space and its declared size are immutable too, so a run cannot be quietly redefined once it exists.

Execution

The worker (job type wiki-folder-run) processes the manifest one source per transaction. Never one transaction for the run: a source that is refused or faults must not roll back the sources the run has already committed.

Each source is judged as it is reached, not as it was previewed — a governance change landing between the two is respected rather than bypassed. A source may therefore be skipped rather than ingested:

Disposition	Why
already_committed	A previous attempt of this run already ingested it.
source_unavailable	Deleted, or reclassified above the run's UNCLASSIFIED execution pin.
outside_run_root	Reparented out of the initiating folder since the run was enqueued.
excluded	Excluded by its own marker or by an ancestor's ingest policy.
already_covered	The knowledge base already covers it.

A skip is a normal outcome and the run continues. A fault — malformed input, an authorization failure, a database error — is different: the run stops there and settles failed, because a failed run can be neither resumed nor group-reverted, so continuing past a genuine fault would only enlarge the set of batches stranded under it.

Attribution

Runs are executed by a machine, and nothing a machine did is attributed to a person.

• Status transitions, and the audit entries that record them, name the system actor. The one entry a person is named on is ingest.folder_run.created, because a person is what made that request.
• Pages and revisions the run creates are owned by and authored by the tenant's seeded wiki agent user — a machine identity, not the person who approved the run. A tenant whose agent user is missing or suspended has its runs refused rather than executed under someone's name: page revisions are append-only, so an attribution recorded once cannot be corrected. The refusal hands the job back to the queue rather than consuming it, so seeding or reactivating the agent user lets the queued run proceed on its next attempt.
• The approving human appears only as approval provenance, never as authorship: on each source's approvedBy, on the queued job that carries it there, and on the ingest.folder_run.created entry for the request that started the run. Everything the run itself then does — the status transitions, the pages, the revisions — is attributed to the machine.

The ledger also binds a run to exactly one queue job, and a worker whose job differs is refused before it writes anything.

Audit

Every entry is written in the same transaction as the change it describes, so "the run moved" and "we recorded that it moved" are atomic:

• ingest.folder_run.created — the request that created the run and enqueued its job, attributed to the person who made it. A replayed idempotency key creates nothing and so records nothing.
• ingest.folder_run.started — pending → running, filed by the worker.
• ingest.folder_run.completed or ingest.folder_run.failed — the terminal transition, and a run receives exactly one of the two, never both.

Each carries the run's size and how its sources came out; none carries the run's classification ceiling, its manifest, or any page or batch id. A run's exact composition is withheld from readers who are not cleared for it, and the audit channel does not reopen that — the ceiling reaches the entry only as the entry's own classification. Every ingested source additionally files an ingest.batch.committed entry.

Creating a run also records a read. The database function that builds the run, wiki.create_folder_run, walks the folder's ancestors and, when a key is retried, checks the stored run's root. Both reads can reach pages above your clearance, so each request files one wiki.folder_run.constructor_check_applied_outside_clearance_scope entry. The entry names the folder you asked for, is classified RESTRICTED, and says nothing about what the walk found. The function writes it itself, so the record does not depend on which caller reached it. Since PLT-1602 the function has one signature, which names the actor type and takes the entry id: the older 6-argument form, kept for one release for the build a deploy replaced, filed every entry as USER and is dropped (migration 20261105175257). When the request is refused, the refusal rolls that write back and the application files the same entry again, so a refused request is recorded exactly like an accepted one.

Large runs: yielding and the walk cursor

The worker runs inside a scheduled invocation that stops after 60 seconds, and a run of up to 50 sources can take longer than that. So a run does not have to finish in one invocation (PLT-1118):

• It yields between sources. Each invocation stops starting new work at its own deadline — the same point at which it stops starting new jobs — and the worker checks it before each source. Once that point has passed, and once it has finished at least one source in this invocation, the run stops and hands its job back to the queue to continue on the next tick. A yield is not a failure: the run stays running, no status changes and no audit entry is filed, and the queue does not count the yield against the job's retry budget.
• It resumes where it stopped. The ledger records how many manifest positions the run has finished (sources_walked, a count — it names no page), advanced after each source. The next invocation takes the run over and continues from there instead of re-checking every source it already finished.

A run of N sources therefore needs at most N invocations, and finishes with exactly the batches and the audit trail of a run that completed in one: one started entry, one terminal entry, at most one batch per source. The terminal entry's per-attempt figures (committedThisAttempt, skippedByReason) describe the invocation that finished the run; committedCount is the whole run's, and a failed entry's walkedCount says how far the run got.

Crash recovery

Job delivery is at-least-once: a worker that dies mid-run is recovered by the queue's stale-claim lease and handed the same job again. The recovered run re-reads its manifest from the cursor — the last source an attempt recorded as finished — and every source a previous attempt committed is recognised and skipped, so the run finishes without re-ingesting anything. A worker killed between finishing a source and recording it re-checks just that one source.

The result is that a crash costs time, never duplicated content and never a duplicated audit trail: at most one ingest batch and one commit entry per manifest source — exactly one for every source actually ingested, and none for one the run skipped — however many attempts it took. A replay of a run that already reached a terminal status writes nothing at all — no status, no audit, no batch — because an audit entry for a transition that did not happen would be a false record.

One attempt drives a run

The lease recovers a worker that died; it cannot tell a dead worker from a slow one. A handler still executing past the lease is not told that a new attempt now holds its job, so two attempts can be executing one run at the same time. The run ledger records which queue attempt currently owns the run, and every write is fenced on it:

• entering a pending run stamps the attempt; a newer attempt resuming a running run takes it over, and an older attempt arriving afterwards stands down without writing;
• every source's transaction verifies, under the run's row lock and immediately before it commits, that the run is still running and still owned by this attempt — otherwise the whole ingest is rolled back, so no batch, page or commit entry from a superseded attempt ever lands;
• the terminal transition is guarded on the same stamp, so a superseded attempt cannot file a terminal entry over the owner's run.

The terminal entry's committedCount is read from the ledger under that same lock rather than added up from the attempt's own walk, so it states what the run holds. Each transition entry also records the attempt that performed it, so a completed entry filed by attempt 2 after a started entry filed by attempt 1 reads as exactly what happened.

A rolled-back source keeps its read audit entries and loses its write entries. A source's transaction may record that its governance walk crossed a classification boundary the worker cannot see; if the transaction is then rolled back — by the fence, or by a fault — those read entries are filed again in a separate transaction, because the read happened whether or not the ingest survived, while ingest.batch.committed and the other write entries stay rolled back with the batch they described.

When the job dies

A run is driven by one queue job, and the queue gives up on a job after its retry budget: the handler failing on its last attempt, or — the common case for a large run, since the worker route stops after 60 seconds — the lease reclaiming a job whose budget is already spent, which is settled dead without running again. Nothing would then ever resume the run.

So once the worker has settled a folder run's job dead, it moves the run to failed, in the run's own tenant, with an ingest.folder_run.failed entry whose failureCode is JOB_DEAD_LETTERED. This applies to a run still pending (its job died before the run was ever entered) as well as to a running one. The entry records the status it replaced, the attempt the job died at, the attempt that last owned the run, and committedCount — the batches the run holds, counted under its row lock, as on every terminal entry. It carries no per-source tally, because nothing walked the sources. A worker that was still executing past its lease finds the run no longer running at its next source and stops without writing.

What is committed stays committed; start a new run to finish the folder. Since a large run now yields rather than being killed (see Large runs), and a yield is never charged to the retry budget, a job dies of its own faults, not of its size. The worker's response reports how many runs it failed this way (deadRunsFailed), and a run it could not fail (deadRunsUnreconciled) is reported as an error.

The worker fails the run only after the job's dead has committed, so a process that dies in between — or a reconciliation that fails — leaves the run open behind a job no later tick sees again. A daily sweep (/wiki/api/cron/sweep-dead-folder-runs, 08:00 UTC) is the backstop (PLT-1759). It finds every active tenant holding a pending or running run, asks the job queue — inside that tenant — which of those runs' jobs are dead, and fails each such run with the same JOB_DEAD_LETTERED entry, so a run ends the same way whichever path ended it. A run whose job is still pending — one that yielded before the worker's deadline and will resume — or running, completed, or missing from the queue is left alone. The sweep requires the tenant's seeded wiki agent and skips, visibly in its audit trail, a tenant that has none; the run's entries are attributed, like every folder-run lifecycle entry, to the system.

Reverting

Any run that is not already reverted can be reverted as a group, undoing every batch it committed in one audited transaction. See source retraction for the single-batch equivalent.

Reverting a pending or running run also stops it (PLT-1055). The revert flips the run to reverted, and the worker's per-source fence admits a commit only into a running run its attempt owns, so a source in flight when the revert lands is rolled back whole and the sources not yet reached are never ingested. The revert and a source transaction take the same tenant and space locks, so every batch the run committed before the revert is reverted with it. A pending run is cancelled with nothing to undo, and its queued job writes nothing when it runs. A failed run's committed batches are reverted as any other; the failed execution is still retried as a new run, not resumed. The group entry, ingest.folder_run.reverted, records the status the run was reverted from as previousStatus.

The cap is set by the revert (PLT-1056). The whole run is reverted in one transaction, and that transaction has to finish inside the revert request's time budget, so a run covers at most 50 pages. The figure is measured rather than chosen: the revert costs about 34 database round trips per batch, and 50 batches fit the budget even at a pessimistic 25 ms a round trip, where the earlier provisional cap of 200 did not. A run created under that earlier cap with more than 50 batches can no longer be reverted as a group: the run history no longer offers Revert on it, and a direct request is refused with 409 and nothing changes. Revert its batches one at a time instead. That also means such a run, if it is still pending or running when the change deploys, can no longer be stopped by a group revert once its 51st batch has committed. Its batches cannot be reverted one at a time while it is active either — a single-batch revert waits for its run to settle — so they can be reverted only once the run has completed or failed.