Skip to content

Runs

Runs are where you execute rulesets against documents and review results. This is where verification actually happens — you see whether your rules pass or fail, and why.

Every ruleset encodes policy requirements as formal logic. A run takes that logic, applies it to real document content, and produces per-rule results showing TRUE, FALSE, or UNKNOWN for each assertion. Whether you are spot-checking a single document or running batch tests across an entire dataset, runs are the verification step.

How Runs Work

When you create a run, the platform executes the following steps:

  1. Select inputs — Choose a ruleset and either input text (interactive) or a dataset (batch)
  2. Ask questions — For each document, the platform asks the ruleset's questions of the document content using an LLM to extract answers
  3. Evaluate DSAIL — The extracted answers are fed into the ruleset's DSAIL code as variable assignments
  4. Solve constraints — The SMT solver evaluates each assertion to determine if it is satisfied (TRUE), violated (FALSE), or indeterminate (UNKNOWN)
  5. Display results — Results are organized by rule, showing each assertion's outcome along with the questions and answers that informed it

One shared claim, one extraction

Step 2 asks the ruleset's questions, not each rule's questions in turn. A registry claim — one written with the @namespace.name sigil — is asked once per document however many rules name it, and the single answer is recorded against every one of them. A claim written as a bare identifier stays private to its rule, so fourteen rules that each declare their own freq_band_mentioned really do ask fourteen questions and can get fourteen different answers.

Two consequences. Rules that share a claim can no longer disagree about it — they are reading one observation, so the value shown for that claim is identical in each rule's Claims table. And a run makes fewer model calls, so it finishes sooner. Metered usage is deliberately unchanged: it counts the verification each rule's verdict rests on, not the number of calls, so it is the same whether a claim is shared or duplicated.

Claims that are not folded together: bare identifiers, the same claim bound to two different input slots (two documents, so two answers), entity-scoped claims (answered per instance), and references pinned to different corpus versions. See One claim, one answer.

Registry claims inherit their question at run time, not at save time

A ruleset rule naming @ns.foo never stores a question of its own: the wording is the registry's, and it is read when the run's configuration is built. Two things ride in that way — the question and the declared reading (what silence means) — and nothing else, because everything else about a claim reaches a run through the version its rules pinned.

That is what makes one edit to a shared claim reach every ruleset that names it, with no copies left behind to go stale. It also means an unversioned reference floats: reword a claim and the next run asks the new wording, with no edit to any ruleset. A rule that pins a version — @ns.foo@2 — gets that version's wording and does not float.

A corpus rule is the opposite: its claim links are frozen at the rule version that wrote them, so verification always describes the versions it compiled. When the two come apart, the claim says so — see Ahead of verification.

Run Types

The platform supports the following run modes:

Interactive

Enter text directly and run a ruleset against it. Use this during development to rapidly check whether rules work correctly on specific content.

Interactive runs offer an optional Streaming Attribution toggle: the input is split into chunks and the platform tracks the text span that caused each claim to resolve. The run's detail view then highlights those source spans, showing exactly which text drove each result. This is also the path where boolean subtypes become observable.

Two settings control how the input is divided, and the first decides what the second means:

Setting What it does
Chunker mode Cumulative prefix (the default) makes each chunk the document up to that point, so chunk 3 contains everything chunks 1 and 2 did plus the next step. Attribution then reads as "by this point in the document, the claim had resolved". Sliding window makes each chunk an independent excerpt of fixed width, so a claim is resolved from that window alone.
Chunk step / Window size How much text one chunk covers, in tokens (roughly ¾ of a word each). On the cumulative chunker this is the step — how much new text each chunk adds. On the sliding chunker it is the window's full width. Smaller values localize attribution more precisely and cost more model calls.
Chunker overlap Sliding mode only: how much of the previous window each new one repeats, so a sentence spanning a boundary is still whole in one of them. Cumulative chunks all start at the beginning of the document, so nothing between them needs overlapping and the field is not offered.

Sliding mode is the one where boolean subtypes start to matter, because its windows do not contain one another and so the same claim can be measured twice and disagree. Every flip a sliding run produces is stamped attribution_mode: "windowed". A cumulative run stamps "cumulative" — except over the tail of a document long enough that the chunker fell back to windows, which is stamped "windowed" as well. That field, not the request you sent, is the record of how each flip was attributed.

A claim that resolves False because the document is silent carries no excerpt on this path, exactly as it carries none on the default path — see the verdict table below.

Where a claim resolved from more than one observation — the document stating the fact in several places, or an entity-scoped claim with several instances — the claim's Evidence shows each distinct supporting sentence, so the excerpts account for every value the claim reports rather than only the last one found. A claim shared by several rules shows the same evidence under each of them, because it is the same observation.

Batch

Run a ruleset against a Dataset of documents. Batch mode supports several test types that can be run together:

  • Evaluate — Runs the ruleset against every document in the dataset, producing per-rule assertion results for each
  • Variance Test — Runs the same evaluation multiple times (configurable 2-10 iterations) to measure consistency
  • Ablation Test — Checks that each claim's answer actually depends on document evidence: the claim is evaluated normally, then again with its supporting evidence redacted from the document. A claim passes when redaction turns its answer UNKNOWN; an answer that survives redaction signals the claim isn't grounded in the document. Two modes are available: Document-free (evaluates against a generated evidence snippet) and Corpus (uses the dataset's own documents).
  • Counterfactual Test — Checks each claim's independence from irrelevant context: an LLM proposes variations of the claim's context, the claim is evaluated under each variation, and the claim fails if its answers diverge.
  • Full Quality Audit — Runs all of the above (Evaluate, Variance, Ablation, Counterfactual) in one action and adds a per-claim quality diagnosis with suggested improvements.

Understanding Results

Run results are organized into tabs:

Evaluation tab — Shows per-document results. Expand a document to see per-rule sections, each containing:

  • Assertions — Each DSAIL assertion shows TRUE (satisfied), FALSE (violated), or UNKNOWN (indeterminate), along with the DSAIL code that was evaluated. A quantified assertion may also show AMBIGUOUS (contradicted) — see below
  • Claims — The questions asked of the document and the answers extracted by the LLM. A registry claim appears under every rule that names it, showing the same value in each — it was asked once and they are all reading that one answer

Results can be exported as CSV or JSON. When any document in the run carries a mark, the CSV also gains mark_status, mark_level, mark_low, mark_high, mark_unknown_driven and mark_unanswered_claims columns — the qualifier travels with the level, because a spreadsheet of bare levels cannot tell a measured mark from the worst case the rules admit.

What a run's numbers mean

Two families of number ride a run's summary and they measure different things:

  • Metering — jvu_count, num_rules, questions_per_rule. This is billing: jvu_count is documents x rules x ceil(questions in rule / 5) x text units, so it is fixed by the shape of the ruleset and the size of the input. It is not a measure of how much of the run landed, and it does not move when claims come back unanswered.

  • Corpus ingest — corpus_ingest: what the run wrote to the claim registry, and why anything was left out. skipped_unanswered is a claim nothing answered; skipped_uninterpretable is an answer the corpus could not read, which is a different fact and is never reported as the first.

    The commonest way to reach skipped_uninterpretable on an ordinary LLM run is a range. "The operating frequency lies between 11 and 14 GHz" is ordinary in a specification, and the numeric parser takes the leading number with the token that follows it — giving 11 and, which is a value the unit converter then refuses ("no conversion from 'and' to the claim's 'GHz'"). The claim is excluded from binding, the run carries a claim_answer_not_bound warning naming the conversion that failed, and the ingest counts it here rather than as unanswered — because the document did answer, and what it said is worth reading. A grail head's raw prediction and an enum answer outside the claim's declared domain reach it the same way. - Coverage — answer_coverage: documents, unevidenced_documents, askable_claims, bound_claims, rate_limited_claims, extraction_failed_claims, never_asked_claims. This is how much of the verdict came from the documents rather than from the rules' completion policies.

The Runs list carries the coverage as a badge beside the status chip, because the banner below is only reachable by opening the run. The status itself stays completed: it describes execution, and a run whose claims all came back unbound did execute successfully.

Badge When
extraction failed Any claim's extraction died, so the model was never asked. Shown ahead of the others because it is an infrastructure failure rather than a fact about the documents — and because a run where every claim failed has nothing left askable, so it reports no unevidenced documents at all.
rate limited The provider throttled claims and nothing else is amiss. Re-running may complete them.
no evidence Every record had askable claims and none bound a value.
N/M unevidenced Some records bound nothing.

The counts are computed when a run's results are first materialized, so a run carries them from completion without anyone opening it — which is the point of having them in the list.

Three reasons a claim has no value

A claim with no value reads Unknown however it got that way, and the three ways are not the same fact. The results say which:

What you see What happened What to do
Unknown The document was read and does not answer the question. For a claim about the world, that is the honest answer; for a claim about the text it should have been False. If the same claim answers Unknown on some runs and False on others, its wording asks about both and the judge is picking one per call — see what a claim asks. Nothing, or declare what silence means on the claim.
False The document answers the question negatively. A False from an explicit denial quotes it; a False from silence on a claim about the text carries no excerpt, because there is nothing to quote. Nothing — this is a finding.
Rate limited The LLM provider throttled the call. Re-run; the answer is recoverable.
Unknown, with Claim extraction failed above the results The model could never be asked — a rejected credential, a timeout, an activity that died. Fix the platform problem. Nothing here is a statement about the document.
Unknown, with no question text noted above the results The claim carries no question, so no call was made by design. Author the question text in the ruleset — the claim is inert until you do.

Above the results you may also see No evidence reached these rules. It fires when a document had askable claims and none of them bound a value, and it is worth reading carefully: a rule's assertion always produces a verdict, and when nothing bound, that verdict came entirely from the rule's completion policy rather than from the document. A pessimistic ruleset reports a confident FALSE over a document it never read a word of, and the run still reports success. The banner is the difference between a result and an artefact of the policy.

The threshold is zero bound claims for that document, not a proportion. At zero, every assertion is decided by the completion policy and no evidence entered any rule; at one, evidence is in play. Claims that were never asked and claims whose extraction failed are excluded, so the banner stays quiet on a ruleset that legitimately asks nothing and fires on one that asked and got nothing.

Why Results Show UNKNOWN

An UNKNOWN result means the solver could not determine whether the assertion is satisfied or violated, usually because one or more input variables were not answered by the LLM. The ruleset's completion policy controls how unknowns are treated — pessimistic mode fails them, optimistic mode passes them, and neutral mode reports them as-is.

Why Results Show AMBIGUOUS

AMBIGUOUS means the document argued with itself: it both supported and refuted the same claim about the same entity, so the assertion could not resolve either way. It is reported separately from UNKNOWN on purpose — UNKNOWN means the document never answered, while AMBIGUOUS means it answered twice, incompatibly, and that conflict is usually the thing worth reviewing.

A rule combines its assertions without ranking TRUE above FALSE or the reverse. UNKNOWN means "no signal", so an assertion that did resolve carries the rule on its own. TRUE and FALSE are equals, so a rule holding both is in conflict with itself and reports AMBIGUOUS — the assertions disagreeing is the contradiction, even when each one resolved cleanly. And an AMBIGUOUS assertion makes the whole rule AMBIGUOUS, whatever its siblings did.

So a rule reports YES only if the assertions that spoke all agreed on TRUE, NO only if they all agreed on FALSE, UNKNOWN if none of them resolved, and AMBIGUOUS otherwise. Each assertion still shows the result it actually reached, so you can see exactly which ones conflicted; it is the rule's own verdict that is withheld rather than guessed. This is deliberate: whether TRUE or FALSE is the "good" outcome depends on how you phrased the rule, and the platform has no way to know, so it will not let one resolved assertion overrule another.

A claim is judged in the company of its ruleset's other claims

A claim is never judged alone. On the streaming path every claim of a ruleset is answered in one call, so the model has always seen the others; on the default path each claim gets its own call and is given the list of the others being asked of the same passage. Either way, the set of claims a ruleset declares is an input to every answer it produces.

That is deliberate, and it is what stops a claim annexing evidence that belongs to its neighbour. Given "The listing agent made no statement of preference, limitation, or discrimination regarding any protected class," a claim asking whether the message mentions a protected class as a neutral matter of fact answers yes when it is judged with no idea that anything else is being asked — the phrase "any protected class" is there, and from inside a single-claim call it is the only relevant text in the passage. Seen beside the claim that asks about stated preferences, it is plainly part of a denial, and the answer is no.

Moving a claim between rulesets can change its answer

The same claim, on the same sentence, can answer one way declared on its own and another declared beside neighbours whose subjects overlap it. This is most visible on a claim whose own wording is doing borderline work — one that asks about "a building, neighbourhood, or area" put to a sentence about a unit, say — where the neighbours are what tip a marginal reading.

It is not run-to-run variance and re-running will not resolve it. If a claim answers differently in a large ruleset than it did in a small one, the claim is asking something the neighbours also plausibly answer; the fix is in the claim's wording, not in the run. Sharpen it until it names its own subject unambiguously, and check it in the ruleset it will actually live in.

A claim answered True on the streaming path is asked a second time on its own before the answer is kept. A True asserts that the document states something, and it is the answer that can suppress a bound — a claim that appears negated in a ceiling's condition switches that ceiling off when it comes back True. Thirteen claims in one response is more per-claim attention than a model reliably gives, and the second opinion is what stops a sentence about one subject being read as an assertion about another. False and the other types are delivered as answered — which is what keeps this affordable, since a claim about the text answers False on silence and False is the majority answer on a typical document.

It is not free: on a thirteen-claim document where five come back True, the second opinions take a chunk from roughly 2,400 tokens to 8,000. That is still about half what asking all thirteen claims individually would cost, which is what the non-streaming path does. Streaming is not there to be cheap in absolute terms — it is there for chunk-level attribution — so a run that does not need attribution should not use it.

Every run records what its judges saw, under summary.extraction_context: one entry per input slot, with the number of claims asked against that slot and a digest of their questions. Two runs of the same document that disagree can then be told apart — same ruleset, different digest means the company the claim kept was different.

The list is per input slot: a claim is only told about the ones sharing its document, because claims read against different documents are not neighbours. Rulesets larger than 60 claims are not given one at all — a roster that long is a substantial addition to every call, and a partial one is worse than none because it implies the claims it left out belong to the call reading it.

Variance Testing

Variance testing measures how consistent a ruleset's results are across repeated runs. Because LLM-based question answering is non-deterministic — the same question asked twice may produce different answers — variance testing reveals how stable your ruleset's results are.

How Variance Testing Works

When you select "Variance Test" in the batch run configuration, the platform:

  1. Runs the ruleset against each document in the dataset multiple times (you choose 2-10 iterations)
  2. For each claim (question), records the answer from every iteration
  3. Computes the mode (most common answer) and variance (how often answers differ from the mode) for each claim

Reading Variance Results

The variance results view shows a summary and a per-claim breakdown:

Summary statistics:

  • Documents — Number of documents tested
  • Claims — Total number of claims across all rules
  • Avg Variance — The average variance percentage across all claims
  • Dissent — The share of individual claim-answers that disagreed with their own claim/document majority, shown with the counts behind it (for example 3/624)

Read Dissent, not Avg Variance, when you need one number to hold a release to. They measure different things. Avg Variance averages the per-claim figures below, so a single claim that split eleven-to-one on one document is divided by every steady claim beside it and reads as very nearly zero. Dissent counts the disagreeing answers themselves, against a denominator of claim-answers — one claim, asked once, is one — which is why it stays comparable between rulesets of different sizes and runs of different iteration counts.

The counts are part of the reading. A rate measured over a few hundred claim-answers carries an interval wide enough that two runs can differ severalfold and mean nothing; widen the dataset or raise the iteration count before treating a movement as real. Iterations that never ran because the provider rate-limited them are excluded from both halves — the model did not answer, so it did not disagree.

The same counts are on the API as answer_stability inside a run's summary, so a build pipeline can read the rate without the UI.

Per-claim table with columns:

Column Description
Claim The claim variable name, identified by both its parent rule name and question key (e.g., "SOX Audit Rule: is_compliant") to distinguish identically-named questions from different rules
Question The data extraction question
Mode The most common answer across iterations
Variance How often answers differed from the mode

A registry claim shared by several rules is listed once under each of them, and those rows are identical — same question, same mode, same variance — because one extraction produced them all. The variance you are reading there is the claim's own instability across iterations, never a disagreement between the rules.

Variance is color-coded:

Variance Color Meaning
0% Green Perfectly consistent — every iteration produced the same answer
< 25% Light green Highly consistent — occasional variation
< 50% Yellow Moderate variation — question or document may be ambiguous
< 75% Orange High variation — results are unreliable
>= 75% Red Very high variation — the claim is essentially non-deterministic

Expanding a claim shows per-document detail: each iteration's answer side by side, with answers that differ from the mode highlighted in orange. This lets you pinpoint which documents and claims are producing unstable results.

Which claims wobble changes between runs; the rate is the stable thing

Repeated measurement of the same corpus finds a different set of claims each time, in different cells, at a roughly constant overall rate. So the per-claim table tells you where to look today — it is not a list to fix and be done with, and a claim absent from it has not been shown to be stable. Hold the release to the rate; use the table to find the next claim worth rewording.

A claim that keeps appearing is usually being asked something its document cannot settle. The most direct fix is not rewording but declaring what its silence means, which takes the choice away from the judge instead of leaving it to be made again on every call.

Which build produced a run

Every run records the processes that did its work, under engines in the run's summary — one entry per process, each naming the service, its version and host, and when that process started.

The version is not what tells two builds apart. Every process reports the platform's release version, so a worker running code from last week and one started a minute ago look identical by that field. process_started_at is the one that separates them, because a process cannot be running code written after it started. Compare it against when the code in question last changed and "was this run produced by the build I think it was?" becomes a lookup rather than an argument.

There is more than one entry because more than one process contributes, and they can be at different builds: the worker that evaluated the run, and the platform service that turned its output into the results you read. host is the container id, which is also the worker identity the workflow engine reports, so a run lines up against its workflow history without translation.

This matters most for a run whose answers you are trying to reproduce. Results are transformed once and served from that point on, so a run's visible shape is fixed by the build that materialised it — not only by the one that evaluated it.

What to Do About High Variance

High variance typically indicates one of:

  • Ambiguous questions — The extraction question is too vague. Make it more specific.
  • Ambiguous document content — The document doesn't clearly state the information the question asks for. Consider improving the document or adjusting the rule.
  • Borderline cases — The answer genuinely depends on interpretation. Consider whether the rule's DSAIL logic needs adjustment to handle edge cases.

Variance results can be exported as CSV or JSON for offline analysis.

Working with Runs

The Runs page shows all completed and in-progress runs for your project.

Runs page

Runs are listed with their ID, type (interactive or batch), status, and start time. Use the filter pills (All, Interactive, Batch) to narrow the list.

Run results also surface outside the project: the workspace-level Activity page is a feed of run summaries from across all your projects, interleaved with data changes, and each run card links back to the run's detail view here.

Note

Runs against a dataset are project-scoped, so the ruleset must belong to a project. An unaffiliated ruleset can still be exercised through the Ruleset Studio's interactive test (manual input) — add it to a project when you're ready to run it against a dataset.

Creating a Run

A ruleset with a rule that does not compile cannot be run

Rules are compiled when the run's configuration is built, and a rule that fails to compile can never produce an assertion. Starting the run anyway produced a completed run with no claims and no assertions — success, to anything reading status. Both POST /runs/interactive and POST /runs/batch now refuse with a 400 naming each rule that does not compile and the compiler's message, and a batch run whose row was already written is marked failed with the same text in error_message.

GET /rulesets/{id}/readiness reports the same thing before you submit, with the line and column. If a run does reach the workflow anyway and every rule in it failed to compile, the run ends failed, with the compiler messages in error_message, rather than completing empty. A rule that fails while running is a different case and does not fail the run: its step carries the diagnostic, and the other rules' verdicts still come back.

Click New Run to open the run configuration dialog.

New run dialog

  1. Choose Interactive or Batch mode
  2. Select a Ruleset from the dropdown
  3. For interactive: provide a document for each input slot the ruleset declares — single-slot rulesets show one text area, multi-slot rulesets show one labeled box per slot — and optionally enable Streaming Attribution. For batch: select a dataset and check which batch modes to run (Evaluate, Variance Test, Ablation Test, Counterfactual Test, or Full Quality Audit)
  4. If variance is selected, choose the number of iterations (2-10); if ablation is selected, choose its mode (Document-free or Corpus)
  5. Click Start Run

A run you open from the list keeps its id in the address — ?run=<id> — so the open run can be refreshed, bookmarked or sent to someone, and closing it clears the parameter again. /projects/{project}/runs/{run} is a valid address too and opens the same view.

The dialog also includes an API snippet section showing how to create the same run via cURL, Python, or JavaScript. The snippet uses the named-slot request shape (input_documents), so it works unchanged for both single-slot and multi-slot rulesets.

Supplying claim answers instead of extracting them

POST /api/v1/runs/interactive accepts an optional supplied_answers map: claim values you already hold, handed to the platform instead of being extracted from a document. Supplied claims are not asked of the model; the rules are evaluated against the values you gave.

{
  "ruleset_id": "<id>",
  "supplied_answers": {
    "@sys.freq_band_mentioned": { "value": "True", "source": "run:<earlier-run-id>" },
    "@scg_alpha.item_waveform_x": "False",
    "@sys.operating_frequency": "Unknown",
    "has_disclaimer": "True"
  }
}
  • Keys are claim keys exactly as the rules declare them: @namespace.name (sigil included) for a registry claim, the bare identifier for a rule-local one. One supplied registry claim binds in every rule that reads it, the same as one extracted answer would.
  • Values are what extraction would have produced: "True" / "False" for a boolean, a number optionally carrying a unit ("12 GHz"), or one of an enum's declared options. "Unknown" is a legitimate value meaning leave this claim free — the rule's completion policy then governs it, exactly as it does for a claim the document never answered.
  • The object form {"value": ..., "source": "..."} attaches provenance. The source is free text: the platform stores it and shows it back, and never dereferences it.
  • Claims you leave out stay Unknown. A partial map is not an error, which is what makes "supply a few, extract the rest" well-defined.

input_documents becomes optional when every claim that would otherwise be extracted is supplied. With every claim supplied, the run makes no model calls at all — it is solver-only. Supply a document and a partial map to have the supplied claims bind and the rest extract, which is how you re-mark a document after correcting one answer without paying to extract the other forty.

In the results, a supplied claim's Evidence reads supplied: <source> (or just supplied) where an extracted claim shows the model's supporting excerpt. The run record keeps the map you sent, verbatim, so the run is its own audit of what was asserted rather than read.

The request is refused with a 400 for a claim key the ruleset version does not have, a value of the wrong type for the claim, an enum value outside the claim's declared options, or a malformed object form — each naming the claim it is about. A value that would not bind is refused rather than dropped, because a dropped value leaves the claim free and its rule decided by completion policy instead of by your evidence.

Entity-quantified rules need a document

Rules quantifying over entity scopes draw their instances from the document text. With no document there are no instances, so those assertions report UNKNOWN — never a vacuously satisfied quantifier — and carry the warning no instances available: no document text. Supplied answers cover scalar claims only; an entity-scoped claim cannot be supplied. Supplying answers is also not available together with Streaming Attribution, which derives its per-claim state from the document as it is chunked.

The mark: one program over the whole ruleset

A ruleset can name an ordered registry claim as its Mark target — typically a classification level, an escalation tier, or a risk band. Doing so is the single opt-in for marking, and it is set on the ruleset version, so retargeting a mark makes a new version. Every run of that ruleset answers one extra question in addition to its per-rule verdicts: what level does that claim take for this document?

Per-rule evaluation cannot answer it. Each rule is solved in its own program, so a claim nothing extracts — which is what a solved-for verdict claim is — enters every program unbound, and every rule that bounds it is satisfiable at any level. Three green rules and no mark. The mark is solved from one program holding the ruleset's rules that bound the level, the axioms that align the vocabularies, and the active precedence set, with the run's own claim answers bound into it.

Which rules take part is decided by what they say, not by an extra setting:

  • A rule whose assertion bounds the target — Or(Not(<condition>), @clas.portion_level >= 2) and the other spellings of an implication — is a floor or a ceiling and joins the program.
  • A rule's ordinary assertions do not. Not(And(band, location)) states what ought to be true; a document that violates it is non-compliant, and that is what the rule's own verdict says. Putting it in the mark's program would make every non-compliant document's mark contradicted.
  • Rules quantifying over entity scopes contribute nothing to the mark, and say so.

Where the floors and ceilings live is up to you. Author them in the Vocabulary Workbench and name their corpus rulesets on the ruleset, or write them in the ruleset itself — the mark is solved over both together.

How it reads. The mark comes back as ordinary DSAIL assert results, one per level of the claim's scale, of which exactly one is TRUE:

assert marked_unclassified { @clas.portion_level == 0 };
assert marked_confidential  { @clas.portion_level == 1 };
assert marked_secret        { @clas.portion_level == 2 };
assert marked_top_secret    { @clas.portion_level == 3 };

The name is built from the level's own display label: lower-cased, with every run of other characters becoming one underscore, so Top Secret becomes marked_top_secret. A scale labelled U / C / S / TS therefore produces marked_u … marked_ts — the label you chose is the name you read. Beside the asserts, the result carries the solved level, its label and the level's gloss as structured fields, together with the identity hash of the corpus slice the answer was computed under.

Three properties the mark obeys.

Prove high, default low. The floors alone define the level: it is the highest level demanded by a floor whose condition holds, or the bottom of the scale when none holds. It rises only when a rule proves a higher one, and nothing raises a mark to be safe except the completion policies you authored.

The safe mark. A claim the run left Unknown is resolved to the worst case, so an unknown can raise the mark and never lower it. This is the same semantics as the Analysis mark. Because that resolution is invisible in a single number, the result also carries low and high: the level the rules force if every unanswered claim resolves favourably, and the level they force if it resolves the other way. When those differ, an unanswered claim moved the mark, and the two numbers say by how much. Do not read the interval as the flag, though: unknown_driven is the field that says an unanswered claim decided this mark, and it can be set with low and high in perfect agreement — see the ceiling case below.

A ceiling is a consistency check, not a cap — and it can still lower the number. A ceiling is never part of the definition of the level — it is a consistency check against it. At the level the floors define, a ceiling is either satisfied, in which case it is slack and changes nothing, or violated, in which case the mark is reported contradicted (below) rather than quietly reduced. There is therefore no run in which a ceiling lowers a level the floors have proved, and no point looking for one. What a ceiling can do is decide an unknown. The safe mark asks for the highest level still reachable under some completion of the unanswered claims that satisfies every rule in scope, and a completion in which a floor and a ceiling collide satisfies neither, so it is excluded before the maximum is taken — together with the level that floor would have demanded. When the only completion that fires a floor is also the one that fires a ceiling contradicting it, that floor can never set the level, and the mark comes back at whatever survives — which may be lower than the same rules give with that ceiling absent, by more than one step. That is the one route by which a ceiling changes a number, and it can happen with low and high in agreement — so the interval is not what tells you: a mark decided by a claim the run left unanswered is reported unknown-driven whether or not a ceiling collapsed low and high onto one level, and assumed_claims names the claims the rules settled rather than evidence did — which is the shortest route from a surprising level to the question worth answering. Recognise it for what it is: the collision the ceiling stands in for would have been reported contradicted had the claim been answered, so a surprisingly low mark here is a prompt to answer that claim, not a cap to reason around.

A floor authored neutral is still treated adversarially by the mark. A rule's [pessimistic] / neutral completion tag governs that rule's own verdict and the corpus checks; the mark is one joint solve under one policy, and nothing in it reads a rule's tag as a solve policy. The effect is always in the over-classifying direction, never under.

Contradicted marks. If the floors and ceilings bearing on a document cannot all be honoured, the mark is reported contradicted — a first-class outcome, not an error. It carries the minimal set of responsible rules, named and versioned, with each rule's own summary sentence, and a witness scenario read off the document's own answers (any subject where the portion reveals that waveform X exists and not the portion states the operating band). The run completes, every per-rule verdict still comes back, the overall pass/fail is unaffected, and in a batch run the document's siblings are untouched. Design-time corpus verification is what makes this rare: the same collision is findable from the rules alone, before any document is uploaded.

A mark with nothing behind it is not shown. When a run bound no claim value the slice reads, the level the rules give is the level they would give for any subject whatsoever, so the run reports the refusal and the claims to answer instead of the number. When something did bind but an unanswered claim still drove the level, the level is shown — with the range the rules admit and the claims that would settle it, because that is a different statement and needs different reading. Both appear above the level rows in the run detail and in the ruleset editor's Test panel, and the Analysis mark chip carries the same qualifier as the word unanswered beside the letter.

Undetermined marks. Some conditions mean no level may honestly be claimed, and each says which: the target claim is not an ordered scale with a level map; no rule in scope mentions it; a rule bounds it in a shape that cannot be read as a floor or a ceiling; the corpus slice could not be resolved for the run's group. The mark reports every level UNKNOWN with the reason attached, rather than a number nobody can stand behind.

A mark the solver did not finish is not a low mark. The mark is a proof over the rules in scope, and a proof has a budget. When the solver spends it without establishing a level, the mark comes back undetermined with timed_out: true, the reason the solver did not settle the level within the time allowed, and the warning mark_not_decided_within_budget at the top of the warnings list. It never comes back as a level: reporting the bottom of the scale for a proof that did not complete would read as a clean subject, and two runs of the same document on the same slice could then disagree because one machine was busier than the other. Re-running is the fix. The same rule holds for every corpus operation — a conflict scan, a what-if, a feasible region or a lever analysis that runs out of budget reports unknown with timed_out: true and a warning naming the stage, never a shorter list of findings presented as the complete one.

Publishing a marking ruleset is gated on the corpus. A mark is a determination, so before one can be published the coherence of the rules behind it has to have been proved. Publishing a ruleset that declares a Mark target requires that the corpus rulesets it names are covered by a scope profile with its publish gate on, that the profile's verification report is current for its slice and ran against today's coverage obligations, that no error finding in it is left undispositioned, and that no rule the mark would use went unchecked. If any of that does not hold the publish is refused with a 409 naming the condition and the fix — and a ruleset that declares a Mark target but names no corpus rulesets is refused too, because there is nothing to verify. See Corpus Health for the profile and the gate, and the Rulesets API for each blocker. Rulesets with no Mark target publish exactly as they always have.

Corpus ruleset refs are deliberately unversioned. A ruleset version names its corpus rulesets by id, not by id and version. One immutable ruleset version can therefore resolve a different slice on two different days, and give two different marks, as the corpus evolves underneath it. That is the intended behaviour for a governance layer — a guide's revision is meant to reach the rulesets that cite it, and pinning would leave every citing ruleset stranded on superseded policy. It is not a loss of reproducibility either: every mark records the identity hash of the slice it was solved over, so the exact rule, axiom and claim versions behind any past answer stay recoverable. There is deliberately no way to pin one from the API: a ref containing @ is refused with a 422 naming it — "drop the @<version> — a bare id names every version of that corpus ruleset, which is what keeps a marking ruleset tracking the corpus it answers to". If a pinned ref reaches the row by some other route it is still used, and the mark carries a corpus_ruleset_ref_pinned warning saying so.

Iterating on Rulesets

Runs are most valuable as part of an iterative development cycle:

  1. Write or modify ruleset rules in Ruleset Studio
  2. Run an interactive test against representative text
  3. Review results — if assertions produce unexpected outcomes, refine the DSAIL code or questions
  4. When rules look correct, run a batch test against a Dataset to test across more documents
  5. Use variance testing to verify consistency before relying on the ruleset
  • Rulesets define the rules that runs execute
  • Documents provide the input text that rulesets analyze during a run
  • Datasets enable batch testing across multiple documents
  • DSAIL Language defines the assertions whose outcomes appear in run results
  • The Corpus owns the ordered verdict claims a run can mark against, and the axioms and precedence the mark is solved with
  • Analysis asks the same question of a finished run's values, under the same safe-mark semantics