# Foundations of agentic engineering Concept-led scholarly instruction. Historical connections are educational comparisons, not claims of direct influence on the Atlas projects. Original exercises are constructed fixtures. ## Contents - Interpret a request without silently changing it - Separate a valid inference from an established claim - Build a small language and test its combinations - Specify what a task preserves and why it stops - Make agent interfaces reject incompatible work - Represent concepts without confusing names and things - Trace evidence through retrieval and revision - Distinguish information, meaning and task value - Infer what a missing result can establish - Give confidence a measurable meaning - Explain what passing tests can establish - Test whether an intervention caused an improvement - Choose among permitted actions under uncertainty - Control revisions when feedback is noisy or delayed - Allocate limited computation and review time - Reason about events across independent processes - Check a narrow result with explicit assumptions - Study the whole task before choosing an agent - Represent plans, commitments, and reconsideration - Divide work through stable, testable interfaces - Separate permission, dispatch, and observed effect - Give reviewers enough evidence and control to intervene - Evaluate outcomes, repeated behavior, and maintenance - Use possible futures to choose a testable next step ## Interpret a request without silently changing it A student asks for the latest approved exhibition plan. The folder contains a newly edited draft and an older signed plan, both called Exhibition Plan. Returning the newest file answers one possible reading of the request, but the student may need the document that actually governs tomorrow’s installation. Before selecting a file, preserve the words of the request and identify what would make one interpretation correct. ### Prerequisites - Read a short source passage and distinguish its words from a proposed interpretation. ### Learning objectives - Separate the expression from its referent - Use conversational context as evidence - Distinguish reports, requests and commitments - Return to information diagnosis Offline worksheet: [Recovery lab](labs/03-recovery/WORKSHEET.md). [Practice-lab guide](labs/README.md). ### Separate the expression from its referent A string is a sequence of characters. A referent is the object an expression picks out in a particular use. The title Exhibition Plan is a string shared by two files; it therefore cannot identify one file without further context. Give each file a stable identifier and record its revision, publication time and approval state, because these fields answer different questions even when a user uses one short phrase to ask about all of them. Meaning often depends on how expressions combine. In “latest approved plan,” approved restricts the eligible plans before latest selects among them, if that is the intended reading. Selecting the latest file first and then ignoring its approval state performs a different operation. A small change in the order of filters can therefore produce a different answer without changing a single document. Represent the first reading as select the most recent member of the set of approved plans. Let P be the available plans and A be the subset approved for the relevant purpose. The candidate must belong to A. A timestamp ordering then chooses a maximal member of A, provided that the chosen timestamp means what the task requires. If A is empty, the procedure has no eligible answer; silently falling back to P changes the contract. ### Use conversational context as evidence H. P. Grice’s 1975 account of conversation examines how people infer more than a sentence literally states. His discussion of cooperation and conversational maxims concerns contributions understood within a shared purpose. It helps explain why a query log records only part of an information need. Someone searching “access” might mean building entry, document permissions or an accessibility requirement, and the surrounding task can distinguish these uses. Treat an inferred purpose as a hypothesis that can be checked. The available evidence might include the project page the student came from, a named event, or an earlier confirmed instruction. When those records do not select an interpretation, ask the smallest question that changes the answer. “Do you mean the newest approved revision for tomorrow’s exhibition?” is more discriminating than asking for every detail of the project. Grice’s distinctions are a useful account of interpretation, not a universal scoring scheme for good communication. Indirectness may be appropriate, and conversational conventions differ. The practical requirement here is narrower. Record what the person said, which interpretation the system proposes, and which evidence or confirmation supports that interpretation. ### Distinguish reports, requests and commitments The sentence topic does not determine what a speaker is doing. “Please export the plan” requests an action; “I will export the plan” makes a commitment; “The plan was exported” reports a result. John Searle’s taxonomy of illocutionary acts develops distinctions among such communicative functions. The force of an utterance is its role as a request, assertion, commitment or other act, rather than the subject matter alone. This distinction became an explicit design concern in agent communication. The historical FIPA Communicative Act Library specification gives separate definitions for Request, Agree, Inform and Failure. Its formal language uses beliefs and intentions within a technical model. Applying those message labels does not demonstrate that a contemporary model has human mental states, and receiving an Inform message does not independently establish that its content is true. A useful task record keeps the request, permission, adopted commitment, reported result and observed result separately. If a source document contains “export the plan,” its imperative grammar does not turn the quoted sentence into a new instruction from the student. In the Atlas inbox and recovery cases, this is the difference between recording a message as evidence and promoting it into an action that changes a destination. ### Return to information diagnosis The Atlas begins with search behavior because failed queries are evidence about work people are trying to complete. A repeated query may reveal an absent document, a naming mismatch or an interface that hides the right document. Those explanations suggest different interventions. Adding an alias can repair a naming mismatch while leaving an absent approval record absent. Construct a diagnosis before automating. Retain a sample of queries, identify the task each query plausibly expresses, inspect whether the collection contains a suitable answer, and compare that result with what the search actually returns. Mark uncertain task interpretations rather than treating every search string as a confirmed specification. The resulting record supports a smaller, more informative experiment than simply generating more content. ### Worked example: Two revisions and one ambiguous adjective This is constructed practice data. Plan D1, revision 4, was approved for the exhibition and published on Monday. Plan D2, revision 5, was edited on Tuesday but is awaiting approval. The request is “latest approved plan for the exhibition.” Filtering on approval leaves D1; selecting the latest member of that eligible set returns revision 4. The newer editing timestamp on D2 does not change its approval status. Now suppose D1 was approved only for budget review, while the request concerns installation. The original answer no longer follows. Approval requires a scope as well as a yes/no field. Record the installation answer as unresolved and seek the relevant approval record. A confident date comparison cannot supply the missing scope. ### Laboratory: Investigate a changed case Create cards for a request, a promise, a source quotation, an approval and an export receipt. Use the same phrase “export revision 4” in each. Classify what each card establishes, then remove the receipt and explain the task’s remaining status. - Write your prediction before inspecting the complete fixture. - Retain the source records and show each transformation. - Change one assumption and explain which conclusions survive. - Compare approval for budget review with approval for installation, and identify which permits the requested use. ### Model answer The request establishes desired work; the promise establishes an adopted commitment; the quotation establishes what a source says. An approval may authorize the named revision and destination if its issuer has authority. A receipt can provide evidence of the result under its own verification procedure. Removing the receipt leaves the reported or intended export without that observation. Do not relabel the promise as completion. In a transfer case, an independently inspected destination may establish completion even when the original receipt is unavailable. ### Scoring - 3 points: Classify request, commitment, quotation, authorization and observed result distinctly. - 3 points: Resolve approval scope before choosing a revision. - 2 points: Keep the result unresolved when the required observation is absent. - 2 points: Explain an alternative observation that could establish completion. ### Applied lessons - [lesson-01](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-01) - [lesson-05](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-05) - [lesson-13](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-13) - [lesson-18](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-18) - [lesson-21](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-21) - [lesson-26](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-26) - [lesson-31](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-31) ### Sources [H. P. Grice (1975), Logic and Conversation](https://web.stanford.edu/class/psych205/papers/Grice-1975.pdf#page=5) Conversational purpose and implicature help distinguish observed query wording from inferred intent. Research inspected the original scan, pp.41–58 with focus on pp.45–47 and implicature discussion. Not a universal cross-cultural communication metric. [John Searle (1975), A Taxonomy of Illocutionary Acts](https://semantics.uchicago.edu/kennedy/classes/f07/pragmatics/searle75a.pdf) Propositional content and communicative force are distinct. Research used §II and §IV. Historical theoretical taxonomy; not an empirical guarantee about model mental states. [FIPA (2001), Communicative Act Library, XC00037H](https://jmvidal.cse.sc.edu/library/XC00037H.pdf) Request, Agree, Inform and Failure have distinct definitions. Historical experimental version dated 10 August 2001; §§3.2,3.7,3.8,3.19 and Annex5.4.1–5.4.2. Not represented as a current standard. ## Separate a valid inference from an established claim A report says that every approved document has been checked, and a document in the folder carries a check mark. An assistant concludes that the document is approved. Before inspecting the software or improving its prompt, inspect that inference. The check mark may establish exactly what it records while the conclusion still fails to follow. A small collection of documents is enough to show why. ### Prerequisites - The claim ledger in lesson-04 - Basic familiarity with sets as collections of objects ### Learning objectives - Construct a counterexample to an invalid inference - Separate the truth of premises from the validity of a conclusion - Declare the scope of a formal representation - Explain selected historical approaches without claiming a single continuous invention ### Make the reasoning small enough to inspect Let A be the set of approved documents and V the set of verified documents. “Every approved document is verified” says that A is contained in V. It does not say that the two sets are equal. The collection may contain verified documents still awaiting an owner’s approval, and one of those documents is enough to defeat the assistant’s conclusion. A countermodel is an arrangement in which the premises are true and the proposed conclusion is false, allowing a reviewer to reject the inference by inspecting a concrete case rather than debating how persuasive its wording sounds. For an agentic research chain, keep the original premise beside the rewritten conclusion and ask whether the rewrite has reversed a relationship, removed a condition or enlarged the group it covers. Aristotle’s Prior Analytics, Book I.1 and I.4, examines necessary consequence from stated premises and distinguishes forms of categorical statements. His treatment supplies a historical way into this question, although his syllogistic should not be presented as modern predicate logic in different handwriting. The connection to the Atlas is educational; it does not establish that its methods were derived from Aristotle. [F-C01] ### Ask why the premises deserve acceptance A different argument may have a valid form. Suppose every externally approved document has a named owner, and a current record establishes that document A is externally approved. It follows that A has a named owner, provided both premises apply to the same document, revision and collection. If the approval record turns out to concern last year’s revision, the logical pattern remains valid, but the available evidence no longer establishes the premise needed to draw a conclusion about the document currently under review. This is why the Atlas’s claim ledger records versions and applicability rather than allowing a well-written explanation to supply its own authority. In Posterior Analytics I.2, Aristotle distinguishes demonstration from deduction alone and imposes epistemic and explanatory conditions on its premises. The Atlas need not adopt his account of first principles or causes to benefit from the separation between what follows and why its starting statements should be accepted. [F-C02] Frege opens the preface to his 1879 Begriffsschrift by separating the route through which someone arrives at a proposition from the grounds that justify it. For a contemporary workflow, a conversation history records how an answer developed; supporting evidence must still show why the answer should be accepted. The inspected Frege source is one translated preface page, so this lesson makes no claim to summarize the complete formal system. [F-C05] ### Choose a representation without hiding its decisions A document selector needs a declared collection. “All relevant sources” might mean every file in one folder, every file the account can access, or every source that exists. Those collections differ. Write down which one the program actually examines, and preserve a separate state for unavailable or uninspected material. Boole’s Laws of Thought develops algebraic operations on classes and discusses the universe of discourse. Selecting the same fixed class twice leaves the class unchanged, represented by his idempotent law x²=x. The calculation depends on fixed interpretation; it does not establish that the classification chosen by a person or model is correct. His nineteenth-century calculus also differs in details from the simplified Boolean algebra now commonly taught. [F-C04] Leibniz’s Preface to a Universal Characteristic describes an ambitious symbolic instrument intended to aid discovery and judgment. Its completion forecasts belong to the proposed project, and the text’s moral and religious ambitions should not be removed from its historical setting. For the Atlas, the useful question is which disagreements explicit notation makes easier to examine. Assigning identifiers still leaves people responsible for deciding what the identifiers mean and which distinctions the records preserve. [F-C03] ### Retain the scope of each claim through the workflow Consider two requirements and two source documents. One document supports the delivery schedule; the other supports the accessibility commitment. The statement “each requirement has a source” can be true even though no single document supports both requirements. In contemporary notation, ∀c ∃s Support(s,c) says that for every claim c there is some supporting source s. The statement ∃s ∀c Support(s,c) requires one source that supports every claim. Read the symbols aloud before using them. A change in their order changes the assertion, and a summarizer that replaces the first with the second has introduced a claim the records do not support. Add revision and applicability when the task needs them. A support relation might connect a claim to a particular source version and a project stage. Merely retaining a document title will then be insufficient. This is a practical extension of the RFP lesson, where an appendix can qualify the main clause and change which requirement applies. ### Worked example: Find the document that defeats the conclusion Use a synthetic collection U={a,b,c,d}, with approved set A={a,b} and verified set V={a,b,c}. Every member of A belongs to V, so the first premise is true. Document c belongs to V, making the second premise true. But c does not belong to A. The conclusion that c is approved is therefore false in a case satisfying both premises. The corrected output is that c is verified and its approval has not been established by this argument. If a current approval record is later found, that new evidence may support approval. It does not repair the original inference retroactively. Now suppose a draft says “all project capabilities are externally approved,” while the evidence table contains only a subset of approved capabilities. Name a capability outside that subset and inspect its status. If its approval is unknown, the draft’s universal statement is unsupported even when every listed example is correct. ### Laboratory: Review a small source-to-claim chain Prepare six synthetic source records, including an outdated approval, a qualified requirement and two claims supported by different documents. - Declare the collection and revision boundary. - Write one valid inference and one invalid inference about the records. - Construct a countermodel for the invalid inference. - Distinguish each claim having a source from one source supporting every claim. - Record which premises still need evidence after the logical form is checked. ### Model answer A satisfactory review identifies the precise record and revision used by each premise. Its countermodel makes the premises true while the conclusion fails, rather than merely stating that the conclusion sounds doubtful. The valid argument is kept separate from the task of verifying its premises. The two-source case satisfies the claim-by-claim requirement if both sources genuinely support their assigned claims. It does not establish that one document supports the whole answer. The outdated approval remains historical evidence, and the qualified requirement retains its condition in the final prose. Before releasing the draft, the reviewer checks the resulting claims against the current records and records any unresolved premise. ### Scoring - 3 points: Constructs a correct countermodel - 3 points: Separates valid form from supported premises - 2 points: Preserves quantifier scope and applicability - 2 points: States the collection and revision boundary ### Applied lessons - [lesson-02](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-02) - [lesson-04](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-04) - [lesson-08](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-08) - [lesson-10](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-10) - [lesson-15](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-15) ### Sources [Aristotle, Prior Analytics, Book I, translated by A. J. Jenkinson](https://classics.mit.edu/Aristotle/prior.1.i.html) I.1, 24a10–24b22, and I.4: premises, deductions and first-figure forms. Book I parts 1, 2 and 4 inspected in the MIT-hosted translation. Historical comparison; no claim of Drew’s direct influence. [Aristotle, Posterior Analytics, Book I, translated by G. R. G. Mure](https://classics.mit.edu/Aristotle/posterior.1.i.html) I.2, 71b9 onward: demonstration and the character of its premises. Book I parts 1–2 inspected. Its account of scientific knowledge is not identical to contemporary evidence review. [G. W. Leibniz, Preface to a Universal Characteristic, 1678–79, in Ariew and Garber, Philosophical Essays (1989), pp. 5–9](https://www.ratiocination.org/Courses/phil3505/wp-content/uploads/2017/08/Leibniz__Preface-to-a-Universal-Characteristic-and-other-fragments-on-logic.pdf) Symbolic reasoning proposal in its historical philosophical and religious context. Scanned translation pp. 5–9 rendered and OCR inspected. This is an aspiration, not a completed modern calculus or an established direct Atlas influence. [George Boole (1854), An Investigation of the Laws of Thought](https://www.gutenberg.org/files/15114/15114-pdf.pdf) Chapters II–III: signs, classes, idempotence and universe; opening of Chapter V on intermediate expressions. Selected chapters and passages inspected in a public-domain transcription. Its pagination is not asserted to reproduce the original edition. [Gottlob Frege (1879), Begriffsschrift, English preface excerpt](https://www.cs.cornell.edu/courses/cs4860/2009sp/Begriffsschrift.pdf) Opening preface distinguishes the discovery of a truth from its justification. One-page English preface excerpt rendered and OCR inspected; not a reading of the complete book. ## Build a small language and test its combinations Write “compare A with B after review” on a card. A person may infer what it means, but a program needs rules for recognizing its parts and deciding what happens first. Building a tiny language makes those rules visible and gives a precise way to diagnose why a plausible-looking instruction fails. ### Prerequisites - Complete Language 1 or be able to distinguish a string, its interpretation and its use. ### Learning objectives - Define a grammar before an interpreter - Understand what formal language history contributes - Test unfamiliar combinations separately from familiar words - Connect parsing to the experimental-language case Offline worksheet: [Language lab](labs/04-language/WORKSHEET.md). [Practice-lab guide](labs/README.md). ### Define a grammar before an interpreter A grammar specifies which arrangements of symbols belong to a language. A parser uses that specification to recover a structure from a string. An interpreter gives the resulting structure an operational meaning. These are different jobs, which is why a program can accept a sentence grammatically and still fail when its named document is missing. Use the following classroom language. A document name is A, B or C. A command is “read document,” “compare document with document,” or “report.” A reviewed command is a command followed by “after review.” Under this grammar, “compare A with B after review” is accepted, while “compare with A B” is rejected. The accepted form contains a compare operation, two ordered operands and a review condition. The grammar does not yet tell the executor what counts as review, where A is stored, whether comparison is symmetric or whether reporting sends information anywhere. Specify those meanings separately. For this exercise, read copies a named public fixture into a working record; compare records agreements and disagreements; report produces a local draft. The review condition requires a completed review record for the operands before the command runs. ### Understand what formal language history contributes In “Three Models for the Description of Language” (1956), Noam Chomsky compares formally defined mechanisms for generating strings, including finite-state and phrase-structure models. The historical contribution relevant here is the importance of structural descriptions and the expressive capacities of different grammar systems. The paper’s claims do not establish that a modern language model implements a particular grammar or that grammaticality guarantees a safe operation. A finite-state description keeps one of finitely many states and changes state as symbols arrive. A phrase-structure grammar can describe nested constituents through rewriting rules. Nesting matters when one expression contains another expression of the same kind, such as a condition containing a condition. A fixed maximum nesting depth can be handled with a sufficiently large finite-state construction; arbitrary depth requires a different account. State the language’s actual requirements before choosing a formalism. For the tiny command language, a deliberately restricted grammar is useful because a learner can enumerate its forms. Adding free-form prose would make the language more convenient for some users while requiring additional interpretation. Neither choice is automatically superior. The engineering question is which meanings must remain unambiguous and how a rejected or uncertain input is handled. ### Test unfamiliar combinations separately from familiar words A system can succeed on familiar examples without reliably recombining their parts. Lake and Baroni’s 2018 SCAN experiments make systematic generalization testable through a controlled navigation language and carefully chosen training and test splits. The results were mixed, including strong zero-shot performance in Experiment 1 and on “turn left,” alongside failures on other composition tests. Their reported findings concern the recurrent sequence-to-sequence systems and conditions they tested; they are not a result about all neural networks or every current agent. The useful experimental distinction is between a random split, which may place very similar combinations in training and test, and a split that withholds a particular combination while exposing its components. Train the classroom interpreter or human participant on read A, read B, read C, compare A with B and report after review. Then test compare B with C after review. Because C was already introduced, a failure is less easily confused with encountering an unknown symbol. Even this design does not isolate every possible cause. A learner may understand C but misunderstand whether “after review” modifies the comparison or the report. Include separate tests for each condition and record the failure category. An overall pass count conceals the distinction between missing vocabulary, wrong scope, reversed order and failure to combine known rules. ### Connect parsing to the experimental-language case The Atlas’s experimental agent-language material describes an exploratory proposal and distinguishes it from a verified compiler. Use that case to ask what evidence each layer would need. A grammar requires accepted and rejected strings; a parser requires structures matching that grammar; a compiler requires a semantics-preserving translation; execution requires observed behavior under stated conditions. A visual or poetic constraint can coexist with these obligations. It may determine line length, token choice or a permitted arrangement of commands, but the constraint must be stated precisely enough to test. A successful aesthetic judgment does not prove execution correctness, and a successful parser does not settle aesthetic quality. Keep separate test records so that failure at one layer leads to the right repair. ### Worked example: Trace a valid command with an unavailable operand Constructed input is “compare A with C after review.” The parser produces Compare(left=A, right=C, condition=Reviewed(A,C)). The document registry contains A and B but no C. Parsing succeeds because C is an allowed document name. Name resolution fails because the current registry lacks its object. The executor must retain that diagnosis rather than pretending the comparison produced no differences. Add C but omit its review record. Name resolution now succeeds; the condition still fails. Add a review record for C revision 2, while C currently names revision 3. A revision-aware condition continues to fail. This trace shows how one accepted string can reach three different operational outcomes as the environment changes. ### Laboratory: Investigate a changed case Write six accepted strings and six rejected strings for the tiny language. Reserve two combinations for a second student, include all component names in the teaching examples, and ask that student to produce a parse and an execution trace. Add one grammatically valid command that cannot execute. - Write your prediction before inspecting the complete fixture. - Retain the source records and show each transformation. - Change one assumption and explain which conclusions survive. ### Model answer A complete answer states the grammar and semantic rules before classifying examples. “read B” is accepted; “read with B” is rejected. “compare B with C after review” is a useful withheld combination only if B, C, comparison and the review condition were each taught separately. A valid but unavailable command should fail at name resolution or its stated precondition, with no invented comparison result. Score structure, meaning and execution independently, and retain the exact split so that another learner can repeat the exercise. ### Scoring - 3 points: Provide explicit grammar and correctly classify accepted and rejected strings. - 3 points: Show parse structure, name resolution and revision-aware review condition separately. - 2 points: Teach every component before testing a withheld combination. - 2 points: Diagnose a valid but unexecutable command without inventing a result. ### Applied lessons - [lesson-02](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-02) - [lesson-14](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-14) - [lesson-17](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-17) - [lesson-25](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-25) ### Sources [Noam Chomsky (1956), Three Models for the Description of Language](https://chomsky.info/wp-content/uploads/195609-.pdf) Formal generative mechanisms and structural descriptions provide a basis for comparing grammars. Research inspected §§2–3 and §6, pp.113–124. No claim that contemporary language models implement these mechanisms. [Brenden Lake and Marco Baroni (2018), Generalization without Systematicity](https://proceedings.mlr.press/v80/lake18a/lake18a.pdf) SCAN tests held-out combinations under controlled train/test splits. Research inspected §2, §4 experiments 1–3 and §5, pp.2873–2882. Findings concern tested recurrent systems, not all current models. ## Specify what a task preserves and why it stops An assistant must review twelve source records and produce a result for each one. After several iterations it announces that the work is complete, but its table contains eleven rows. A detailed prompt may have named every desired quality while leaving this simplest completion condition untested. Start with the input list, the current position and the rows already produced; those records are enough to define a more exact task. ### Prerequisites - Variables, whole numbers and simple inequalities - Bounded task contracts in lesson-05 ### Learning objectives - Write preconditions, postconditions and an invariant - Use a decreasing quantity to justify termination - Distinguish a general computability limit from a bounded engineering problem - Keep coverage guarantees separate from semantic correctness Offline worksheet: [Verification lab](labs/06-verification/WORKSHEET.md). [Practice-lab guide](labs/README.md). ### Describe the state that matters A state is the information needed to describe the process at a chosen moment. For the review task, use an immutable input list, its length N, an index i and a results list. Begin at i=0 with no results. Each step examines the next input, appends a supported, contradicted or unresolved result, and increments i by one. The task contract should state what must hold before execution, called its precondition, and what must hold afterward, called its postcondition, so that the reviewer can connect the starting assumptions to the promised result. Here the input is a finite frozen list, each examination terminates, and the available storage can hold the required results. At completion, every input must have exactly one corresponding result. Hoare’s 1969 paper formalizes relationships between initial assertions, program execution and final assertions, explicitly crediting Floyd’s earlier work. Its treatment also makes arithmetic assumptions visible. A proof using unlimited integers does not automatically cover an implementation whose index can overflow. The original paper’s correctness assertion does not by itself promise that execution terminates. [F-C07] ### Check what remains true after each step An invariant is an assertion that remains true at the chosen checkpoints. For this loop, require 0≤i≤N and exactly one result for each of the first i input records. Check the invariant before the loop begins. Since i=0 and the results list is empty, no record has been skipped or duplicated. Next suppose the invariant holds and i0. The rule follows because P(H and E) can be calculated either as P(E|H)P(H) or as P(H|E)P(E). The equation does not supply its own inputs. For document review, the prior P(H) might come from comparable documents, while P(E|H) describes how often the extraction and search process produces the observation when the hypothesis is true. Gelman and colleagues place model construction, conditioning and checking together in Bayesian Data Analysis, Chapter 1. The probability identity is exact; the document model remains an assumption that needs checking. [F-P01] ### Retain the state that the task actually needs In the RFP lesson, record whether each appendix was available, extracted and searched, because “no exception found” after complete inspection supports a different inference from the same words produced after the extractor omitted an entire appendix. A final answer should preserve that difference even when a downstream summarizer would prefer one short conclusion. The recovery lesson introduces Boris, a specified architecture for durable contracts and action records. Its recovery mechanisms are design evidence, not a reproduced runtime demonstration. To reason about a saved report, retain the destination and revision alongside the attempted action. “Some report exists” cannot establish that revision R4 was saved, and a receipt for R4 cannot certify the different contents of R5. Kaelbling, Littman and Cassandra formalize related problems with partially observable Markov decision processes. A state describes the world; an observation supplies incomplete information about that state; a belief assigns probabilities to possible states. Their state-update results assume the specified transition and observation models. Using these distinctions to analyze Boris does not claim that Boris implements their planner. [F-P08] ### Account for dependent failures Suppose eight workflow stages each have a failure probability no greater than 0.02 under a declared operating distribution. Let F_i mean stage i introduces a material unrecovered failure. The chance of at least one F_i is at most the sum of the eight probabilities, or 0.16. This union bound needs no independence assumption, although it may be loose. If the eight success events were independent, their joint probability would be at least 0.98^8, about 0.85076. Equality requires every stage failure probability to equal 0.02; the stated upper bounds alone give a lower bound on joint success. Independence is a stronger claim. A missing appendix may affect extraction, classification and final synthesis together. Another valid product multiplies success probabilities conditional on every preceding stage having succeeded, but those conditional rates must be justified. Merely collecting eight percentages from eight different test sets will not establish them. Evidence can be dependent too. Three summaries repeating one report retain one reporting origin. Multiplying three likelihood ratios as if three independent investigations occurred would overstate what the sources establish. ### Worked example: Search the same document through two imperfect channels All numbers in this example are constructed. Among 1,000 comparable documents, assume 200 contain an exception and 800 do not. A functioning search misses 10% of the exceptions and correctly reports no exception for 95% of documents without one. The no-finding group therefore contains 20 documents with exceptions and 760 without them. Given no finding, the exception probability is 20/780, about 2.56%. Now leave an appendix unread often enough that the search misses 80% of exceptions. The no-finding group contains 160 documents with exceptions and the same 760 without them. Its exception probability is 160/920, about 17.39%. Identical visible wording supports different inferences because the observation process changed. For an uncertain save, assume success probability 0.9, missing-receipt probability 0.1 after success, and missing-receipt probability 1 after failure. Bayes’ rule gives 0.09/(0.09+0.10), about 47.37%, for success given no receipt. These invented rates cannot be assigned to a real application without evidence. They demonstrate why a missing receipt cannot by itself establish failure. ### Laboratory: Build a coverage-aware evidence record Use twenty short synthetic documents, five with exceptions. Keep the answer key separate from the search operator. - Using the stated RFP model, calculate the posterior probability after no finding for search sensitivities 90% and 20%, retaining the same prior and specificity. Show both denominators. - Give two documents missing appendices and one a deliberately unreadable table. - Record available, extracted and inspected sections before searching. - Distinguish explicit denial, no finding, contradictory evidence and uninspected content in the output. - Explain the recovery decision when a save has no receipt and its destination cannot be inspected. ### Model answer A sound record identifies the document revision, inspected sections, supporting spans and unresolved coverage. It may give a conditional conclusion about inspected material, but it cannot infer that an unavailable appendix contains no exception. Literal quotations must still be checked against source spans; a probability does not repair an invented quotation. After the uncertain save, the action remains dispatched with an unknown outcome. If destination inspection becomes available, match the intended revision and target before retrying. If inspection is unavailable, report the uncertainty and pause dependent mutations that might duplicate the effect. The calculation explains the need for observation; it does not authorize another save or override a required human decision. ### Scoring - 3 points: Separates source state from extraction and search observations - 3 points: Preserves all four evidence statuses without collapsing unknown into false - 2 points: Calculates both document posteriors correctly - 2 points: Keeps an uninspectable save unresolved and avoids an unjustified retry ### Applied lessons - [lesson-02](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-02) - [lesson-04](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-04) - [lesson-08](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-08) - [lesson-18](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-18) ### Sources [Gelman et al. (2013), Bayesian Data Analysis, third edition, corrections through 20 February 2025](https://sites.stat.columbia.edu/gelman/book/BDA3.pdf) Conditional probability and the model-conditioning-checking workflow; Sections 1.1 p.3, 1.3 p.6, 1.5 p.11 and 1.8 p.19. Downloaded title page, contents and opening chapter inspected. General statistical foundation; none of the Atlas document or save probabilities is a source finding. [Kaelbling, Littman and Cassandra (1998), Planning and acting in partially observable stochastic domains](https://cs.brown.edu/courses/csci2951-k/papers/kaelbling98.pdf) Artificial Intelligence 101:99–134; Sections 3.1–3.4, printed pp.105–108, define observation models, belief states and their updates. Primary paper sections inspected. Correct-model assumptions bound the sufficiency and optimality results. Boris is a pedagogical application of the distinction, not a claimed POMDP implementation. ## Give confidence a measurable meaning A taxonomy worksheet contains a confidence column beside each proposed relationship. One row says 0.8. Before using that number to decide which relationships to publish, ask what would count as its prediction coming true. A reviewer’s support weight, a model’s word probability and an estimated chance of expert acceptance can share the same decimal representation while answering different questions. ### Prerequisites - Fractions and averages - Candidate versus accepted concept in lesson-03 ### Learning objectives - Define the event attached to a probability forecast - Calculate and interpret a binary Brier score - Distinguish calibration, discrimination and quality of the evaluation labels Offline worksheet: [Probability lab](labs/05-probability/WORKSHEET.md). [Practice-lab guide](labs/README.md). ### Define the event before collecting a number For a taxonomy candidate, a usable event could be “accepted by the specified review panel under rubric R,” provided the panel receives the source context, proposed broader relation, definition and applicable governance policy needed to make that decision. Record its decision rather than replacing the candidate silently. The resulting label measures acceptance under that procedure; it should not be presented as a direct measurement of every possible interpretation of semantic truth. The original taxonomy work contains observed workbook counts and an authored account of a larger expansion process. Neither provides a measured semantic accuracy rate by itself. A new confidence study requires labelled cases and a sampling plan. In the feature-value lesson, confidence is explicitly a judged support weight. Preserve that meaning unless an additional study establishes a probability interpretation. ### Score probability forecasts against outcomes A proper scoring rule rewards a forecast distribution according to the outcome that occurs. Gneiting and Raftery define strict propriety as a uniquely best expected score at the true distribution within the stated class. That result concerns the scoring rule, and does not establish that a person or model knows the true probabilities. Their 2007 paper supplies the general framework. [F-P02] For a binary outcome y, use y=1 for acceptance and y=0 for rejection. The binary Brier loss is (p−y)^2, where p is the forecast acceptance probability. Lower loss is better. Assigning 0.9 to an accepted candidate incurs 0.01; assigning 0.9 to a rejected candidate incurs 0.81. Confident errors receive a larger penalty. Suppose the true acceptance probability for a case is q. Expected loss is q(1−p)^2+(1−q)p^2. Expand the terms to obtain p²−2pq+q, then rearrange them as (p−q)^2+q(1−q). The second term does not depend on the forecast p, while the first is nonnegative and vanishes only at p=q. This elementary derivation explains why the binary score favors an honest probability estimate. ### Inspect groups as well as averages Calibration asks whether outcomes occur at the frequencies predicted. If many genuinely comparable cases receive forecasts near 0.8, approximately 80% should be accepted over a sufficiently large evaluation population. A group of five cases is too small to make an exact 80% requirement sensible, and broad groups can conceal variation within them. Discrimination asks whether forecasts distinguish cases with different outcomes. A constant forecast equal to the overall acceptance rate can be calibrated across repeated samples and still provide no useful ranking for review. Report an overall-frequency baseline beside the proposed method, and inspect performance for ambiguous concepts, rare families and changed source conditions. A self-reported model confidence is only a candidate forecast until tested. Fluent certainty supplies no confusion matrix. Agreement among several agents may reflect shared training, prompts or sources, so do not treat agreement as independent evidence without examining how the answers were produced. ### Keep the review procedure visible Prepare the rubric before inspecting which system performs best. Reviewers should know what counts as an unsupported definition, a circular hierarchy, a wrong family assignment and a legitimate multiple assignment. Preserve disagreement and its reasons. If one reviewer treats every multiple assignment as an error while another permits it, calculating a single accuracy number first will conceal the policy difference, because the reviewers are applying different definitions to the same recorded output. Keep development candidates separate from the cases used for the final assessment. Once a team changes its prompts after reading the assessment results, that feedback has become part of development. A later report should state which data influenced the changes and reserve fresh evidence for claims about unseen cases. ### Worked example: Compare forecasts with a simple baseline Four synthetic forecasts are 0.9, 0.8, 0.6 and 0.2. The corresponding outcomes are 1, 1, 0 and 0. Squared errors are 0.01, 0.04, 0.36 and 0.04, so the mean Brier loss is 0.1125. A constant prediction of 0.5 produces mean loss 0.25 on the same four cases. The first method scores better on this fixture. Four cases do not establish its calibration or deployment performance. The rejected case assigned 0.6 deserves inspection: was the forecast poorly estimated, did the source omit a qualification, or did the reviewers disagree about the acceptance rule? The score identifies a contribution to error without diagnosing its cause. If forecasts will determine escalation, examine the cases near the decision threshold separately. A method can have a favorable average score while making costly errors exactly where release decisions are made. ### Laboratory: Run a small forecast audit Use a frozen set of at least twenty synthetic taxonomy proposals with an explicit acceptance rubric. - Write the acceptance event and designate the reviewer procedure. - Record forecasts before revealing the review labels. - Calculate each squared error and the mean loss for both the forecasts and a constant baseline. - Group predictions into broad ranges, display case counts, and inspect disagreements before declaring calibration. ### Model answer A satisfactory audit retains the candidate identity, source, forecast, reviewer label and reason. It calculates the mean over the same cases for both methods and does not exclude difficult cases after seeing their scores. The constant baseline must be specified without exploiting hidden evaluation labels; a development-set acceptance frequency is one defensible choice. Report the limited sample directly. If a group contains three predictions around 0.8 and only one acceptance, the observed frequency is one third, but the sample is too small for a precise general conclusion. The next step is to inspect the rubric and collect additional representative cases, while retaining the disagreement rather than revising labels to improve the score. ### Scoring - 2 points: Defines a reproducible forecast event - 3 points: Calculates losses and an honestly specified baseline - 3 points: Distinguishes calibration from discrimination and label validity - 2 points: Reports sampling and group-size limitations ### Applied lessons - [lesson-03](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-03) - [lesson-10](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-10) - [lesson-16](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-16) - [lesson-17](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-17) ### Sources [Gneiting and Raftery (2007), Strictly Proper Scoring Rules, Prediction, and Estimation](https://doi.org/10.1198/016214506000001437) JASA 102:359–378; Introduction and Section 2 explain proper and strictly proper scoring rules. Publisher description and author-hosted indexed paper inspected. The binary squared-loss derivation and four-case fixture are original instructional calculations, not reported results from the paper. Author copy: https://sites.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf ## Explain what passing tests can establish A verification report lists thousands of successful checks. Open two of them. If they test intermediate masks produced from the same input, they may provide valuable coverage while sharing nearly every cause of failure. Counting them as thousands of independent encounters with future user tasks would give the report a statistical meaning that its test design never supplied. ### Prerequisites - Probability multiplication and percentages - The claim ledger in lesson-04 ### Learning objectives - Separate invariant tests, regression tests and representative evaluation - Derive a zero-failure binomial bound - Recognize how repeated tuning on an evaluation set changes its role Offline worksheet: [Coverage lab](labs/01-coverage/WORKSHEET.md). [Practice-lab guide](labs/README.md). Offline worksheet: [Probability lab](labs/05-probability/WORKSHEET.md). [Practice-lab guide](labs/README.md). ### Choose the claim before choosing the count The WITNESS lesson reports 8,523 reproduced checks for the recovered mathematical kernel, including interval cases and intermediate masks. This supports a bounded statement about the tested kernel and fixtures. The source boundary explicitly excludes a general claim of complete PDF understanding or reliable performance on every real drawing. An invariant test checks a property that should hold for a defined class of inputs. A regression test checks that a known failure stays repaired. A representative evaluation estimates performance on a stated task population through a sampling procedure, and although a single test can serve more than one purpose, its recorded result supports only the claims justified by that procedure. A hand-built boundary case may be exceptionally good at revealing a defect while being unsuitable for estimating how often users encounter it. ### Derive the meaning of zero observed failures Assume n independent tasks drawn from the same fixed task distribution, each with a common failure probability p. One task succeeds with probability 1−p, so all n succeed with probability (1−p)^n. Independence and the fixed distribution are substantive assumptions; repeated copies of one file do not automatically satisfy them. For a one-sided 95% upper confidence bound after zero failures, solve (1−p_U)^n=0.05. Taking the nth root gives p_U=1−0.05^(1/n). The NIST exact-binomial reference explains confidence limits obtained by inverting binomial probabilities and the problems with normal approximations when counts are small. [F-P03] The confidence level describes the long-run coverage of the interval procedure under the model. It does not mean there is a 95% posterior probability that the fixed unknown p lies below this bound. A Bayesian probability statement would require a prior and a corresponding posterior calculation. Both approaches still depend on a defensible observation process. ### Check the task distribution and dependency structure Suppose the deployment population includes scanned tables, long appendices, revised drawings and plain text. Testing only short clean files changes the population. The estimated failure rate would concern those clean files even if the arithmetic was flawless. List input classes, selection rules and exclusions alongside the result. Near-duplicate tasks can fail together because they share a template, source, extraction path or ambiguity. Treating them as independent makes intervals artificially narrow. Grouped sampling or analysis may be appropriate, but first identify what groups were actually selected. A generic adjustment cannot recover information that the test set never contained. The same principle applies across revisions. A receipt for one extractor version cannot establish end-to-end performance after that extractor changes. Retain relevant hashes, configuration, labels and acceptance criteria so that the claim can be recomputed for the version being considered. ### Protect the assessment from repeated selection A team that tests a hundred prompt variants and reports the best score on the same cases has used those cases to select its method. The winning score is partly a result of that selection. Dwork and colleagues analyze adaptive data analysis and specialized mechanisms for holdout reuse; their guarantees belong to those mechanisms and assumptions. Calling an ordinary repeatedly queried set a holdout does not supply them. [F-P04] Keep known failures available for development, since hiding them would make repair harder. Separately govern the evidence intended to support a final generalization claim. Record who saw evaluation results, which revisions followed and whether labels influenced the final choice. A small honest assessment is more interpretable than a large score whose role changed without documentation. ### Worked example: Read a perfect score with its assumptions attached For 100 independent representative trials with zero failures, p_U=1−0.05^(1/100), approximately 0.02951. The fixed-model upper confidence bound is therefore about 2.95%, despite the observed failure fraction being zero. Substituting 8,523 into the same expression gives approximately 0.000351, or 0.0351%. That calculation is arithmetically valid and evidentially unjustified for the reported WITNESS count unless the checks truly meet the sampling assumptions. Intermediate masks, repeated fixtures and deterministic assertions must not quietly become independent deployment tasks. Now suppose a 100-case evaluation includes ninety copies of one clean document and ten distinct difficult documents. Its raw pass fraction is still a useful description of those 100 rows. It needs a separate argument before it describes a future user’s chance of success, and it cannot support the independence-based bound merely because the report contains 100 rows. ### Laboratory: Design three different suites Build or specify a test plan for a source-to-answer workflow, using synthetic documents rather than private production material. - Create a regression suite for known defects. - Create a boundary suite containing missing appendices, conflicting clauses and ambiguous terminology. - Specify a separate representative evaluation population, sampling unit and version freeze. - Write one allowed claim and one unsupported claim for each suite. - Derive the one-sided zero-failure upper bound from (1−p)^n=0.05, calculate it for 100 independent representative trials, and explain why a selected regression count cannot simply replace n. ### Model answer The regression suite can establish that particular known defects were not reproduced under its conditions. The boundary suite can expose failures around important constraints and document how the workflow responds to missing evidence. Neither supplies a population-rate estimate solely from its pass count. The representative evaluation must explain how tasks were selected, what counts as failure, which version was frozen and how dependencies are handled. For the illustrative 100 independent zero-failure trials, report the approximately 2.95% upper bound and its assumptions. Do not convert the result into a universal guarantee. If the team fixes a failure after inspecting the evaluation, identify that revision as informed by the evaluation and obtain appropriately independent evidence for the new claim. ### Scoring - 3 points: Separates the three test purposes and allowed claims - 2 points: Derives or correctly calculates the zero-failure bound - 3 points: Identifies sampling, dependency and revision assumptions - 2 points: Records the effect of adaptive evaluation feedback ### Applied lessons - [lesson-04](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-04) - [lesson-23](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-23) - [lesson-24](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-24) - [lesson-25](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-25) ### Sources [NIST, EXACT BINOMIAL confidence-limit reference](https://itl.nist.gov/div898/software/dataplot/refman2/auxillar/exacbino.htm) Description section explains exact confidence limits by inversion of binomial probabilities and small-count approximation limitations. Reference page inspected. Supports a fixed binomial model, not independence or representativeness of arbitrary software checks. The zero-failure calculation is explicitly derived in the chapter. [Dwork et al. (2015), Generalization in Adaptive Data Analysis and Holdout Reuse](https://proceedings.neurips.cc/paper_files/paper/2015/file/bad5f33780c42f2588878a9d07405083-Paper.pdf) Introduction and holdout-reuse construction address adaptive use of evaluation feedback. Primary proceedings abstract and indexed paper inspected. Specialized formal guarantees do not apply to an ordinary repeatedly reused holdout merely because it is called a holdout. ## Test whether an intervention caused an improvement The new search tool completes 82% of its test tasks. The old tool completed 50%. Before crediting the new tool, inspect the questions in each test. If the second set contains many more easy requests, the improvement in the chart may describe the changed workload rather than anything the search tool did. ### Prerequisites - Percentages and weighted averages - Task completion as an outcome in lesson-01 ### Learning objectives - Distinguish an observed difference from a causal effect - Use standardization to compare systems under the same task mix - Specify an experiment and the assumptions needed for a causal interpretation Offline worksheet: [Probability lab](labs/05-probability/WORKSHEET.md). [Practice-lab guide](labs/README.md). ### Specify the alternative outcomes A causal question asks what would happen under alternative interventions. For task i, let Y_i(1) be completion under the new tool and Y_i(0) completion under the old tool, using 1 for success and 0 for failure. The individual effect is Y_i(1)−Y_i(0), but a task attempt ordinarily reveals only the outcome under the tool actually used. Hernán and Robins develop this counterfactual framework in Causal Inference: What If. Their identification conditions include consistency, exchangeability and positivity. Consistency links observed outcomes to the specified intervention; exchangeability concerns comparability of the groups; positivity requires the relevant alternatives to occur within the population groups being compared. These are conditions for an argument, rather than achievements conferred by fitting a regression. [F-P05] ### Separate source claims from proposed evaluations The Atlas’s findability work connects search observations with retrieval tuning, navigation, missing-content repair and course planning. It proposes outcome-linked revision rather than documenting a measured improvement for every intervention. The client-segmentation research includes an evaluation proposal but supplies no recovered measurement of campaign lift or causal attribution. Keep those boundaries when designing a new study. An external report that a technique worked elsewhere can motivate a local test. It cannot establish that the local organization adopted the technique or benefited from it. A final proposal should preserve explicit negative evidence about current use and keep planned outcomes separate from observed outcomes. Before collecting data, write the intervention, comparison, task population, unit of assignment, outcome and observation period. “Better search” is too broad for an experiment: changes to ranking, aliases and source coverage could have different effects and maintenance costs. Specify which change the result will describe. ### Compare like task groups before combining them Standardization compares systems under a common distribution of task characteristics. For easy and hard tasks, choose weights w_easy and w_hard that sum to one. A standardized success rate is w_easy × success_easy + w_hard × success_hard. The weights should describe the target workload or another declared comparison population, rather than being selected because they favor a preferred tool. Stratifying by difficulty can remove the case-mix difference in the worked example. In a real study, it does not automatically remove every source of confounding. Experienced users may choose the new tool more often, or difficult questions may be sent to a separate review team. State which variables were observed and why adjustment for them is expected to address the comparison problem. More data can narrow sampling uncertainty while leaving systematic bias unchanged. A precisely estimated difference between unlike groups remains a difference between unlike groups. ### Design assignment and measurement together Random assignment can make tool groups comparable in expectation. Assign suitable task instances to the old or new tool before outcomes are known, and record completion, wrong-result selection, abandonment and review effort under the same rubric. If the same person repeats the same question, the first attempt may teach the answer, so balance the order or use equivalent task instances and explain how any remaining learning effect limits the comparison between tools. Randomization alone does not guarantee an unbiased recorded result. A tool may cause users to abandon tasks, making it misleading to remove abandoned sessions from the denominator. Reviewers who know which tool produced an answer may score it differently. Shared advice between participants can also create interference between assignments. Choose the assignment level and review procedure with these effects in mind. For observational comparisons, a causal argument may still be possible, but it needs explicit assumptions and sensitivity to their failure. The appropriate conclusion is often an observed association with named limitations, especially when the records do not identify why users received different tools. ### Worked example: Remove an apparent gain caused by task mix All counts are synthetic. The old tool succeeds on 90 of 100 easy tasks and 10 of 100 hard tasks, for 100 successes in 200 attempts, or 50% overall. The new tool succeeds on 162 of 180 easy tasks and 2 of 20 hard tasks, for 164 successes in 200 attempts, or 82% overall. Within each difficulty group the tools have identical rates: 90% for easy tasks and 10% for hard tasks. Under a common 50:50 workload, each standardized rate is 0.5×0.9+0.5×0.1=0.5. The apparent 32-percentage-point gain disappears. Under an 80:20 workload both would have a standardized rate of 0.8×0.9+0.2×0.1=0.74. The arithmetic demonstrates how task mix creates the observed difference in this fixture. It does not show why a real workload changed, and it does not prove that difficulty is the only variable needed for a real causal comparison. ### Laboratory: Write an intervention study that could reject the proposal Prepare a pilot for an alias-and-navigation change intended to help users find approved sources. - Define the target task population and a completion rubric. - Specify assignment to the existing and revised routes, including how repeated-user learning is handled. - Retain abandoned tasks and wrong-source selections in the outcome record. - Calculate both raw and 50:50 standardized rates for the fixture and state the strongest supported conclusion. ### Model answer A satisfactory pilot names the alias/navigation change, freezes the source collection where feasible, and uses a common task outcome rubric. It randomizes appropriate task instances or participants while accounting for learning and communication between them. It records review effort and abandonment rather than treating a shorter interaction as automatic success. For the fixture, the raw difference is 82%−50%=32 percentage points; both standardized rates are 50%, so the standardized difference is zero. The supported explanation is that the entire raw difference is accounted for by the supplied easy/hard mix and identical within-group rates. For a new real study, no improvement has been established until outcomes are collected. The protocol should specify a result that would reject the intervention, such as unchanged completion together with increased wrong-source selections or review burden. ### Scoring - 2 points: Defines intervention, population and outcome precisely - 3 points: Calculates the raw and standardized contrasts - 3 points: Addresses assignment, learning, missing outcomes and interference - 2 points: Preserves the proposal-versus-result evidence boundary ### Applied lessons - [lesson-01](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-01) - [lesson-07](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-07) - [lesson-08](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-08) - [lesson-28](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-28) ### Sources [Hernán and Robins (2020 citation), Causal Inference: What If, electronic edition 19 August 2026](https://miguelhernan.org/s/hernanrobins_WhatIf_19aug26.pdf) Sections 1.5 p.9, 2.3 p.19, 3.1–3.5 pp.27–38, 3.6 p.41; Chapters 7–9 address confounding, selection and measurement. Current author link, downloaded edition date and detailed contents inspected; indexed author material on identification conditions checked. The easy/hard task counts are synthetic. Stable author landing page: https://miguelhernan.org/whatifbook ## Choose among permitted actions under uncertainty A library-service assistant has enough information to draft an answer but still needs to choose whether to answer, inspect another approved source or request human review. The same chance of error can support different choices when one mistake costs a minute and another sends an incorrect request to an external recipient. Specify the consequences and the allowed actions before setting a confidence threshold. ### Prerequisites - Expected value as a probability-weighted average - Hard constraints in lesson-05 and lesson-16 ### Learning objectives - Calculate an action threshold from a declared loss model - Compute the value of information from the decisions it changes - Separate constrained optimization from permission and subjective value judgments ### Keep outcomes, preferences and permission separate A decision model lists available actions, possible outcomes, probabilities and the value or loss attached to each outcome. Expected loss is the probability-weighted average of losses. The decision-analysis formulation in the Stan User’s Guide explains these elements separately and chooses the action with lowest expected loss, equivalently highest expected utility. The mathematics is platform-independent. The utility scale and outcome model remain stated assumptions. [F-P06] For the Atlas service-routing scenario, first remove actions that are unavailable or unauthorized, including a paid-standard request that lacks a verified identifier, a configured destination or the human approval required for the particular external action. A favorable numerical score cannot supply any of these. If a preservation requirement is non-tradeable, treat it as a condition of eligibility rather than a penalty that a large benefit can outweigh. Expected value also reflects a chosen treatment of risk. Equal average losses can conceal different worst cases. Where policy imposes a hard safety condition, retain that condition explicitly instead of assuming that an average adequately represents it. ### Derive a review threshold Let p be the probability that an answer is correct under an agreed rubric. In a synthetic loss model, automatic release costs zero if correct and 100 units if wrong. Review costs 4 units and initially is assumed to remove every error. Automatic expected loss is 100(1−p); review loss is 4. Automatic release has lower expected loss when 100(1−p)<4, giving p>0.96. The perfect-review assumption is deliberately strong. If review leaves residual error probability r, its expected loss becomes 4+100r, and the threshold becomes p>0.96−r. If r depends on the case, retain that dependence. Do not insert a single average review error rate where difficult cases are systematically routed to review. A probability used near this threshold needs relevant evaluation evidence. The judged confidence weight in the Atlas feature-value fixture is not such a calibrated probability. Without a defensible p, the threshold calculation remains a model of the decision rather than a validated release procedure. ### Ask whether more information can change the choice Suppose a proposed intervention yields benefit 50 if suitable and loss 30 otherwise, with suitability probability 0.5. Acting now has expected value 0.5×50−0.5×30=10; doing nothing has value zero. A perfect test would permit action only when suitable, producing expected value 25 before test cost. The expected value of perfect information is therefore 25−10=15. A test costing 4 would leave value 21 if it were perfect. An imperfect test requires a calculation for each possible result, the revised probabilities and the action selected afterward. Reducing uncertainty without changing any available decision may have no immediate decision value. An unavailable or unauthorized action also cannot contribute benefit to the comparison. The calculation describes this fixture’s agreed units and probabilities, leaving the team responsible for explaining whose interests the utility scale represents and how the task contract handles a possible irreversible harm that cannot be traded against routine benefits. Those decisions must remain visible in the task contract. ### Identify the mathematical problem you actually have Boyd and Vandenberghe show that a local minimum of a convex optimization problem is global. Convexity means the objective and feasible set have a specific mathematical structure; the result does not imply uniqueness or apply automatically to prompt search and feature selection. Their Section 4.2.2 supplies the bounded theorem. [F-P07] For a small set of indivisible projects, enumerate feasible combinations directly. This is often easier to inspect than a complicated scoring procedure. Include upkeep over a declared period and keep units compatible. Hours cannot be subtracted from an ordinal benefit score without a stated conversion or policy. When plausible changes to an estimate reverse the selected project, report the decision as sensitive and identify the observation that could resolve the uncertainty, because adding decimal places to the estimate will not supply the missing evidence. Consider a reversible trial whose result could distinguish the alternatives, provided its cost and permissions are included. ### Worked example: Allocate a review budget with a preservation constraint A synthetic budget permits ten review-hours. Proposal A requires six hours and offers nine expected benefit units. B requires five hours and offers eight; C requires four hours and offers six. Feasible pairs are A+C, costing ten and yielding fifteen, and B+C, costing nine and yielding fourteen. A+B costs eleven and is ineligible. Under these estimates A+C is best. If A fails a required preservation check, remove A and A+C from consideration. B+C is then best among the remaining feasible choices. If A instead remains eligible but its benefit estimate falls from nine to seven, A+C yields thirteen and B+C yields fourteen. The ranking depends on the estimate. The enumeration proves the best combination within this finite fixture and declared objective. It establishes neither the accuracy of the benefit estimates nor the appropriateness of the utility scale. ### Laboratory: Write a reviewable decision note Use the service-answer loss model and the ten-hour allocation fixture. - Calculate the automatic-release threshold with perfect review and with residual error r=0.01. - Explain the effect of missing external-release permission. - Enumerate all feasible subsets of A, B and C. - Calculate the value of a perfect suitability test and state what an imperfect test would require. ### Model answer The perfect-review threshold is p>0.96. With residual review error 0.01, review loss is 5 and the threshold becomes p>0.95. These comparisons apply only among authorized actions and within the declared loss model. Missing release permission excludes automatic release regardless of p. Feasible subsets are none, A, B, C, A+C and B+C. Their benefits are zero, nine, eight, six, fifteen and fourteen. The best eligible combination is A+C until A fails its hard gate or its estimate changes as specified. The perfect suitability test has value fifteen before its cost; an imperfect test needs sensitivity, specificity, outcome probabilities and the decisions taken after each result. A strong note also names the person who approved the benefit scale and the owner of future maintenance. ### Scoring - 3 points: Calculates thresholds and information value correctly - 3 points: Keeps authorization and preservation outside compensable scores - 2 points: Enumerates feasible combinations accurately - 2 points: Discloses estimate sensitivity and value judgments ### Applied lessons - [lesson-05](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-05) - [lesson-16](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-16) - [lesson-18](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-18) - [lesson-26](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-26) - [lesson-31](https://recluse.studio/research/agentic-engineering-atlas/#learn=lesson-31) ### Sources [Stan User’s Guide, Decision Analysis](https://mc-stan.org/docs/stan-users-guide/decision-analysis.html) Outline of decision analysis separates decisions, outcomes, distributions and utilities before expected-utility optimization. Complete current page inspected. Used for the mathematical formulation, not software-specific instructions or a claim that its model grants action authority. [Boyd and Vandenberghe (2004), Convex Optimization](https://www.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf) Sections 4.1–4.2; Section 4.2.2, print p.138, establishes local-to-global optimality for convex problems. Official author book page and PDF passage inspected; downloaded copy retained for verification. The finite feature-choice example is enumeration, not a convex-optimization application. ## Control revisions when feedback is noisy or delayed A team changes search aliases after every batch of failed queries. Some users improve immediately; others are still working from cached pages, and their outcomes arrive a day later. If the team treats every delayed failure as evidence about the latest revision, it can reverse a useful change before the change has been properly observed. ### Prerequisites - Signed numbers, multiplication and geometric sequences - Observation versus state in probability-01 - Advanced stability derivation only: quadratic equations and complex roots. The numerical trace can be completed without this extension. ### Learning objectives - Describe the parts of a feedback process - Derive stability for a simple discrete update - Explain how observation delay changes the model and its conclusions Offline worksheet: [Probability lab](labs/05-probability/WORKSHEET.md). [Practice-lab guide](labs/README.md). ### Name what is measured and what is changed Feedback uses observed results to choose later actions. For a maintained knowledge collection, the action might revise aliases, navigation or source content. The measured result should concern an agreed task outcome, such as finding an applicable approved source. A query count is only an intermediate measurement, because fewer queries can accompany either successful completion or abandonment. Classical control theory studies the combined behavior of a process, observations and a controller. Åström and Murray’s Feedback Systems develops state models, dynamic behavior and robustness. The indexed public manuscript includes discrete-time dynamics in Chapter 3. Its full text could not be retrieved for this edition check, so the calculations below are derived here rather than presented as a quotation or a theorem about a particular agent. [F-P09] The Atlas query-feedback and curation methods can be analyzed using these distinctions. That comparison does not establish that their dynamics are linear or that a numerical control rule has been implemented. Before applying a fitted controller, a team would need evidence about response times, changing task distributions and what its measurements actually detect. ### Solve a simple update completely Let e_t be the signed gap between a desired value and the observed value at step t. Assume an update removes fraction k of the current gap, with no delay, noise or changing target. The resulting recurrence is e_(t+1)=(1−k)e_t. Repeated substitution gives e_t=(1−k)^t e_0. The equilibrium e=0 is asymptotically stable for all initial errors exactly when |1−k|<1, or 02, its magnitude grows. These are exact properties of the recurrence. They do not recommend a gain for a real knowledge-maintenance process whose assumptions have not been checked. ### Advanced derivation / Add a one-step observation delay Suppose the update uses the previous gap instead of the current one. The equation becomes e_(t+1)=e_t−k e_(t−1). It now needs two starting values. Guess a solution proportional to λ^t and substitute it into the equation; after division by λ^(t−1) where appropriate, the characteristic equation is λ²−λ+k=0. For 01 it exceeds one, and for k≤0 at least one root has magnitude at least one. Thus the delayed model is asymptotically stable for all initial conditions exactly when 0