{
  "schemaVersion": "1.0",
  "type": "research-article",
  "runtimeAvailable": false,
  "id": "proof21:journal:audit-the-batch-before-the-sample:en",
  "inLanguage": "en",
  "slug": "audit-the-batch-before-the-sample",
  "title": "Audit the batch before the sample.",
  "description": "A reproducible sample can still miss an omitted population, a weak evaluation, or a poorly chosen statistical policy.",
  "datePublished": "2026-09-07",
  "dateModified": "2026-09-07",
  "url": "https://proof21.xyz/journal/audit-the-batch-before-the-sample/",
  "markdownUrl": "https://proof21.xyz/journal/audit-the-batch-before-the-sample/article.md",
  "markdown": "# Audit the batch before the sample.\n\n**Research note · 7 September 2026 · Sampling workflows are proposed, not a released audit service.**\n\nA marketplace has more completed jobs than its reviewers can inspect. Sampling looks like an obvious way to reduce work: commit a batch, select some jobs, evaluate them, and publish a receipt. But an operator could omit its worst jobs before committing the batch. A mathematically perfect sample of the remaining records would never select the missing jobs.\n\nThis is why Proof21's Sample research separates population completeness, selection, evaluation, and acceptance. Each layer needs its own evidence. A portable artifact should make those boundaries inspectable rather than compress them into a broad statement that an entire marketplace has been “audited.”\n\n## Define the population before the lottery\n\nStart by naming the unit being sampled. Is it a task, an invoice, a model response, a customer session, or a group of records? Define the time window, inclusion and exclusion rules, stable identifiers, duplicates policy, and the point at which the population closes. Otherwise, two parties can use the word “batch” while referring to different collections.\n\nFor a proposed pilot, reconcile the declared batch against an independently maintained intake or completion ledger. Record counts and digests, but also record the limits of that ledger. A ledger controlled by the same operator may help detect accidental omissions without independently proving that deliberately hidden work never existed. Do not turn a reconciliation convenience into an absolute completeness claim.\n\nMissing selected records should remain visible. Replacing an unavailable item with the next convenient item changes the sampling procedure. Define that failure path before selection and preserve the original selection record. Any permitted replacement should be an explicit policy action, not a silent repair by an agent.\n\n## Commitments protect a set, not its relevance\n\nA Merkle commitment can bind a particular collection, and an inclusion proof can show that a record belongs to that committed structure under the chosen construction. Certificate Transparency's RFC 9162 is a primary example of carefully specified Merkle inclusion and consistency mechanisms. It does not make an arbitrary application batch complete, nor does citing it make Proof21 a Certificate Transparency implementation. [1]\n\nThe proposed report should retain the construction version, leaf encoding, leaf count, root, selected positions, and proof material. Order and duplicate handling matter. A consumer needs to know whether the root represents a sequence, a set, or some other explicitly defined structure. “It has a hash” is not enough to reproduce the question asked.\n\nThe commitment also needs an ordering boundary relative to the source used for selection. If the operator can see the selection source and then build a favorable batch, committing the result afterward does not rescue the procedure. The companion article on [choice without rerolls](https://proof21.xyz/journal/choice-without-rerolls/) explains this lifecycle requirement.\n\n## Put a number on a limited question\n\nConsider an illustrative fixed population of 1,000 records containing exactly 20 defective records. Select 100 distinct records uniformly without replacement, and assume the evaluation detects every defect in a selected record. The probability of detecting at least one defect is:\n\n```text\n1 - C(980, 100) / C(1000, 100) = 0.8809980814752082\n```\n\nHere `C(n, k)` counts combinations. The calculation is derived for this example, not measured Proof21 performance. Even with those favorable assumptions, the chance of missing all 20 defects is about 11.90%. If the actual defect count is unknown, this calculation does not magically reveal it. If evaluation misses defects or selection is not uniform, the assumptions no longer hold.\n\nNIST's acceptance-sampling guidance distinguishes a defined sampling plan and lot decision from a general guarantee of quality. Sample size, decision thresholds, and acceptable risks belong to that plan; “we sampled ten percent” is not a complete policy. [2][3] Grouping related jobs can also change what the sample teaches: selecting one cluster of similar records is not the same design as independently selecting individual records across the population.\n\n## Evaluate the reviewer as well as the selection\n\nA faithfully chosen reviewer can still make mistakes, have a conflict, or apply the wrong evaluation rubric. Preserve the rubric version, required evidence, reviewer identity or authorized role, and the result with reasons. Where the claim is deterministic, an independent consumer should reproduce the supported check. Where it involves human judgment, the artifact should say so.\n\nA pilot can use known synthetic defects to measure detection behavior and disagreement. Keep those tests separate from real customer observations. Do not claim an operational fraud rate from a fixture set whose defect rate was chosen by the test author. A reviewer who sees the answer key is not an independent evaluation, even if their selection was random.\n\nEscalation rules should be predeclared: what happens after one finding, an unresolved item, or a pattern of disagreement? Additional sampling can be legitimate under a specified design, but repeatedly drawing until a clean sample appears is a different and misleading practice. Retain unsuccessful and incomplete attempts rather than publish only the pleasant ending.\n\n## A report that helps a consumer decide\n\nThe useful output is a chain of scoped statements: this population was declared; these completeness checks were available; this selection reproduces; these selected items were evaluated under this rubric; these findings remain unresolved. A consumer then decides whether its own acceptance requirements are satisfied.\n\nThat workflow can reduce repeated evidence gathering without promising certainty about all unsampled work. Proof21's proposed role is to make the procedure portable and replayable. It is not to replace independent auditing expertise, certify an entire business from a small sample, or authorize financial actions by printing a favorable sampling result.\n\n## Frequently asked questions\n\n### Does a Merkle root prove every eligible job was included?\n\nNo. It binds the collection used to construct it under specific encoding and hashing rules. Evidence about completeness relative to the intended population is separate.\n\n### Does a clean sample prove the entire batch is defect-free?\n\nNo. A clean sample is an observation under a sampling design. Its interpretation depends on the population, selection procedure, evaluation accuracy, and chosen statistical policy, not merely the sample's appearance.\n\n### Can the operator replace a selected record that is unavailable?\n\nOnly under an explicitly defined and authorized policy that records the original selection and replacement. A missing item should not disappear from the evidence. Otherwise the operator can bias what is reviewed.\n\n## Primary references and next reading\n\n- [1: RFC 9162 — Certificate Transparency Version 2.0](https://www.rfc-editor.org/rfc/rfc9162.html)\n- [2: NIST — What is Acceptance Sampling?](https://www.itl.nist.gov/div898/handbook/pmc/section2/pmc21.htm)\n- [3: NIST — Choosing a Single Sampling Plan](https://www.itl.nist.gov/div898/handbook/pmc/section2/pmc23.htm)\n- [Proof21 Choice and Sample](https://proof21.xyz/docs/capabilities/choice/)\n",
  "articleBody": "Research note · 7 September 2026 · Sampling workflows are proposed, not a released audit service. A marketplace has more completed jobs than its reviewers can inspect. Sampling looks like an obvious way to reduce work: commit a batch, select some jobs, evaluate them, and publish a receipt. But an operator could omit its worst jobs before committing the batch. A mathematically perfect sample of the remaining records would never select the missing jobs. This is why Proof21's Sample research separates population completeness, selection, evaluation, and acceptance. Each layer needs its own evidence. A portable artifact should make those boundaries inspectable rather than compress them into a broad statement that an entire marketplace has been “audited.” Define the population before the lottery Start by naming the unit being sampled. Is it a task, an invoice, a model response, a customer session, or a group of records? Define the time window, inclusion and exclusion rules, stable identifiers, duplicates policy, and the point at which the population closes. Otherwise, two parties can use the word “batch” while referring to different collections. For a proposed pilot, reconcile the declared batch against an independently maintained intake or completion ledger. Record counts and digests, but also record the limits of that ledger. A ledger controlled by the same operator may help detect accidental omissions without independently proving that deliberately hidden work never existed. Do not turn a reconciliation convenience into an absolute completeness claim. Missing selected records should remain visible. Replacing an unavailable item with the next convenient item changes the sampling procedure. Define that failure path before selection and preserve the original selection record. Any permitted replacement should be an explicit policy action, not a silent repair by an agent. Commitments protect a set, not its relevance A Merkle commitment can bind a particular collection, and an inclusion proof can show that a record belongs to that committed structure under the chosen construction. Certificate Transparency's RFC 9162 is a primary example of carefully specified Merkle inclusion and consistency mechanisms. It does not make an arbitrary application batch complete, nor does citing it make Proof21 a Certificate Transparency implementation. [1] The proposed report should retain the construction version, leaf encoding, leaf count, root, selected positions, and proof material. Order and duplicate handling matter. A consumer needs to know whether the root represents a sequence, a set, or some other explicitly defined structure. “It has a hash” is not enough to reproduce the question asked. The commitment also needs an ordering boundary relative to the source used for selection. If the operator can see the selection source and then build a favorable batch, committing the result afterward does not rescue the procedure. The companion article on choice without rerolls explains this lifecycle requirement. Put a number on a limited question Consider an illustrative fixed population of 1,000 records containing exactly 20 defective records. Select 100 distinct records uniformly without replacement, and assume the evaluation detects every defect in a selected record. The probability of detecting at least one defect is: 1 - C(980, 100) / C(1000, 100) = 0.8809980814752082 Here C(n, k) counts combinations. The calculation is derived for this example, not measured Proof21 performance. Even with those favorable assumptions, the chance of missing all 20 defects is about 11.90%. If the actual defect count is unknown, this calculation does not magically reveal it. If evaluation misses defects or selection is not uniform, the assumptions no longer hold. NIST's acceptance-sampling guidance distinguishes a defined sampling plan and lot decision from a general guarantee of quality. Sample size, decision thresholds, and acceptable risks belong to that plan; “we sampled ten percent” is not a complete policy. [2][3] Grouping related jobs can also change what the sample teaches: selecting one cluster of similar records is not the same design as independently selecting individual records across the population. Evaluate the reviewer as well as the selection A faithfully chosen reviewer can still make mistakes, have a conflict, or apply the wrong evaluation rubric. Preserve the rubric version, required evidence, reviewer identity or authorized role, and the result with reasons. Where the claim is deterministic, an independent consumer should reproduce the supported check. Where it involves human judgment, the artifact should say so. A pilot can use known synthetic defects to measure detection behavior and disagreement. Keep those tests separate from real customer observations. Do not claim an operational fraud rate from a fixture set whose defect rate was chosen by the test author. A reviewer who sees the answer key is not an independent evaluation, even if their selection was random. Escalation rules should be predeclared: what happens after one finding, an unresolved item, or a pattern of disagreement? Additional sampling can be legitimate under a specified design, but repeatedly drawing until a clean sample appears is a different and misleading practice. Retain unsuccessful and incomplete attempts rather than publish only the pleasant ending. A report that helps a consumer decide The useful output is a chain of scoped statements: this population was declared; these completeness checks were available; this selection reproduces; these selected items were evaluated under this rubric; these findings remain unresolved. A consumer then decides whether its own acceptance requirements are satisfied. That workflow can reduce repeated evidence gathering without promising certainty about all unsampled work. Proof21's proposed role is to make the procedure portable and replayable. It is not to replace independent auditing expertise, certify an entire business from a small sample, or authorize financial actions by printing a favorable sampling result. Frequently asked questions Does a Merkle root prove every eligible job was included? No. It binds the collection used to construct it under specific encoding and hashing rules. Evidence about completeness relative to the intended population is separate. Does a clean sample prove the entire batch is defect-free? No. A clean sample is an observation under a sampling design. Its interpretation depends on the population, selection procedure, evaluation accuracy, and chosen statistical policy, not merely the sample's appearance. Can the operator replace a selected record that is unavailable? Only under an explicitly defined and authorized policy that records the original selection and replacement. A missing item should not disappear from the evidence. Otherwise the operator can bias what is reviewed. Primary references and next reading 1: RFC 9162 — Certificate Transparency Version 2.0 2: NIST — What is Acceptance Sampling? 3: NIST — Choosing a Single Sampling Plan Proof21 Choice and Sample",
  "citations": [
    "https://www.rfc-editor.org/rfc/rfc9162.html",
    "https://www.itl.nist.gov/div898/handbook/pmc/section2/pmc21.htm",
    "https://www.itl.nist.gov/div898/handbook/pmc/section2/pmc23.htm"
  ],
  "faq": [
    {
      "question": "Does a Merkle root prove every eligible job was included?",
      "answer": "No. It binds the collection used to construct it under specific encoding and hashing rules. Evidence about completeness relative to the intended population is separate."
    },
    {
      "question": "Does a clean sample prove the entire batch is defect-free?",
      "answer": "No. A clean sample is an observation under a sampling design. Its interpretation depends on the population, selection procedure, evaluation accuracy, and chosen statistical policy, not merely the sample's appearance."
    },
    {
      "question": "Can the operator replace a selected record that is unavailable?",
      "answer": "Only under an explicitly defined and authorized policy that records the original selection and replacement. A missing item should not disappear from the evidence. Otherwise the operator can bias what is reviewed."
    }
  ],
  "sourceSha256": "89bfa2dace266f680ae27c7be4ab5b087e2e045f79bc97fc30f68e70bb4a1025",
  "markdownSha256": "17eba3b9b490bd15efe5d51ac6e3a96bd771e473ca3e5549805a795996fa4c1f",
  "translationReview": "Original English edition; not an independent technical audit.",
  "image": {
    "url": "https://proof21.xyz/assets/journal/audit.png",
    "caption": "Selected cells stand out from a larger grid. The illustration shows a sample inside a declared batch, not proof that the batch contains every eligible record.",
    "sha256": "a1af0c208529ed83888996953f9440de44911adcb5b69ce4401c5c3aacc58d4a"
  },
  "motion": {
    "url": "https://proof21.xyz/assets/journal/audit.mp4",
    "engine": "Remotion",
    "sourceSha256": "13a909f4267d8147386045cbd90df732a42679342c16e04c5725e09e953d3728",
    "sha256": "f41c56f2ac90236ca6cd9ca8f80b16f6f461a474309a852816e334b28e77855b",
    "seconds": 7.2,
    "loop": true,
    "audio": false
  },
  "availableLanguages": [
    "en",
    "zh-Hans",
    "th",
    "ar"
  ],
  "editions": {
    "en": "https://proof21.xyz/journal/audit-the-batch-before-the-sample/",
    "zh-Hans": "https://proof21.xyz/zh-hans/journal/audit-the-batch-before-the-sample/",
    "th": "https://proof21.xyz/th/journal/audit-the-batch-before-the-sample/",
    "ar": "https://proof21.xyz/ar/journal/audit-the-batch-before-the-sample/"
  }
}