Restore full model card (LOC1, links) with search tags

README.md

@@ -6,6 +6,11 @@ tags:
 - code-generation
 - self-improvement
 - software-engineering
+- ai-agents
+- program-repair
+- autonomous-agents
+- llm
+- open-source
 ---
 
 # Runesmith 1.0
@@ -16,11 +21,14 @@ This card is written from the paper, *Beyond the Model* (see "How to cite"), and
 
 **What is and is not shown, in short.**
 
-- Runesmith improved on its own previous generation under a sealed, preregistered test: a generation of its repair step that its own Kaizen loop produced repaired 57 of 162 sessions against its predecessor's 35 of 162, with the same cheap repair model, on 54 fresh tasks from two repositories that supplied no task to the organ's development (exact p = 0.000845; within the larger repository alone p = 0.0020), in about half the time per success. The predecessor, an earlier generation from the same loop, had not beaten the shipped one, and the new generation has not yet been compared with the shipped one in a sealed test.
+- Runesmith changes its own code only on evidence, and it has passed a sealed test of doing so. It improved on its own previous generation under a sealed, preregistered test: a generation of its repair step that its own Kaizen loop produced repaired 57 of 162 sessions against its predecessor's 35 of 162, with the same cheap repair model, on 54 fresh tasks from two repositories that supplied no task to the organ's development (exact p = 0.000845; within the larger repository alone p = 0.0020), in about half the time per success. The predecessor, an earlier generation from the same loop, had not beaten the shipped one, and the new generation was then compared with the shipped one in two later sealed studies: in a smaller one on small public repositories (U1) it led both the shipped generation and its predecessor in direction, without a significant difference, and in LOC1 (below), on three large public repositories, it repaired more than the shipped one.
+- LOC1 (sealed 2026-10-06), the sealed test of that new generation (C7) against the shipped one (g0): on 56 fresh tasks in three large public repositories (pygments, sqlglot and astroid), with one free repair model (gpt-oss-20b) and one envelope for every arm, C7 repaired 42 of 168 sessions and g0 17 of 168 (exact one-sided p = 5835/2097152 = 0.00278, below its Bonferroni level of 0.0083 over its three tests; verdict PASS). A hand-written, trace-aware rule, HEUR, written by the study's designer after C7 existed, repaired 65 of 168, more than C7 (mirrored p = 101/131072 = 0.00077; verdict HEUR_BETTER): a gain that the loop found by itself, a careful designer can also write. A replication on a second model (gemini-3.5-flash) completed one task and could detect no difference (NO_DIFFERENCE_DETECTABLE). The protocol was sealed at 2026-10-06T10:41:04Z and the wording of the result at 11:34:30Z, both before any outcome, and their fingerprints were anchored in Bitcoin blocks 970170 and 970174 (OpenTimestamps) hours before the analysis (16:36:53Z). With SR7 this is a sealed two-step record, and SR7's p-value survives Bonferroni correction across all 23 sealed or registered comparisons the program has run.
 - A second self-improvement (faster memory retrieval, authored by a free model) was replicated on a fresh cohort; a frozen generation repaired its own test configuration; the predecessor kernel promoted target-hidden corrections on revisions of a production service and supervised that service's operations across restarts, continuing without observer intervention after its start.
-- Free-tier models were the last author of every applied change to Runesmith Motion, whose engine (at 52 milestones) rendered the graphics of the launch film, and which ran unattended by the project lead's definition on two occasions; in the acceptance test, run by an AI in a newcomer's seat, one free Google key took a project to a checked, working feature in 16 minutes.
-- Six of the seven sealed tests in the study series (the paper's "SR" studies, SR1 to SR7, each a preregistered test of a change to Runesmith's own machinery) did not show their effect.
-- Not yet measured: whether Runesmith makes a given model better than that model is on its own (no study compares the same model with and without Runesmith); whether its competence survives a named change of model; learning across users (release 1 does not do it).
+- Free-tier models were the last author of every applied change to Runesmith Motion, whose engine (`motion.mjs`, at 52 milestones) computed the graphics of every frame of the launch film, and which ran unattended by the project lead's definition on two occasions; in the acceptance test, run by an AI in a newcomer's seat, one free Google key took a project to a checked, working feature in 16 minutes. On the build just before 1.0.0, with free models as both repair model and improver, Runesmith's scheduled repair rounds repaired 69 of 89 single-line regressions injected one at a time into a project with a real test suite, each accepted by a held-out judge (one project, observational, no comparison arm).
+- Six of the seven sealed tests in the study series (the paper's "SR" studies, SR1 to SR7, each a preregistered test of a change to Runesmith's own machinery) did not show their effect; they shaped the loop that then passed (its held-out validation, and the affordance audit in its author's packet, came from them).
+- After that series, three sealed studies, U1 (2026-10-05), U2 and U2b (to 2026-10-06), compared the improved generation with SR6-W's fixed scaffold, which U1's protocol calls "the strongest fixed scaffold the record has defined", driving the same model in the same kernel (formally, "the same model with a simple fixed script in place of Runesmith's repair organ"), on fresh tasks from two public repositories. Across the six model-by-study comparisons, the improved generation was ahead in direction in three (2.6B model: 3 of 17 against 0 of 17; 20B: 33 against 23 of 60; 120B: 34 against 28 of 63), level in two (120B: 10 of 17 each; 2.6B: 9 of 51 each) and behind in one (U1, 20B: 20 against 26 of 51 sessions; per task 5 better, 6 worse, 6 tied, in a study the remaining credit cut to a size with 80% power only for an advantage of about 36 points). None is significant in either direction; this is a descriptive tally, not a registered analysis, and the six are not pooled. U1's replication attempt of SR7 (20 of 51 against the predecessor's 15 of 51) went in SR7's direction and was not significant either.
+- Starved of every model in a sealed study (SI, 2026-10-05: 104 opportunities across 13 states, with no model configured or with every model failing), no request reached a model, no change was unauthorized and every ledger chain stayed intact. The sealed harness counted 18 failures (17.3%, upper bound 24.6%): 15 effects recorded without a ledger event of their own (a check-autopilot failure counter, and own-log lines for the owner's one-more-try request and restart decision), and 3 made by a defect in the harness's own detector.
+- Not yet measured: whether Runesmith makes a given model better than that model is on its own (no study compares a model working alone with the same model inside Runesmith); whether its competence survives a named change of model; learning across users (release 1 does not do it).
 
 ## Model details
 
@@ -34,7 +42,7 @@ This card is written from the paper, *Beyond the Model* (see "How to cite"), and
 | Model type | A non-traditional model: a fixed policy plus persistent, evidence-bearing project state. No weights, nothing trained. |
 | Runs on | Python 3.11 or newer with the standard library only, and a web browser for the Studio. An earlier Core snapshot passed 88/88 tests on Windows 11 with Python 3.13, and 84 tests with 4 platform skips on Linux; macOS is not yet verified. |
 | Paper | *Beyond the Model: Runesmith, an Open Runtime That Improves Software and Itself, and the Instrument–Substrate Hypothesis*, 6 October 2026. Preprint; not peer reviewed. |
-| Links | Code: [repository](https://github.com/Veristria/runesmith). Paper: [PDF](https://aithinklab.com/paper/beyond-the-model.pdf). Guide: [guide](https://aithinklab.com/guide/). |
+| Links | Website: [aithinklab.com](https://aithinklab.com). Code: [repository](https://github.com/Veristria/runesmith). Paper: [PDF](https://aithinklab.com/paper/beyond-the-model.pdf), DOI [10.5281/zenodo.23196763](https://doi.org/10.5281/zenodo.23196763). Technical report: [PDF](https://aithinklab.com/paper/beyond-the-model-technical-report.pdf). Guide: [guide](https://aithinklab.com/guide/). Evidence and data: [aithinklab/runesmith-evidence](https://huggingface.co/datasets/aithinklab/runesmith-evidence) (session tables, recompute scripts, seal records). Film: [YouTube](https://youtu.be/cmb_zbqrKvA). |
 
 ## What kind of model this is
 
@@ -96,13 +104,13 @@ Runesmith runs on any OpenAI-compatible chat endpoint. The Studio has presets fo
 
 **Within a project.** State accumulates: the ledger, the environment and self maps, memory of what was tried, experience records, and the approved acceptance checks that judge every later build. In the paper's terms this is adaptation ("any history-dependent change in behavior"). It is not learning in the paper's stricter sense, which requires a versioned state change that improves a predeclared future outcome against a matched fixed-state baseline.
 
-**Its own repair step.** The starting generation of the repair organ is g0 (the repair step as shipped). C7 (the improved repair step that the sealed test SR7, described under "Evidence", compared with its predecessor B, an earlier improved generation) ships in a library with its caveat shown to you: it was proven against B, not against g0, on synthetic single-line bugs with one model (`gpt-oss-20b`), which of its steps carried the gain was not tested, and its module-path step recognises only the package names of the two repositories used in development and does not fire on yours. On your work it must win an online trial before it becomes active, or you may activate it by explicit choice, which ends any open trial and is recorded.
+**Its own repair step.** The starting generation of the repair organ is g0 (the repair step as shipped). C7 (the improved repair step that the sealed test SR7, described under "Evidence", compared with its predecessor B, an earlier improved generation) ships in a library with a caveat shown to you. The evidence: C7 was shown better than B in SR7 and better than g0 in LOC1 (in which a hand-written trace-aware rule did better than C7), on synthetic single-line bugs with one model (`gpt-oss-20b`); which of its steps carried the gain was not tested, and its module-path step recognises only the package names of the two repositories used in development and does not fire on yours. On your work it must win an online trial before it becomes active, or you may activate it by explicit choice, which ends any open trial and is recorded.
 
 **Across users.** Release 1 does not learn across users. No mechanism sends experience or organ changes between installations. The only path is a generation archive that one person exports and another imports by hand; an imported generation is frozen and inactive, confined like a local one, and becomes active only if it wins an online trial on the receiver's own work or the receiver explicitly chooses it.
 
 ## Evidence
 
-All figures are from the paper, which reports every sealed result. The p-values are nominal and one-sided under each study's declared design. No result has been rerun by a distinct investigator: the "recomputations" are second scripts written inside the project from the same raw receipts. The SR5 to SR7 studies (sealed, preregistered tests around Runesmith's own repair step) ran on the open-source Core; the object-improvement result, S1 (one self-correction of a generation's own configuration) and M1 (a model-authored memory-retrieval improvement, tested for speed) ran on the earlier kernel and subject machinery, which is a different code base. All were run by AI ThinkLab with AI collaborators in named roles.
+All figures are from the paper and its technical report, which together report every sealed result. The p-values are nominal and one-sided under each study's declared design. No result has been rerun by a distinct investigator: the "recomputations" are second scripts written inside the project from the same raw receipts. The SR5 to SR7 studies (sealed, preregistered tests around Runesmith's own repair step) ran on the open-source Core; the object-improvement result, S1 (one self-correction of a generation's own configuration) and M1 (a model-authored memory-retrieval improvement, tested for speed) ran on the earlier kernel and subject machinery, which is a different code base. All were run by AI ThinkLab with AI collaborators in named roles.
 
 ### The one sealed positive for the system's own change: SR7
 
@@ -122,19 +130,37 @@ A frozen successor generation of the repair organ (C7), written inside Runesmith
 
 Boundaries, all stated in the paper:
 
-1. The comparator is B, not g0 and not a conventional scaffold. In an earlier study (SR5), B's point estimate was below g0's (36/180 against 50/180), so C7 beating B does not show C7 beating g0.
+1. SR7's comparator is B, not g0 and not a conventional scaffold. In an earlier study (SR5), B's point estimate was below g0's (36/180 against 50/180), so SR7 alone does not show C7 beating g0; LOC1 (next section) later compared C7 with g0 directly, and U1, U2 and U2b compared it with a fixed scaffold (no significant difference in any of six comparisons).
 2. The author was a paid model (though below today's free-tier Gemini Flash models on a public coding benchmark). Free-class authors produced no transferring change in SR5 or SR6.
 3. One of the five accepted tiers (the module-path tier) was inert in the study, and no ablation shows which tiers carried the effect.
 4. Errors and false promotions were equal between the arms, not lower. The time figure is per strict success, not per repair.
-5. No replication, no second improving generation, no natural bugs, and no model-free localization arm. Fifty of the 54 tasks come from one repository (within it alone, p = 0.0020).
+5. SR7 itself had no replication, no second improving generation, no natural bugs and no hand-written localization comparator (LOC1, next section, is a later study, and in it a hand-written trace-aware rule repaired more than C7). Fifty of the 54 tasks come from one repository (within it alone, p = 0.0020).
 6. The target was chosen by a declared rule, but the Kaizen machinery was built by a consultant who knew earlier results (the paper calls this a hybrid lineage).
 7. The improvement is of the repair organ. The Kaizen loop, the target rule and the trial machinery did not change, so this is not an improvement of the improver. The paper calls it "a nudge towards recursive self-improvement": the system's own loop rewrote part of the system, a sealed test confirmed the new version better than the one it replaced, and the loop itself did not change.
 
 The paper's earned name for this result: a bounded subject-improving adaptive runtime within the SR7 envelope.
 
-### All sealed or registered confirmatory comparisons (paper, Table 1)
+### The second sealed test: C7 against the shipped organ (LOC1)
 
-Eleven comparisons: three passed (SR7, M1 confirmation 02 and M1 replication 03, which test the same frozen candidate on two cohorts), one failed a preregistered guard, seven did not show their effect or were null. Within the seven SR tests, one passed. The second column says what each row compared.
+SR7 compared C7 with its predecessor B, not with the organ Runesmith shipped with (g0), and the October studies (U1, U2, U2b) used repositories small enough that C7's file-ranking steps could not act. LOC1 was sealed on 2026-10-06 to close both gaps: fresh synthetic single-line regressions in three large public Python repositories (pygments, sqlglot, astroid), where the navigation index exceeds its 12,000-byte cap, one free repair model (gpt-oss-20b), three rounds per task and one envelope for every arm. Three arms ran: g0, C7 and HEUR, a hand-written trace-aware localization rule written by the study's designer after C7 existed.
+
+| | g0 (shipped) | C7 (self-improved) | HEUR (hand-written) |
+|---|---|---|---|
+| Strict successes (56 tasks × 3 rounds = 168 sessions per arm) | 17 of 168 | 42 of 168 | 65 of 168 |
+| Sessions with the faulty file in the model's edit view | 27 | 86 | 99 |
+
+- Primary test, C7 against g0: exact one-sided sign-flip p = 5835/2097152 = 0.00278 (mirrored 0.9988), below its Bonferroni level of 0.0083; at task level 15 better, 6 worse, 35 tied; verdict PASS. By repository, C7 against g0: 27 against 13 (pygments), 4 against 3 (astroid) and 11 against 1 (sqlglot), descriptive.
+- Secondary test, C7 against HEUR: p = 0.9998 (mirrored 101/131072 = 0.00077); at task level 2 better, 16 worse, 38 tied; verdict HEUR_BETTER. The hand-written rule repaired more than the self-improved organ, so LOC1 does not show C7 above a careful hand-written rule.
+- C7 took about half of g0's time per strict success (1,520 against 2,806 seconds).
+- Replication on gemini-3.5-flash (free tier): one task completed by the stop time, 3 of 3 sessions each; verdict NO_DIFFERENCE_DETECTABLE. It replicates nothing.
+- Sealed in two steps: the protocol at 2026-10-06T10:41:04Z, the wording of the result at 11:34:30Z, the first session at 12:05:45Z, the analysis at 16:36:53Z. The fingerprints were submitted to OpenTimestamps and anchored in Bitcoin block 970170 (12:12:09Z) and block 970174 (12:48:21Z). The evidence dataset ships the session table, the recomputation and the seal records.
+- Deviation: at 12:22Z, before any outcome was read, the gateway's concurrency cap for the study's calls was raised from 8 to 16 to raise throughput; the sealed protocol had said it would not change. The paper reports it.
+
+What it licenses: on fresh synthetic single-line regressions in three large public Python repositories, with one free model and an identical envelope, C7 repaired more than the shipped organ g0. With SR7 this is a sealed two-step record: the loop produced a generation that beat its predecessor, and that generation also beat the shipped organ on fresh public repositories. It does not license any claim that C7 beats the hand-written rule, a scaffold or a model working alone, any claim about natural bugs, other task families or other models, or recursive self-improvement beyond "a nudge towards". HEUR was written after C7 existed and developed on SR7's analysed pool; one model carried the test; three repositories.
+
+### All sealed or registered confirmatory comparisons (technical report, Table 1)
+
+Eleven comparisons: three passed (SR7, M1 confirmation 02 and M1 replication 03, which test the same frozen candidate on two cohorts), one failed a preregistered guard, seven did not show their effect or were null. Within the seven SR tests, one passed. U1's two tests and LOC1's three, run after the record's cut, are listed at the foot and are not in that count. The second column says what each row compared.
 
 | Study (2026 date) | Comparison | Result | Verdict |
 |---|---|---|---|
@@ -151,6 +177,11 @@ Eleven comparisons: three passed (SR7, M1 confirmation 02 and M1 replication 03,
 | SR5b (09-24) | same, 46 medium-repository tasks × 3 rounds | 58/138 vs 60/138; p = 0.685 | not shown |
 | SR6-W (09-24/25) | shipped organ g0 vs a strong fixed scaffold, one 2.6B model, 68 tasks | 5/68 vs 12/68; p = 0.994; mirrored p = 0.0327 | NULL |
 | SR7 (09-24/25) | C7 vs B, 54 tasks × 3 rounds | 57/162 vs 35/162; p = 0.000854 (exact 0.000845) | PASS |
+| U1 primary (10-05) | C7 vs SR6-W's fixed scaffold, same model and kernel, 17 public-repository tasks × 3 rounds | 20/51 vs 26/51; p = 0.873; mirrored p = 0.2095 | NULL |
+| U1 secondary (10-05) | C7 vs B, same tasks | 20/51 vs 15/51; p = 0.1875 | NULL |
+| LOC1 primary (10-06) | C7 vs the shipped organ g0, gpt-oss-20b, 56 tasks in three large public repositories × 3 rounds | 42 of 168 vs 17 of 168; p = 5835/2097152 = 0.00278; mirrored p = 0.9988 | PASS (Bonferroni level 0.0083) |
+| LOC1 secondary (10-06) | C7 vs HEUR, a hand-written trace-aware rule, same tasks | 42 of 168 vs 65 of 168; p = 0.9998; mirrored p = 101/131072 = 0.00077 | HEUR_BETTER |
+| LOC1 replication (10-06) | C7 vs g0, gemini-3.5-flash, 1 task completed | 3 of 3 vs 3 of 3 sessions; p = 1 | NO_DIFFERENCE_DETECTABLE |
 
 Notes on the nulls. SR4 also showed that a keep-best loop scored on tasks whose answers its author sees memorizes them: the tool passed 29/32 development tasks and 1/46 fresh ones; Kaizen now scores only on held-out validation. SR6-W is the one preregistered weak-model test: with a 2.6B model the shipped organ did worse in direction than a strong fixed scaffold, and the mirrored p-value (0.0327) did not meet the declared 0.025, so the verdict is NULL, not a demonstrated harm. The paper does not test why; a plausible reading, not tested, is that model-led navigation spends part of a five-call budget that a weak model needs for edits. SR6-KAIZEN2 (a second free-class generation) closed before confirmation and is not a confirmatory test.
 
@@ -175,22 +206,22 @@ Supervised field construction of two new projects produced 16 applied changes: 1
 
 ### Test evidence, as logged
 
-The release's operator logged a full suite of 1,609 tests passed with 202 of 202 browser loops for the release of 2026-10-04, and a later run on the merged tree of 1,648 passed and 4 failed, the failures fixed in the next commit. These are logged runs, not reruns for this card. The earlier Core snapshot passed 88/88 tests on Windows 11 with Python 3.13 and 84 passed with 4 platform skips on Linux.
+The release's operator logged a full suite of 1,609 tests passed with 202 of 202 browser loops for the release of 2026-10-04, and a later run on the merged tree of 1,648 passed and 4 failed, the failures fixed in the next commit. On the release code (commit `05fa549`; the release commit `c94f01f` changes only the README) the full suite passed on 2026-10-06: 1,924 tests passed, 2 skipped and none failed, and every browser use-loop passed. The earlier counts are logged runs, not reruns for this card. The earlier Core snapshot passed 88/88 tests on Windows 11 with Python 3.13 and 84 passed with 4 platform skips on Linux.
 
 ## What is not yet measured
 
-- **Uplift.** No study runs the same model with and without Runesmith. SR7 runs both arms inside Runesmith; SR6-W compares the shipped organ with a strong fixed scaffold, and the scaffold did better in direction; the other comparisons are between Runesmith components. Any statement that Runesmith makes a given model better than it is alone is unsupported by a measurement. A preregistered uplift study at equal budget (fresh sealed tasks, at least two named models, a minimal harness and a strong conventional scaffold as comparators) is designed and not yet registered.
+- **Uplift.** No study runs a model working alone against the same model inside Runesmith. SR7 and LOC1 run every arm inside Runesmith; SR6-W compares the shipped organ with a strong fixed scaffold, and the scaffold did better in direction; U1, U2 and U2b compared the SR7 generation with that scaffold driving the same model inside the same kernel, with no significant difference in any of six comparisons (the tally is under "What is and is not shown"); the other comparisons are between Runesmith components. Any statement that Runesmith makes a given model better than it is alone is unsupported by a measurement. A preregistered uplift study at equal budget (fresh sealed tasks, at least two named models, a minimal harness and a strong conventional scaffold as comparators) is designed and not yet registered.
 - **Named-model substitution.** Whether competence kept as state survives the removal or replacement of the model that produced it has not been tested. The one substitution-like comparison (SR1-MIN, a model-authored selector against its predecessor) returned equal success in both arms.
 - **State carryover.** The one measurement (73 vs 65, interval crossing zero) was inconclusive.
 - **Weak-model amplification.** Tested once (SR6-W) and null, with the direction favouring the scaffold.
-- **Replication of SR7**, a free-class author reproducing it, an ablation of C7's five steps, a second improving generation, and natural bugs.
+- **Replication of SR7 and LOC1** on other models (LOC1's replication on gemini-3.5-flash completed one task and could detect no difference), a free-class author reproducing SR7, an ablation of C7's five steps, a second improving generation, and natural bugs.
 - **Real local servers** (Ollama, LM Studio, llama.cpp), and **first-time, non-technical users**: no journey measured either.
 
 ## Limitations
 
 - Every result sits inside a stated envelope (the repair organ of the open-source Core, a rule-selected target, a paid author, one cheap repair model, synthetic single-line regressions). None licenses "general intelligence", "safe system", "recursive self-improver" or "commercial moat".
 - Evidence independence is limited. The collaborators who built the studies' machinery, operated the journeys and edited the record are AI collaborators in named roles, and the same model family (Claude Opus) authored SR7's change, built the studies' machinery and operated the journeys. There has been no external peer review.
-- The paper lists 41 threats to validity. Those most relevant here: evaluators and baselines that omit what a conventional workflow already does; development scores on seen tasks that do not transfer; free capacity that varies by the hour; text checks that cannot see rendered output; and the same AI acting as operator and owner, so that every owner audit in the journeys is self-review.
+- The technical report lists 41 threats to validity. Those most relevant here: evaluators and baselines that omit what a conventional workflow already does; development scores on seen tasks that do not transfer; free capacity that varies by the hour; text checks that cannot see rendered output; and the same AI acting as operator and owner, so that every owner audit in the journeys is self-review.
 - Several SR task pools come from the laboratory's private repositories, and one evaluator is not hermetic; the paper states what a reader can and cannot reproduce.
 - Results depend on the model. Free tiers have rate limits and a round may wait or move to the next model you listed.
 - "Done" means that approved checks passed, not that a feature works. A second model that cross-checks the checks can approve wrong ones. Whole-file drafts fail once one file grows past what a model can be shown or can return (targeted edits are planned).
@@ -217,7 +248,7 @@ Release 1 moves generations between installations only by hand. The design the a
   title  = {Beyond the Model: Runesmith, an Open Runtime That Improves Software and Itself, and the Instrument--Substrate Hypothesis},
   year   = {2026},
   month  = oct,
-  note   = {AI ThinkLab. Release draft prepared for publication on 2026-10-06. Not peer reviewed.}
+  note   = {AI ThinkLab. Preprint, 2026-10-06. Not peer reviewed.}
 }
 ```