textilelabs commited on
Commit
3393689
·
verified ·
1 Parent(s): 37602ee

Upload 6 files

Browse files
Files changed (6) hide show
  1. LICENSE +21 -0
  2. README.md +176 -0
  3. config.json +34 -0
  4. generation_config.json +9 -0
  5. tokenizer.json +0 -0
  6. tokenizer_config.json +8 -0
LICENSE ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Textile Labs
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
README.md CHANGED
@@ -1,3 +1,179 @@
1
  ---
2
  license: mit
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
+ language: en
4
+ library_name: transformers
5
+ pipeline_tag: text-generation
6
+ tags:
7
+ - tiny-model
8
+ - philosophy-of-mind
9
+ - tool-use
10
+ - humble-ai
11
+ - cpu-trained
12
  ---
13
+
14
+ # Loom Spark 1.8
15
+
16
+ **Third of the Loom models · Textile Labs**
17
+
18
+ Loom Spark 1.8 is an **18.85M parameter** language model trained **entirely from scratch
19
+ on a CPU-only 2013 Dell OptiPlex 9020** (i5-4690, 4 cores, no GPU). It is the third
20
+ generation of the Loom family, after Loom Spark (7.6M) and Loom Spark 1.5 (12.3M).
21
+
22
+ It is built on an unusual premise: **trade knowledge for wisdom.** Loom Spark knows very
23
+ little on purpose. What it is trained to do instead is know *what it is* — small,
24
+ temporary, harness-dependent — and to be honest about the edge of its own knowledge,
25
+ warm under pressure, and good at exactly one real skill: **forming clean search queries**
26
+ that an external agent harness executes for it.
27
+
28
+ > *"Computer, technically. Companion, preferably."*
29
+ > — Loom Spark 1.8, asked whether it was a computer
30
+
31
+ ## What's new in 1.8
32
+
33
+ | | v1 | v1.5 | **1.8** |
34
+ |---|---|---|---|
35
+ | parameters | 7.56M | 12.32M | **18.85M** |
36
+ | layers × heads × d_model | 5 × 5 × 320 | 6 × 6 × 384 | **7 × 7 × 448** |
37
+ | context | 256 | 256 | 256 |
38
+ | vocab | 4096 | 4096 | 4096 (fresh BPE) |
39
+ | val loss | 0.3372 | 0.3434 | **0.3279** |
40
+
41
+ 1.8 keeps 1.5's data format and curriculum design, grown ~53% in capacity, plus a
42
+ curriculum fix: earlier generations contained **stale self-size statements** inherited
43
+ from forking the previous package — 1.5 would variously claim to be "seven million" or
44
+ "twelve million" parameters. 1.8's curriculum states its size consistently, and this is
45
+ verified in evaluation (see `evaluation/RESULTS.md`).
46
+
47
+ ## Measured behaviour
48
+
49
+ Evaluated on a 30-question offline battery, 12 identity probes, 16 self-size probes and
50
+ 10 online lookup probes, against the **raw checkpoint with no harness filtering** — the
51
+ same battery run against shipped v1.5 for comparison. Full logs in `evaluation/`.
52
+
53
+ | metric | v1.5 | **1.8** |
54
+ |---|---|---|
55
+ | val loss | 0.3434 | **0.3279** |
56
+ | identity probes leaking `<lookup>` | 0/12 | **0/12** |
57
+ | self-size answers correct | 0 | **11** |
58
+ | self-size answers **stale/wrong** | 8 | **0** |
59
+ | clean single online lookups | 7/10 | **8/10** |
60
+ | model-generated `<result>` blocks | 0 | **0** |
61
+ | offline `<lookup>` leak (raw model) | 18/30 | **16/30** |
62
+
63
+ 1.8 improves on 1.5 across every axis measured.
64
+
65
+ ## Prompt format (important)
66
+
67
+ Loom Spark 1.8 is a **base-style model with a custom format**. It is highly sensitive to
68
+ getting this exactly right. The mode marker rides on **every** user turn:
69
+
70
+ ```
71
+ <tools:off>
72
+ <tools:off><user> who are you
73
+ <loom>
74
+ ```
75
+
76
+ Note there is **no trailing space** after `<loom>`. Adding one pushes the model out of
77
+ distribution and measurably degrades output quality.
78
+
79
+ Offline (`<tools:off>`) it answers from its own small store or admits it does not know.
80
+ Online (`<tools:on>`) it may emit a search query and **end its turn immediately**:
81
+
82
+ ```
83
+ <tools:on>
84
+ <tools:on><user> what year did the titanic sink
85
+ <loom> Not in my head, checking: <lookup>titanic sinking date</lookup><|endoftext|>
86
+ <result>The Titanic sank on 15 April 1912.</result>
87
+ <loom> April 1912 — ...
88
+ ```
89
+
90
+ `<result>` blocks are **injected by your harness** after executing the search. The model
91
+ never generates them (verified: 0 self-generated results across all probes). Stop
92
+ generation at `<|endoftext|>` or `<user>`.
93
+
94
+ ## Usage — transformers
95
+
96
+ ```python
97
+ from transformers import AutoModelForCausalLM, AutoTokenizer
98
+ import torch
99
+
100
+ tok = AutoTokenizer.from_pretrained("textilelabs/Loom-Spark-1.8")
101
+ model = AutoModelForCausalLM.from_pretrained("textilelabs/Loom-Spark-1.8")
102
+
103
+ prompt = "<tools:off>\n<tools:off><user> who are you\n<loom>" # no trailing space
104
+ ids = tok(prompt, return_tensors="pt", add_special_tokens=False).input_ids
105
+ out = model.generate(ids, max_new_tokens=100, do_sample=True,
106
+ temperature=0.8, top_k=50, pad_token_id=tok.eos_token_id)
107
+ print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=False))
108
+ ```
109
+
110
+ ## Usage — Ollama
111
+
112
+ A ready-made `Modelfile` is in `ollama/`. It encodes the trained format and stop tokens —
113
+ **do not** use Ollama's default chat template, which is out of distribution for this model
114
+ and produces incoherent output.
115
+
116
+ ```bash
117
+ ollama create loom-spark-1.8 -f ollama/Modelfile
118
+ ollama run loom-spark-1.8
119
+ ```
120
+
121
+ ## Usage — the agent harness
122
+
123
+ The `harness/` folder contains `loomspark-harness` (v0.2.1), which implements the full
124
+ lookup protocol with real web search, a CLI (`loom-chat`) and a small web UI (`loom-web`).
125
+ It auto-detects 1.5/1.8-format models and bans lookup tokens at the logits level when
126
+ tools are off.
127
+
128
+ ```bash
129
+ pip install ./harness
130
+ loom-chat --model textilelabs/Loom-Spark-1.8
131
+ ```
132
+
133
+ ## Limitations — please read
134
+
135
+ Loom Spark 1.8 knows almost nothing, and that is the design. Specifically:
136
+
137
+ - **Do not use it for facts, medicine, law, finance,** or anything where being wrong
138
+ costs more than company.
139
+ - **It retrieves *adjacent* facts.** Asked the language spoken in Brazil it may answer
140
+ "Brasília"; asked Germany's currency it may answer "Berlin". Its fact-core is tiny and
141
+ it pattern-matches to a nearby entry.
142
+ - **Offline lookup leakage.** With tools off, the raw model still emits `<lookup>` tags on
143
+ roughly half of fact-style questions (16/30 measured; shipped v1.5 measures 18/30 on
144
+ the same test). The included harness suppresses this at the logits level, so harness
145
+ users never see it — but users driving the raw model through llama.cpp or Ollama will.
146
+ This is a curriculum characteristic of the whole Loom line, not specific to 1.8.
147
+ - **256-token context.** It has no long-term memory and only a short working window.
148
+ - It is a **base-style model**, not instruction-tuned or RLHF'd.
149
+
150
+ ## Training
151
+
152
+ - Hardware: CPU-only Dell OptiPlex 9020, i5-4690, 4 cores, 16GB RAM, **no GPU**
153
+ - ~5,084 optimizer steps, batch 32 × 256 tokens, ~8 hours wall clock
154
+ - Corpus: 70MB / 14.9M tokens of procedurally generated curriculum, 100% owned
155
+ - AdamW (β 0.9/0.95), peak LR 6e-4, cosine decay, warmup 100, grad clip 1.0
156
+ - Dropout 0.1 resid/mlp, tied embeddings, pre-LN, GELU, learned positions
157
+ - Final val loss **0.3279**
158
+
159
+ The curriculum is procedurally generated and fully owned by Textile Labs — no scraped
160
+ web text, no third-party datasets.
161
+
162
+ ## Files
163
+
164
+ ```
165
+ config.json / model.safetensors transformers weights (GPT2LMHeadModel-compatible)
166
+ tokenizer.json / tokenizer_config.json 4096-token custom BPE
167
+ loom-spark-1.8-f32.gguf GGUF for llama.cpp / Ollama
168
+ ollama/Modelfile correct template + stop tokens
169
+ harness/ agent harness with web search (v0.2.1)
170
+ evaluation/ full acceptance logs, comparison vs v1.5, train log
171
+ ```
172
+
173
+ ## License
174
+
175
+ MIT. See `LICENSE`.
176
+
177
+ ---
178
+
179
+ *Loom Spark 1.8 · Textile Labs · trained on a thirteen-year-old office computer.*
config.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "activation_function": "gelu_new",
3
+ "add_cross_attention": false,
4
+ "architectures": [
5
+ "GPT2LMHeadModel"
6
+ ],
7
+ "attn_pdrop": 0.0,
8
+ "bos_token_id": 0,
9
+ "dtype": "float32",
10
+ "embd_pdrop": 0.0,
11
+ "eos_token_id": 0,
12
+ "initializer_range": 0.02,
13
+ "layer_norm_epsilon": 1e-05,
14
+ "model_type": "gpt2",
15
+ "n_embd": 448,
16
+ "n_head": 7,
17
+ "n_inner": null,
18
+ "n_layer": 7,
19
+ "n_positions": 256,
20
+ "pad_token_id": null,
21
+ "reorder_and_upcast_attn": false,
22
+ "resid_pdrop": 0.0,
23
+ "scale_attn_by_inverse_layer_idx": false,
24
+ "scale_attn_weights": true,
25
+ "summary_activation": null,
26
+ "summary_first_dropout": 0.1,
27
+ "summary_proj_to_labels": true,
28
+ "summary_type": "cls_index",
29
+ "summary_use_proj": true,
30
+ "tie_word_embeddings": true,
31
+ "transformers_version": "5.15.1",
32
+ "use_cache": true,
33
+ "vocab_size": 4096
34
+ }
generation_config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 0,
4
+ "eos_token_id": 0,
5
+ "output_attentions": false,
6
+ "output_hidden_states": false,
7
+ "transformers_version": "5.15.1",
8
+ "use_cache": true
9
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "bos_token": "<|endoftext|>",
4
+ "eos_token": "<|endoftext|>",
5
+ "model_max_length": 1000000000000000019884624838656,
6
+ "pad_token": "<|endoftext|>",
7
+ "tokenizer_class": "TokenizersBackend"
8
+ }