eclipse

Token-level distillation assumes the student shares the teacher’s tokenizer. A byte-level student does not, so the teacher’s next-token distribution has to be converted into a next-byte distribution with a well-defined terminal state.

This project tests whether that conversion can be exact, whether a small byte student can learn from it, and whether End-Of-Token targets beat Marginalize-It and plain byte cross-entropy in a controlled comparison.

The teacher is Ornith-1.5-9B, a Qwen3.5 hybrid 9B model, quantized to Q5_K_M for bulk extraction and Q8_0 for the precision audit. The student is a ~48M dense byte Transformer with a 1,024-byte context and a 260-symbol vocabulary. The corpus is ~20 MB of NFC-normalized text and code: 6,592,718 training tokens and 136,916 validation tokens. Training runs on a single RTX 3070 8 GB.

Teacher logits stream from llama.cpp and are converted against a byte-trie of the 248k-token vocabulary in DFS order, so each prefix subtree is a contiguous range. Byte distributions come from child subtree masses; the terminal node receives p(token) / mass(prefix) on the EOT symbol.

Marginalize-It is derived from the same 260-dimensional row by zeroing EOT and renormalizing. Raw logits are never persisted; targets are stored as fp16 shards with explicit next-symbol labels.

A float32 cumulative sum in the Python converter could misallocate rare-prefix targets (up to 0.789 against a float64 reference). The C++ converter is the reference path and matches float64 to 1.3e-7.

The 1M extraction produced 4,014,129 positions from 1,000,497 teacher tokens: 2.087 GB, 431 tokens/s, 38.7 minutes. Five positions underflow in fp16; the shard otherwise validates with exact row sums and counts.

Next: a 24 GB GPU path with a 100–300M student and on-the-fly targets, a generation-focused objective, and a second seed or token-student control. The current result is a feasibility study, not a trained product.