olaflaitinen / knap
A byte level BPE tokenizer written from scratch in pure Mojo. Implements the seven encodings OpenAI models use, runs with no Python interpreter in the process, and publishes its evidence: 191.7 million tokens and every token id checked against the reference implementation.
bpebyte-pair-encodingdifferential-testingfuzzinggptllmmaxmodularmojomojo-langnlpopenaisimdtext-processingtiktokentokenizerunicode
Stars4when found
Forks0
Created2026-09-072 months ago
Lines of Mojo17,42155 files
Commits752026-09-07 to 2026-09-11
Mojo version1.0.0from lock file
Toolingpixi
LicenseEUPL-1.2
Lines of Mojo
2026-09-07 to 2026-09-11; peak 17,421 lines
Stars per month
- stars
- forks
2 stars over the months of GH Archive loaded so far (8; months in between aren't loaded yet), counted from its events.
Releases and tags
| Date | Tag | Name | From | Mojo version | Commit |
|---|---|---|---|---|---|
| 2026-09-11 | v1.0.0 | – | git tag | 1.0.0 | 4c5b07a909 |
Migrations
| Date | What | From | To | Commit |
|---|---|---|---|---|
| 2026-09-07 | mojo_version | – | 1.0.0 | 571c26a698 |
| 2026-09-07 | tooling | none | pixi | 571c26a698 |
Commits
| Date | Commit | Lines of Mojo | Tooling | Mojo version |
|---|---|---|---|---|
| 2026-09-11 | b1c30f3ab8 | 17,421 | pixi | 1.0.0 |
| 2026-09-11 | 67056e08fd | 17,421 | pixi | 1.0.0 |
| 2026-09-11 | 4c5b07a909 | 17,421 | pixi | 1.0.0 |
| 2026-09-11 | 05aceb717a | 17,421 | pixi | 1.0.0 |
| 2026-09-11 | e83f0ea34f | 17,421 | pixi | 1.0.0 |
| 2026-09-10 | d90b59d4c4 | 17,421 | pixi | 1.0.0 |
| 2026-09-10 | 0008794d5b | 17,421 | pixi | 1.0.0 |
| 2026-09-10 | 3a2ce50309 | 17,421 | pixi | 1.0.0 |
| 2026-09-10 | 3927e1dbf3 | 17,421 | pixi | 1.0.0 |
| 2026-09-10 | 3bc1e184e1 | 17,421 | pixi | 1.0.0 |
| 2026-09-10 | 3fde614cd2 | 17,324 | pixi | 1.0.0 |
| 2026-09-10 | 3fbf271be5 | 16,565 | pixi | 1.0.0 |
| 2026-09-10 | 54afd98f91 | 16,565 | pixi | 1.0.0 |
| 2026-09-10 | 0bcc85d015 | 16,461 | pixi | 1.0.0 |
| 2026-09-10 | bacb031baf | 16,461 | pixi | 1.0.0 |
| 2026-09-10 | 8342c79ce3 | 16,461 | pixi | 1.0.0 |
| 2026-09-10 | efdf05c43d | 16,461 | pixi | 1.0.0 |
| 2026-09-10 | 92853d574f | 16,461 | pixi | 1.0.0 |
| 2026-09-10 | 1bcee0c364 | 16,461 | pixi | 1.0.0 |
| 2026-09-10 | 0982d712a1 | 16,461 | pixi | 1.0.0 |
| 2026-09-10 | cc2617a765 | 16,461 | pixi | 1.0.0 |
| 2026-09-10 | 4734f8c609 | 16,461 | pixi | 1.0.0 |
| 2026-09-10 | 899e0442aa | 16,461 | pixi | 1.0.0 |
| 2026-09-10 | eb910bb9a7 | 15,586 | pixi | 1.0.0 |
| 2026-09-09 | 304ec2c092 | 14,851 | pixi | 1.0.0 |
| 2026-09-09 | 7405a02614 | 14,851 | pixi | 1.0.0 |
| 2026-09-09 | 04f50b66ba | 14,851 | pixi | 1.0.0 |
| 2026-09-09 | eebb252a41 | 14,786 | pixi | 1.0.0 |
| 2026-09-09 | fa89bae14e | 14,786 | pixi | 1.0.0 |
| 2026-09-09 | 8f17fd6aa9 | 14,786 | pixi | 1.0.0 |
| 2026-09-09 | a7a45a73d6 | 14,532 | pixi | 1.0.0 |
| 2026-09-09 | 0814805655 | 12,894 | pixi | 1.0.0 |
| 2026-09-09 | fb45b7c0b0 | 12,894 | pixi | 1.0.0 |
| 2026-09-09 | 71003bfd65 | 12,894 | pixi | 1.0.0 |
| 2026-09-09 | d7bcacb321 | 12,894 | pixi | 1.0.0 |
| 2026-09-09 | 6ec02ea191 | 12,894 | pixi | 1.0.0 |
| 2026-09-09 | b1d2ee2547 | 12,775 | pixi | 1.0.0 |
| 2026-09-09 | 82075426c4 | 12,775 | pixi | 1.0.0 |
| 2026-09-09 | 45fc6d3df4 | 12,775 | pixi | 1.0.0 |
| 2026-09-09 | ff3d7bdd76 | 12,775 | pixi | 1.0.0 |
| 2026-09-09 | 7f87cd73fb | 12,775 | pixi | 1.0.0 |
| 2026-09-09 | 64c1d6d437 | 12,775 | pixi | 1.0.0 |
| 2026-09-09 | cd59ada95f | 12,775 | pixi | 1.0.0 |
| 2026-09-09 | f762cdb4d6 | 12,775 | pixi | 1.0.0 |
| 2026-09-08 | 9faff20ba4 | 11,449 | pixi | 1.0.0 |
| 2026-09-08 | 4c50132f06 | 11,449 | pixi | 1.0.0 |
| 2026-09-08 | 00d0c794ee | 11,449 | pixi | 1.0.0 |
| 2026-09-08 | 413264e6da | 11,449 | pixi | 1.0.0 |
| 2026-09-08 | b070918786 | 11,449 | pixi | 1.0.0 |
| 2026-09-08 | 5dc05024f7 | 11,422 | pixi | 1.0.0 |
The latest 50 of 75 commits; all of them are in the JSON.