ratulb / mbpe
mbpe — A BPE tokenizer engine in Mojo, generic over a compile-time pre-tokenizer. Trains vocabularies from scratch, reads/writes the standard .tiktoken format, and reproduces OpenAI's GPT-2/GPT-4/GPT-4o vocabularies byte-for-byte. Outperforms tiktoken-rs (Rust) on every benchmark.
Stars9when found
Forks7
Created2026-07-263 months ago
Lines of Mojo6,25120 files
Commits1912026-07-23 to 2026-10-28
Mojo version1.2.0from lock file
Toolingpixi
LicenseMIT
Lines of Mojo
2026-07-23 to 2026-10-28; peak 7,931 lines
Stars per month
- stars
- forks
0 stars over the months of GH Archive loaded so far (8; months in between aren't loaded yet), counted from its events.
Releases and tags
| Date | Tag | Name | From | Mojo version | Commit |
|---|---|---|---|---|---|
| 2026-07-27 | mbpe | – | git tag | 1.0.0 | d35dfb4b97 |
Migrations
| Date | What | From | To | Commit |
|---|---|---|---|---|
| 2026-07-23 | mojo_version | – | 1.0.0 | 71a98eda31 |
| 2026-07-23 | tooling | none | pixi | 71a98eda31 |
| 2026-09-19 | mojo_version | 1.0.0 | 1.1.0 | 55e168bbcc |
| 2026-10-28 | mojo_version | 1.1.0 | 1.2.0 | 9d36bb2d58 |
Commits
| Date | Commit | Lines of Mojo | Tooling | Mojo version |
|---|---|---|---|---|
| 2026-10-28 | 1e7aa381f3 | 6,251 | pixi | 1.2.0 |
| 2026-10-28 | 9d36bb2d58 | 6,251 | pixi | 1.2.0 |
| 2026-10-08 | ca06186383 | 6,257 | pixi | 1.1.0 |
| 2026-09-22 | 8cef990adb | 6,257 | pixi | 1.1.0 |
| 2026-09-21 | 925da3a01f | 6,257 | pixi | 1.1.0 |
| 2026-09-21 | 38edafe0ce | 6,257 | pixi | 1.1.0 |
| 2026-09-21 | 5c2ad578c3 | 6,257 | pixi | 1.1.0 |
| 2026-09-21 | e7b7f9b397 | 6,257 | pixi | 1.1.0 |
| 2026-09-21 | 26688bf097 | 6,257 | pixi | 1.1.0 |
| 2026-09-20 | 67b2780654 | 6,257 | pixi | 1.1.0 |
| 2026-09-20 | b0b62d20c0 | 6,257 | pixi | 1.1.0 |
| 2026-09-20 | 28dcb27cd1 | 6,257 | pixi | 1.1.0 |
| 2026-09-20 | aeeaaeb3ee | 6,257 | pixi | 1.1.0 |
| 2026-09-20 | 753c05a5bd | 6,257 | pixi | 1.1.0 |
| 2026-09-20 | 65f9ec5bed | 6,257 | pixi | 1.1.0 |
| 2026-09-20 | c60a6085c9 | 6,257 | pixi | 1.1.0 |
| 2026-09-19 | 3e1ed1c48c | 6,257 | pixi | 1.1.0 |
| 2026-09-19 | 55e168bbcc | 6,257 | pixi | 1.1.0 |
| 2026-09-19 | c90ada3b2a | 6,257 | pixi | 1.0.0 |
| 2026-09-19 | 42d586808b | 6,257 | pixi | 1.0.0 |
| 2026-09-15 | b2da942764 | 6,268 | pixi | 1.0.0 |
| 2026-09-15 | 0207f008e6 | 6,268 | pixi | 1.0.0 |
| 2026-09-12 | 9745ad535d | 6,268 | pixi | 1.0.0 |
| 2026-09-12 | deeb1b365d | 6,268 | pixi | 1.0.0 |
| 2026-09-10 | c0dde361df | 6,268 | pixi | 1.0.0 |
| 2026-09-10 | 713b54e83c | 6,268 | pixi | 1.0.0 |
| 2026-09-09 | e67c4ca3df | 6,229 | pixi | 1.0.0 |
| 2026-09-09 | a6ae5075a3 | 6,229 | pixi | 1.0.0 |
| 2026-08-09 | df8c710012 | 6,220 | pixi | 1.0.0 |
| 2026-08-09 | 6f88548cbb | 6,220 | pixi | 1.0.0 |
| 2026-08-07 | c2c3868818 | 6,220 | pixi | 1.0.0 |
| 2026-08-07 | 98c5710401 | 6,220 | pixi | 1.0.0 |
| 2026-08-07 | 4b3a3fed26 | 6,220 | pixi | 1.0.0 |
| 2026-08-06 | 606509737c | 5,823 | pixi | 1.0.0 |
| 2026-08-06 | bc47e0b071 | 5,823 | pixi | 1.0.0 |
| 2026-08-06 | 50df7c930b | 5,823 | pixi | 1.0.0 |
| 2026-08-06 | 32c748d202 | 5,823 | pixi | 1.0.0 |
| 2026-08-06 | 5d4e84138c | 7,585 | pixi | 1.0.0 |
| 2026-08-06 | 98e1eee616 | 7,585 | pixi | 1.0.0 |
| 2026-08-06 | e662d5ff3b | 7,575 | pixi | 1.0.0 |
| 2026-08-05 | 6d4470e291 | 7,575 | pixi | 1.0.0 |
| 2026-08-05 | 15be30d221 | 7,930 | pixi | 1.0.0 |
| 2026-08-05 | 4983ea0234 | 7,931 | pixi | 1.0.0 |
| 2026-08-05 | b4e2ca710b | 7,931 | pixi | 1.0.0 |
| 2026-08-05 | 80030a5206 | 7,931 | pixi | 1.0.0 |
| 2026-08-05 | 66f51b4221 | 7,664 | pixi | 1.0.0 |
| 2026-08-05 | cdfa22342d | 7,664 | pixi | 1.0.0 |
| 2026-08-05 | b126eb5296 | 7,713 | pixi | 1.0.0 |
| 2026-08-05 | 91c3085eab | 7,713 | pixi | 1.0.0 |
| 2026-08-04 | b66b09d067 | 7,682 | pixi | 1.0.0 |
The latest 50 of 191 commits; all of them are in the JSON.