Rendered at 18:33:30 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
maxdo 1 days ago [-]
Interesting :
Q: Did you just way over-optimize for a specific CPU and tokenizer? How is it so fast?
No, I way over-optimized for every combination of these! The results are very consistent across CPUs (modern x86 and ARM), and across specific tokenizers.
The major improvements are in optimizing heavily an implementation that usually is outsourced to a Regex engine (pretokenization) using SIMD, minimizing branching and other tricks, as well as heavily optimizing caching of pretoken mappings (if a word has been seen before, look it up its encoded tokens efficiently). Caching is a very hard problem in this domain since the cache grows very quickly, and pretoken distributions are very long-tailed.
Finally, interactions with Python are minimized, and threads have minimal interactions with each other.
cschmidt 17 hours ago [-]
Can I say this seems to be fantastic work. I cloned your repo earlier today after seeing it on the tokenization discord. I know everyone in the tokenization community wants to absorb the lessons of how you got such a speedup. The caching and replacing the regex for pretokenization seem like generally useful ideas.
And screw all the 0.1% haters on here, this is great stuff.
marcelroed 15 hours ago [-]
Thanks for the kind words, Craig! I'm planning to do a technical writeup+paper and a presentation video on the project in the near future. Will make sure to share it with the Discord!
njjkkjjdhs 14 hours ago [-]
[dead]
cs702 16 hours ago [-]
That is my reaction too. It looks like great work!
Valuable not only for inference, but for training too (think proprietary datasets).
I would add, a single individual did this.
One person can make a difference :-)
victor106 15 hours ago [-]
> tokenization discord
How can I join this? Sounds interesting
cschmidt 7 hours ago [-]
Send me an email (my address is in my profile)
glenngillen 15 hours ago [-]
also subscribing to this!
cschmidt 7 hours ago [-]
Send me an email (my address is in my profile)
gghfez 14 hours ago [-]
>the tokenization discord
Could I join this?
cschmidt 7 hours ago [-]
Send me an email (my address is in my profile)
asdf88990 16 hours ago [-]
[flagged]
cschmidt 16 hours ago [-]
I’m not sure why you think this is ai slop. I work on tokenization research full time. My name is Craig Schmidt and I have a number of papers in the field. This researcher has done some very impressive work and I’m trying to defend him from the HN dismissive hoards.
There is a serious research community on tokenization, and we are quite interested in this work.
asdf88990 16 hours ago [-]
Which part of “doesn’t seem to be aislop signs” made you think I believe your comment was aislop?
As I said, It just might be the low information density approach to talking recently that seems off to me.
To explain a bit more, I kept reading and waiting for the penny drop but nothing.
“I cloned your repo” — okay then what happened? Nothing? I put my shoes on this morning.
“I know the tokrniziation community wants to absord tje lessons” okay? Tell me the point please! I know the AI community wants to understand the universe.
“These are useful ideas”. Yes.
Of course, you could argue that my comment falls in the same category in the sense that I am not actually contributing to the topic at hand but I am peeved with all low the information noise.
XCSme 10 hours ago [-]
Congrats, I love performance optimizations!
Hardware nowadays is so powerful, but our code so inefficient... I think most libraries/apps could easily be 10x-100x faster if we really try to optimize them.
The good thing is, that now with AI, we'll probably have the time to implement those optimizations rather quickly.
lionkor 10 hours ago [-]
The issue is really with testing! You can do such optimizations but you have to be sure that the result is the same in ALL your use cases, so you need very good test coverage and quite good tests as well.
LLMs can help with the amount of test, and somewhat with the quality, but that's not quite enough for doing heavy optimizations in existing, deployed products, in a normal sprint somewhere.
XCSme 9 hours ago [-]
I think sometimes optimizations are more about using the right data structure/ideas/libraries.
In my case, I recently optimized for https://uxwizz.com the session playback: before, it was saving the entire recording in one row, with updates, and loading it client-side in one chunk. I asked thr AI to, instead, store the recorded chunks in separate rows in db, and when replaying, to stream those chunks as needed. It implemented it perfectly, and everything is much snappier and lighter now.
So, the idea was "store in chunks and stream the chunks", and the LLM does it. Yes, there were a few bugs, but those systems genetally either work or don't. So testing for a 30 minutes after was enough, and now the system is live, and a client actually told me in person yesterday he loved the playback improvements.
I said in another comment somewhere, now knowing how to code isn't important, what's important is that you know what to ask, and what to look for when testing.
embedding-shape 10 hours ago [-]
> The good thing is, that now with AI, we'll probably have the time to implement those optimizations rather quickly.
Haha, yeah, product/executives will surely now see the benefits of optimizations instead of piling new features on top of new features with no cohesive idea about the design or architecture :)
XCSme 9 hours ago [-]
In my experience, when adding new features with LLMs, most of those optimizations come automatically.
Good models now already follow best practices when implementing, better than junior devs.
I wrote a bit about this, I call it "AI slap", lol:
We're clearly thinking of very different "optimizations" here I think :) Do you have any concrete examples of this sort of optimizations you'd get automatically? In my experience, you get what you prompt for, if I don't include to think about performance, they won't think about performance, not sure what model would automatically consider things like that. Most of the time I use whatever SOTA OpenAI has on maximum reasoning level, fwiw.
XCSme 8 hours ago [-]
Not only performance optimizations, but also UI/UX.
If you ask to implement a custom dropdown that does something, it often comes with good spacing, aria-accessible tags, keyboard accessibility, etc. A junior dev wouldn't think of all of those.
Also, it will probably choose the right HTML elements to use for it (i.e. maybe the modern native popover functionality instead of implementing it with custom JS, which would indeed be more efficient and less code).
I am not saying it would add caching by default (even though, it might suggest that), but it's more likely to choose whatever the best options are and to use them as they should, including going around knowing limitations and gotchas.
onlyrealcuzzo 1 days ago [-]
This is awesome, but tokenization is typically <0.1% of total inference time.
Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
marcelroed 19 hours ago [-]
Author here: Actually, depending on the nature of the inference you're doing it can be quite significant. Here are some numbers for time-to-first-token (time to process the entire input and produce the first token of output) for an 8B Qwen3 model running on a single B200. Obviously these numbers are more significant with smaller models and on faster GPUs. Credit to fastokens [0] for the benchmark.
sglang_speed [huggingface]: mean=10.31ms median=6.48ms p99=45.98ms rps=96.8
sglang_speed [gigatoken]: mean=10.13ms median=6.54ms p99=45.16ms rps=98.4
input_len= 2048: TTFT mean 30.74 -> 29.05 ms (+5.5% reduction) | median 31.00 -> 28.80 (+7.1%) | p99 33.02 -> 32.02 (+3.0%)
input_len= 8192: TTFT mean 105.20 -> 96.36 ms (+8.4% reduction) | median 103.87 -> 95.49 (+8.1%) | p99 126.88 -> 113.84 (+10.3%)
input_len= 32768: TTFT mean 687.05 -> 633.66 ms (+7.8% reduction) | median 708.14 -> 657.35 (+7.2%) | p99 728.95 -> 678.79 (+6.9%)
These are preliminary numbers, so I will need to do some more testing before including this in the README.
I run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request.
jaggederest 22 hours ago [-]
To concur it's "latency critical", not "performance critical", people often confuse those two - optimize it all, but especially the chained critical path latency!
dataflow 18 hours ago [-]
Latency isn't performance? Maybe you mean "not throughput-critical"?
jaggederest 16 hours ago [-]
Just to clarify, latency is one form of performance, and a separate thing to optimize from total resource usage in more classic "performance critical" situations. That performance might be energy, space, or other dimensions besides latency. It might also be something like reliability, accuracy, precision, or even the very human factors like simplicity, modifiability, and visibility.
Heck, even latency alone you can just reduce the standard deviation and get smoother flows. Little's law is a great callout here too, one of my favorite computer science principles.
rockwotj 17 hours ago [-]
but according to Little’s law, if you improve latency, you also improve throughput, right?
If I have the same number of CPU cores and they all can do their work in half the time they can double the number of requests now
scheme271 17 hours ago [-]
I don't think that's accurate. If tokenization takes say 10ms and the rest of the inference steps take 50 ms then, improving tokenization will improve the time to first token but won't affect throughput much. After the first token, the inference steps effectively hide the tokenization time.
ludston 10 hours ago [-]
It depends on whether or not latency is the constraint of throughput in your circumstances.
flockonus 19 hours ago [-]
Same here, as we're sitting in the middle between requests and what budget constraints are allowed given a particular token allowance there can be 10 ~ 100 milliseconds improvement in the UX (TTFT) given such massive tokenization speed up.
Xalutiono 9 hours ago [-]
Whats the prefered LLM runtime to use? vLLM?
Tips & Tricks on parameters/settings?
What happens at peak? Do people have to wait now? Increase of latency?
scottcha 4 hours ago [-]
We use vllm as it generally has the best ecosystem support.
Parameters are largely dependent on what type of requests you are serving (concurrency, input/output ratios, cached hit patterns).
We've never had a limitation at the tokenizer step. Limitations at peak tend to manifest more on slower time in vllm doing prefill or decode though we actively try and minimize this.
quietfox 20 hours ago [-]
‘I run an AI platform’
I have so many genuine questions I don’t even know where to start.
NuclearPM 20 hours ago [-]
Pick one. I would like to hear it.
pipsterwo 24 hours ago [-]
1/1000 of inference compute is a non-trivial workload at scale. Gartner estimates ~$28B in inference spend for 2026 making this a $28 million dollar per year workload (edit: based on the assumption above)
The issue is it’s cpu compute which is underutilized in gpu clusters anyway, so practically it’s not really 1/1000.
pipsterwo 23 hours ago [-]
Totally, edited my comment to specify "based on the assumption above." The main takeaway I was going for was 0.1% is not a small number in this context
noahbp 22 hours ago [-]
Time to first token, especially for smaller models, can be sharply reduced.
Latency can be just as important as overall throughput, especially for inference providers like Groq and Cerebras.
fastball 22 hours ago [-]
Tokenization is <0.1% of the inference time for the first token in the same way it is <0.1% for the last.
marcelroed 22 hours ago [-]
Time to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower when doing prefill, so the significance of tokenization is higher.
dingdingdang 20 hours ago [-]
Have you done preliminary numbers on replacing tokenizer on, say, llama-server?
My understanding is that tokenization is largely serial, so for a large initial prompt it can make up a large chunk of input processing time since after handing it off to the model inference it's (able to be) fully parallel across all tokens.
GenerocUsername 24 hours ago [-]
Always good to make it 0.001%
brcmthrowaway 23 hours ago [-]
[dead]
ubedan 13 hours ago [-]
Spectacular... Reminds me of the SimdJson algorithm in terms of jaw dropping nearly unbelievable speeds through creative programming. I hope this code get popular, as it will save tons of electricity, money, CO2, etc.
Have you considered publishing a rust crate as well? (If not, I volunteer.)
o_m 8 hours ago [-]
According to Jevons paradox this will likely lead to more electricity used and more CO2 being released to the atmosphere, because it makes it more profitable to build another data center.
bob1029 12 hours ago [-]
> I hope this code get popular, as it will save tons of electricity, money, CO2, etc.
I don't think tokenization has ever been a meaningful bottleneck. JSON being fast falls into the same bucket much of the time. We spend way more energy on I/O and storage than we do on serialization and tokenization.
If you are concerned with economics and the environment, request batching would make a bigger impact. The most expensive part of this whole thing is GPU underutilization. You can save 50% with OAI right now if you can figure out how to make your workload fit the batch pattern. Do your users always need answers right now or can we afford to wait a few days in some cases? Tool calling doesn't "time out". Wall clock does not exist in the LLM. It took me a while to get used to this.
a_c 9 hours ago [-]
Is there any write up regarding the SimdJson Algo? Definitely love to read more of it!
Cool stuff. From my understanding, this is less valuable at inference time and more useful when running offline pre-training data prep.
When tokenizing terabytes of text for your training corpus, the speedup here is probably doing real work in saving you time (and money?). You get a faster iteration cycle when figuring out and adjusting your datasets.
It will be nicer if the README focuses more on per-core performance.
About the actual algorithm - will something like matching in a perfect hash table help?
Antibabelic 12 hours ago [-]
"AI Use Disclosure: A majority of this code base was crafted by hand without any use of AI (which can be seen from the project's Git history)."
So much for "human programming is obsolete".
porridgeraisin 8 hours ago [-]
For context:
In the final stages of the project, AI was used to assist:
Implementing the user-facing API
Widening of compatibility, for instance generalizing and porting the pretokenizer implementations to support more tokenizers, less interesting features like padding/truncation/unicode normalization
Porting SIMD strategies between AVX512/AVX2/NEON
Final profiling stages and the last ~4x worth of performance from eliminating branching and improving the pretoken cache hierarchy
Refactoring and code reuse
Cthulhu_ 9 hours ago [-]
Nobody except people trying to sell AI are claiming human programming is obsolete though.
luciana1u 22 hours ago [-]
engineering effort to make something 1000x faster that accounts for 0.1% of total runtime is the most software developer thing imaginable
minimaxir 20 hours ago [-]
I initially had a Rust-based word cloud generator that generates word clouds in high resolution in ~100 milliseconds whereas it would take other generators a couple seconds to do the same thing. Does the world need a super-fast word cloud generator? No. Do I want a super-fast word cloud generator? Yes.
I later found enough optimizations to reduce the generation speed all the way down to ~16ms. Do I need a word cloud generator that fast? No. But if I have a word cloud generator it's going to be as fast as possible dammit.
netniuq 5 hours ago [-]
links or it didn't happen ;)
piker 22 hours ago [-]
"The pursuit of excellence does not need justification."
This depends on what your workflow is. There are use cases for tokenization that don't always involve immediately feeding the text into a model.
casey2 4 hours ago [-]
There are more usecases, for this class of tokenizer, now that it's 1000x faster as well. RAG being one example.
hansvm 18 hours ago [-]
1000x improvements unlock qualitatively new capabilities, even when only applied to subcomponents. Moreover, the fact that it's only 0.1% of the runtime is usually [0] an artifact of the entire project being treated that way -- why write this the right way when it won't move the needle on the composite project?
[0] Yes, for LLMs we're closer to the bound than 1000x. Even there though, basic pytorch operations are often 2x slower than simple rewrites, better scheduling algorithms have a history of closer to 5x-10x gains, and all of that supposes you don't have further architectural improvements over time. Moreover, who's to say that legitimately fast tokenization doesn't unlock additional capabilities elsewhere which people have ignored because it was never close to viable?
brookst 19 hours ago [-]
If you're tokenizing to run a tiny SLM for routing purposes, it can be way more than 0.1%.
This is the "GPU driver optimizations don't matter because PC's sit idle at the desktop most of the time" mindset.
Xalutiono 9 hours ago [-]
AI now consumes TWh. This is more energy than a lot of countries on the whole planet.
If this increases efficiency, it has real impact. 0.1% is A LOT at scale.
yrds96 18 hours ago [-]
This can make a huge difference for local models on modest hardware
alansaber 22 hours ago [-]
It's pretty funny but then again, why not if it's as trivial to simplify as it appears
Cthulhu_ 9 hours ago [-]
I don't think this was particularly trivial, but I do think that thanks to AI assisted coding there's more capacity for making improvements that "don't seem worth it" at first or when you look at it as a percentage of total.
But look at e.g. Biome; optimizing the formatter didn't seem worth it for a long time because it only took <1s to format most files. But it's <1s times millions of files, billions of times a day when you add up every developer that used Prettier for their code formatting - it adds up.
And I'm convinced Biome triggered or was part of a bigger effort to convert JS based tools to native code. This saved time and energy, which in turn allows for faster and / or more feedback loops, which in turn allows faster turnaround cycles for software development (bet it human or LLM assisted), etc. It's a compound effect.
I don't know enough about tokenization or whatever to judge this one, but if it's 1000x as fast as it used to be, there will be less need to try and avoid or minimize tokenization which may lead to new applications.
NuclearPM 20 hours ago [-]
1000x faster on 0.1% of runtime = 0.1% saved. Amdahl remains undefeated.
Globally that’s ~50 GWh/yr, or ~5.7 MW continuous:
• 4,700 American homes
• a week of British tea
antonvs 13 hours ago [-]
Except, their benchmarking shows this can speed up time to first token by up to 10%.
0xnyn 1 days ago [-]
I had to stare at that chart for a minute just to let the numbers sink in. It's genuinely mind-bending, incredible ship OP
swiftcoder 23 hours ago [-]
So the question becomes, how many other parts of the inference pipeline have left 1000x optimization opportunities lying on the table?
fastball 22 hours ago [-]
The problem with the rest of inference is that changes are not trivially correct or incorrect, as they are with the tokenization layer.
michaelmior 19 hours ago [-]
Some changes certainly can be. If the model produces the exact same output for a fixed seed across a variety of inputs after a code change, I think it's reasonable to expect that the change is correct. There are also mathematical transformations that can be applied in some cases that are provably correct. (Not suggesting there's necessarily anything of this nature that will lead to 1,000x improvement though.)
janwas 13 hours ago [-]
hm, maybe not so trivially correct here. Do I understand correctly that incorrect results can happen as a result of a 42-bit hash collision?
That could happen after less than one MB of input, given the simple one-mul hash.
BTW throughput is measured for a 12 GiB file. Would be interesting to see the throughput for something more like 32 KiB, with cold start (token cache not yet populated).
nixon_why69 21 hours ago [-]
Eh, linear algebra changes are still easy to measure correctness, it's just that you're competing with 50 years of research for most of them, less low hanging fruit.
ProofHouse 22 hours ago [-]
the answer is many! This would take hours to write. Full teams and research on nearly every part. So many 'unlocks' coming.
parineum 22 hours ago [-]
I'm sure there's been a lot more effort put into the other, more consequential, portions of inference time.
fwip 1 days ago [-]
What sort of setups do people have that are bounded by the speed of the tokenizer?
marcelroed 1 days ago [-]
Author here! In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM.
From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) as reported by fastokens.[0]
I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache. If you have a long prefix that's been seen before (say a system prompt), the time for tokenizing that will be a large part of your TTFT. The tokenizer cache should be warmed up in this case, so the throughput for Gigatoken would be significantly higher than reported in the repo.
> I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache
Is this necessary? Tokenisation is deterministic, so for a hit/miss check you can lookup on (a hash of) the source text instead of the tokens. You only need the tokens once you're seeking for the exact token index having determined there is a hit. That means tokenisation can proceed in parallel with your cache query, and since these caches are distributed in production systems I imagine the query itself could be slow.
I'm not trying to undermine the utility, and this is obviously excellent work. Being able to tokenise faster on the client also seems useful (precise token counts for context pruning heuristics, instead of `chars / 4`), and on a phone your work translates directly to energy savings. I'm just curious about the cache lookup point.
marcelroed 21 hours ago [-]
It's usually not as binary as "hit" or "miss" with a prefix cache, and you need to know the token boundaries to know where the cache hit ends.
The current structures used for KV-caching in vLLM and SGLang work by chunking the KV-cache tokens into prefix trees, and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.
Again I have no idea what proprietary engines are doing, but this is why open source stuff needs to tokenize before cache lookup at least.
wren6991 19 hours ago [-]
> and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.
The hash just has to uniquely identify the contents. I still don't see what stops you from walking the chunk tree by chunks of characters instead of chunks of tokens, then lazily finding the token boundary once you've found the longest common chunk prefix and also (in parallel) tokenized the input.
fwip 1 days ago [-]
Very cool, thanks.
lostmsu 1 days ago [-]
Can't you tokenize in preloading on demand?
marcelroed 24 hours ago [-]
You can, but this usually results in sequences with padding/truncation, since you won't know how many tokens your inputs map to before you actually tokenize them. This also makes shuffling difficult.
In practice every training project I've worked on does tokenization in a separate data processing phase.
andersa 1 days ago [-]
Wait, since when does it matter whether something being hyper-optimized is useful? The computer going brrrr on an interesting problem is in itself the goal!
fwip 1 days ago [-]
That's fair, I just figure there are useful scenarios as well. Apologies if I came off as dismissive!
ac2u 24 hours ago [-]
It didn’t come off as dismissive to me. I was curious as well as to where such optimizing helps and knew that the answers to your question would help me discover use cases I didn’t think of
janalsncm 24 hours ago [-]
If you are training an LLM, you need to tokenize the text before it’s trained on. A lot of time this can be done in parallel with the GPU though.
I have spent way too much time waiting 10-15 minutes tokenizing my training dataset only for the run to crash over some minor bug after that. (If I was smarter, I’d test on a smaller batch first.)
antonvs 13 hours ago [-]
> … only for the run to crash over some minor bug after that
Ah, Python.
janalsncm 10 hours ago [-]
An example would be setting batch size too high causing OOM which isn’t really a python problem.
SnowflakeOnIce 20 hours ago [-]
I worked on a system a couple years ago with a BERT-based model (64M parameters) used for classification. The rest of the system could process data at gigabytes per second, and so here tokenization at a measly few megabytes per second really slowed things down. The model inference was more expensive than tokenization, but tokenization was still >10% of total runtime.
rhdunn 1 days ago [-]
It can be useful for checking input token usage before sending it to the model, e.g. preventing calls above a given token bound or grouping requests into batches.
It can also be used by the LLMs to provide the input and output token counts on the different APIs, though I'm not sure if this is how llama.cpp or other OpenAI-like APIs calculate the input/output tokens of a request.
charcircuit 1 days ago [-]
But are those bounded on the speed of tokenization?
avereveard 24 hours ago [-]
I've data where i cannot store metadata that i need to search semantically so i embed it on the fly at every search with static embedding and tokenizing was more than 99% of the cpu time. Granted that was due the naive implementation of the default tokenizer which was o^2 with document length and just switching to a proper scanner solved most of it without going to simd and whatnot, but still.
imperio59 1 days ago [-]
Pre-training data is pre-tokenized ahead of time before being used to not waste any GPU compute.
A massive speedup like this is a nice efficiency savings on some of these data pipelines for sure.
chocrates 22 hours ago [-]
Practically I would need to wait for hugging face models to adopt this? My harness tokenizer is just an estimate since the model tokenizes on my api calls?
hansvm 17 hours ago [-]
For the lazy among us (not me of course), is there a small number of core techniques which enabled this for even a single architecture and single CPU core?
mcpindex-ai 16 hours ago [-]
Tokenization is one of the most under appreciated and under optimized part of the agentic stack- not sure if this is truly production grade, and applicable across all hardware+stack combo but this for sure can help inspire a lot of that work. Good work!
sashank_1509 1 days ago [-]
This is really cool, great work!
nxpnsv 18 hours ago [-]
Surely it should be kilotoken
anonymousmoos 23 hours ago [-]
Quality software here.
phplovesong 14 hours ago [-]
Is tokenizing really the bottleneck? If we go from 20ms to 15ms does it really matter?
gnabgib 14 hours ago [-]
You sound like someone who used to write fastruby
phplovesong 13 hours ago [-]
Never heard of fast ruby. My point is tokenizing is rarely a bottleneck, as 99% of time is spent in inference.
So you speed up 1% of the pipeline by some factor, and the end result is unobservable for a human.
antonvs 12 hours ago [-]
Time to first token is observable by a human, and they’re reporting up to 10% reduction there.
Plus, inference is not the only place tokenization happens. This can make a big difference during development of ML models.
dmezzetti 24 hours ago [-]
Very interesting project! Are there benchmarks for the "compatibility mode" or are all the numbers for the Gigatoken API?
marcelroed 24 hours ago [-]
Numbers are for the Gigatoken API, but compatibility mode just means eating a bunch of Python overhead (creating lists, reading strings to bytes). You can expect a modest ~200-300x speedup with compatibility mode depending on how you use it.
robotresearcher 21 hours ago [-]
> a modest ~200-300x speedup with compatibility mode
marcelroed is modest, this speedup is not. Good work.
marcelroed 24 hours ago [-]
I can add some benchmarks for compatibility mode in the future. I have a little more juice to squeeze out of the Python interop though, so not quite ready for it yet.
zerolines 24 hours ago [-]
wow, best release all week.
semiinfinitely 24 hours ago [-]
quite excellent software
fenestella 8 hours ago [-]
[flagged]
Timmyzzz 10 hours ago [-]
[flagged]
wacjiomv 13 hours ago [-]
[dead]
saschag 13 hours ago [-]
[flagged]
luck7710 21 hours ago [-]
[dead]
kiaansaraiya 19 hours ago [-]
[dead]
tim_tihub 6 hours ago [-]
[dead]
histiq 8 hours ago [-]
[dead]
sghn 12 hours ago [-]
[flagged]
17 hours ago [-]
wacjiomv 13 hours ago [-]
[flagged]
marcelroed 13 hours ago [-]
The output tokens are identical in either case, but there are quite a few additional settings and formats that huggingface compat mode can generate. In general it also requires inputting Python lists of Python strings.
The Gigatoken API instead prefers input bytes or paths to iterate, and returns an Awkward array by default. Again, no difference in correctness.
vmware508 23 hours ago [-]
We should just rewrite everything in Rust, especially bloated Python code, and the world would be a better place. ;) Disclosure: I'm a Rust advocate!
minimaxir 23 hours ago [-]
Both the example libraries compared (tokenizers and tiktoken) are Rust-based with Python bindings. There's just a few levers in Rust that can speed it up even more particularly with LLM assistance as the AI Use Discloure here notes:
> Final profiling stages and the last ~4x worth of performance from eliminating branching and improving the pretoken cache hierarchy
SOLAR_FIELDS 23 hours ago [-]
1 year ago everyone would have called you insane for suggesting this. Now we all shrug and say yeah maybe we can do this and it’s actually a good idea?
22 hours ago [-]
weberer 22 hours ago [-]
We should rewrite all Rust code in Python. Not for any technical reason. I'm just sick of the Rust cult at this point.
Q: Did you just way over-optimize for a specific CPU and tokenizer? How is it so fast? No, I way over-optimized for every combination of these! The results are very consistent across CPUs (modern x86 and ARM), and across specific tokenizers.
The major improvements are in optimizing heavily an implementation that usually is outsourced to a Regex engine (pretokenization) using SIMD, minimizing branching and other tricks, as well as heavily optimizing caching of pretoken mappings (if a word has been seen before, look it up its encoded tokens efficiently). Caching is a very hard problem in this domain since the cache grows very quickly, and pretoken distributions are very long-tailed.
Finally, interactions with Python are minimized, and threads have minimal interactions with each other.
And screw all the 0.1% haters on here, this is great stuff.
Valuable not only for inference, but for training too (think proprietary datasets).
I would add, a single individual did this.
One person can make a difference :-)
How can I join this? Sounds interesting
Could I join this?
There is a serious research community on tokenization, and we are quite interested in this work.
As I said, It just might be the low information density approach to talking recently that seems off to me.
To explain a bit more, I kept reading and waiting for the penny drop but nothing.
“I cloned your repo” — okay then what happened? Nothing? I put my shoes on this morning.
“I know the tokrniziation community wants to absord tje lessons” okay? Tell me the point please! I know the AI community wants to understand the universe.
“These are useful ideas”. Yes.
Of course, you could argue that my comment falls in the same category in the sense that I am not actually contributing to the topic at hand but I am peeved with all low the information noise.
Hardware nowadays is so powerful, but our code so inefficient... I think most libraries/apps could easily be 10x-100x faster if we really try to optimize them.
The good thing is, that now with AI, we'll probably have the time to implement those optimizations rather quickly.
LLMs can help with the amount of test, and somewhat with the quality, but that's not quite enough for doing heavy optimizations in existing, deployed products, in a normal sprint somewhere.
In my case, I recently optimized for https://uxwizz.com the session playback: before, it was saving the entire recording in one row, with updates, and loading it client-side in one chunk. I asked thr AI to, instead, store the recorded chunks in separate rows in db, and when replaying, to stream those chunks as needed. It implemented it perfectly, and everything is much snappier and lighter now.
So, the idea was "store in chunks and stream the chunks", and the LLM does it. Yes, there were a few bugs, but those systems genetally either work or don't. So testing for a 30 minutes after was enough, and now the system is live, and a client actually told me in person yesterday he loved the playback improvements.
I said in another comment somewhere, now knowing how to code isn't important, what's important is that you know what to ask, and what to look for when testing.
Haha, yeah, product/executives will surely now see the benefits of optimizations instead of piling new features on top of new features with no cohesive idea about the design or architecture :)
Good models now already follow best practices when implementing, better than junior devs.
I wrote a bit about this, I call it "AI slap", lol:
https://x.com/XCSme/status/2079115230567686263?s=20
We're clearly thinking of very different "optimizations" here I think :) Do you have any concrete examples of this sort of optimizations you'd get automatically? In my experience, you get what you prompt for, if I don't include to think about performance, they won't think about performance, not sure what model would automatically consider things like that. Most of the time I use whatever SOTA OpenAI has on maximum reasoning level, fwiw.
If you ask to implement a custom dropdown that does something, it often comes with good spacing, aria-accessible tags, keyboard accessibility, etc. A junior dev wouldn't think of all of those.
Also, it will probably choose the right HTML elements to use for it (i.e. maybe the modern native popover functionality instead of implementing it with custom JS, which would indeed be more efficient and less code).
I am not saying it would add caching by default (even though, it might suggest that), but it's more likely to choose whatever the best options are and to use them as they should, including going around knowing limitations and gotchas.
Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
[0] https://github.com/crusoecloud/fastokens
Heck, even latency alone you can just reduce the standard deviation and get smoother flows. Little's law is a great callout here too, one of my favorite computer science principles.
If I have the same number of CPU cores and they all can do their work in half the time they can double the number of requests now
Tips & Tricks on parameters/settings?
What happens at peak? Do people have to wait now? Increase of latency?
Source: https://www.gartner.com/en/newsroom/press-releases/2026-07-2...
Latency can be just as important as overall throughput, especially for inference providers like Groq and Cerebras.
Have you considered publishing a rust crate as well? (If not, I volunteer.)
I don't think tokenization has ever been a meaningful bottleneck. JSON being fast falls into the same bucket much of the time. We spend way more energy on I/O and storage than we do on serialization and tokenization.
If you are concerned with economics and the environment, request batching would make a bigger impact. The most expensive part of this whole thing is GPU underutilization. You can save 50% with OAI right now if you can figure out how to make your workload fit the batch pattern. Do your users always need answers right now or can we afford to wait a few days in some cases? Tool calling doesn't "time out". Wall clock does not exist in the LLM. It took me a while to get used to this.
It showcases an especially ingenious scheme for escaping json strings, as well as the other tokens necessary to parse json.
Daniel Lemire - One of the authors at QCon 2019: https://www.youtube.com/watch?v=wlvKAT7SZIQ
Another Youtube video that explains it: https://www.youtube.com/watch?v=vd9J9PPmAMM
When tokenizing terabytes of text for your training corpus, the speedup here is probably doing real work in saving you time (and money?). You get a faster iteration cycle when figuring out and adjusting your datasets.
It will be nicer if the README focuses more on per-core performance.
About the actual algorithm - will something like matching in a perfect hash table help?
So much for "human programming is obsolete".
In the final stages of the project, AI was used to assist:
Implementing the user-facing API Widening of compatibility, for instance generalizing and porting the pretokenizer implementations to support more tokenizers, less interesting features like padding/truncation/unicode normalization Porting SIMD strategies between AVX512/AVX2/NEON Final profiling stages and the last ~4x worth of performance from eliminating branching and improving the pretoken cache hierarchy Refactoring and code reuse
I later found enough optimizations to reduce the generation speed all the way down to ~16ms. Do I need a word cloud generator that fast? No. But if I have a word cloud generator it's going to be as fast as possible dammit.
https://x.com/mitchellh/status/2074225453217505494
[0] Yes, for LLMs we're closer to the bound than 1000x. Even there though, basic pytorch operations are often 2x slower than simple rewrites, better scheduling algorithms have a history of closer to 5x-10x gains, and all of that supposes you don't have further architectural improvements over time. Moreover, who's to say that legitimately fast tokenization doesn't unlock additional capabilities elsewhere which people have ignored because it was never close to viable?
This is the "GPU driver optimizations don't matter because PC's sit idle at the desktop most of the time" mindset.
If this increases efficiency, it has real impact. 0.1% is A LOT at scale.
But look at e.g. Biome; optimizing the formatter didn't seem worth it for a long time because it only took <1s to format most files. But it's <1s times millions of files, billions of times a day when you add up every developer that used Prettier for their code formatting - it adds up.
And I'm convinced Biome triggered or was part of a bigger effort to convert JS based tools to native code. This saved time and energy, which in turn allows for faster and / or more feedback loops, which in turn allows faster turnaround cycles for software development (bet it human or LLM assisted), etc. It's a compound effect.
I don't know enough about tokenization or whatever to judge this one, but if it's 1000x as fast as it used to be, there will be less need to try and avoid or minimize tokenization which may lead to new applications.
Globally that’s ~50 GWh/yr, or ~5.7 MW continuous:
• 4,700 American homes
• a week of British tea
BTW throughput is measured for a 12 GiB file. Would be interesting to see the throughput for something more like 32 KiB, with cold start (token cache not yet populated).
From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) as reported by fastokens.[0]
I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache. If you have a long prefix that's been seen before (say a system prompt), the time for tokenizing that will be a large part of your TTFT. The tokenizer cache should be warmed up in this case, so the throughput for Gigatoken would be significantly higher than reported in the repo.
[0] https://github.com/crusoecloud/fastokens
Is this necessary? Tokenisation is deterministic, so for a hit/miss check you can lookup on (a hash of) the source text instead of the tokens. You only need the tokens once you're seeking for the exact token index having determined there is a hit. That means tokenisation can proceed in parallel with your cache query, and since these caches are distributed in production systems I imagine the query itself could be slow.
I'm not trying to undermine the utility, and this is obviously excellent work. Being able to tokenise faster on the client also seems useful (precise token counts for context pruning heuristics, instead of `chars / 4`), and on a phone your work translates directly to energy savings. I'm just curious about the cache lookup point.
The current structures used for KV-caching in vLLM and SGLang work by chunking the KV-cache tokens into prefix trees, and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.
Again I have no idea what proprietary engines are doing, but this is why open source stuff needs to tokenize before cache lookup at least.
The hash just has to uniquely identify the contents. I still don't see what stops you from walking the chunk tree by chunks of characters instead of chunks of tokens, then lazily finding the token boundary once you've found the longest common chunk prefix and also (in parallel) tokenized the input.
In practice every training project I've worked on does tokenization in a separate data processing phase.
I have spent way too much time waiting 10-15 minutes tokenizing my training dataset only for the run to crash over some minor bug after that. (If I was smarter, I’d test on a smaller batch first.)
Ah, Python.
It can also be used by the LLMs to provide the input and output token counts on the different APIs, though I'm not sure if this is how llama.cpp or other OpenAI-like APIs calculate the input/output tokens of a request.
A massive speedup like this is a nice efficiency savings on some of these data pipelines for sure.
So you speed up 1% of the pipeline by some factor, and the end result is unobservable for a human.
Plus, inference is not the only place tokenization happens. This can make a big difference during development of ML models.
marcelroed is modest, this speedup is not. Good work.
The Gigatoken API instead prefers input bytes or paths to iterate, and returns an Awkward array by default. Again, no difference in correctness.
> Final profiling stages and the last ~4x worth of performance from eliminating branching and improving the pretoken cache hierarchy