📊 Tokenization & Statistical Compression Playground¶
Estimated time to read: 8 minutes
Welcome to the interactive playground for LAB-01 (Tokenization Compression).
This notebook allows you to:
- Interactively tokenize text using different algorithms (BPE vs WordPiece).
- Compare models like GPT-4o (
o200k_base), GPT-4 (cl100k_base), GPT-2 (gpt2), and BERT (bert-uncased). - Compute token footprint, compression ratios, and cost implications across different languages and structured data domains.
0. Environment Setup¶
If you are running in Google Colab, this cell will automatically clone the repository and install all required dependencies (like tiktoken and transformers). If running locally, it does nothing.
In [1]:
Copied!
import os
import sys
import sys
from pathlib import Path
try:
import google.colab
IN_COLAB = True
except ImportError:
IN_COLAB = False
if IN_COLAB:
print("🤖 Running in Google Colab. Setting up environment...")
!pip install -q numpy scikit-learn matplotlib loguru
# Clone repository to access custom local source modules
if not os.path.exists("production-ai-systems"):
!git clone https://github.com/RogerioLS/production-ai-systems.git
sys.path.append("production-ai-systems")
print("🎉 Colab environment successfully initialized!")
# Ensure project root is in the path when running locally
if not IN_COLAB:
# Localiza dinamicamente a raiz do repositório procurando por 'pyproject.toml'
current_dir = Path.cwd()
root_dir = current_dir
for parent in [current_dir] + list(current_dir.parents):
if (parent / "pyproject.toml").exists() or (parent / "projects").exists():
root_dir = parent
break
if str(root_dir) not in sys.path:
sys.path.insert(0, str(root_dir))
print(f"📁 Project root added to sys.path: {root_dir}")
import os
import sys
import sys
from pathlib import Path
try:
import google.colab
IN_COLAB = True
except ImportError:
IN_COLAB = False
if IN_COLAB:
print("🤖 Running in Google Colab. Setting up environment...")
!pip install -q numpy scikit-learn matplotlib loguru
# Clone repository to access custom local source modules
if not os.path.exists("production-ai-systems"):
!git clone https://github.com/RogerioLS/production-ai-systems.git
sys.path.append("production-ai-systems")
print("🎉 Colab environment successfully initialized!")
# Ensure project root is in the path when running locally
if not IN_COLAB:
# Localiza dinamicamente a raiz do repositório procurando por 'pyproject.toml'
current_dir = Path.cwd()
root_dir = current_dir
for parent in [current_dir] + list(current_dir.parents):
if (parent / "pyproject.toml").exists() or (parent / "projects").exists():
root_dir = parent
break
if str(root_dir) not in sys.path:
sys.path.insert(0, str(root_dir))
print(f"📁 Project root added to sys.path: {root_dir}")
📁 Project root added to sys.path: /home/runner/work/production-ai-systems/production-ai-systems
1. Setup & Imports¶
Let's import our custom libraries and standard requirements. Make sure you are running this in your active Conda environment.
In [2]:
Copied!
# Ensure project root is in the path when running locally
if not IN_COLAB:
sys.path.append(os.path.abspath("../../../"))
from projects.a_llm_basics.src.lab_01_tokenization.tokenizer_math import (
HuggingFaceTokenizer,
TiktokenTokenizer,
TokenizationAnalyzer,
)
print("✅ Tokenizer modules imported successfully!")
# Ensure project root is in the path when running locally
if not IN_COLAB:
sys.path.append(os.path.abspath("../../../"))
from projects.a_llm_basics.src.lab_01_tokenization.tokenizer_math import (
HuggingFaceTokenizer,
TiktokenTokenizer,
TokenizationAnalyzer,
)
print("✅ Tokenizer modules imported successfully!")
/opt/hostedtoolcache/Python/3.11.16/x64/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
✅ Tokenizer modules imported successfully!
2. Loading Tokenizers¶
We load wraps for different models representing different vocabularies and base algorithms:
- tiktoken GPT-4o (
o200k_base): Expanded BPE vocabulary of 200,000 tokens. - tiktoken GPT-4 (
cl100k_base): Standard BPE vocabulary of 100,000 tokens. - tiktoken GPT-2 (
gpt2): Legacy BPE vocabulary of 50,000 tokens. - Hugging Face BERT (
bert-uncased): WordPiece algorithm vocabulary of 30,000 tokens.
In [3]:
Copied!
tokenizers = [
TiktokenTokenizer("o200k_base"),
TiktokenTokenizer("cl100k_base"),
TiktokenTokenizer("gpt2"),
HuggingFaceTokenizer("google-bert/bert-base-uncased")
]
analyzer = TokenizationAnalyzer(tokenizers)
print("✅ All tokenizers initialized!")
tokenizers = [
TiktokenTokenizer("o200k_base"),
TiktokenTokenizer("cl100k_base"),
TiktokenTokenizer("gpt2"),
HuggingFaceTokenizer("google-bert/bert-base-uncased")
]
analyzer = TokenizationAnalyzer(tokenizers)
print("✅ All tokenizers initialized!")
2026-09-30 21:12:15 | INFO | projects.a_llm_basics.src.lab_01_tokenization.tokenizer_math:__init__:95 - Loaded TiktokenTokenizer for 'o200k_base' using encoding 'o200k_base'.
2026-09-30 21:12:16 | INFO | projects.a_llm_basics.src.lab_01_tokenization.tokenizer_math:__init__:95 - Loaded TiktokenTokenizer for 'cl100k_base' using encoding 'cl100k_base'.
2026-09-30 21:12:17 | INFO | projects.a_llm_basics.src.lab_01_tokenization.tokenizer_math:__init__:95 - Loaded TiktokenTokenizer for 'gpt2' using encoding 'gpt2'.
2026-09-30 21:12:17 | INFO | projects.a_llm_basics.src.lab_01_tokenization.tokenizer_math:__init__:121 - Loading HuggingFaceTokenizer for 'google-bert/bert-base-uncased'...
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-09-30 21:12:17 | INFO | projects.a_llm_basics.src.lab_01_tokenization.tokenizer_math:__init__:128 - Successfully loaded HuggingFaceTokenizer 'google-bert/bert-base-uncased'.
✅ All tokenizers initialized!
3. Interactive Text Tokenization Analyzer¶
Enter a custom string below to benchmark the tokenizers. Look at the token count and the compression ratio (UTF-8 Bytes / Token).
A higher compression ratio means the tokenizer compresses the text into fewer tokens, translating directly to lower API costs and shorter context windows.
In [4]:
Copied!
# Enter your custom text to test below
custom_text = "IA para produção exige robustez, pipelines claros e monitoramento constante em ambientes de nuvem. 🚀"
print(f"Original text: {custom_text}")
print(f"Total length: {len(custom_text)} characters | {len(custom_text.encode('utf-8'))} UTF-8 bytes\n")
# Perform analysis
results = analyzer.analyze_corpus({"interactive_input": custom_text})
print("| Tokenizer Name | Tokens Count | Compression Ratio (Bytes/Token) |")
print("|----------------|--------------|---------------------------------|")
for t_name, metrics in results["interactive_input"].items():
print(f"| {t_name:<14} | {metrics['token_count']:<12} | {metrics['compression_ratio']:<30.3f} |")
# Enter your custom text to test below
custom_text = "IA para produção exige robustez, pipelines claros e monitoramento constante em ambientes de nuvem. 🚀"
print(f"Original text: {custom_text}")
print(f"Total length: {len(custom_text)} characters | {len(custom_text.encode('utf-8'))} UTF-8 bytes\n")
# Perform analysis
results = analyzer.analyze_corpus({"interactive_input": custom_text})
print("| Tokenizer Name | Tokens Count | Compression Ratio (Bytes/Token) |")
print("|----------------|--------------|---------------------------------|")
for t_name, metrics in results["interactive_input"].items():
print(f"| {t_name:<14} | {metrics['token_count']:<12} | {metrics['compression_ratio']:<30.3f} |")
2026-09-30 21:12:17 | INFO | projects.a_llm_basics.src.lab_01_tokenization.tokenizer_math:analyze_corpus:188 - Analyzing category 'interactive_input'...
Original text: IA para produção exige robustez, pipelines claros e monitoramento constante em ambientes de nuvem. 🚀 Total length: 100 characters | 105 UTF-8 bytes | Tokenizer Name | Tokens Count | Compression Ratio (Bytes/Token) | |----------------|--------------|---------------------------------| | o200k_base | 21 | 5.000 | | cl100k_base | 28 | 3.750 | | gpt2 | 30 | 3.500 | | google-bert/bert-base-uncased | 30 | 3.500 |
4. Code & Character Footprint (Visual Comparison)¶
Let's see what the tokens actually look like. We will print the list of token IDs and their decoded string representations to understand where BPE splits text (like splitting Portuguese syllables, structured JSON, or finance digits).
In [5]:
Copied!
test_word = "anticonstitucionalmente"
print(f"Comparing tokenization segments for word: '{test_word}'\n")
for t in tokenizers:
ids = t.encode(test_word)
# Decode each token individually to visualize splits
decoded_parts = []
for token_id in ids:
try:
part = t.decode([token_id])
decoded_parts.append(part)
except:
decoded_parts.append("")
print(f"Tokenizer: {t.name:<15} | Tokens: {len(ids)} | Split: {decoded_parts}")
test_word = "anticonstitucionalmente"
print(f"Comparing tokenization segments for word: '{test_word}'\n")
for t in tokenizers:
ids = t.encode(test_word)
# Decode each token individually to visualize splits
decoded_parts = []
for token_id in ids:
try:
part = t.decode([token_id])
decoded_parts.append(part)
except:
decoded_parts.append("")
print(f"Tokenizer: {t.name:<15} | Tokens: {len(ids)} | Split: {decoded_parts}")
Comparing tokenization segments for word: 'anticonstitucionalmente' Tokenizer: o200k_base | Tokens: 5 | Split: ['ant', 'icon', 'stit', 'ucional', 'mente'] Tokenizer: cl100k_base | Tokens: 6 | Split: ['ant', 'icon', 'stit', 'uc', 'ional', 'mente'] Tokenizer: gpt2 | Tokens: 8 | Split: ['ant', 'icon', 'st', 'it', 'uc', 'ional', 'ment', 'e'] Tokenizer: google-bert/bert-base-uncased | Tokens: 8 | Split: ['anti', '##con', '##sti', '##tu', '##cion', '##al', '##ment', '##e']
5. Domain Corpus Analysis¶
Let's run a benchmark across various specialized text corpora to verify general tokenizer performance trends.
In [6]:
Copied!
corpus = {
"English (Plain)": "Large Language Models learn statistical distributions of text to predict next tokens.",
"Portuguese (PT-BR)": "Modelos de Linguagem de Grande Porte aprendem distribuições estatísticas de texto para prever os próximos tokens.",
"Structured JSON": '{"model": "gpt-4o", "parameters": {"temperature": 0.7, "top_p": 0.9}, "tags": ["llm", "prod"]}',
"Numeric / Financial": "$1,245,678.90 + $98,765.43 - $43,210.00 = $1,301,234.33",
"Emojis & Special": "🤖🚀🧠✨🔥🌈💻🛠️"
}
analysis = analyzer.analyze_corpus(corpus)
for domain, tok_results in analysis.items():
print(f"\nDomain: {domain}")
print(f"UTF-8 Bytes: {len(corpus[domain].encode('utf-8'))}")
print("-" * 50)
for tok_name, metrics in tok_results.items():
print(f" - {tok_name:<14} : {metrics['token_count']} tokens ({metrics['compression_ratio']:.3f} B/T)")
corpus = {
"English (Plain)": "Large Language Models learn statistical distributions of text to predict next tokens.",
"Portuguese (PT-BR)": "Modelos de Linguagem de Grande Porte aprendem distribuições estatísticas de texto para prever os próximos tokens.",
"Structured JSON": '{"model": "gpt-4o", "parameters": {"temperature": 0.7, "top_p": 0.9}, "tags": ["llm", "prod"]}',
"Numeric / Financial": "$1,245,678.90 + $98,765.43 - $43,210.00 = $1,301,234.33",
"Emojis & Special": "🤖🚀🧠✨🔥🌈💻🛠️"
}
analysis = analyzer.analyze_corpus(corpus)
for domain, tok_results in analysis.items():
print(f"\nDomain: {domain}")
print(f"UTF-8 Bytes: {len(corpus[domain].encode('utf-8'))}")
print("-" * 50)
for tok_name, metrics in tok_results.items():
print(f" - {tok_name:<14} : {metrics['token_count']} tokens ({metrics['compression_ratio']:.3f} B/T)")
2026-09-30 21:12:17 | INFO | projects.a_llm_basics.src.lab_01_tokenization.tokenizer_math:analyze_corpus:188 - Analyzing category 'English (Plain)'...
2026-09-30 21:12:17 | INFO | projects.a_llm_basics.src.lab_01_tokenization.tokenizer_math:analyze_corpus:188 - Analyzing category 'Portuguese (PT-BR)'...
2026-09-30 21:12:17 | INFO | projects.a_llm_basics.src.lab_01_tokenization.tokenizer_math:analyze_corpus:188 - Analyzing category 'Structured JSON'...
2026-09-30 21:12:17 | INFO | projects.a_llm_basics.src.lab_01_tokenization.tokenizer_math:analyze_corpus:188 - Analyzing category 'Numeric / Financial'...
2026-09-30 21:12:17 | INFO | projects.a_llm_basics.src.lab_01_tokenization.tokenizer_math:analyze_corpus:188 - Analyzing category 'Emojis & Special'...
Domain: English (Plain) UTF-8 Bytes: 85 -------------------------------------------------- - o200k_base : 13 tokens (6.538 B/T) - cl100k_base : 13 tokens (6.538 B/T) - gpt2 : 13 tokens (6.538 B/T) - google-bert/bert-base-uncased : 14 tokens (6.071 B/T) Domain: Portuguese (PT-BR) UTF-8 Bytes: 117 -------------------------------------------------- - o200k_base : 22 tokens (5.318 B/T) - cl100k_base : 28 tokens (4.179 B/T) - gpt2 : 40 tokens (2.925 B/T) - google-bert/bert-base-uncased : 36 tokens (3.250 B/T) Domain: Structured JSON UTF-8 Bytes: 94 -------------------------------------------------- - o200k_base : 41 tokens (2.293 B/T) - cl100k_base : 40 tokens (2.350 B/T) - gpt2 : 43 tokens (2.186 B/T) - google-bert/bert-base-uncased : 53 tokens (1.774 B/T) Domain: Numeric / Financial UTF-8 Bytes: 55 -------------------------------------------------- - o200k_base : 31 tokens (1.774 B/T) - cl100k_base : 31 tokens (1.774 B/T) - gpt2 : 31 tokens (1.774 B/T) - google-bert/bert-base-uncased : 33 tokens (1.667 B/T) Domain: Emojis & Special UTF-8 Bytes: 34 -------------------------------------------------- - o200k_base : 17 tokens (2.000 B/T) - cl100k_base : 23 tokens (1.478 B/T) - gpt2 : 24 tokens (1.417 B/T) - google-bert/bert-base-uncased : 1 tokens (34.000 B/T)