Reduce LLM Context Costs by 90% with Lossless Memory

The problem with growing context

Every additional turn in an AI agent conversation increases token cost linearly. A 20-turn debugging session can easily blow past 30k tokens of context — and at GPT-4-class pricing, that adds up fast. The standard mitigations all have trade-offs:

• Summarization loses information irreversibly. The agent forgets the exact wording of earlier user preferences.
• Embedding retrieval only helps when you know what to retrieve; it doesn't compress the working set.
• Sliding windows drop turns entirely, breaking tool-call continuity.

What we actually want is compression that is bit-exact on decompression — the same property ZIP gives you for files, but tuned for the short, structured turns that agents produce.

The Memorizer-LM approach

Forge's Memorizer model is a 2-layer Transformer encoder with a discrete bottleneck. The pipeline:

1. Tokenize the input text with a byte-level BPE tokenizer (vocab 8192, max 256 tokens).
2. Encode with two Transformer layers (hidden 512, 8 heads).
3. Bottleneck through a linear projection to 4096 logits, then Gumbel-Softmax with straight-through estimator. Keep the first 4 indices — that's your compressed representation.
4. Decode non-autoregressively: prepend the 4 code embeddings, fill the rest with [MASK] tokens, and predict all original token IDs in one forward pass.

The training loss combines token cross-entropy with a commitment loss (MSE, weight 0.25) on the bottleneck embeddings, exactly as specified in the VQ-VAE literature.

The cost math

Consider a 20-turn agent conversation at an average of 150 tokens per turn = 3,000 tokens of context. At GPT-4o pricing of ~$5/M input tokens, every single user request costs ~$0.015 just for the context.

With Forge, you compress all but the last 2 turns into 4 integers each = 8 × 4 = 32 bytes of state. The 2 retained turns add ~300 tokens. So your effective context drops from 3,000 to ~300 — a 90% reduction in input token cost.

Across 1M requests per month, that's $15k → $1.5k. The compression itself is free.

Why "lossless" matters

The crucial difference between Memorizer and a summarizer is reversibility. When the agent decides it needs the exact text of turn 7 — say, a JSON payload the user pasted — it can decompress codes[7] and get the literal bytes back. No hallucinated summary. No rephrased preference.

This is especially important for:

• Tool arguments that must round-trip exactly (URLs, IDs, code snippets)
• Compliance scenarios where you need to prove what was said
• Debugging — you can always see the original conversation

Performance

The encoder is ~12ms on a single GPU and ~80ms on an edge CPU (with the int8 ONNX export). Decompression is ~18ms on GPU and ~120ms on edge CPU. Both fit comfortably within the Vercel Edge function budget.

Try it

Open the playground, paste a paragraph, hit Compress, and watch 4 integers come back. Hit Decompress and verify the round-trip. Then read the docs to wire it into your agent.