The End of 'Thinking for Minutes': Taalas Targets 17,000 Tokens/Second with Custom Silicon

Two-and-a-half-year-old startup Taalas published “The Path to Ubiquitous AI,” a technical essay that earned 107 points on Hacker News (as of February 20, 2026). The core argument is direct: AI coding assistants are far from ubiquitous due to their latency and cost structure — and this problem cannot be solved at the software level.

When Coding Assistants Become Unusable

Taalas opens with a problem familiar to most developers:

“Coding assistants can think for several minutes, breaking programmers’ flow state and impeding effective human-AI collaboration.”

Flow state, as defined by psychologist Mihaly Csikszentmihalyi, is the peak-performance mode of focused concentration. In programming, maintaining flow state is directly correlated with productivity. Waiting several minutes for an AI response reliably destroys it.

For agentic systems, the problem compounds:

“Autonomous agent systems require millisecond-level latency. Human-paced responses are completely inadequate.”

When an agent calls other agents across a multi-step parallel task, latency accumulates at each step. An agent that takes 10 seconds per step will introduce 100 seconds of waiting across a 10-step task chain.

The Structural Problem with Inference Hardware

The root cause, according to Taalas, lies in the architecture of modern inference hardware.

The 1,000x Gap Between Memory and Compute

Current AI inference systems store model parameters (weights) in DRAM, while computation happens on GPU chips. This architecture creates a critical bottleneck:

• On-chip memory (SRAM): Fast, but limited capacity
• DRAM: High capacity, but 1,000x slower than on-chip memory

Generating each token requires transferring model weights from DRAM to the GPU. For models with 17B to 700B parameters, this transfer cost determines virtually all of the inference speed.

The Physical Reality Behind “Cloud AI”

Taalas makes explicit what cloud abstraction hides:

“Deploying modern models requires room-sized supercomputers, hundreds of kilowatts of power consumption, liquid cooling, advanced packaging, stacked memory, complex I/O, and miles of cables.”

This is not hyperbole. Every API call to GPT-4 or Claude triggers this scale of infrastructure activity in a data center. High costs are a direct consequence — and they represent the primary barrier to truly ubiquitous AI.

Taalas’s Solution: Integrating Storage and Compute

Taalas addresses the separation of storage and compute directly.

”Hardcore Models” via Custom Silicon

Taalas has developed a platform that can receive any AI model and convert it to custom silicon within two months. The resulting “Hardcore Models” are:

• An order of magnitude faster (10x or more) than software-based implementations
• An order of magnitude cheaper (less than 1/10 the cost)
• Lower power consumption

Target: 17,000 Tokens/Second

Current major inference servers (NVIDIA A100/H100 GPU clusters) deliver roughly 100–150 tokens/second (varying by model size and parallelism). Taalas targets 17,000 tokens/second — approximately 100–170x faster.

At this throughput, a GPT-4-class model would generate 17,000 characters per second. The “minutes of waiting” from coding assistants would shrink to seconds. Multi-agent chains would execute at speeds that don’t disrupt human workflow.

Inference Costs and AI Democratization

The significance of Taalas’s argument goes beyond technical performance.

Current AI inference costs are prohibitive for individual developers, SMBs, and organizations in developing economies at meaningful scale. The cost of sustained GPT-4 or Claude 3.5 Sonnet-equivalent usage for individuals is simply impractical for most of the world.

A 10x or greater reduction in inference costs changes this fundamentally. If AI coding assistants become accessible without per-query anxiety about costs, software development democratization moves into a different dimension entirely.

Assessing the Claims

HN commenters raised reasonable skepticism:

• “Two months to custom silicon — does this require models to be locked to specific architectures?”
• “17k tokens/sec for which model size? Small models already approach this.”

These are valid questions. However, Taalas presenting concrete throughput targets while operating in the custom silicon space is notable in itself.

Anthropic, Google, and Meta are all making substantial investments in custom silicon for inference. That a startup is competing in this space with a specific approach and numbers warrants attention.

Conclusion

The problem Taalas identifies is real. Most developers would readily agree that AI coding assistants are “too slow and too expensive.” Whether the custom silicon approach can reach 17,000 tokens/second remains to be seen, but the diagnosis — inference speed and cost are the primary bottleneck to ubiquitous AI — is accurate.

As competition in this space intensifies through 2026 and beyond, inference efficiency players like Taalas are worth tracking.

---

Source: Taalas - The Path to Ubiquitous AI