Can ASICs Overtake NVIDIA in AI Inference?

Etched’s Sohu Chip Promises Incredible Price/Performance for Transformer Architectures

Sohu, the world’s first specialized ASIC for transformers. Image Credit: Etched.

Nvidia has been the hottest stock on Wall Street, capturing the imagination of investors with its skyrocketing growth. The company’s shares have surged 155% this year alone, making Nvidia a key player in the tech industry. Its GPUs are fundamental to modern AI, especially in AI infrastructure and data centers, where they hold more than 90% of the market share. This hot commodity status has also made NVIDIA GPUs increasingly difficult to obtain, often feeling as elusive as the last ice cube at a summer barbecue. However, a new development in AI hardware could challenge Nvidia’s dominance: the rise of AI-inference ASICs (Application-Specific Integrated Circuits).

ASICs are specialized chips designed for specific computational tasks, and they can offer a significant performance boost over general-purpose GPUs. Etched’s Sohu chip, announced yesterday, is an astounding example, designed specifically for transformer models, which are at the heart of advanced AI applications such as ChatGPT and Stable Diffusion 3. By focusing solely on transformer inference, the Sohu chip achieves remarkable efficiency and speed, processing over 500,000 tokens per second for models like Llama 70B - which they claim to be an order of magnitude faster and cheaper than even NVIDIA’s next-generation Blackwell (B200) GPUs.

This shift towards ASICs in AI inference hardware parallels the transformation seen in cryptocurrency mining. Initially, GPUs were used to mine Bitcoin due to their parallel processing capabilities. However, as mining difficulty increased, ASICs took over, offering superior efficiency and performance. This transition rendered GPUs obsolete for Bitcoin mining. Similarly, as long as transformer architectures remain supreme, the Sohu chip could make GPUs less relevant for AI inference tasks by offering specialized performance that general-purpose GPUs cannot match.

Transformers have become the backbone of modern AI due to their efficiency and scalability. From GPT-2 to Llama-3, these models have nearly identical architectures with minor tweaks like SwiGLU activations and RoPE encodings used across various applications, including LLMs, embedding models, image inpainting, and video generation. This similarity has allowed AI labs to invest heavily in optimizing GPUs for transformer models, creating a vast ecosystem of optimized software libraries like TensorRT-LLM, vLLM, and Huggingface TGI​.

However, GPUs have inherent limitations. The NVIDIA H200, for example, boasts 989 TFLOPS of FP16/BF16 compute (for comparison, this is twice as fast as the world’s fastest supercomputer in 2007 - in a single chip!), but according to Etched, only about 3.3% of its transistors are dedicated to matrix multiplication - the core operation for transformer models. This is because the majority of a GPU’s real estate is devoted to programmability, which supports a wide range of AI models but results in inefficient use of silicon for any single task.

In contrast, ASICs like Sohu are designed with a single purpose in mind, eliminating the need for extensive control flow logic and maximizing the number of math blocks. Sohu achieves over 90% FLOPS utilization, compared to approximately 30% for GPUs running transformer models. This high efficiency allows Sohu to outperform GPUs dramatically (20x according to their claims), processing massive amounts of data quickly and cost-effectively​.

Despite the promise of ASICs, GPUs remain essential due to their versatility. Nvidia’s CUDA programming language and its ecosystem of software libraries are deeply integrated into AI development, supporting a wide range of applications beyond transformers, most notably in AI training. This broad applicability ensures that GPUs will continue to play a critical role in AI, even as specialized ASICs rise. So, I wouldn’t bet against Nvidia just yet.

An interesting player in this space is Groq, which offers a middle ground between GPUs and ASICs. Groq’s chips are designed to be more specialized than GPUs but more flexible than ASICs. This intermediate approach allows for high performance in specific AI tasks while maintaining some level of versatility beyound transformer models, potentially bridging the gap between the general-purpose nature of GPUs and the specialized efficiency of ASICs.

The future of AI hardware is likely to involve an interplay between these different types of chips. While ASICs like Etched’s Sohu chip offer astounding performance for transformer models, GPUs’ flexibility ensures their continued importance. Groq’s intermediate solution further adds to the complexity, providing an additional layer of choice for AI developers.

Picture stepping into an AI-generated virtual reality where you navigate epic adventures, engage in dynamic dialogues, and influence the plot’s direction with your decisions. ASICs like the Sohu chip can offer the extraordinary efficiency, speed and cost reduction needed to power these experiences, enabling seamless and instantaneous content creation.

However, the incredible pace of AI software innovation could render algorithm-specific chips obsolete in a heartbeat. The risk is monumental, but so is the potential reward.

As AI inference hardware evolves, the possibilities will expand beyond our wildest dreams, making personalized, interactive AI-generated worlds an everyday reality.

Are you ready to embrace this brave new world?