The AI chip market has been highly competitive in recent years, and various manufacturers are constantly launching new products and technologies in an attempt to gain a foothold in this field. And in this competition, there is a startup called Groq, which has recently attracted a lot of attention in the industry. Groq has launched a new AI chip, the LPU (Language Processing Unit), which claims to be the "strongest inference on the surface" - the inference speed of running large models on Groq is 10 times faster than that of NVIDIA GPUs, and the cost is only one-tenth of that. Is this really true? What makes groq's technology unique? And how will it impact the AI space? This article will reveal the secret for you.
What are Groq and LPUs?
Groq is an AI chip startup founded in 2016 with a founding team from Google's TPU (Tensor Processing Unit) project and rich experience in AI chip design. GroQ's goal was to create a chip built for AI inference that would surpass traditional GPUs and CPUs in terms of speed, cost, and power efficiency.
The LPU is Groq's first AI chip and the industry's first inference chip dedicated to natural language processing (NLP) and other sequential data. The LPU is designed with "software-defined hardware" in mind, i.e., a single-core configuration of compute and storage units, with all operations set up in software. This architecture, known as a TSP (Tensor Streaming Processor), is designed to be relatively simple from a hardware perspective, removing all unnecessary control logic and leaving all control to the software compiler, thereby optimizing chip area allocation and achieving higher computing power per unit area.
How fast is the Groq LPU?
The performance of the Groq LPU is impressive. According to data published by GroQ, the LPU has an integer (8-bit) operation speed of 750 TOPS (trillion operations per second) and a floating-point (16-bit) operation speed of 188 TFLOPS (trillion floating point operations per second). Nvidia's latest A100 GPU has an integer (8-bit) speed of 624 TOPS and a floating-point (16-bit) speed of 312 Tflops. This means that the LPU is 20% faster than the A100 on integer operations and 40% faster than the A100 on floating-point operations.
However, computing speed is not the only measure of the performance of an AI chip, but more important is inference speed, which is how quickly the chip can complete an AI task, such as generating a piece of text or recognizing a one**. In this regard, the performance of the Groq LPU is even more impressive. According to artificialanalysisAI's data, the Groq LPU is capable of processing about 430 tokens per second (the smallest unit of text), while NVIDIA's GPU can only process about 40 tokens per second. This means that LPUs are up to 10x faster than GPUs in terms of inference speed.
The reason why the inference speed of the Groq LPU is so fast is mainly due to its unique technical advantages. On the one hand, Groq LPUs don't require the same fast data transfer as NVIDIA GPUs. Unlike GPUs, which utilize high-bandwidth memory (HBM), the Groq LPU utilizes SRAM for data processing, which is approximately 20 times faster than the memory used by GPUs. This helps to avoid HBM shortages and reduce costs. On the other hand, a key advantage of the TSP architecture used by the Groq LPUs is that they reduce the frequency of loading data from memory, which not only helps to alleviate memory bandwidth bottlenecks, but also reduces power consumption and latency. At the heart of this architecture is a large MXM module containing 409,600 multipliers that leverages on-chip data parallel processing to deliver more than 1 teraops per square millimeter of compute density.
How will GroQ LPUs impact the AI space?
The emergence of the Groq LPU has undoubtedly brought a revolution to the AI field. As a chip designed for AI inference, LPUs can meet the needs of users in terms of speed and cost, especially in scenarios of large model inference, LPUs can provide lower latency and higher throughput, providing users with a smoother experience and higher efficiency. For example, in Q&A and dialogue scenarios, users experience little to no delay from asking a question to receiving an answer, and the delay of the first word output is only 0In 2 seconds, more than 500 words are all generated in about a second, while the same amount of content, NVIDIA GPUs take nearly 10 seconds to generate, and the output of the first word is measured in seconds. This speed advantage makes LPUs the king of AI inference.
The impact of the GroQ LPU extends beyond AI inference to AI innovation and applications. The Groq LPU supports inference through standard machine learning frameworks such as PyTorch and TensorFlow, and Groq also provides a compilation platform and localized deployment solutions, allowing users to compile their own applications with the Groq compiler for better performance and latency metrics based on specific scenarios. This flexibility and customizability makes it easier for users to develop and deploy their own AI applications, driving AI innovation and adoption. For example, in healthcare, finance, education, entertainment, and other fields, Groq LPUs can be leveraged to achieve more efficient AI solutions that bring more convenience and value to people's lives and work.
The emergence of the GroQ LPU has also brought huge challenges to NVIDIA GPUs. NVIDIA GPUs have always been the leaders in the AI chip market, and their GPUs have a wide range of applications and excellent performance in AI training and inference scenarios. However, as AI models continue to grow and become more complex, GPU performance and cost bottlenecks are becoming increasingly apparent. The GroQ LPU is designed to address the weaknesses of GPUs, which are difficult to compete with in terms of speed and cost for AI inference. If the Groq LPU can be widely recognized and adopted in the market, then it will have a serious impact on the market position of NVIDIA GPUs. It remains to be seen whether NVIDIA GPUs will be able to meet this challenge.
The Groq LPU is a chip designed for AI inference, and its advantages in speed and cost make it a revolution in AI inference, a nightmare for NVIDIA GPUs. The emergence of the Groq LPU has not only brought a revolution to the AI field, but also brought new possibilities to the innovation and application of AI. The success of GroQ LPUs in the market also depends on their interaction and collaboration with users and partners. We'll keep an eye on the development of the Groq LPU and bring you the latest coverage.