A comprehensive comparative analysis of HBM memory technology.
HBM, a breakthrough 3D memory technology, uses advanced packaging to stack multiple DRAM chips vertically, significantly increasing memory bandwidth and reducing power consumption.
Achieve large-capacity, high-bandwidth storage to meet the stringent memory requirements of high-performance computing, artificial intelligence, and other fields.
By employing advanced packaging technologies such as through-silicon vias (TSVs) and micro-bumps, HBM breaks the bottleneck of traditional memory bandwidth and power consumption.
The integrated package with the GPU reduces the signal transmission path, reduces latency, and further improves performance. As a high-performance DRAM based on 3D stacking technology, it breaks the bottleneck of memory bandwidth and power consumption. HBM (High Bandwidth Memory) is a combination of high-capacity and high-bandwidth DDR arrays that stack multiple DRAM chips with advanced packages (such as TSV through-silicon vias and micro-bumps) and package them together with GPUs.
HBM technology enables a compact connection to the computing chip through the same "interposer" intermediate layer as the processor, which minimizes chip area and significantly reduces data transfer time.
In addition, the HBM is 3D stacked using the TSV process, which effectively improves bandwidth, reduces power consumption, and achieves a higher level of integration. The compact connection to the computing chip is achieved through the same "interposer" intermediate layer as the processor, which not only saves the chip area on the one hand, but also significantly reduces the data transmission time on the other hand. In addition, the HBM is 3D stacked using the TSV process, which not only significantly improves the bandwidth, but also reduces power consumption and achieves a higher level of integration.
HBM performance far exceeds GDDR, making it an ideal solution for current GPU storage units. Compared to GDDR, HBM has faster speed, higher bandwidth, and lower power consumption. HBM delivers up to 460GB of bandwidth, more than four times the bandwidth of GDDR, while consuming half the power of GDDR. HBM has become an ideal choice for current GPU memory units due to its superior performance, and is widely used in high-performance computing, artificial intelligence, graphics processing and other fields. The performance far exceeds GDDR, making it an ideal solution for current GPU memory units. GPU memory generally uses GDDR or HBM solutions, but HBM performance far exceeds GDDR.
HBM memory, higher bandwidth and stronger performance.
HBM memory is an upgraded version of GDDR5 memory, which has greatly improved memory bit width, clock frequency, and memory bandwidth.
Video memory bit width: HBM is 4 times that of GDDR5.
The bit width of the HBM memory is 1024-bit, which is 4 times the 32-bit of GDDR5 memory, which means that the HBM memory can transmit more data at the same time.
Clock frequency: HBM is lower than GDDR5
The clock frequency of the HBM memory is 500MHz, which is lower than the 1750MHz of the GDDR5 memory. However, since the bit width of HBM memory is 4 times that of GDDR5 memory, the actual bandwidth of HBM memory is much higher than that of GDDR5 memory.
Memory bandwidth: HBM is much higher than GDDR5
The bandwidth of a stack of HBM memory is greater than 100 Gb s, while the bandwidth of a chip in GDDR5 memory is only 25 Gb s. As a result, the data transfer rate of HBM memory is much higher than that of GDDR5 memory.
Summary: HBM memory is an upgraded version of GDDR5 video memory, with higher bandwidth and stronger performance. AMD data, in terms of memory bit width, GDDR5 is 32-bit, and HBM is four times that of it, reaching 1024-bit; From the perspective of clock frequency, HBM is 500MHz, which is much smaller than GDDR5's 1750MHz; From the perspective of memory bandwidth, a stack of HBM is greater than 100GB s, while a chip of GDDR5 is only 25GB s, so the data transfer rate of HBM is much higher than that of GDDR5.
HBM: a high-bandwidth, high-capacity, low-power GPU storage solution.
HBM (High Bandwidth Memory) is a high-bandwidth, high-capacity, low-power GPU memory technology. Compared to traditional GDDR5 memory, HBM offers the following advantages:
Smaller chip area: The HBM chip area is only one-third of the GDDR5 chip area, which allows the HBM to achieve greater capacity in a smaller space.
Higher bandwidth: HBM has a bandwidth of up to 1TBS, which is 8 times that of GDDR5. This enables HBMs to meet the bandwidth requirements of high-performance graphics processors.
Lower power consumption: HBM consumes only half the power of GDDR5, making HBM more promising in power-constrained environments such as mobile devices and laptops.
Therefore, HBM is seen as an ideal solution for GPU storage units. At present, HBM has been widely used in high-performance graphics processors, artificial intelligence accelerators and other fields. The HBM is packaged in a GPU, which greatly reduces the space of the graphics card PCB, and the GDDR5 chip area is three times that of the HBM chip, which means that the HBM can achieve more capacity in a smaller space. Therefore, HBM can achieve high bandwidth and high capacity while saving chip area and power consumption, and is regarded as an ideal solution for GPU memory cells.
HBM has become the standard configuration of AI servers to meet the massive computing power requirements.
The rise of large AI models has created massive computing power requirements, which require higher chip memory capacity and transmission bandwidth.
HBM 3 provides up to 460GB of transfer bandwidth, which is 3-4 times that of GDDR6, meeting the high bandwidth requirements of AI servers.
HBM 3 can have a capacity of up to 32GB per machine, which is 4-8 times that of GDDR6, meeting the large-capacity memory requirements of AI servers.
HBM 3 consumes less power and generates less heat, making it ideal for high-density AI server deployments. Driven by the demand for GPUs, HBM has now become the standard equipment for AI servers. The rise of large AI models has given rise to massive computing power requirements, and the significant increase in data processing capacity and transmission rate has made AI servers have higher requirements for chip memory capacity and transmission bandwidth.
HBM (high-bandwidth memory) has become the standard configuration of GPUs in AI servers due to its advantages of high bandwidth, high capacity, low latency, and low power consumption. A number of chips launched by NVIDIA for AI training, A100, H100 and H200, all use HBM video memory:
The A100 uses 80GB HBM2 video memory with a peak bandwidth of up to 16tb/s。
The H100 uses 80GB HBM3 video memory with a peak bandwidth of up to 32tb/s。
The H200 uses 96GB HBM3 video memory with a peak bandwidth of up to 512tb/s。
The application of HBM memory has greatly improved the speed and efficiency of AI training, and is an indispensable and important technology in the field of AI computing. With the advantages of high bandwidth, high capacity, low latency, and low power consumption, it has gradually become the standard configuration of GPUs in AI servers. A number of chips launched by NVIDIA for AI training, A100, H100 and H200, all use HBM video memory.
Optimized article:
NVIDIA's latest H200 chip features faster HBM3E video memory and capacities up to 80GB.
AMD's MI300 series also features HBM3 technology, with the MI300A having a capacity of 128GB while the MI300X has a capacity of 192GB, which is 100 of the H2 capacity4 times.
The capacity of the Mi300X is 50% higher than that of its predecessor, which is 2 of the capacity of the H1004 times to provide more memory options for the industry. The A100 and H100 chips are equipped with 40GB of HBM2E and 80GB of HBM3 video memory, and the latest H200 chips are equipped with the faster HBM3E with higher capacity. AMD's Mi300 series also uses HBM3 technology, with the Mi300A having the same capacity as the previous generation at 128GB, while the higher-end Mi300X boosts the capacity to 192GB, a 50% increase, which is equivalent to 2 of the H100 capacity4 times.
HBM: A revolutionary advancement in high-bandwidth storage technology.
The steady development of HBM (High Bandwidth Memory) technology has set off a revolution in storage technology. Since the birth of the first generation HBM1 in 2016, HBM has been iterated to the fifth generation - HBM3E. Looking at the performance changes of the fifth-generation HBM products, it can be found that they have made significant breakthroughs in bandwidth, IO rate, capacity, process nodes, etc.
Bandwidth: HBM1: 128GB s
hbm3e：1tb/s
io Rate:
hbm1：1gbps
hbm3e：8gbps
Capacity: HBM1: 1GB
HBM3E: Up to 36GB
Process node: HBM3E: 5nm
HBM technology continues to innovate and provide strong support for high-performance computing, artificial intelligence, machine learning, and other fields. In the future, HBM technology is expected to continue to push the envelope, bringing faster speeds, larger capacities, and lower power consumption, driving the construction of next-generation computing platforms. The market competition is fierce, and HBM products are accelerating iteration towards low energy consumption, high bandwidth, and high capacity. Since the release of the first generation HBM1 in 2016, HBM has been iterated to the fifth generation of products - HBM3E, looking at the performance changes of the fifth generation of HBM products, it can be found that HBM has made great breakthroughs in bandwidth, IO rate, capacity, process nodes, etc., among which the bandwidth has been iterated from the first generation of 128GB S to 1TB S of HBM3E, the IO rate has been iterated from 1Gbps to 8Gbps, the capacity has increased from 1GB to up to 36GB, and the manufacturing process has made further breakthroughs. Up to the 5nm level.
HBM3E data processing speed up to 115TBS to provide strong power for AI technology.
HBM series products continue to focus on low energy consumption, high bandwidth, and high capacity.
The high performance of HBM series products will drive further innovation and transformation of AI technology. HBM3E data processing speed can reach up to 1The update and iteration of the 15TBS HBM series products will continue to focus on low energy consumption, high bandwidth, and high capacity, and further innovate AI technology with high performance.
With the blessing of HBM3E, the performance of AI chips has been upgraded!
On November 13, 2023, NVIDIA released the H200, the first GPU chip equipped with the most advanced storage technology HBM3E, bringing a significant increase in video memory capacity and bandwidth.
Compared with the previous generation H100, the H200 has a 76% increase in memory capacity to 141GB, and a 43% increase in memory bandwidth to 48TBS to provide more powerful data processing capabilities for AI applications.
The introduction of HBM3E marks another leap in the performance of AI chips, which will bring better performance in deep learning, graphics rendering and other fields. Product iteration helps to upgrade the performance of AI chips. On November 13, 2023 local time, NVIDIA released the first GPU chip H200 equipped with the most advanced storage technology HBM3E. As the first GPU equipped with the most advanced storage technology, the H200 has 141GB of video memory and 48TBS memory bandwidth, 80GB with H100 and 3Compared with 35TBS, the memory capacity is increased by 76%, and the memory bandwidth is increased by 43%.
Blockbuster upgrade: NVIDIA H200 GPU creates a new height of AI large model computing!
With larger capacity and higher bandwidth memory, the H200 has achieved significant improvements in the calculation of large AI models.
Single-card performance is unrivaled:
llama2 13 billion parameters training speed increased by 40%.
GPT-3 175 billion parameters training speed increased by 60%.
llama2 70 billion parameter training speed increased by 90%.
Energy consumption and cost both decreased:
TCO (Total Cost of Ownership) can be reduced by up to half.
Cut energy consumption in half.
The H200 provides stronger performance and lower energy consumption with better**, becoming a leader in the field of AI large model computing. The GPU core has not been upgraded, but the H200 still achieves significant improvements in AI large model computing with larger capacity and higher bandwidth memory. According to NVIDIA's official data, in terms of single-card performance, H200 is 40% faster than H100 in LLAMA2's 13 billion parameter training, 60% in GPT-3's 175 billion parameter training, and 90% in LLAMA2's 70 billion parameter training; When it comes to reducing energy consumption and costs, the H200 achieves a new level of TCO (Total Cost of Ownership), which can reduce energy consumption by up to half.
HBM market structure and development trend.
At present, it is dominated by the three original factories of Hynix, Samsung and Micron, and Hynix has a leading share, accounting for 50%.
In 2023, the generative AI market will explode, driving a surge in demand for HBM.
High-bandwidth and high-capacity HBMs have become key components of AI servers, especially for large model training and inference. The market is currently occupied by three major OEMs, among which Hynix has the leading share and occupies a dominant position in the HBM market. According to TrendForce data, the market share of the three original manufacturers Hynix, Samsung, and Micron in 2022 is %. Since the beginning of 2023, the generative AI market has shown explosive growth, with the number of large model parameters and pre-trained data rising, driving the demand for high-bandwidth and high-capacity HBM for AI servers to increase rapidly.
Hynix: HBM3 mass production leads the AIGC industry.
As a pioneer of HBM chips, Hynix took the lead in realizing the mass production of HBM3 at the time of the rapid development of the AIGC industry and seized the market opportunity. In the second half of 2023, NVIDIA's high-performance GPUs H100 and AMD MI300 will be equipped with Hynix's HBM3, further consolidating its market leadership. In 2023, Hynix, Samsung, and Micron are expected to have a share of about % and 9% of the HBM market, respectively. Hynix, the HBM chip, was able to seize the opportunity in the context of the rapid development of the AIGC industry, taking the lead in realizing the mass production of HBM3 and seizing the market share. In the second half of 2023, NVIDIA's high-performance GPUH100 and AMD MI300 will be equipped with Hynix's HBM3, and Hynix's market share will further increase, and it is expected that Hynix, Samsung, and Micron will have a market share of % in 2023.
TSV technology, also known as TSV technology, is an advanced packaging technology that enables the electrical interconnection of through-silicon vias through vertical stacking between chips and wafers. TSV technology offers significant advantages in terms of storage capacity, bandwidth, and reduced power consumption. It dramatically increases storage capacity in a limited space by stacking multiple DRAM chips vertically. At the same time, TSV technology can shorten the signal transmission path between chips, reduce power consumption, and increase bandwidth.
The application of TSV technology has expanded to a variety of fields, including high-performance computing, mobile devices, and automotive electronics. With the development of technology, TSV technology is expected to play an important role in more fields and become a key technology to promote the development of electronic devices in the direction of smaller, lighter and faster. By stacking multiple DRAMs vertically, the technology can significantly increase storage capacity, bandwidth, and reduce power consumption. TSV (Through-Silicon Via) technology realizes vertical electrical interconnection of through-silicon vias by making vertical conduction between chips and between wafers and filling with conductive materials such as copper, tungsten, and polysilicon.
TSV's advanced packaging technology is leading the miniaturization of semiconductor devices.
As one of the key technologies to enable 3D advanced packaging, TSV advanced packaging technology can provide higher interconnect density and shorter data transmission paths, thereby improving chip performance and transmission speed. At a time when Moore's Law is slowing down and chip feature sizes are approaching physical limits, the miniaturization of semiconductor devices is increasingly dependent on advanced packaging with integrated TSVs.
In the DRAM industry, 3D-TSV DRAM and HBM have successfully produced TSVs, breaking through capacity and bandwidth limitations. The application of TSV advanced packaging technology provides a new way for the miniaturization and performance improvement of semiconductor devices, and has broad application prospects. One of the key technologies of 3D advanced packaging, compared to Wire Bond stacked packaging, TSV can provide higher interconnect density and shorter data transmission paths, so it has higher performance and transmission speed. As Moore's Law slows down and chip feature sizes approach physical limits, the miniaturization of semiconductor devices is increasingly reliant on advanced packaging with integrated TSVs. In the DRAM industry, 3D-TSVDRAM and HBM have successfully produced TSVs, overcoming capacity and bandwidth limitations.
TSV: HBM packaging core technology, high cost proportion.
As the core process of HBM, TSV accounts for about 30% of the cost of HBM3D packaging. Compared to traditional POP packages, the 3D TSV process can save 35% of package size, reduce power consumption by 50%, and bring an 8x increase in bandwidth.
TSV Cost Analysis: Formation and Exposure Percentage.
Taking the cost of 3D stacking of 4-layer memory chips and 1-layer logic bare cores as an example, the total cost of TSV formation and exposure accounts for 30% and 28% respectively, exceeding the cost proportion of the front and back processes. This indicates that TSVs are the most cost-intensive part of HBM3D packages. It is the core process of HBM, accounting for about 30% of the cost of HBM3D packaging. According to Samsung, the 3D TSV process saves 35% of package size, reduces power consumption by 50%, and brings an 8x increase in bandwidth compared to traditional POP packages. The cost of 3D stacking of 4-layer memory chips and 1-layer logic bare core is analyzed, and the total cost of TSV formation and exposure accounts for 99The 5% and 99% bond yields are 30% and 28%, respectively, which exceeds the cost proportion of the front-end process and is the highest cost proportion of the HBM3D package.
TSV technology process.
Deep Hole Etching: Through-holes are made using the Deep Reactive Ion Etching (DRIE) method.
Deposition: The dielectric layer is deposited using the chemical vapor deposition (PECVD) method and the barrier and seed layers are deposited using the physical vapor deposition (PVD) method.
Electroplated Copper: Select Electroplated Copper (CU) for hole filling.
Thinning polishing: Chemical and mechanical polishing (CMP) is used to remove excess copper.
Wafer thinning and bonding: After the copper filling is complete, wafer thinning and bonding are required. The technology mainly involves key processes such as deep hole etching, deposition, thinning and polishing. TSV first used the deep reactive ion etching (DRIE) method to fabricate the vias. Then, the dielectric layer was deposited by chemical vapor deposition (PECVD), and the barrier layer and seed layer were deposited by physical vapor deposition (PVD). Then select electroplated copper (Cu) for hole filling; Finally, the excess copper is removed using chemical and mechanical polishing (CMP) methods. In addition, due to the need for chip stacking integration, wafer thinning and bonding are also required after the copper filling is completed.
HBM's multi-layer stacking structure is driving demand growth for packaging equipment.
Front-end link: The vertical connection of TSV increases the demand for TSV etching equipment.
The HBM process adds TSV, micro-bumps, silicon interposers and other processes, driving the growth of demand for front-end inspection and measurement equipment.
Back-end link: The increase in HBM stacking structures requires wafer thickness to be continuously reduced, and the demand for thinning and bonding equipment is increasing.
Ultra-thin wafers and copper-copper hybrid bonding processes increase the need for equipment such as temporary bonding and debonding.
The increased requirements for the protective material of the various layers of DRAM die place higher demands on the injection or compression molding equipment.
In summary, the HBM multi-layer stacking structure improves the process steps and drives the continuous increase in the demand for packaging equipment. The multi-layer stacking structure improves the process steps and drives the demand for packaging equipment to continue to increase. (1) Front-end link: HBM needs to be vertically connected through TSV, which increases the demand for TSV etching equipment, and at the same time, TSV, micro-bump, silicon interposer and other processes in HBM have greatly increased the front-end process, which brings increments to the front-end inspection and measurement equipment; (2) Back-end links: the increase in HBM stacking structures requires the wafer thickness to be continuously reduced, which means that the demand for thinning, bonding and other equipment increases; HBM's multi-layer stacking structure relies on ultra-thin wafers and copper-copper hybrid bonding processes, increasing the need for equipment such as temporary bonding, debonding, etc.; (3) The protective material of each layer of DRAM die is also very critical, which puts forward high requirements for injection molding or compression molding equipment.
- What are your thoughts on this? -
- Welcome to leave a message** and share in the comment area. -