Alibaba Cloud ditches Nvidia's interconnect in favor of Ethernet — tech giant uses own High Performance Network to connect 15,000 GPUs inside data center

Data center network connections.

Alibaba Cloud engineer and researcher Ennan Zhai shared his research paper via GitHub, revealing the Cloud provider’s design for its data centers used for LLM training. The PDF document, entitled “Alibaba HPN: A Data Center Network for Large Language Model Training,” outlines how Alibaba used Ethernet to allow its 15,000 GPUs to communicate with each other.

General cloud computing generates consistent but small data flows with speeds lower than 10 Gbps. On the other hand, LLM training produces periodic bursts of data that can hit up to 400 Gbps. According to the paper, “This characteristic of LLM training predisposes Equal-Cost Multi-Path (ECMP), the commonly used load-balancing scheme in traditional data centers, to hash polarization, causing issues such as uneven traffic distribution.”