AI Vocabulary · 02 AI Infrastructure
What is Interconnect (NVLink, InfiniBand, Ethernet)?
The fast network that links thousands of GPUs so that they can work on one AI job together.
Large AI models are trained on thousands of GPUs that constantly exchange intermediate results. Inside a server or rack they talk over a very fast private link such as NVIDIA's NVLink, called scale-up. Between racks, a data centre network connects them, called scale-out. It usually runs on NVIDIA's InfiniBand or on Ethernet with RoCE, a technique that lets machines write directly into each other's memory.
Both work at the largest scale. In 2024 Meta built two clusters of 24,576 H100 GPUs each, one on RoCE Ethernet and one on InfiniBand, each with 400 Gbps per GPU, and trained Llama 3 on the Ethernet cluster, by its own account without network bottlenecks.
Why it matters for you
The interconnect decides how much of a cluster's power is actually used. Meta reports that before tuning, the utilisation of its large clusters ranged from about 10% to 90%, and above 90% afterwards.
Own it
When you rent a GPU cluster for training, ask for the network type, the bandwidth per GPU and how many GPUs share one fast domain. For inference through an API, this is the provider's job.
Where it comes from
Meta describes both clusters in Building Meta's GenAI Infrastructure (Meta Engineering, March 2024, vendor).
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.