Hopp til hovedinnhold

Meta: Prometheus now has hundreds of thousands of Nvidia GPUs, target one million

Meta says the first phase of Prometheus in Ohio is complete and comprises hundreds of thousands of Nvidia GPUs, with a target of about one million. It publishes its own utilisation and interruption figures, which are not independently verified.

Av Joachim HøgbyKilde: Meta Engineering

Meta said on Monday that the first phase of Prometheus is complete. The cluster in New Albany, Ohio, now comprises "hundreds of thousands" of GPUs. The target is roughly one million. What is new in the blog post is not that Prometheus is running. Meta said that on 12 October, when it stated that the Muse Wonder models were trained there. What is new are the first concrete figures on size, power density and how well the machine is actually used.

All figures come from Meta itself. They have not been independently verified.

From tens of thousands to hundreds of thousands of GPUs

In February, Meta's engineers described Prometheus as a cluster that would interconnect "tens of thousands of GPUs" and deliver one gigawatt of capacity once complete. Meta now writes that the cluster already comprises hundreds of thousands of GPUs across several interconnected buildings. These include traditional data center buildings, AI-optimised buildings and rapid deployment structures.

At full scale, Prometheus will comprise roughly one million GPUs working in lockstep, according to Meta. That is a target, not a status. Meta does not say when it will be reached, what the build-out costs or how much power the cluster draws today.

The network spans distances from 2 to more than 100 kilometres. Meta describes millions of miles of fibre strand, tens of thousands of switches and millions of optics.

One supplier at the base

All the compute chips in Prometheus come from Nvidia. Meta currently runs two platforms: GB200 NVL72, which Meta calls Catalina, and GB300 NVL72, called Clemente. A third generation, based on Nvidia's Vera Rubin NVL72, is arriving "soon", according to Meta. The network also uses equipment from Arista and silicon from Broadcom and Nvidia.

Meta describes its work with Nvidia as going beyond a traditional vendor relationship. According to the post, Nvidia teams are embedded in Meta's data centers. The two share real-time telemetry and review every system halt together. Meta handles scheduling and infrastructure, while Nvidia addresses firmware, signal diagnostics and memory handling.

Power density is high. One two-rack Catalina pair draws about 120 kilowatts. One Clemente rack draws about 135 kilowatts. The cluster uses closed-loop liquid cooling.

How much of the machine actually works

Meta also publishes reliability figures. When hundreds of thousands of GPUs train one model together, a single faulty machine can halt the whole job. Meta therefore measures "goodput": the share of time spent on useful training rather than restarts and waiting.

According to Meta, goodput was above 85 percent during Muse Wonder pre-training and above 90 percent during reinforcement learning. Mean time between interruptions was about 47,000 GPU hours during training. After firmware fixes, Meta reports more than 54,000 GPU hours. For each large training job, Meta also holds roughly 13 percent extra GPUs in reserve. In Meta's own example, a job with 32,768 active GPUs requires 37,656 GPUs in total.

The post does not give comparable figures for Meta's earlier clusters, so the size of the improvement cannot be assessed.

What it means for business leaders

The news changes no product or price today. It shows how large and supplier-dependent the resources are behind the models companies are adopting.

  • Capacity for the largest models rests on a single chip supplier. That is a concentration risk across the whole value chain, including for those who only buy the service.
  • Reserves, interruptions and utilisation are part of the cost of training models. Such figures indicate what providers must recover through pricing, but Meta gives no costs.
  • The energy need is large. Meta describes the cluster as gigawatt-scale but does not state current consumption. Leaders who report on climate and supply chains should ask providers for actual figures.

Sources and media

Mer om Meta

Oppskriften. Strategien. Hver dag.

AI-nyheter for ledere. Kuratert av en CIO som bygger det selv. Daglig i innboksen.