Claude Shannon, the founder of information theory, once remarked: “The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point.” In the early days of computing, we treated communication as a trivial post-script to computation. We designed faster arithmetic logic units, packed more transistors onto silicon, and assumed that as long as we could compute fast enough, our machines would keep pace.
However, as we scale artificial intelligence to cluster-level systems, Shannon’s observation becomes a stark warning. The limit of our AI systems is no longer how fast we can perform arithmetic, but how quickly we can reproduce messages across those systems.
To think and reason is to move data. How else does one process information but by shifting it from memory to compute?
The limit of our AI systems is no longer how fast we can perform arithmetic, but how quickly we can reproduce messages across those systems.
If you read the financial press today, the headlines are filled with noise: soaring GPU sticker prices topping $30,000 to $40,000, persistent reports of global RAM and High Bandwidth Memory (HBM) shortages, and a historic stock market bull run that has propelled companies like NVIDIA, TSMC, SK Hynix, and Micron to unprecedented valuations. To the casual observer, this looks like a typical gold rush fueled by speculative excess.
In reality, these news items are symptoms of a deeper, structural transition. The economy is pricing in a shift from compute-centric hardware to communication-centric systems. The traditional mental model—where more AI simply meant more FLOPS and transistors—is becoming increasingly incomplete. To find the underlying signal in this noise, we must turn to a simple but elegant framework from computer architecture: the Roofline Model.
The Roofline Model and Arithmetic Intensity
Originally developed by researchers at Berkeley to describe the performance limits of individual processors, the Roofline Model plots a processor’s throughput against its arithmetic intensity. Arithmetic intensity measures the number of operations performed per byte of data transferred:
\[I = \frac{\text{Operations (FLOPs)}}{\text{Memory Access (Bytes)}}\]When a workload has high arithmetic intensity, the processor can perform many calculations on each byte retrieved from memory. In this regime, the system is compute-bound. However, when arithmetic intensity is low, the processor spends most of its time waiting for data to arrive from memory. The workload is memory-bound.
| Metric | Compute-Bound Workload | Memory/Communication-Bound Workload |
|---|---|---|
| Primary Constraint | Peak FLOPS (Processor Speed) | Bandwidth (Memory & Interconnect) |
| Arithmetic Intensity (\(I\)) | High | Low |
| Optimization Focus | Maximizing ALU utilization | Minimizing data movement |
Recently, researcher Reiner Pope extended this classic model beyond a single chip to entire AI clusters. At cluster scale, the relevant equation becomes surprisingly simple:
\[\text{Time} = \max(\text{Compute Time}, \text{Communication Time})\]The critical observation here is not mathematical but architectural. An AI accelerator is only as fast as its slowest bottleneck. If computation takes longer than moving data, then adding more compute helps. But once communication becomes slower than computation, adding more arithmetic units accomplishes very little. The bottleneck has moved.
In other words, our state-of-the-art accelerators are spending most of their time waiting.
Why This Reshapes the Semiconductor Industry
Too often, memory and packaging have been treated as commodity sub-sectors of the semiconductor industry—low-margin components whose only job was to support the central GPU.
However, as communication bottlenecks dominate execution time, the boundaries between these categories are beginning to dissolve. Value is migrating across the supply chain:
- Memory Stops Being a Commodity: Large language models require streaming billions of parameters repeatedly during inference. Consequently, bandwidth matters far more than capacity. This explains why HBM has become one of the industry’s scarcest resources. The persistent RAM and HBM shortages reported in the news are not temporary supply chain hiccups; they represent the structural bottleneck of AI itself. As a result, memory manufacturers like SK Hynix and Micron are experiencing massive stock bull runs because they control the gating factor of AI performance.
- Packaging Becomes Architecture: Advanced packaging technologies like TSMC’s CoWoS (Chip-on-Wafer-on-Substrate) or Intel’s EMIB are no longer manufacturing details. Because we must connect HBM stacks to logic dies with thousands of microscopic interconnects, packaging packaging density now dictates memory bandwidth. This is why foundries are commanding premium pricing, driving up GPU cluster costs.
- Networking Becomes Compute: As training clusters grow to tens of thousands of GPUs, communication between chips becomes as critical as compute inside them. NVIDIA’s primary competitive moat is not raw FLOPS; it is NVLink, NVSwitch, and optical interconnects that treat the cluster as a single computer.
- Software as a Communication Solution: Funnily enough, the massive shifts we see in model architecture—such as the transition to Mixture of Experts (MoE) in models like GPT-4, or the adoption of Grouped-Query Attention (GQA)—are often discussed as purely algorithmic breakthroughs. In reality, they are software solutions to memory-bandwidth bottlenecks. They reduce active parameter traffic and memory footprint. The fastest model is not the one that computes the most, but the one that waits the least.
The Strategic Path Forward
What does this mean for organizations and investors trying to navigate the next wave of AI development?
The short-term solution for most organizations has been to secure as many raw GPUs as possible. But this approach runs the risk of building “AI factories” with immense theoretical FLOPS that sit idle due to insufficient networking or memory throughput.
A more forward-thinking solution is to focus on systems-level efficiency. To succeed in this new era, data science teams and infrastructure architects must place communication front and center. Foundries must compete on packaging density, cloud providers must optimize their network topologies, and software engineers must design compilers (like XLA or Triton) to optimize physical memory layout rather than just mathematical operations.
The history of computing is a sequence of shifting bottlenecks. Early machines were constrained by transistor counts, then by clock speeds, and then by power. Today, the scarce resource is no longer arithmetic; id est, the bottleneck is movement.
The companies that dominate the next decade may not be those that build the fastest chips. They will be those that build the fastest systems.
Victor Blancada is a data scientist focused on deriving actionable insights for clients. Visit his LinkedIn page here.