Artificial intelligence is advancing at a blistering pace, but the hardware powering AI systems need help to keep up. GPUs have become the standard for training and running neural networks, but their architecture was designed with something other than AI workloads in mind. Now, IBM has unveiled a revolutionary new AI chip called NorthPole that could shake up the AI hardware landscape.
NorthPole represents a radical departure from traditional computing architectures. It does away with the separation between memory and processing by embedding memory directly into each of its 256 cores. This allows NorthPole to sidestep the von Neumann bottleneck completely, the shuttling of data back and forth between memory and processors that creates significant inefficiencies in standard chips. By integrating computing and memory, NorthPole can achieve unprecedented speeds and energy efficiency when running neural networks.
In initial tests, NorthPole has shown mind-blowing performance compared to existing GPUs and CPUs on AI workloads. On the popular ResNet-50 image recognition benchmark, NorthPole was 25x more energy efficient than even the most advanced 12nm GPUs and 14nm CPUs. It also handily beat these chips in latency and compute density. Remarkably, NorthPole achieved this using 12nm fabrication technology. If it were built today with 4nm or 3nm processes like the leading edge chips from Nvidia and AMD, its advantages would be even more significant.
Nvidia has dominated the AI chip market with its specialized GPUs, but NorthPole represents the most significant challenge yet. GPUs excel at the matrix math required for neural networks, but they suffer from having to move data back and forth from external memory. NorthPole's integrated architecture tackles this problem at the hardware level. The efficiency gains in speed and power consumption could be game-changing for AI applications.
However, NorthPole is not going to dethrone Nvidia for a while. The current version only has 224MB of on-chip memory, far too little for training or running massive AI models. It also cannot be programmed for general purposes like GPUs. NorthPole is tailored for pre-trained neural network inference, applying already learned networks to new data. This could limit its real-world applicability, at least in the near term.
That said, NorthPole's efficiency at inference could make AI viable in a whole new range of edge devices. From smartphones to self-driving cars to IoT gadgets, running AI locally is often impossible with today's chips. The low power consumption and tiny size of NorthPole opens the door to putting AI anywhere. Embedded AI chips based on NorthPole could make AR/VR glasses practical or enable real-time video analysis in security cameras. These applications only need a small, specialized neural network rather than an all-purpose AI model.
NorthPole's scale-out design also shows promise for expanding its capabilities. By connecting multiple NorthPole chips, more extensive neural networks could be run by partitioning them across the distributed on-chip memories. While yet to be feasible for massive models, this could make NorthPole suitable for a much more comprehensive range of AI tasks. And, of course, Moore's Law expects fab processes to continue improving, allowing greater memory capacities on future iterations of NorthPole.
The efficiency genes are clearly in NorthPole's DNA. Any real-world product based on it must be tested across various AI workloads and applications. However, the theoretical concepts have been proven. By integrating computing and memory, NorthPole delivers far superior efficiency on neural network inferencing compared to traditional architectures.
Nvidia will retain its AI chip crown for the foreseeable future, especially for training colossal models. But in AI inferencing, NorthPole represents the most promising challenge yet to the GPU giant's dominance. It opens up revolutionary possibilities for low-power, ubiquitous AI in edge devices. If NorthPole's capabilities can grow exponentially over generations as Moore's Law expects, it may one day become the standard AI compute architecture across the entire stack from edge to cloud.
The AI hardware landscape is shifting. An architecture inspired by the human brain has shown vast untapped potential compared to today's computer chips. NorthPole heralds the dawn of a new era in AI hardware, where neural networks are computed with unprecedented speed and efficiency. The implications for embedding advanced AI into everyday technology could be world-changing.