"The misconception here is that this is as much, if not more, a software problem to solve than it is a hardware problem," said Nigel Toon, chief executive officer of AI computer startup Graphcore.
Graphcore is known for making a computer system that has hundreds of thousands of processor cores inside every "intelligence processing unit," or IPU, chip, to crunch artificial intelligence problems.
Its latest machines have an IPU chip, "Bow," that consists of two semiconductor die stacked one on top of the other.
And yet, Toon, who was in town last week from Graphcore's headquarters in Bristol, England, told ZDNet that software is at the heart of the very large challenge of increasingly large AI problems, whereas the hardware, though by no means trivial, is, in a sense, secondary.
"You can build all kinds of exotic hardware, but if you can't actually build the software that can translate from a person's ability to describe at a very simple level into hardware, you're not really producing a solution," said Toon over lunch at The Grey Dog cafe in Manhattan's Union Square neighborhood.
It was Toon's first time on the road since the pandemic. He's relishing getting back to meeting with customers. "It's good to be traveling again," he said, and getting face to face.
Among the points to be emphasized on his swing through the U.S. is the software factor. Specifically, the capability of Graphcore's Poplar software, which translates programs written on AI frameworks such as PyTorch or TensorFlow to efficient machine code.
It is, in fact, the act of translation that is key to AI, Toon argues. No matter what hardware you build, the challenge is how to translate from what the PyTorch or TensorFlow programmer is doing to whatever transistors are available.
A common conception is that AI hardware is all about speeding up matrix multiplications, the building block of neural net weight updates. But, fundamentally, it's not.
"Is it just matrix multiplication, and is it just convolutions that we need, or are there other operations that we need?" asked Toon, rhetorically.
In fact, he said, "it's much more about the the complexity of the data."
A large neural net, said Toon, such as GPT-3, is "really an associated memory," so that connections between the data are what are essential, and movement of things in and out of memory becomes the bottleneck for computing.