Why AI Runs on GPUs: From CUDA 2007 to AlexNet 2012

The answer is not that GPUs are smarter. AI runs on GPUs because training a neural network is the opposite of how an ordinary processor works.

In 2012, AlexNet, built by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, won ImageNet by an absurd margin. Its top-5 error was 15.3 percent against 26.2 percent for the runner-up, and it trained on two CUDA cards in Toronto.

Five years earlier, in 2007, the GPU looked nothing like that. The chip could only be programmed through a C-like shader language, and data had to be copied by hand between video and CPU memory.

2007: The Year a GPU Learned to Speak Code

CUDA was made public in February 2007. Before that, using a GPU outside graphics and games felt like ordering food by painting a menu. To get one matrix, a programmer faked the result as a texture, ran it through the graphics API, then read the pixels back one at a time.

What you could not do was write one block of C and run it across thousands of cores. CUDA removed that obstacle: you write C-like syntax, then declare how many threads run together. The hardware does not change fundamentally. How you get in does.

Behind it were Ian Buck, a Stanford PhD student, and John Nickolls, who ran GPU computing at NVIDIA. Buck had built Brook, an experimental language for general-purpose GPU work. NVIDIA hired him in 2004, and Brook became CUDA.

Picture the area of 10,000 plots of land. Through the graphics API you draw the answer on screen, read every pixel back, then compute. Through CUDA you write it like ordinary C and thousands of cores do the arithmetic.

2009: The First Published Proof

In June 2009, Rajat Raina, Anand Madhavan and Andrew Ng from Stanford published an ICML paper titled Large-scale Deep Unsupervised Learning using Graphics Processors. It was full of numbers, not claims.

They used one GeForce GTX 280 card with 240 cores. Deep belief network learning ran up to 70 times faster than a dual-core CPU. A 45 million parameter model updated in roughly 29 minutes per million examples, where a multicore CPU needed over a day.

Picture marking a class register sheet. The old way reads one name at a time and adds them up. The GPU way scans the whole page at once. It is not five percent faster, it is an order-of-magnitude jump.

What changed was who got to own a large model. If one card could already handle a model worth training, model size stopped being capped by budget.

2012: AlexNet and the Cliff

ImageNet holds more than 15 million labeled images across roughly 22,000 categories. ILSVRC started in 2010 with 1.2 million training and 150,000 test images.

Krizhevsky and Hinton submitted as SuperVision on 30 September 2012. AlexNet had eight layers, about 60 million parameters and 650,000 neurons. Training took five to six days on two 3 GB GeForce GTX 580 cards.

The reason was practical. One card had only 3 GB of memory, so a network that size would not fit. The team split it across two GPUs, and the paper records that the split cut top-1 error by 1.7 percent and top-5 error by 1.2 percent.

Interconnect mattered even more. The two GPUs read and wrote each other's memory directly, without routing through CPU memory. Every data movement became a queue that slowed the arithmetic.

Those bars are the top-5 error of each ILSVRC winner. The 2010 and 2011 figures come from a chart in the 2015 Deep Residual Learning paper; the 2012 figure comes from the AlexNet paper, at 15.3 percent on the test set. The first two years barely moved. The three after fell every year.

Why Thousands of Tiny Cores Beat a Few Big Ones

An ordinary processor spends many transistors on caches and branch prediction, all aimed at finishing one large instruction fast. A GPU picks the opposite: caches deliberately small, control logic deliberately simple, the rest left to arithmetic. It is an odd bargain if you have only met one kind of processor.

Training a neural network is almost entirely matrix multiplication. Inside one multiply, every output element stands on its own, so thousands of identical small operations run side by side.

One caveat: backpropagation does run in order from layer to layer. That ordering sits between layers, not inside the multiply. What gets parallelized is the most repetitive part of the job.

Picture a catering kitchen feeding a thousand guests. The processor is one master chef cooking one excellent dish. The GPU has thousands of clueless apprentices, each chopping one simple ingredient. Far more food, which is what the job needed.

The second constraint is often data movement, not arithmetic. Model weights must reach every core without queueing. If the road is narrow, more cores do not raise throughput. That is why interconnect gets its own lane.

GPU, TPU, and NPU: Three Different Animals

Everyday computers ship GPUs because their jobs vary. Google's TPU was built for large matrix multiplication, and an NPU is that idea shrunk onto a phone: less power, narrower work.

The difference is not raw power, it is flexibility. A GPU runs anything a library exists for. A TPU or NPU is only fast at what it was designed for, which is why the two run alongside each other.

Modern AI work is dominated by matrix multiplication, so the specialist leads there. The moment you need a new feature, an adapter can keep you waiting.

Behind the Numbers: Tensor Cores, NVLink, and Cold Water

Four numbers explain the shift after 2012, and every one comes from an official document.

In 2017, NVIDIA's Volta architecture introduced the Tensor Core. The V100 carries 640 of them and reaches 120 TFLOPS of deep learning. Inputs are FP16 and accumulation is FP32, so precision survives without losing speed. The Volta whitepaper claims up to 12 times higher peak performance for training than a P100 on FP32, and six times for inference over FP16.

In 2016, NVLink arrived with Pascal P100, giving each GPU 160 GB/s bidirectional, roughly five times PCIe Gen3 x16. The DGX-1 document reports something heavier: ResNet-50 at batch size 64 across eight GPUs over PCIe reached under 35 percent parallel efficiency, while the NVLink hybrid cube mesh hit about 54 percent.

Cooling is the third layer. AI chips run hot, and a dense rack is no longer something air cooling can handle. The Uptime Institute Cooling Systems Survey from 2024 records 22 percent of operators already using direct liquid cooling, with most putting the switch above 20 kW per rack. That is an opinion, not a measurement. No standard PUE figure exists.

Those numbers deserve honest framing. The 2012 and 2018 figures come from NVIDIA's own documents, so they are vendor claims. The 22 percent comes from an independent survey of 964 respondents.

For readers in Indonesia one question follows: how far apart are an expensive rack and a cheap one, once cooling costs are counted?

What This Means for a Small Team in Indonesia

Many teams in Indonesia do not need to train a laboratory-sized model. They need to run somebody else's model at a sensible cost, and two decisions control everything.

• Pick open weights rather than a closed model. Open weights can be downloaded, inspected, and hosted yourself.
• Quantize. FP16 takes 2 bytes per weight, INT8 takes 1 byte, INT4 takes half a byte.
• Work out the memory requirement before choosing a GPU.
• Rent by the hour rather than buying. Rental prices change every month and differ between providers.

Quantization is like storing a color photo in black and white. Quality drops a little, while size and electricity cost drop a lot, and for office work the small drop goes unnoticed.

The arithmetic is simple. A model with 70 billion parameters needs roughly 140 GB of weights at FP16, about 70 GB at INT8, and about 35 GB at INT4. Those are calculated, not measured.

Avoid inventing price numbers. GPU rental prices differ by provider and change monthly, and figures in chat groups are often months out of date. Open the price page and do the sum.

Numbers Nobody Has Published

Be honest here: three things still have no standard number anyone can quote.

• A typical PUE for an AI rack. It moves with cooling design, workload and location.
• GPU rental prices in Indonesia. They differ by provider and change every month.
• The real saving from quantization. It depends on model, task, and how you measure it.

None of that is rare. Plenty of technical reports stop just short of the number, which tells you it belongs to each operator rather than to the industry. Rack policy differs between providers too, so the numbers cannot be lined up.

Closing

So the question of why AI uses GPUs has an answer unchanged since 2012. Thousands of small operations that do not depend on each other, executed across thousands of cores. AlexNet proved it on two cards, not one.

By 2018 a DGX-2 could train AlexNet in 18 minutes. That figure is the vendor's own, but it still reads as a direction.

The question now is not whether GPUs are useful. The question is when a job stops being divisible into thousands of small pieces.