So wouldn't this be like 2.6 TFLOPS? I'm thinking if this can replace NVidia V100s to train something like ImageNet purely on CPU. However, V100 has 100 TFLOPS which seems 50x more than 3990X. Perhaps, I'm reading the specs wrong?
PS: Although FLOPS is not a good way to measure these stuff, it's a good indication of possible upper bound for deep learning related computation.
DDR4 has a bandwidth of about 25 Gigabytes per second. The memory on a V100 does about 900 Gigabytes per second. Cerebras has 9.6 Petabytes per second of memory bandwidth. For stochastic gradient descent, which typically requires high-frequency read/writes, memory bandwidth is crucial. For ImageNet, you're trying to run well over 1TB of pixels through the processing device as quickly as possible while the processor uses a few gigabytes of scratch space.
DDR has a bandwidth of about 25GBps per channel. You can hit around 100-200GBps on Epyc processors if you're utilizing ram efficiently. GPUs tend to enforce programming models that ensure more sequential accesses, but CPU can do it too.
These stats are true, but the CPU's biggest advantage is L1, L2, and L3 cache.
In particular, the 3990x, 64-core Threadripper will have 256MBs of aggregate L3 cache, and 512kBs L2 cache per core (32MBs of L2 cache). Highly-optimized kernels may fit large portions of data within L3 cache and rarely even touch DDR4!
Note: Each L3 cache is only 16MBs between 4-cores. It will take some tricky programming to split a model into 16MB chunks, but if it can be done, Threadripper would be crazy fast.
True, GPUs have really fat VRAM to work with, but CPUs have really fat L3 cache and L2 cache to work with. And the CPU caches are coherent too, simplifying atomic code. GPUs do have "shared memory" and L2 caches, but they're far smaller than CPU-caches.
More accurately, Cerebras has 9.6 "bullshitobytes" per second. If you can't verify this, it doesn't exist. You could claim insane "bandwidth" by considering your register file to be your "memory". But that doesn't make it so.
Nah, you’re probably not reading the specs wrong. GPU-type devices severely outclass CPUs in raw compute power. This has been true for years, and it’s why deep learning depends on them. But CPUs and GPUs fulfill very different niches computationally; GPUs are incredibly parallel but aren’t well-suited for serial tasks or tasks that require unpredictable branching/looping, which is pretty much exactly what CPUs are good at.
CPU’s do general computing. They’re super flexible but if you have a specific workload you might be able to use a different piece of silicon to get more performance.
GPU’s do more parallelized computing but they don’t do as many different operations. They’re really good at doing a small not super complex fast but massively parallelized (like updating an array of pixels on a screen, for example).
TPU’s are even more parallelized but the operations they do are even more specific and often simpler than the operations GPUs do.
That's not really why they're slower. CPUs are significantly more complicated. Things like branch prediction, transactions, sophisticated prefetching all take up a lot of silicon.
The more specific you get on the circuit the less flexible it is and the more bandwidth you get at a specific task.
The tradeoff is flexibility for application specific performance. CPUs can do hella stuff but they can’t do a specific thing faster than specialty hardware.
GPUs are surprisingly flexible. GPUs are a full on turing machine.
The thing is, GPUs have horrible latency characteristics compared to CPUs. Whenever a GPU "has to wait" for RAM, it only switches to another thread. In contrast, CPUs will search your thread for out-of-order work, speculative work, and even prefetch memory ("guessing" what memory needs to be fetched) to help speed up the thread.
--------
Consider speculative execution. Lets say there is a 50% chance that an if-statement is actually executed. Should your hardware execute the if-statement speculatively?
Since CPUs are latency optimized, of course CPUs should speculate.
GPUs however, are bandwidth optimized. Instead of speculating on the if-statement, the GPU will task switch and operate on another thread. GPUs have 8x to 10x SMT, many many threads waiting to be run.
As such, GPUs would rather "make progress on another thread" rather than speculate to make a particular thread faster.
---------
What problems can be represented in terms of a ton-of-threads ? Well, many simple image processing algorithms operate on 1920 x 1080 pixel entries, which immediately provides 2,073,600 pixels... or ~2-million items that often can be processed in parallel.
When you have ~2-million items of work (aka: "CUDA Threads") waiting, the GPU is the superior architecture. Its better to make progress on "waiting threads" than to execute speculatively.
But if you're a CPU with latency-optimized characteristics, programmers would rather have that if-statement speculated. The 50% chance of saving latency is worth more to a CPU programmer.
Remember that these millions of items must actually be able to do work independently, i.e. you need very few (or regular and localized) dependencies between the data processed by each thread, so that threads can just wait on things like memory accesses rather than waiting on _other threads_.
Hmm, I think I see what you're trying to say, but maybe more precise language would be better here.
GPU cores have extremely efficient thread-barrier instructions. NVidia PTX has "barrier", while AMD has "S_BARRIER". Both of which allow the ~256 threads of a workgroup to efficiently wait for each other.
-------
The other aspect, is that "Waiting on Memory" (at least, waiting on L2 memory) is globally synchronized in both AMD and NVidia systems. Waiting for an L2 atomic operation to complete IS a synchronization event, because L2 cache has a total memory ordering on both AMD and NVidia platforms.
Tying L2 cache to higher levels allows for memory coherence with the host CPU, or other GPUs even. That is to say: "dependencies" are often turned into memory-sync / memory-barrier events at the lowest level.
Synchronizing threads is one-and-the-same as waiting on memory. (Specifically: creating a load-and-store ordering that all cores can agree upon).
---------
I think what you're trying to say is that dependency chains must be short on GPUs, and that there are many parallel-threads of dependency chains to execute. In these circumstances, an algorithm can run efficiently.
If you have an algorithm that is an explicit dependency chain from beginning to end (ironically: Ethereum hashing satisfies this constraint. You work on the same hash for millions of iterations...), then that particular dependency chain cannot be cut or parallelized.
But Ethereum Hashing is still parallel, because there are trillions of guesses that can all work in parallel. So while its impossible to parallelize a singular ETH Hash... you can run many ETH Hashes in parallel with each other.
It's reading the specs wrong for apples to apples. OP is using the tensor core numbers, which is half precision and only for matrix multiplies. Operations that don't fit that will use the standard fp32/16 performance of the chip, which is around 13TFLOPS. still higher than 2, but nowhere near 100.
64 cores * 2.9 GHz * 8 single-precision lanes * 2 issue * 2 (FMA) = 5.9 TF. This compares with 14 TF for V100 (costs more and needs a host). The 100 TOPS for V100 refers to reduced precision (which may or may not be useful in a given ML training or prediction scenario). The V100 has much (~9x) higher memory bandwidth, but also higher latency.
* Radeon VII GPU is 14.2 TFlops for $600 right now.
* NVidia RTX 2070 Super is 9 TFlops for $500
True, Radeon VII and RTX 2070 are "consumer" GPUs... but Threadripper is similarly a "consumer" CPU and commands a lower price as a result.
"Enterprise" products cost more. EPYC costs more than Threadripper, V100 costs more than RTX 2070 Super. If you're aiming at maximum performance at minimum price, you use consumer hardware.
Similarly, Threadripper loses on RDIMMs, LRDIMMs, and have 1/2 the memory channels and 1/2 the PCIe lanes. Most people don't need that either.
Of course consumer products have fewer features than enterprise products. The chip manufacturers need to leave some features for "enterprise". The general idea is to extract more wealth from the people who can afford it, while providing consumers the features they care about at a lower cost.
-----
Really, the feature enterprise GPUs need seems to be SR-IOV, or other PCIe-splitting technologies. This allows a singular GPU to be split over many VMs.
Double-precision floats is niche to the scientific fields, also "enterprise" but I don't think most enterprise customers use double-precision floats.
PS: Although FLOPS is not a good way to measure these stuff, it's a good indication of possible upper bound for deep learning related computation.