Tiny Corp., the startup that maintains tinygrad, a neural network framework, aims to democratize PetaFLOPS-class performance for artificial intelligence. To that effect, it has unveiled its TinyBox system that packs six AMD's Radeon RX 7900 XTX graphics cards into a 12U rack case and will sell for $15,000. That figure includes all the necessary software and hardware.
The TinyBox system uses six AMD Radeon RX 7900 XTX boards (one of the best graphics cards) connected using 'full fabric' PCIe 4.0 x16 links to ensure maximum bandwidth. Apparently, these consumer-grade GPUs fully support the peer-to-peer interconnections necessary for large language models (unlike Nvidia's GeForce RTX 4090), so the company chose to use these GPUs over Nvidia's more popular options.
From a performance point of view, the TinyBox can offer up to 738 FP16 TFLOPS (0.738 FP16 TFLOPS) of performance along with 96 GB of GDDR6 memory that delivers 21 TB/s of aggregated memory bandwidth. To put the numbers into context, one TinyBox offers 37% of Nvidia H100 compute performance (FP16) but slightly more memory (96 GB instead of 80 GB) and considerably higher peak memory bandwidth (21 TB/s instead of 3.35 TB/s).
The system is powered by an AMD EPYC 7532 CPU mated with 128 GB of RAM. As for storage, the TinyBox has five Western Digital SN850X 1TB SSDs, four configured in RAID for enhanced performance and one dedicated for booting the system. Additionally, the machine features an empty 16x OCP 3.0 slot for networking, allowing for flexible connectivity options.
Tiny says that the cost of its TinyBox machine is about $10,000, and it sells it for $15,000, which is considerably lower than Nvidia's H100. As a result, the company has received 583 preorders and plans to begin shipping in April. A production run of 100 units is already underway, the company indicated.
On the software side, the TinyBox ships with Ubuntu 22.04 and initially will include only the tinygrad framework. However, it is compatible with other machine learning frameworks such as PyTorch and JAX on AMD hardware, providing users with the flexibility to choose their preferred tools.
Since Tiny wants to democratize PetaFLOPS-class performance for AI workloads, the TinyBox is not just a product, but rather a testament that this can be done. By providing a standardized piece of hardware, the company aims to enable developers to explore new frontiers in software and algorithm development, ultimately leading to breakthroughs in the field. As AMD, Intel, and Nvidia introduce new high-performance compute architectures based on data center-grade hardware, we can only wonder whether TinyBox, based on consumer-grade hardware, will gain popularity.
