News coverage surrounding artificial intelligence almost invariably focuses on the deals that send hundreds of billions of dollars flying, or the latest hardware developments in the GPU or datacenter world. Benchmarking efforts have almost exclusively focused on the silicon, though, and that's what SemiAnalysis intends to address with its open-source InferenceMax AI benchmarking suite. It measures the efficiency of the many components of AI software stacks in real-world inference scenarios (when AI models are actually "running" rather than being trained), and publishes those at the InferenceMax live dashboard.
InferenceMax is released under the Apache 2.0 license and measures the performance of hundreds of AI accelerator hardware and software combinations, in a rolling-release fashion, getting new results nightly with recent versions of the software. As the project states, existing benchmarks are done at fixed points in time and don't necessarily show what the current versions are capable of; nor do they highlight the evolution (or regression, even) of software advancements across an entire AI stack with drivers, kernels, frameworks, models, and other components.
