Nvidia built its AI empire on GPUs. But its $20 billion bet on Groq suggests the company isn’t convinced GPUs alone will dominate the most important phase of AI yet: running models at scale, known as inference.
The battle to win on AI inference, of course, is over its economics. Once a model is trained, every useful thing it does—answering a query, generating code, recommending a product, summarizing a document, powering a chatbot, or analyzing an image—happens during inference. That’s the moment AI goes from a sunk cost into a revenue-generating service, with all the accompanying pressure to reduce costs, shrink latency (how long you have to wait for an AI to answer), and improve efficiency.
