For much of the past decade, companies moved data into their analytics systems with pipelines written as code: Spark or Python jobs that pulled records out of operational databases, reshaped them, and loaded the results into a warehouse. The approach works, but it is brittle. A single change upstream, such as a renamed or newly added column, can stop a job and send an engineer digging through custom code to find out why.
That brittleness has become more expensive. Companies now want the same data to feed dashboards, machine-learning models and AI agents, and the condition of that data is increasingly the limiting factor. In a July 2024 survey of 1,203 data management leaders, Gartner found that 63 percent of organizations either do not have, or are unsure whether they have, the right data management practices for AI. The research firm predicted that through 2026, organizations will abandon 60 percent of AI projects that are not supported by AI-ready data.
