The arrival of Islam forced the caste order of India to reinvent itself. Between 1300 and 1600 AD, Sultanate and Mughal rule remade Indian society from the ground up. The Muslim ruler did not care for caste. He cared whether you could fight, trade, farm or pay tax. Shudras, meant to be service providers to the elite, became peasants with land, merchants with capital, artisans with guilds, sometimes even rulers with armies. The old four-fold varna map, established in Gupta times (400 AD), no longer matched the territory.
Brahmanical scholars faced a problem. Dharma claims to be eternal. Eternal things are not supposed to change. But a Shudra landowner living better than a starving Brahmin priest is a fact that eternal texts cannot simply ignore. So Sanskrit scholars did what they always do when reality embarrasses doctrine. They wrote their way around it.