I think the problem of caste is much deeper than people consider. The difference between a bad idea and a really bad idea is that bad ideas dont work and people walk away from it as it doesnt work, a really bad idea is an idea that actually works by tapping into cognitive biases leading us into a cognitive quicksand.
The problem is much deeper than simply one of dogmatic belief with regards to hierarchy. Unlike other places, Jains, Buddhists competed with Hindus, many Hindu traditions also competed as well , some egalitarian. Lingayats for example rejected both vedas and caste for example , there was also ramanujacharya as well.
Question is, what were the utilitarian features of caste that allowed it to exist until Industrial revolution. It was never the case that people did not have choice to choose sampradaya . Infact the biggest feature of India and Hinduism was diversity of beliefs itself.
If one would assume to study the evolving nature of changing paradigms in connection to the actual choices they could actualize, One could consider that actual egalitarianism in practice was possible only after Industrial revolution. Before that, societies made compromises on it. Some went further to bring in slaves from elsewhere, here too Hinduism is different, there is not much evidence of them bringing slaves from elsewhere.
If caste did not offer any relative localized material benefit, it would perhaps not have persisted. some might choose to believe that for thousands of years , people were brainwashed into believing their own submissiveness and were basically turned into serfs with total lack of selfishness for wealth, power and individuality. I trust biology over beliefs and think this is ridiculous. I think that men of all backgrounds would prefer to have access to wealth, female partners of their choice and power.
Even now in democratic period, caste offers political networks and material benefit as well. Especially for lower castes, without which they could not have achieved their relative success given Indian economy was pathetic overall for most of post Independence period.
The reason for existence and persistence of x in face of competition and also when its traditional guardians were out of power with the islamic period and later british period as well and even now in democratic period points to relative localized benefits. One answer could be that lingayats,ramanujacharya, buddhists,jains ,atheists and later Islam and christianity ,marxists and liberals were not sufficiently evangelical. Or lack of study of history in India made people ignorant of possible choices they could actualize. But that only points once again to the effort it would take to undo it. I assume only when India is much richer will that challenge the material side of benefits leaving the political side of it and dogmas.
The hierarchical feature tallying with it simply has to do with the first mover advantage. It is easier for individuals to rise and fall over time than entire chunks of different communities. If one focused on explaining persistence of x in face of competition and also when its traditional guardians were out of power , it explains a lot. It just is the case that sometimes societies fall into stable configurations that persist for long even if it hurts them overall, it hurt Hindus and Hinduism more than it did good. I think Human cognitive biases in favor of kinship tribalism accounts for this. And idea of caste was a really bad idea that they stepped into a cognitive quicksand. I think scientists from field of evolutionary psychology should find it interesting to study.