Food in Beast of Reincarnation is actually two separate systems bound by a single purpose. Field food keeps Emma and Koo’s hunger meters filled so they can passively recover HP outside combat, while the meals prepared by Kagura provide stat improvements before your next journey.
Farming inside the Cleanse Walker supplies many of the ingredients for those meals, so ignoring it will eventually catch up with you.