One thing that has dogged Xbox users since the beginning of time is the inconsistent speeds games often download at.
The same seems true across both Xbox PC and Xbox Series X|S (and older) consoles, where download speeds often fail to come close to what you're actually paying for via your ISP.