Why would adding a cart in your system suddenly speed up the C64 or the 1541 for loading games?
The VIC-20 and the C64 had horribly slow disk IO because everything was done by the CPU over a serial link, i.e. the CPU was reading in the data bit by bit. Both software and cartridge "turbo loaders" worked by replacing the protocol with something more efficient, for example by doing synchronization only once per byte instead of once per bit.
I think I read somewhere once that some also used the clock wire as a second data wire, but I'm not sure if this was something that could be done purely in software/cartridge or if that required a direct connection between the drive and something plugged into the user port of the C64.