For historical reasons most computers treat a byte as the smallest unit of data they operate on. A byte only contains 8 bits, so to store a 32 bit value you need 4 bytes. The question is what order the bytes should be stored in.
The N64 has a MIPS CPU which is Big Endian. That means it stores the most significant byte first, a lot like how we write numbers in English. For example if I write 1234 then 1 is the most significant digit, it's the thousands digit. 4 is the last significant digit, the units, and comes last.
Most PCs are Little Endian, meaning they put the least significant byte first. That would be like writing the above number as 4321, i.e. units first and thousands last. There are historic and technical reasons why computers do this, and it was one of the great holy wars back in the day. The name itself refers to Gulliver's Travels, in which there is a war over which end of the egg to break open (the big end or the little end).
The V64 format is like writing the number as 2143, or hundreds thousands units tens. That rather strange order came about because of the design of the Doctor V64, a peripheral for the N64 that allowed ripping ROMs to a PC and is the source of many early files. It's not clear why they selected the unusual byte order, it may have been a primitive form of protection to prevent the ripped ROMs being used on competing devices, or it may simply have been a mistake.