The N64 came with 4MB of RAM soldered on the board. With the Expansion Pak, you could add an additional 4MB of RAM to bring it to a total of 8MB. Was 8MB the maximum amount of RAM that the N64 could support or could a person in theory develop an Expansion Pak with more than 4MB of RAM? Why or why not?
In theory, yes it could.
The RDRAM Interface (RI) inside the Reality Coprocessor translates accesses to the first 63 MiB of address space into RDRAM accesses, and provides enough access to RDRAM registers using the next 1 MiB of address space to let you configure all 63 MiB as usable RAM.
However, the Nintendo-supplied RAM initialization routine only attempts to initialize 8 "modules", each of which can be 1 MiBsize or 2 MiBsize of DRAM. This makes the upper limit 16 MiB - 4 MiB from the two onboard modules, and up to 12 MiB from the expansion pak connector. To go beyond this limit, you'd need new RAM initialization routines to start up more modules. If you used 1 MiBsize modules, then you'd be limited to 8 MiB without new code.
The console itself contains one or two chips providing RDRAM. The single-chip option is internally 2 modules of 2 MiBsize each; it is not clear from online resources whether the two-chip option is also 2 modules, or if it's 4 modules of 1 MiBsize each. If it can be 4 modules, then the upper limit for commercial games would go down to 12 MiB - 4 MiB from the 4 internal modules, plus 4 modules of 2 MiBsize in the expansion pak for an extra 8 MiB.
It is also unclear whether the RI was validated with more than 8 modules; it's entirely possible that the limit is 12 or 16 MiBsize because the initialization routines reflect the upper limits on how much RDRAM released RIs can reliably drive, and new initialization routines would simply expose RI bugs.
size: The modules are actually 9 bits wide, but one bit is hidden from the CPU and only used by the 3D engine. A "1 MiB" module is in fact 1 * 9 * 1024 * 1024 bits of DRAM, and a "2 MiB" module is 2 * 9 * 1024 * 1024 bits of DRAM.
In theory you certainly can, just by paging the RAM in the expansion pack. Basically you would connect a register to the last byte(s) of the 4MB address space and use it to hold high address bits of the address space. Whenever you want to use a different 4MB page, you would write the higher 8 bit of the new address range into this register. This is very simple to implement.
The harder part is to write a game to support memory paging and to take advantage of additional memory.