I can answer the last part of your question: yes, the Mac II could use a 68851.
For a time, Virginia Tech required incoming Computer Science freshmen to purchase a (heavily discounted) Mac II running A/UX, Apple's first in-house-developed UNIX. These machines shipped with 2MB RAM, an 80MB HD (huge for the time), and a 68851 pre-installed; A/UX required the 68851 to work.
Somewhere, in a dusty corner, I have a prototype Mac II that has a small daughtercard in that socket instead of a 68851. I may also have some documentation from the beta-testing period that would shed light on the "alternate" MMU.
I believe the Mac II first shipped with Mac OS 6, which didn't support virtual memory. I don't remember if VM support in System 7.(whatever) required the 68851.
As I recall, the Mac II shipped with 256KB SIMMs, supported 1MB SIMMs (up to 8MB total RAM), and was expected to support 4MB SIMMs when they became available. Unfortunately, manufacturers changed something in the electrical interface for 4MB SIMMs, so they didn't work in the Mac II. I think someone may have made a 4MB SIMM that could work in the Mac II, but they were low-volume and prohibitively expensive, and the Mac product line had moved on by then. So, "maxing out your RAM" probably means 8MB -- but that still exceeds the 24-bit address limit in early Mac OS, so you may need a "real" MMU to take advantage of it.