VIC-20 keyboard seems to have become sort of a standard for [...] the MAX Machine (in membrane form), C64, C16 and (with additional keys added) the 128.
The C64 decision was easy, as it was done to save development time - existing part. Similar for the C16. For Max as well for the C128 it was a compatibility issue.
What keyboards and layouts did Commodore use up until the VIC-20 and how did they compare?
There are basically three keyboard setups, with some variations in usage (assignment of sybols/functions):
PET Chiclet (*1) keyboard

(Taken from Steve Gray's page on 6502.org)
That's the well known 73-key (*2) chiclet (*3) one we all love and hate. It sets the tone for all keyboards to follow.
PET/CBM Graphics keyboard

(Taken from Steve Gray's page on 6502.org)
Next in line was the graphics keyboard introduced with the PET 2001-N. Essentially the chiclet moved into more standard form with 70 keys (*4), featuring a real space bar and all keys of the fifth row moved to the sides of the main section. This was the primary Keyboard for all 40 column models.
Despite the name, the keyboard, used thruout the 3/4000 series, was available with and without graphics symbols printed on.
PET/CBM Business Keyboard

(Taken from Steve Gray's page on 6502.org)
Commodore was, before becoming a (home) computer company a competitive business supplier, so it was natural to place the PET for the office. Just leaving of the graphics symbols from the front of the graphics keyboard, as offered with the graphics keyboard, wasn't really doing it. So a 'new' machine, the CBM 2001-16B was introduced, featuring a keyboard version more fitting to office user expectations.
For the now (again only) 73 key (*5) version this meant foremost creating a numerical block in line with desktop calculators. To accommodate this the numeric block was striped of everything but digits and period (the zero key was double width), and moved in line with the main keyboard, removing any keys at 'space level'. Much like with the transition from chiclet to graphics, all unique (*6) key got again moved to the sides of the main keyboard.
Later it became the standard for the 8000 (80 column business) series.
Later Keyboards
PET II keyboards moved ahead to an even more conventional layout with lots of extras adding up.
VIC-20 keyboard is based on the main section of the business keyboard with some keys moved for eronomic reasons (cursor) and CTRL instead of TAB.
C64 keyboard is the same as VIC-20
C16 (TED) Keyboard is a shaken up VIC-20/C64 Keyboard, with ESC added and cursor keys up again but finally on seperate keys. Over all more following the at the time establishing standard while still using the same small layout.
C128 is a complete redesign, and for sure the most weird modern (*7) keyboard. Of course that's in part due the imperative of C64 compatibility.
In particular, which keyboards ever had CTRL and ESC keys?
CTRL: When ignoring international layouts, it was the VIC-20 introducing a control-modifier key - instead of the TAB key present on PET machines.
ESC: The CBM business keyboard did first feature an escape key.
Did any have keys or characters not available on the VIC-20 keyboard?
To start with TAB of course, as well as OFF/RVS and Repeat. More important here would be different symbol assignment. So for writing keyboard based action games you may want a different assignment table per model.
it may be simpler to ignore these [country-specific keyboards] and focus on just the U.S. keyboards.
Too bad, as that excludes the eventual first occurance of ESC
as well as CTRL
on CBM keyboards with the (well, a) German variant of the business keyboard.
Further reading:
Both hosted on the ever great 6502.org.
*1 - Often as well named as chicklet, as it fits the typing style - picking like a chicken - quite good. Also, the gum isn't known in many places.
*2 - Key caps that is - but as well decoded keys, as it was a full scan with no duplicates
*3 - While the term chiclet keyboard has been used for the PET keyboard, it isn't one. It's not basedn a rubber mat, but 'real' keys, build from switches and springs. So technology wise most modern keyboards are chiclet disguised with hard keycaps instead :)
*4 - Interestingly the 70 keys come down to again 73 unique switches for decoding.
*5 - While it got 73 keys again, they now produce 78 unique switch positions.
*6 - Meaning only the keys that did not already occure on the main section, like operators (/, *, +, -, =), got moved
*7 - By 1985 the search for default layouts was basically done even for home computers