Let's first establish that all computers that came with a programming language in ROM could also load a programming language, take a (E)(P)ROM you burned, or either had a cartridge port or something similar like a bus or expansion slot. (As in, the presence of a capable boot ROM was highly desired, and not only could you replace it if you didn't like it, but you could use it to load what you wanted instead of starting blank and entering bits/octals/hex/bytes into RAM just to load from paper tape.)
Home computers of the 80s vs Personal computers of the 70s
Personal computers of the 70s were marketed towards professionals or certainly adults already interested in electronics and hardware programming. You were expected to homebrew parts of the solution, both software and hardware with very few exceptions, or else it came with a cost few families could afford.
Home computers by contrast sold millions, and they were about the cost of a small color TV or record player - a household appliance, not a really good used car. This allowed not only some kids to get one, but there would be TV series to teach you programming, and schools could also afford enough of them, to become part of this nationwide computer education - "let's bring everyone into the future!" - no-one should feel left out of the computer revolution, feel left behind the times. (This was the concern then, not only as countries but individuals - socially, economically, intellectually.)
Why boot to a programming language?
Most if not all of the 8-bit computers of the 80s booted to a programming language prompt precisely in order to become directly programmable. (See last section.)
Licensing. Computer manufacturers didn't hire programmers to develop their own built-in languages (there are 3 known exceptions). Microsoft has stayed true to using the code of others or buying a port, then license it as their own product with adaptations to computer manufacturers. They recognized the success of BASIC since the 1960s, as did from necessity those homebrewers and hardware programmers called hobbyists in that open letter.
OP is asking for a system that came with a competitor to BASIC. (Not for systems lacking a programming language in ROM, etc.)
The Jupiter Ace came with Forth, as mentioned in the question. I've heard of French, Polish, and Japanese home computers that came with different languages, but I can't substantiate details at the time of writing, as these are one of the few I don't own. As a consequence of the dominance of the licensed built-in commercial monopoly, it's now rare to find these systems as well as detailed information about them.
Partial language support
Most boot ROMs of any system will have so-called routines. In the 1970s, because of ROM and RAM sizes, these would be to provide very basic functions indeed, such as I/O interfacing, but no complete interpreter or compiler - because a big enough ROM to hold it would be prohibitively expensive.
Obversely, this is the strength of a language that is truly built in: without anything but the computer, it could be immediately be put to use - to run your programs. This is what makes home computers more personal than today's personal computers, who turn users into only that, disempowering them.
Late 80s and 90s
The routines (kernel if a reasonably complete set of routines) would again challenge the size and cost of ROM chips, so that no language could be built-in.
ROM sizes did increase for some systems offering a richer operating system than a Shell or language with disk operations added: The Lisa and Mac offered a GUI OS, and the Amiga a modern multitasking one with many features of Unix and Linux, but no longer could you boot to a built-in programming language.
The IBM PC appealed to the 1970s by restricting it to just I/O routines again. This was kept for the CP/M clone branded MS-DOS, and for Linux.
The Acorn Archimedes is currently the last platform to have both a GUI OS (if not modern multitasking) and a built-in language in ROM, even though it might not be a valid answer to the question as the language is called BASIC. (I would pay attention here though, as it's quite different from all other BASIC dialects in merit, and deserves consideration.)
What's a built-in language good for?
The strengths of booting to a programming prompt (over a Shell prompt with limited script and file commands, or a desktop paradigm that you can only click on) is not to be underestimated. Certainly, computers are (sometimes) easier to use than ever - but mostly only to those who would rather not program, and just remains users.
Today, we have to set up things just right on computers (with actually some quite special software and elaborate preparations) to make it as easy as it once was for kids to instantly pick up that "this is a computer, this is how it works, and this is how you make it do what you want".
I think it's educationally powerful, and much more social and democratic, embracing all walks of life, rather than leaving it to only nerds and professionals again, like in the 70s. It really wasn't like that in the 80s with the home computers, we all typed in programs and learned because it was fun.