What books and approaches did people take to writing adventure/RPG games?
Alright, you asked for history. Here we go.
Basically (sic), one would start with some of the books/magazines that had listings of games in them. All kind of games, not only adventure/RPG.
Then you'd sit down and type those listings in. It took a while, you made errors, but at the end you'd have typed (and thereby also internalized) the whole source code. In about 75-90% of cases the game, then, even worked and you didn't have to go through all of it, multiple times, to find that one typo.
Later, they went and changed it to hex encoded strings with checksums, especially if machine code was involved, but at that time there was not much of anything to learn anymore (the magazines I had tended to include the assembler files as well, IIRC, so you could at least try to follow what happened).
After a while, you would dabble in your first own games or other programs. There were printed books about all parts of the 8-bit computers (BASIC, assembler, graphics/sound chips, overall system architecture, the actual hardware, and so on) in technical reference formats - what an engineer would expect; very detailled but you had to kind of figure out how to use all of it. Magazines regularly had great series where they sat down and walked you through major components and how to program them (e.g., video processors, sound etc.).
Magazines also were a great source for specific components of specific game types - e.g., one could expect a series about data management in a RPG, how to parse the crude text languages in an adventure, or how to store the "rooms" and such. Most of these was concerned not with advanced things like OOP (what we would have today), but how to use shortcut after shortcut to implement it in the very limited space and with no particular programming libraries in existance.
Then there was the demo scene, which gave you ideas about what the upper limit was regarding video and sound effects. The demos usually did not come with source code, so you had to use a disassembler. The demo code was also in my experience not much use for anything else as it usually relied on very exact timings (the code was timed to run in sync with the actual monitor being built up not only frame-by-frame but line-by-line or even pixel-by-pixel, often controlling the "actual" color of the electron beam more or less directly without even bothering to store any image in video RAM, which would have been much too slow to do it for every refresh cycle).
After a while, you just sat down and "did it". The complexities then started to arise when RAM got full - that happened rather quickly with BASIC; you had to start paging 256-byte chunks of RAM in and out and whatnot. Assembler was an option, but it got overwhelming pretty quickly - there were none of the comfortable toolchains around; having a text file editor (instead of something that used line numbers) was not necessarily standard. There were fables about wild guys using PASCAL or even C dialects on 8-bits, but I'd guess something like 99.999% of all games were either straight BASIC (maybe precompiled/obfuscated), assembler or a mixture. Wikipedia says that Ultima 1 was BASIC and Ultima 2 and on were 100% assembler. That should also explain why Richard Garriott made it into space and is a half maniac. :-)
Keep in mind that even the most complex of games back then were incredibly simple by todays standard. Of the thousands of 8-bit games, there are a handful of exceptions (Ultima, Elite ...). If one of the other games was great, then due to the artful execution (game idea, graphics, sound...) within the limits imposed.
Also, all hardware components of an 8-bit computer are vastly, vastly simpler than even the most mediocre PC you have today. There is really almost nothing to it. Sit any 15-year-old down with a lot of curiosity and a heap of books and a lot of time, and he or she can easily grasp everything there ever is to know about them.
I want to write a game myself since I'm a much better programmer now than I was when I learned BASIC on my Apple II.
I did not write such a complex game as Ultima or Elite myself, obviously, but I did dabble a bit; I recall breaking the BASIC RAM limit plenty of times, and having incredible fun (sic) in assembler, frequently freezing the machine on which I was developing while running the program under development, occasionally losing all source code, and so on.
So I'd say the biggest challenge aside from physical limitations was to keep track of all the data without growing crazy, while writing the code in assembler language from scratch, with no meaningful toolchains, debuggers and whatnot.
If you edit on modern machines, cross-compiling into that environment, with all kinds of resources at your fingertips (see the links in other answers), maybe even a useful debugger, you have a huge benefit already. From there on, I'd say it's the same as learning any new environment these days - figure out how to do stuff step by step, and then simply start doing it.