Debuggers are carefully written programs that peek and poke other programs while they run. In retrocomputers, programs could use any part of the memory they could access.
So how did debuggers insert themselves into memory so they could be executed, without overwriting - or being overwritten by - the target program or its data?
Without hardware support, there is no way for a debugger to protect itself from the program being debugged. A debugger needs to protect its code and working state, as well as any hooks it's set up for its debugging (e.g. single-stepping interrupts); but in typical "retro" systems (8-bit machines), a debugger would just insert itself into memory and hope for the best.
Debuggers hosted on the same system as the code being debugged, and implemented entirely in software, would only be able to rely on the cooperation of the program being debugged and the person doing the debugging. Most debugging isn't adversarial: the code being debugged isn't actively taking measures against the debugger, so the debugger doesn't need to actively protect itself. (If a bug destroys the debugger as well as the program being debugged, well...) The debugger can offer configuration options to the end-user to facilitate things of course, but there will always be limitations that the end-user will have to live with (using a debugger leaves less RAM for the program being debugged, it might use some interrupts that the program would want for itself, etc.) — all told though the gain from having a hosted debugger makes it worthwhile.
Some early debuggers used extra hardware, without special CPU support, to reduce the level of compromise:
but, without a MMU, all this still happens in the same address space (even if banking is involved) so it's only reliable in non-adversarial debugging.
Adversarial debugging, or any form of complex debugging really (kernel debugging etc.) requires hardware support. This can take a number of forms:
Hosted debuggers, especially for adversarial debugging, only really took off with the arrival of MMUs and CPU-based protection; e.g. Turbo Debugger 386, SoftICE... Multi-host debugging is still used in many cases, e.g. embedded development, development on mobile phones (using emulators or real devices), kernel debugging...
There were some systems where the whole operating environment was constantly debuggable, e.g. Lisp Machines, but again that's non-adversarial debugging in most cases.
A debugger that runs inside the debugged machine is a program, so it does need memory.
Sometimes the debugger is loaded as a ROM cartridge, usually with its own RAM, so it doesn't need to take any RAM from the running program. This is the case with, for example, the Action Replay modules for the Amiga.
Sometimes, it's a regular program that takes some RAM, but it can be reallocated at load time so if you know in advance that the program to examine won't use a certain block of RAM, you can load the debugger there. This is the case with, for example, the MONS 3 debugger for the ZX Spectrum
Sometimes, the debugger cannot find any "normal" RAM to execute from, so there are debuggers that load themselves into the screen memory, like this one:
Yes, that's a program. You can try to dump this screen into a ZX Spectrum emulator screen and do a PRINT USR 16384 to execute it :)
The only kind of debuggers that don't really need any memory at all from the debugged machine are the so called ICE (In Circuit Emulators), which either take the place of the main CPU, emulating its behaviour and reporting data to a external machine, or are logic analyzers that can decode bus cycles and present you a disassembly dump of real instructions actually executed in real time by the machine being debugged.
I developed Logo for the Commodore 64, based on work we did at MIT for the Apple ][ and TI 99/4. Apple debugging was done with the ROM, via assembled-in breakpoints. For the C64, Andy Finkelstein at Commodore ordered me a 6510 CPU with an extra pin that signalled the I/D line, and a clamp-on connector that led to a Nicolet-Paratronics logic analyzer, connected to a PET with a disassembler in BASIC. I was having trouble with usage of page zero registers, both by the ROM and by Logo, which deferenced Nil (empy list [] in Logo), but on the 6510 locations 0 and 1 were the parallel port to peripherals such as the 1541 disk.
So, I could set a breakpoint when dinner code wrote or even read an address. What's even better is that I could set the breakpoint to start up to 255 bytes before the errant access occurred. It was the only time I could set a breakpoint in the past before an error occurred. And yes, it was more miraculous that the Lisp Machine debugger!
On 8-bit machines any debugging support would just be a normal program in memory, and the application could accidentally overwrite the debugger. To single step through a program involved replacing some code in the application with a call or software interrupt that returned control to the debugger. It was all too easy to get a breakpoint in the wrong place and completely lose control of the debugger and the application.
Later on (in the early PC era), this principle was still the same. MS-DOS had a DEBUG application, and this would load in memory first, in turn loading the application on top of itself higher in memory. The 8088/8086 chips had no way to protect one segment of memory from another, so MS-DOS was not designed to manage memory to protect applications from each other. Consequently it was quite possible for the application under test to accidentally clobber the debugger and prevent it from working properly. Sometimes the machine just locked up and you just had to start back at the beginning.
ROM Monitor - effectively a very similar program, but ever present in EPROM rather than loaded into RAM as needed.
Jun 6, 2016 at 20:36
A debugger may be able to defend his integrity and memory space even in arcane architectures.
Granted that in the most common implementations, where you want to save speed and memory, and simplify the debug code, you are dependent on hardware protections provided by more advanced architectures.
The software can very well have anti-debug measures built-in for known debuggers or accidentally overwrite it in the course of loading or even in the normal operation of the program.
However, if you are willing to sacrifice memory and speed, it is rather easy to do "emulation" in the same CPU/interpreting critical commands and running others in a controlled area, at least more easy than implementing an emulation in a foreign architecture, as you are implementing actual arithmetic and bit operations using the native machine code instructions .
Taking the ZX Spectrum as an example, think it as an emulator for Z80 written in Z80 binary code just for the purpose of having an advanced debugger - think of it like pretending Z80 code is a foreign CPU, or as p-code and interpreting it (similar to how a BASIC program works).
To be fair, before I wrote my own Spectrum emulator for Windows, I was toying around for long about writing a native Z80 interpreter/emulator in the actual Z80 machine as a proof of concept, and even wrote the basics of this idea for debugging in a real ZX Spectrum 48K behaviour of undocumented Z80 opcodes for my emulation.
Even more interestingly yet, in machines like the TC/TS 2068 where you can control RAM paging from the 48K mode, you might get away with debugging full 48K programs if you get your paging code right.
Actually with such scenario you might even page in a screen with the debug interface and at the when running the commands write to the actual addresses/pages of the "real" screen.
Such a tool, besides the obvious debugging capabilities, would perhaps be more interesting as an educational tool if used contemporarily before emulation development systems in more advanced architectures became common place to hobbyists.
You might even extend such a tool with help of hardware, where you load a very protected piece of software (a turbo loader with the Alkatraz protection for instance), and later on take control of it with a NMI interrupt, passing control to such debugger in another page of RAM.
P.S. The comment of TS/TC 2068 might also apply to Spectrum +2 hardware, never had one so not sure.
The concept is also not new, and it was implemented in the past, albeit in more advanced systems. One of the oldest (from 1994) and better example I know of, is Bochs
Bochs (pronounced "box") is a portable IA-32 and x86-64 IBM PC compatible emulator and debugger mostly written in C++ (...) It supports emulation of the processor(s) (including protected mode), memory, disks, display, Ethernet, BIOS and common hardware peripherals of PCs.
Many guest operating systems can be run using the emulator including DOS, several versions of Microsoft Windows, BSDs, Linux, Xenix and Rhapsody (precursor of Mac OS X). Bochs runs on many host operating systems, including Android, iOS, Linux, Mac OS X, PlayStation 2, Windows, Windows Mobile.
Bochs is mostly used for operating system development (when an emulated operating system crashes, it does not crash the host operating system, so the emulated OS can be debugged) and to run other guest operating systems inside already running host operating systems. It can also be used to run older software – such as PC games – which will not run on non-compatible, or too fast computers.
A concept that was sometimes useful in the 8-bit era, was the "In Circuit Emulator". These devices were very expensive, but had a header that would plug into a CPU socket in place of the CPU. The emulator would contain a CPU along with some logic for bus monitoring and bus switching, as well as either its own display (e.g. six hexadecimal digits) and keyboard (20 buttons or so) or else a means of connecting a separate computer. The emulator would contain its own RAM and ROM, but they would be completely "invisible" to the system under test, since they would only be banked into the host processor's address space at times when the emulator was running its own code.
Emulators would allow transparent debugging of code on systems that did not contain any provision for debugging features beyond the presence of a CPU socket rather than a soldered-on CPU chip. A computer system and an emulator would together cost much more than a computer system that was designed to facilitate debugging, but if one was trying to debug code for a mass-market device, and no system that was compatible with it supported debugging features, an in-circuit emulator could fill that gap.
Nowadays many microcontrollers contain some hardware which allows a simple PC-attached microcontroller with a couple of I/O pins to supply most of the functionality that would have previously required an in-circuit emulator, and thus many people who started working with microcontrollers within the last decade may never have had any reason to want a real ICE, but in the 1980s and 1990s they were extremely useful to those who could afford them.
Many debuggers were limited to setting breakpoints by patching in trap instructions; some would keep track of the original instructions at the breakpoints, while others would require users to manually patch in trap instructions and then manually restore the original instructions.
Some debuggers supported single-stepping of application code either by using hardware-single-step interrupts or by interpreting the instructions being executed. Debuggers without hardware assistance couldn't generally provide any sort of memory read/write traps, but MacsBug circa 1991 included a feature that would combine a single-step with a memory-compare operation. Code execution in step-spy mode was at best somewhere around 1/10 normal speed (I never measured it, and I think it could go much slower depending upon selected options) but the application worked, and I was able to make it stroll along to the point where something was getting incorrectly overwritten, whereupon it fell back into the debugger and I could see what was going on.
For non-virtual-memory systems, the answer for 'insertion into memory' was usually simple: you linked your program with the debugger, just like linking with any other object module.
The linker or loader might provide some minor support, such as transferring control to the debugger's entry point rather than the usual entry point, on program startup.
And, of course, as in many systems today, you got to say whether debug information is left in the program image file.
Absent specific kernel support for cross-address-space access, the debugger had to be in the same address space as the code being debugged. Absent hardware protection (such as read-only code), the debugger was at risk of corruption just like any other code. On the other hand, in my experience, things rarely got that bad. Bad references were usually of the "off the end of the array" or "use after free" type.
