Speaking from my personal experience of writing a PET emulator, a C64 emulator and a Sinclair Spectrum emulator,, here are the issues I had:
Getting the Speed Right
It's no good just making a processor go as fast as it can because, frequently, application code depends on timing. For old 8-bit machines, it's easy to write an emulator that runs at many times the speed of the original. The trouble is that has a knock on effect. For example, the PET Space Invaders program goes way too fast to be playable. Not that it matters because its key scanning code is similarly speeded up which means when you press a key, your gun is uncontrollable.
The same issue applies to the C64. The interrupt is driven by one of the IO chips which needs to be synchronised to the CPU's clock. This means pressing a key for even a brief time is the same as pressing it for several seconds on a real C64.
So you need to throttle the performance to something like the original speed. Unfortunately, that means having a clock with microsecond accuracy. Such things do not really exist in modern general purpose PC operating systems. Your CPU thread just has to be scheduled out of the processor and it will miss several microseconds probably. One way to get around this is to raise some event every 1/60th of a second (the probable refresh rate of your monitor and handily NTSC TVs) and when the CPU has executed 1/60th of a second's worth of instructions, just make it wait until the event occurs. Unfortunately, that makes doing sound on a PET or a Spectrum difficult because they both rely on the CPU toggling bits in IO registers at the right frequency.
Parallelism
In a real computer, there are several components that all operate concurrently. For example, the C64 has a VIC II chip for the display, a sound chip, some IO chips ands a 6510 and they are all synchronised by the same clock. The easiest way to deal with this is to have a loop in which you execute an instruction in the CPU and then update all the other components with the new clock time. Unfortunately, this is by nature serial and you have to be careful about do complex stuff in case it makes your emulation too slow.
An alternative is to put each component in its own thread, taking advantage of the multiple cores of modern computers, but then you have the problem of synchronisation. All of your components will need to access your emulation of the memory bus and they all need an access to the same copy. So, you might emulate the clock with a boolean that is toggled every 0.5 microseconds (in emulation time, see above) by the CPU. Unfortunately, modern processor cores have caches between themselves and the main memory. If the thread emulating the CPU core toggles a boolean representing the clock, it may only actually be altering the cached version of that variable and the other components won't see it. There are OS functions that allow you to force the cached version of a variable to main memory, but they incur a significant performance penalty. It's about 100 times slower to access main memory than L1 cache.
Documentation
Documentation for old computers can be quite hard to find and may not be detailed enough for constructing an accurate emulator. For example, if you want an accurate Z80 emulation, you need to understand that there is an undocumented "w" register which affects the behaviour of some of the undocumented Z80 instructions. In theory you don't need to care about those, but in practice, some popular game might have used them. The behaviour of the W register has been painstakingly reverse engineered by enthusiasts, but sometimes they get it wrong.
The other problem with old documentation is that it frequently contains errors. A popular book on the 6502 was the Zaks book, Programming the 6502. I remember that my Dad's copy of it was festooned with hand written annotations correcting all the errors that he discovered by bitter experience.
Graphics
Getting the graphics right is pretty hard. I started by just taking a dump of the graphics memory every 1/60th of a second and drawing it in a window. I progressed to doing that in the GPU but it is still not right. C64 programmers were adept at changing the graphics mode on the fly so they could use mixed modes on the screen. Even the Spectrum effect of the rapidly moving stripes in the border when the tape is loading is done by rapid changes to the background colour. You can't just snapshot the state and render it every 60th of a second, you effectively have to know the state at the end of every scan line on the VDU and, in fact, on the C64 I believe it was possible to split the screen vertically by carefully timed mode changes during a scan line. I haven't solved that yet.
Sound
Timing is actually more important as far as sound is concerned than graphics. A film is projected at 24 frames per second and our brains easily fill in the gaps. Try something similar with sound and you'll soon notice. For this reason, a haven't even attempted to emulate sound on the PET or the Spectrum. The C64 should be easier because it had a sound chip that you sent commands to rather than having to toggle an output wire very fast, but I haven't done that bit yet.
Development Tools
You'll need to create test programs for your emulation which means having a development suite. I was lucky in that Z80 and 6502 are both relatively well supported. Other architectures are not so fortunate. Not finding a good toolchain for 68000 stopped me from bothering with that architecture.
Responses to Comments
I woke up this morning and mistook the number of comments I had as my accumulated overnight score. (I nearly fell off my chair when I saw what my actual score was, thank you.) There are many points to answer which I don't feel I can do in the comments easily, so I will put my answers here.
I feel [your answer] understates how big of an issue parallelism is ~ Austin Hemmelgarn
Yes it does. But I have so far only attempted to write emulators from systems from the 80's which tend to be slow by today's standards and also very simple. Thus, you can get away with serialising a lot of the parallel components and, in fact, that might be desirable given that running too fast is often too fast.
Ironically, I have found that components that run off different clocks are sometimes easier to work with because the protocols between them cannot rely on clock synchronisation, so there is usually some hand shaking mechanism that means they don't rely on absolute timings.
As a simple example that expands on Luaan's reply, the PET interrupt is driven by the vertical sync of the monitor (or it appears to be), so I simply had a task triggered by the actual monitor refresh which is 60 per second that raises an interrupt. This meant that the keyboard scanned at the right speed and the cursor blinked at the right speed even though the CPU was running at about 80MHz. By contrast, on the C64, the same interrupt is driven by a timer in one of the IO chips which itself is driven from the CPU clock, which means it looks like the C64 runs ridiculously fast.
This is only true if you decide to suspend your thread. And you actually miss somewhere around 1 millisecond which makes thread sleeping very undesirable for game dev in general. You can create your desired delay without releasing the thread. This allows you to get microsecond precision. ~~ Chris Rollins
On a modern general purpose PC, you have no choice about when your threads get suspended. My laptop has 8 cores and 16 hyperthreads, but there are way more threads than that running on it at any one time. Most are suspended, at any one time, but it's quite possible for the emulated CPU thread to be pre-emptively suspended at periods of high load without you doing anything. Furthermore, in order to time things at the microsecond level, you need a microsecond accurate clock. Note: precision is not accuracy. My laptop has a nanosecond precision clock, but it would be a stretch to assume you can measure time periods accurately to the nanosecond with it. For one thing, it requires a system call and system calls are relatively expensive and also lay you open to rescheduling your thread. Your milage mat vary depending on your operating system.
Don't modern sound cards buffer sound data so you don't have to bit-bang them? ~ snips n snails
I had thought of counting the number of toggles in a certain period and counting the CPU clock cycles to get a frequency to send to the sound card, but I don't know enough about programming modern sound cards to say if that is reasonable or would give good fidelity. I guess you actually don't need it for a Spectrum or a PET :) . Having just read the next comment by Ilmari Karonen, this is the same approach. I think you'd need a shorter period than a frame (I assume you mean a video frame).
Without a barrier, your later loads don't wait for your earlier stores to be visible to other threads, ~ Peter Cordes
Yes I admit I was running fast and loose with the actual mechanics of cache and memory synchronisation in the interests of brevity. The point is that ensuring correct sequencing of memory updates requires special expensive synchronisations like memory barriers. This is over and above the penalty of going to main memory relative to the cache.