I am asking this here because from the history the principles might become much clearer than looking at the latest and greatest CPUs. Of course I know that "cache speeds up memory access" and "pipelines allow multiple CPU instructions to be handled concurrently, each at a different stage", etc. etc. But most of those explanations are handwaving and I do not get the hard core principle in light of the following:

What good is cache if I need memory access cycles to fill it? If fetching data from memory takes one clock cycle (1) assert address, activate output-enable, (2) clock the data word appearing on the data bus into register. Took one cycle. How can a cache speed that up? It seems to me that the untold story here is that actual large size (D)RAM is slow, requiring perhaps several GHz clock cycles to select "rows" and extract columns? And so it's not really that cache are making SRAM access faster, but instead, a cache is a "band aid" to slow(er) RAM, not unlike a RAM cache of an even slower disk, with SSD blurring the lines between "hard drive" and "memory".

Is this about right? How many CPU cycles are going to waste for a RAM access today?

But then it's still not clear why the cache helps if I can't fill it fast enough. I figure there are two issues:

(1) If row access is slow on DRAM, then once a row has been accessed it is better to read the entire row into the cache SRAM rather than asking for parts of it now, and another part later when the row has to be accessed again.

(2) If you have read a location once, there is a good likelihood you will have to access it once again, if your registers run out for reading, or quite possibly for writing. This later then raises the question of write-through cache (how a cache can speed up write) and the problem of committing writes to DRAM ... especially in multi-processor systems! In that case, would it not be better if the DRAM module held its own SRAM cache? Maybe not, because you still need to bottleneck accesses on the common data bus, which gets worse in a multi-CPU situation. But maintaining consist transactions becomes a nightmare.

Caching seems similar to a hard drive cache which optimizes head changes to different cylinders (analogy DRAM ROW = disk cylinder). I think the original BSD FFS was making these disk geometry based optimizations, filling the buffer cache in RAM with data that is available from a track even though it had not been requested yet. Putting the blocks of the same file on the different tracks of the same cylinder to allow fast reading, etc. But then for disks this has become largely irrelevant as all "sectors" and "cylinders" have now become virtual. Is what happened just a cache on the disk, including "virtual disk" remapping of (bad) sectors. So RAM cache has moved into the storage device, buffer cache is of course still in main RAM, but the disk access optimizing cache is on the storage device.

In that sense, wouldn't slow DRAM memory modules also benefit from a SRAM cache right on them?

I don't intend this question to become long and meandering leading down all sorts of rabbit holes. I am sure about every sentence I wrote above one can write books, first nit-picking my terminology and then putting it together complete and correct. But I am only looking for the key principle here. The one critical ah-ha inventive step. The one missing piece that is always lost when everybody nods their head about how "L2 cache" magically makes stuff faster. To me, that missing piece seems to be that DRAM is actually quite slow, along with the problem of bottleneck on the single memory address and data bus.

Now this moves us right to pipelining. The two are related because the thing I don't get is how pipelines can improve anything if the execution is memory access bound? If I need to read 2 or 3 instruction words to even define a memory access, I just had 3 memory accesses just to set up the 4th. The talk about pipelines always makes a big deal about that "instruction decoding" as if "decoding" of an instruction is something that takes more than one half of a clock cycle. The rate limiting step is memory access! So how can having multiple instructions being executed help here? How can I even get those multiple instruction into the CPU if not by memory access, exacerbating that memory access bottleneck problem even more?

When I ask this question elsewhere and before, someone will invoke the magic of the "cache". But how will the cache fill up with all these look-ahead instructions if for each word we need at least one clock cycle? If the CPU could handle 5 instructions in parallel, say, each taking 3 CPU cycles to execute, but for each instruction I need 3 cycles to load them, my CPU will always have to wait for the memory anyway.

It seems to me there is that single point, single inventive step that I am missing. And it might all come down to

"(1) DRAM is slow to access a block of memory locations, and (2) single thread instruction handling is slow, requiring many clock cycles each. So we apply a band-aid by (1) reading the entire block of DRAM into the internal SRAM which may have multiple ports (or register pipelines) and (2) then have each pipeline on the CPU handle the instructions in parallel, each at a different stage, and hence we can parallelize the activity internally, to make sure that every resource (the next one to talk about would be the ALU) is never idle, and all waiting is minimized. Resources don't wait for jobs, and jobs don't wait for resources."

Maybe it could even be boiled down to a shorter principle: "there is no faster execution than one instruction per clock cycle, ever, no matter how many parallel pipelines and cores you put on the memory bus. So all that caching and pipelining does is to reduce idle time to get as close to one instruction per clock cycle as possible."

Is that about right? Can someone say it even simpler? More concisely, and precisely perhaps, without having to write the entire text book on computer engineering?

  • 6
    This question is kinda problematic, bein off topic on multiple levels. It explicit asks about general CS knowledge, not specific to any historic implementation. It as well asks specific about 'modern RAM', about 'modern CPU' and multi GHz. At the same time it's beeing way to broad by mixes several topics (Memory speed, Caching, multi threading, pipelining, etc). Either way, not RC.SE, but CS or SO related. (P.S.: Try to think of a 4 MHz 6502 in a 1 MHz Apple II. What would the CPU do, and what would a small 4 MHZ RAM - like used in a Zip-Chip - do?)
    – Raffzahn
    Commented Aug 16, 2022 at 7:41
  • Be aware that the latency for DDR RAM is typically in the tens of cycles, so cache that can be accessed in a relatively few cycles will make a big difference.
    – Frog
    Commented Aug 16, 2022 at 9:38
  • 1
    I read through everything you wrote here, and it sounds to me like your understanding is pretty much correct. There are some details that are wrong (I don't know if DRAM takes a significant time to change from one row to another, and CPUs can execute multiple instructions per clock cycle if they can find a way to do them all in parallel). I don't think there are any important concepts that you're missing. Commented Aug 16, 2022 at 12:58
  • @TannerSwett thank you for reading and checking so I guess I answered my question. And I was going to fix the 1 per clock cycle with a big qualification and then decided to leave it so blunt. Even what constitutes an "instruction" depends on the ISA. Commented Aug 16, 2022 at 13:14
  • Caching is a strategy for bringing some small portion of the program (code or data) closer to the CPU using a faster memory of substantially limited size. Caching doesn't guarantee to speed up a program, but it has a great probability of doing so based on universal access patterns of programs (e.g. code is generally sequential, arrays are sequential). Some programs / use cases benefit more than others so CPU designers study access patterns to get the best average performance given a certain transistor budget.
    – Erik Eidt
    Commented Aug 16, 2022 at 19:01

1 Answer 1



Is this about right? How many CPU cycles are going to waste for a RAM access today?

Firstly, memory accesses are very slow. Processors typically operate a much higher speeds than memory modules and I would guess that this has always been the case apart from a brief period during the 8-bit micro processor heyday.

In the 70's main memory was typically of little iron donuts strung to together in grids. Bits were encoded by the way the donuts (or "cores") were magnetised. Even in the best core memory, you could see the cores with a magnifying glass. Memory made with semi conductors was much faster but also much more expensive, and, during the discrete transistor era, much bulkier.

Modern memory is also much slower to access than cache directly on the processor chip. Here, even physical distance matters. In one cycle of a 2GHz clock, even light can only travel about 15 cm.

There's a further issue in that main memory actually takes more than one memory cycle to access in the worst case. For example, a 386 CPU has a two level page table, so it takes two memory access to find the physical address of the data you want.

This is why caches are so important.


as if "decoding" of an instruction is something that takes more than one half of a clock cycle.

In a 6502 it might do, certainly. But then executing the instruction might take several cycles. More sophisticated processors typically use micro programs to implement their instructions. Even a processor as simple as a Z80 may take several cycles. For example, its ALU is only 4 bits wide, so a simple addition takes two uses of the ALU.

You have at least three stages: fetch, decode, execute. Assuming each stage takes one cycle, an individual instruction takes three cycles and, without pipelining, two thirds of the circuitry is unused in any one cycle. With pipelining, you can aspire to a throughput of instructions of one per cycle by making use of all the circuitry all the time.

Obviously, in real life it's very much more complicated than that, but that is the basics and it works. Your modern computer would be much slower without both.

  • Thanks, great answer. I don't know why it was downvoted. If I had two votes I'd upvote it twice. Your reference to the speed of light was especially eye-opening, because indeed in one ns light travels only 29 cm in vacuum. That is quite amazing actually, just how slow light travels! This should allow even relativistic time warp effects to become relevant on the nanosecond scale. Never thought about that. Commented Aug 16, 2022 at 13:17
  • I also add the comment that your answer is why this question does belong into this SE, because it specifically asks for history (when the inventive step was made) and your answer referencing the 6502 as an exception when memory speed and CPU speed had briefly reached an equilibrium. I heard someone say that the 6502 was quasi-pipelined, at least it did one memory access per clock cycle with only very few such cycles wasted. Commented Aug 16, 2022 at 13:19
  • 1
    @GuntherSchadow I don't sweat the downvotes, unless there are comments posted to explain why the downvote was done and that give me a chance to rectify the issues.
    – JeremyP
    Commented Sep 1, 2022 at 9:03

Not the answer you're looking for? Browse other questions tagged .