7

Recently there is a huge interest in "processing-in-memory" solutions, where they put some CPU, AI processor or even a DBMS inside a RAM chip.

Indeed, current computers appear to be very inefficient: one of the most common software operations "x++" requires multiple operations, something like:

  • CPU sends select row command to RAM

  • CPU send read command to RAM

  • RAM sends multiple bytes (a cache line) to CPU

  • RAM performs precharge (stores selected row back in capacitors)

  • CPU performs "add 1" operation in the cache

  • CPU send select row command again

  • CPU send write command along with bytes from the the cache line

  • RAM stores selected row back to capacitors

This begs to be optimized:

  • CPU sends select row command to RAM

  • CPU sends some hypothetical "ADD" command to RAM along with value "1"

  • RAM adds received data "1" to the row

  • RAM stores selected row back to capacitors

I guess it should be relatively easy to implement in RAM simple commands such as ADD, AND, OR (much simpler than the currently proposed DBMS or AI).

My question is whether any similar smart RAM architecture was ever implemented in computers in the past?

P.S: Several people pointed that cached data is written back at a much later time and multiple operations are made in CPU on the same cached "x". Therefore, I should clarify that the above optimization makes more sense when "x" is an element of large data set, like an image or AI data. And in this case placing "x" in cache does more harm (like evicting other data) than good. And RAM should support more specialized image operations rather than simple ADD in the example above.

6
  • 2
    The step between "CPU performs 'add 1' operation in the cache" and "CPU send select row command again" can be very, very, very long and are not linked at all immediately. The cache decouples the write from the read and from the operation. With the smartRAM, you will lose that, making maybe that one operation in and on itself faster, but will slow down the whole computer. Jun 27 at 5:37
  • 3
    There's plenty of retro architectures that have main RAM or I/O "RAM" (I/O address space) do "interesting" stuff (like autoincrement, or extended arithmetic). There is also a "theoretical" CPU that uses only move commands, all operations are carried out on specific addresses. But you seem to be ignoring that given today's speed differential between "CPU internal" and "RAM access" operations, and the cache hierarchy needed, this approach doesn't make much sense today, so I am not sure if your question belongs on retrocomputing...
    – dirkt
    Jun 27 at 6:29
  • DRAM chips from the early times had huge pages (like page of 128 bits in 4116) so operating data in-memory in page-sized chunks could have given significant speedup for some tasks. And that could have been made already in 70ies or 80ies. So I believe there is retrocomputing flavour in this question.
    – lvd
    Jun 27 at 8:02
  • Also just out of curiosity: there is an undocumented way to copy the whole pages in SDR SDRAM -- within a single bank, just open the rows in a sequence without ever closing: the contents of the row first opened will be copied to all following rows. A kind of in-memory processing.
    – lvd
    Jun 27 at 8:03
  • 1
    @PatrickSchlüter I love you. Half my write up in a single comment :))
    – Raffzahn
    Jun 30 at 15:16

4 Answers 4

9

TL;DR:

There was only one moment in time when a small subset of 'in memory' computation made sense and could be delivered without any penalty:

       When Core was King


Various computers of the 1960s, added 'in memory computation' based on the fact that core memory has a destructive read. But even then usage was limited to certain simple cases.

For example the PDP-8 implemented incrementing indirect memory references within the memory logic(*1,*2). This enabled fast array handling like the classic C *p1++ = *p2++. The Nova extended this to generic increment and decrement instructions with or without being a pointer referenced, the x++ case mentioned.

IBM used the write after read feature of core to implement atomic locking. The Test and Set (TS) instruction was implemented at the memory interface as a read variant. When issues the memory system delivered the value of a byte, but always wrote it back with all bits set (*3).


How it Worked

While magnetic memory is by default non volatile, it has the basic issue of destructive read. That is, whenever content of a cell is read out, its destroyed at the same time, thus it needs to be written back again. These are two dedicated cycles, but done by the memory logic itself, thus it seems static to any CPU/Memory interface. Access is, viewed from the CPU atomic.

By inserting some simple logic into the memory above instructions could be performed without needing any additional memory bandwidth. Just a few lines to indicate the additional function.

Limitations

As so often it's all about bandwidth. While performing certain operations in memory, during rewrite, comes (almost) for free, as the (re)write-cycle is to be performed anyway, it requires a wider memory interface to encode the additional function (instruction) beyond read. While one or two additional signals may not be an issue, it grows fast from there.

More so, when the result is needed within the CPU for follow up computing only operations with implied operands (like increment/decrement) can be done. Two operand operations will need a follow up read cycle to gather the result, negating the intended savings. That is, unless a second data path is added, which is an incredible cost for a quite small gain.

It's Gotten Worse Since Then!

Introduction of semiconductor memory has taken away the timing advantage of core for such tasks, so any data modifying operation would have to be a read/modify/write spanning two memory cycles. Except today, no system of acceptable performance does still access memory synchronous to operation. There are many levels of caching, access reordering and even instruction reordering, that have cut any direct hard relation between memory access and execution.

It's Full of Wires

So unless we talk about RAM modules being their own computers (rather pointless, see below), the idea is about including (more or less) simple ALUs int RAM chips, isn't it? Just, RAM chips do not store consecutive data. OnePC-DDR-Stick usually contains 8 independent chips operated in parallel. Each chip will only see it's 8 bits (usually size).

For simplicity let's just assume that there are only two chips and the internal memory row is only 64 bits (8 bytes).

When the CPU writes a sweries of 16 bit words to such a two chip module, one would receive the lower half of each word, the other the higher.

Now, if the task would be to add 'a row' of 16 bit values, none of the chips could do it on it's own, as the higher half does need to see if the lower half had a carry or not. So we need to have at least a carry out and another carry in pin. Except, with only one pin each we can only do one operation at a time. To do it really in parallel, we need to have this for every 8 bit within the row, wouldn't we? That's a whopping 16 lines, 8 out and 8 in, per chip.

Hold your breath, it's getting worse. Sure, the calculation might be possible to be done in a nano second (*4), but we also need to wait at least 4-5 ns for the carry to propagate between chips. Now multiply that by 8 as the number of parallel chips per RAM module. Or add 8 outputs and 8 inputs per chip for some carry look ahead, which is even worse for chip design.

So far we even haven't touched the complexity added thru interleaved memory modules and modules connected to different channels - and all of that in addition covered and moved around by virtual memory management.

Poor guy who has to write any kind of memory allocation for that :))

Or Rather Better?

In fact, when looking at the first example, it only looks cumbersome compared to the second when seen in a strict sequential order repeated once per data item. But modern CPUs don't work like that. They already work like in the second example:

  1. Read is done once for a whole cache line.
  2. The CPU iterates over that line (e.g. adding 1)
  3. Write is done once per cache line of modified data

#1 and #3 already work at maximum RAM speed (*5), while #2 is even faster than RAM can deliver data. Actual DDR delivers two data items per clock of 200 MHz, effective 400 MHz. At the same time a modern CPU rund somewhere between 3 and 4 GHz, up to 10 times faster with the ability to deliver 1-2 'simple' things like adds per cycle. So the time a RAM can deliver two 32 bit words (64 bit module size), the CPU can add twenty of them.

Thus, considering data prefetch, the whole operation already works at maximum RAM speed. CPU isn't, as the idea of smart RAM implies, the limiting factor. It's RAM access time. The very basic time a RAM cell needs to be read. And that one hasn't really improved since more than 30 years. What has improved is transfer speed, realized thru ever wider data rows and highly sophisticated access protocols (*6).

Describing modern systems in terms of such simple steps is extreme simplified and not a great base to be used to think about improvements.

But One Could do it Between tRCD and tRP

And yes, I know, there are possible ways to wrangle with that, like doing the computation between readout and writeback, much as core did. Except it won't add much performance, but slow down over all RAM access.

All the modern multi level interleaving of parallel data streams makes wait times that seem so obvious neglectable - if not turning them into an advantage.

Modern RAM is like a juggler keeping 20 balls in motion. While each of his throws is much much slower than a baseball is thrown, he'll outperform by simply doing more at the same time.

But ...

... where they put some CPU, AI processor or even a DBMS inside a RAM chip

Well, yes, but that isn't having RAM doing stuff, but more like a (slave) compute node with its own CPU and RAM, connected over a memory bus allowing DMA into that RAM. Isn't it?

It comes with all the burden of a distributed architecture. Not at least the synchronisation issue. There is a good reason tight coupled node based architectures, like Transputers, have mostly vanished. They are simply too complex. Today it's either

  • symmetric multi core systems,

where several equal CPUs/cores work close on the same data set with high data exchange or

  • rater independent full figured systems of equal structure

communicating over a (comparably) slow network, working on partiality independent data sets with low data exchange.

The only exception are dedicated special purpose processors like DSP/GPU. Their use is essentially a special case of the second part. They are dedicated systems, coupled over a slow connection to perform a very specific but high level function. So exactly the opposite of "adding some simple logic to RAM".

To add to add insult to injury, integrating a CPU into RAM wouldn't speed up much, as that CPU will still be bound by the access time of that RAM. it doesn't get faste due being closer. Each access within will still be as slow.

Bottom line:

"Smart RAM" sounds great at first, but doesn't work out. Modern systems have implemented optimizations that go way past what can be archieved that way. As usual, very special cases might exist as an exception.


*1 - It also handled indirect reference within memory on its "own" (*2)

*2 - The PDP-8 and related CPUs are a bit of a special case as core handling was part of the CPU. To save money (PDP were intended to be cheap) DEC didn't use a dedicated/off the shelf memory controller, but build their own, integrating it as part of the CPU.

*3 - Being atomic guaranteed that it always delivered the previous state and set the new one, without any interference due other CPUs or I/O. While the memory returned the value read, it only influenced the condition code, telling if it was prior all ones or not. To set a lock one issued a TS and looked if the prior value was not all ones. if so, one owned that lock, otherwise one hat to wait for it to be cleared, using regular instructions.

*4 - Yes, that long, as even the most simple adder does take it's time.

*4 - In fact, it operates already faster, as modern DDR type RAM can deliver data faster than a row can be read. And all of that in parallel with various levels of open pages, parallel modules and different levels of cache.

*5 - Time for a complete random access cycle (tRC)of a 1995 50ns EDO RAM was about 85ns. the same for a modern DDR4-3200 (CL22 22-22-52) is ~26ns, so barely three times faster. What differs is the transfer speed when an access has happened. DDR4-3200 transfers about 100 times faster than 1995 EDO. A cache line of 64 bytes is transferred in just 20 ns by a standard RAM module. That's already faster than it can be read - leaving room for parallel access over the same bus.

15
  • Very interesting, thank you. It will be nice if somebody with knowledge will write this info to Wiki. Currently "en.wikipedia.org/wiki/In-memory_processing" only says that this is just an "emerging technology for processing of data stored in an in-memory database", but it is clearly more than that. Also, I am not sure this is correct: "integrating a CPU into RAM wouldn't speed up much, as that CPU will still be bound by the access time of that RAM.". But accessing RAM on-die or working on selected row when it is in registers should be faster than transfer between chips, right?
    – jhnlmn
    Jun 28 at 20:07
  • @jhnlmn Faster? No, not really. As mentioned, the RAM cells haven't gotten much faster over the last 30 years. Time for a complete random access cycle (tRC)of a 1995 50ns EDO RAM was about 85ns. the same for a modern DDR4-3200 (CL22 22-22-52) is ~26ns, so barely three times faster. What differs is the transfer speed when an access has happened. DDR4-3200 transfers about 100 times faster than 1995 EDO. A cache line of 64 bytes is transferred in just 5 ns. Any attempt to save there is saving at the wrong end. Not to mention the realRAm is build from multiple chips used in parallel and ...
    – Raffzahn
    Jun 28 at 22:58
  • @jhnlmn ... and interleaved. So the sequence the CPU (and any user program) sees is quite different from what each chip will see. Thus any operation wider than a chips data lines (often a byte) would need to have the RAMs not only knowing it's place within the virtual memory, but also within all other RAM chips, using that information to output/input carry from other chips - that at least impractical, if not impossible.
    – Raffzahn
    Jun 28 at 23:02
  • And no, I'm not going for Wiki. I decided many years ago to restrict myself to only makign obvious and undeniable changes - there are way to many fanbois out there (not just Commodore) each defending his pet topic and trying to force his wishful thinking into Wiki. Not taking that fight.
    – Raffzahn
    Jun 28 at 23:04
  • DRAM memories supported page mode operations already in the beginning of 80ies (if not earlier) and those ops were not separate cycles as a single row was kept open. It allowed reads and writes to the single row faster than full memory cycle per operation. Nowadays synchronous DRAM memories do support that too. Hence techically the statement of 'spanning two memory cycles' is wrong.
    – lvd
    Jun 29 at 11:07
4

Mitsubishi's 3D-RAM targeted graphics memory and supported simple 8-bit ALU operations and 8-bit blend operations (32-bit SIMD) as well as 32-bit comparisons with one source provided from outside. From a databook: "The essence of the 3D-RAM architecture is: (1) an optimized array architecture that minimizes the average memory cycle time when rendering and (2) a selective on-chip logic that converts the interface with the rendering controller from a read-modified-write mode to a write-mostly mode."

In addition to the obvious bandwidth benefit of not using read bandwidth when performing simple pixel operations, this reduced the read-write bus turnaround overhead by reducing the frequency of such data movement direction changes.

Mitsubishi also produced an M32R processor with 16 Mibits of DRAM ("The M32R/D, A 32b RISC Microprocessor with 16Mb Embedded DRAM"", Hideo Tsubota and Toshifumi Kobayashi, 1996). More than 70% of the chip is dedicated to DRAM (quick estimate based on die photo in the referenced paper). While this still used caching and a traditional processor, it did avoid the off-chip bandwidth issue and provided more integration.

2

Many computers were designed around the idea of having memory cycles take a fixed amount of time independent of the address involved. Further, while it may seem logical to view each DRAM row as being a chunk of sequential addresses, the requirement that no row go too long without being accessed led to many machines using the lower portion of each address to represent a row and the upper portion to represent a column since in many cases sequential addresses would naturally get accessed during normal operation, either because the display subsystem was designed to fetch things from groups of consecutive addresses (e.g. with the Apple II), because a CPU included circuitry to sequentially cycle through the bottom address bits (e.g. the Z80's refresh mechanism), or because a DMA controller channel was used to cycle through addresses (e.g. the one on the original IBM PC).

That having been said, there were a couple of missed opportunities:

  1. Display subsystems could easily be designed to always access bytes in even/odd pairs. If the bottom address bit was wired to a DRAM column wire, an even/odd access pair would take substantially less than twice as long as two individual accesses. If the display systems always behaved in this fashion, the extra hardware would be minimal, and it would probably be possible to support two display updates and one CPU update in each cycle with little or no increase in RAM speed grade.

  2. Most RAM chips could rather easily support a single-cycle bitwise-AND or bitwise-OR operation with relatively simple external hardware, though I don't know of any common discrete-logic chips that would make this especially convenient (one could use three 74HC373 chips, sixteen Schottky diodes, and eight discrete resistors, but that would be rather a nuisance). Use one of the 74HC373 chips along with the resistors in a "bus keeper" arrangement (so that any bus pin which is sitting high will be weakly pulled high and any that is low will be pulled low, use eight diodes to make it so one 74HC373 chip can only pull the bus pins high, and the other eight to make it so the other one can only pull the pins low. During the first part of each cycle, enable the output enable on the RAM chip. Then assert write enable and enable one, both, or neither 74HC373 chip depending upon whether the goal is to write one bits, write zero bits, write a whole byte, or leave memory unaffected. Then release write enable.

On common SRAM chips, data will become available as soon as possible once the address, chip-enable, and output-enable signals are valid. Data which is to be written, however, will be sampled at the tail end of a cycle. If the only thing that needs to be done to the data bus is to selectively set or clear some bits thereof without regard for their previous state, the only external propagation delay that might matter would be for the bus keeper. Otherwise, because the write takes place at the tail end of a cycle, read-modify-write cycles of this style could be done in a single cycle.

Incidentally, I've used read-modify-write cycles as described in a practical design for an Atari 2600 cartridge. During the first half of every memory access cycle, a byte would be read from SRAM onto the 2600 data bus and captured by the bus-keeper circuits of a Xilinx (IIRC) XC36XL. Then the cart would assert /WE throughout most of the rest of the cycle. If the 6502 performed a write operation, its value would replace the value that was placed on the bus by the RAM, but if it didn't the RAM would simply write back the value it had just read. The effect of this was that even though the cartridge port doesn't include the CPU's read/write pin, the cartridge could handle read and write instructions without requiring programmers to use the special programming techniques needed to access RAM on every other Atari 2600 cartridge design I'm aware of.

2

One direct example was the one-off Cray-3/SSS. It was not unique among designs, but the only one I know of that was implemented in this particular fashion. The idea here was that the Cray-3 would be the front-end to a massively parallel machine implemented in the RAM. It was basically an enormous machine made out of bit-slice processors, 1-bit per processor.

2
  • Was it ever fully built? Jul 1 at 17:25
  • No, just the one module. Still big for the era! Jul 2 at 22:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.