There was only one moment in time when a small subset of 'in memory' computation made sense and could be delivered without any penalty:
When Core was King
Various computers of the 1960s, added 'in memory computation' based on the fact that core memory has a destructive read. But even then usage was limited to certain simple cases.
For example the PDP-8 implemented incrementing indirect memory references within the memory logic(*1,*2). This enabled fast array handling like the classic C
*p1++ = *p2++. The Nova extended this to generic increment and decrement instructions with or without being a pointer referenced, the
x++ case mentioned.
IBM used the write after read feature of core to implement atomic locking. The Test and Set (TS) instruction was implemented at the memory interface as a read variant. When issues the memory system delivered the value of a byte, but always wrote it back with all bits set (*3).
How it Worked
While magnetic memory is by default non volatile, it has the basic issue of destructive read. That is, whenever content of a cell is read out, its destroyed at the same time, thus it needs to be written back again. These are two dedicated cycles, but done by the memory logic itself, thus it seems static to any CPU/Memory interface. Access is, viewed from the CPU atomic.
By inserting some simple logic into the memory above instructions could be performed without needing any additional memory bandwidth. Just a few lines to indicate the additional function.
As so often it's all about bandwidth. While performing certain operations in memory, during rewrite, comes (almost) for free, as the (re)write-cycle is to be performed anyway, it requires a wider memory interface to encode the additional function (instruction) beyond read. While one or two additional signals may not be an issue, it grows fast from there.
More so, when the result is needed within the CPU for follow up computing only operations with implied operands (like increment/decrement) can be done. Two operand operations will need a follow up read cycle to gather the result, negating the intended savings. That is, unless a second data path is added, which is an incredible cost for a quite small gain.
It's Gotten Worse Since Then!
Introduction of semiconductor memory has taken away the timing advantage of core for such tasks, so any data modifying operation would have to be a read/modify/write spanning two memory cycles. Except today, no system of acceptable performance does still access memory synchronous to operation. There are many levels of caching, access reordering and even instruction reordering, that have cut any direct hard relation between memory access and execution.
It's Full of Wires
So unless we talk about RAM modules being their own computers (rather pointless, see below), the idea is about including (more or less) simple ALUs int RAM chips, isn't it? Just, RAM chips do not store consecutive data. OnePC-DDR-Stick usually contains 8 independent chips operated in parallel. Each chip will only see it's 8 bits (usually size).
For simplicity let's just assume that there are only two chips and the internal memory row is only 64 bits (8 bytes).
When the CPU writes a sweries of 16 bit words to such a two chip module, one would receive the lower half of each word, the other the higher.
Now, if the task would be to add 'a row' of 16 bit values, none of the chips could do it on it's own, as the higher half does need to see if the lower half had a carry or not. So we need to have at least a carry out and another carry in pin. Except, with only one pin each we can only do one operation at a time. To do it really in parallel, we need to have this for every 8 bit within the row, wouldn't we? That's a whopping 16 lines, 8 out and 8 in, per chip.
Hold your breath, it's getting worse. Sure, the calculation might be possible to be done in a nano second (*4), but we also need to wait at least 4-5 ns for the carry to propagate between chips. Now multiply that by 8 as the number of parallel chips per RAM module. Or add 8 outputs and 8 inputs per chip for some carry look ahead, which is even worse for chip design.
So far we even haven't touched the complexity added thru interleaved memory modules and modules connected to different channels - and all of that in addition covered and moved around by virtual memory management.
Poor guy who has to write any kind of memory allocation for that :))
Or Rather Better?
In fact, when looking at the first example, it only looks cumbersome compared to the second when seen in a strict sequential order repeated once per data item. But modern CPUs don't work like that. They already work like in the second example:
- Read is done once for a whole cache line.
- The CPU iterates over that line (e.g. adding 1)
- Write is done once per cache line of modified data
#1 and #3 already work at maximum RAM speed (*5), while #2 is even faster than RAM can deliver data. Actual DDR delivers two data items per clock of 200 MHz, effective 400 MHz. At the same time a modern CPU rund somewhere between 3 and 4 GHz, up to 10 times faster with the ability to deliver 1-2 'simple' things like adds per cycle. So the time a RAM can deliver two 32 bit words (64 bit module size), the CPU can add twenty of them.
Thus, considering data prefetch, the whole operation already works at maximum RAM speed. CPU isn't, as the idea of smart RAM implies, the limiting factor. It's RAM access time. The very basic time a RAM cell needs to be read. And that one hasn't really improved since more than 30 years. What has improved is transfer speed, realized thru ever wider data rows and highly sophisticated access protocols (*6).
Describing modern systems in terms of such simple steps is extreme simplified and not a great base to be used to think about improvements.
But One Could do it Between tRCD and tRP
And yes, I know, there are possible ways to wrangle with that, like doing the computation between readout and writeback, much as core did. Except it won't add much performance, but slow down over all RAM access.
All the modern multi level interleaving of parallel data streams makes wait times that seem so obvious neglectable - if not turning them into an advantage.
Modern RAM is like a juggler keeping 20 balls in motion. While each of his throws is much much slower than a baseball is thrown, he'll outperform by simply doing more at the same time.
... where they put some CPU, AI processor or even a DBMS inside a RAM chip
Well, yes, but that isn't having RAM doing stuff, but more like a (slave) compute node with its own CPU and RAM, connected over a memory bus allowing DMA into that RAM. Isn't it?
It comes with all the burden of a distributed architecture. Not at least the synchronisation issue. There is a good reason tight coupled node based architectures, like Transputers, have mostly vanished. They are simply too complex. Today it's either
- symmetric multi core systems,
where several equal CPUs/cores work close on the same data set with high data exchange or
- rater independent full figured systems of equal structure
communicating over a (comparably) slow network, working on partiality independent data sets with low data exchange.
The only exception are dedicated special purpose processors like DSP/GPU. Their use is essentially a special case of the second part. They are dedicated systems, coupled over a slow connection to perform a very specific but high level function. So exactly the opposite of "adding some simple logic to RAM".
To add to add insult to injury, integrating a CPU into RAM wouldn't speed up much, as that CPU will still be bound by the access time of that RAM. it doesn't get faste due being closer. Each access within will still be as slow.
"Smart RAM" sounds great at first, but doesn't work out. Modern systems have implemented optimizations that go way past what can be archieved that way. As usual, very special cases might exist as an exception.
*1 - It also handled indirect reference within memory on its "own" (*2)
*2 - The PDP-8 and related CPUs are a bit of a special case as core handling was part of the CPU. To save money (PDP were intended to be cheap) DEC didn't use a dedicated/off the shelf memory controller, but build their own, integrating it as part of the CPU.
*3 - Being atomic guaranteed that it always delivered the previous state and set the new one, without any interference due other CPUs or I/O. While the memory returned the value read, it only influenced the condition code, telling if it was prior all ones or not. To set a lock one issued a TS and looked if the prior value was not all ones. if so, one owned that lock, otherwise one hat to wait for it to be cleared, using regular instructions.
*4 - Yes, that long, as even the most simple adder does take it's time.
*4 - In fact, it operates already faster, as modern DDR type RAM can deliver data faster than a row can be read. And all of that in parallel with various levels of open pages, parallel modules and different levels of cache.
*5 - Time for a complete random access cycle (tRC)of a 1995 50ns EDO RAM was about 85ns. the same for a modern DDR4-3200 (CL22 22-22-52) is ~26ns, so barely three times faster. What differs is the transfer speed when an access has happened. DDR4-3200 transfers about 100 times faster than 1995 EDO. A cache line of 64 bytes is transferred in just 20 ns by a standard RAM module. That's already faster than it can be read - leaving room for parallel access over the same bus.