In the microcontroller world, there's already asymmetric RAM like this, although it's all based on the same (SRAM) technology, and the distinction is around the topology. You have TCM directly coupled to the core, then you generally have a few SRAM blocks attached to an AXI cross-bar (so that if software running on different µc cores don't simultaneously access the same block, you have non-interference on timing; but simultaneous access is allowed at the cost of known timing), and then a few more SRAM blocks attached a couple of AXI bridges away (from the point of view of a core; for example, closer to a DMA engine, or a low power core, or another peripheral that masters the bus). You can choose to ignore this, but for maximum performance and (more importantly) maximum timing determinism, understanding what is in which block is key. And that's without getting into EMIFs and off-chip SRAM and DRAM, or XIP out of various NVM technologies...
What they seem to want is fast-read, slow-write memory. "Primary
applications include model weights in ML inference, code
pages, hot instruction paths, and relatively static data pages".
Is there device physics for cheaper, smaller fast-read slow write memory cells
for that?
For "hot instruction paths", caching is already the answer. Not sure about locality of reference for model weights. Do LLMs blow the cache?
Cheaper / smaller? I would say not likely. There is already an enormous amount of market pressure to make SRAM and DRAM smaller.
Device physics-wise, you could probably make SRAM faster by dropping the transistor threshold voltage. It would also make it harder / slower to write. The bigger downside is that it would have higher leakage power, but if it's a small portion of all the SRAM, it might be worth the tradeoff.
For DRAM, there isn't as much "device" involved because the storage element isn't transistor-based. You could probably make some design tradeoff in the sense amplifier to reduce read times by trading off write times, but I doubt it would make a significant change.
NOR is usually limited to <30MHz, but if you always want to fetch an entire cacheline, and design the read port, you can fetch the entire cacheline at once so that's pretty neat.
I don't know if anyone has applied this to neutral networks.
> "The key insight motivating LtRAM is that long data lifetimes and read heavy access patterns allow optimizations that are unsuitable for general purpose memories. Primary applications include model weights in ML inference, code pages, hot instruction paths, and relatively static data pages—workloads that can tolerate higher write costs in exchange for lower read energy and improved cost per bit. This specialization addresses fundamental mismatches in current systems where read intensive data competes for the same resources as frequently modified data."
Essentially I guess they're calling for more specific hardware for LLM tasks, much like was done with all the networking equipment for dedicated packet processing with specialized SRAM/DRAM/TCAM tiers to keep latency to a minimum.
While there's an obvious need for this for traffic flow across the internet, whether or not LLMs are really going to scale like that, or there's a massive AI/LLM bubble about to pop, would be the practical issue, and who knows? The tea leaves are unclear.
I'm not seeing the case for adding this to general-purpose CPUs/software. Only a small portion of software is going to be able to be properly annotated to take advantage of this, so it'd be a pointless cost for the rest of users. Normally short-term access can easily become long-term in the tail the process gets preempted by something higher priority or spend a lot of time on an I/O operation. It's also not clear why if you had an efficient solution for the short-term case you wouldn't just add a refresh cycle and use it in place of normal SRAM as generic cache? These make a lot more sense in a dedicated hardware context -- like neural nets -- which I think is the authors' main target here.
I imagine it would be straightforward to support this for codebases which already define multiple allocators (game engines, programs written in zig, bunch of other examples I’m less familiar with). If you’re already in memory management land you already you already have multiple implementations of malloc and free. Adding more of them is trivial.
If you’re not in manual memory management land, then you probably don’t care about this optimization just like you barely think of stack vs heap. Maybe the compiler could guess something for you, but I wouldn’t be worrying about it in that problem space.
I'm totally in manual memory management land. But it's very difficult for me to think of a case where time-limited retention is something I'd feel safe with, and with limited-endurance memory I'd worry about wearing it out with iterating or debugging.
A bunch of applications should be able to annotate data as read-heavy. Without any change of application code, operating systems can assume that pages mapped read-only should be mapped. This imidietly gives you the majority of executable code and all data files that are mmapped as read only.
I'm not sure how good applications are at properly annotating it, but for most applications assets are also effectively read only.
You don't even need most of the ram usage to be able to take advantage of this. If you can reasonably predict what portion of ram usage will be heuristically read-heavy, then you can allocate your ram budget accordingly, and probably eak out a measurable performance improvement. In a world with Moore's law, this type of heterogeneous architecture has proven to not really be worth it. However that calculus chagnes once we lose the ability to throw more transistors at the problem.
> Only a small portion of software is going to be able to be properly annotated to take advantage of this
The same could be said for, say, SIMD/vectorization, which 99% of ordinary application code has no use for, but it quietly provides big performance benefits whenever you resample an image, or use a media codec, or display 3D graphics, or run a small AI model on the CPU, etc. There are lots of performance microfeatures like this that may or may not be worth it to include in a system, but just because they are only useful in certain very specific cases does not mean they should be dismissed out of hand. Sometimes the juice is worth the squeeze (and sometimes not, but you can't know for sure unless you put it out into the world and see if people use it).
Presumably, the cost would be roughly the cost of traditional memory. In most consumer devices, memory is bottlenecked by monetary cost, not space or thermal constraints.
However, dedicate read-optimized memory would be instead of a comparable amount of general purpose memory, as data stored in one need not be stored in the other. The only increase in memory used would be what is necessary to account for fragmentation overhead when your actual usage ratio differs from what the architect assumed. Even then, the OS could use the more plentiful form of memory as swap-space for more in demand form (or, just have low priority memory regions used the less optimal form). This will open up a new and exciting class of resource management problems for kernel developers to eek out a few extra percentage points of performance.
The existence of SIMD has knock-on effects on the design of the execution unit and the FPUs, though, since it's usually the only way to fully utilize them for float/arithmetic workloads. And newer SIMD features like AVX/AVX2 have a pretty big effect on the whole CPU design; it was widely reported that Intel and AMD went to a lot of trouble to make it viable, even though most software probably isn't even compiled with AVX support enabled.
Also SIMD is just one example. Modern DMA controllers are probably another good example but I know less about them (although I did try some weird things with the one in the Raspberry Pi). Or niche OS features like shared memory--pipes are usually all you need, and don't break the multitasking paradigm, but in the few cases where shared memory is needed it speeds things up tremendously.
Generational collectors need to record when older generation objects are modified (by card marking, for example), so they can distinguish mutated from unmutated.
Afaik card marking and other similar schemes do not care about (or track) mutated objects. They track cross-generational references, which have to have been caused by mutation but only a very selective subset thereof. And card marking does not even track at the object level, it tracks pages which have at some point contained a pointer to the newgen.
In the microcontroller world, there's already asymmetric RAM like this, although it's all based on the same (SRAM) technology, and the distinction is around the topology. You have TCM directly coupled to the core, then you generally have a few SRAM blocks attached to an AXI cross-bar (so that if software running on different µc cores don't simultaneously access the same block, you have non-interference on timing; but simultaneous access is allowed at the cost of known timing), and then a few more SRAM blocks attached a couple of AXI bridges away (from the point of view of a core; for example, closer to a DMA engine, or a low power core, or another peripheral that masters the bus). You can choose to ignore this, but for maximum performance and (more importantly) maximum timing determinism, understanding what is in which block is key. And that's without getting into EMIFs and off-chip SRAM and DRAM, or XIP out of various NVM technologies...
I'll put the Tandem 5 minute rule paper here, it seems very relevant.
https://dsf.berkeley.edu/cs286/papers/fiveminute-tr1986.pdf
and a revisit of the rule 20 years later (It still held).
https://cs-people.bu.edu/mathan/reading-groups/papers-classi...
What they seem to want is fast-read, slow-write memory. "Primary applications include model weights in ML inference, code pages, hot instruction paths, and relatively static data pages". Is there device physics for cheaper, smaller fast-read slow write memory cells for that?
For "hot instruction paths", caching is already the answer. Not sure about locality of reference for model weights. Do LLMs blow the cache?
Cheaper / smaller? I would say not likely. There is already an enormous amount of market pressure to make SRAM and DRAM smaller.
Device physics-wise, you could probably make SRAM faster by dropping the transistor threshold voltage. It would also make it harder / slower to write. The bigger downside is that it would have higher leakage power, but if it's a small portion of all the SRAM, it might be worth the tradeoff.
For DRAM, there isn't as much "device" involved because the storage element isn't transistor-based. You could probably make some design tradeoff in the sense amplifier to reduce read times by trading off write times, but I doubt it would make a significant change.
> Do LLMs blow the cache?
Sometimes very yes?
If you've got 1GB of weights, those are coming through the caches on their way to execution unit somehow.
Many caches are smart enough to recognize these accesses as a strided, streaming, heavily prefetchable, evictable read, and optimize for that.
Many models are now quantized too to reduce the overall the overall memory bandwidth needed for execution, which also helps with caching.
Probably not what they want, but NOR flash is generally directly addressable, it's commonly used to replace mask roms.
NOR is usually limited to <30MHz, but if you always want to fetch an entire cacheline, and design the read port, you can fetch the entire cacheline at once so that's pretty neat.
I don't know if anyone has applied this to neutral networks.
> device physics for cheaper, smaller
And lower power usage. Datacenters and mobile devices will always want that.
Yes, this from the paper:
> "The key insight motivating LtRAM is that long data lifetimes and read heavy access patterns allow optimizations that are unsuitable for general purpose memories. Primary applications include model weights in ML inference, code pages, hot instruction paths, and relatively static data pages—workloads that can tolerate higher write costs in exchange for lower read energy and improved cost per bit. This specialization addresses fundamental mismatches in current systems where read intensive data competes for the same resources as frequently modified data."
Essentially I guess they're calling for more specific hardware for LLM tasks, much like was done with all the networking equipment for dedicated packet processing with specialized SRAM/DRAM/TCAM tiers to keep latency to a minimum.
While there's an obvious need for this for traffic flow across the internet, whether or not LLMs are really going to scale like that, or there's a massive AI/LLM bubble about to pop, would be the practical issue, and who knows? The tea leaves are unclear.
I'm not seeing the case for adding this to general-purpose CPUs/software. Only a small portion of software is going to be able to be properly annotated to take advantage of this, so it'd be a pointless cost for the rest of users. Normally short-term access can easily become long-term in the tail the process gets preempted by something higher priority or spend a lot of time on an I/O operation. It's also not clear why if you had an efficient solution for the short-term case you wouldn't just add a refresh cycle and use it in place of normal SRAM as generic cache? These make a lot more sense in a dedicated hardware context -- like neural nets -- which I think is the authors' main target here.
I imagine it would be straightforward to support this for codebases which already define multiple allocators (game engines, programs written in zig, bunch of other examples I’m less familiar with). If you’re already in memory management land you already you already have multiple implementations of malloc and free. Adding more of them is trivial.
If you’re not in manual memory management land, then you probably don’t care about this optimization just like you barely think of stack vs heap. Maybe the compiler could guess something for you, but I wouldn’t be worrying about it in that problem space.
I'm totally in manual memory management land. But it's very difficult for me to think of a case where time-limited retention is something I'd feel safe with, and with limited-endurance memory I'd worry about wearing it out with iterating or debugging.
A bunch of applications should be able to annotate data as read-heavy. Without any change of application code, operating systems can assume that pages mapped read-only should be mapped. This imidietly gives you the majority of executable code and all data files that are mmapped as read only.
I'm not sure how good applications are at properly annotating it, but for most applications assets are also effectively read only.
You don't even need most of the ram usage to be able to take advantage of this. If you can reasonably predict what portion of ram usage will be heuristically read-heavy, then you can allocate your ram budget accordingly, and probably eak out a measurable performance improvement. In a world with Moore's law, this type of heterogeneous architecture has proven to not really be worth it. However that calculus chagnes once we lose the ability to throw more transistors at the problem.
> Only a small portion of software is going to be able to be properly annotated to take advantage of this
The same could be said for, say, SIMD/vectorization, which 99% of ordinary application code has no use for, but it quietly provides big performance benefits whenever you resample an image, or use a media codec, or display 3D graphics, or run a small AI model on the CPU, etc. There are lots of performance microfeatures like this that may or may not be worth it to include in a system, but just because they are only useful in certain very specific cases does not mean they should be dismissed out of hand. Sometimes the juice is worth the squeeze (and sometimes not, but you can't know for sure unless you put it out into the world and see if people use it).
That's fair, I'm implicitly assuming the area cost for this dedicated memory would be much larger than that of e.g. SIMD vector banks.
Presumably, the cost would be roughly the cost of traditional memory. In most consumer devices, memory is bottlenecked by monetary cost, not space or thermal constraints.
However, dedicate read-optimized memory would be instead of a comparable amount of general purpose memory, as data stored in one need not be stored in the other. The only increase in memory used would be what is necessary to account for fragmentation overhead when your actual usage ratio differs from what the architect assumed. Even then, the OS could use the more plentiful form of memory as swap-space for more in demand form (or, just have low priority memory regions used the less optimal form). This will open up a new and exciting class of resource management problems for kernel developers to eek out a few extra percentage points of performance.
The existence of SIMD has knock-on effects on the design of the execution unit and the FPUs, though, since it's usually the only way to fully utilize them for float/arithmetic workloads. And newer SIMD features like AVX/AVX2 have a pretty big effect on the whole CPU design; it was widely reported that Intel and AMD went to a lot of trouble to make it viable, even though most software probably isn't even compiled with AVX support enabled.
Also SIMD is just one example. Modern DMA controllers are probably another good example but I know less about them (although I did try some weird things with the one in the Raspberry Pi). Or niche OS features like shared memory--pipes are usually all you need, and don't break the multitasking paradigm, but in the few cases where shared memory is needed it speeds things up tremendously.
Sounds a bit like Intel's Optane which was seemed great in principle but I never had a use for it.
https://www.intel.com/content/www/us/en/products/details/mem...
https://en.wikipedia.org/wiki/3D_XPoint
Used a lot with giant SAP HANA systems
Are there new physics on the horizon that could pave the way for new memory technologies?
Wouldn't a generational garbage collector automatically separate objects into appropriate lifetime categories?
Garbage collectors typically do not differentiate live-and-mutated from live-but-unmutated, which is what is needed here.
Generational collectors need to record when older generation objects are modified (by card marking, for example), so they can distinguish mutated from unmutated.
Afaik card marking and other similar schemes do not care about (or track) mutated objects. They track cross-generational references, which have to have been caused by mutation but only a very selective subset thereof. And card marking does not even track at the object level, it tracks pages which have at some point contained a pointer to the newgen.