Caching Simplifies Software: A Point of Contention
A central theme in the discussion is whether caching, by its nature, simplifies software. The original premise suggests that caching can be a tool for simplification, but many users pushed back on this, arguing that it inherently introduces complexity.
- "I think now caching is probably best understood as a tool for making software simpler" - ckdot2
- "By definition a system or piece of code on where you add a condition where something else happens (cache) that behaves differently than the uncached path increases complexity." - meesles
- "The author is arguing that the latter is the default way we should think about the concept of caching, which is a valid opinion to have." - hatthew
- "If you have a system with 'slow storage', caching is a way to optimize that to 'storage that is sometimes fast'. If you have a system with 'slow storage' and 'fast storage', caching is a way to abstract that away to just 'storage'." - hatthew
- "ckdot2: That abstraction is another layer though. And additional layers are additional complexity. So, if you add another layer, the software is less simple than before."
- "The author is arguing that the latter is the default way we should think about the concept of caching, which is a valid opinion to have." - hatthew
The "Two Hard Things" and Their Nuance
The classic Computer Science aphorism about "cache invalidation and naming things" being the only two hard problems is a recurring topic. While some find it a bit of a joke, many agree with its underlying truth, particularly regarding the practical difficulties of these tasks in large-scale software. The discussion also touched on whether this aphorism is still relevant or if other challenges have emerged. The addition of "off-by-one errors" to the quote is noted as a humorous embellishment.
- "There's that famous quote 'There are only two hard things in Computer Science: cache invalidation and naming things.', and, sure, it's a bit ironical, but there's some truth in there." - ckdot2
- "You forgot off-by-1 errors." - bell-cot
- "It's a little bit tongue in cheek; no one is seriously suggesting it's harder than P=NP or the problem of consciousness. But there's something a bit 'death and taxes' to the inevitability that any large enough project is going to have some corner cases involving these old chestnuts." - gryfft
- "For 'only two hard problems,' read 'two candidates for among the hardest problems (but we feel strongly that these are indeed good candidates),' or something along those lines, more or less." - quuxplusone
- "The joke form of this quote goes along the lines of:
There are only two hard things in Computer Science: cache invalidation, naming things, and off-by-one errors.
:-D" - AdieuToLogic - "In my experience, the larger the software you write, the truer these become. At some point all obvious names will have collisions, and getting caching right is crucial to do but difficult to achieve because it transcends the entire stack." - Valodim
The Difficulty of Cache Invalidation
A significant portion of the conversation delves into the inherent challenges of cache invalidation. While some argue that it's not as difficult as the meme suggests, others emphasize that it's a subtle but critical problem that is easy to get wrong and hard to test, especially in distributed systems.
- "Cache invalidation isn't hard in theory. It's just one of those things that is very easy to get subtly wrong and difficult to test." - IshKebab
- "Think about all those times your program isn't building and
make clean
fixes it." - throwaway150 (This example was later contested as not being directly related to cache invalidation.) - "Namings things is of course a bit tongue in cheek. But cache invalidation is hard. For example, allegedly MESI is one of the hardest things to validate in processor design." - gpderetta
- "Getting cache keys or caching events wrong is easy and a nightmare." - moritzwarhier
- "The hard part is in the last word, invalidation. Even that is manageable for a single process. But as soon as you have multiple (threads / processes / nodes / data centers) updating the data, it does get quite complex, pretty fast." - heikkilevanto
- "I don't think that's a good example. I've worked with more than 20 different build tools by now, and I cannot recall a single instance where the problem actually came down to cache invalidation. Every time I dug into it, the real cause was something else: a mistake in the build script, an incorrectly declared dependency, or something similar." - throwaway150
- "Here's one: everybody invalidating and refreshing their cache at the same time can cause a thundering herd problem." - porridgeraisin
- "You don’t support read-your-own-write and your cache data might be stale for arbitrarily long. These relaxed consistency constraints make caching a lot easier. If that’s acceptable to your use cases then you’re in a great place! If not… well, at scale you often need to find a way for it to be acceptable anyway" - williamdclt
Caching as an Abstraction vs. an Optimization
A nuanced debate emerges around whether caching is primarily an abstraction or an optimization. Some argue that caching, by hiding retrieval costs, acts as an abstraction layer, while others contend that it's fundamentally an optimization technique, and the abstraction comes from how it's implemented or used. The discussion also touches on "leaky abstractions" when caching fails.
- "My understanding of the article is that it's about how caching algorithms can abstract the concern of minimising retrieval cost." - movpasd
- "So in some ways you're coming at it from opposite directions. You're talking about a prior of 'disk by default' and saying that a good abstraction lets you insert cache layers above that, whereas for the author the base case is 'manually managing the layers of storage'." - movpasd
- "This is correct, I appreciate you for putting it so coherently :). I think I didn’t make it clear enough in the piece that I’m coming from a stance of fast access being table stakes, and the question being about how that’s accomplished." - foldU
- "When code has abstract interfaces for data access, introducing caching can be simpler (but not simple) by localizing it in the abstraction implementation which has or doesn't have caching. But it is not an abstraction (you can perfectly well do caching without any abstractions, and it's frequently done exactly that way)." - necovek
- "If you hide caching away as an implementation detail behind an abstraction, it comes back and bites you as a leaky abstraction later." - jameshart
- "The problem is that eventually delivering that message will result in clients assuming that it will always be the same, when it’s not." - hmottestad (referring to SQS delivery guarantees and caching)
- "This reminds me of the use of materialized views as both a cache strategy and as an abstraction helper." - taeric
- "And they too can slow things down. Like all caches can. Like Redis can. Cache is a leaky abstraction." - bravesoul2
- "The author means that caching can be used as an implementation detail in an (abstracted) storage access system, as opposed to a baseline of having multiple storage systems (fast, medium, slow) and managing them directly." - klabb3
- "Most optimizations require you to think about how your code is structured, so as a side-effect you make the code more understandable. In this article, it's cache levels forcing you to separate different types of data because they're accessed at different frequencies. Another example is Rust's borrow checker, whose main purpose is arguably to facilitate both safe and efficient memory management, but which can also be used to enforce invariants that aren't clearly memory-related (e.g. builder pattern, temp files that auto-delete after they're dropped). These aren't abstractions though. An abstraction is the opposite, hiding structure when it's noisy and making it easier to change." - armchairhacker
- "IPC is like caching in that it's an optimization that also feels like it adds a ton of complexity. But it's also an abstraction." - pclmulqdq
- "Use of a better abstraction is an optimization, though." - pclmulqdq
The Domain and Context of "Hard" Problems
Several participants brought up the idea that the difficulty of naming things and cache invalidation is highly dependent on the context and scale of the software project. What might be simple in a small, single-process application can become exceedingly complex in large, distributed systems with multiple teams and dependencies.
- "They also tend not to be very hard. Except when they're part of some base assumptions in the domain or dozen of layers of abstractions below you. They are hard to prevent from happening." - TeMPOraL
- "All software has to name things, and count. Caching (including invalidation) is best understood as a liability." - bell-cot
- "What you describe as 'caching algorithms' are not really caching algorithms, but cached object lifetime management algorithms (LRU, LFU...)." - necovek
- "The problem is that 'naming things' that's hard. Beyond picking a common identifier format in the team, there are at least two dimensions that are much rarer: The language dimension - choice of words, that are good enough for the purpose, and not confusing. The epistemological dimension - whether the word you chose correctly names the concept you meant, and whether the concept you meant is actually the right one to describe the thing you're trying to describe." - TeMPOraL
- "Like all caches can. Like Redis can. Cache is a leaky abstraction. (Although a materialised view is more like an index than a cache. The view won't expire requiring you to rebuild.)" - bravesoul2
- "The difficulty of naming things is deciding how much context. A name might begin inside an organization but need to endure a wider area. If you make all names so long and context-free that they can work in any context, they become unwieldy." - gblargg
CPU Caching as an Example
The discussion frequently referenced CPU caching as a positive example of how caching can simplify things, particularly for developers who don't need to manage it directly. It's seen as an effective abstraction that boosts performance transparently.
- "CPUs are still a great example for how caching simplifies things." - atq2119
- "There's a long history in computer architecture of cores and accelerators that don't have a cache but instead rely on explicitly programmed local scratchpads. They are universally more difficult to program than general purpose CPUs because of that." - atq2119
- "Even more obvious if you think about the case of hardware-managed caches! The ISA typically exposes some simple cache control instructions (and I guess non-temporal loads/stores?), but apart from that, the actual choice of storage location is abstracted away from you (and your compiler)." - eigenform
- "Even CPU cache is no longer an abstraction if you deal with very high performance code." - LudwigNagasena