Add maxmemory-reserved parameter for evicting key earlier to avoid OOM

[hwware]

Reference: #742 and https://learn.microsoft.com/en-us/azure/azure-cache-for-redis/cache-best-practices-memory-management (Azure)

Generally, when clients set maxmemory-policy as allkeys-lru or other memory eviction policies, and maxmemory as well, If server runs as write-heavy workloads, the data stored in memory could reach the maxmemory limit very quickly, then OOM message will be reported.

If we have maxmemory-reserved parameter, we can set amount of memory away from maxmemory, then key eviction process could begin earlier to avoid used memory to catch up the maxmemory before OOM happens. Thus, we can see the benefit is we can delay OOM time.

[codecov]

Codecov Report

Attention: Patch coverage is 83.01887% with 9 lines in your changes missing coverage. Please review.

Project coverage is 71.03%. Comparing base (aced268) to head (4be3b3c).
Report is 9 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/config.c	69.23%	4 Missing ⚠️
src/module.c	0.00%	3 Missing ⚠️
src/evict.c	87.50%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable     #831      +/-   ##
============================================
+ Coverage     70.87%   71.03%   +0.16%     
============================================
  Files           121      121              
  Lines         65203    65282      +79     
============================================
+ Hits          46212    46375     +163     
+ Misses        18991    18907      -84     

Files with missing lines	Coverage Δ
src/server.c	87.72% <100.00%> (+0.09%)	⬆️
src/server.h	100.00% <ø> (ø)
src/evict.c	98.10% <87.50%> (-0.76%)	⬇️
src/module.c	9.59% <0.00%> (ø)
src/config.c	78.24% <69.23%> (-0.16%)	⬇️

... and 23 files with indirect coverage changes

[enjoy-binbin]

this seem like a interesting feature, did not review, just drop a comment that approve the concept.

[madolson]

I also like the idea. I just haven't really spent enough time thinking about it. Memory management is a big area in Valkey we need to improve.

[PingXie]

Took a quick look.

[PingXie]

+1 on introducing "back pressure" earlier. I feel that this could be used with `maxmemory_eviction_tenacity" to give an even smoother eviction experience.

[PingXie]

Thanks @hwware! LGTM overall. Can you add some tests too?

[hwware]

Thanks @hwware! LGTM overall. Can you add some tests too?

Sure, ready to work, Thanks

[PingXie]

Thinking about this more, I feel that it is more preferable to model this new setting as a "soft" maxmemory, which can trigger key eviction earlier but won't cause OOM to be returned before hitting the actual maxmemory. Otherwise, we effectively create an alias of maxmemory. More specifically, I think performEviction should only return EVICT_FAIL when the memory usage goes beyond the real maxmemory.

Additionally, since getMaxmemoryState is also used outside of the OOM prevention path such as in VM_GetUsedMemoryRatio, we should consider parameterizing maxmemory and having it passed in by the caller instead, so that we can maintain the same semantics in these externally facing scenarios.

Thoughts?

[hwware]

Like what you suggested, if we can think when "maxmemory_available" is available, it is a soft maxmemory. Then you maybe think we should not return OOM, in the case, how we can return message to client, any idea?

[PingXie]

right - how about just processing the command normally?

1. if used_memory is below soft max, no change to the existing logic
2. if used_memory is above soft max but below hard max, trigger key eviction and continue with the normal processing
3. if used_memory is above hard max, no change to the existing logic, i.e., trigger key eviction and fail the command if the used_memory is still above hard max after the eviction)

[zuiderkwast]

The INFO field mem_not_counted_for_evict should include the reserved memory, right? In our docs, we say that eviction happens when

used_memory - mem_not_counted_for_evict > maxmemory

so we should make sure this formula is still correct. Docs here: https://valkey.io/topics/lru-cache/

[zuiderkwast]

Backticks? This is not markdown. I think we can skip the backticks, like how the other configs are described.

[hwware]

I notice in valkey.conf, if a command is mentioned in the comment, Backticks is used. Although I have no idea how it origins, I just follow this tradition.

[zuiderkwast]

For most of the configs, there is a real example here, usually with the default value. I think it's better. We can mention in the text above that it's in bytes and that the default is 0 to be extra clear.

Suggested change

	# maxmemory-reserved <bytes>
	# maxmemory-reserved 0

[hwware]

Here is aligned with maxmemory

[zuiderkwast]

server.key_eviction_memory is redundant information. Maybe we don't need to store it in a global? We can calculate server.maxmemory - server.maxmemory_reserved every time when we need it, or use a small function for it.

[hwware]

I do not think so.
In function getMaxmemoryState(), we need the server.key_eviction_memory value multiply times. And performEvictions() always call getMaxmemoryState() to check if key eviction process is needed.

Here, the server.maxmemory and server.maxmemory_reserved are fixed values if nobody update them by config set command.

Thus, I think calculating server.key_eviction_memory once and work as a global variable is better.

[hwware]

@soloestoy good finding that the Azure service already has maxmemory-reserved. It's described like this:

The maxmemory-reserved setting configures the amount of memory, in MB per instance in a cluster, that is reserved for non-cache operations, such as replication during failover.

Further, here it's documented as

The maxmemory-reserved setting configures the amount of memory in MB per instance in a cluster that is reserved for noncache operations, such as replication during failover. Setting this value allows you to have a more consistent Redis server experience when your load varies. This value should be set higher for workloads that write large amounts of data. When memory is reserved for such operations, it's unavailable for storage of cached data. The minimum and maximum values on the slider are 10% and 60%, shown in megabytes. You must set the value in that range.

I assume it is a hard limit for data size, but IIUC we want a soft limit to start early eviction but not reject writes. Other than that, it seems very similar to our idea, I think. Maybe it's OK?

The "reserved" memory in this PR is also for keys when we have bursts of writes and we can't evict fast enough...? It seems OK to me.

I implemented this in my first commit 3b9fb9f, it is a percetage of the maxmemory, well match with
the Azure doc.

[hwware]

The INFO field mem_not_counted_for_evict should include the reserved memory, right? In our docs, we say that eviction happens when

used_memory - mem_not_counted_for_evict > maxmemory

so we should make sure this formula is still correct. Docs here: https://valkey.io/topics/lru-cache/

No, i do not think so. The mem_not_counted_for_evict (aka called by freeMemoryGetNotCountedMemory())
is the size of replicas output buffers and AOF buffer from the count of used memory.

However, the reserved memory is the size memory stepped away from the maxmemory. It should not include the AOF and replica output buffer part.

[zuiderkwast]

The INFO field mem_not_counted_for_evict should include the reserved memory, right? In our docs, we say that eviction happens when

used_memory - mem_not_counted_for_evict > maxmemory

so we should make sure this formula is still correct. Docs here: https://valkey.io/topics/lru-cache/

No, i do not think so. The mem_not_counted_for_evict (aka called by freeMemoryGetNotCountedMemory()) is the size of replicas output buffers and AOF buffer from the count of used memory.

However, the reserved memory is the size memory stepped away from the maxmemory. It should not include the AOF and replica output buffer part.

So.... instead we should update the docs to explain the formula like this?

used_memory - mem_not_counted_for_evict > maxmemory - maxmemory_reserved