Halving MySQL's Memory Footprint: glibc Arenas and the Thread Cache
A while ago I wrote about migrating the legacy PHP app behind a hobby project of mine, a community driven subtitles website. Three years later it runs on PHP 8 and still talks to the same MySQL 8.0.32 container. The VPS is still a small one, and MySQL is the most expensive tenant on it. Recently I noticed that the database container was sitting at around 1.17 GiB of RSS during peak hours, with a hard mem_limit of 1.5 GiB. That is not a comfortable margin, so I spent a couple of evenings figuring out where the memory was going. Today the same container peaks somewhere between 550 and 650 MiB under the same traffic, and it stays there. The fix was two lines of configuration, and neither of them has anything to do with queries.
The memory MySQL doesn’t know about
The first question was simple: is MySQL actually using that memory, or is something else holding on to it? MySQL 8 can answer this itself. The Performance Schema instruments most of the server’s allocations, and the memory summary tables expose them by event name, by thread, by account and so on. The sys schema wraps all of that in a single number through sys.memory_global_total:
| |
The answer was around 478 MiB, with the InnoDB buffer pool (still on its 128 MiB default) at the top of the list. The kernel, on the other hand, reported 1.17 GiB of resident memory for the mysqld process. Roughly 700 MiB was resident but not accounted for by the server.
Where the other 700 MiB went
Performance Schema doesn’t see everything, but a gap that large usually points below MySQL, at the allocator. mysqld in the official Docker image links against glibc’s malloc, and glibc’s allocator is built around arenas. An arena is a pool of memory with its own lock and its own free lists. The main arena grows the regular process heap; every other arena is built out of heaps that glibc maps separately, reserving 64 MiB of address space at a time on 64-bit systems and making it resident as it gets used. Each thread is attached to an arena, and when a thread finds its arena locked by someone else, glibc prefers to give it a new arena over making it wait. This goes on until the limit, which on 64-bit systems defaults to eight times the number of cores.
That last part has a Docker specific twist. glibc counts the cores the kernel reports, not the cpus limit of the container. A container restricted to one vCPU on a host with more cores still gets the arena allowance of the whole host.
Memory freed inside an arena goes back to that arena’s free lists, but it is only returned to the kernel when the free space sits at the top of the heap. If one long lived allocation remains near the top, everything below it stays resident. With a handful of arenas this is mostly noise. With dozens of them, each holding on to its own partially used heap, it adds up quickly. The mallopt(3) man page describes M_ARENA_MAX and the default sizing, and the glibc manual on memory allocation tunables documents the same knob as glibc.malloc.arena_max.
Now add the way the PHP side talks to MySQL. Every FPM request opens a fresh connection and closes it at the end, and I don’t use persistent connections on purpose. MySQL’s default connection handling is one thread per connection, so every page view creates a thread, that thread allocates its per-connection buffers, runs a few queries and then goes away. At peak this happens constantly, and with many short lived threads allocating at the same time, the lock collisions that make glibc create new arenas are exactly what you get. Each new thread may land on a different arena, allocate, free, and leave holes behind. The memory isn’t leaked in the strict sense, glibc would reuse it eventually, but it is never handed back to the kernel, and that is exactly the kind of fragmentation that shows up as a growing gap between what the application tracks and what the kernel reports as resident. The MySQL manual has a page on how the server uses memory that is worth reading next to this, especially the part about which buffers are allocated per connection.
If you want to see the arenas for yourself, they are visible from outside the process. Each arena heap shows up in the memory map as an anonymous region, usually split in two: a readable and writable part that has been used, followed by a reserved remainder with no permissions, the two adding up to 64 MiB:
| |
A long tail of those pairs, each with tens of MiB in the RSS column, is a good sign that you are looking at the same problem.
Two lines of configuration
The fix came in two parts. The first one is capping the number of arenas through the environment of the MySQL container:
| |
MALLOC_ARENA_MAX is the older spelling and still works; on recent glibc versions GLIBC_TUNABLES=glibc.malloc.arena_max=2 does the same thing. With a limit of two, there is the main arena and one more, and every thread shares them. Freed chunks are reused by the next connection instead of sitting in an arena that may not see another allocation for a while, so the heaps stop growing in parallel.
The usual argument against a low arena count is contention: fewer arenas means more threads fighting over the same allocator locks. With a single vCPU assigned to the container, that argument doesn’t really hold, since there was never much parallelism to protect. If you run MySQL on a larger box with real concurrency, measure before copying this value.
The second part attacks the churn itself. thread_cache_size controls how many threads the server keeps around for reuse after a client disconnects. When a new connection comes in, MySQL picks up a cached thread instead of creating one, so the thread’s stack and a good part of its allocations simply stay where they are. The default is autosized to 8 + max_connections / 100, which with the default max_connections of 151 comes out at 9. That is easy to exceed when a burst of page views arrives at the same time, and every connection above it means a thread created and destroyed again:
| |
You can check whether the cache is doing its job by comparing Threads_created against Connections:
| |
Without a cache the two grow in lockstep; with a cache that is large enough, Threads_created should barely move once the server has warmed up. On production, after 46 hours of uptime, the counters read 454,107 connections and 10 threads created. Practically every connection since the restart ran on a recycled thread, and with Max_used_connections at 10, a cache of 32 leaves plenty of room. The two settings work together: the arena cap limits how many heaps can fragment, and the thread cache cuts down the allocate, free, exit cycle that fragments them in the first place.
An alternative I considered and skipped was swapping the allocator for jemalloc or tcmalloc, which mysqld_safe supports through its --malloc-lib option. Both are less prone to this kind of fragmentation, but they mean a custom image or a preloaded library, and I wanted to try the cheap option first. It was enough.
How it settled
After the restart, resident memory climbed as the buffer pool warmed up and the thread cache filled, and then it flattened out. That flat line is the part I care about most. Before the change, RSS followed the traffic up during the evening and never followed it back down, which is what fragmentation looks like from the outside: every peak left a bit more memory behind. Now the container peaks between 550 and 650 MiB, falls back when traffic drops, and the next peak lands in the same place.
The gap between sys.memory_global_total and what the kernel reports went from roughly 700 MiB to about 150 MiB. After 46 hours of uptime, mysqld had a VmRSS of 620 MiB against 470 MiB tracked by Performance Schema. Some of it will always be there: thread stacks, the mysqld binary and its shared libraries, and the allocator’s own bookkeeping are not instrumented by Performance Schema. That is the baseline you should expect, not something to chase. With the buffer pool still at its default, the process has around 900 MiB of headroom under the container’s mem_limit, which is the margin I was missing.
Measuring the right number
One thing tripped me up while verifying all this. Two days after the change, docker stats showed the container at 880 MiB, and for a moment it looked like the fix had stopped working. It hadn’t. On cgroup v2, docker stats reports the cgroup’s memory usage minus inactive_file, so the active part of the page cache and some kernel memory are still in that number. The cgroup’s memory.stat file, described in the kernel’s cgroup v2 documentation, splits it up, and /proc gives you the process itself:
| |
Out of those 880 MiB, about 600 MiB was anonymous memory and around 210 MiB was active page cache. The mysqld process had a VmRSS of 620 MiB, nearly all of it RssAnon. The page cache is expected for this setup. With the default innodb_flush_method on Linux, InnoDB reads and writes its data files through the OS cache. MyISAM doesn’t cache data rows at all, only index blocks in its key cache, so it relies on the page cache for everything else. The kernel reclaims those pages under pressure before the container runs into its limit. If you are chasing allocator fragmentation, RssAnon is the number to watch; docker stats mixes in memory that MySQL doesn’t own.
Results
The MySQL container went from 1.17 GiB of RSS at peak to a stable 550 to 650 MiB, with the buffer pool unchanged and no query touched. One environment variable for the C allocator and one line in my.cnf closed most of the gap between tracked and resident memory, and stopped it from coming back.
What I take from this round is that the database was mostly fine and its surroundings weren’t. The fix was found by comparing two numbers that should have been close and weren’t. If the resident memory of mysqld is a lot higher than what sys.memory_global_total reports, and your application opens a connection per request, look at the arenas and the thread cache before you touch a single query.