Over one hundred percentage memory usage on OpenWRT

OpenWrt Memory Mystery: When 796% Isn't Really 796%

Troubleshooting apparent memory overcommit on a Belkin RT3200 running OpenWrt 24.10.3

The Problem

I was checking my Belkin RT3200 router running OpenWrt 24.10.3, and both LuCI and the top command were showing something alarming: three processes were reporting a combined memory usage of nearly 800%.

Mem: 418740K used, 81752K free, 1528K shrd, 0K buff, 146184K cached
  PID  VSZ %VSZ %CPU COMMAND
 6562 1371m 281%   0% AdGuardHome
 3055 1287m 263%   0% tailscaled
12325 1231m 252%   0% cloudflared

On a router with only 500MB of RAM, how were three processes using nearly 4GB? The system was stable, running for 13 days without issues, and no swap was configured. Something didn't add up.

Understanding Virtual vs. Physical Memory

The key to this mystery lies in understanding the difference between virtual memory size (VSZ/VmSize) and resident set size (RSS/VmRSS).

Quick Definitions

• VmSize (Virtual Memory Size): The total amount of virtual address space a process has reserved. This is not physical RAM.
• VmRSS (Resident Set Size): The actual physical RAM currently being used by the process.

The Investigation

I dug deeper into /proc/[pid]/status for each process to see the real memory usage:

Process	VmSize	VmSize %	VmRSS	VmRSS %	Ratio
AdGuardHome	1,404,572 kB	281%	121,284 kB	24.2%	11.6x
tailscaled	1,318,608 kB	264%	84,164 kB	16.8%	15.7x
cloudflared	1,261,440 kB	252%	32,524 kB	6.5%	38.8x
Total	3,984,620 kB	796%	237,972 kB	47.5%	-

The revelation: While these three processes appeared to use 796% of available memory, they were actually only using 47.5% of physical RAM.

Why Go Programs Show High Virtual Memory

All three of these processes (AdGuardHome, Tailscale, and Cloudflared) are written in Go. The Go runtime has specific characteristics that lead to high virtual memory allocation:

1. Aggressive address space reservation: Go's garbage collector and memory allocator reserve large chunks of virtual address space upfront for performance reasons.
2. 64-bit address space abundance: On 64-bit systems (like this ARM64 router), virtual address space is essentially unlimited, so Go doesn't hesitate to reserve what it might need.
3. Memory-mapped data structures: Go uses memory mapping for various internal structures, which shows up as virtual memory.
4. Lazy allocation: Just because virtual memory is reserved doesn't mean physical pages are allocated. Linux only allocates physical RAM when memory is actually touched.

Why LuCI and Top Show VSZ

By default, OpenWrt's top command and LuCI show the VSZ metric because it's simpler to read from /proc and provides a quick overview. However, for Go programs, this metric is misleading.

The Better Commands

To see actual memory usage, use these commands:

# View RSS for all processes
ps aux | sort -k6,6n | tail -n 10

# Detailed memory breakdown for a specific process
cat /proc/[PID]/status | grep -E 'VmSize|VmRSS|RssAnon|RssFile'

# Quick snapshot of real memory usage
free -h

Should You Be Concerned?

In most cases, no. Here's when virtual memory size matters and when it doesn't:

Don't Worry If:

• RSS (actual memory usage) is reasonable
• The system is stable and responsive
• You have plenty of available memory according to free -h
• No OOM (Out of Memory) kills are occurring

Do Investigate If:

• RSS is consuming most of your physical RAM
• The system is swapping heavily (if swap is enabled)
• You see OOM killer messages in dmesg
• The system becomes unresponsive

System Configuration Notes

My router configuration that's relevant here:

• Total RAM: 500MB (489 MiB)
• Architecture: aarch64 (ARM Cortex-A53)
• Overcommit setting: Mode 0 (heuristic)
• Swap: None configured

The system uses vm.overcommit_memory = 0, which means the kernel uses heuristics to decide whether to allow memory allocations. This is perfect for this use case, as it allows Go programs to reserve virtual address space while preventing actual overcommit of physical memory.

Conclusions

What initially appeared to be a catastrophic memory problem turned out to be completely normal behavior for Go programs on Linux. The key lessons:

1. Virtual memory ≠ Physical memory: VSZ shows address space reservations, not actual RAM usage.
2. Look at RSS, not VSZ: For real memory consumption, always check RSS values.
3. Go is verbose with address space: This is by design and not a problem on 64-bit systems.
4. Monitor what matters: Focus on available memory, RSS usage, and system stability rather than VSZ percentages.

In my case, with 186MB of available memory and no OOM events over 13 days of uptime, the system is running perfectly fine despite the scary-looking 796% number.