A service's memory keeps growing

Symptoms

nodejs_heap_used_bytes climbs over minutes or hours and garbage collection does not bring it back down. Near the limit (nodejs_heap_limit_bytes) the process slows down and then crashes with "out of memory"; the platform restarts it.

Causes, most common first

1. A cache or map with no eviction that grows with traffic.
2. Listeners or timers that are added per request and never removed.
3. A change in traffic shape: a new client sending much larger payloads into code that keeps them.

Triage

• Check whether the growth started with a deploy. If no release changed recently, look at traffic: which routes and which clients grew at the same time.
• Look for gauges that count entries in in-memory structures; one that grows with the heap is the leak.

Mitigation

Restart the service to release memory now. This is a stopgap: it buys hours or minutes depending on the rate. Then fix the code (eviction, a size limit) or throttle the client sending the traffic.