Leaks

I’ve posted about leaks several times in the past _ _. Let’s take a look at another, shall we?

This set of inGraphs tells a pretty good story (thanks Vishnu C N!) - the longer the service is up, the more memory it consumes…either until someone comes in on Monday morning and deploys/restarts it, it tops out, starts GCing and 5XXing. What was the cause? In vcn’s own words “a cache to store or info about non critical endpoints for load shedding” (that was growing unbounded). What’s that look like? Welp, vcn helped us out there, too:

One big fuckall hash with 29M items holding onto ~2.6GB of memory (and growing). Look familiar?