After spending a fair bit of time triaging my ruined post-break inbox yesterday my thoughts turned toward igotw. I’ve got a bunch of stuff saved off, but what _should I post? …and then Michael Kehoe came along like a _gleaming Australian superhero and saved the day! He shot me a graph from an inGraphs dashboard for omnibot - the New Hotness slackbot that prod-sre has been working on.
Before showing you the inGraphs, I want to issue a disclaimer: omnibot is beta status. omnibot is not production-ready.
With that out of the way, let’s take a look at a few things. I poked around at a few of the graphs in the dashboard. One of the first thing I noticed was a regular spike in file_deleted events:
This is probably some kind of log rotation/aging. These happen just after midnight Eastern time, every day (yes, every day - the spikes are just smaller on weekends…I’m assuming there are less logs to clean up due to less activity).
Huh. Heartbeat looked okay through the break, then started dipping when people got back and started screwing around with it deploying. Making sense. In related news:
Huh. Okay. So maybe one of these deployments resulted in omnibot using more plugins. Much more likely: the deployment resulted in omnibot emitting metrics about the number of plugins it’s using (whereas before it was emitting no such metric). Cool.
After playing around with the dashboard for the New Hotness for a bit I started wondering about the Old-n-Busted. “Does notabot have a dashboard?” I wondered? As it turns out, it does One of the dashboards caught my eye. :
Just a guess - I’d have to spend a little more time thinking about what this means - but at a glance i’d say maybe something upstream with a ~24-hour reconnect policy.
At any rate, thank you Michael Kehoe for giving me confidence that I still remember how to inGraphs, and to prod-sre for the super-helpful Slackbot integrations.
…and also: welcome back from break, folks. Happy 2019.