https://thingly.net/igotw/ Recent content in inGraph of the Week on thingly.net Hugo -- 0.129.0 en-us Sun, 04 Feb 2024 00:00:00 +0000 https://thingly.net/igotw/fin/ Sun, 04 Feb 2024 00:00:00 +0000 https://thingly.net/igotw/fin/ tl;dr: This is the 423rd and final (for now) post of igotw. I’ve been struggling to find a way to start this post. I’ve come up with a handful of ideas. Each has its own flaws/fallacies and I can’t quite land on one I prefer. Some ideas were: I’ve said all I have to say … (improbable, if not impossible) I don’t want to do this any more … (not wholly accurate) I don’t have time for this … (callous - somehow “mean”? https://thingly.net/igotw/unintentional-art-gcn-40140/ Sat, 27 Jan 2024 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-gcn-40140/ Submitted in (mostly) no particular order and (mostly) without comment… …and my favorite one of all: https://thingly.net/igotw/postmortems/ Sun, 21 Jan 2024 00:00:00 +0000 https://thingly.net/igotw/postmortems/ I recently stumbled upon this collection of postmortems on Github. It’s a treasure trove exploring interesting incidents across a bunch of different domains - not just tech! For instance: Sweden. Use of different rulers by builders caused the Vasa to be more heavily built on its port side and the ship’s designer, not having built a ship with two gun decks before, overbuilt the upper decks, leading to a design that was top heavy. https://thingly.net/igotw/computer-science-major-arcana/ Sat, 13 Jan 2024 00:00:00 +0000 https://thingly.net/igotw/computer-science-major-arcana/ I had to re-parse the title of Computer Science Major Arcana after reading the first sentence (“Ohhhhh! Its a ‘Major Arcana based on Computer Science’, not ‘an Arcana, for Computer Science Majors’.”) I also had to look up what Major Arcana meant. Having done so, it bugs me a bit that NULL isn’t 0, and that 1 corresponds to The Fool…but there are some goodies in there, like: Undefined Behavior*: You have transgressed the Law, but were not lucky enough to Crash. https://thingly.net/igotw/over-the-break/ Sun, 07 Jan 2024 00:00:00 +0000 https://thingly.net/igotw/over-the-break/ Once every coupla-few years I take a Journey - walk a Path of Pain, if you will. It always starts at the same place: the gentoo.org Downloads page. I grab my ISO, thinking to myself “Surely, this time it will be better. My machine has more cores and RAM now than ever before, compilation should be fast, I’ll bet they’ve improved the overall ergonomics of the experience, …” etc. And so, I set off partitioning disks and creating filesystems and setting USE flags and things, and get down to compiling… https://thingly.net/igotw/i-m-a-teapot/ Sun, 31 Dec 2023 00:00:00 +0000 https://thingly.net/igotw/i-m-a-teapot/ This last post of 2023 is igotw #418, which just so happens to be one of my favorite HTTP response codes. (Yes, there’s an RFC and everything.) Happy New Year, folks. I’ll catch you on the flippy-flop. https://thingly.net/igotw/twas/ Sun, 24 Dec 2023 00:00:00 +0000 https://thingly.net/igotw/twas/ …the night before Christmas. Err…or the morning/afternoon before Christmas, anyway. Have a Merry, y’all. https://thingly.net/igotw/flame-graphs/ Sun, 17 Dec 2023 00:00:00 +0000 https://thingly.net/igotw/flame-graphs/ This week I wanted to pass along a write-up about a gnarly performance bug Stufflebiscuit tracked down. He’s got a couple of inGraphs in there that are good examples of banding: …and he also includes a couple of flame graphs: It made me realize I’ve never posted anything about them before. Invented by Brendan Gregg[citation needed], these charts are a way of visualizing where a given call stack is spending its time. https://thingly.net/igotw/fun-bug/ Sun, 10 Dec 2023 00:00:00 +0000 https://thingly.net/igotw/fun-bug/ A recent GCN caught my eye - not because of its impact (although it was a Medium), and not because it was a repeater (although it’s worth noting that it was, in fact, a repeat…and the previous one was Major). The reason it caught my attention is the bug itself (patched here many moons ago, if you’re interested). The bug is in some code that configures Percona audit log rotation. https://thingly.net/igotw/friends-don-t-let-friends/ Sat, 02 Dec 2023 00:00:00 +0000 https://thingly.net/igotw/friends-don-t-let-friends/ _I’ve talked about bad dataviz and chartjunk here before. The _Friends Don’t Let Friends Make Bad Graphs github repo is something a little different. It’s not that these charts have unnecessary or distracting decoration, it’s more that they’re not designed or leveraged in a way that most effectively conveys the data they represent. For example: Some of them are fairly obvious (like the above), some are a little more subtle. In addition to pretty pictures there is a wealth of information and resources on how to make charts that don’t suck. https://thingly.net/igotw/family-night-redux/ Wed, 22 Nov 2023 00:00:00 +0000 https://thingly.net/igotw/family-night-redux/ I’ve talked about Family Night before. I thought the night before (US) Thanksgiving might be a good time to swing back around to it. So far as I’m concerned Thanksgiving is an annual version of what I strive for weekly with Family Night (albeit with decidedly less turkey and fewer awkward conversations about politics with That One Uncle). …so…check this out: This is my niece Madigan. She had a school project to present her “favorite family tradition or day of the year”…and what she picked was Family Night. https://thingly.net/igotw/serendipity/ Fri, 17 Nov 2023 00:00:00 +0000 https://thingly.net/igotw/serendipity/ When I went full-time remote in 2017 I used to travel back to the Bay Area pretty frequently. (In retrospect it was entirely too frequently, but that’s a story for another time.) The first couple of times I came out, I loaded up my calendar with meetings and 1:1s and face time with a (fairly large) number of people I wanted to see. I was sensitive to the fact that being remote would mean a substantial investment in maintaining relationships - an investment that I would’ ve gotten “for free” had I remained in situ and been able to interact with all of these folks as a part of my regular day-to-day. https://thingly.net/igotw/unintentional-art-november-2023/ Sat, 11 Nov 2023 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-november-2023/ Just a few bits ’n’ bobs I’ve got laying around in my stash this week. First up, this depiction of GCN-39427 that looks like a stack of upside-down cowboy hats: I’m almost certain this next one was also from a GCN - notice how the metric just completely stops metric-ing toward the right edge - but I’m at a bit of a loss as to which one: _This one is from Observe. https://thingly.net/igotw/discord/ Fri, 03 Nov 2023 00:00:00 +0000 https://thingly.net/igotw/discord/ Earlier this year I came across a write-up on How Discord Stores Trillions of Messages. The author talks through scaling challenges Discord had with Cassandra, their subsequent architecture change and migration to ScyllaDB, and some pretty impressive performance and maintainability gains they achieved by doing so. …but the main reason it caught my eye was this: This is Discord’s message send graph during the 2022 World Cup final, punctuated by events of interest in the match. https://thingly.net/igotw/fe2o3/ Fri, 27 Oct 2023 00:00:00 +0000 https://thingly.net/igotw/fe2o3/ I’ve been watching a handful of talks from last month’s Strange Loop conference. One that piqued my interest in particular was Andrew Black’s “Why Programming Languages Matter”. His central thesis is that the programming languages you know are a major influence on the way that you think as a programmer - a weak form of linguistic relativity, applied to programming. I’ve experienced this firsthand; the way I might think about solving a problem in C is almost certainly going to differ from how I would think about solving it in Python. https://thingly.net/igotw/wellness/ Fri, 20 Oct 2023 00:00:00 +0000 https://thingly.net/igotw/wellness/ I reckon this week is a good one to end on Wellness inDay. Right now I have many thoughts, few words. So I’ll just say this: take care of yourselves, folks. https://thingly.net/igotw/sitecon23/ Thu, 12 Oct 2023 00:00:00 +0000 https://thingly.net/igotw/sitecon23/ LinkedIn’s eighth annual SiteCon (formerly, SRE[in]con_) was last week, and the recordings and _slides are now available. For those interested in the history _of the conference, Carolyn Blood put together a great talk on _The Evolution of SiteCon. From relatively humble beginnings in 2016, the conference has grown from a one-day offsite with 17 talks and 175 attendees to a 3-day fully-virtual event featuring 55 talks and 6 workshops for ~1300 attendees. https://thingly.net/igotw/expertise/ Fri, 06 Oct 2023 00:00:00 +0000 https://thingly.net/igotw/expertise/ A recent update on my phone put me in mind of this xkcd about experts in a given field misjudging non-experts’ understanding of topics in that field: blocked URL The update is pretty straightforward: my phone asked “Would you like me to automatically clean up OTPs after 24 hours?” I had a bit of a chuckle thinking about the reaction that most people would likely have to this question. Now, maybe The Goog is trying to uplevel folks’ understanding of infosec terminology by injecting it into their products. https://thingly.net/igotw/old-dogs/ Thu, 28 Sep 2023 00:00:00 +0000 https://thingly.net/igotw/old-dogs/ My Uncle Rick was infamous for mispronouncing words. Sometimes it was a classic common misuse - “death” in place of “deaf” - but some words were uniquely his own. A chef became a “cheft”. Bratwurst was abbreviated to “brocks”. He might call me up and ask me about a new show on “The Netflick” (which I actually prefer, and have wholly adopted), or whether I’d seen “Van Halen” yet (“It’s the vampire hunter one with the guy who played Wolveree. https://thingly.net/igotw/unintentional-art-september-2023/ Wed, 20 Sep 2023 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-september-2023/ _Today’s igotw comes 100% from a recent KBMI (Key Business Metrics Intraday) follow-up email. For those uninitiated with KBMI (and too lazy to click the _ link), it’s a report tracking important business metrics. It is sent out hourly via email, and is highly-scrutinized for deviations. Also, sometimes follow-up investigation on these reports produces a set of inGraphs that kinda-sorta look like animals. Let’s dive in, shall we? To me this one looks like a unicorn with a badonkadonk trying to hide under a blanket (or maybe behind a bush): https://thingly.net/igotw/learning-inday-2023/ Fri, 15 Sep 2023 00:00:00 +0000 https://thingly.net/igotw/learning-inday-2023/ In honor of Learning inDay (and at the end of ICCDW_, no less) I figured I’d follow the lead of my past self and post a handful of links to some things I’ve _ found educational. First off, we’ve got this article on Why Do Things Go Right? Authored in 2018, this one has been sitting in my igotw “scratch list” of ideas to write about for quite a while now. https://thingly.net/igotw/event-timeline/ Fri, 08 Sep 2023 00:00:00 +0000 https://thingly.net/igotw/event-timeline/ A recent incident clued me in to a feature of Observe that I didn’t realize existed. If you click through that, the “base” view includes things like this: …which is a nice at-a-glance view of the duration of the incident (with the red bits being when the alert(s) were firing). The thing I didn’t know about was the “Event Timeline” tab: It’s a bit small, and doesn’t show up directly on the graph the way informed overlay would - e. https://thingly.net/igotw/on-glue/ Fri, 01 Sep 2023 00:00:00 +0000 https://thingly.net/igotw/on-glue/ A couple of years ago I read an O’Reilly Radar (no, not that one) article entitled “Thinking About Glue” (no, not that kind). It made me think of something I heard once along the lines of “All the code that needs to be written has already been written”, with the implication being that all the “interesting” bits have already been solved and now it’s simply a matter of composing/wiring them together. https://thingly.net/igotw/milestones/ Thu, 24 Aug 2023 00:00:00 +0000 https://thingly.net/igotw/milestones/ Last Sunday my oldest daughter turned 11. Yesterday was her first day of middle school. She changes classes and has a locker and…well…she’s still a kid, but she’s kinda gone Full Corinthians on me (so to speak). In May we celebrated LinkedIn’s 20th birthday. My 10-year work anniversary is right around the corner - I started in September of 2013 - so I’ve been around for (slightly) more than half of the company’s existence. https://thingly.net/igotw/unintentional-art-august-2023/ Thu, 17 Aug 2023 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-august-2023/ It’s been a coupla-few months - time for another UA post! First up we’ve got this lil guy, which I’ve entitled Feed Me, Seymour! Next is this beauty Andrew Guirguis sent my way - one part Seuss one part “Hidden Picture” puzzle. (It’s not a schooner it’s a sailboat!) This one reminds me of my days supporting stork/flock, when the email campaigns we were sending would reliably break pieces of the site once an hour, on the hour: https://thingly.net/igotw/load-balancing/ Wed, 09 Aug 2023 00:00:00 +0000 https://thingly.net/igotw/load-balancing/ _Over the years I’ve posted a handful of times about load balancing strategies and _when they break, but I don’t think I’ve ever talked about what load balancing is - what its fundamentals are, how it Works. Today I’d like to allow Sam to do so for me viz his ‘splainer on load balancing. In it, he explores the concept and outlines several algorithms, including nifty little visualizations that demonstrate how they work (albeit on a small scale) and what their tradeoffs are. https://thingly.net/igotw/misfeatures/ Fri, 04 Aug 2023 00:00:00 +0000 https://thingly.net/igotw/misfeatures/ A feature is just a bug in a tuxedo. - David Henke As a bit of a thought exercise, I’ve been kicking around ideas for “bad social media features” - features that I could very well imagine a Product Manager at a social media company coming up with, but that end up not being such a great idea after all. Here are a few of them, in no particular order: https://thingly.net/igotw/range-intersection/ Thu, 27 Jul 2023 00:00:00 +0000 https://thingly.net/igotw/range-intersection/ The idea for this post came out of a question in #site-reliability; specifically: “Is there a quick way to check if two services are colocated on the same host?” The person who asked the question went on to say “I was thinking of getting the hosts of the two services from range and just find the intersection.” There are multiple ways to answer this question, but if you’re comfortable with using range it might be one of the quickest/simplest. https://thingly.net/igotw/playgrounds/ Fri, 21 Jul 2023 00:00:00 +0000 https://thingly.net/igotw/playgrounds/ _It’s Play inDay, so I thought I might pass along this _list of programming playgrounds I happened across recently. Some of them you may already be familiar with (e.g., JSFiddle), but for me a lot of these playgrounds were novel. I’m particularly impressed with the compiler explorer, and crontab guru had me shouting “My God, where was this 15 years ago? I could’ve really used this!” I think the next one I’m going to dive into is Oh My Git xkcd 1597 ( is a little too real for comfort). https://thingly.net/igotw/worst-practices/ Fri, 14 Jul 2023 00:00:00 +0000 https://thingly.net/igotw/worst-practices/ I’ve recently become enamored with Worst Practices - a YouTube channel (and podcast) of brief (4-5 minute) interviews with developers sharing things they do that are commonly considered “sub-optimal” in industry. Covering things from eschewing modern IDE features to not reading documentation, every one of the videos to date covers something I’ve either done in the past or, in fact, still do. In an industry rife with imposter syndrome, I love the humanity of a channel dedicated to demonstrating that yes, there are engineers out there - good, seasoned engineers! https://thingly.net/igotw/rockets-red-glare/ Fri, 07 Jul 2023 00:00:00 +0000 https://thingly.net/igotw/rockets-red-glare/ If you squint, this just might look a little bit like fireworks being launched: …or Missile Command, I s’pose. Either way… Happy Fourth, folks! https://thingly.net/igotw/hidden-gems/ Fri, 30 Jun 2023 00:00:00 +0000 https://thingly.net/igotw/hidden-gems/ I’ve known about python -mjson.tool for a while now, but this post about CLI tools hidden in the Python standard library opened my eyes to other lil squirreled-away goodies. After a bit of grepping and awking similar to what’s in the TFA, I had a list of candidates to try out. First of all, I gotta know what turtle.py is, right? python3 -mturtle –help Traceback (most recent call last): File “/usr/lib/python3. https://thingly.net/igotw/englobing/ Fri, 23 Jun 2023 00:00:00 +0000 https://thingly.net/igotw/englobing/ TIL: Wiktionary defines englobing as the present participle of englobe, meaning “to surround as if by a globe”. It also notes that this term is used almost exclusively in the context of phagocytosis, but I’m mostly just tickled that it’s an anagram of belonging. I’m no poet, but to me this is about as linguistically euphonious as it gets. To what do you “belong”? What tribe, what family, what set or genus makes you think “People like us do things like this. https://thingly.net/igotw/myths/ Fri, 16 Jun 2023 00:00:00 +0000 https://thingly.net/igotw/myths/ A recent post about blogging myths by the inestimable Julia Evans* caught my eye. A couple of the myths she points out - “you need to be original”, “you need to be an expert” - align super-strongly with “You should write more” (referenced _here some 6 months ago), and resonate with me just as much now as they did then. I _love that she calls out not necessarily needing to explain every concept in detail, as a sort of facet of knowing your audience; if I weren’t writing this for - broadly-speaking - “the LinkedIn eng community” I’d need to weave in way more details in some places and leave out way more details in others. https://thingly.net/igotw/yumi/ Fri, 09 Jun 2023 00:00:00 +0000 https://thingly.net/igotw/yumi/ I recently came across a post by Jeff Geerling (and an associated video) about how Raspberry Pis are manufactured. I was skimming through the post - mostly looking at the pictures, if we’re being honest - when this one stood out: blocked URL It caught my eye because I have a buddy who used to work at ABB - the company who makes these badboys (in case the big bold red letters didn’t make that clear). https://thingly.net/igotw/go-links/ Fri, 02 Jun 2023 00:00:00 +0000 https://thingly.net/igotw/go-links/ Over time I’ve amassed a decently large number of go links - 90 of them in all, if I’m counting correctly: A fair number of these are defunct by this point - I’d guess about half. Of the ones that aren’t either pointing at something outdated or just outright broken, I’d say perhaps a bit less than half are work-related, and a bit more than half are “fun”. If I had to choose a favorite it might be go/circlek…which actually has a fair bit of situational utility, in a meme-ish sort of way. https://thingly.net/igotw/on-alerting/ Thu, 25 May 2023 00:00:00 +0000 https://thingly.net/igotw/on-alerting/ The below is an inGraph of a Thing that broke: Said Thing broke for about an hour, and unfortunately in this particular case nobody noticed until a couple of hours later. The Thing was an important one, so some questions were asked around gaps in monitoring/alerting. Those kinds of questions are common and reasonable - I mean, if the above graph exists then surely we should be able to set up an alert on that metric so we can catch this the next time, right? https://thingly.net/igotw/opportunity-cost/ Fri, 19 May 2023 00:00:00 +0000 https://thingly.net/igotw/opportunity-cost/ Dan McKinley’s “Choose Boring Technology” is a personal favorite of mine. The article is decidedly an “oldie but goodie” on a Tech Timeline, but it still resonates with me some 8 years after it was originally written. In it, McKinley introduces the notion of innovation tokens - a currency that every company has a limited number to spend - as a simplistic mechanism for illustrating why companies should be selective about where they decide to invest in innovation. https://thingly.net/igotw/unintentional-art-may-2023/ Fri, 12 May 2023 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-may-2023/ Due to a bit of a home re-org I’m displaced this week - which is to say, sans office (I’m writing this at my dining room table). So…opportunity to unload (unleash?) a few bits ’n’ bobs from my inGraph backlog. Let’s see what we’ve got. First up - in rapid succession, a few examples of the Moiré Effect_ that were sent my way by John Wingenbach__ and Mykyta Gubenko :_ https://thingly.net/igotw/cinco-de-mayo/ Fri, 05 May 2023 00:00:00 +0000 https://thingly.net/igotw/cinco-de-mayo/ In honor of Cinco de Mayo, I present you with this lovely chart describing types of tacos: Also, an interesting tidbit from Wikipedia: “According to Nielsen, in 2013 more than $600 million worth of beer was purchased in the United States for Cinco de Mayo, more than for the Super Bowl or St. Patrick’s Day.” Happy birthday, LinkedIn; you’re not quite old enough to drink yet (in the United States, at least). https://thingly.net/igotw/maps-april-2023/ Fri, 28 Apr 2023 00:00:00 +0000 https://thingly.net/igotw/maps-april-2023/ If you’ve been reading for a while it should come as no surprise that I have a bit of a penchant for maps (see: here _here _here here _here, , , , __…etc.) I have a _ few in my backlog of igotw ideas that I’d like to share today. First off, we’ve got Watercolor Maptiles - an open-source mapping tool that displays map data in the style of a watercolor painting. https://thingly.net/igotw/aplomb/ Fri, 21 Apr 2023 00:00:00 +0000 https://thingly.net/igotw/aplomb/ In addition to sporting the most impressive unibrow since Bert Paul Vixie, has a pretty incredible tech résumé. So much so that he was inducted into the Int ernet Hall of Fame in 2014. A good deal of his body of work has been in the space of security, as his IHoF nickname “Internet Sheriff” would seem to indicate. …but I think my favorite of his contributions is this single email. https://thingly.net/igotw/saturation/ Sat, 15 Apr 2023 00:00:00 +0000 https://thingly.net/igotw/saturation/ This week’s igotw brought to you by Nick Garvey and this write-up of GCN-38509 (network saturation caused by kafka cluster rebalances). First, we’ve got switch interface output: …and then, the network topology demonstrating where the bottleneck is: I’m going to keep this one short - just a teaser, really - but there are some other goodies in the doc. Enjoy! https://thingly.net/igotw/volume/ Fri, 07 Apr 2023 00:00:00 +0000 https://thingly.net/igotw/volume/ An amusing twitter thread caught my eye this week. It’s a picture dump of a 2017 competition to come up with the worst possible volume control interface. The entries - for the most part - fall into one (or more) of a few buckets: Mini-games: 1, 2, , 9, 11, 12, 175 Presenting every possible option: 3, 21 RNG: 4, 6, 18 Diabolically (perhaps even physically) difficult to adjust: 7, 10, 14, 15, 16, 20, 22 Pure absurdism: 13, 19 https://thingly.net/igotw/incident-response/ Thu, 30 Mar 2023 00:00:00 +0000 https://thingly.net/igotw/incident-response/ This past Saturday started out pretty normal, as Saturdays go. Got up, made coffee, shuffled around the kitchen cleaning up this and that. My wife went to yoga and hit the road for a day trip to visit friends - “Be back tonight.” My girls asked if they could go play at a neighbor’s house…which lasted for all of 3 minutes, and then turned into them bringing the neighbor kid back over here to play. https://thingly.net/igotw/on-sponsorship/ Fri, 24 Mar 2023 00:00:00 +0000 https://thingly.net/igotw/on-sponsorship/ Last October I gave a _SiteCon 2022 talk on _Being Staff. The talk covered some of the mechanics of the promotion process, but it was largely in the space of what being a staff engineer - what it means, what it looks like on a day-to-day basis. More recently, isBrian Wilcox hit me with this guide about “What do Staff engineers actually do?”…and I was frankly blown away by how much it resonated. https://thingly.net/igotw/ireland/ Fri, 17 Mar 2023 00:00:00 +0000 https://thingly.net/igotw/ireland/ In honor of St. Patrick’s Day I wanted to try and find an inGraph having something to do with Ireland. Grepping around a bit yielded some defunct Cedexis dashboards, some defunct holden Catchpoint dashboards, and a handful of IDB2 dashboards…also - you guessed it - defunct. (Side Note: I’d be curious to know what percentage of existing dashboards are actually funct at this point; my suspicion is that they’re in the minority. https://thingly.net/igotw/readme-md/ Fri, 10 Mar 2023 00:00:00 +0000 https://thingly.net/igotw/readme-md/ I’d like to take a moment to mourn the passing of a beloved open source convention: the README file. The humble README was so simple, so elegant - just a little text file, placed directly in-tree right alongside the code - with a name that demanded precisely what was meant to be done with it. A few instructions on how to build/use the thing, some warnings about caveats or gotchas or outright bugs, perhaps an email address or pointer to a website where one might look for help. https://thingly.net/igotw/magic/ Sat, 04 Mar 2023 00:00:00 +0000 https://thingly.net/igotw/magic/ I recently switched ISPs. This meant updating the wifi on every device I own. In this day and age that’s a non-trivial amount of work - hell, I’m surprised my fucking socks still work without having to enter an SSID and passphrase. Perhaps the most painful part of this Great Reset: my Sonos speakers. We’re talking two, mebbe two-and-a-half hours of jiggery-fuckery in the Sonos app, and tapping out the Bible in Morse Code on the 3 buttons on top of every Sonos device in my house. https://thingly.net/igotw/most-awesome/ Fri, 24 Feb 2023 00:00:00 +0000 https://thingly.net/igotw/most-awesome/ AC/DC is the most awesome rock band of All Time. “…but what about The Stones? Pink Floyd? etc. etc.” Well, yes, those are all excellent bands - certainly in the running for Greatest of All Time. …but I wasn’t talking about greatness or or influence or the Rock & Roll Hall of Fame, now, was I? “Most Awesome” needn’t have anything to do with those things. This differentiation between “awesomeness” and “bestness” was introduced to me by a buddy of mine who, upon seeing Alien vs. https://thingly.net/igotw/unintentional-art-february-2023/ Fri, 17 Feb 2023 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-february-2023/ Let’s take a peek through the backlog, shall we? I really like the aesthetic of this one. tl;dr: it’s a single-primary service with a manual mechanism for changing what fabric the primary is in. To be clear: You should not architect your service in this way if you can help it. However, it makes for some nifty inGraphs: I actually have no idea what this one is, but I dig its static: https://thingly.net/igotw/cause-effect-ii/ Fri, 10 Feb 2023 00:00:00 +0000 https://thingly.net/igotw/cause-effect-ii/ It’s been a little over 5 years since I wrote inGraph of the Week - Cause & Effect, talking about inGraphs that show…well…a cause and a subsequent effect. Sometimes it can be difficult to disentangle those two things, though. I’d like to walk through a recent GCN that was an example of this phenomenon. Take a look at this: Yikes - that’s a pretty big spike in errors. Someone should probably look into that, eh? https://thingly.net/igotw/the-essence-of-yaml/ Fri, 03 Feb 2023 00:00:00 +0000 https://thingly.net/igotw/the-essence-of-yaml/ A coupla-few weeks ago The yaml document from hell popped up on lobste.rs (and subsequently surfaced in #sre), outlining some of the footguns of YAML. At first read I suppose one might say “Man, YAML is really terrible, no one should ever use this” - or, at the very least, “Wat.” (The bit about sexages imal is where my head started to hurt.) …but for me, it reminded me of nothing so much as one of my favorite quotes; from the 2003 paper The Essence of XML: “…the essence of XML is this: the problem it solves is not hard, and it does not solve the problem well. https://thingly.net/igotw/single-serving/ Fri, 27 Jan 2023 00:00:00 +0000 https://thingly.net/igotw/single-serving/ I’ve known for a long time that they existed, but I didn’t know that Single-serving sites (SSSes) were a “named thing” until just recently. Let’s take a look at a few of my faves, shall we? http://www.hasthelargehadroncolliderdestroyedtheworldyet.com/ might be the first SSS I can recall seeing. If you’re not big on science news: when the Large Hadron Collider first came online, there was a big to-do about it potentially causing a black hole that would swallow up All of Everything We Know. https://thingly.net/igotw/on-pronunciation/ Fri, 20 Jan 2023 00:00:00 +0000 https://thingly.net/igotw/on-pronunciation/ Jargon, abbreviations, and acronyms abound in tech. This is probably true of just about any specialized field. These peculiar vocabularies can lead to differences of opinion on how various things are pronounced. Examples of this can be found in common system directory names. Some of them are fully spelled out and so have non-ambiguous pronunciation - /home, /local, /include. Some are abbreviated, but follow the convention of being pronounced “as they look” - /usr is user, /etc is etsy (although I suppose have heard “slash e t c” before). https://thingly.net/igotw/slices/ Fri, 13 Jan 2023 00:00:00 +0000 https://thingly.net/igotw/slices/ No, not that kind this kind…! Since 2014 Liam Quigley has meticulously documented every slice of pizza he’s eaten (and posted them on his insta). A few days ago he published the data, including a map of the locations, purchase date and price, etc. The page links to a Google doc used to create the map, or if you want to grep it you can grab the CSV the feeds the price table. https://thingly.net/igotw/welcome-to-2023/ Fri, 06 Jan 2023 00:00:00 +0000 https://thingly.net/igotw/welcome-to-2023/ Last year my wife committed to regularly doing yoga. She signed up some time in the spring (maybe April-ish?) and started doing a few classes a week. She got into a regular routine/rhythm and really stuck with it. Last month her FitBit spat out an end-of-year report revealing that she’d attended something on the order of 120 classes in 2022. Most impressive! …and then January came along, with its usual bevy of New Years’ Resolutionistas. https://thingly.net/igotw/frozen-iii/ Thu, 29 Dec 2022 00:00:00 +0000 https://thingly.net/igotw/frozen-iii/ I’m not sure what the weather is like where y’all are at, but last Friday here was a bit of a doozy: …and, because this is igotw, here it is in graph form (courtesy of World Weather): Hoo boy. I’m not 100% sure how “RealFeel” is computed - some combination of windchill and humidity and other factors, I suppose - but what I can tell you is about the time that number hits thirty below the RealFeel should probably just say “get your ass inside a tauntaun and wait it out. https://thingly.net/igotw/achievement-unlocked/ Fri, 23 Dec 2022 00:00:00 +0000 https://thingly.net/igotw/achievement-unlocked/ This week’s post is a bit of a milestone: by my count it is the 365th igotw. If you were a new hire and you wanted to catch up, at a rate of one post per day it would take you an entire year to read through the entire “backlog”. …and by that time there would be 52 new posts awaiting you. For the pedants among us saying “Well, what if it were a leap year? https://thingly.net/igotw/unintentional-art-random-edition/ Thu, 15 Dec 2022 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-random-edition/ I was poking around my stash of graphs today looking for a little inspiration when it struck me that I could just do this: ls | sort –random-sort | tail -5 | xargs open After fiddling around trying to get xargs to do proper quoting/escaping (some of the filenames have spaces in them) I said “fukkit” and just opened the 5 files manually (because, this). Anyhow, this is what the “random sort” came up with: https://thingly.net/igotw/aoc-2022/ Fri, 09 Dec 2022 00:00:00 +0000 https://thingly.net/igotw/aoc-2022/ No, not that one this one - ! Advent of Code is a nifty little set programming puzzles that Eric Wastl has been publishing since 2015. In Advent calendar fashion, new puzzles unlock each day starting on December 1 and going through December 25. If you’ve got a little free time and feel like flexing your programming muscles a bit, check it out! https://thingly.net/igotw/the-netherlands/ Fri, 02 Dec 2022 00:00:00 +0000 https://thingly.net/igotw/the-netherlands/ I was poking around traffic inGraphs looking for World Cup impact when something interesting caught my eye. Check this out: I’m a little surprised at the overall QPS - who knew we took so much traffic from the Netherlands? - but, more than that: the monotony! I see no meal times, only slight variability on weekends, and super-regular points of inflection at 09:00 and 21:00 (Central European Time). Folks, this is Not Normal. https://thingly.net/igotw/gobble-gobble/ Wed, 23 Nov 2022 00:00:00 +0000 https://thingly.net/igotw/gobble-gobble/ Thanksgiving is tomorrow, so I thought I’d (1) get the igotw out a day early, and (2) make it about turkeys! I gave “graphs about turkeys” a quick Goog and picked out a few choice tidbits. From a survey about the most popular common Thanksgiving dishes: Perhaps unsurprisingly turkey tops the list, closely followed by mashed potatoes and gravy. …although I do wonder about the 6% of people who selected mashed potatoes and did not select gravy. https://thingly.net/igotw/submarine-cables/ Fri, 18 Nov 2022 00:00:00 +0000 https://thingly.net/igotw/submarine-cables/ I recently came across this submarine cable map. It’s zoomable, pannable, searchable, and at time of writing tracks “530 active and planned submarine cables” (according to the FAQ). So I started fiddling with it. In particular, it lets you search by year ready for service all the way back to 1989…so you can kind of browse through different years and see when cables between various parts of the world were lit up. https://thingly.net/igotw/lego/ Fri, 11 Nov 2022 00:00:00 +0000 https://thingly.net/igotw/lego/ I’ve been reading a book about toy history to my youngest at bedtime, and a recent chapter focused on the history of Lego. One of the factoids at the end of the chapter: Lego claims that only 26 out of every 1,000,000 Lego bricks produced is considered defective. A little poking around online for other interesting Lego facts revealed that number to be reported at something more like 18 out of every 1,000,000. https://thingly.net/igotw/double-jeopardy/ Fri, 04 Nov 2022 00:00:00 +0000 https://thingly.net/igotw/double-jeopardy/ When I came across this video of a person getting hit twice by the same stray tire, I have to admit that my first reaction was to laugh. It’s not that I’m some unsympathetic monster - that’s a human being who probably got hurt, and I suspect everyone has had “days like that” - it’s just the damnable unfairness of the thing. My second reaction was “I wonder if I have an inGraph like this. https://thingly.net/igotw/pizza/ Thu, 27 Oct 2022 00:00:00 +0000 https://thingly.net/igotw/pizza/ Last week I ordered a pizza. I want to talk through my experience, as a sort of expressive therapy. I pulled up the pizza place’s mobile web site and flicked through the menu. It wasn’t easy to navigate (see: below) but I sorted it. One large Supreme. Coolcoolcool. I tapped in my credit card and hit “Order”. …and then. “Upgrade”, huh. On its face that sounds cool. …but…well…what am I “upgrading” ? https://thingly.net/igotw/relax/ Fri, 21 Oct 2022 00:00:00 +0000 https://thingly.net/igotw/relax/ _SiteCon22 was this week. It’_s easily my favorite work event of the year, and serves as an annual reminder why I’m proud to work for LinkedIn. (Yes, I am directly plagiarizing my feed post about it.) I consider myself to be an extrovert, thriving off of interaction with other humans - which conferences in general (and SiteCon, in particular) provide in spades, even in spite of being fully virtual for the last couple of years. https://thingly.net/igotw/sharkbite/ Thu, 13 Oct 2022 00:00:00 +0000 https://thingly.net/igotw/sharkbite/ I stumbled across a “parable about software development” earlier today on lobste.rs. It’s an amusing little tale that reminded me of the classic Malcom in the Middle lightbulb replacement scene. It also alerted me to the existence of the SharkBite product line, and that fact that SharkBite doesn’t gibe with galvanized pipe. This made me curious - I’m not hip to newfangled pipe technology - so I gave it a Goog. https://thingly.net/igotw/safety-third/ Sun, 09 Oct 2022 00:00:00 +0000 https://thingly.net/igotw/safety-third/ Last Friday Site Eng did a Tech Debt Blitz. I was asked to speak for a few minutes at the kickoff for the event. I accepted…and then immediately went into panic mode trying to figure out what I was actually going to talk about. Luckily, Brian Wilcox had my back (thanks, buddy!) bwilcox sent me a handful of different ideas, but the one that resonated is a piece Mike Rowe wrote about Safety Third. https://thingly.net/igotw/oreg-o-no/ Fri, 30 Sep 2022 00:00:00 +0000 https://thingly.net/igotw/oreg-o-no/ Last week a bit of a Thing happened - a “power event” in our Oregon data center whereby we lost an entire suite of machines, all at once. What did that look like? Well… This was taken during the incident itself, so it isn’t (exactly) accurate. What I mean: various pieces of the stack that collect the metrics were busted at the time, so the very metrics themselves were unreliable at the time. https://thingly.net/igotw/inquery/ Thu, 22 Sep 2022 00:00:00 +0000 https://thingly.net/igotw/inquery/ A recent GCN (no, not that one) got me poking around at inGraphs trying to satisfy my curiosity. While I was doing so I stumbled across this: This is an inGraph of ProcessCPUTime, which according to the docs is “the CPU time used by the process on which the Java virtual machine is running in nanoseconds.” Leaving aside the bit where it gets all wibbly-wobbly (that would be the GCN) notice how those lines are nice and tight up until just after 07: 00, when they diverge. https://thingly.net/igotw/eleven/ Fri, 16 Sep 2022 00:00:00 +0000 https://thingly.net/igotw/eleven/ Sometimes I stumble across answers to questions I didn’t know I had. This blog post about why speedometers have speeds going up to 160MPH is a recent example. The majority of folks don’t have vehicles capable of achieving anywhere near that speed, and even if they do they’re unlikely to ever actually drive that fast (sorry, Sinclair). …so why have an instrument with a top-end measurement that’s so far outside of likely - or even possible - operating parameters? https://thingly.net/igotw/icymi/ Fri, 09 Sep 2022 00:00:00 +0000 https://thingly.net/igotw/icymi/ Seth Godin is next week’s Speaker Series speaker! He’ll be talking about The Carbon Almanac - an effort to educate and drive action on climate change. I have to admit that I haven’t read the book…but also TIL: you can order a free copy of the book here. (I just ordered mine.) Seems like a reasonably good way to spend 45 minutes on a Learning inDay. https://thingly.net/igotw/before-you-panic-trust-but-verify/ Fri, 02 Sep 2022 00:00:00 +0000 https://thingly.net/igotw/before-you-panic-trust-but-verify/ Earlier this week my xbar caught my attention: 21M QPS, eh? Is Conan_ messing with us again? (For context: we only just recently crossed the 1M QPS mark.) Time to take a peek at go/stickyrouting_: Okay, so we actually peaked around 70M? With the vast majority being in prod-lor1 (I love the distribution graph): If I wasn’t suspicious before, now I most certainly was. By this time there was some chatter in #noc - “Are we under a DDoS attack? https://thingly.net/igotw/asimov/ Thu, 25 Aug 2022 00:00:00 +0000 https://thingly.net/igotw/asimov/ There was an incident earlier this week that made me sit up and pay attention. I don’t have an inGraph for it - that’s my bad - but the “root cause” is the bit that got me really thinking. If you dig around the ticket trail you’ll find a link to a bit of code called Poison Pill. In general the idea is that if nodes think that they aren’t processing incoming messages when they think they should be then they self-terminate in order to (hopefully) make their clients’ lives better. https://thingly.net/igotw/on-design/ Fri, 19 Aug 2022 00:00:00 +0000 https://thingly.net/igotw/on-design/ Earlier this week I stumbled across this post by designer Aral Balkan. The bulk of the post is about terminal application theming in GNOME, which I suppose I have some vested interest in as both a user of command line interfaces and of GNOME…but not nearly enough interest to read 2500+ words on the subject. What I did find interesting enough to both read about and to watch the embedded 49 minute talk was his take on design. https://thingly.net/igotw/takeout/ Fri, 12 Aug 2022 00:00:00 +0000 https://thingly.net/igotw/takeout/ I love to cook, but we also like to go out to eat. Unfortunately Covid-19 put the kibosh on that until relatively recently, so we turned to DoorDash. In fact, we started ordering so much that it actually made financial sense for us to subscribe to DashPass. If you order so much takeout that a $10 monthly subscription actually saves you money on it…well…you may just order too much takeout. https://thingly.net/igotw/moar-maps/ Fri, 05 Aug 2022 00:00:00 +0000 https://thingly.net/igotw/moar-maps/ I know I just did a post about maps last week…but this week I came across this wonderful world map of notable people throughout history. I’m a little embarrassed to admit how long I’ve spent panning around and zooming in/out on this lil guy. Of course, one of the first things I wanted to know: who are /were the most famous people from my home state of Ohio. Presidents out the ass, astronauts, a coupla-few actors/entertainers whose names you might recognize…not bad, Ohio. https://thingly.net/igotw/cartography-redux/ Fri, 29 Jul 2022 00:00:00 +0000 https://thingly.net/igotw/cartography-redux/ Terrible Maps is the brand of thing that sort of exists on the fringes of my awareness and then seems to pop into visibility every so often. It’s been a little while since it last resurfaced for me, so I thought I’d peruse what New Hotness they might have to offer. This time around I started to note a few general themes. For instance, maps that are “all one color”; e. https://thingly.net/igotw/the-good-old-days/ Fri, 22 Jul 2022 00:00:00 +0000 https://thingly.net/igotw/the-good-old-days/ If you’re over a certain age, you may remember a time when you could go buy a Thing - a television, a video game, a toaster, a doorbell, a smoke detector, or what-have-you - and when you got home the steps required were: Plug it in (or put batteries in it). Turn it on. Use it. Well, gentle reader, I’m afraid those days are long gone. Steps #1 and #2 still generally hold…but man oh man, is there ever a whole lot of fuss before getting to #3 these days. https://thingly.net/igotw/brian-wilson/ Thu, 14 Jul 2022 00:00:00 +0000 https://thingly.net/igotw/brian-wilson/ In the eponymous song by Barenaked Ladies there’s a great line that I think about - and reference - often. (If you know me you’ve probably heard me say it once or twice.) It goes as follows: I’m lying here just staring at the ceiling tiles, and I’m thinking ‘bout…what to think about. Roughly akin to writer’s block, to know the feeling of “thinking about what to think about” is pretty deeply human…and with company shutdown over and the _SiteCon ‘22 CFP deadline approaching in about a week-and-a-half, that line has been swimming ‘round back of my head this week. https://thingly.net/igotw/on-food-briefly/ Thu, 07 Jul 2022 00:00:00 +0000 https://thingly.net/igotw/on-food-briefly/ Last weekend I spent 4 days barbecuing. For 3 of them, I was the guy behind the grill (or at the stove). I’m not sure it made me any more proud about being an American - particularly not right now…but I reckon people getting together to eat food in celebration bridges damn near every culture at any point in human history. I suppose there’s a kind of comfort in the pragmatism of the thing. https://thingly.net/igotw/4th-of-july-shutdown/ Sat, 02 Jul 2022 00:00:00 +0000 https://thingly.net/igotw/4th-of-july-shutdown/ It’s that time of year again - the grills are going (even though it’s probably hot enough to cook the meat by just letting it sit outside for 5 minutes), the kids have popsicles, the dog is scared pantsless about all the fireworks going off. Happy 4th, folks! Enjoy. https://thingly.net/igotw/contagion/ Fri, 24 Jun 2022 00:00:00 +0000 https://thingly.net/igotw/contagion/ I recently came across this Taxonomy of Tech Debt write-up. Written a few years ago, it presents 3 major axes along which to evaluate technical debt: impact, cost to fix, and contagion. The latter - the idea of tech debt that “infects” its surrounding landscape - is the most interesting to me. I’ve seen this brand of tech debt before, but it was a bit of an “oh yeah!” moment in that I’ve never mentally framed it in quite this way. https://thingly.net/igotw/diminishing-returns/ Sat, 18 Jun 2022 00:00:00 +0000 https://thingly.net/igotw/diminishing-returns/ _Chris Carinirecently sent me _this blog post covering how Github thinks about browser support. It’s particularly timely, given this week’s retirement of Internet Explorer. Chris sent it along because it has a handful of “pretty colorful graphs”, but perhaps the one that’s most striking to me is this one: Yes, this is a PNG screenshot of the original - it would appear that Confluence doesn’t support SVG as an image type. https://thingly.net/igotw/tradeoffs/ Fri, 10 Jun 2022 00:00:00 +0000 https://thingly.net/igotw/tradeoffs/ voyager-web serves what LinkedIn has historically dubbed “Homepage” - i.e., The Site. It fell over a coupla times this week (see: GCN-37000 GCN-37014, ) and I thought it might be interesting to talk about the technique for mitigation used while investigating root cause. Check this out: I’ve set an upper bound on this inGraph so that the latency spikes don’t completely blow out the scale and make it inscrutable. The incidents are visible as the perturbations on Monday morning, and then again on Tuesday morning. https://thingly.net/igotw/unintentional-art-june-2022/ Fri, 03 Jun 2022 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-june-2022/ It’s that time of year again - the kids are out of school, the pools are open, the grills are lit, the flip-flops are flippin’ and floppin’. Ah, Summer. Let’s take a look at a few bits ’n’ bobs from my stash. First up is one I like just due to its overall energy: …and another, for its regularity (while having enough variation to not become monotonous): Summer break is likely to mean baseball games…and this lil guy reminded me of nothing so much as a foam finger: https://thingly.net/igotw/progress/ Thu, 26 May 2022 00:00:00 +0000 https://thingly.net/igotw/progress/ Once upon a time I worked in a shop where our hardware provisioning “process” was for a human being (i.e., me, or someone from my team) to go down to the datacenter, roll a crash cart over to the machine to be imaged, hook it up, drop in a RedHat CD-ROM, and spend an hour or so clicking through the installer. After that, we could ssh into the machine and install packages, set up filesystems, etc. https://thingly.net/igotw/mfa/ Thu, 19 May 2022 00:00:00 +0000 https://thingly.net/igotw/mfa/ Once upon a time, I had a Symantec VIP token on my phone. After wrecking a couple of phones (turns out I’m decidedly hard on mobile devices) and going through the pain of getting a new phone re-registered, I requested a hard token. It’s a little keyfob I carry around on my keychain. I always have my keys with me, so for a little while this was a pretty acceptable state of affairs. https://thingly.net/igotw/calltree/ Thu, 12 May 2022 00:00:00 +0000 https://thingly.net/igotw/calltree/ In all the years I’ve been posting these I don’t recall ever having talked about calltree. Let’s remedy that, shall we? Feast your eyes on this badboy: This is a subset of a calltree for a thing that broke recently. The observant…err…observer will note the response times and see that for this particular calltree voyager-api-growth returned after 3130ms, while its immediate downstream (tscp-admin-core) returned after 4540ms. Smells like a timeout to me! https://thingly.net/igotw/viola/ Thu, 05 May 2022 00:00:00 +0000 https://thingly.net/igotw/viola/ Last night my oldest daughter had her spring strings concert. The mechanics: in the fourth grade you can opt into a strings instrument (violin, viola, cello, or bass). In the fifth grade, you can either choose to stick with strings, OR you can opt to play a different instrument in the sort of “not-strings” band (woodwinds, horns, percussion, etc.) My oldest is in the fourth grade, and this year she decided to try the viola. https://thingly.net/igotw/right-vs-left/ Fri, 29 Apr 2022 00:00:00 +0000 https://thingly.net/igotw/right-vs-left/ This is decidedly not a political post. I am a long-time wearer of corrective lenses. I think I got my first pair of glasses when I was 5 or 6, and went through a steadily-increasing set of prescriptions that would make a Coke bottle jealous. (Sight-y thicc.) Some 5-6 years after getting my first pair of glasses I looked into contact lenses. The doc told me that my astigmatism was so severe that the only contacts that would actually work to correct my vision were gas permeable - i. https://thingly.net/igotw/d-t-o/ Sun, 24 Apr 2022 00:00:00 +0000 https://thingly.net/igotw/d-t-o/ Little DTO You’re really lookin’ fine Five nights in a cabin Should be a real good time A couple hours in the van, man, The kids are gonna whine C’mon and gas it up, turn it on, blow it out! DTO — Completing the rest of the verses is an exercise left to the reader. Many thanks to Ronny & the Daytonas for the inspiration, and to whatever genius invented Spring Break. https://thingly.net/igotw/unintentional-art-april-2022/ Fri, 15 Apr 2022 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-april-2022/ This week I’ve just got a few bits ’n’ bobs from my stash before I head off for DTO (yay, Spring Break!) This first one reminds me of nothing so much as a P-51 Mustang, in profile. Also worth noting: this may be the first prod-us-e inGraph to appear in an igotw? Next up is…hmmm…a sort of bookend? Or…perhaps Duder standing there at midnight on the 9th needs to get out of the way? https://thingly.net/igotw/moire/ Sun, 10 Apr 2022 00:00:00 +0000 https://thingly.net/igotw/moire/ Every now and again conditions will be just right for inGraphs to exhibit a sort of moiré effect. Here are a couple of examples: It’s interesting - and one might say lovely - but I have to admit that I don’t quite grok how this actually happens. Any ideas? https://thingly.net/igotw/whats-in-a-name/ Sat, 02 Apr 2022 00:00:00 +0000 https://thingly.net/igotw/whats-in-a-name/ Some months ago I wrote a proposal for handling how things get named. It was a bit tongue-in-cheek (in a “just kidding…unless you’re gonna do it” sort of way), but more recently I’ve been giving some thought to how I might actually go about the business of being the Naming Tsar. For at least some things I think I’d take the approach of “nonsense names” - names that have no semantic meaning attached, that have nothing to do with the thing they represent. https://thingly.net/igotw/what-s-broke-how/ Fri, 25 Mar 2022 00:00:00 +0000 https://thingly.net/igotw/what-s-broke-how/ In the thick of the pandemic when both my girls were Remote Learning and my wife needed a little distance, I carved out some space in my home office and set up two computers, one for each of them. Both hand-me-down machines that I rebuilt/repurposed from hardware I’d once used for some other purpose, they are mostly used for playing Roblox and watching YouTube videos (of other people playing Roblox). Sometimes these computers stop working - as computers tend to do - and being the resident Computer Guy I’m called upon for tech support. https://thingly.net/igotw/questions/ Sat, 19 Mar 2022 00:00:00 +0000 https://thingly.net/igotw/questions/ Frequently inGraphs are used to find answers - When did the Thing break? What does the latency look like? What’s the current traffic distribution? But often they can also prompt questions. Take this beaut’ Joe Gillotti sent my way: What we’re looking at here is the state of all autoalerts in prod-ltx1. …and it makes me wonder… There are something like 330K alerts in prod-ltx1. Assuming even distribution among prod fabrics that means upward of a million alerts for all of production. https://thingly.net/igotw/unintentional-art-march-2022/ Thu, 10 Mar 2022 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-march-2022/ TIL The lil Java mascot dude has a name: Duke. …and…well…this inGraph decidedly took a turn for the Duke: Following that, this one that looks kind like the ass-end of a whale: It leaves me wanting more, wondering what the front half of this feller looks like. And then there’s this: When I look at it I hear something like sproi-oi-oing! …and then there’s this one, which is kinda boring but feels like it should be entitled “Get With the Program” or “You Had One Job” or suchlike: https://thingly.net/igotw/bloody-marys/ Fri, 04 Mar 2022 00:00:00 +0000 https://thingly.net/igotw/bloody-marys/ Several months ago we made plans to go to brunch with some of my wife’s old college buddies. We met at what can only be described as a “kid-friendly dive bar” (don’t look at me that way, some of you know exactly the kind of place I’m talking about). It was a lovely establishment - one might go so far as to call it “charming” - with a great brunch menu, friendly staff, and…a bloody mary bar. https://thingly.net/igotw/creativity/ Fri, 25 Feb 2022 00:00:00 +0000 https://thingly.net/igotw/creativity/ Some years ago I spent a little time exploring different representations of the relative ages of myself, my wife, and my kids. The “bubble chart” at the end of _this post was one of the results. I thought for Creativity inDay I might swing back around to the concept, this time representing our birth dates as quadratic _ functions of the form (month)x2 + (day)x + (year). Twenty-ish minutes fiddling around with Function Plot produced this: https://thingly.net/igotw/douglas-mcilroy/ Fri, 18 Feb 2022 00:00:00 +0000 https://thingly.net/igotw/douglas-mcilroy/ A coupla-few weeks ago I was pretty deep down the wikihole, poking through bits of Unix lore. Everyone knows Ken Thompson and Dennis Ritchie (for certain value of “everyone”) - their names are Legend - but the name that caught my attention was one I wasn’t familiar with: Douglas McIlroy. Okay, Wikipedia…teach me who’s a Dougie. First up: he “…developed several Unix tools, such as spell, diff, sort, join, graph, speak, and tr”. https://thingly.net/igotw/kaboom/ Fri, 11 Feb 2022 00:00:00 +0000 https://thingly.net/igotw/kaboom/ Who remembers the humble bathroom magazine caddy? Situated alongside the commode, it contained some few lightly-fondled periodicals issued some time in the past decade or so. In those halcyon ante-Zynga days one needn’t wonder too hard about who touched the Sports Illustrated last or what they were doing - one knew - its dappled pages at least served as an alternative to re-learning the formula for Pantene Pro-V by rote. https://thingly.net/igotw/unintentional-art-february-2022/ Thu, 03 Feb 2022 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-february-2022/ I remember sitting in front of the TV the morning after a big snowfall watching the school closings scroll across the screen when I was a kid, hoping and praying for a Snow Day. Nowadays, the school district just calls everyone…and sends an SMS…and sends an email. Except it’s not called a Snow Day any more; it’s called - get this - a Calamity Day. I have to admit it sounds way more badass than Snow Day. https://thingly.net/igotw/shims/ Fri, 28 Jan 2022 00:00:00 +0000 https://thingly.net/igotw/shims/ A buddy of mine recently took a position with a new employer. Most of his meetings at his former gig had a primarily non-technical business cast of characters - finance, accounting, et al. The new-new requires much more direct interaction with engineers, and for him that has been illuminating. Perhaps even disillusioning. His perception of engineers is (was?) that they work in the space of uniformity and precision: mathematics and schematics, micrometers and arithmetic, and so forth. https://thingly.net/igotw/cobras/ Sat, 22 Jan 2022 00:00:00 +0000 https://thingly.net/igotw/cobras/ Wikipedia defines a perverse incentive as one “that has an unintended and undesirable result that is contrary to the intentions of its designers”. The classic anecdotal example of this corollary to Goodhart’s Law is The Cobra Effect. Legend has it that Way Back When the British wanted to do something about the population of highly-venomous cobras in Delhi, so they started offering a bounty on cobra heads. On its face this may not seem like an unreasonable approach, but once cobras became a monetizable commodity the natural thing to do was to become a snake breeder (thus increasing the overall cobra population). https://thingly.net/igotw/y2k22/ Fri, 14 Jan 2022 00:00:00 +0000 https://thingly.net/igotw/y2k22/ It strikes me that we are likely hiring folks who are too young to remember the Y2K Bug - who were still wee babbies in nappies when it was all over the news. For their benefit, I’d like to present a similar bug (albeit with a much smaller scope and scale). The tl;dr: software attempting to convert a date formatted as YYMMDDHHMM into a 32-bit signed integer. (2201010000 > 2147483647) https://thingly.net/igotw/41/ Fri, 07 Jan 2022 00:00:00 +0000 https://thingly.net/igotw/41/ I have an early January birthday (yesterday, in fact). Birthdays and New Years are typically times for reflection/introspection, so one might think I’ve been just reflecting and introspecting my ass off all week. However, I also took the week after the company shutdown off (this week). Beyond “huh, my age is a prime number again” about the only reflection I’ve done is deciding when (and whether!) I might consider changing out of my sweatpants in the morning. https://thingly.net/igotw/pinkeye/ Thu, 30 Dec 2021 00:00:00 +0000 https://thingly.net/igotw/pinkeye/ For Christmas this year I got…pinkeye! Hooray for me. Before you proceed with reading this post, go ahead and take a little time to make your jokes about how I got pinkeye. Victim-blame away, seriously. I’m pretty sure I’ve heard them all by now. The jokes are all funny and I can appreciate that brand of humor. Don’t worry, I’ll wait. It’s okay. Really. Coincidentally I had an appointment with my optometrist scheduled for 12/29 so I suppose things kind of worked out (insofar as having pinkeye can “work out”), but there were a couple of aspects of the whole experience that I thought might be interesting to talk about. https://thingly.net/igotw/christmas-eve/ Fri, 24 Dec 2021 00:00:00 +0000 https://thingly.net/igotw/christmas-eve/ Stir onions until you can stir no more. This is the Good Life. Merry Christmas, folks! https://thingly.net/igotw/log4shell/ Fri, 17 Dec 2021 00:00:00 +0000 https://thingly.net/igotw/log4shell/ Welp…it’s been some kinda week, hasn’t it? Unless you’ve been living under a rock you’ve probably at least heard about the Log4Shell vulnerability by now…and as vulns go, this one’s a doozy. Prior to this week, this is the shortlist of the most critical vulnerabilities I might have expected out of a logging library: logs nothing (null attack) logs too much (possible (D)DOS attack) logs ASCII-art pictures of dicks, exclusively (PUNKT attack) some combination of the above - like, ERRORs log nothing but INFOs log pictures of dicks, exclusively (perhaps we could call this one “Kutcher- DOS”) https://thingly.net/igotw/gratitude-2021/ Sun, 12 Dec 2021 00:00:00 +0000 https://thingly.net/igotw/gratitude-2021/ This week’s inDay theme was Gratitude and I’ve been thinking about that a bit. Last year I posted about a bunch of things I was grateful for…but being grateful isn’t the same as expressing gratitude. “I like hot dogs” isn’t quite the same thing as expressly telling the inventor of the hot dog “I am grateful that you invented the hot dog”, yeah? So I thought this year I might call out Carolyn Bloodspecifically. https://thingly.net/igotw/drip-drip/ Fri, 03 Dec 2021 00:00:00 +0000 https://thingly.net/igotw/drip-drip/ When we’re talking about a resource leak, often the resource we’re talking about is memory. However, this needn’t always be the case. Sometimes it’s threads. The scale might be a little hard to grok, but the above is an example of a slow thread leak. How slow? Something on the order of a few threads an hour per instance. With a jetty threadpool size of 150 this particular leak would take a couple of days to chew through the entire threadpool on a given instance and render it useless until restarted. https://thingly.net/igotw/thanksgiving-2021/ Thu, 25 Nov 2021 00:00:00 +0000 https://thingly.net/igotw/thanksgiving-2021/ Last weekend I asked my brother-in-law if he could swing by for a bit Sunday morning before the Browns game to give me a hand moving some furniture around. He couldn’t make it - he had plans to take his mom to brunch - but he told me he had some free time Thursday morning. “Thursday” meaning today. Thanksgiving Day. …and he’s hosting. Signals get lost in SMS, so I don’t know if he was joking (“haha I’m free on Thursday”), or trolling me (“screw you and your couch, bro”), or genuinely gung- ho about screaming over here for an hour while the turkey was cooking and then rushing on back home to finish out watching the Macy’s Thanksgiving Day Parade and getting the rest of the food ready. https://thingly.net/igotw/poka-yoke/ Fri, 19 Nov 2021 00:00:00 +0000 https://thingly.net/igotw/poka-yoke/ TWIL: poka-yoke - a Japanese term for “mistake-proofing”. I’ve been thinking about it quite a bit. One example is a car that won’t let you shift out of Park (or perhaps won’t even let you even start it) without the brake pedal being depressed. Seth Godin calls out microwaves, which won’t turn on unless the door is closed. Similarly, the lever on my toaster won’t engage unless it’s plugged in; this one is less about safety, and more about not giving you the false sense that your bread is going to be toasted unless the toaster is actually capable of toasting the toast. https://thingly.net/igotw/mr-kerwin/ Thu, 11 Nov 2021 00:00:00 +0000 https://thingly.net/igotw/mr-kerwin/ Mr. Kerwin’s name has been mentioned here before. He was one of my high school English teachers. I can’t say who my “favorite” teacher was - frankly, I don’t remember all of them (which is a data point in itself) - but Bill Kerwin is certainly in the top three. Bill wrote and published a novel. Big Sleep Boogie is a fun bit of sci-fi, written in the style of old-timey private dick novels…but set in the afterlife. https://thingly.net/igotw/unintentional-art-november-2021/ Fri, 05 Nov 2021 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-november-2021/ Let’s take a peek at a few bits ’n’ bobs from my “stash” today. First up, a nice lil angular rainbow - one that I guess decided to “bend” instead of “curve”: Next, there’s one I’ve entitled “The Espresso Guillotine”: The next pair of graphs are actually related to one another. I was trying to identify (automated) restart behavior on a particular host. I started off looking at this: Okay, so that looks kinda cool, but it’s a little hard to figure out exactly what’s going on. https://thingly.net/igotw/percentiles-part-i/ Fri, 29 Oct 2021 00:00:00 +0000 https://thingly.net/igotw/percentiles-part-i/ The following is adapted from a section of a monitoring talk I’m putting together. I thought these bits might be interesting/relevant here. Operational metrics are frequently expressed in terms of percentiles - most notably latency metrics. If you’re looking at an inGraphs dashboard that includes latencies it’ll most likely include 95th percentile (p95), or p99, or perhaps some combination therein. Some dashboards incorporate each of p50 /p95/p99 in an attempt to give some high-level idea about the overall shape of the latency distribution. https://thingly.net/igotw/the-pentagon-wars/ Fri, 22 Oct 2021 00:00:00 +0000 https://thingly.net/igotw/the-pentagon-wars/ This clip from the movie The Pentagon Wars should be required viewing material for anyone who has ever managed a project. In it, Cary Elwes and Viola Davis piece together the history of the Bradley Fighting Vehicle. Originally designed to be a lightweight troop transport, over the course of 17 years and multiple design changes the vehicle morphs into “a troop transport that can’t carry troops, a reconnaissance vehicle that’s too conspicuous to do reconnaissance, and a quasi-tank that has less armor than a snowblower, but has enough ammo to take out half of D. https://thingly.net/igotw/learning-inday/ Mon, 18 Oct 2021 00:00:00 +0000 https://thingly.net/igotw/learning-inday/ In observance of Learning inDay I thought I might put together a linkdump of things I consider to be “educational”. While this list is curated (by me) it’s also a decidedly unorganized curriculum; it runs the gamut from tech to marketing to biology to…well, you’ll see. I’ll start this off with Carolyn Van Slack’s GopherCon ‘19 talk “Design Command-Line Tools People Love”. Imminently practical even for non-golang developers, this is the talk that I want anyone who will ever write a CLI to watch - and then watch again, and take notes, and maybe watch a couple more times - before ever putting hands to keyboard. https://thingly.net/igotw/here-comes-everyone/ Fri, 08 Oct 2021 00:00:00 +0000 https://thingly.net/igotw/here-comes-everyone/ ICYMI Facebook, Instagram, and WhatsApp experienced a major outage earlier this week. How major? Welp, they were hard down for something on the order of 6 hours, so…pretty Major. What did that look like for us? Well, at a high level it would appear that Facebook going down is good for (our) business: Lookit all those extra queue pee esses! If you’re interested, there are also some data scientists off sciencing the data (like they do), teasing out additional interesting little tidbits - things like sessions increase on mobile being roughly 3X the increase we saw on desktop, or a disproportionate amount of this additional traffic going to Messaging (which saw a 62. https://thingly.net/igotw/unintentional-art-synesthesia/ Thu, 30 Sep 2021 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-synesthesia/ It’s been (roughly) a million years since I read A Wrinkle In Time, but there are parts of it that I still remember vividly. One of those bits is the planet with the creatures that had no eyes. When I was a kid, the idea of trying to describe the sense of sight to someone who had never had it - to an entire species who had never seen anything - completely blew my mind. https://thingly.net/igotw/this-is-sparta/ Sat, 25 Sep 2021 00:00:00 +0000 https://thingly.net/igotw/this-is-sparta/ This week marks the 300th inGraph of the Week. The very first “real” post was on January 7, 2016 - 2,088 days ago. It was a different time. President Obama was in the final year of his second term. Tom Petty was still alive. So was Prince Muhammad Ali Alan Rickman. . would pass the following week. (RIP, all) Since that time, 15,493 GCNs have been opened, and 5,123,476 tickets have been created across all JIRA projects…and counting. https://thingly.net/igotw/seinpost-quotes/ Fri, 17 Sep 2021 00:00:00 +0000 https://thingly.net/igotw/seinpost-quotes/ When I was growing up my dad didn’t want my sisters and I to watch Friends (for reasons I won’t get into here). Somehow my mom talked him into watching the episode that aired after Superbowl XXX. JCVD - then at the height of his fame - was in that episode, and somewhere along the way he uttered the phrase “I can crush a walnut with my butt.” Dad thought that was the funniest thing in the world. https://thingly.net/igotw/depths-of-wikipedia/ Wed, 08 Sep 2021 00:00:00 +0000 https://thingly.net/igotw/depths-of-wikipedia/ For a long time I struggled trying to figure out exactly what Instagram was. I mean, sure, pictures of stuff…but like…what is it for? Does it “fit” into my life? … how? …and then, I came across Depths of Wikipedia. As someone who frequently falls down the wikihole, this is an Insta of Greatness. From the hilarious to the bizarre to the downright fascinating…it’s all there. So far as I’m concerned, this is what Instagram is for. https://thingly.net/igotw/patterns-banding/ Wed, 08 Sep 2021 00:00:00 +0000 https://thingly.net/igotw/patterns-banding/ band·ing /bandiNG/ - noun - the presence or formation of visible stripes of contrasting color. _While I’ve never talked explicitly about banding as a distinct pattern, I’ve posted inGraphs demonstrating the phenomenon _on multiple occasions. I suspect that most of the time banding is a pattern that’s indicative of something not-so-good happening, and most (all?) of graphs I’ve posted point out some issue with load balancing - an uneven distribution of traffic among nodes. https://thingly.net/igotw/unintentional-art-august-2021/ Fri, 27 Aug 2021 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-august-2021/ I’m not entirely sure what happened to Summer. It’s still hot, to be sure…but the kids are back in school, college football is starting up again, and Labor Day is right around the corner. Ah, well. Let’s take a look at a few bits of UA, shall we? I dig the aesthetic of this one, but I’m not entirely sure what’s going on in it. If I had to hazard a guess I’d say maybe one misbehaving host generating a client retry storm on the hosts in good working order? https://thingly.net/igotw/olives/ Fri, 20 Aug 2021 00:00:00 +0000 https://thingly.net/igotw/olives/ There are very few foods I dislike. For the most part, if a thing is edible then I’ll eat it. Not olives. If olives are in a pasta salad I’ll eat around them. I might accidentally get one of the filthy bastards on my fork. I won’t spit it out, but I’ll be less-pleased about my pasta salad experience than I would’ve been otherwise. If olives are on a pizza I won’t pick them off, but if I’m the one picking the toppings olives aren’t going to be among them. https://thingly.net/igotw/on-curation/ Thu, 12 Aug 2021 00:00:00 +0000 https://thingly.net/igotw/on-curation/ When I was 17 a close friend made me a mixtape. This may be showing my age a bit, but I’m talking about an actual-ass cassette tape. She labeled it “Songs by Jane”; it had songs on it, and her name was (and still is) Jane. Clever gal, Jane. I listened to this mixtape on repeat for months. The tape is now lost to time and I don’t recall precisely what songs were on it (I really wish I would’ve written down the setlist somewhere along the way), but it was a huge part of my personal zeitgeist. https://thingly.net/igotw/two-problems/ Fri, 06 Aug 2021 00:00:00 +0000 https://thingly.net/igotw/two-problems/ There’s an old trope about “…now they have two problems.” (No, not those two.) But what about when the problems don’t stop at just two? If you’ve been around a while you’ve probably seen this kind of thing - one problem leading to another, which leads to another… Suppose you’ve got a misbehaving service. In an attempt to get more information about what’s going on you turn on debug logging. Welp… https://thingly.net/igotw/unintentional-art-alternative-edition/ Fri, 30 Jul 2021 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-alternative-edition/ A funny thing happened to me yesterday. I sat down to write the igotw, and I already had a pretty good idea of what I wanted to say so it was coming pretty easily. Keyboard’s clattering, I’m In the Zone, right? I got about 3/4 of the way into the post and I started to get a bit suspicious; this is an amusing little story that I think about often and that I’ve told before, sure, but…is this too easy? https://thingly.net/igotw/belonging/ Fri, 23 Jul 2021 00:00:00 +0000 https://thingly.net/igotw/belonging/ Most of the folks reading this are members of a “team” of some kind…but how do you know? What makes you feel like you’re a member of a team, and not just a collection of individuals off doing their own thing? Happy hours, offsites, occasional lunches and water-cooler conversation - all of these things can contribute to a feeling of “teamliness”. In keeping with today’s inDay theme of “Belonging” I’d like to talk about something _Ryan Dohertyinstituted for a team _ I’m on that has been super-helpful in this regard. https://thingly.net/igotw/context/ Fri, 16 Jul 2021 00:00:00 +0000 https://thingly.net/igotw/context/ I suspect most folks have seen or heard the (in)famous Bill O’Reilly “We’ll Do It Live” clip. The man comes off as an unhinged nutjob - a raging psycho whose cheese done slid off his cracker. Not to defend Mr. O’Reilly, but in the name of empathy I’d like to point out that this clip is presented without any context. Why is that important? Shouldn’t he always comport himself with professionalism, irrespective of what’s going on around him? https://thingly.net/igotw/form-vs-function/ Thu, 08 Jul 2021 00:00:00 +0000 https://thingly.net/igotw/form-vs-function/ I’ve spent a fair bit of this 4th of July break educating myself about woodworking - a couple of books, a whole shitload of YouTube videos. One unfamiliar term I came across (among several; see also: chamfer, spelching) was “spalted maple”. So I Googled it, like ya do, and according to Wikipedia spalting is “any form of wood coloration caused by fungi”. As one might expect, this has a generally negative effect on the wood itself - weakens it, softens it…too much fungus and you’ve basically got rotted sawdust. https://thingly.net/igotw/sierpinski-triangles/ Thu, 01 Jul 2021 00:00:00 +0000 https://thingly.net/igotw/sierpinski-triangles/ Last week I came across this post about creating Sierpiski triangles by drawing dots in relation to three points of a starting triangle. The thing that stood out to me is that these dots are randomly selected (within a set of constraints). It doesn’t matter which starting triangle you choose, where you start drawing your dots, etc…it always converges on the same pattern. My mind was blown; I think my initial reaction was something like “Whoa”, followed by something like “Nuh-uh”, and then sort of devolving into “Hmmmmm”. https://thingly.net/igotw/panda-acts-of-kindness/ Fri, 25 Jun 2021 00:00:00 +0000 https://thingly.net/igotw/panda-acts-of-kindness/ I recently got all of my desk crap from the office. It was a little like opening a time capsule - nearly 8 years’ worth of history packed into a couple of cardboard boxen. The card signed by my former teammates that was on my desk my first day, a snakes’ nest of cables and dongles and adapters, conference swag dating back to 2014 or so…you get the picture. Of the things I didn’t throw away, I passed off a lot of these “goodies” to my kids - the LinkedIn-branded Rubix cube, the squirt gun from some cloud vendor or other, etc. https://thingly.net/igotw/euro-2020/ Thu, 17 Jun 2021 00:00:00 +0000 https://thingly.net/igotw/euro-2020/ It was delayed a year due to The ‘Vid, but the the UEFA European Championship is once more upon us. I thought I’d do a bit of a reprise of the last time and take another peek at its impact to site traffic. I also thought it might be interesting to compare two traffic from two teams’ respective countries while they were playing a match. …and so I give you Italy vs. https://thingly.net/igotw/messaging/ Fri, 11 Jun 2021 00:00:00 +0000 https://thingly.net/igotw/messaging/ Earlier this week I came across Kehoe’s post about some analysis he did on his LinkedIn Messaging inbox (which mirrored a similar analysis that _Benjamin _ Kane had done). I decided I’d go ahead and take a look at my own LinkedIn messages over time: This diverges from Kehoe & Kane’s work in a couple of ways. First of all: I generated the chart in a spreadsheet. (It was just plain the simplest way to get what I wanted in ~10-15 minutes. https://thingly.net/igotw/the-bradilator/ Thu, 03 Jun 2021 00:00:00 +0000 https://thingly.net/igotw/the-bradilator/ An old friend and former colleague from a previous life is stopping by tonight. Brady likes drinking beer on front porches, I have beer and a front porch…the thing was basically Meant to Be. I haven’t seen Brady in probably 10-15 years - he’s spent the past decade or so of his semi-retirement wandering the Earth like Kane from Kung Fu - so when he let me know he was going to be in town it triggered some old memories. https://thingly.net/igotw/todo/ Wed, 26 May 2021 00:00:00 +0000 https://thingly.net/igotw/todo/ I recently came across a post about plotting the TODO history of a handful of popular open source projects. For example, here’s what the TODOs for the Linux kernel look like over time: I thought it might be interesting to do something similar with LinkedIn source code. We’ve got quite a bit of code lying around, so for the purposes of scoping I decided to select a relatively-large multiproduct with a reasonably-long commit history. https://thingly.net/igotw/naming-tsar/ Thu, 20 May 2021 00:00:00 +0000 https://thingly.net/igotw/naming-tsar/ I could’ve sworn I did a post about this at some point in the past, but all I could find digging through the archives was a post on Two Hard Things from a few years back (and it turns out that post was actually about the other Hard Thing). So let’s talk about Naming Things. Naming matters - it’s important. …but also, it doesn’t, and it’s not. (Fun semi-related side note here: Who remembers when the . https://thingly.net/igotw/on-inertia/ Fri, 14 May 2021 00:00:00 +0000 https://thingly.net/igotw/on-inertia/ I recently came across a post about “low-priority alerts” that resonated pretty strongly with me. Coincidentally, I read it shortly after (re-)joining a team and getting (re-)added to all of that team’s email distros, which led to an influx of email “alerts” that I didn’t have existing filters for. …and when I say “influx” I’m talking about hundreds of emails a day. Here’s the thing: that sudden flood of emails was not a surprise to me. https://thingly.net/igotw/one-liners/ Fri, 07 May 2021 00:00:00 +0000 https://thingly.net/igotw/one-liners/ I came across a post about one-liners recently. It got me thinking about the commands I use the most. An incomplete list, in no particular order: emacs -nw - Yes, I am an emacs user (who knows just enough vi to get by when emacs is unavailable.) And yes, I could alias this so that I don’t have to type the -nw every time. This is largely a muscle-memory holdover from a Previous Life, when sometimes I’d log into a machine that (1) had an X server running with forwarding enabled, and (2) didn’t have an alias set up. https://thingly.net/igotw/on-interviewing/ Thu, 29 Apr 2021 00:00:00 +0000 https://thingly.net/igotw/on-interviewing/ I’ve done a fair number of interviews in my time as a LinkedIn employee. While it takes time away from my “day job”, I reckon it’s pretty rad that I get to directly contribute to selecting the incredible individuals that I get to work with on a daily basis. The process has room for improvement (like any process ever in the history of processes), but on the whole I’d say it’s pretty solid and provides a reasonably consistent rubric for evaluating incoming talent. https://thingly.net/igotw/unintentional-art-april-2021/ Sat, 24 Apr 2021 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-april-2021/ This week we’ve got a few bits ’n’ bobs I pulled out of my backlog. This one reminds me of a retro-futuristic rendering of a train: This one from Joel De Gan reminds me of power lines drooping down off the pole toward houses: What I see here is a pile of praying mantises crawling all over one another, trying desperately to get out of the right side of the frame: https://thingly.net/igotw/cyanoacrylate/ Thu, 15 Apr 2021 00:00:00 +0000 https://thingly.net/igotw/cyanoacrylate/ I’m assuming most folks have used superglue before, but for the uninitiated here’s a step-by-step guide: Rummage through the junk drawer for the half-tube of superglue you have left over from the last time you used it 6-12 months ago. If you have more than one junk drawer, no sweat - just pick one, then try another. By law, the superglue will always be in the second drawer (local statutes may vary). https://thingly.net/igotw/restup/ Fri, 09 Apr 2021 00:00:00 +0000 https://thingly.net/igotw/restup/ It’s been a few years since I posted _Boring inGraphs, but I thought in honor of _RestUp! Week it might be appropriate to drop something like this error rate inGraph: : …and now as before, to all y’all oncall out there: hopefully your error rates have looked similar this week. Happy RestUp! https://thingly.net/igotw/ever-given/ Fri, 02 Apr 2021 00:00:00 +0000 https://thingly.net/igotw/ever-given/ Anyone who has paid even a little bit of attention to the news over the past couple of weeks will recognize the name of the ship that blocked the Suez Canal. This ain’t no lil dinghy, folks - in fact, the numbers on this one are difficult for me to wrap my head around. The Ever Given is 400m-long (1,312ft) and weighs 200,000 tonnes… Man…where was this thing when those guys had that problem with that shark that one time? https://thingly.net/igotw/learning/ Fri, 26 Mar 2021 00:00:00 +0000 https://thingly.net/igotw/learning/ Folks with kids will sometimes talk about how their children teach them things. Patience, unconditional love, a renewed appreciation of “the smaller things in life” - that’s the brand of claptrap they’re usually talking about. That’s fine, all those things are grand. But check this out: This photo contains a few subtle indicators that my children have been adding things to the grocery list. The suboptimal handwriting and spelling, the preponderance of ice cream and ice-cream-related products…and, crucially, the positioning of the text. https://thingly.net/igotw/happy-st-pi-tty-s-day/ Thu, 18 Mar 2021 00:00:00 +0000 https://thingly.net/igotw/happy-st-pi-tty-s-day/ I have a pi problem. Every time a new version of the Raspberry Pi comes out I’m compelled to buy it. I have at least a dozen of ’em, several of which are on or about my desk in the office. I’ve bought accessories hats, …I backed a Kickstarter for a RasPi tube amp (also currently sitting on or about my desk in the office). Yes, I bought the 400. Yes, it’s sitting here right in front of me, collecting dust. https://thingly.net/igotw/leaks/ Sat, 13 Mar 2021 00:00:00 +0000 https://thingly.net/igotw/leaks/ I’ve posted about leaks several times in the past _ _. Let’s take a look at another, shall we? This set of inGraphs tells a pretty good story (thanks Vishnu C N!) - the longer the service is up, the more memory it consumes…either until someone comes in on Monday morning and deploys/restarts it, it tops out, starts GCing and 5XXing. What was the cause? In vcn’s own words “a cache to store or info about non critical endpoints for load shedding” (that was growing unbounded). https://thingly.net/igotw/omphaloskepsis/ Sat, 06 Mar 2021 00:00:00 +0000 https://thingly.net/igotw/omphaloskepsis/ About a month ago I had a touch of writer’s block. I reached out to The Wolffe and he gave me a few helpful suggestions - none of which I’ve actually used…until today! One of them was “How you write regularly - why do you do it? What’s fun/hard/bad about it?” Why do you do it? I’ve been asked some variant of this question a handful of times in the past - “Why do you do igotw? https://thingly.net/igotw/frequency/ Fri, 05 Mar 2021 00:00:00 +0000 https://thingly.net/igotw/frequency/ A little while back someone in some slack channel mentioned looking forward to igotw being posted on Fridays. “Nonsense”, said I, “I make it a point to post every Thursday, and only spill over into Friday if something comes up and I end up being too busy.” But then I started thinking about it, and I couldn’t actually remember the last time I’d posted on a Thursday. Time to look at the data. https://thingly.net/igotw/unintentional-art-february-2021/ Fri, 19 Feb 2021 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-february-2021/ Last week I posted about the impact the Superbowl had on (US) site traffic. Welp, ‘Murricans most certainly aren’t the only ones who like a good party - here’s what the impact of Carnival looks like: The connection is a little thin here…but if you had to imagine what the music was like, maybe the equalizer would look something like this? …and utterly unrelated: elephants have come up more than once in this space. https://thingly.net/igotw/superbowl-lv/ Fri, 12 Feb 2021 00:00:00 +0000 https://thingly.net/igotw/superbowl-lv/ I’ve posted before about the impact of world events - notably, the Superbowl - on site traffic. I thought I’d take another peek this year: Huh. That really doesn’t look like much. Maybe if I tighten up the window to align more closely with the time of the game? Zoom. Enhance. Still just minor perturbations - a blip on the radar, nowhere near the kind of impact we’ve seen in previous years. https://thingly.net/igotw/jargon/ Sat, 06 Feb 2021 00:00:00 +0000 https://thingly.net/igotw/jargon/ In my 7+ years at LinkedIn I’ve had the opportunity to work with some supremely-talented folks. When I was a member of the growth-sre team Jerrod Lowmaster was one such individual. He’s long since moved on to his Next Play, but this dude was (and likely still is) a Data Scientist par excellence in all the ways I’ve had the luxury of coming to expect from LinkedIn engineers. Supreme tech chops, driven, deep subject matter knowledge, etc. https://thingly.net/igotw/right-vs-best/ Fri, 29 Jan 2021 00:00:00 +0000 https://thingly.net/igotw/right-vs-best/ The topic of org charts came up in a meeting I was in a little while back. Cinco has some super-basic org chart data, but it doesn’t really do a good job of telling you which teams/frontline managers map onto which specific product(s) within any given org. Picking on _Nicholas Han to give a specific example: _ _Cinco will tell you who reports to him, who his peers are, and who his direct manager is…but it doesn’t give any indication of what teams or products his _ crew actually supports. https://thingly.net/igotw/vizplay/ Fri, 22 Jan 2021 00:00:00 +0000 https://thingly.net/igotw/vizplay/ Over the years I’ve played around with various bits of data viz. Sometimes it’s a point solution for a bespoke problem space. Sometimes I want to see if I can compress as much data into as small a space as possible. Sometimes I just plain wanna make “something that looks cool” - viz gratia artis, if you will. Unfortunately, I’ve done a completely awful job of keeping any of it organized. https://thingly.net/igotw/martin-luther-king-jr/ Sat, 16 Jan 2021 00:00:00 +0000 https://thingly.net/igotw/martin-luther-king-jr/ Every year on MLK Day I make it a point to watch or listen to the “I Have a Dream” speech. Sometimes I’ll also listen to one or two of King’s other well- known speeches. At the risk of understatement: the man had a gift. Even some 57 years later I suspect I’ll be rapidly blinking before he’s done speaking. When I told my girls we were going to do this on Monday, I got a less-than-enthusiastic response; at their age I reckon having a day off school has more impact than the prospect of watching an old-timey video of “some guy talking”. https://thingly.net/igotw/pantsdrunk/ Sat, 09 Jan 2021 00:00:00 +0000 https://thingly.net/igotw/pantsdrunk/ Maybe some of you have asked yourself questions about what service do during the year-end moratorium/company shutdown. Left alone to their own devices with nobody poking and prodding them, do they take the time to catch up on their reading lists? Do they reflect on the happenings of the past year and auld lang syne? Perhaps a little Kalsarikännit? Well, this lil beaut’ that Vishnu C Nsent my way might shed some light on the matter: https://thingly.net/igotw/twenty-twenty-one/ Fri, 01 Jan 2021 00:00:00 +0000 https://thingly.net/igotw/twenty-twenty-one/ This year’s NYE celebration was a tame one - ball drop, TV off, in bed by 12:03. Glass-half-full hot take: no hangover when I was burning my fingertips off this morning peeling boiling-hot cabbage leaves for the cabbage rolls. Let’s ease on into this new year. Let’s just sit down and watch a whole bunch of college football. Happy New Year, folks. I’ll see y’all soon! https://thingly.net/igotw/ship-it-or-don-t/ Thu, 24 Dec 2020 00:00:00 +0000 https://thingly.net/igotw/ship-it-or-don-t/ By now I reckon most of y’all have seen the shipping memes floating around the interwebitubes - “It’s in God’s hands now” and suchlike. Welp. I’m here to tell ya, the memes are real. About a month ago my wife ordered us a few books from a local bookstore. Amazon could’ve had ’em to us in a coupla days, but we like to buy local when we can (especially if it’s supporting something as rad as a bookstore). https://thingly.net/igotw/mobaa/ Fri, 18 Dec 2020 00:00:00 +0000 https://thingly.net/igotw/mobaa/ Earlier this week a couple of folks brought a recent post about Google’s Museum of Borgmon Abstract Art to my attention. In the vein of “imitation is the sincerest form of flattery” let’s take a peek at a couple of their graphs, shall we? Their fraying tapestry graph… …made me think of prior posts about load balancing. It’s also a good example of the inverse of the Tighten Up. But the one that really got me going was their “k8s elephant”: https://thingly.net/igotw/gratitude/ Fri, 11 Dec 2020 00:00:00 +0000 https://thingly.net/igotw/gratitude/ Today is Gratitude inDay, so I’ve spent a little time thinking about the things I’m grateful for. The top of that list is pretty standard: Family & Friends - the people who are there for me, who I can depend on, who I try desperately not to take for granted and celebrate whenever I can. Expanding a little way beyond that in the category of People, I’m grateful for anyone who’s ever taken a chance on me, gone to bat on my behalf, given me the benefit of the doubt, or cut me a little slack when I fucked up. https://thingly.net/igotw/black-friday/ Fri, 04 Dec 2020 00:00:00 +0000 https://thingly.net/igotw/black-friday/ 2020-12-04 - I went to write the igotw today and realized that last week’s post was missing. I looked around and apparently Confluence decided to publish it under Ada Ma*’s blog? Bizarre. Anyhow…in case you aren’t on the email list, here it is…* Folks, I gotta say this has easily been the weirdest Thanksgiving in memory. I made a buncha turkey and stuffing and mashed taters and green beans and gravy - that bit was more-or-less the same as usual. https://thingly.net/igotw/glitch-gallery/ Fri, 04 Dec 2020 00:00:00 +0000 https://thingly.net/igotw/glitch-gallery/ https://thingly.net/igotw/satisfaction/ Thu, 19 Nov 2020 00:00:00 +0000 https://thingly.net/igotw/satisfaction/ For the better part of 20 years I’ve built my own PCs. Once every few years I get tired of the machine I’m using - maybe it’s perceivably “slow”, maybe som ething broke, maybe there’s some New Hotness out that I want to be able to try - and I start looking around at components to start a new build. My current machine is approaching the age when I’d typically start doing this…but… https://thingly.net/igotw/unintentional-art-elephant-edition/ Fri, 13 Nov 2020 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-elephant-edition/ Recently _Ben Goldsbury_hit me up with this lil beaut': It very clearly resembles an elephant, and it got me thinking: where have I seen something like this before? Oh, right! A previous UA post that included “The Little Prince”: If that’s not a picture of a snake swallowing an elephant then frankly I don’t know what is. …and then I was scrolling around my inGraph catalog and - lo and behold! https://thingly.net/igotw/red-vs-blue/ Thu, 05 Nov 2020 00:00:00 +0000 https://thingly.net/igotw/red-vs-blue/ I’d wager that most of the folks reading this have spent some amount of time looking at red/blue maps of the United States this week. Allow me to add a few bits of slightly different red & blue to the mix. This is the map generated for me by a dialect quiz that _Tyler Elliott recently sent my way. How it works: you answer a handful of questions about how you _ pronounce this-or-that word, and it spits out a heatmap showing you the regions of the country that you are most/least likely to have come from based on those pronunciations. https://thingly.net/igotw/beggar-s-night/ Thu, 29 Oct 2020 00:00:00 +0000 https://thingly.net/igotw/beggar-s-night/ I grew up with the concept of “Beggar’s Night” so I just kind of assumed it was a Thing everywhere, but I’ve been asking around and nobody seems to know what it is. I thought I might explain. In Columbus, OH kids do not go trick-or-treating on 10/31. Instead, there is a separate date the week prior that is designated for trick-or-treating. It can vary by specific locality/suburb, but in general it avoids the weekends and Halloween itself. https://thingly.net/igotw/the-pen-cup/ Fri, 23 Oct 2020 00:00:00 +0000 https://thingly.net/igotw/the-pen-cup/ I have a sister who is almost exactly ten years younger than me (she was “The Surprise”). In her teen years, she got in trouble for running up $350 in long- distance charges calling a friend of hers in Indiana. Mom wasn’t too pleased about the bill, but she really couldn’t go too hard on her; from the time my sister was old enough to use a phone she’d always had a mobile phone available. https://thingly.net/igotw/sre-in-con-2020/ Fri, 16 Oct 2020 00:00:00 +0000 https://thingly.net/igotw/sre-in-con-2020/ Another successful incon wrapped up yesterday afternoon. The format was decidedly different this year - 100% virtual over Zoom webinar - but other than a couple of minor technology hiccups (and a swag shipping delay) I’d say it went off without a hitch and maintained the essential spirit of incons past. As in previous years, the Fireside Chats stood out to me in particular. Fireside Chat #1 Who: David Henke, former SVP of Engineering/Operations at LinkedIn. https://thingly.net/igotw/quarter-millennial/ Thu, 08 Oct 2020 00:00:00 +0000 https://thingly.net/igotw/quarter-millennial/ This is the 250th inGraph of the Week. To commemorate the “occasion” I thought I might revisit every 50th post, from the beginning….perhaps with a little color commentary (not just a link-dump - I’m a little more loquacious than that). So let’s see what we’re working with here… Post #1: inGraph of the Week - (Not) Learning from Mistakes (1/7/2016) The inaugural igotw is one of my all-time favorites. I’ve referenced it in multiple other posts and it continues to be relevant nearly 5 years later. https://thingly.net/igotw/fun-with-ps1/ Fri, 02 Oct 2020 00:00:00 +0000 https://thingly.net/igotw/fun-with-ps1/ My .bashrc tends to be a slow-moving beast; other than the occasional new tool requiring a FOO_HOME variable it’s pretty much “set-it-and-forget-it”. But once every coupla-few years I get the “itch” to fiddle with it and see what I can improve. Last week was one such time. There are a few things I wanted to get out of my PS1 - the variable that determines what my bash prompt looks like. https://thingly.net/igotw/footprint/ Fri, 25 Sep 2020 00:00:00 +0000 https://thingly.net/igotw/footprint/ A while back someone asked me a question about server footprints. It kinda stuck with me, so last inDay I spent an hour or two fiddling around with d3js treemaps and came up with this: It’s kinda slick in that you can click into a given environment and get a breakdown: …but it gets pretty illegible pretty quickly outside of the top few things installed. Clicking into an individual fabric: https://thingly.net/igotw/on-buttons/ Thu, 17 Sep 2020 00:00:00 +0000 https://thingly.net/igotw/on-buttons/ I was making dinner last week when the microwave button panel caught my eye. It was a bit of a Seth Godin “This Is Broken” moment; to wit: why on Earth are there so many buttons on this thing? I started scrutinizing. So far as I’m concerned this could all be replaced by one big “Add 30 sec” button. (“Add 30 seconds” is the single greatest innovation in microwave technology mankind has ever seen. https://thingly.net/igotw/unintentional-art-september-2020/ Fri, 11 Sep 2020 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-september-2020/ This week I’ve just got a few bits ’n’ bobs that folks sent my way. First up, a nice little escalator to nowhere, courtesy ofNick Brown: _Next up is this great example of the Tighten Up from _Anthony Gargiulo: _Then there’s this cool little oscillator that Ryan Underwood_sent my way: …and last but not least, this praying mantis (with entirely too many legs), pointed out by Brandon Matthews: Happy Friday…y’all stay safe out there. https://thingly.net/igotw/family-night/ Fri, 04 Sep 2020 00:00:00 +0000 https://thingly.net/igotw/family-night/ There’s a thing I do called Family Night. If you know me you’ve probably heard me talk about it and can safely skip the next paragraph or two. Here’s how it works: Every Friday night I make a meal. A big meal. There is a standing open invite for family and friends, and anyone can come on any given Friday. Sometimes it’s just a dozen or so people - my mom and sisters (with husbands and kids) are the usual suspects. https://thingly.net/igotw/scratch-txt/ Fri, 28 Aug 2020 00:00:00 +0000 https://thingly.net/igotw/scratch-txt/ I have a file called scratch.txt that I’ve used as a running scratchpad since I started at LinkedIn. It contains little snippets of logs/stack traces, partially- drafted emails, bits of yaml, shell commands (so much grep!), etc. It’s around 5800 lines, and every now and again I’ll spend a few minutes scrolling around it in a sort of trip down memory lane/archaeology project. For instance: control-deployment -f ela4 rollback-product –product nux –tag nux. https://thingly.net/igotw/stoxart/ Fri, 21 Aug 2020 00:00:00 +0000 https://thingly.net/igotw/stoxart/ I have to be honest, I don’t entirely understand the stonks meme. What I do understand: blocked URL Other than that messy business around the March timeframe it’s nice to see MSFT marching along up-and-to-the-right. Also nice to see: stoxart, wherein Gladys - a “visual insights innovator” - turns data: blocked URL …into art: blocked URL Lovely. Happy inDay, folks! :-) https://thingly.net/igotw/oscillation/ Fri, 14 Aug 2020 00:00:00 +0000 https://thingly.net/igotw/oscillation/ Earlier this week _Brandon Matthews_posted these two beaut’s in #sre: Oh man. I was curious, so I thought I’d do a little digging into the first one and see what I could see. I didn’t (and still don’t) know exactly where these graphs came from, but I took an educated guess that they were graphs of redliner runs based on the legends. $ redliner test show 349246 Product: espresso-router Application: espresso-router Fabric: prod-lor1 Application Version: 2019. https://thingly.net/igotw/the-nineties/ Thu, 06 Aug 2020 00:00:00 +0000 https://thingly.net/igotw/the-nineties/ I stumbled across a thing on the Interwebitubes last week - an attempt to define “the ‘90s music canon” by mining some data to figure out which hit songs from the ‘90s are standing the test of time and which are fading into obscurity. Having been a teenager for most of that decade this had me feelin’ emotions (heh - see what I did there?): Joy: Oh man, I haven’t heard this tune in a minute*! https://thingly.net/igotw/on-refactoring/ Fri, 31 Jul 2020 00:00:00 +0000 https://thingly.net/igotw/on-refactoring/ My wife (Kate) bought a new piece of furniture - just a bit of flat pack kit for additional storage in the living room. After we got the kids to bed I threw in my earbuds and went about knocking it together. Straightforward set of instructions, I know my way around a screwdriver…this should just take an hour or so, right? Then the subject of placement came into play: If we put the new-new in this corner, then that means we’ll have to move the end table to accommodate the doors… https://thingly.net/igotw/what-is-cliff-doing/ Fri, 24 Jul 2020 00:00:00 +0000 https://thingly.net/igotw/what-is-cliff-doing/ I occasionally get asked questions about how I organize my work. For me, one important facet of that is having some idea of where my time is committed. To that end, for several years I’ve maintained a super-simple Google doc that helps me to track time commitments. Here’s a peek at what the table of contents looks like: blocked URL A few things to note: I leverage headers so I can auto-(re)generate a table of contents at the top for an at-a-glance idea of total time (over-)commitment. https://thingly.net/igotw/unintentional-art-july-2020/ Fri, 17 Jul 2020 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-july-2020/ I wanted to find something a little “light” to post today - it’s the week after break, let’s ease into it, shall we? So I was trolling through my inGraphs backlog and was a little surprised to discover that I haven’t saved off anything new since around the April/May timeframe. Not to worry! GCNs are typically a treasure trove of rad inGraphs, right? So I cracked open go/ds3dash and immediately stumbled upon a couple of gems. https://thingly.net/igotw/good-morning/ Fri, 10 Jul 2020 00:00:00 +0000 https://thingly.net/igotw/good-morning/ My sister pointed out how life is a bit like The Truman Show right now. It’s not exactly Groundhog Day - notably, there’s no Ned Ryerson - but there is admittedly a certain kind of quality that might make someone who has seen the movie wonder “Am I on TV right now?” Stuck in a “protective bubble”? Check. Monotony of everyday life? Check. Carefully-constructed set of global circumstances, travel restrictions, and personal phobias that prevent one from going to Fiji? https://thingly.net/igotw/informed-overlay-redux/ Thu, 02 Jul 2020 00:00:00 +0000 https://thingly.net/igotw/informed-overlay-redux/ The loyal reader might recall a post I did a few years back about how to use the Informed Overlay. Now that inGraphs has cut over to inFormed v2 I thought it might be worth revisiting and talking a bit about what’s changed. Let’s dig into it a bit and take a look at a couple of examples that I think might help folks out for common use cases. Let’s start off with what the new overlay UX looks like: https://thingly.net/igotw/social-contract/ Thu, 25 Jun 2020 00:00:00 +0000 https://thingly.net/igotw/social-contract/ I’ve been talking to an old buddy of mine about trying to get together “some time soon” for a little while now. Earlier this week he shot me a text; paraphrasing: “Hey, it’s nice out, how ‘bout we sit on your front porch - an appropriate distance away from one another, natch - and drink some beers?” Compelling. I took him up on it. …and then panic sets in. I mean, he’s gonna be here in a half hour - what if he has to pee? https://thingly.net/igotw/hyrum-s-law/ Fri, 19 Jun 2020 00:00:00 +0000 https://thingly.net/igotw/hyrum-s-law/ This week Ben Weirfloored me with a reference to Hyrum’s Law. As a concept it felt familiar - perhaps in the category of something I “felt” but hadn’t thought too much about - but…man… With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody. This is some deep observational shit, folks. https://thingly.net/igotw/cardboard-castle/ Fri, 12 Jun 2020 00:00:00 +0000 https://thingly.net/igotw/cardboard-castle/ We order a ton of stuff online. One time I was on my front porch when our mailman was making his rounds and he hit me with a little jab - “What, no packages today?” …and that was before COVID-19. Now that we’re ‘Rona Hermits and a casual trip to Target is a decidedly unattractive proposition I’d guesstimate that our online purchasing has doubled. I don’t think we’re actually buying more crap (the occasional retail therapy flare-up notwithstanding), we’re just having more of it shipped as opposed to buying it in a brick-and-mortar. https://thingly.net/igotw/we-re-gonna-make-it/ Thu, 04 Jun 2020 00:00:00 +0000 https://thingly.net/igotw/we-re-gonna-make-it/ I thought about posting nothing this week. Not my style. I thought about posting a black square, ala Insta. I have to admit I decided against it because I don’t exactly know what it means. I thought about posting this whole big long thing about how shitty this week has been. It didn’t seem necessary. If you’re a human being, you already get it. I thought about posting the song This Year by the Mountain Goats. https://thingly.net/igotw/what-do/ Thu, 28 May 2020 00:00:00 +0000 https://thingly.net/igotw/what-do/ I took a walk around my block tonight. By myself, after the kids were in bed. Not a long walk - just the “short block”. I’ve seen this neighborhood in the daytime a jillion times, I’d never quite seen it in this way. It’s a lovely night. Shorts-and-a-t-shirt weather (for a typically jeans-and-a-fleece kinda guy). We’ve had some rain, so greenery is lush. Also, I live in the Midwest so things are mowed, things are trimmed. https://thingly.net/igotw/unintentional-art-may-2020/ Fri, 22 May 2020 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-may-2020/ This week’s post is just a few bits ’n’ bobs from my graph stash. First up, a graph that Luke Young_ put together to visualize deployment convergence of managed SSH configs_: Then there’s this wonderfully-oscillating inGraph of inbound DNS queries from Ryan Underwood. Possible theory: some batch job that spins up at the top of every hour and starts making calls? Next is one that I meant to post ~2 months ago…and completely forgot until today (sorry, Vishnu C N). https://thingly.net/igotw/we-used-to-wait/ Fri, 15 May 2020 00:00:00 +0000 https://thingly.net/igotw/we-used-to-wait/ Last week I made a lamb tagine and some flatbreads for dinner. (Side note: Remind me to tell y’all about Family Night some time.) The flatbreads fit with the cuisine - and it’s a good thing, since the ‘Rona-bakers have made sure to snatch up every last packet of yeast in my local grocery store for weeks now. …but I digress. I’ve spent a fair chunk of time working with dough. https://thingly.net/igotw/the-covids/ Thu, 07 May 2020 00:00:00 +0000 https://thingly.net/igotw/the-covids/ I’ve been avoiding posting about COVID-19 - I thought maybe igotw could be about Something Else. As it turns out, posting about anything else this week is simply impossible. And it’s my wife’s fault. Let me explain. Kate and I played (virtual) cards with some friends tonight. Afterward, she and I talked a bit about what this week’s igotw might be about. I jotted down some notes, and the shortlist looks something like: https://thingly.net/igotw/cartography/ Fri, 01 May 2020 00:00:00 +0000 https://thingly.net/igotw/cartography/ I recently stumbled across Terrible Maps on Facebook. I have some interest in data viz and cartography is one of its oldest forms, so finding this led me down a Sunday afternoon rabbithole poking through photos. Some of them are, in fact, terrible (as advertised). Some are actually pretty interesting. On the whole, though, I’d say the overall flavor of the page is “mild amusement”. For example, this heatmap of the usage of the word “dang” across the United States: https://thingly.net/igotw/olympus-draft/ Fri, 24 Apr 2020 00:00:00 +0000 https://thingly.net/igotw/olympus-draft/ I’ve been working on a collaborative series of Eng Blog posts with Manny Lavery Armen Hamstra*,* , and Tyler Grant on the new Messaging architecture (Olympus). In the absence of infinite time to write blog posts, I thought I’d “cheat” a bit and drop the (very raw) first draft of my contribution here. So…here it is. When I started at LinkedIn (September 2013) Messaging was a very different product than the one we know today. https://thingly.net/igotw/perspective/ Fri, 17 Apr 2020 00:00:00 +0000 https://thingly.net/igotw/perspective/ In a previous life I worked for a small non-profit tech company that provided services to libraries around the world. I loved the vision & mission of the company, but from a technology standpoint the company was (and likely still is) decidedly less-mature than LinkedIn. A few examples: Deployments Notify all customers that there would be a “maintenance window” on Sunday from 02:00 - 06:00 Eastern time. On Sunday at 02:30, shut down the entire site. https://thingly.net/igotw/synchronicity/ Fri, 10 Apr 2020 00:00:00 +0000 https://thingly.net/igotw/synchronicity/ I’ve been full-time remote for a few years now. A conversation I’ve had in a handful of different forums over the past several weeks has been along the lines of “So this must just be pretty much life-as-usual for you, right?” Well…sort of. In terms of how and where I do my work day-to-day? Sure. I’ve had 3+ years to adjust to working out of my home office. I’ve already got a decent chair, I’ve spent some time getting my A/V kit set up, I already grok that every meeting will be over VC and I’m not going to be having lunch with colleagues tomorrow (for any given tomorrow). https://thingly.net/igotw/big-numbers/ Fri, 03 Apr 2020 00:00:00 +0000 https://thingly.net/igotw/big-numbers/ Richard Waid recently pointed out a new inGraphs graph type to me: Big Numbers. Here’s how you get to it: …and here’s what it looks like: This is the Big Number view of main partition requests to the site (https://ingraphs.prod.linkedin.com/dashboard/stickyrouting/graph/main%20partition% 20requests%20per%20fabric?fabric_groups=prod), and the numbers being displayed are the “Last” value - the most recent available data point. How might this be useful? Well, suppose for that same graph you wanted to compute the total number of queries in that time period. https://thingly.net/igotw/falsehoods/ Fri, 27 Mar 2020 00:00:00 +0000 https://thingly.net/igotw/falsehoods/ Way back in 2010, a guy named Patrick wrote a post about Falsehoods Programmers Believe About Names. I believe it may have been the first of its kind, and led to many subsequent authors coming up with similar “Falsehoods…” on all different kinds of topics. Somewhere along the way, Awesome Falsehood was created - *"*A curated list of falsehoods programmers believe in." Per the site: “…Falsehood articles are a suite of wordy unit-tests covering extensive edge-cases provided by real-world usage. https://thingly.net/igotw/unintentional-art-first-day-of-spring-edition/ Fri, 20 Mar 2020 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-first-day-of-spring-edition/ Edition) Thank you to the folks who contributed to last week’s experiment On Toil (go/ontoil)! I’m letting it percolate for a bit for now; I’m going to try and tease out some themes and pull it into something coherent Soon. :-) Just a few bits ’n’ bobs from my backlog of ingraphs. _Here’s a nifty one that Minhaz Mahmud_pointed out. If memory serves I believe it’s demonstrating the Median feature? …but mostly I just like its unique aesthetic: https://thingly.net/igotw/on-toil/ Fri, 13 Mar 2020 00:00:00 +0000 https://thingly.net/igotw/on-toil/ [Ed: Ugh. I accidentally created this under “Page” instead of “Blog”. I’ll leave both around so as to not have any dangling pointers.] I’ve given a fair bit of brainspace to this idea of “toil” lately - trying to wrap arms around it, trying to figure out what it means and how to make it better. I have thoughts and opinions and ideas…and ultimately no direct conclusions, but something that I keep coming back to is that the very term “toil” has become detrimental to trying to solve for it. https://thingly.net/igotw/on-isolation/ Thu, 05 Mar 2020 00:00:00 +0000 https://thingly.net/igotw/on-isolation/ If you haven’t seen it yet go check your inbox for an email from Nina McQueen, Subject: “Important coronavirus update on working from home in the Bay Area” (don’t worry, I’ll wait). I suspect a lot of folks might have an initial reaction of “Woohoo! No more commute, and I don’t even have to put on pants if I don’t wanna!” Some may even be thinking along the lines of today’s well-timed xkcd. https://thingly.net/igotw/new-features/ Fri, 28 Feb 2020 00:00:00 +0000 https://thingly.net/igotw/new-features/ ICYMI Monitoring Infra sent a newsletter earlier this week. I’d recommend taking a peek at it if you haven’t already - don’t worry, I’ll wait - but I thought I’d mention a couple of the new features they announced here. Consider the following inGraph: blocked URL Okay, so kafka partitions sizes marching onward up-and-to-the-right, pretty straightforward stuff. …but what if you wanted to know the rate of growth? I suppose you could pick a couple of values at time t0 and t1 and then do a little arithmetic, but what if you wanted to see it? https://thingly.net/igotw/pep-talk/ Fri, 21 Feb 2020 00:00:00 +0000 https://thingly.net/igotw/pep-talk/ A while back I noticed flake8 spitting out warnings about W504 while building a multiproduct: Line breaks should occur before the binary operator to keep all operators aligned. I found the offending lines, PEP-ified them accordingly - one small contribution to hygiene, a minor reduction in kipple - and I kicked off another build. BUILD FAILED Oops. I must’ve messed something up - a stray character, or maybe I didn’t quite get the whitespace right. https://thingly.net/igotw/ship-it/ Fri, 14 Feb 2020 00:00:00 +0000 https://thingly.net/igotw/ship-it/ I recently read Tina Fey’s Bossypants. Not only is it deeply funny - Fey is one of the funniest writers alive - but there are also a few nuggets of wisdom in there, bits of advice for aspiring comedians and writers. One of them is something that Lorne Michaels said about Saturday Night Live. I’m probably butchering the exact quote, but it was something like “The show (SNL) doesn’t go on because it’s ‘ready’, it goes on because it’s 11:30. https://thingly.net/igotw/end-of-an-era/ Thu, 06 Feb 2020 00:00:00 +0000 https://thingly.net/igotw/end-of-an-era/ Some time in my first year at LinkedIn something happened behind 2029 on the old Stierlin Court campus. I was behind the building and J-Dubs was walking in. He saw a piece of trash on the ground - a plastic cup or a crumpled fast food bag or whatever it was. He bent down, picked it up, and carried it to the trash can on his way to the door. https://thingly.net/igotw/o-que-e-tinder/ Thu, 30 Jan 2020 00:00:00 +0000 https://thingly.net/igotw/o-que-e-tinder/ Around the time I started at LinkedIn (September 2013) Conan O’Brien decided that he was going to conquer LinkedIn. This resulted in some consternation on our end as metrics like cold signups and connection requests suddenly showed unexpected spikes. I couldn’t find a GCN for this event (although I did find an old ticket for Conan’s subsequent first post as an Influencer) and I don’t have any inGraphs for it, but I reckon the metrics probably looked a little like the following: https://thingly.net/igotw/rolling-reboots-and-the-last-tranche/ Thu, 23 Jan 2020 00:00:00 +0000 https://thingly.net/igotw/rolling-reboots-and-the-last-tranche/ [The following is a guest post written by Brandon Matthews link to his original (). It’s a little longer (and decidedly more math-y) than the typical igotw, but I think he presents some findings that are super-interesting. Thanks, Brandon!] The other day, I happened across the following graph: This is a graph of QPS per host during a rolling restart. That’s a really interesting load pattern, isn’t it? A lot of traffic gets shunted into just a few nodes! https://thingly.net/igotw/on-kipple/ Fri, 17 Jan 2020 00:00:00 +0000 https://thingly.net/igotw/on-kipple/ Once Upon a Time I spent 4 nights a week making pasta and all day Saturday selling it. I took cleanliness seriously. First and foremost: I was making food for people to eat and I wasn’t trying to get anyone sick. …but also, making pasta by hand is a decidedly messy business and I was doing it in my home kitchen. The floors were a problem - seriously, flour gets everywhere - but the counter was my focus since that’s where the actual food touched. https://thingly.net/igotw/data-is-beautiful/ Fri, 10 Jan 2020 00:00:00 +0000 https://thingly.net/igotw/data-is-beautiful/ Every now and then I’ll poke around in the /r/dataisbeautiful subreddit. There’s a fair bit of chartjunk and I’m not sure I’d necessarily trust any of the data being represented, but there are some real gems in there. From the beautiful: blocked URL …to the mildly interesting: blocked URL …to the just plain fun: blocked URL If you’re into data viz (or even just wanna look at the pretty pictures/animations) you should check it out! https://thingly.net/igotw/birthday-words/ Fri, 03 Jan 2020 00:00:00 +0000 https://thingly.net/igotw/birthday-words/ I have a birthday coming up and it got me thinking about a post I saw a while back about Merriam-Webster’s “Time Traveller”. The site lets you pick a year and take a peek at the (English) words that were first used in print in that year. Pretty nifty! A few gems from my birth year include: boom box, camcorder, LAN, snowboard, and sysop. I like to compare that to much more-recent words - like blockchain (2011), or 2014’s manspreading. https://thingly.net/igotw/merry-christmas-happy-new-year/ Fri, 27 Dec 2019 00:00:00 +0000 https://thingly.net/igotw/merry-christmas-happy-new-year/ I know I’ve posted this one before, but it’s been a while so I don’t feel too bad about recycling it: Apropos, don’t you think? Happy Holidays, folks. https://thingly.net/igotw/holiday-trough/ Thu, 19 Dec 2019 00:00:00 +0000 https://thingly.net/igotw/holiday-trough/ It’s been a while since I originally posted about the trough pattern. There was a GCN earlier this month that got me thinking about it again. blocked URL That’s one helluva trough. For those not quite sure what they’re looking at: that’s our site being hard down (globally) for ~45 minutes or so. Oof. I’m not actually going to talk about this GCN. (You can go read about it if you like. https://thingly.net/igotw/go-percent/ Fri, 13 Dec 2019 00:00:00 +0000 https://thingly.net/igotw/go-percent/ This month’s inDay theme is Reflection - makes sense for December, I reckon - and I’ve been mentally looking back over the 6+ years I’ve spent at LinkedIn. Some of that thinking is about things I’ve done - things I’ve worked on, things I’ve built, things I’ve accomplished, etc. - but mostly it’s about people. The folks I’ve had the opportunity to work with. The ones I’ve been on the interview panel for (I maintain a mental toplist of “Proudest Hires”). https://thingly.net/igotw/absolute-time/ Fri, 06 Dec 2019 00:00:00 +0000 https://thingly.net/igotw/absolute-time/ It’s been a while since the last time I posted about an inGraphs feature, but I recently discovered a nifty bit of (new?) hotness that I thought I’d share. Suppose you’re looking at an inGraph: You happen to think it’s pretty interesting and you’d like to share it, so you click the button in the upper righthand corner to generate a snapshot: At this point inGraphs presents you with a dialog that looks like this: https://thingly.net/igotw/happy-thanksgiving-2019/ Wed, 27 Nov 2019 00:00:00 +0000 https://thingly.net/igotw/happy-thanksgiving-2019/ This week I’ve got a graph that _Patrick Fairbank_posted recently in some slack channel somewhere. It looks like a shark fin: What do sharks have to do with Thanksgiving? Not a damn thing. …or do they? I suspect a lot of folks have already seen Jessica’s Daily Affirmation - it’s an “oldie but a goodie” - and if you haven’t it’s worth a watch (and under a minute long). I reckon Jessica sound pretty thankful for a whole lot of things…and she just so happens to start her process with “Look, I can be a shark! https://thingly.net/igotw/give-back/ Fri, 22 Nov 2019 00:00:00 +0000 https://thingly.net/igotw/give-back/ My buddy’s dad (Dan) is a high school guidance counselor at my alma mater. He reached out to me to see whether I’d be interested in participating in a career workshop that he’s conducting with a handful of students. It sounded like a super-interesting opportunity, so I said “Yes.” …and then I spent a week fretting about it. Someone once told me that overthinking things was my “superpower”, and this is a prime example of that. https://thingly.net/igotw/unintentional-art-family-inday/ Fri, 15 Nov 2019 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-family-inday/ This morning I was thinking about Family inDay last year, which also happened to be Bring In Your Parents Day (TIL: it got moved from November to May this time ‘round). My mom participated last year and I had her pick out the graphs for that week’s igotw. Something I didn’t mention in that post: my youngest sister was 9 months pregnant at the time. I think my sister was a little bit salty that I flew her mother all the way to the other side of the country that close to her due date. https://thingly.net/igotw/11-foot-8-the-final-chapter/ Fri, 08 Nov 2019 00:00:00 +0000 https://thingly.net/igotw/11-foot-8-the-final-chapter/ Previously: 11 Foot 8 (Part 1) 11 Foot 8 (Part 2) Leejay Wu_ and Nick Brown_ first drew my attention to this tweet a few weeks ago. It’s happening, folks…they’re raising the bar. (heh See what I did there?) There are a bunch of different takes from the Internet Peanut Gallery on this decision to correct a decades-long bit of physical infra tech debt, the most common of which is of the “sadness”/”end of an era” variety. https://thingly.net/igotw/sres-are-people-too/ Fri, 01 Nov 2019 00:00:00 +0000 https://thingly.net/igotw/sres-are-people-too/ I had a conversation at Sports Page a while back with one of the SWEs I supported and a new hire that he’d he’d taken under his wing. Old Hand to FNG “This is Cliff. He’s an SRE. SREs are the Most Powerful!”: FNG (eyes wide as saucers) “Ohhhhhhhh!”: It was kind of funny at the time - a little embarrassing, and admittedly a little flattering. I’ve thought about it quite a bit since then - about this idea that Engineers in general (and SREs, in particular) are some kind of techno-wizards with Special Powers. https://thingly.net/igotw/keepin-it-200/ Thu, 24 Oct 2019 00:00:00 +0000 https://thingly.net/igotw/keepin-it-200/ Something I hear from time to time: “you don’t post graphs any more”. Well, here’s one: blocked URL It’s a graph of cumulative igotws over time. I posted a graph something like the above a while back, but SRE[in]con was this week and I’m writing this amongst a pile of folks hanging around at the tail end of the conference so I’m not going to track down the link. This is the 200th igotw post. https://thingly.net/igotw/on-mentorship/ Thu, 17 Oct 2019 00:00:00 +0000 https://thingly.net/igotw/on-mentorship/ I’ve mentored - and been mentored by - a handful of people. I don’t claim to be an expert on the subject. In fact, I’m not sure “mentorship experts” exist outside of the mental health profession. That said, I’ve walked a little way down this path so I thought I might share a few ideas. Side note: I think that traditional 1-on-1 mentorship has value, but I’m increasingly drawn to something a little more distributed - something more like a support network - a personal “Board of Advisors” if you will. https://thingly.net/igotw/11-foot-8-part-2/ Fri, 11 Oct 2019 00:00:00 +0000 https://thingly.net/igotw/11-foot-8-part-2/ A while back I posted some thoughts based on the 11foot8 FAQ_. Today I thought I might swing back around to that. For those who didn’t read the original _ post, 11foot8.com is a website that posts footage of trucks wrecking into a low bridge in Durham, NC. Since I wrote that (April 2019) there have been 4 more posts on the site, for a total of 7 crashes tagged in 2019. https://thingly.net/igotw/thundering-barracudas/ Thu, 03 Oct 2019 00:00:00 +0000 https://thingly.net/igotw/thundering-barracudas/ The people who write computer programs are typically not good at making those programs write sensible logs. I say this with great fondness for that set of folks, and in fact as someone who makes a living writing and understanding code I am among them. I could probably do an entire weekly blog about logs, but for now I’m just going to give some of the “flavor”. I can remember a time in a previous life when I was oncall for a Thing and I would get phone calls in the middle of the night when that Thing broke. https://thingly.net/igotw/the-making-of/ Thu, 26 Sep 2019 00:00:00 +0000 https://thingly.net/igotw/the-making-of/ I thought I might talk a bit about the “creative process” around the experiment I tried a couple of weeks ago with audio recording. Let’s see if I can paint that picture. It was late and I was restless. There’s a line in a Barenaked Ladies song about “just thinkin’ bout what to think about”…and this was that. I’d been kicking around the idea of producing an igotw podcast for a while so I figured mebbe this was the time to take a crack at it. https://thingly.net/igotw/unintentional-art-september-2019/ Tue, 24 Sep 2019 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-september-2019/ Hoo boy, it’s been A Week. I had a few ideas for posts, but ultimately I decided to fall back on the old, reliable UA format. Let’s get through this inDay and into the weekend, shall we? This first one caught my eye because it looks quite a bit like something I posted in the past: Perhaps Ze Germans just really like Groups, mmm? The rest are various takes on a rainbow theme. https://thingly.net/igotw/podcast-edition/ Thu, 12 Sep 2019 00:00:00 +0000 https://thingly.net/igotw/podcast-edition/ I’ve wanted to try this out for a while now. …so here it is… https://thingly.net/igotw/yahoo/ Thu, 05 Sep 2019 00:00:00 +0000 https://thingly.net/igotw/yahoo/ I saw some chatter in slack this morning about folks not being able to resolve DNS for www.yahoo.com, so I started doing a little poking around. My first visit was to Downdetector, which looked a little like this: Okay, so people on the Internet noticed it. …but did we (meaning LinkedIn) notice it? There may be other places where we leverage Yahoo APIs, but the one that jumped most immediately to mind was contact imports. https://thingly.net/igotw/the-monty-hall-problem/ Thu, 29 Aug 2019 00:00:00 +0000 https://thingly.net/igotw/the-monty-hall-problem/ About once every year or two I serendipitously stumble across the Monty Hall Problem, stated thus: A game show host presents you with three doors. Behind one of those doors is a brand new car. Behind the other two doors are goats. The host asks you to select a door. After you’ve made your selection, the host opens one of the other two doors to reveal a goat. The host then asks you whether you want to keep the door you originally chose or whether you would like to switch to the other unopened door. https://thingly.net/igotw/chart-xkcd/ Thu, 22 Aug 2019 00:00:00 +0000 https://thingly.net/igotw/chart-xkcd/ I stumbled across chart.xkcd earlier this week - a nifty little library that draws charts in a “hand-drawn” style, ala xkcd. Initial thought: “Oh, cool!” … immediately followed by “Welp. There goes two hours of my life.” The first one I decided to tackle was the line chart. A bit of fiddling around on Yahoo Finance, some massaging to make the x-axis labels non-overlapping, et voila! Next up I thought I’d try using the donut chart to show the distribution of titles in Bruno Connelly’s org. https://thingly.net/igotw/nines/ Fri, 16 Aug 2019 00:00:00 +0000 https://thingly.net/igotw/nines/ Uber Pool recently changed. By “recently” I mean “within the last 6 months or so.” When you get an Uber Pool you now have the option to “walk a short distance to meet your driver” and “walk a short distance to your destination”. Presumably this eases Uber’s driver routing algorithms and they knock a coupla bucks off the price of the ride, so it’s a win-win, right? Well… The pickup bit wasn’t so bad. https://thingly.net/igotw/unintentional-art-august-2019/ Fri, 09 Aug 2019 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-august-2019/ _This week I’ve just got a few bits and bobs from here and there. First up, an example of The Flatline _: Next up, a wonderfully-jumbled Raptor dashboard that - in the abstract - kinda looks like a series of constellations of…hmmm…a horse? Mebbe a guy riding a horse? There’s definitely a horse in there. Then there’s this cool prism rainbow from Alexsandra McMahan: …and this super high-energy jobbie that Mark Feinstein sent my way: https://thingly.net/igotw/memory-lane/ Thu, 01 Aug 2019 00:00:00 +0000 https://thingly.net/igotw/memory-lane/ A little while back Someone in Some Slack Channel Somewhere posted this little gem. For those too lazy to click it’s a “Disaster Recovery Runbook” circa 2012 - well before multi-master services and trafficshift failouts and all of that goodness that we more or less take for granted today. A few specific callouts: There are 11 manual tasks with multiple sub-tasks, all taking between 30 and 90 minutes. I love how well-documented it all is…but… https://thingly.net/igotw/c-for-k/ Fri, 26 Jul 2019 00:00:00 +0000 https://thingly.net/igotw/c-for-k/ Saturday Morning Breakfast Cereal is one of my favorite webcomics. In a recent comic the author talks about why the Welsh language doesn’t use the letter ‘k’. I actually laughed out loud at this one, because it’s a great example of what I’ve referred to as “Getting Sebenicked” in the past. What does it mean? Well, the general idea is that some Senior Guy makes a decision about technology/policy/etc. It’s a reasonable decision - possibly even a good decision - for the state of the world at the time. https://thingly.net/igotw/this-one-goes-to-37/ Thu, 18 Jul 2019 00:00:00 +0000 https://thingly.net/igotw/this-one-goes-to-37/ Earlier this week Someone in Some Channel Somewhere mentioned having a (bad) habit of iterating on product features based on how the software has already been designed/built, as opposed to developing what the customer needs. In a bit of Baader-Meinhof, not 2 weeks ago I was describing this phenomenon to a buddy of mine (who is not in tech) using the analogy of refrigerator design. I’ve captured it here for your amusement: https://thingly.net/igotw/consistency/ Fri, 12 Jul 2019 00:00:00 +0000 https://thingly.net/igotw/consistency/ There’s a blog post I stumbled across a little while back that resonated with me. The Grand Canyon is a sort of classic example of this. J-Dubs has tweeted about it in the form of Trust, which is a pretty powerful form of Greatness. …but mostly, it reminded me of one of my high school English teachers. Mr. Kerwin was one of the best teachers I ever had. One time he talked about love. https://thingly.net/igotw/independence-day/ Thu, 04 Jul 2019 00:00:00 +0000 https://thingly.net/igotw/independence-day/ I spent some time last night writing, got through about 3/4 of a post before going to bed…and ultimately decided to save it for next week and just ship this, instead: _Chris Carini sent this one my way (and IIRC _Josiah Bradley sent it to him). Looks a bit like flags fluttering in the breeze, no? Happy Fourth, folks! https://thingly.net/igotw/walking-on-water/ Fri, 28 Jun 2019 00:00:00 +0000 https://thingly.net/igotw/walking-on-water/ I want to talk a little bit about how a recent GCN came to be. This GCN is relatively unique in that it caused zero impact to Members. It was bumped to Major due to the potential impact. I’ll cover that a bit below, but the details of the incident itself are actually less interesting than some of the context around how it arose. Let’s dig in. The story begins with the Voyager dev oncall trying to get a release out. https://thingly.net/igotw/data-viz-clarity-indc/ Thu, 20 Jun 2019 00:00:00 +0000 https://thingly.net/igotw/data-viz-clarity-indc/ This week I thought about doing a post on a recent Major GCN of Greatness. I mean, look at the thing! I count no fewer than 75 inGraphs links in there. … but ultimately I wasn’t feeling a “Things got broke and then they got better” story, so I decided to share a couple of internal examples of different data visualization tools that folks might be interest in. First up is Clarity: https://thingly.net/igotw/ups-n-downs/ Thu, 13 Jun 2019 00:00:00 +0000 https://thingly.net/igotw/ups-n-downs/ A common pattern for migrating from one data source/platform/endpoint to another in a non-disruptive way is to do dual reads. The concept is pretty basic: you make calls in parallel to both the “old” and the “new” endpoint. For some period of time you rely on the result from the old endpoint and discard the result from the new one, and then you have some mechanism (typically - but not always - a lix) for reversing that logic and considering the new endpoint to be the source of truth. https://thingly.net/igotw/a-modest-proposal/ Thu, 06 Jun 2019 00:00:00 +0000 https://thingly.net/igotw/a-modest-proposal/ I read an article a while back about a video game studio that had taken a novel approach to protecting their product against piracy. I don’t recall the studio or the specific game, but the approach stuck with me. The game was a first-person shooter, and it had a mechanism whereby it could detect whether the copy being played had been legitimately purchased. If the game suspected foul play - that is, decided that it had been pirated - its first step was to take no action. https://thingly.net/igotw/infrastructure/ Fri, 31 May 2019 00:00:00 +0000 https://thingly.net/igotw/infrastructure/ Once every few years I get the itch to build a new computer. I don’t build with bleeding edge (read: crazy-expensive) parts, but I don’t cut corners, either. I do my research to find the knee in the price-performance graph, and that sweet spot is where I spend my money. I went through this hardware refresh exercise a couple of years ago, and the most expensive single component ended up being - no surprise - the graphics card. https://thingly.net/igotw/unintentional-art-may-2019/ Thu, 23 May 2019 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-may-2019/ I reckon it’s been a little while since I posted any kind of an inGraph. Let’s take a peek at a few UA bits ’n’ bobs, shall we? First up is a nice little stacked graph that, as Josh Briefman pointed out, looks a bit like a city skyline: The thing I like about this one is that it’s effectively a pseudo-random skyline generator; if you fiddle with the timelines, you can generate a whole multitude of different skylines. https://thingly.net/igotw/radon/ Fri, 17 May 2019 00:00:00 +0000 https://thingly.net/igotw/radon/ My wife and I moved a couple of years ago. We found the house we wanted, made an offer in November, and closed in late December. Due to timing and circumstances we weren’t actually going to be able to move in until February. The way it was going to work: my wife would fly out and stay with the kids at her mom’s place while we waited on all of our furniture to make its way across the country, I would make a business trip to Redmond and then fly out to meet them about a week later. https://thingly.net/igotw/chartjunk/ Thu, 09 May 2019 00:00:00 +0000 https://thingly.net/igotw/chartjunk/ If there’s one person who I consider to be an expert on data visualization, it would be Edward Tufte. I’m pretty sure I’ve referenced him here before, but for those who haven’t heard of him Tufte quite literally wrote the book on The Visual Display of Quatitative Information. In this book he coins the term chartjunk to refer to visualizations with unnecessary elements of decoration that add zero value; i.e., ornamentation that conveys no additional information. https://thingly.net/igotw/customer-service/ Thu, 02 May 2019 00:00:00 +0000 https://thingly.net/igotw/customer-service/ FWIW - I was going to post about something completely different today, but a “conversation about a conversation” reminded me about a particular thing that I couldn’t get out of my head (and that I thought was worth talking about a bit). Many thanks to Xiao Li*, and to* Ben Goldsbury for bringing it back up in my mind… I travel pretty frequently on business. I’m no globetrotter - I’m not jetsetting around Europe on a conference circuit or anything like that - but I’ve spent my fair share of nights in hotels. https://thingly.net/igotw/reminiscence/ Thu, 25 Apr 2019 00:00:00 +0000 https://thingly.net/igotw/reminiscence/ When I first started at LinkedIn (waaaay back in the Fall of ‘13) I was a member of a team called CNC. (No, not that one _CNC.) was Content & Community, _ _and the SRE team was led by The Venerable Tony Quan. We owned a pile of consumer-facing products - things like Homepage (the sorta-kinda rough _ equivalent of/predecessor to what we now know as Voyager), NUS/USCP (Feed), Anet (Groups), and Comm. https://thingly.net/igotw/holy-week-2019/ Fri, 19 Apr 2019 00:00:00 +0000 https://thingly.net/igotw/holy-week-2019/ I woke up this morning wondering to myself “What might site traffic look like in predominantly-Catholic countries - the ones whose demography is composed of a plurality (if not an outright majority) of Catholic folks?” Well, as it turns out, there’s an inGraph for that: This is roughly what I expected; site traffic starting off inline week-over-week and starting to diverge going into Easter Sunday. The keen eye might pick out a sharper-than-usual dip in the evening on 4/15 (times are in UTC). https://thingly.net/igotw/11-foot-8-part-1/ Thu, 11 Apr 2019 00:00:00 +0000 https://thingly.net/igotw/11-foot-8-part-1/ I’ve stumbled across http://11foot8.com multiple times in the past. For the uninitiated: there is a low-clearance train trestle in Durham, NC with a traffic cam that captures footage of trucks - specifically, trucks with trailers taller than 11’ 8" - ignoring all warning signs and wrecking into the bridge, typically in spectacular fashion. The videos are all pretty similar. Truck meets bridge. Truck falls in love with bridge. Truck gets its top ripped off and blocks the intersection for hours while folks try to un-fuck the situation. https://thingly.net/igotw/on-writer-s-block/ Thu, 04 Apr 2019 00:00:00 +0000 https://thingly.net/igotw/on-writer-s-block/ I’ve written plenty. igotw has marched onward once a week for over 3 years. 170 posts by my count (this is the 171st)…and man, this week I’m having a tough time figuring out what to post. It’s not that I don’t have inGraphs - I’ve got a bountiful backlog of screenshots and snapshots. It’s not that there’s nothing interesting going on - I assure you that there are plenty of interesting things happening, things that I might sort of “report on”. https://thingly.net/igotw/unintentional-art-srecon19-edition/ Fri, 29 Mar 2019 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-srecon19-edition/ I just got back home from Brooklyn where I was attending SREcon19 Americas. The conference was a good one, but I’m glad to be back home…and I’m als o glad to be heading into the weekend and unlikely to hear “observability” or “SLO” for the next few days. Anyhow, I’ve got a few bits ’n’ bobs of UA that I’ve picked up along the way. _First up are a couple on inGraphs Dmitrii Sutiagin sent my way. https://thingly.net/igotw/invisible-infra/ Fri, 22 Mar 2019 00:00:00 +0000 https://thingly.net/igotw/invisible-infra/ A recent GCN got me thinking about invisible infrastructure - the quiet magic that we take for granted…until it ceases to exist. _PIE is still trying to figure out _ exactly what happened, but my understanding is that a restart of named - which is typically a routine operation with zero impact - went…well…went all atypical on us. DNS resolution is such a fundamental underpinning technology that when it goes away - even briefly! https://thingly.net/igotw/unintentional-art-with-eagles/ Wed, 20 Mar 2019 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-with-eagles/ https://thingly.net/igotw/a-vignette/ Thu, 14 Mar 2019 00:00:00 +0000 https://thingly.net/igotw/a-vignette/ There was a little Mexican market down the street and around the corner from the apartment where I used to live. I used to walk down there every now and then to pick up a few things - a gallon of milk, a few onions, some beer. They also served hot food, but the grill kept odd hours so I almost never ate there (more on that in a bit). https://thingly.net/igotw/postmortem-action-items/ Thu, 28 Feb 2019 00:00:00 +0000 https://thingly.net/igotw/postmortem-action-items/ Fair warning: this is going to be a bit of a boring post. No pretty pictures, no fun story…just a few things I’ve learned over the course of some several dozen postmortems that I thought might be worth sharing. Some of what I’m about to say might seem super-obvious - particularly for those who have participated in postmortems - but I’m explicitly trying not to take too many things for granted for the sake of the uninitiated. https://thingly.net/igotw/lemons-lemonade/ Thu, 21 Feb 2019 00:00:00 +0000 https://thingly.net/igotw/lemons-lemonade/ One thing about me that does not appear on my LinkedIn profile: in a previous life I was co-owner of a pasta business. I don’t remember how the conversation got started - I’m assuming can pretty much guarantee it was over beers - but the tl;dr is that a good buddy of mine (JoJo) talked me into rushing off to become an Entrepreneur. We would seek our fortune making and selling handmade pasta at a local farmer’s market on Saturday mornings. https://thingly.net/igotw/happy-valentine-s-day/ Thu, 14 Feb 2019 00:00:00 +0000 https://thingly.net/igotw/happy-valentine-s-day/ I spent some time poking around my inGraph backlog this morning trying to find a heart-shaped graph. Sadly, I could not find one…so I turned to The Google. After getting momentarily sidetracked listening to the first suggestion for “heart-shaped”, I found what I was looking for: I found this remarkable. I love how it progresses from the simplest possible equation that produces something heart-like (even if it does also look a little like a butt) and moves through more complex alternatives, each giving a new shape - a different degree of “heartliness”. https://thingly.net/igotw/wins/ Wed, 06 Feb 2019 00:00:00 +0000 https://thingly.net/igotw/wins/ This week I am totally ripping off an email that Nimesh Chakravarthi sent out to his team about an operational win. I mean, just shameless plagiarism. So… here it is in its mostly-unadulterated form (with minor edits for formatting and to include attribution). I wanted to send out a report on a change that Lenny Dong made and also demoed last Friday because we have seen some outstanding results from it. https://thingly.net/igotw/maps/ Fri, 01 Feb 2019 00:00:00 +0000 https://thingly.net/igotw/maps/ I’d like to showcase a couple of interesting maps I came across this week. One of them - inVisualize - leverages inGraphs data to map out relationships between services for different parts of the site. Here’s an example of what inVisualize looked like during a recent GCN: Huh. That’s…somethin’. I have to admit that I haven’t spent enough quality time with inVisualize to really grok what it’s trying to tell me. https://thingly.net/igotw/the-holy-hand-grenade/ Fri, 25 Jan 2019 00:00:00 +0000 https://thingly.net/igotw/the-holy-hand-grenade/ My oldest daughter is in first grade. The school she goes to provides free breakfast. Milk & juice, fresh fruit, yogurt, cereal, etc. served up buffet-style. The kids come in, put what they want to eat in a little cardboard tray, grab a seat at one of the tables, and eat. (Oh, and socialize and shout and spill and lose one of their gloves and…in short, all the other things kids do in such a setting. https://thingly.net/igotw/unintentional-art-january-2018/ Thu, 17 Jan 2019 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-january-2018/ Today we’ll take a peek at a handful of inGraphs from the backlog that caught my eye. This first one reminded me of an AI class I took once upon a time. There was an exercise in searching for a global maximum given a landscape (imagine programming a robot to do this), and I remember a picture super-similar to this one demonstrating local vs. global minima/maxima: This next one could be used in a toothpaste commercial. https://thingly.net/igotw/consolidate-histogram/ Thu, 10 Jan 2019 00:00:00 +0000 https://thingly.net/igotw/consolidate-histogram/ This week I’d like to talk a little bit about an inGraphs consolidation that is used relatively infrequently: Histogram. Backing up a step: inGraphs provides a handful of different ways to consolidate data in meaningful/helpful ways. Consolidation can either be defined in the dashboard or manipulated using the inGraphs UI; see: Most of these consolidation options are pretty straightforward. Aggregate (sum each series), AggregateAll (sum the sums), Max (the maxima of each series), etc. https://thingly.net/igotw/omnibot/ Thu, 03 Jan 2019 00:00:00 +0000 https://thingly.net/igotw/omnibot/ After spending a fair bit of time triaging my ruined post-break inbox yesterday my thoughts turned toward igotw. I’ve got a bunch of stuff saved off, but what _should I post? …and then Michael Kehoe came along like a _gleaming Australian superhero and saved the day! He shot me a graph from an inGraphs dashboard for omnibot - the New Hotness slackbot that prod-sre has been working on. Before showing you the inGraphs, I want to issue a disclaimer: omnibot is beta status. https://thingly.net/igotw/happy-holidays-2018/ Fri, 28 Dec 2018 00:00:00 +0000 https://thingly.net/igotw/happy-holidays-2018/ https://thingly.net/igotw/unintentional-art-dto-edition/ Thu, 20 Dec 2018 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-dto-edition/ Last year I took DTO the week before the company shutdown. I liked having the last 2 weeks of the year off so much that I decided to go for a repeat performance this year. So far so good - I’ve stayed away from Slack/email, it’s been quiet & relaxing…but ask me how I feel about it on, say, the 30th when the kids have been out of school for a week or so and are bouncing off the walls. https://thingly.net/igotw/on-moratoria/ Fri, 14 Dec 2018 00:00:00 +0000 https://thingly.net/igotw/on-moratoria/ _A few times a year we enforce a production-wide moratorium - a planned pause on all deployment activity. The intent is to reduce the likelihood that _ something will break during major holidays and/or periods when there won’t be a whole lot of folks around to dig in and fix issues, and the approach is largely effective (except when the moratorium schedule mysteriously vanishes and hilarity ensues). That said, anyone who’s been around for a while knows that the period immediately following a moratorium is a time to hold your breath, cross your fingers, and hold onto your butts. https://thingly.net/igotw/small-world/ Thu, 06 Dec 2018 00:00:00 +0000 https://thingly.net/igotw/small-world/ By now folks have probably seen the news about Google purchasing LinkedIn’s old headquarters for a cool $1 billion in “the largest Bay Area real estate sale in 2018.” Some of the old-timers might remember when LinkedIn shared that space with other companies. Some of the really old-timers might remember Research Libraries Group (RLG), in particular. RLG eventually ended up relocating their staff to San Mateo after merging with OCLC ~10 years ago, but once upon a time they occupied space in 2029 Stierlin Ct. https://thingly.net/igotw/fractals/ Thu, 29 Nov 2018 00:00:00 +0000 https://thingly.net/igotw/fractals/ This week’s post comes from Josh Briefman. He came across an interesting inGraph that exhibits patterns within patterns. Let’s take a look. In Josh’s own words: At 8 week’s, it looks kinda blah: https://ingraphs.prod.linkedin.com/container/notifications-service/graph/notifications-service/notifications-service.i001.R2_Client_Sensor. R2_Client_Sensor_-http-r2d2DefaultClient-_lixClientContexts.CallCountTotal.rrd?fabric_groups=prod&filter=D2&duration_unit=weeks But if you zoom in to 8 days, an interesting pattern begins to emerge: https://ingraphs.prod.linkedin.com/container/notifications-service/graph/notifications-service/notifications-service.i001.R2_Client_Sensor. R2_Client_Sensor_-http-r2d2DefaultClient-_lixClientContexts.CallCountTotal.rrd?fabric_groups=prod&filter=D2&duration_unit=days It only gets better at 8 hours, look at those colors: https://ingraphs.prod.linkedin.com/container/notifications-service/graph/notifications-service/notifications-service.i001.R2_Client_Sensor. R2_Client_Sensor_-http-r2d2DefaultClient-_lixClientContexts.CallCountTotal.rrd?fabric_groups=prod&filter=D2 And finally, at 8 minutes, we can see the pattern that builds that beautiful graph above: https://thingly.net/igotw/happy-thanksgiving/ Fri, 23 Nov 2018 00:00:00 +0000 https://thingly.net/igotw/happy-thanksgiving/ You may have noticed that the office was a bit of a ghost town on Wednesday. Welp…as it turns out there was an Espresso storage node that decided to d uck out a bit early for the 4-day weekend, too. To horribly mangle a Dostoyevsky quote*: “In spite of all possible moratoria you are in complete slavery to your storage layer.” Ah, well. Hopefully this was the only GCN to mar an otherwise quiet holiday weekend. https://thingly.net/igotw/unintentional-art-biyp-edition/ Fri, 16 Nov 2018 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-biyp-edition/ Today is Bring In Your Parents Day. I brought my mom in to see the campus, see what I do, and partake in the festivities. In that spirit I thought it might be fun for her to participate in this week’s igotw. So…I sat her down in front of my MBP, pointed her toward a directory with something on the order of 1100+ graphs, and asked her to pick out whichever ones she thought I ought to post. https://thingly.net/igotw/flatline-redux/ Thu, 08 Nov 2018 00:00:00 +0000 https://thingly.net/igotw/flatline-redux/ I’ve got a couple of recent inGraphs that were sent my way by Antonio Gurgel. Both are examples of the _flatline pattern. Let’s start off with The Bad One: _ In addition to reminding me of Sonic the Hedgehog, this is a graph you probably don’t want to see. Shit can get weird when filesystems fill up - particularly the /var filesystem - so free space going to 0 is most likely undesirable. https://thingly.net/igotw/ethics-in-computing/ Fri, 02 Nov 2018 00:00:00 +0000 https://thingly.net/igotw/ethics-in-computing/ This close to November 6 you might be expecting a post with a title like the above to be about election tampering. This is not that post, folks…at least not in the way you might be expecting. This is a story about the first lesson I ever learned about ethics in computing. I learned this lesson when I was fairly young. I first put hands to keyboard was when I was 6. https://thingly.net/igotw/war-story/ Fri, 26 Oct 2018 00:00:00 +0000 https://thingly.net/igotw/war-story/ _Ever since I saw Kathleen Shannon’s _post on venturebeat (adapted from a talk she gave at SREincon17) I’ve been thinking about some of my own war stories. One in particular stands out in my mind. It’s not a story I tell often - frankly, it’s super-embarassing - but thinking back on it the experience was super-formative for me, so I’d like to talk about it a bit here. Very shortly after I started at LinkedIn - perhaps 6 months in, certainly less than a year - I was working on collecting metrics from memcached using v1 of OBHC. https://thingly.net/igotw/two-hard-things/ Fri, 19 Oct 2018 00:00:00 +0000 https://thingly.net/igotw/two-hard-things/ There’s an old trope I refer to more than anyone probably should: There are only two hard things things in Computer Science: cache invalidation and naming things. The vast majority of the time when I bring it out I’m talking about the latter (host/fabric naming conventions, anyone? ), but at least some of the time I’m talking about the former. This is one of those times. Folks who spend any significant amount of time using inGraphs are likely to have noticed that sometimes the inGraphs don’t load, or don’t show the data that were expected. https://thingly.net/igotw/ceci-n-est-pas/ Fri, 12 Oct 2018 00:00:00 +0000 https://thingly.net/igotw/ceci-n-est-pas/ Every now and again I get a hankerin’ to fiddle around with data viz that isn’t inGraphs. This week, I wanted to take a peek at some redliner numbers to see if I could tease something interesting out of them. My goals were pretty simple: Generate a scatter plot that showed all of the data for each run (not just the Redline Number), and Hack it together in 90 minutes or fewer (cuz I’ve got Shit to Do) https://thingly.net/igotw/unintentional-art-chili-cookoff-edition/ Thu, 04 Oct 2018 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-chili-cookoff-edition/ Edition) Hopefully most of the folks reading this are at the 2018 Chili Cookoff today. I’m super-bummed I couldn’t make it this year - it’s my favorite annual LinkedIn event - but I hope y’all are having fun. At any rate, let’s take a peek at a handful of inGraphs, shall we? First up: a nifty little inGraph-cake from a recent GCN: In addition to being a good demonstration of the informed overlay (note the two faint blue lines representing the lix activation that caused the GCN and the subsequent termination that resolved it. https://thingly.net/igotw/cornucopia/ Thu, 27 Sep 2018 00:00:00 +0000 https://thingly.net/igotw/cornucopia/ This week was a veritable Horn of Plenty for interesting inGraphs. There were 3 Major GCNs at this morning’s _DS3 alone, and while not every igotw is _ about an incident the frequent reader will grok that GCNs are a consistent source of rad inGraphs. There was one incident in particular this week that stood out: GCN-27616. I have to admit, upon initially seeing this GCN my gut reaction was that maybe it was an exercise in efficiency. https://thingly.net/igotw/on-observability/ Thu, 20 Sep 2018 00:00:00 +0000 https://thingly.net/igotw/on-observability/ Over the past couple of years I’ve seen a handful of talks and read a number of…err…reads relating to Observability. As a concept, Observability makes my mouth water. Yes, I want it. Yes, I want to be able to understand precisely what’s going on with the systems I support. Yes, I want everyone around me - everyone I work with - every engineer I come in contact with - to have that same ability to deeply grok What’s Hatnin’. https://thingly.net/igotw/redliner/ Thu, 13 Sep 2018 00:00:00 +0000 https://thingly.net/igotw/redliner/ For those of you who don’t know what redliner is, it’s a tool for estimating the maximum per-node QPS that a given service can handle using empirical evidence. The fuck does that mean? Well, here’s how it works: Redliner selects one node providing the service you want to “redline” Redliner gradually increases the percentage of overall traffic that goes to that particular node After each increase, redliner evaluates the relative health of the service on that node https://thingly.net/igotw/unintentional-art-september-2018/ Thu, 06 Sep 2018 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-september-2018/ Let’s take a look at a few bits ’n’ bobs I’ve picked up along the way. This first one from _Jayesh Lahori is a great example of the _moiré effect (oblig xkcd): One might look at this and ask “Why is it alerting?” Maybe it’s because there were just too damn many “uppity_downis”? I mean, like, more than a trillion of them! I don’t know what they are, but that seems like more than anyone could possibly need or want. https://thingly.net/igotw/espresso-triage/ Thu, 30 Aug 2018 00:00:00 +0000 https://thingly.net/igotw/espresso-triage/ This week I’ve got a series of inGraphs and a little “color” provided by _Nick Brown. These inGraphs are especially timely in light of _Greg Banks’ recent SRE [in]con talk on triaging Espresso problems - in fact, the very first question I asked was “did you find this because you went to the espresso triage talk at incon”. So…let’s dive right in: Raised to learning-sre from invisualize [link] Found via espresso-client-triage dynamic dashboard (useful! https://thingly.net/igotw/convergence/ Thu, 23 Aug 2018 00:00:00 +0000 https://thingly.net/igotw/convergence/ I’ve posted about load balancing a handful of times in the past. Sometimes the post is about a _break-fix. Sometimes it’s an algorithm that perhaps _doesn’t balance load as well as one might hope. Sometimes it just plain looks cool. This time? Welp…I’d say this time it’s a tale of triumph. Feast your eyes on this: Breathtaking. Once you’re done marveling at this beaut’ (go ahead and take another look…don’t worry, I’ll wait) you might be wondering what happened? https://thingly.net/igotw/unintentional-art-sre-in-con/ Fri, 17 Aug 2018 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-sre-in-con/ [Note: This post has nothing at all to do with incon other than mentioning it, but Confluence apparently no longer allows posts with duplicate titles (despite the fact that I have a dozen or so prior posts with the UA title). So I had to find something else to put in there to make it unique…] _After a long week (including another successful SRE[in]con) I’m pretty wiped out, so let’s just take a look at a few recent bits of UA. https://thingly.net/igotw/go-links/ Thu, 09 Aug 2018 00:00:00 +0000 https://thingly.net/igotw/go-links/ Earlier today I was prepping to write a post I’ve been thinking about for a couple of weeks now. …and then Confluence decided to shit the bed. That got me thinking about go links that answer a question, so I decided to defer the post I was going to write for yet another week (teaser: it had to do with load balancing) and post about something else entirely. Unless this is your first day at LinkedIn it’s very likely that you know what a “go link” is. https://thingly.net/igotw/tracking/ Thu, 02 Aug 2018 00:00:00 +0000 https://thingly.net/igotw/tracking/ The frequent igotw reader will know that GCNs are often an excellent source of interesting inGraphs. Yesterday’s mobile-tracking-frontend GCN (GCN- 27108) was no exception. What happened? Per the ticket: “a tracking event was added in Callisto without the schema being registered. tracking frontend has the ability to do dynamic topic whitelisting and while trying to add that topic it failed to get the schema and spiraled out of control” Just how “out of control” did it spiral? https://thingly.net/igotw/power-shutdown/ Thu, 26 Jul 2018 00:00:00 +0000 https://thingly.net/igotw/power-shutdown/ https://thingly.net/igotw/gaps/ Fri, 20 Jul 2018 00:00:00 +0000 https://thingly.net/igotw/gaps/ I posted about an improvement to azmon a while back, and I thought this week I’d go share another one that I think is interesting. A tl;dr refresher on what azmon is: a python multiproduct for gathering Azure resource metrics from Azure’s monitoring API and re-emitting them into autometrics so we can visualize/alert on them in inGraphs. Let’s take a peek at what needed improvement: Per the graph title, this is an inGraph of average % CPU used for a particular VM running in Azure. https://thingly.net/igotw/family-vacation/ Fri, 13 Jul 2018 00:00:00 +0000 https://thingly.net/igotw/family-vacation/ I’m coming off a week of post-shutdown DTO at the beach. I was combing through my inGraph screenshot backlog a bit trying to find something to post and I came across this lil guy: What it actually is: Something Happened that impacted member-to-member (M2M) invitations. It happened around a month ago. I spent about half of the time since it happened on DTO, so I’m a little fuzzy recalling exactly what it was. https://thingly.net/igotw/happy-4th-of-july/ Wed, 04 Jul 2018 00:00:00 +0000 https://thingly.net/igotw/happy-4th-of-july/ I poked around some of the inGraphs in my backlog today looking for something patriotic - not for too long, mind you (DTO and all that). Here’s what I came up with: This little gem (provided by _Matt Knecht - thanks, Matt!) both looks pretty rad _and can be interpreted as looking like a flag waving in the breeze. (Bonus exercise for the reader: if you fiddle around with the color palette and stacking you can also make it look like nyan cat! https://thingly.net/igotw/break-fix-redux/ Thu, 28 Jun 2018 00:00:00 +0000 https://thingly.net/igotw/break-fix-redux/ A couple of weeks ago I posted a little something about an interesting unsolved problem I happened to be working on at the time. Well, that particular problem has since been resolved (a bit on that below). …and, in classic fashion, solving that problem revealed yet another problem (for which I have an inGraph, natch). On the original issue: we were seeing intermittent timeouts between client and server when making calls through a load balancer in Azure. https://thingly.net/igotw/intentional-art/ Thu, 21 Jun 2018 00:00:00 +0000 https://thingly.net/igotw/intentional-art/ [This week’s post is yet another contribution by Chris Carini Intentional Art (**). Enjoy…and thanks, Chris! ] The What The Why The What Cliff Snyder gets a lot of awesome submissions from folks where inGraphs displays something that looks like art, unintentionally. Cliff Snyder appropriately titles them “Unintentional Art”, and to-date, there have been 19 such posts over the nearly 2.5 years IGOTW has been going! Every time I see one of the UA posts, I think “It’d be pretty neat if some of these were intentional”. https://thingly.net/igotw/break-fix/ Fri, 15 Jun 2018 00:00:00 +0000 https://thingly.net/igotw/break-fix/ By way of apology for not posting on-time: I was laser-focused on tacos and tasty booze-drinks last night. It’s summer, tacos are awesome, booze-drinks are awesome…and I just plain spaced. Anyhow, I have a bit of a sticky wicket I’d like to share. Fair warning: we haven’t managed to unstick this particular wicket just yet, so to speak, so this may not be the most satisfying of posts. But I think it’s at least interesting. https://thingly.net/igotw/ups-downs/ Thu, 07 Jun 2018 00:00:00 +0000 https://thingly.net/igotw/ups-downs/ Man. It’s been A Week. It started off with my toilet flushing into my dining room (quite literally). I reckon here’s what that looked like metaphorically: …and then the plumber came out, re-seated the toilet, fixed the leak, and that looked kinda like: All “Yay! No more shit dripping into the place where my family eats!”, leveling off into something like normal. …and then the rest of the week was kind of: https://thingly.net/igotw/unintentional-art/ Thu, 31 May 2018 00:00:00 +0000 https://thingly.net/igotw/unintentional-art/ Now, I hesitate to call this week’s igotw random - these are of course all hand-selected, curated, bespoke, artisanal, gourmet graphs for your consumption - but…well, there’s just plain no theme this week. I’ve got a backlog of interesting things I’ve picked up over the past coupla months, and it’s time for that dam to burst. So…it’s just plain all over the place. Pandemonium. Let’s Dive right in! Max Wolffe shot this one my way a little while back: https://thingly.net/igotw/before-you-panic-context/ Fri, 25 May 2018 00:00:00 +0000 https://thingly.net/igotw/before-you-panic-context/ I’m certain I’ve posted about this kind of thing before but I reckon it bears repeating: you’re going to want to have a little context around what your metrics actually mean before pressing the panic button. Let’s take a peek at an example, shall we? This a Latency inGraph of Greatness. Speaking broadly a graph like this probably should evoke a response of “Hrrrnnnnnng what the shit?!” In fact, that’s what I think I said when I first saw it. https://thingly.net/igotw/before-you-panic-limits/ Thu, 17 May 2018 00:00:00 +0000 https://thingly.net/igotw/before-you-panic-limits/ So, a Thing Happened yesterday: There are a couple of interesting tidbits to note here: NoNameSpecified is probably not the most descriptive name for this sensor to have; we’ll wanna fix that. It is an excellent example of The Dreaded Plateau. This is an ErrorRate sensor (errors divided by requests) and it’s plateauing at 1.0 (i.e., 100%). This is probably not desired behavior. Since we’re taking a peek at this post hoc I happen to know what caused this - specifically, the lix sn4x-use-k2-api - so it’s a good opportunity to revisit the informed overlay: https://thingly.net/igotw/unintentional-art-pycon-edition/ Thu, 10 May 2018 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-pycon-edition/ I’m heading to PyCon 2018 a little later today, so I wanted to do a post on something Python-related. My first thought was to grep around in inGraphs dashboards. Perhaps unsurprisingly, that approach didn’t really provide particularly fruitful. I did have to chuckle a bit when I saw the note “If /tmp is filled, Python starts throwing Bus Errors. Please clear up /tmp.” in one dashboard - yepyep, seen that one before - but I still didn’t really have any inGraphs to show for my efforts. https://thingly.net/igotw/steps-to-improvement/ Thu, 03 May 2018 00:00:00 +0000 https://thingly.net/igotw/steps-to-improvement/ A bit of background: the k2-sre team is working on a Python multiproduct called azmon. The purpose of azmon is to MONitor AZure resources (clever naming, I know _). It came about in order to meet some of the monitoring needs of the K2 project. The mechanics are straightforward: pull metrics from _ Azure’s monitoring API and stuff them into AMF such that we can put them on inGraphs dashboards, alert on them, etc. https://thingly.net/igotw/the-good-ship-lodbrog/ Thu, 26 Apr 2018 00:00:00 +0000 https://thingly.net/igotw/the-good-ship-lodbrog/ Per Wikipedia: Lodbrog is a 10,243 GT cable laying ship that was built as a freight ferry. She was built in 1983 as Siegelberg and completed in 1985 for Romanian owners as Tuzla. In 1996 she was renamed Bolero. The ship was renamed Lodbrog after conversion to a cable layer in 2001. The fuck does this have to do with anything? Well…I was chatting a bit with some folks earlier this week and it triggered a memory of a DS3 from Way Back When. https://thingly.net/igotw/travel-time/ Thu, 19 Apr 2018 00:00:00 +0000 https://thingly.net/igotw/travel-time/ [Today’s igotw is a guest post written by Chris Carini (the original can be found here: How long does it take to travel across campus?**) Thanks, Chris!] Right. So, any idea on how long it takes to get from Unify in 950 (R), to Mercury in 580 (J)? Maybe.. 5-10 minutes? I suppose it depends if you’re walking, biking, driving, or maybe taking the shuttle. Well, I was walking, so maybe closer to 10 minutes? https://thingly.net/igotw/unintentional-art-nyc-edition/ Thu, 12 Apr 2018 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-nyc-edition/ I’m working from our New York office this week, so I thought I’d try to pull together a UA post that’s NY-themed. Some of these might be a bit of a stretch, but bear with me: This is what things look like when sysops does testing by artificially introducing 1ms of latency for periods of time…but it also looks a bit like a (super- regular) skyline. Next up: This one (from Richard Waid) is a little abstract, but when looked at the right way I kinda see the subway. https://thingly.net/igotw/correlation-causation/ Fri, 06 Apr 2018 00:00:00 +0000 https://thingly.net/igotw/correlation-causation/ Today’s post is a couple of fun little graphs. Neither is actually an inGraph, but both are good reminders that correlation does not imply causation. Let’s take a peek: I stumbled across this pair of graphs showing a pretty tight correlation between progress on the GDPR Online Deletions horizontal initiative and Dow Jones performance (IIRC thanks go to Xin Cai for throwing this one together). The astute reader will note that the timelines are not at all aligned…but it’s a fun one all the same. https://thingly.net/igotw/fear-of-commitment/ Thu, 29 Mar 2018 00:00:00 +0000 https://thingly.net/igotw/fear-of-commitment/ Yesterday _Juan Grande came to me with a question about a service that had barfed trying to allocate memory for a new thread. For folks who have seen a _ JVM blow up before, this log snippet should look pretty familiar: There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory. # An error report file with more information is saved as: # logs/hs_err. https://thingly.net/igotw/don-t-cross-the-streams/ Thu, 22 Mar 2018 00:00:00 +0000 https://thingly.net/igotw/don-t-cross-the-streams/ Aatif Awan recently moved on to his Next Play. That got me thinking about one of the very first igotw posts I ever wrote, referencing Aatif’s blog post about LinkedIn sending fewer emails. For those too lazy to click, here’s what that looked like from the perspective of the deduplication cache that stork uses: …and here’s what it looks like today (on a much shorter time scale - 2 weeks instead of ~7 months): https://thingly.net/igotw/unintentional-art/ Fri, 16 Mar 2018 00:00:00 +0000 https://thingly.net/igotw/unintentional-art/ Folks send me dashboard/snapshot links and screen caps of inGraphs all the time. Somewhere along the way I decided it would be a good idea to try and keep track of who sent what so I could provide proper attribution. Typically this takes the form of throwing the submitter’s name into the filename of the image, along with a string that is at least reasonably descriptive of what the thing is. https://thingly.net/igotw/ups-downs/ Thu, 08 Mar 2018 00:00:00 +0000 https://thingly.net/igotw/ups-downs/ This week’s igotw comes from Ben Weir: There are a handful of things I really like about this inGraph. In many cases you don’t want to see flatlines in your graphs, but this is a great example of a place where it’s perfectly fine (in fact expected) behavior. I also love the sort of Newtonian symmetry of the thing. Oh, and then there’s the fact that it kinda reminds me of a swingset. https://thingly.net/igotw/mystery-edition/ Thu, 01 Mar 2018 00:00:00 +0000 https://thingly.net/igotw/mystery-edition/ I got nerd-sniped earlier this week by Kathleen Shannon. She sent this inGraph my way: Huh. Okay. Interesting. First and foremost: it looks super-cool, right? …but what’s going on here? Well, a coupla bits of data here: A snapshot link to this inGraph: https://ingraphs.prod.linkedin.com/snapshot/k2-nlp-snapshot_20180302_003619/ It’s backed out to 8 days’ worth of data, which should be at least roughly apparent via the legend but it’s worth calling out. It’s a count…and as counts go, 600u (0. https://thingly.net/igotw/backlog/ Thu, 22 Feb 2018 00:00:00 +0000 https://thingly.net/igotw/backlog/ Matt Knecht_ got me kind of thinking with his comment on last week’s post - what exactly _do I have lying around in my igotw “stash”? Welp, a rough count: $ find graphs | wc -l 874 Huh, okay. Well…what are they named? Most of them take the following form: voyager-api-frontend_voyager-api-frontend_i002_HTTP_Input_Sensor_HTTP_Input_Sensor_- _filter_segmentedInboundSensor_Overall_CallCountTotal_rrd.png That should be pretty recognizable to folks who’ve use inGraphs - name-mangled metric strings tend to look something like that. https://thingly.net/igotw/love-hate-relationship/ Fri, 16 Feb 2018 00:00:00 +0000 https://thingly.net/igotw/love-hate-relationship/ I’ve spent a little time thinking about this inGraph that Loren Carvalho shot my way (thanks, lo-lho!) Link: http://ingraphs.prod.linkedin.com/dashboard/range-server/graph/classified?fabrics=prod- lva1&use_time_selector&start_time=1518573600000&end_time=1518832800000 (Seriously, guys, why the fuck haven’t I been linking to these all along? Click it. Play around with it a bit.) Anyhow…there are things I love about this inGraph and things I hate about it. What do I love? For one: it’s named classified. Whoa. I’m not sure whether I have sufficient security clearance to even be looking at this inGraph. https://thingly.net/igotw/a-thousand-words/ Thu, 08 Feb 2018 00:00:00 +0000 https://thingly.net/igotw/a-thousand-words/ This week’s igotw is centered around an inGraph provided by Ian Liu-Johnston (thanks, Ian!) What I love about this inGraph (other than the fact that it looks fucking awesome): it tells a story. It’s super-busy and there are probably too many colors, but it demonstrates a specific thing happening. I have a bit of insider knowledge about what that thing is, but if you gaze upon it for a while things start to pop out. https://thingly.net/igotw/unintentional-art/ Fri, 02 Feb 2018 00:00:00 +0000 https://thingly.net/igotw/unintentional-art/ I know, I know - I just did a UA post _a few weeks ago. Maybe I’m like Bill Murray, _doomed to repeat the same post over and over until I get it right. This first inGraph makes me think of some kind of peasant uprising or something: Thrice they tried to breach the castle walls…and thrice they were repelled. So sad. Next up: a classic example of _the plateau pattern (which seems to find _ its way into igotw almost as often as I see it “in the wild”): https://thingly.net/igotw/cause-effect/ Thu, 25 Jan 2018 00:00:00 +0000 https://thingly.net/igotw/cause-effect/ An inGraph that Matt Knecht sent my way last week made me think about the graphs I’ve seen that demonstrate cause & effect. A pretty common and accessible example of this is stickyrouting traffic distribution graphs during traffic shifts: When the percentage of traffic goes up in one place, it goes down in another. Straightforward enough. Another one that I love: One node starts freaking out and throwing errors, so you try The IT Crowd Fix…and 2 other nodes start freaking out and throwing errors. https://thingly.net/igotw/unintended-consequences/ Thu, 18 Jan 2018 00:00:00 +0000 https://thingly.net/igotw/unintended-consequences/ This week’s post - courtesy of Jon Sorenson (username pronounced “jay-so-n-so”) - reminds me of a couple of previous igotw _posts. When Jon first sent _ this my way, my first thought was “some kind of load balancing algorithm somewhere was busted and has since been un-busted”. As it turns out, that’s not actually the case. Let’s take a peek: Now, I don’t pretend to understand the specifics of how this played out, but in Jon’s own words: https://thingly.net/igotw/unintentional-art/ Fri, 12 Jan 2018 00:00:00 +0000 https://thingly.net/igotw/unintentional-art/ It’s time to take a peek at a few more bits of unintentional inGraphs art, folks. First up, a little danger: This piece makes me feel a touch uneasy. It’s just this immense slab of an impossibly-spikey thing, precarious atop crumbling pillars. (Is this the first bit of UA in which the legend is actually a part of the artwork? I think it may be…) Next up, a hummingbird: Not convinced it looks like a hummingbird? https://thingly.net/igotw/blackouts/ Thu, 04 Jan 2018 00:00:00 +0000 https://thingly.net/igotw/blackouts/ It’s no secret that LinkedIn sends a lot of emails, but what folks might not know is that we observe a set of dates on which we do not send certain classes of email (for example, invitation reminders). These dates are compiled on a per-country basis once a year by the BizOps team and reflect major holidays in those countries. We also observe a handful of blackout dates that are global as opposed to per-country, and there is one 8-day stretch in particular that is static/“fixed” every year - 12/24 through 12/31, and 1/1 (of the following year). https://thingly.net/igotw/christmas-theme/ Thu, 28 Dec 2017 00:00:00 +0000 https://thingly.net/igotw/christmas-theme/ It may be a bit late for this, but for those of you who want to extend the Christmas Spirit a few days after the fact in your inGraphs this might prove useful. Take a look in the inGraphs UI for the Theme dropdown; it looks like this: Set it to “Christmas” as in the screenshot above and voila! It’s not obvious from the screenshot, but those little white dots are animated to give the appearance of falling snow. https://thingly.net/igotw/carpinteria/ Thu, 21 Dec 2017 00:00:00 +0000 https://thingly.net/igotw/carpinteria/ Folks are probably already aware of the Southern California wildfires. What folks may not be aware of: there is a LinkedIn facility in Carpinteria that had to be shut down and evacuated due to the approaching Thomas Fire (TIL: wildfires are named). _“The campus was officially closed by Dec. 6, 2017 at 9 am.” _ What does that look like? Well, there aren’t a whole lot of inGraphs for LCP1, but I was able to dig up this inbound/outbound network graph: https://thingly.net/igotw/calories/ Thu, 14 Dec 2017 00:00:00 +0000 https://thingly.net/igotw/calories/ In one of my favorite talks of all time Dr. Robert Sapolsky, talks about the things that make humans unique (and not-so-unique) in the animal kingdom. One of the things he brings up toward the beginning of the talk is the enormous amount of calories that individual human beings can expend just thinking; specifically, the amount of calories expended by chess grand masters. The numbers he gives - something on the order of 6,000-7,000 calories a day - are hard for me to believe…but he’s a well-respected neurobiologist who has spent his career studying stress responses, so I reckon I’ll take his word for it. https://thingly.net/igotw/load-testing/ Thu, 07 Dec 2017 00:00:00 +0000 https://thingly.net/igotw/load-testing/ Most folks reading this are probably aware of the recent load testing war-room effort…and if you’re not aware of it then you’ve either been on extended DTO or haven’t been paying attention. At any rate, the idea is the following: given traffic projections going into 2018 we need to be able to support X traffic (for a particular value of X - seriously, read your email) and we aren’t quite able to do that just yet. https://thingly.net/igotw/wfh/ Fri, 01 Dec 2017 00:00:00 +0000 https://thingly.net/igotw/wfh/ I know I already did a Thanksgiving post_, but Michael Kehoe pointed out an inGraph that was just too good to pass up (thanks, Michael!):_ Thi s is an inGraph of total connections to the VPN. Notice that big bump in the middle there? That would be the Wednesday before Thanksgiving. Taking a consolidated look comparing to WoW: Perhaps unsurprisingly, it would appear that working from home is a fairly popular trend on that particular day. https://thingly.net/igotw/happy-turkey-day/ Thu, 23 Nov 2017 00:00:00 +0000 https://thingly.net/igotw/happy-turkey-day/ Q: How long does it take a frozen 16-pound turkey to thaw in the refrigerator? A: Trick question…it doesn’t thaw. It never thaws! (Oh God why isn’t it thawing?) Expressed in terms of inGraphs: Happy Turkey Day, folks. Here’s hoping it doesn’t turn into “Happy Late-November Family Pizza Night”. https://thingly.net/igotw/cubism/ Thu, 16 Nov 2017 00:00:00 +0000 https://thingly.net/igotw/cubism/ I’ve spent a little time playing around with visualization tools like cubism.js. I love me some inGraphs (see: like every post here ever), but I’m interested in whether there might be some more interesting/meaningful/helpful ways to look at The Data. As a side project - something to hack around on inDays - it’s interesting to me. Take a look at this: This is another way of looking at downstream calls for a particular service (comm-inbox-bps). https://thingly.net/igotw/unintentional-art/ Thu, 09 Nov 2017 00:00:00 +0000 https://thingly.net/igotw/unintentional-art/ The first bit of UA comes from _Nick Brown (thanks, Nick!) and is a pretty good demonstration of what a slow memory leak might look like in the absence of _ occasional restarts: Next up is an interesting scribble of abstract art that I’m thinking of entitling “Laser Caterpillar” (or something): The li’l guy below is a demonstration of some interesting kafka consumer lag behavior. I’m pretty sure it’s a pair of nodes continually trading consumption of the same kafka partition (or set of partitions). https://thingly.net/igotw/mailbag/ Thu, 02 Nov 2017 00:00:00 +0000 https://thingly.net/igotw/mailbag/ [This week I thought I’d answer a few questions I got from Alexsandra McMahan (thanks, mickmayhay!) I like this idea of having some kind of a recurring “Mailbag” segment for helping folks understand How Shit Works, so if you have a question then please hit me up! ] First up: “wtf does it mean/wtf do I do when my ingraph has a little error thingy?” Well, let’s see what “little error thingy” we’re talking about here: https://thingly.net/igotw/duplicates/ Fri, 27 Oct 2017 00:00:00 +0000 https://thingly.net/igotw/duplicates/ I thought it might be constructive to do a breakdown of a recent GCN while it’s still fresh in my mind. Let’s jump into this with how it started - with Shubhans hu Nagar coming to me with an inGraph: Huh. So all of a sudden the p95 delivery time of this particular push notification jumped to around 600M milliseconds (something on the order of 7 days). Well, that seems pretty bad - it’s probably Not Okay to send out push notifications 7 days after the fact - so let’s try and figure out what happened. https://thingly.net/igotw/rrds-and-granularity/ Thu, 19 Oct 2017 00:00:00 +0000 https://thingly.net/igotw/rrds-and-granularity/ [The idea for today’s post came out of a conversation I had with Unknown User (jgong). Thanks, Jian! ] One of my engineers recently noticed something peculiar in inGraphs. He was looking at an inGraph of the error percentage for some endpoint. When he looked at data for one day (“today”), the graph spiked up to a maximum of 35%. …but then when he backed it out to look at one week, the maximum was only 21%. https://thingly.net/igotw/reciprocal-graphs/ Fri, 13 Oct 2017 00:00:00 +0000 https://thingly.net/igotw/reciprocal-graphs/ The inGraphs for this week’s post were submitted by Drew Dibble (Thanks, Drew! ) Let’s take a peek: These are just lovely. In Drew’s own words: “This one looks like stalactites/stalagmites to me, and shows the relationship between downstream latency and the TotalPointsInHashRing metric.” Let’s unpack that a bit. The bottom graph is fairly straightforward - it’s showing latency in the 90th percentile. …but what’s going on with this “TotalPointsInHashRing” thing? https://thingly.net/igotw/meta/ Thu, 05 Oct 2017 00:00:00 +0000 https://thingly.net/igotw/meta/ [This post was inspired by a conversation I had with Natalia Goreva*. Thank you, nat-nat, for bringing this up.]* I recently had a colleague ask me why I do igotw. I thought it might be interesting to dive into that as a sort of meta-subject. It’s gotta start off with a graph of some kind so I’ll go ahead and give this one: This is not an inGraph - it’s a thing I threw together in Excel - but it depicts the number of igotw posts over time. https://thingly.net/igotw/a-series-of-tubes/ Thu, 28 Sep 2017 00:00:00 +0000 https://thingly.net/igotw/a-series-of-tubes/ As we all know, the Internet is a series of tubes. Sen. Ted Stevens being a bit of a numnutz aside, sometimes this view of How Shit Works can be helpfully leveraged - in particular, when trying to think about how asynchronous/“offline” flows work. In that vein, let’s take a high-level look at how GCN-24373 played out. The initial report: a particular push notification had gone well outside of its delivery time threshold: https://thingly.net/igotw/unintentional-art/ Thu, 21 Sep 2017 00:00:00 +0000 https://thingly.net/igotw/unintentional-art/ I realized today that it’s been a good long while since the last time I did a UA post. Well, let’s rectify that, shall we? This one is guest partition requests during a traffic shift, but it kind of made me think of the light cycles from Tron: An impressive one - 200% errors! - that also happens to look a fair bit like a bulldozer: …and continuing the “vehicle” theme, this lil guy reminded me of an ice cream truck: https://thingly.net/igotw/out-of-band-monitoring/ Thu, 14 Sep 2017 00:00:00 +0000 https://thingly.net/igotw/out-of-band-monitoring/ Earlier today I came across some chatter in a slack channel about one service getting errors calling another service. (I know, I know - the sky is blue, the grass is green, it’s Thursday and something is broken on the Internet…this isn’t news, who gives a fuck? Bear with me here.) Taking a look at errors metrics for the offending service didn’t really reveal anything, but a little poking around in the logs revealed that on one node the service had blown through its open file descriptor limit. https://thingly.net/igotw/before-you-panic-wow-multifabric-consolidation/ Thu, 07 Sep 2017 00:00:00 +0000 https://thingly.net/igotw/before-you-panic-wow-multifabric-consolidation/ Consolidation) It should come as no surprise to the regular igotw reader that it can take some time to get familiarized with inGraphs - to understand what it’s trying to tell you, to really grok what you’re looking at. Let’s take a peek at a (seemingly-)scary inGraph: Someone without any familiarity with inGraphs might look at this and say “Shit! We saw spikes in errors two weeks in a row! What’s going on here? https://thingly.net/igotw/mystery-revealed/ Thu, 31 Aug 2017 00:00:00 +0000 https://thingly.net/igotw/mystery-revealed/ [As promised, here’s the reveal writeup from last week’s mystery post by Max Wolffe. Thanks again, Max! ] Last week Cliff posted a mysterious rainbow waterfall graph, of hosts for a service suddenly receiving no traffic, one after another. blocked URL It almost looks like a deployment, except that the traffic drop doesn’t occur for all hosts, and there’s no informed overlay event indicating that a deployment occurred. Upon looking at the graphs for a single host, we can see that there’s a big single spike in 5xx errors across several endpoints that realtime-dispatcher exposes. https://thingly.net/igotw/mystery-post/ Thu, 24 Aug 2017 00:00:00 +0000 https://thingly.net/igotw/mystery-post/ This inGraph was brought to my attention by none other than The Wolffe - Max Wolffe. Apologies to Max for putting him on the spot, but he has committed to writing up next week’s post. So! In that vein: What do you think might be going on here? I’d love to hear guesses/speculation/opinions/etc. in the comments. If it’s helpful, here’s the link to the actual dashboard (including timeboxing): http://ingraphs.prod.linkedin.com/dashboard/realtime-dispatcher-qps-dashboard/graph/Synchronous%20publish%20event%20QPS? legend=false&override&auto_refresh=1&use_time_selector&start_time=1503521220000&end_time=1503550020000&consolidate=None&fabrics=prod- ltx1&width=734 https://thingly.net/igotw/periodicity/ Fri, 18 Aug 2017 00:00:00 +0000 https://thingly.net/igotw/periodicity/ Site traffic tends to follow fairly regular patterns. Barring holidays, world events, sports, etc. traffic generally adheres to a particular cadence. Let’s take a peek at a week of site traffic in the US and Germany, respectively: No huge surprises here. Traffic dips when we’d expect traffic to dip - on weekends, at lunch/dinner, when folks are asleep - and then picks up when everyone is awake and engaged. There is a bit of a shift due to time zone offsets, but overall the pattern is sensible and readily identifiable. https://thingly.net/igotw/metric-discovery/ Thu, 10 Aug 2017 00:00:00 +0000 https://thingly.net/igotw/metric-discovery/ [Shout-out to Jacky Wang for giving me the idea for this post yesterday! ] I recently had one of my Engineers ask me how to find a metric in inGraphs - specifically, he wanted to know what the QPS to the RecipientSuggestionResource#getAll endpoint looked like. I love inGraphs, but I reckon I’d say it’s a bit lacking in the discoverability department. If you know exactly what you’re looking for (and how to look for it) then you can find what you need, but if you don’t then it can be a bit of a challenge. https://thingly.net/igotw/what-s-that-dip/ Thu, 03 Aug 2017 00:00:00 +0000 https://thingly.net/igotw/what-s-that-dip/ [Once again, this week’s igotw comes from Chris Carini. Thanks again, Chris - keep this up and I’ll have to rename this blog “The Weekly Carini”! ] Below is a metric my team follows fairly closely. It looks like we’ve got a normal pattern here, a fairly consistent oscillation, and then all of a sudden, it flat lines for a bit, and resumes it’s normal pattern. In general, we’re trending upwards https://thingly.net/igotw/informed-overlay/ Fri, 28 Jul 2017 00:00:00 +0000 https://thingly.net/igotw/informed-overlay/ In this week’s post I’d like to highlight a Killer Feature of inGraphs that I think I’ve demonstrated before but never posted about explicitly (at least not that I can find): the informed overlay. Informed is a tool at LinkedIn that accepts messages about events from various sources - deployments, lix ramps, etc. - and stashes them away in the hopes that if someone wants to figure out what happened after the fact that information will be available. https://thingly.net/igotw/below-threshold/ Wed, 19 Jul 2017 00:00:00 +0000 https://thingly.net/igotw/below-threshold/ [This week’s post was submitted by Chris Carini. Thanks for the awesome find and great write-up, Chris!] Today ( 18 Jul 2017 ) at TS3, there was an Alfred GCN that had happened during the night and our wonderful colleagues in Bangalore took care of the issue. Awesome! The input Found at go/ts3dash this morning, the closed GCN; cool, everything is better now that it’s closed, right?… The trail Of course… the link and screenshot to the inGraph looks like it did before! https://thingly.net/igotw/before-you-panic-metric-lag/ Thu, 13 Jul 2017 00:00:00 +0000 https://thingly.net/igotw/before-you-panic-metric-lag/ It’s almost a rite of passage for inGraphs newbies to run across an inGraph like the following and flip out: “Oh my God! This metric has dropped right to the floor in the past couple of minutes! Spin up a GCN, let’s Engage Dudes and get to the bottom of this!” They’re correct - it certainly does look alarming! - but hopefully they run it by someone with a little more experience using inGraphs before rounding up the cavalry. https://thingly.net/igotw/independence-and-canada-day/ Thu, 06 Jul 2017 00:00:00 +0000 https://thingly.net/igotw/independence-and-canada-day/ Just like last year, I thought we could take a look at what US site traffic looked like around the Fourth of July: “But Cliff,” my Canadian friends want to know, “Canada happens to have a major holiday around this time, too! Why don’t you show us Canadians a little love? (And no, that one time you put a side note about how Canadians tuned into the Superbowl to watch Coldplay doesn’t count. https://thingly.net/igotw/canary-redux/ Thu, 29 Jun 2017 00:00:00 +0000 https://thingly.net/igotw/canary-redux/ I’ve come back to the innaugural igotw post on more than one occasion. It never ceases to amaze me how relevant it continues to be. A similar situation - in the form of a Major GCN - happened earlier this week. I say “similar” in that some canaried code showed the errors (and was promoted all the same). Let’s dive into just how similar it was (or wasn’t). First up, let’s take a peek at error QPS for the busted service in all fabrics: https://thingly.net/igotw/duplicate-mbeans/ Thu, 22 Jun 2017 00:00:00 +0000 https://thingly.net/igotw/duplicate-mbeans/ Once a month or so I get a Slack DM from one of my engineers. (S)he is typically - but not always - new to the oncall rotation, and trying to shepherd the canary & subsequent promotion of a new release of some jankitudinous old code with a handful of new commits. The ask typically takes the following form: “Hey, can you please help me take a look at this EKG? https://thingly.net/igotw/cadence/ Thu, 15 Jun 2017 00:00:00 +0000 https://thingly.net/igotw/cadence/ Consider the following inGraph: The first thing that might come to mind when looking at the legend is “Lord Almighty, look at all the QPS!” Sure.This is an inGraph of inbound traffic to identity-mt - one of the highest-traffic mid-tiers at LinkedIn. If you’re doing anything at all that requires member profile data then you’re talking to identity-mt, and that use case covers a pretty broad swath of the site. Looking a little more closely there are these little “perturbations” on hourly boundaries. https://thingly.net/igotw/unintentional-art/ Fri, 09 Jun 2017 00:00:00 +0000 https://thingly.net/igotw/unintentional-art/ Kicking it off with something other than an inGraph, here’s a latency heatmap Kurt Andersen shot my way: I’m still chewing on it a bit, trying to figure out the reasons for the pretty obvious convergence around 6/1…but it’s here in a UA post because it looks fucking sick. Next up: This nifty lil feller is what happens when you uplift a service that has explicitly-defined ulimits. Unfortunately, the service comes up before range has sync’d, which means the cfengine machinery that pushes out those limit changes hasn’t had a chance to kick in yet (and thus you get the default values on the newly-added nodes). https://thingly.net/igotw/less-is-more/ Thu, 01 Jun 2017 00:00:00 +0000 https://thingly.net/igotw/less-is-more/ This week I stumbled across an inGraph of Beauty almost completely by accident: It actually started out as an inGraph of Panic - I wasn’t expecting something quite this dramatic - so I reached out to Swapnil Ghike_ and Aniruddh _ Chaturvedi (the oncall) to figure out what was going on (thanks, guys!) As it turns out (o continue to name-drop): _Jeffrey Schang ramped a lix that actually _ decreased the overall QPS to this endpoint: https://thingly.net/igotw/what-s-that-downstream/ Thu, 25 May 2017 00:00:00 +0000 https://thingly.net/igotw/what-s-that-downstream/ In _last week’s post we took a look at the dynamic dashboards feature of inGraphs. One of those was the downstream-traffic dashboard, which gives a _ bunch of useful information about endpoints that our service is calling. …but what if we wanted to know what the actual name of the service serving that endpoint is? For instance, suppose it’s busted and we want to look it up in go/owners so we can escalate to the appropriate person. https://thingly.net/igotw/dynamic-dashboards/ Thu, 18 May 2017 00:00:00 +0000 https://thingly.net/igotw/dynamic-dashboards/ This week’s post is (hopefully) an educational one for folks who aren’t super-familiar with inGraphs. The concept came from _Alexsandra McMahan, and the _ idea is the following: “Suppose LinkedIn is down, what are the first three inGraphs I should look at?” Hmm…damn near infinite scope. I could give you any three inGraphs and they would be the wrong ones to look at for any given site outage. So let’s scale it back a bit: https://thingly.net/igotw/subtlety/ Thu, 11 May 2017 00:00:00 +0000 https://thingly.net/igotw/subtlety/ One of the things I love about inGraphs (or really, graphs/visualizations in general) is the information density - the amount of information that can be conveyed in a relatively small amount of space. Granted, it takes some time and experience to really grok what you’re looking at, but once you’ve invested that time inGraphs like this one really hit home: See that dip there in the middle? Subtle, right? Well…that “little” dip is LinkedIn dropping something on the order of 20% of all traffic at the network edge. https://thingly.net/igotw/not-even-wrong/ Thu, 04 May 2017 00:00:00 +0000 https://thingly.net/igotw/not-even-wrong/ I’m an O365 early adopter and guide…which has fuckall to do with anything, except that as Outlook has been syncing a number caught my eye: Huh. Over 200K alerts since I started at LinkedIn. I dumped some numbers into a spreadsheet to take a look at some averages: I got a little curious - is this Just Me, or do other folks see this same kind of email volume? Well, David De Maagd hooked me up with this inGraph: https://thingly.net/igotw/datavault-d2/ Thu, 27 Apr 2017 00:00:00 +0000 https://thingly.net/igotw/datavault-d2/ [Attribution for the meat of this week’s post goes to Akhilesh Gupta*, whose email I ripped off more or less verbatim below. Also, huge shout-out to Max* Wolffe and the Datavault team for the investigation leading up to that email.] Here are a couple of interesting inGraphs: Don’t adjust your dial, folks…these are two different inGraphs of two different metrics. The first is a datavault metric representing the HTTP message count to a service. https://thingly.net/igotw/unintentional-art/ Thu, 20 Apr 2017 00:00:00 +0000 https://thingly.net/igotw/unintentional-art/ You know the drill - yet another UA post - so let’s jump right into it! Here’s a rad helix graph: Here’s one that makes me think of some kind of space battle over the face of a contested planet (that perhaps harbors some kind of super-weapon): A spectacular mustache: …and for those of you who enjoyed The Dark Knight, an inGraph I think of as “Why So Serious?”: https://thingly.net/igotw/go-routine-leak/ Thu, 13 Apr 2017 00:00:00 +0000 https://thingly.net/igotw/go-routine-leak/ [This week’s igotw is a guest post by Ahmed Sharif original wiki page (**). Thanks for sharing, Ahmed!] In _obhc-server we monitor the number of currently running go-routines among other things. For the last several _months we observed the pattern below for the number of go-routines (inGraphs snapshot link). The number of go-routines was always going up and up until the application is restarted because of deployment. Clearly there was a go-routine leak. https://thingly.net/igotw/cascading-failure/ Thu, 06 Apr 2017 00:00:00 +0000 https://thingly.net/igotw/cascading-failure/ There have been a handful of failures of the voyager-api-frontend messaging cluster this past week. Three of these failures have had the same/similar characteristics, and we now think we have a fairly good understanding of what was actually happening. I’ll outline the events below below. Keep in mind that while I’m putting these in numeric order, all of these things happened so quickly, so close together that they were for all intents and purposes “immediate”. https://thingly.net/igotw/unintentional-art/ Thu, 30 Mar 2017 00:00:00 +0000 https://thingly.net/igotw/unintentional-art/ _I’ve got UA starting to overrun my stockpile of inGraphs snapshots, so it’s time to get a few of ’em out there. First up: some diamonds, courtesy of Richard _ _Waid and a bit similar to some UA from _a prior post: Next, one from Matt Knecht that both has an interesting backstory and makes me think of a chainsaw: One of the most rock-n-roll inGraphs I think I’ve ever seen: https://thingly.net/igotw/stall-time/ Thu, 23 Mar 2017 00:00:00 +0000 https://thingly.net/igotw/stall-time/ _In a Java World garbage collection (GC) is a fact of life. I’ve posted graphs similar to the below on at least two previous occasions __ but I’ve never given a _ whole lot of detail about what they actually mean. I feel like these particular metrics are relevant enough to warrant their own post, so without further ado: Here are a couple of stall time inGraphs. You can get to them fairly quickly using filters. https://thingly.net/igotw/happy-holi/ Thu, 16 Mar 2017 00:00:00 +0000 https://thingly.net/igotw/happy-holi/ In keeping with the igotw tradition of posting site traffic during holidays, here’s what overall site traffic from India looked like during Holi: You may have noticed the colorful title bar, which I thought was very much in the spirit of the holiday. Well, you can have this color in your life if you like, too! Just go into Controls and select “Advanced” from the Theme dropdown. [Warning: Use of the Advanced theme may be seizure-inducing! https://thingly.net/igotw/unintentional-art/ Thu, 09 Mar 2017 00:00:00 +0000 https://thingly.net/igotw/unintentional-art/ Just a few interesting bits of unintentional art this week. First up, a piece dubbed “Rainbows (But No Unicorns)” by Kyle Goedtel: Along fairly similar visual lines, a refractive piece that fans of Floyd might find familiar: …and, for something a little different, a little more organic, there’s this lil guy: I thought it looked kinda like an armadillo, then maybe an anteater…but my wife ultimately convinced me it looks most like a shrew. https://thingly.net/igotw/escalator-to-and-from-nowhere/ Thu, 02 Mar 2017 00:00:00 +0000 https://thingly.net/igotw/escalator-to-and-from-nowhere/ In a previous Unintentional Art post I referenced a great moment from an episode of The Simpsons (without attribution - shame on me!) Well, this week I’ve got an escalator that not only goes nowhere, but also returns! Feast your eyes on this one: This is an inGraph of what GCN-22862 looked like. It shows l1proxy failing to connect to realtime-frontend, in terms of failed bytes’-worth of requests. A graph from the same timeframe that looks less like an escalator to nowhere but is nonetheless still interesting: https://thingly.net/igotw/decom-lessons-learned/ Thu, 23 Feb 2017 00:00:00 +0000 https://thingly.net/igotw/decom-lessons-learned/ _This week’s post is really more of a re-blog of Ting Sun’s Eng Blog post about _Lessons Learned from Decommissioning a Legacy Service. I know, I should be posting my own shit…but this is an opportunity to highlight an excellent & positive follow-on to a previous igotw. If you happened to read that then this should look familiar to you: Well done, Ting Sun! https://thingly.net/igotw/patterns-the-flatline/ Thu, 16 Feb 2017 00:00:00 +0000 https://thingly.net/igotw/patterns-the-flatline/ I almost feel like this is a duplicate post in that it’s super-similar to _The Dreaded Plateau, only in reverse. Similar to that pattern, it’s not generally something _ you want to see. Like when kafka production to a particular set of partitions ceases for some reason, and your consumers have nothing to consume: …or one node in your jankitudinous wreck of a legacy service decides it’s toly way more important to GC itself into oblivion than it is to serve requests: https://thingly.net/igotw/a-tale-of-terror/ Thu, 09 Feb 2017 00:00:00 +0000 https://thingly.net/igotw/a-tale-of-terror/ I’d like to talk about a recent GCN that is one of the scariest I’ve seen at LinkedIn. The actual impact was limited; it only hit services that were deployed during the incident, and it happened relatively late in the day so only a handful of things had been deployed in that timeframe. …but had it happened earlier in the day…oh, man, could it ever have been worse. I only have a single inGraph to share from this one: https://thingly.net/igotw/before-you-panic-aggregation/ Thu, 02 Feb 2017 00:00:00 +0000 https://thingly.net/igotw/before-you-panic-aggregation/ Suppose you see a spike in your service’s latency. It might look something like this: Well, okay. This inGraph shows the average of a bunch of nodes’ 95th percentile latencies. Averaging percentile values makes the Baby Jesus cry, so let’s remove that averaging function and take a look at each individual host’s latency. Hmmm…a whole bunch of nodes are seeing that additional latency and it’s actually worse than we’d thought wrt magnitude (look at the scale); not terribly surprising when you torture statistics by doing something like averaging per-host 95th percentile values. https://thingly.net/igotw/dotted-line/ Thu, 26 Jan 2017 00:00:00 +0000 https://thingly.net/igotw/dotted-line/ This week’s igotw comes courtesy of Jerilyn Franz. The metric is Couchbase “Actual Replicas”. What that means: Couchbase can be configured to maintain zero or more replicas of data. This can be super- useful if you want to be able to take a node out of rotation and not lose access to data while it’s OOR. This particular cluster is apparently configured to hold one replica. For reasons I don’t entirely understand - I am far from being a Couchbase expert - sometimes replicas can become unevenly distributed. https://thingly.net/igotw/a-cautionary-tale/ Thu, 19 Jan 2017 00:00:00 +0000 https://thingly.net/igotw/a-cautionary-tale/ Ever wondered what it might look like if your service went down in a fabric for two days (and nobody noticed)? Well, friend, have I got some inGraphs for you! There’s the QPS graph. Note how it goes to zero…on account of no userp instances being available to receive traffic (and thereby emit metrics). Another: There’s the uptime graph. The corresponding OBHC graph: Well, shit. \ There should absolutely have been some kind of alert here - especially for a service that is pretty much wholly SEO-specific (read: bot-facing, not member- facing). https://thingly.net/igotw/the-thundering-herd/ Thu, 12 Jan 2017 00:00:00 +0000 https://thingly.net/igotw/the-thundering-herd/ This week’s igotw is a fun little anecdote I’d forgotten about until browsing through my cache of saved-off inGraphs. Most folks reading this are probably aware of the thundering herd problem - a process/service wakes/starts up and all of the clients who have been waiting on it subsequently beat it to death. Well, this is a sort of meat-space example of that. On occasion (read: all the damn time) some poor engineer will wander into a slack channel and let folks know “Hey, I’m trying to do something on eng-portal but it’s telling me it’s out of disk space, can someone please help me? https://thingly.net/igotw/happy-new-year/ Thu, 05 Jan 2017 00:00:00 +0000 https://thingly.net/igotw/happy-new-year/ Today we’ll be taking a look at some inGraphs that Michael Kehoe passed along. These inGraphs show incoming site traffic for the month of December. The first one should be pretty familiar - current week with week-over-week overlay: It should come as no surprise that overall site traffic shows a decline as compared to WoW for the holiday weekend. …but how does it compare to last year? Well, we can look at year-over-year in inGraphs, too: https://thingly.net/igotw/happy-holidays/ Thu, 29 Dec 2016 00:00:00 +0000 https://thingly.net/igotw/happy-holidays/ This inGraph seemed apropos for this week (thanks, Ben Weir!) https://thingly.net/igotw/boring-ingraphs/ Thu, 22 Dec 2016 00:00:00 +0000 https://thingly.net/igotw/boring-ingraphs/ Today’s igotw is super-lame (on purpose). For all you folks oncall out there: I wish you all super-boring inGraphs like this for the duration of the company shutdown. Happy holidays! https://thingly.net/igotw/un-graceful-restart/ Thu, 15 Dec 2016 00:00:00 +0000 https://thingly.net/igotw/un-graceful-restart/ I’ve been thinking about this inGraph a bit today: This shows the median latency of badging-mt over the course of a restart. On its face, it looks pretty bad - “OMG, when we restart each individual node’s p50 goes close to a minute!” …but I wonder whether that’s actually accurate in terms of actual member impact. I’d like to outline a couple of possibilities (not an exhaustive list): The service does not remove itself from d2 as it’s shutting down and/or announces to d2 as it’s coming up before it’s ready to take traffic. https://thingly.net/igotw/stacked-graphs/ Thu, 08 Dec 2016 00:00:00 +0000 https://thingly.net/igotw/stacked-graphs/ Generally speaking I’m not a fan of the stacked graph. It has limited utility and more often than not has a tendency to distort perception rather than enhance understanding. (PSA: If anyone out there is currently relying on stacked graphs of latency, please reach out to me. You are not alone in this world. I can help.) That said, there are times when I can appreciate a stacked graph. Sometimes they’re beautiful, like this SCN age inGraph that James Luck sent my way recently: https://thingly.net/igotw/unintentional-art/ Thu, 01 Dec 2016 00:00:00 +0000 https://thingly.net/igotw/unintentional-art/ Thin post this week…but it looks enough like a space ship or an absurdly-long sci-fi laser rifle that I hope you’ll forgive me. I’m certain that this inGraph tells an interesting story. It actually makes me think a bit of Tufte and the statistical graphic of Napoleon’s march that he seems to be enamored with. I’d love to tell that story…but sadly, I don’t actually know what it is or why I took this screenshot to begin with. https://thingly.net/igotw/happy-thanksgiving/ Thu, 24 Nov 2016 00:00:00 +0000 https://thingly.net/igotw/happy-thanksgiving/ I scoured through my pile of inGraphs snapshots for one that looked like a turkey. Sadly, I did not find one. (My wife helpfully asked whether I had one that looks like mashed potatoes.) Instead, I’ve got one that I think looks a little like a dimetrodon: …which I think paleontologists likely agree was the turkey of the dinosaur world. Happy Thanksgiving, folks. https://thingly.net/igotw/the-long-tail-of-sadness/ Thu, 17 Nov 2016 00:00:00 +0000 https://thingly.net/igotw/the-long-tail-of-sadness/ We all deal with tech debt every day. This inGraph speaks to that: The time scale: 2 years. 24 months of an all-but-defunct service just kind of lingering out there, all of its “important” functionality re-routed to the New Hotness but some endpoints left hanging. 730 days of “Well, but it does still cover this one use case so we need to keep it around.” 17,520 hours of potential for a wrecked oncall when it breaks in some weird way. https://thingly.net/igotw/load-un-balancing/ Thu, 10 Nov 2016 00:00:00 +0000 https://thingly.net/igotw/load-un-balancing/ I stumbled across some rad inGraphs recently by accident that I thought might be interesting to share. Yay, serendipity! I know I’ve posted _something _ similar before, but let’s take a look: This is traffic from identity (which I’ll generically refer to as ISB) to Espresso. It more or less maps directly to inbound QPS. I only chose this particular graph because ISB exposes multiple endpoints and there’s no good way to show total per-host QPS for all endpoints. https://thingly.net/igotw/spammy-loggers/ Thu, 03 Nov 2016 00:00:00 +0000 https://thingly.net/igotw/spammy-loggers/ [This week’s post comes from Jon Sorenson - Manager, Security SRE - whose username (jsorenso) is properly pronounced “j-so-and-so”. It doesn’t contain any actual inGraphs, but the visualizations he came up with in Kibana were too good to pass up. Thanks for the lovely bits of art, Jon!] I was recently looking into making some summary dashboards in Kibana around spammy loggers. I was looking for a breakdown of log level (INFO, WARN, ERROR) per service (Login-server, UCV, etc), so I created this: https://thingly.net/igotw/farewell-mountain-view/ Thu, 27 Oct 2016 00:00:00 +0000 https://thingly.net/igotw/farewell-mountain-view/ Here’s a rad inGraph: I have no reason for posting it other than the fact that it looks a little like a page being turned - sort of being peeled back from the upper righthand corner - and that seemed appropriate today. So just enjoy the pretty picture and don’t think twice about it. https://thingly.net/igotw/before-you-panic-split-fabrics/ Thu, 20 Oct 2016 00:00:00 +0000 https://thingly.net/igotw/before-you-panic-split-fabrics/ [This is the first of a periodic series of posts meant to address the following brand of question: “I saw something fucked-up in an inGraph. What else should I look at before I panic?”] Suppose you come across the following pair of inGraphs relating to your cache. The first from the perspective of the client: …and the second from the perspective of the cache itself: Okay, so there are a couple of points of inflection that sorta-kinda align, but the client thinks hit rate has bottomed out and the cache doesn’t seem to think the same thing…right? https://thingly.net/igotw/this-is-fine/ Fri, 14 Oct 2016 00:00:00 +0000 https://thingly.net/igotw/this-is-fine/ Earlier this week there was a Major GCN. The root cause was a configuration change that behaved in a way other than what was expected, and the impact was that Voyager was hard down in one fabric for about an hour. When bad things happen to the site it generally results in a whole mess of interesting inGraphs, and this time was no exception. First up, let’s take a look at l0’s view of the world during the timeframe of the GCN: https://thingly.net/igotw/patterns-the-trough/ Thu, 06 Oct 2016 00:00:00 +0000 https://thingly.net/igotw/patterns-the-trough/ _Flip The Plateau over and you have The Trough. Sometimes it’s expected behavior, as in the case of a traffic shift out of a particular fabric; I’ll leave finding _ an example of that up to the reader, since it happens almost daily. …but sometimes your metrics will “bottom out” in ways that you don’t particularly like. Perhaps a transactional email flow you care about was humming along just fine and then dropped to the floor: https://thingly.net/igotw/presidential-debate/ Thu, 29 Sep 2016 00:00:00 +0000 https://thingly.net/igotw/presidential-debate/ Brian Sherwin passed along a most excellent idea to me earlier today: ask folks how much they think tonight’s presidential debate would impact site traffic. …so that’s exactly what I’m doing. What do you think site traffic might have looked like during the debate? The Big Reveal will be on Thursday, same as always. (Note: Feel free to speak up in the comments and/or DMs on slack/email, but if you happen to know where to look to find the answer then please https://thingly.net/igotw/the-38th-inning-stretch/ Thu, 22 Sep 2016 00:00:00 +0000 https://thingly.net/igotw/the-38th-inning-stretch/ According to whatweekisit it’s the 38th week of the year…which has nothing to do with anything, but I’m selecting that as my arbitrary reason to give you a graph without any commentary at all. A hint: it’s related to a Major GCN from this week. There are related inGraphs which I can share upon request; hit me up if you wanna see ’em. https://thingly.net/igotw/unintentional-art-symmetry/ Thu, 15 Sep 2016 00:00:00 +0000 https://thingly.net/igotw/unintentional-art-symmetry/ After two pattern posts in a row I thought maybe we could take a look at some wonderfully-symmetrical inGraphs. A super-common example: The bytes come in, the bytes go out. Bonus: It looks a bit like an alligator! Along similar visual lines: This one may look familiar to regular readers and is intentionally designed to look symmetrical under “normal” circumstances. Also similar: This is a series of graphs that show a redline test in action. https://thingly.net/igotw/patterns-the-loner/ Thu, 08 Sep 2016 00:00:00 +0000 https://thingly.net/igotw/patterns-the-loner/ This post will contain some inGraphs that I’ve posted previously, mostly because I am lazy but at least partly because some of them can illustrate an interesting new facet I’d like to talk about. Let’s take a peek: IIRC Rylee Fowler sent me this one a while back. It’s been long enough that I don’t recall the specific circumstances, but I think what was going on is that an elasticsearch node was being removed from rotation and distributing all of its indices to other nodes. https://thingly.net/igotw/patterns-the-tighten-up/ Thu, 01 Sep 2016 00:00:00 +0000 https://thingly.net/igotw/patterns-the-tighten-up/ We don’t only sing, we dance just as good as we want (apologies to Archie Bell & the Drells). The Tighten Up is a pattern I see frequently. The specific instance I had in mind was garbage collection - un-collectable heap use increasing due to some event. For instance: Note the nice long slow regular GC pattern early on, followed by the sudden tighten up. A more drastic example: I know this one is a bit messy, but just bear with me for a minute. https://thingly.net/igotw/the-speed-of-light/ Thu, 25 Aug 2016 00:00:00 +0000 https://thingly.net/igotw/the-speed-of-light/ _I’ve posted about this kind of thing before, but I’ve gotten a lot of questions in the past week or two about how the Messaging (n_ée Inbox) data layer “works” so I thought it might be worth talking about a bit. I was asked recently “What is the name for this architecture?” and I did not have an answer to that question. It is not single-master (as we think of it). https://thingly.net/igotw/order-of-operations/ Thu, 18 Aug 2016 00:00:00 +0000 https://thingly.net/igotw/order-of-operations/ I was poking around my stash of inGraphs snapshots earlier today and I stumbled across one that I couldn’t immediately place: It took me a little while to recall exactly what had happened in the Ancient Past (lol at least 2 months ago) but I think I’ve pieced it together. First of all, the graph title gives a strong clue: inbound QPS to a comm-inbox-bps endpoint. The hostnames give yet another: these are all hosts in the prod- lsg1 fabric. https://thingly.net/igotw/unintentional-art/ Thu, 11 Aug 2016 00:00:00 +0000 https://thingly.net/igotw/unintentional-art/ I haven’t done a UA post in a little while so I’ve got a handful of interesting inGraphs for you today. First up, a little something Mark Feinstein sent my way: I’ve been kicking around some theories in my head about what’s actually going on here, but one way or the other it makes a nice piece of art. Next, a little inGraph I like to think of as The Caterpillar: https://thingly.net/igotw/catchpoint-edition/ Thu, 04 Aug 2016 00:00:00 +0000 https://thingly.net/igotw/catchpoint-edition/ Busted canaries aren’t exactly new territory for igotw, but this week’s post is a little different. As you may have already noticed - gasp! there is no inGraph - in this post! _Don’t worry, it’s not going to become a habit…but when Jim Ockers - one of our Edge Performance SREs - pointed out the scatterplot below I _ couldn’t resist the opportunity to share it. Let’s see what we’re working with here: https://thingly.net/igotw/hindsight/ Thu, 28 Jul 2016 00:00:00 +0000 https://thingly.net/igotw/hindsight/ In this week’s post I’d like to share a little personal embarrassment that will hopefully be a learning opportunity. We’ll start off with the inGraph: Those familiar with inGraphs will recognize this as being a very old snapshot. For those less-familiar: same bright green for current week, same weird pinkish color for previous week. It depicts the send rate of a high-volume email campaign. For (hopefully) obvious reasons I’ve somewhat affectionately entitled it “storky-no-sendy. https://thingly.net/igotw/more-lies/ Fri, 22 Jul 2016 00:00:00 +0000 https://thingly.net/igotw/more-lies/ I’ve posted before about how your metrics can lie to you. This post is another example. The inGraph: The metric shown here is phone-fe latency for Wonton (Messaging). The peaks correspond to deployments. Now, you might be tempted to say “Oh my God, that’s terrible! Why do deployments cause latency to go up so much?!” …and if this inGraph were telling the truth I’d be inclined to agree with you. But let’s take a look at what’s really going on here. https://thingly.net/igotw/cache-replacement/ Thu, 14 Jul 2016 00:00:00 +0000 https://thingly.net/igotw/cache-replacement/ For this week’s post I’m going to borrow from a CBVT newsletter article I wrote a while back. (I know it’s kind of “cheating”, but I’m on DTO this week and it’s quicker to re-hash an old post). The tl;dr summary: couchbase does not implement any kind of replacement policy, so if you don’t either (1) set a TTL on your records or (2) explicitly delete records when you no longer need them, then it will happily hold onto all of your data indefinitely. https://thingly.net/igotw/4th-of-july/ Thu, 07 Jul 2016 00:00:00 +0000 https://thingly.net/igotw/4th-of-july/ US site traffic on the 4th: Happy US Independence Day, folks! https://thingly.net/igotw/successful-canary/ Thu, 30 Jun 2016 00:00:00 +0000 https://thingly.net/igotw/successful-canary/ I’ve posted before about busted canaries that wind up getting promoted. This post is about a canary success story. Make no mistake, it was one sick bird… but it was caught and rolled back while limiting overall member impact. Check it out: This inGraph shows errors/second for the rollcall service. Notice that There. Are. Four. Lines. That’s because we have four production fabrics, and when we canary we deploy the new code to one node per fabric. https://thingly.net/igotw/compounded-complexity/ Thu, 23 Jun 2016 00:00:00 +0000 https://thingly.net/igotw/compounded-complexity/ Sometimes outages occur because of a single specific thing - someone made a mistake, something broke somewhere, perhaps something out of our control just plain Happened. Often it’s a little more complex than that - multiple factors that push us into a site-down situation. This post is about one such occasion. Factor #1: A GCN that should give you the heebie-jeebies because it has the words “data loss” in the title. https://thingly.net/igotw/euro-2016/ Thu, 16 Jun 2016 00:00:00 +0000 https://thingly.net/igotw/euro-2016/ I posted before about (American) football and its impact on site traffic. Now let’s take a look at what kind of impact what the rest of the world considers to be real football has: This is UK site traffic during the Euro 2016 second round game between England and Wales. The super-awesome thing about this: the UEFA site has a mi nute-by-minute recap, so you can actually align the timing of the 3 goals scored in the game with the sharp dips in site traffic. https://thingly.net/igotw/acquisition-edition/ Mon, 13 Jun 2016 00:00:00 +0000 https://thingly.net/igotw/acquisition-edition/ It has been a supremely weird day, folks. I haven’t the faintest idea what the future holds - I’m maintaining a standpoint of “cautious optimism” for the now - but what I can provide is a couple of interesting data points in the short term. The first is actually not an inGraph, but I’ll bet you’ve stumbled upon it: Step function changes in metrics are typically undesirable. Speaking strictly as a shareholder this one is decidedly desirable. https://thingly.net/igotw/patterns-up-and-to-the-right/ Thu, 09 Jun 2016 00:00:00 +0000 https://thingly.net/igotw/patterns-up-and-to-the-right/ There are some metrics you want to see go up and to the right. An example that I think will resonate: LNKD share price. That said, there are many times when up-and-to-the-right is something you do not want to see. For instance: This is what a memory leak looks like. Things are humming along just fine, and then some new code is deployed and memory use just climbs and climbs until the JVM seizes up in GC Hell, stops emitting metrics, and falls over altogether. https://thingly.net/igotw/fire/ Thu, 02 Jun 2016 00:00:00 +0000 https://thingly.net/igotw/fire/ Everything is breaking on the Internet all the time and this week was no exception. This post is not about that, mostly because it’s been a Long Week in spite of being a short week. Instead let’s take a look at some bitchin’ inGraphs that happen to look kinda like fire. First up: I entitle this one “login-server on fire”. It’s a stacked latency graph and I don’t understand that, but it sure does look nifty. https://thingly.net/igotw/troubled-canary/ Thu, 26 May 2016 00:00:00 +0000 https://thingly.net/igotw/troubled-canary/ This week’s igotw is fairly similar to the very first igotw, in which I talked about production code deployment at LinkedIn and how it can go badly. We saw along those lines this week: These are inGraphs of error rates. Canaries in all 4 fabrics pretty clearly show an increase…and yet the promotion still happened. How does this happen? Well, the first post I mentioned hearkens back to the Bad Old Days. https://thingly.net/igotw/unintentional-art/ Thu, 19 May 2016 00:00:00 +0000 https://thingly.net/igotw/unintentional-art/ [Thank you to everyone who contributed the inGraphs in this post. I don’t recall specifically who submitted all of these, but most (all?) of them were provided by someone other than me.] Just a few interesting bits of inGraph art this week. First up, some databus consumer abstract art: Next: Kind of makes me think of something one might see on the side of a van in the 70s. …and finally: https://thingly.net/igotw/the-number-s-of-the-beast/ Thu, 12 May 2016 00:00:00 +0000 https://thingly.net/igotw/the-number-s-of-the-beast/ _This week’s igotw comes from an incident that Kathleen Shannon and I looked at a while back. One of the services Katie supports - _guidednav-backend - had been running for months without incident, and then all of a sudden just up and fell over. She bounced it and it came back up, but we were curious: What the devil had happened? While we were poking around through metrics this graph jumped out: https://thingly.net/igotw/fallout/ Thu, 05 May 2016 00:00:00 +0000 https://thingly.net/igotw/fallout/ This week’s post is a follow-on to last week’s post about Mistaken Assumptions. It’s a bit of a study in the fallout that can happen post-incident. Let’s take a look at the inGraph: It’s worth taking a look at the snapshot and scrolling over the legend here. QPS showing up then dropping off, more or less at random…what fresh hell is this? Well, as it turns out there is a fairly serious kernel bug specific to the cgroup oom-killer that we’ve come across as a result of the migration to Rain. https://thingly.net/igotw/mistaken-assumptions/ Thu, 28 Apr 2016 00:00:00 +0000 https://thingly.net/igotw/mistaken-assumptions/ This post is about a Major GCN that happened earlier this week. It’s an unfortunate fact that site outages tend to provide super-interesting inGraphs, and this week’s incident was no exception. Take a look at this: This is an uptime graph. It’s worth spending some time to unpack what’s going on here. The dropoff around 16:15 is a canary promotion. It looks about like what one might expect - uptime drops off sequentially as the deploy rolls through each node and then starts moving up-and-to-the-right. https://thingly.net/igotw/unintentional-art/ Thu, 21 Apr 2016 00:00:00 +0000 https://thingly.net/igotw/unintentional-art/ Prince died today and I’m kinda torn up about it, so we’re going to take a look at a couple of inGraphs that really just look kinda cool. Thanks to _Loren _ Carvalho for this one, which I entitle “The Escalator to Nowhere”: I’m not entirely sure what’s going on here, but presumably this up-and-to-the-right load average trend was disconcerting to some Tool somewhere. Next up: _Sick. Thanks to Richard Waid and __Karrick McDermott - I’m not sure who first discovered it. https://thingly.net/igotw/failure-mode-redux/ Thu, 14 Apr 2016 00:00:00 +0000 https://thingly.net/igotw/failure-mode-redux/ This week’s igotw comes from Lester Haynes - Search SRE - who has pointed out some super-interesting inGraphs that illustrate stale search results. I know very little about the Search stack, but these inGraphs resonate pretty strongly with me because they mirror metrics I’ve seen in the past. First up: If this looks familiar then you’ve been paying attention - a very similar graph appeared in a _previous post. Latet SCN Age is the rough equivalent of kafka _ lag in databus-land. https://thingly.net/igotw/reverse-polish-notation/ Thu, 07 Apr 2016 00:00:00 +0000 https://thingly.net/igotw/reverse-polish-notation/ [Apologies for the relatively “thin” post this week; I’ve been at SREcon all day. Hopefully this awesome graph - and potentially discovering a feature of inGraphs you may not have been aware of - makes up for it. ] Today’s post is a two-for-one sale: both a beautiful inGraph and a useful feature of inGraphs. Let’s start off with the graph: As per the label this an edge cache uptime graph and appears to be two related processes in ATS. https://thingly.net/igotw/mystery-edition/ Thu, 31 Mar 2016 00:00:00 +0000 https://thingly.net/igotw/mystery-edition/ Another mystery! Apologies for the differing timescales and weird differences in aspect ratio (I suck at cropping). What’s going on here? What do these two metrics have to do with one another? What is the common thread here? All will be revealed on Thursday. Update: A quick point of clarification here: the timeframes do not line up at all because these two metrics have nothing to do with one another in terms of a specific incident/Thing that Happened. https://thingly.net/igotw/failure-modes-memcached-edition/ Thu, 24 Mar 2016 00:00:00 +0000 https://thingly.net/igotw/failure-modes-memcached-edition/ At LinkedIn we’ve migrated off of memcached in favor of _Couchbase for most of our caching use cases, but I thought this story might be useful in the _ greater context of how important it can be to understand how a particular piece of infra works and what assumptions we can make about it. I touched on this subject in a CBVT Newsletter a while back (which I will likely plagiarize for a future post) but I think it’s worthwhile to delve into some detail here. https://thingly.net/igotw/patterns-the-dreaded-plateau/ Thu, 17 Mar 2016 00:00:00 +0000 https://thingly.net/igotw/patterns-the-dreaded-plateau/ If you spend enough time looking at metrics you start to develop a sense for the patterns you do and don’t want to see. Consistency is desirable most of the time; unexpected points of inflection can be alarming (“It just went through the roof/dropped to the floor”). Things moving “up and to the right”/“down and to the right” can also be bad (“It’s fallen behind and isn’t catching up/it’s degrading over time”). https://thingly.net/igotw/is-it-plugged-in/ Thu, 10 Mar 2016 00:00:00 +0000 https://thingly.net/igotw/is-it-plugged-in/ [Today’s inGraph of the Week is a guest submission from fellow SRE Michael Kehoe*. Thanks, Michael! Reminder: folks can always submit graphs, posts,* anecdotes, or links about whatever metrics you think are interesting to igotw-submissions@linkedin.com] On August 28th 2015, there was a standard load-test of our LVA1 datacenter. Unfortunately the load test failed due to latency from the Identity Super Block (ISB). Doing a simple search on key dashboards show that there is some high thread pool utilization on identity-mt hosts. https://thingly.net/igotw/unintentional-art/ Fri, 04 Mar 2016 00:00:00 +0000 https://thingly.net/igotw/unintentional-art/ I had a recent conversation with a colleague about igotw. One of the things that came up was (paraphrasing based on recollection): “I love it when inGraphs unintentionally produces something that looks like an art piece.” I happen to have a few examples of this. I don’t necessarily recall all of the details of what each one actually represents, but for the most part these are screenshots of inGraphs that I took largely because they were aesthetically pleasing in some way. https://thingly.net/igotw/tradeoffs/ Thu, 25 Feb 2016 00:00:00 +0000 https://thingly.net/igotw/tradeoffs/ In this week’s post I’d like to focus on an example of something every engineer should be familiar with: tradeoffs. This is a screenshot I took of some inGraphs from early 2015. The metric being represented is latency from identity-mt to invitations. It’s a 12-month timeline of Ancient History so there is a bunch of stuff going on here (I actually named the snapshot “wow_lots_of_things”). Take some time to digest it - it’s a pile of information - and then we’ll break down some of the more interesting bits. https://thingly.net/igotw/lies-damned-lies-and-statistics/ Thu, 18 Feb 2016 00:00:00 +0000 https://thingly.net/igotw/lies-damned-lies-and-statistics/ This week I’d like to share a story about how metrics can lie to you. Take a look at this: This is the graph of the 95th percentile (p95) latency for pprofile-services. (If you’re unfamiliar with how percentiles work or why they might be useful feel free to either read up or ask me directly.) The course of events was something like this: A sudden drastic uptick in p95 latency set off alerts around 04:45, triggering an escalation to SRE https://thingly.net/igotw/the-superbowl/ Thu, 11 Feb 2016 00:00:00 +0000 https://thingly.net/igotw/the-superbowl/ While inGraphs give us insight into how our services are doing operationally, they can also show us what’s going on in the external world. Nights & weekends, holidays, major sporting events - all of these things can have an impact on site traffic and thus on site metrics. Even meal times can be seen, in some cases; in a future post I might compare how seriously the people in different countries take their lunch. https://thingly.net/igotw/mystery-edition/ Thu, 04 Feb 2016 00:00:00 +0000 https://thingly.net/igotw/mystery-edition/ In lieu of the normal Thursday post I thought I’d try something a little different this week - perhaps a bit of a challenge. Take a look a this: What’s going on in this graph? I’ll talk more about it on Thursday, but in the mean time I’d like to hear any discussion/guesses/conjecture in the comments about what you think it might be. Feel free to ask for additional details; I’ll provide what I can so long as those hints don’t give it away entirely. https://thingly.net/igotw/where-did-the-metrics-go/ Thu, 28 Jan 2016 00:00:00 +0000 https://thingly.net/igotw/where-did-the-metrics-go/ Sometimes the most interesting thing about a metric can be its absence. Take a look at this: I don’t have screenshots of all of the metrics that went missing for this particular service over this time period but they’re not necessary; they all fell off the map. Every metric this service was emitting just ceased to exist. The point of at which the metrics disappeared coincided with a deployment of the service, but at the time we didn’t really know why. https://thingly.net/igotw/failure-modes-kafka-client-edition/ Fri, 22 Jan 2016 00:00:00 +0000 https://thingly.net/igotw/failure-modes-kafka-client-edition/ [Note: There is how a go link for the inGraph of the Week! go/igotw*]* This week’s iGotW describes an interesting failure mode of a piece of technology that is heavily leveraged by LinkedIn: the kafka consumer client. Before getting too deep into the details I want to spend a moment talking about how awesome kafka is - partly because it is, in fact, awesome, and partly because quite literally every service I support uses it in some way and I would prefer to stay in the good graces of the world-class SREs and SWEs who support it. https://thingly.net/igotw/less-is-more/ Thu, 14 Jan 2016 00:00:00 +0000 https://thingly.net/igotw/less-is-more/ I ripped off this post’s title from a post that Aatif Awan did last July about how LinkedIn will be sending its members less email. This post really resonated with people (including Jimmy Fallon). It resonated with me in particular for a handful of reasons; here are two of them: I’m tired of telling people where I work and getting the question “Why do you guys send me so damn many emails? https://thingly.net/igotw/not-learning-from-mistakes/ Thu, 07 Jan 2016 00:00:00 +0000 https://thingly.net/igotw/not-learning-from-mistakes/ [This is the first in a weekly series of posts about interesting inGraphs I’ve stumbled across in my time as an SRE at LinkedIn] One of the things I love about inGraphs is how it can tell a story if you know how to look. Sometimes that story is one of victory, sometimes it’s one of sorrow and woe. Comedy, tragedy, drama…to co-opt a familiar saying, an inGraph is worth 1000 words.