Introduction
It has been over five years since I have added to this blog. The reason at the highest level is a predictable one – it fell from the top of my list of priorities in favor of new personal interests and professional demands.
But beyond changes in how much time I have had to write, another factor contributing to the prolonged silence has been that I’ve just been doing less good old fashioned tinkering with computers and the Internet for personal interest, and so my source material dried up. The sorts of investigations and projects I did through about 2018, like my demonstration of accelerating image-rich webpage load times using a technique that works fantastically, but that I never saw the industry adopt, or the months of time I spent building up my own redundant, Puppet-driven hosting platform, are the mainstay of my subject matter. Stopping that kind of hobbyist activity was not for lack of time so much as a depletion of spare technical and creative energy.
Tug on the string of why I stopped tinkering, and you touch on career change.
Since it largely accounts for the absence of new posts, and since there is really just as much to say about career as there is about code, I think it’s time for a series about my experiences working as a software engineer. After all, whether I believe it or not, the calendar says I’ve been doing it for 14 years now.
This will be the most personal I’ve gotten on this blog by far, and besides just talking of career growth and chronologies of employers, it’s also a good opportunity for some open and frank self-reflection.
When I set out to resume my blog with this post, I planned to mostly write about and reflect on my time at Pantheon, but that felt like telling a story from the middle. I set out to fill in the beginning, but quickly realized there was too much to say for an introduction to a post about something else. So, here I cover only what I took from my first two employers. I hope to follow this up with a post about a self-directed open source project I worked on some years ago and a post about Pantheon (my original plan 😂).
The employers
I’ve yet to actually reference many employers I’ve worked for anywhere on this blog. A quick overview of the whole lineup is a good overview to this series:
“EnergyCo”
I unfortunately have few good things to say about the company that offered me $job fresh out of college, which lasted from 2009 to 2013. Because of this, I have with some consideration opted not to identify them by name here, as protection from the unlikely possibility that this post would have otherwise provoked them to involve me in some kind of legal action. Let’s call them EnergyCo.
At least at that time, the company confined their developers to work in a very small ecosystem of languages and frameworks, some of which had no or few other users in the world. And although they were fundamentally a SaaS company, developers were not to think about, influence, or configure the third-party components that would be used in new products; you made it work on what you were given as best you could, and large swaths of standard SaaS platform tooling like observability or message queues were absent as choices entirely.
I don’t think these restrictions were intentionally put in place to trap new college hires (by far their preferred type of new engineering hire) into accruing a critical mass of non-transferrable skills, but even at four years there, I see now that I was starting to become heavily disadvantaged in my professional development compared to best-in-class peers in the broader industry. I did not appreciate it at the time – it was my first job out of college. With a few exceptions, as far as I knew, this was the normal and healthy technical experience for a SaaS professional. I attribute to pure good fortune that circumstances transpired when they did to motivate finding my next opportunity.
One useful outcome of my time here was that their woefully premature choice to promote me to management at 25 or 26, with both technical leadership and employee oversight responsibilities, taught me that I do not have managerial ambitions or a natural managerial acumen. I’ve been upfront with every potential employer who has asked since then that I would prefer not to help in this capacity. That year I spent in a managerial role was also the catalyst for my feeling the need to get out.
Some weaknesses I still have today as a developer were on full and unmitigated display here, and certainly the responsibility for the problems my team faced lie with me as much as with the aforementioned organizational factors. I have a tendency to prefer skipping over rough-draft, just barely usable designs of a product – these inevitably feel to me like either a waste of time if we’re going to do it differently later, or if it is never going to be improved, like I am contributing to something akin to the enshittification of everything. I think the jury’s still out on whether this attitude in itself is a bad thing. At the same time, when the thing I jump to right out of the gate is the product that perhaps should be built out after several years of usage and marketplace exposure, I am all but ensured of over-engineering for a particular problem before I have data to prove what the most important problems really will be.
In my final year at EnergyCo, I singlehandedly prescribed for my team a design incorporating a substantial list of open-source frameworks previously unknown to the company or any of the (even more junior) developers I was managing, along with a fairly heavy dose of overly DIY solutions for backend components, a result of the cultural limitations and my blind spots on server software choices. It was far too ambitious a plan.
When upper management eventually realized what was in progress was not going to work within their desired timeframe, they responded by retracting their support for using any new, open-source software tools, and restricted things back to the limited set of technology they had always built all their products with before. This felt like a betrayal to me. It was also a disagreement between me and upper management on top-level architectural design, a theme that would recur as a repulsive force to drive me away from an organization later in my career.
I hope this experience didn’t traumatize EnergyCo into solidifying their avoidance of externally developed tools. That would definitely not be the right ending to this story for their best interests, and they don’t deserve to continue to be haunted by the poor judgments of an ill-equipped young manager.
On reflection, it’s also clear to me now that there were some truly ugly culture issues. I was not really valued as a member of their workforce: I found this out when a personable new hire with no prior programming work experience and substantially less formal computer programming education volunteered to me her starting salary one day, and it turned out to be a figure that on her first day was substantially higher than mine after years of their employ. When I raised to my manager that it seemed likely my compensation was determined by factors other than the relative value I brought to their organization, the company’s response was to begrudgingly match salaries, have a vice president make an unnecessary comment directly to me about how I could probably cover some expense for the team now, and to march everyone into a room to inform us that there had been an incident with an employee compelling the company to adjust that employee’s compensation, and that henceforth employees were forbidden from discussing compensation amongst themselves – an attempt to cultivate a workplace environment that appears likely to be straight up unlawful at the federal and state levels. Contrasted with the beer taps and ping-pong tables also on offer in the technology industry, it’s a puzzle how they successfully retain anybody.
Employee wellbeing was not such a much, either: the culture encouraged and didn’t correct choices of mine like sitting at a computer for so long that I developed a seat-contact related health condition requiring medical treatment. Even worse, in that instance I continued to work anyway by staying at home and laying on my stomach or working on my knees at a low table. (Yeah, this really happened.) To be clear, nobody told me to do that, it wasn’t Musk “extremely hardcore” bad. Much of this behavior is down to my own lack of perspective and poor decision making at this stage of my career. But my lack of perspective made me easily influenced, and had there been more healthy systems of recognition for good work and proactive messaging about employee wellbeing, I likely would have made other choices.
MSI
The Minnesota Supercomputing Institute was by contrast a very welcome reprieve and culture shift, and in 2013 was exactly what I needed to pivot my career into a more suitable direction for me. I was fortunate to be hired: they required a Linux systems programmer, but I’d mostly been sitting on my butt writing web apps served off Microsoft servers I didn’t even have admin rights on.
To MSI, I looked like not an absolutely insane choice because they needed someone to babysit their Drupal website with custom PHP integrations into their accounting systems, and my tinkering and hobbying had led me to have informal experience in both Drupal and PHP. My enthusiasm for Linux also enabled me to accurately assert that I had been operating various Linux machines as headless servers in my spare time continuously since I was in high school, so I was at least not a total stranger to the command line. These credentials were probably not everything they dreamed of, but EnergyCo and public higher ed shared the feature of paying below industry rates, which helped out my cause insofar as their candidate pool was about 2.
To me, MSI was supercomputers, so that had to be cool, right?!? I actually had a bit of prior supercomputer programming fundamentals experience from college coursework, and one assignment from that course in particular remains the most memorable thing I did in college. Credit to the University of Minnesota system for making these machines available for certain undergraduate coursework, and to the UMD computer science department for putting it in their curriculum. With these fond memories, the chance to work at the facility where I had four or five years earlier run my hyper-optimized C MPI code definitely held a certain cachet in my mind.
More practically speaking, MSI was crucial to my professional development because it allowed me to pivot my skillset from writing proprietary applications that only ran in Microsoft Internet Explorer to professionally developing and operating Linux. A respectable cohort of highly capable engineers and sysadmins I worked with at MSI massively broadened my knowledge of the force Linux, and I owe them for the knowledge they imparted. While those folks doubtless could have gotten more lucrative jobs elsewhere, higher ed and direct support of scientific research is still able to retain some highly experienced and skilled professionals who are attracted to benefits beyond salary.
I think the Linux knowledge I absorbed from them turned into a long-term asset for me, as Linux is everywhere in SaaS, but isn’t universally mastered. Because it is so far down the stack, it is easy to avoid really having to learn much about it when you’re focused on this week’s sprint. But, as my Operating Systems professor once said, if you understand the capabilities, goals, and feature set of your operating system, you will be able to write code that works with your system. It’s a no-brainer, but there’s truth to it, especially with Linux’s built-in capabilities for containerization, security, filesystems, tracing, and more having become so expansive. The start of a solution to many problems nowadays is knowing which Linux feature to reach for.
MSI is a unique facility among its peers in higher-ed HPC in that compute resources are not necessarily allocated based on how many research grant dollars a lab can pay to the center, but instead based on an individual’s faculty status with a Minnesota institution of higher education and a review of the proposed research by a panel at the University. Sufficient funding to maintain respectable amounts of modern compute hardware - likely much more than they could afford through a lab grant funding model - is instead provided directly through a line item in the State of Minnesota’s annual budget. Basically any faculty member who asks (I forget at what rank) is guaranteed a base resource allocation. I understand that this model is even used as a recruiting tool when the University is courting faculty hires. But for all the academic independence and research freedom promised by this model, at a practical level it also creates a small thicket of administrative and bookkeeping challenges. Modernizing the automation of these was my primary task. But, since programming computers to automate bookkeeping tasks is an eminently achievable task, it did not demand unbounded amounts of my work output. I discovered in time that these tasks were really the only things they wanted. While there were plenty of interesting problems that I could find to work on, I think the Institute (understandably) didn’t want one or two developers baking any more bespoke complexity into their operations than necessary. The time eventually came where I had learned all I could in this role, and had given MSI all they really wanted from me.
There was also time at MSI to nurture my interest in the Drupal CMS. This led to an unsuccessful personal project, and also to the next chapter of my career. Both are good topics for future posts in this self-reflective series.