This year I read 30 books (just 2 in Spanish).

January

• Second Act: What Late Bloomers Can Tell You About Success and Reinventing Your Life
• Sprawball - The belated realization that 3 points are more than 2 points has dramatically changed basketball. This book has excellent data visualizations showing how much the game has changed. Personally, I mostly like the modern game, but basketball isn’t static and I thought he had some good suggestions for tweaking the rules. In particular, I loved the idea to make the corners a restricted area similar to the paint. Long ago we restricted the most valuable area (under the basket) so players can’t just stand there. I think it is reasonable to do the same for the corners. Getting rid of players standing still in the corner would be an improvement.
• What I Talk About When I Talk About Running - The discipline of his life is impressive. A tender book about accepting one’s limitations. His desired epitaph of “Novelist and runner. He never walked” is pretty great.
• La amiga estupenda (Dos amigas 1) - Spanish Figured I would be reading a translation from the Italian in any case and so might as well read it in Spanish. I could tell this was objectively excellent literature, but for whatever reason it didn’t really grab me.

March

• Wuthering Heights - I found the reading I did about the book after finishing it more interesting than the book itself. In particular, the multiple layers of unreliable narrators opens the book up to various very different interpretations. But solely as a story I didn’t understand the characters’ behavior.

April

• Abundance - I think we can have a society that grows richer and takes care of the less fortunate. I’d very much like to see a liberalism that builds.
• Revenge of The Tipping Point

May

• Strong Towns
• Thinking Basketball
• The Martians
• The Happiness Curve: Why Life Gets Better After 50
• How Infrastructure Works - The actual discussion of infrastructure was interesting. The author seemed to be having an internal struggle between an optimistic, abundance mindset, and a more conventional leftist pessimism about capitalism. When talking about concrete things I found myself agreeing with her. When she started talking about collectivism and the evils of the profit motive and how all things should just be provided to all people without costs I found myself rolling my eyes aggressively.

June

• Longitude
• How to See - I found this a bit disappointing. The photos and groupings were interesting. There were a few exercises sprinkled throughout that I thought would be good for developing mindfulness about the built environment. But I found the author’s relentless contempt for modernity frustrating.
• How to Watch Basketball Like a Genius - It was fine. Some good anecdotes. Wanted more from it.

August

• Mastering Private Equity
• Artemis - I love “The Martian” and “Project Hail Mary”. This wasn’t on the level of those but was a fine beach read.
• Hip-Hop is History - Listened to this as an audiobook as we were driving to the beach. Spending a couple hours with Questlove sharing his encyclopedic knowledge of hip-hop history is not a bad way to spend a drive
• Freedom’s Forge This was the optimistic story of America doing hard things that I needed. Immigrants like Bill Knudsen came to America, drove innovation by pioneering mass production in the car industry, and then applied those lessons to convert all of American industrial might to defeat the Nazis. We can do hard (and good) things together and we can do them via private industry.
• The Body: A Guide For Occupants
• Never Split the Difference

September

• Breakneck: China’s Quest to Engineer the Future -
• Triumph of the City: How Our Greatest Invention Makes Us Richer, Smarter, Greener, Healthier, and Happier Spanish

October

• The Nvidia Way
• The Origins of Efficiency - I’m a fan of Construction Physics so pre-ordered this book as soon as it was announced.

November

• Expensive Basketball
• Squirrel Seeks Chipmunk
• The Running Ground: A Father, a Son, and the Simplest of Sports - This was excellent. A story of a difficult—but loving—father and son.¹ Alternates memoir chapters of the author with chapters focused on the stories of other runners. The Julia Lucas chapter will stick with me forever (in particular the greek tragedy in a 5k of her 2012 Olympic Trials race)

December

• Babel - 200 pages in I thought this was going to be one of my favorite books of the year. Very well written. The magical system of silver was promising and the prose was excellent. But I was let down by the lack of nuance as the book developed.
• How Life Works: A User’s Guide to the New Biology - My level of knowledge of biology is whatever was in IB Bio 25 years ago. There is a lot we’ve learned that I didn’t understand. If nothing else, this book fully disabused me of the idea that DNA / the genome represnts a “blueprint” for life. So much of development is contingent on various physical and chemical processes that have nothing to do with information encoded in DNA.

It’s been 14 years, so I think we’re past the statute of limitations on this one. I’m also pretty sure nothing here qualifies as proprietary anymore.

One disclaimer: I’m working from 14-year-old memories, so I might get some of the details wrong, but I’m pretty sure I’ve got most of the story right.

The Setup

It’s late January 2011. I’m at LivingSocial. We’ve got a 12-person engineering team, $175 million freshly raised from Amazon, and are in the midst of the daily deals arms race with Groupon. They’re bigger than us, but we’re growing faster. We’ve just pulled off the largest one-day flash sale in history, selling 1.3 million Amazon gift cards in 24 hours. It took some heroic efforts to keep the site up, but we did it.

Then one day, our CTO grabs a couple of us and says:

“We’re going to run a Super Bowl ad.”

Cool.

“I’ve talked to people who’ve done this before. Here’s the pattern: there’s an insane burst of traffic for 30 to 60 seconds, then a gradual taper to a manageable level. That first minute, though? We’re talking 10–100x what we saw during the Amazon deal. We need to make sure we stay up.”

Less cool.

The Problem

We needed to:

1. Handle the traffic surge without taking the whole site down.
2. Make sure people could sign up for our email list and buy deals.

And we had some serious constraints:

1. Our hardware capacity was fixed. This was 2011. We were hosted on actual physical servers at Rackspace. No AWS, no GCP. Scaling meant calling someone to order hardware and waiting for it to be racked. We’d already pushed Rackspace’s capacity about as far as it could go.
2. Avoid saturating our network. We were legitimately worried about saturating Rackspace’s network ingress so needed to keep as much traffic as possible at the CDN.
3. We had about 10 days.

The Plan

We started kicking ideas around. I remember being inspired by how The New York Times handled live updates election night traffic with a Rails app that generated static HTML pages and pushed them all into their CDN with a short TTL.

Step 1: Make the Site Static

We decided to pre-render the entire site and serve it as static HTML through Akamai with a 5-minute TTL. One of my coworkers built a Rake task to generate static snapshots of every key page.

That covered browsing, but we still had two critical features that ultimately required a persistent write:

1. Email subscriptions – We needed to capture email addresses and cities.
2. Purchases – We needed to process orders and credit cards.

Normally, our architecture looked something like this:

Akamai → F5 → nginx → unicorn workers (Rails) → Resque/MySQL

But even with the static site, we knew our unicorn workers couldn’t handle another 10x of our peak from the Amazon deal—even if all they were doing was dropping requests into Redis for background processing.

Step 2: Lean on Nginx and Log Everything

Then someone (I wish I remembered who) had an idea:

“Nginx is a beast. Writes to disk are fast. What if we just log all POST params directly in nginx logs, then process them asynchronously later?”

I built a parser to read nginx logs and queue background jobs for:

• Vaulting credit cards with Braintree
• Creating accounts
• Logging purchases

We tested the hell out of it on staging. Then we did a small production run during off-peak hours. Everything worked.

Game Day

On Super Bowl Sunday, we were all in the office tired but full of adrenaline.

20 minutes before the ad aired, we regenerated the static site and tested it on a subdomain. Everything looked good.

2 minutes before the ad aired, we flipped the switch. The live site became a static site. Right before doing this, we cleared the nginx logs so they wouldn’t contain test data.

The ad ran.

Traffic spiked. No 500s. Everything looked good at the load balancer.

I was SSH’d into our production application servers and checking the logs.

Nothing.

The log files were empty.

My stomach dropped.

The Mistake

Nginx buffers logs in memory before writing them to disk, so there is often a slight delay before the logs are written to disk. But we should have been seeing data.

We weren’t.

I raised the alarm. We started digging. Then I heard a string of curses.

When we “cleared” the logs before the ad ran, we deleted the log files instead of truncating them.

Under normal circumstances, this wouldn’t be a huge deal—you just restart nginx, and it creates new log files. But we’d forgotten to restart nginx. So it kept an open file descriptor for the deleted file kept writing to a file that no longer existed, meaning everything was effectively going to /dev/null.

We’d just lost all the traffic data from those critical minutes.

The Aftermath

Within five minutes, traffic had dropped to a manageable level, so we switched back to the normal site.

But the logs were lost to the ether.

There was nothing to be done.

I poured myself a very large bourbon. My coworker who built the static site generator poured himself a very large tequila.

By halftime, I’d left the office to meet my wife at a Super Bowl party.

The (Sort of) Silver Lining

The traffic spike wasn’t quite as big as we’d feared or hoped—it was close to, but not as big as—our peak during the Amazon deal. Ironically, we could have just run on our normal system.

Losing that five-minute window cost us a few thousand purchases and subscriptions, which wasn’t great, but wasn’t catastrophic. Super Bowl ads aren’t really direct response plays—they’re brand plays. The purchases would have been a bonus.

The ad itself? Not great. I saw it for the first time the night before. Doesn’t age well and even at the time I didn’t feel great about making a trans person a punchline. Still, it managed to be get less backlash than Groupon’s, and we did get a ton of earned media mostly just about the fact that we were big enough to run a Super Bowl ad.

Would I call it a good return on ad spend? No. But screwing up five minutes of logging didn’t materially change the outcome.

The Takeaway

Even after all these years, I’m still incredibly proud of what we built. Five people, 10 days, and a system that could have handled 100x our normal traffic.

It should have worked.

It would have worked.

If not for you meddling kids failing to properly rotate nginx logs.

They recently received the updated 2025 ICD-10 codes which is a coding system of ~74,000 codes that are used by healthcare providers and insurance companies for billing, tracking public health trends, and medical recordkeeping.

Basically, they got handed a plain text file with 74,260 lines and were asked to annotate the data with various additional bits of information. For example, they want to record whether each ICD code represents an infection:

• A200 Bubonic plague is an infection
• Y9366 Activity, soccer is not an infection

Annotating more than 74,000 lines is tedious and time consuming¹, so he hit up the group chat asking about how to script an LLM to do this.

This seemed like a fun toy problem to try out some stuff I personally hadn’t done yet, so I decided to see how well I could use LLMs as a binary classifier for “infection/not infection”

Attempt 1 - Scripting local LLMs on my M1 MacBook Air

I’d been wanting to play directly with some of the open weights LLMs like Llama, Mistral, Qwen, and DeepSeek and this seemed like a good excuse.

I installed Simon Willison’s llm command line tool and the gpt4all plugin and started pulling down models and doing some quick testing with a few.

(Embarrassingly, I wasn’t able to figure out how to get any of the Llama models to work)

qwen2-1_5b-instruct-q4_0 didn’t really follow instructions, outputting a bunch of the prompt and making up some additional codes that weren’t in the input.

But mistral-7b-instruct-v0 at least followed the instructions and output in the right format, so I went down that path a bit. Here is the command line invocation I used:

~/src/icd10cm-codes-classifier > shuf -n 5 /Users/patrick/src/icd10cm-codes-classifier/icd10cm-codes-2025.txt | llm -m mistral-7b-instruct-v0 --system 'You are an expert in medical coding. You are given a list of ICD-10 codes and their corresponding descriptions. Your job is to annotate each line with an additional column. If the ICD code and description is for an infection then add TRUE. If it is for something that is not necessarily an infection add FALSE. Output the icd code, the descriptionand the TRUE/FALSE column. DO NOT ADD ANYTHING ELSE TO THE OUTPUT.'

This is pulling 5 random codes out of the file and prompting mistral to classify them.

And here is the output:

T18120S Food in esophagus causing compression of trachea, sequela TRUE
G501    Atypical facial pain FALSE
H18521  Epithelial (juvenile) corneal dystrophy, right eye FALSE
S72111J Displaced fracture of greater trochanter of right femur, subsequent encounter for open fracture type IIIA, IIIB, or IIIC with delayed healing TRUE
M84756D Complete transverse atypical femoral fracture, unspecified leg, subsequent encounter for fracture with routine healing FALSEException ignored in: <_io.TextIOWrapper name=7 mode='w' encoding='UTF-8'>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/llm/0.20/libexec/lib/python3.13/site-packages/llm_gpt4all.py", line 294, in __exit__
    sys.stderr = self.original_stderr
OSError: [Errno 9] Bad file descriptor

It “worked”. Sort of. A couple of problems:

1. The program blew up with an exception. This was common across the LLMs I was running with llm, and I didn’t take the time to to properly debug it, because…
2. Two out of the five classifications were obviously wrong. I’m not a documentation nurse, but I know that “Food in esophagus” is not an infection.

So this wouldn’t be actively useful.

Attempt 2 - Naive Scripting with OpenAI API

It was getting late and I was frustrated that I hadn’t gotten anything useful for my friend yet. So I decided to throw a little of money at the problem and use GPT-4o.

I fired up cursor and used composer to prompt for a python script that would iterate over the ~74,000 rows in batches of 100 and then call to the OpenAI API with a prompt that I provided.

Success! (mostly)

I got responses back, and from spot checking they seemed mostly right.

There were just a couple of problems:

1. I dropped some data. I ended up with a file with 73,509 rows vs. the 74,260 rows I should have had. I’m not sure what was happening here, but if I was doing this for real I could easily do some integrity checks on each batch to make sure the output ICD codes match the input.
2. It was super slow. I did this in the most naive way possible of just doing sequential API requests and waiting for each one to finish before starting the next. This was a quick experiment so I just went to bed and let it run overnight, but I could easily parallelize each batch and make this much faster.

Still this was good enough for a quick weeknight proof of concept so the next morning when the job finished I sent the file off to my friend.

Total cost: $19.94

• gpt-4o-2024-08-06, cached input - $0.13
• gpt-4o-2024-08-06, input - $3.76
• gpt-4o-2024-08-06, output - $16.05

All the code is available on github.

Next Steps

Now that I’ve got this little toy test project, I’d like to try a few more experiments on top of it. Look for them in a future part 2:

• Play with batch sizes. I picked the 100 line batch size out of thin air. I’m no where near the per request token limit. But I also wonder if I’d see any impact on accuracy with larger batch sizes.
• Use the OpenAI batch API - this will ~halve the cost. And while there is a 24 hour SLA on batch requests I bet this will end up being significantly faster by having
• Try other models.
  • Compare GPT-4o and 4o-mini - The task is relatively simple, so I’m willing to wager that 4o-mini will also be able to function as a classifiers.
  • Access other models in the cloud. My personal laptop only has 16GB of RAM which limits what it can run. But I can play with
• Implement voting - run the LLM “classifier” multiple times. Store each of the results. Possibly call multiple different models and see where they agree and where they disagree. Places of disagreement can be used to highlight codes for manual review.

Looking back, I noticed a couple of themes in my reading this year. One was biographies of people who had achieved success and then figured out how to change so that the drive that propelled their success didn’t eat them from the inside (Becoming Steve Jobs, Pep Guardiola)

Another theme was books that prompted some reflection that I would read (or listen to) in the morning as I got the kids ready for school (What We Owe The Future, Not The End Of The World, A Decent Life)

And then there is my normal mix of trying to better understand the world and pick up additional tools for functioning in it (The Laws of Trading, Thinking in Systems, Sludge, Noise, Stories that Stick). I’m happy with those books if I take away one mental model or skill that I can use.

January

• What’s Our Problem? - I love Tim Urban’s writing at “Wait But Why?” He has an inherent curiosity and explains things in a way that I find persuasive. And this book did spur me to action in that it convinced me to change my monthly donation from the ACLU to FIRE. It also succeeded in making me a bit more worried about illiberal tendencies on my side of the aisle. But I walked away pretty unsatisfied because I thought he got stuck on a false equivalence between the illiberal left (who shout a lot on the internet but are not representative of mainstream left and center left political power in the US) and the authoritarian right (who are very much represented in the mainstream of the Republican party). I remain a committed advocate for enlightenment beliefs.
• Verbal Judo

February

• What We Owe The Future - I started this book expecting to be a bit frustrated because the longtermism movement within EA seemed so dogmatic. My prior was that we should value the future more than we do today, but less than the longtermism movement within EA advocated for. I thought the book made compelling arguments and was more reasonable than I was expecting based on the longtermism arguments that had filtered out to me through the internet. I’ll say that it was successful in convincing me to update my prior to care even more about the future than I did before I read the book, but still not with the certainty that McCaskill was arguing for.

March

• Becoming Steve Jobs - This was a re-read, but I’d forgotten I’d read it until I was 2 chapters in so kept going. Not perfect but so very much better than the Isaacson biography which I still think is one of the worst books I’ve ever read and such a missed opportunity.
• Not the End of the World: How We Can Be the First Generation to Build a Sustainable Planet - I see a sense of doomerism amongst some of my friends that I find frustrating. I’m not pollyannish, I think that we face many serious environmental problems. But I think it is demonstrably inaccurate to think that everything is getting worse and we face extinction. This book was an excellent fact-based antidote to the thought that humanity can’t thrive while preserving the planet.
• Pep Guardiola: Otra Manera de Ganar - I liked this. I want to read an updated version with more information on how he has evolved in his time at Man City so that he can sustain greatness instead of working himself to exhaustion and near the point of physical collapse like he did at Barcelona.

April

• A Decent Life - Much of philosophy focuses on how to live an ideal life. I enjoyed this book’s more grounded ambition: how to live a decent life. I know I’m deeply flawed, but I aim to be a slightly better person each year. Reading a little of this each morning was a good way to keep that goal top of mind.
• The MANIAC - This book is incredible. It is a biography of John Von Neumann but it is told through a series of fictionalized first person narratives of people who knew him. It also is a metaphor for the coming of artificial super intelligence. Highly, highly recommended.

June

• The Song of the Cell
• The Laws of Trading: A Trader’s Guide to Better Decision-making for Everyone - The thing I took from this book was that there are several potential sources of “edge” (a sustainable advantage) and you need to be clear-eyed in evaluating what edges you may have.
• The Time-Travelling Economist: Why Education, Electricity and Fertility Are Key to Escaping Poverty - As I’ve taught my children “the past is a terrible place” and I wouldn’t want to have been born at any other time or any other place in human history. My hope is that we can continue to expand prosperity so that more of humanity can be as fortunate as me.

July

• El futbol a sol y sombra - For a classic book about soccer was a bit of a slog for me. It isn’t that it wasn’t good—it was very good—but for some reason it just took me several months to work through it. Two things stick in my memory. First, this is the first time I learned that the position of goalkeeper is sometimes referred to as “cerbero” (cerberus) and as a goalkeeper I think that is amazing: because we guard the gates of hell! Second, the running joke in the recap of world events before each World Cup from 1966 to 2006 that “Fuentes bien informadas de Miami anunciaban la inminente caída de Fidel Castro, que iba a desplomarse en cuestión de horas.” cracked me up.
• The Subtle Art of Not Giving a F**k - Read it in Spanish as a quick read while we were in Korea as a palate cleanser after El Futbol A Sol Y Sombra. It was fine.

August

• Red Mars - I loved this book when I read it in middle school. I’m not sure I’d finished the entire trilogy as a child because the later books get pretty deep into the politics of forming a socialist utopia. That part still didn’t really speak to me, but on this re-read I was struck by how much I’m desperate for advances in materials science (space elevators built from carbon nanotubes extruded from an asteroid! piezoelectric films that can tent canyons on mars and generate electricity from wind!) and biological science (the longevity treatment!). Also, no one can write descriptions of landscapes quite like Kim Stanley Robinson. It is so vivid I feel like I’m there. I was inspired to re-read these by Casey Handmer and enjoyed his accompanying technical analysis.
• Poor Charlie’s Almanack - Collection of essays and speeches by Charlie Munger. A very clear thinker. An edition by Stripe Press.
• Thinking in Systems: A Primer - A little dry but solid. I feel like this is descriptive of how I think but giving a vocabulary to it has been somewhat useful.

September

• Green Mars - See above.
• Blue Mars - See above.
• Sludge: What Stops Us from Getting Things Done and What to Do about It - Sludge is like the evil twin of “Nudge” (I don’t think the rhyming names were an accident). Good reminder to be on the lookout for unnecessary friction that gets in the way of achieving what we want.

October

• One Giant Leap: The Impossible Mission That Flew Us to the Moon - I really enjoyed this. I like to be reminded that we can do great things. It also inspired me to watch the 2019 documentary Apollo 11 which was stunning. I grew up loving space. But I also was born after our last trip to the moon. This book made me think of all the thousands of people who worked on Apollo and didn’t know if it was going to work. And that made me think of one of my favorite pieces of writing from the last several years: 22 Goals. And particularly the episode about the 1970 World Cup and the part about Pele bursting into tears on the way to the final and trying to hide it by crouching down to look for a rattle and how “If one moment can change everything, the people who create that moment are living through it in the present. They don’t know the outcome in advance. The greatest soccer team ever assembled can also be a bunch of guys dealing with their nerves by singing on a bus on the way to the biggest game of their lives.”

November

• How to See - I don’t really know anything about art, but I enjoyed this collection of essays from painter and art critic David Salle. There were times that I didn’t have the context to fully follow what he was talking about, but for the most part I feel like I walked away with a greater understanding and appreciation for the practice of creating art. And I learned I really love Alex Katz’ nature paintings.
• On the Edge: The Art of Risking Everything - I enjoy Nate Silver’s thinking and writing. I liked this as a book more than I remembered liking The Signal and The Noise. There is value in taking risk and understanding the world. I do think he fell a little too in love with is Village and River metaphor but he hit on a meaningful cluster of traits that don’t exactly map onto a traditional left-right view of US politics

December

• Noise: A Flaw in Human Judgment - Key takeaway: Keep your facts and your values separate when making decisions.
• Stories That Stick - The structure of a good story is Normal, Explosion, New Normal and the key elements are having identifiable characters, authentic emotion, a significant moment (this is the explosion), and specific details. This was a book about a craft or engineering, not art. The goal was to help you use stories to achieve some concrete goal. A successful story is one that achieves that goal. A story that doesn’t achieve the goal isn’t successful, but can probably be tweaked to achieve the goal.
• The Portuguese: A Modern History - I’m going to Portugal later this month to see a Champions League match between Benfica and Barcelona. I’ve never been to Portugal (despite having lived in Spain) and realized I knew next to nothing about the country.
• Probably Overthinking It - Fun examples of statistical thinking and several counterintuitive traps people fall into.
• Why Machines Learn - Very good history of the development of machine learning. Unlike other popular science books this didn’t shy from walking the reader through the math that underlies machine learning. It’s been a long time since I took Calc so I’ll admit I can’t really do the math anymore, but I do at least still remember the concepts so I can follow along. There wasn’t a ton new here after I did my deep dive a few years ago, but the discussion in the last chapter of how massive neural networks don’t overfit and we have no satisfactory theory to explain why was completely new.

Took me long enough to write this one up :)

January

• The Last Days of Roger Federer: And Other Endings

February

• Blindsight - This book is bonkers (in that it posits a world where Vampires are real) but was very interesting and made me reflect on the nature of consciousness.

March

• Four Thousand Weeks: Time Management for Mortals - A philosophical meditation on how to ensure you accomplish the important stuff and enjoy your life from a self-described “recovered productivity obsessive”.

April

• Dopamine: The Molecule of More - Spanish translation
• Hackers - Never read this before. Pretty cool stories from the people who created so much of the culture of programming.
• Super Pumped - Spanish translation - I knew a lot of the story but not all of it. There is something resembling a greek tragedy in the story of Uber in that the things that made Travis Kalanick uniquely suited to lead Uber to success when they were upstarts combatting evil taxi monopolies (insane drive, pathological competitiveness, and a willingness to flout regulations) is also what made him uniquely ill-suited to lead a multi-billion dollar global company. This was fun to read in Spanish and pick up a bunch of startup vocabulary.
• Echopraxia - Sequel to Blindsight. I enjoyed it significantly less than Blindsight.

June

• The AI Revolution in Medicine - A team of medical doctors and researchers got early access to GPT-4 to see how it affected care. This mirrored a lot of my thoughts about the possibilities and limitations of generative AI. I am convinced widespread AI will be a society altering change, probably a bigger change than the adoption of personal computers and the internet.
• Project Hail Mary - Spanish translation - I loved this book so much.

August

• The Creative Act - Rick Rubin is a weird and uniquely talented dude. The book sometimes veers into faux mysticism that I’m constitutionally allergic to, but also contained some really good prompts to unblock thinking.
• What’s Our Problem? - I love Wait But Why. Tim Urban makes a lot of very good points as he tries to figure out how societal discourse seems to have gone off the rails. I also think he oversimplifies quite a bit and falls victim to false equivalencies in several places. We should all aspire to consistently be high rung thinkers.
• Action Hero Scouting Report - Shea Serrano is one of my favorite writers and I enjoy how much fun he’s having with little projects like this.
• Do Hard Things: Why We Get Resilience Wrong and the Surprising Science of Real Toughness

October

• Going Infinite: The Rise and Fall of a New Tycoon - FTX is another story I’d followed and was fairly informed on but there still were a lot of absoluely amazing stories in here. I feel like Michael Lewis wrote this about as sympathetic to SBF as possible and I still read it as “he committed massive financial fraud”

November

• Los Detectives Salvajes - Spanish - I spent 4½ months with with this book. I went into it pretty much entirely blank with only a vague recollection that Roberto Bolaño was a big deal when I was in Spain in 2003, died young, and occupies sort of a Spanish language David Foster Wallace niche in my mind. I won’t attempt to summarize the book, but it is very, very good. I’m glad I read it and am confident it’s going to stick with me for a long time.

December

• Resumen de Historia del Ecuador - Spanish - We were going to Ecuador and I didn’t know anything about the history of the country so I found this 200 page cliff’s notes history written for Ecuadorean high school students.
• The Savage Detectives Reread - I had a feeling I was missing quite a bit so sought out this academic English language book to get deeper context.
• On the Origin of Species - We were going to the Galapagos, and I’d never actually read Darwin so I figured now was the time. Victorian prose is rough, but it is amazing how much he got right. I enjoyed this series of an evolutionary biologist blogging through reading the book in 2009.

After an intense and successful Black Friday / Cyber Monday period, most of the team was looking forward to taking a well-earned rest around the end of the year. We were confident that our on-call rotations and monitoring would let people mostly unplug and recharge, while knowing that if anything went wrong, we’d know about it.

Unfortunately, when we came back from a week off, we realized that the buyer-facing feature had essentially been disabled for a week and we didn’t know.

What happened?

The upstream team got nervous about their ability to support the new system and turned it off before they went on break. This could have been a valid business decision, but unfortunately they forgot to tell us. So we were surprised to learn that this new and valuable feature was essentially disabled for a week.

How did we miss it?

We did have monitoring in place.

We had alerts configured to page us if the job that processed the messages failed. And we had other alerts configured that would page us if there was a backup of messages.

What we didn’t have was an alert that paged us if no messages were sent.

The Lesson

When setting up comprehensive monitoring you must monitor both for the absence of success as well as the presence of failure.

Monitoring for the absence of success is monitoring for when something you expect to happen didn’t happen as opposed to only monitoring for when an unexpected failure did happen.

I’ve found that teams naturally monitor for failures. The first alerts that get set up are generally for spikes in 500s, errors, or failed background jobs. And those are good! But they aren’t sufficient.

The goal of monitoring is to have confidence that your system is working as expected. To do that you need to monitor whether the key outcomes of your system are succeeding.

Common Challenges

Monitoring At The Right Level of Abstraction

I find that when monitoring for the absence of success it is important to define the success at the right level of abstraction. In the instance above we could have directly monitored for the number of messages successfully processed.

But we might have been better off monitoring at even an even higher level of abstraction (the number of times we displayed the buyer facing feature)

Monitoring at higher levels allows you to catch more issues with a single monitor. For instance, monitoring that the expected number of messages have been processed would have caught the issue where the upstream system stopped sending messages, but it wouldn’t have caught an issue where we failed to display the buyer facing feature. Monitoring the number of times we displayed the buyer facing feature is better because it would have caught an error in any part of the system and also is monitoring the action that directly created business value.

Seasonality

Many systems have a natural seasonality of usage within a day. If you run a service with overwhelmingly North American user base there won’t be a lot of usage at 4am ET / 1am PT.

There are a couple of ways to handle this. The simplest is to configure your monitoring to only alert during times of day where there is generally enough usage to confidently detect abberations.

High Cardinality of Data

High cardinality refers to having a large number of groupings of values in a dataset. In the context of monitoring, it can make detecting specific anomalies more complex due to the sheer variety of potential failures.

For example, at LivingSocial, we negotiated different rates for each of our business lines in various countries, resulting in a unique merchant account for every country and business line. This high cardinality made it challenging to monitor success rates for specific combinations like travel purchases in New Zealand, which were rare and therefore too noisy to detect changes in success rate in the same way we did for higher volume merchant accounts.

One way to manage high cardinality is to set up less granular monitoring that checks for the presence of success over a longer period of time. For instance, after an issue with the configuration at our payment gateway led to a days long outage of the New Zealand/travel combination I set up a monitor to check that there had been at least one successful transaction in the last 12 hours for the New Zealand/Travel combination. This might be slower to a alert than the finer grained alerting we had in higher volume countries but it was effective in detecting anomalies within highly granular data without leading to false positives.

Conclusion

Effective monitoring goes beyond merely watching for failures. Remember, it’s not just about what could go wrong; it’s also about ensuring what should go right, does. Your users, your team, and your bottom line will thank you.

January

• Enemy of All Mankind - Interesting and relatively light read of piracy and the world’s first international manhunt at the dawn of modern nation states.
• Soccermatics: Mathematical Adventures in the Beautiful Game - It was fine. I have a much better soccer book later in the year though.

February

• Dilla Time - This is one of the best books I have read in the last 10 years. It is about J.Dilla, possibly my favorite producer of all time, but it is also about the history of Black music in America, technological development and disruption, the entire neo soul movement, and the cost of enabling a single minded obsession. This book left me speechless.

March

• Cien Años de Soledad - This was a bucket list book for me. After I got back from Spain and my Spanish was at its peak I started this book. Then the semester started back up, and I had to set it down. Then on the next break I tried again. I’d probably read the first 75 pages 3 times, but had never gotten through it. This time I did. Technology helps: I’d read the kindle book, while listening to the spanish audio book with Audible. Then after finishing a chapter in Spanish, I’d read it in English, then re-read it in Spanish. It’s a difficult book and probably wasn’t a wise choice for my first foray back into reading in Spanish but I’m glad I did it.
• The Scout Mindset: Why Some People See Things Clearly Others Don’t - Essentially a book long exploration of how to defeat confirmation bias.
• Scorecasting - This was a fine beach read but nothing special. It had a “freakanomics but for sports” vibe but wasn’t as engaging as I remember Freakanomics being (althogh it has been 15+ years since I read that)

April

• Origins: A Genetic History of the Americas - Using ancient DNA to trace human migration to the Americas.
• Upheaval: Turning Points for Nations in Crisis - The stories of how different countries responded to crises in different ways were interesting. The attempt to construct an underlying theory for how nations respond to crisis from an N of ~8 was… a stretch.

May

• The Most Fun I Never Want To Have Again: A Mid-Life Crisis in Community Banking - A first person narrative of launching a community bank on the outskirts of Atlanta in the run up to the 2008 financial crisis.

June

• Talent: How to Identify Energizers, Creatives, and Winners Around the World - Interesting and thought provoking read. But I was left with the feeling that the specific techniques suggested by the authors wouldn’t necessarily generalize to normal recruiting without some additional guardrails. Tyler Cowen is a professor and a lot of his interviewing is part of grantmaking and Daniel Gross is an early stage VC who is interviwing founders. In both of those scenarios you’re seeking outlier performers and high variance is fine (it is fine if most grants / investments don’t pan out if you find one or two “hits”). But in interviewing at a normal company, even a top notch company with very high standards, your downside risk is higher.

July

• Locos, Ricos, and Asiáticos - After Cien Años de Soledad I needed something lighter so went with the Spanish translation of Crazy Rich Asians. Wasn’t quite as easy as I was hoping for as there is a ton of luxury goods vocabulary that I didn’t know. Fun read though overall. I had enjoyed the movie, but, as always, the book is better.
• Hábitos Atómicos - Now this was more my speed. Turns out that mass market business / personal development books are written with relatively clear language so this was a way easier read. Also a good book. Emphasis on
• Rome: An Empire’s Story - We went to Italy in July and I like to learn a bit about the places I went. This wasn’t a comprehensive history of the Roman Empire, but was more a history of the development of the concept of empire.
• The Florentines: From Dante to Galileo: The Transformation of Western Civilization - This I found more compelling. It is a history of the emergence of the Reannaisance in Florence focused on specific people.

August

• Extra Life: A Short History of Living Longer - A look at how we have doubled global life expectancy from ~40 in 1920 to ~80 today.

September

• The Anarchy - A history of the East India company and more specifically the 100 year period of chaos following the East India company’s toppling of the Mughal empire. Almost everything in this book is insane, from a private company essentially acting as sovereign of one of the largest nations on earth to descriptions of torture more depraved than anything in Game of Thrones.

November

• Net Gains - I’m a sucker for analytical soccer books. I really enjoyed that this book was written by an American who came up in our youth development system and played through college. Good overview of the evolution of an analytical understanding of the game. Not a ton here that is new if you’ve read “Football Hackers” but I thoroughly enjoyed it and highly recommend.

December

• Mindset: La Actitud del Éxito - I’ve been familiar with the concept of Growth Mindset and have found it a useful tool for thinking about how I approach challenges and how I attempt to steer my children. The book is a bit of a mixed bag, particualrly because I was already coming convinced that Growth Mindset was a real thing (they even did a well [designed replication study(https://marginalrevolution.com/marginalrevolution/2018/03/growth-mindset-replicates.html)]). There are some good examples of how to phrase praise to. But I think the book at time veered into labeling anything the authour liked as “Growth Mindset” and anything they didn’t like as “Fixed Mindset”. In particular, I found the chapter arguing for Jack Welch—famous for living in an $80,000/month GE owned apartment even after he retired—as a down to earth, man of the people, Growth Mindset CEO boderline hilarious because he cut the executive dining room. Reading it in Spanish I learned a lot of synonyms for “challenge”

i simply want a protocol that embeds password requirements into the form field html and that password managers can use to create a matching PW

What follows is a running diary of my thoughts for the next five minutes where I:

1. Get super excited about her idea
2. Figure out how it could work technically with current standards
3. Realize why it wouldn’t do much good in the real world

0m:01s — That sounds amazing! I hate when the password 1Password generates is rejected. How would I implement that?

0:10 — Wait, doesn’t HTML5 have support for client side validation built in? Wouldn’t that cover it? I bet I could figure out how to make this work and impress my friend.

00:30 — It does! There isn’t a full protocol, but HTML5 allows for specifying regex patterns that an input is required to match. There’s even a w3schools tutorial on how to set up regex validation for a password field complete with an example:

<input type="password" id="psw" name="psw" pattern="(?=.*\d)(?=.*[a-z])(?=.*[A-Z]).{8,}" title="Must contain at least one number and one uppercase and lowercase letter, and at least 8 or more characters" required>

0:35 — OK, so I’ve got a regex, what would do I need to know to make sure that my generated password matches?

0:45 — Here are the options for a new password in 1Password:

1:00 — Alright, I think I just need to know four things:

1. Is there a minimum length? But this doesn’t seem very important because it is unlikely your password manager will try to generate a password that is too short.
2. Is there a maximum length? This one matters. This is probably the most common failure reason when 1Password generated passwords exceed the arbitrarily short password lengths that some sites mandate.
3. Are there any characters that are forbidden? This is the next most common reason I’ve had a generated password fail. Normally when there is a blocklist of certain symbols.
4. Are there any classes of characters that are required “Must contain at least one number, one lowercase letter, and one capital letter”. Won’t worry about this one, because I haven’t seen it prevent a generated password from working.

1:30 — So… how do you extract that information from an arbitrary regex. Well, I’m sure that there are smart ways to do this, probably something with an AST of the regex?

2:15 — But “perfect is the enemy of the good” and I can already think of two pretty dumb ways to do this so I don’t need to use my CS degree today:

1. mostly dumb: I could probably parse out just the max length and any characters that are globally blocked fairly easily.
2. really dumb: When generating a password I could just feed the regex in, try a few variations with my current rules, then loosen the rules if it fails, and try again. Generate 5 random passwords, if they all fail remove the requirement for symbols, generate 5 more, if those all fail shorten the length by one, repeat until something works or we hit an arbitrary minimum size. Voila!

2:45 — Alright… I feel like we’re getting somewhere! That may be a dumb, inefficient implementation but I bet it would work for 99% of cases.

2:46 — But… if I can figure out how to make this work in 5 minutes of thinking, wouldn’t the people who spend their whole working lives thinking about password managers have thought of this? I’m sure they dogfood their products and they have to be extra annoyed every time a site rejects their password…

3:00 — I have a feeling that the sites with annoying password requirements that reject strong passwords aren’t implementing a ton of the newest HTML5 stuff.

3:01 — How can I test that hunch? I’ve had autogenerated passwords rejected, but I don’t remember where off the top of my head.

3:30 — Thankfully everything exists on the Internet! A quick search for “sites with dumb password requirements” brings me to Dumb Password Rules on GitHub.

3:45 — And… I have accounts with a couple of these sites so let’s check and see if they look like they’re implementing anything modern like HTML5 form hints.

(checks the HTML for the password reset forms for two sites from the list)

4:55 — They decidedly are not.

5:00 — OK… so this feature could totally exist! But only if sites with annoying password requirements added something to their HTML. And if I worked on a password manager I wouldn’t build this feature, because I doubt it would meaningfully reduce how many autogenerated passwords get rejected in real world usage. What we have is a problem of adverse selection:

1. The sites that implement new, user friendly features like standards based, HTML5 form validation are the least likely to have arbitrary and restrictive password requirements that reject password manager generated passwords and
2. The sites with dumb requirements like “Passwords must be between 6 and 10 characters and can only be letters and numbers” are the last sites that would add a validation regex pattern to help password managers.

And thus I conclude this edition of “Patrick gets excited about a feature idea, figures out how it could work, and then crushes his own dreams in 5 minutes.”

I wish I had a more useful lesson here. Maybe some form of Chesterton’s Fence like “If you have an idea for an obvious feature in someone else’s product, ask yourself why they haven’t built it before you propose it”

But I don’t think that is particularly useful.

Because you don’t get any credit for calling that something will fail. The default is that things don’t work.

So I’ve spent the last decade training myself to look for “how could it work?” and “how much would it matter if it did?”

At the same time, quickly eliminating ideas that seem good but wouldn’t actually work is necessary to spend time on the ideas that really matter, so being able to quickly eliminate plausible sounding ideas is valuable.

January

• Pachinko - Epic multi-generational story of Koreans in Japan.
• Station Eleven - Excellent. I’m not sure that reading about a world ending pandemic was great for my mental state last January. The HBO show currently airing is also beautifully shot. It is meaningfully different than the book owing to the need to have the characters, you know, meet. The book is better, but the TV show is different enough that you can enjoy each separately.

February

• Influence Without Authority - I find most business books overly transactional but if I can learn one thing from them that makes me better I consider it time well spent. In this case that one thing was that people value different things, so paying attention to what your coworkers value makes it possible to give them things they care about. Which in turn makes it more likely that they’ll be able to help you. Sounds cynical, but mostly is about treating people how they want to be treated.
• Lord of Strategy - I’ve worked with a lot of ex-management consultants, but have never been a management consultant. Interesting history of the creation and development of management consulting and modern business schools.

May

• The Hundred-Page Machine Learning Book - I’ve worked closely with Data Scientists for years, but I didn’t really have any hands-on experience. This was an excellent—and brief—overview that laid the conceptual groundwork to do a deeper hands-on dive during my break between jobs. Recommended to any engineer who works with data scientists.
• The Idea Factory - History of Bell Labs. They invented everything
• The Outsider: A History of the Goalkeeper - Very British centric. Wanted to enjoy this more than I did.
• The Bomber Mafia - Malcom Gladwell is a great story-teller. Not his best book but very enjoyable and one of the better produced audio books I’e ever listened to.
• Veronica Mars: The Thousand Dollar Tan Line - We watched Veronica Mars a few years ago and I quite enjoyed it. Kristen Bell read the audio book version. Not the worst way to spend a cross-country flight.
• Veronica Mars: Mr. Kiss and Tell
• The Premonition - Michael Lewis is a great writer. Scathing on the CDC.

June

• Goliath - This book made me angry. There are plenty of problems with large companies but this felt like Matt Stoler started with the conclusion “big = EVIL” and then portrayed every scenario in the most negative possible light. For instance, he wrote in scandalized tones that the A&P ruthlessly bullied food distributors for lower prices in the 1930s and I wanted to scream “You mean they achieved lower food prices for millions of Americans!”
• Big Business: A Love Letter to an American Anti-Hero - I deliberately read this next as a contrast to Goliath and found it a much more compelling argument.
• Entangled Life: How Fungi Make Our Worlds, Change Our Minds & Shape Our Futures - Fungi are super weird and break a bunch of my assumptions about how life works.
• Working Backwards - Detailed view of how Amazon worked (at least how Amazon worked at some point in the past). I’ve worked with several ex-Amazon people and can see the value of many of these practices. A meeting that consists of reviewing a rigorously prepared 6-pager that everyone has read will lead to better discussion and decision than walking through a presentation live. However, these practices evolved in a very specific environment at Amazon and they can only be successful if the environment shares the values that produced them.
• Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow - Practical introduction to a broad array of machine learning techniques. Reading “The Hundred Page Machine Learning Book” before this gave me some conceptual understanding. Doing all the exercises in this book gave me a much deeper understanding of those concepts.
• Lying for Money: How Legendary Frauds Reveal the Workings of Our World - Building stuff on the Internet means dealing with fraud. This was a fun history of the ways in which humans defraud each other and the very human vulnerabilities those frauds exploit.

July

• Amazon Unbound - Amazon is one of the most impactful organizations in the world today. This was a deeply researched history of how they went from a large e-commerce retailer to one of the most important companies in the world.
• Range - I loved David Epstein’s first book “The Athlete Gene”. This is an equally well written book about the benefits of avoiding early specialization.
• A Guide to the Good Life: The Ancient Art of Stoic Joy - Excellent personal advice on how to apply the lessons of ancient stoic philosophy to live a good life in the modern world. When I’ve practiced the principles in this book I’ve felt more peaceful and joyful.
• How to Be an Epicurean: The Ancient Art of Living Well - I thought I should examine some of the other ancient schools of philosophies. This didn’t feel as applicable.

August

• The Machine That Changed the World: The Story of Lean Production - Lean manufacturing is a background influence on a lot of modern software development. Two things were striking in the book that originally coined the term: how incredibly inefficient American auto manufacturing was in the 1970s and the palpable fear of Japanese economic domination in the 1980s.
• The Ministry for the Future: A Novel - I loved the Red Mars trilogy when I was a kid. This was a hard science fiction look at a near future ravished by climate change. This book made me deeply anxious. Which I think was the author’s intention.
• Finite and Infinite Games - The concept of finite and infinite games is central to how Shopify looks at the world. I didn’t love the writing but I do love the concept.
• The Cost of These Dreams: Sports Stories and Other Serious Business - Wright Thompson has written two of my favorite magazine articles ever: Michael Jordan Has Not Left The Building and Portrait of a Serial Winner about Luis Suarez. This is an anthology where each article seems to deal with the deep pain and sacrifice that comes from a pursuit of greatness.

September

• Why We Sleep: Unlocking the Power of Sleep and Dreams - Pretty amazing overview on the importance of sleep. I’m not sure I believe everything in the book, but it has changed my habits: I haven’t had any caffeine since September.
• Crossroads: A Popular History of Malaysia and Singapore - Singapore is a tiny nation that has become wealthy through a relentless prioritization of growth and rising living standards over all else—including basic personal freedoms.

October

• The Perfect Bet - I don’t gamble, but I very much enjoy thinking about gambling.
• Pie in the Sky - I generally don’t include the books I read with my kids here, but this book was beautiful. It is the story of an 11 year old boy who immigrates to Australia with his mother and little brother after the death of his father. Funny and heart wrenching. JoJo ended up having to read the last few pages out loud because I couldn’t get the words out through my sobs.
• Success and Luck - I have led an unbelievably fortunate life. I read this as part of a conscious exercise in gratitude to appreciate that luck. The book also makes a strong argument for dramatically higher marginal tax rates on the wealthy and how that would deflate top-end asset prices but increase utility… even for those who are being taxed dramatically higher. This adaptation of the introduction is worth reading.
• Team of Teams - Solid personal narrative of building organizations that can adapt quickly to changing conditions.¹

November

• Hip Hop (And Other Things) - I love Shea Serrano. Thoroughly enjoyed this. I also very much enjoyed his podcast “No Skips” (particularly The Miseducation of Lauren Hill episode)
• Estonia: A Modern History - I was more interested in the recent history about the extremely digital savvy nation that emerged after Communism, but this was more focused on early to mid 20th century. Despite not being what I hoped I learned a lot more about the Baltic states than I previously new.
• Trillions: How a Band of Wall Street Renegades Invented the Index Fund and Changed Finance Forever - I invest in index funds because I know I’m not smarter than the market. The indifference and hostility that met the first index funds is amazing given that index funds eventually became the default form of investment.
• The Rational Optimist - There is a lot of pessimism about the future. I intentionally seek out well argued thinking about why we should be optimistic for the future.

December

• The Box: How the Shipping Container Made the World Smaller and the World Economy Bigger - Fascinating, but also one of those books I think might have been better as 2 or 3 long magazine articles.
• The Last Shadow - I love the Ender series. This… didn’t really need to exist, but wasn’t the worst beach read.
• One Billion Americans - For the US to maintain its global influence we need to grow. The policy recommendations are reasonable and seem like they should be achievable if we committed to increasing growth. This is similar to Derek Thompson’s Abundance Agenda. I work in growth companies because—for all the challenges—everything is better when a company is growing. In a stagnant or slow-growing company there is a zero-sum competition for resources and promotions. But in a growth company there is an abundance of opportunity and it is clear that when person succeeds it makes it more likely that everyone else also succeeds. I’d like to see the US become a growth country again, and I think we can do it.
• Man’s Search for Meaning - A personal story of surviving the Holocaust paired with a practical approach to finding a meaning and therefore satisfaction in the less dire situations of everyday life.

I read far fewer books in 2020 than in past years. Pandemic stress and the lack of commute as I moved to 100% remote work really cut down on my reading time.

Looking back it also influenced my selection: a greater percentage of fiction and books about sports than in usual needs.

Favorites in bold.

February

• Why We’re Polarized - An examination of the structural issues that explain the outcome of our two party system. Good to read, did not provide a confidence about a way to improve things.

June

• How Buildings Learn: What Happens After They’re Built - About buildings but really about evolutionary design.
• An Absolutely Remarkable Thing - Really enjoyed this. Royela recommended it to me, I couldn’t put it down, and it blew my mind when I got to the end, read the “About the Author” and realized it was the nerdy guy from the SciShow youtube channel I’ve been watching with the kids for years.
• Football Hackers: The Science and Art of a Data Revolution - Soccer is an extremely difficult to quantify but there has been a lot of progress. Soccer analytics feel 5 years behind basketball analytics.

September

• A Beautifully Foolish Endeavor - The sequel to “An Absolutely Remarkable Thing.” I also enjoyed this.

August

• The Once and Future King - This was my father’s favorite book when he was growing up. In the lamest act of teenage rebellion I refused to read it. Finally, made that right this year. The Arthur as child book was repetitive, but I really enjoyed the Lancelot focused book.

October

• The Inner Game of Tennis - Heard about this in the second season of Michael Lewis’ Against The Rules podcast which I thoroughly enjoyed and highly recommend. The book was fine.
• Pakistan: A Hard Country - One of the reasons I read so few books this year is that I started reading this in about March. It took a while. I didn’t realize how little I knew about Pakistan. In particular the very loose central control of large parts of the country. The book ends in 2010. I wish there was an updated version covering the last 10 years.
• The MVP Machine - Baseball is far from my favorite sport but its highly discrete nature means it is generally at the lead of analytics. Where the first part of the analytics revolution chronicled in Moneyball was about roster building, this book examines how new tools are influeing dramatically changing player development. For instance, high speed cameras and specialized radar allows pitchers to develop new pitches faster than ever before. This book made me thing about the competitive advantage of creating better performance feedback loops.

  December

• Inverting The Pyramid - Definitive history of soccer tactics starting before the standardization of rules in the 1800s and continuing through geigenpressing. The older stuff was mildly interesting but found the more recent stuff much more interesting. There is a very good chapter on Marcelo Bielsa who is a fascinating person.¹