Marc Brooker's Blog

The Essential Barbara Liskov Some of my favorite Barbara Liskov publications. Barbara Liskov is one of the greats of computer science. Over a research career nearing 45 years, she’s had a resounding impact on multiple different fields, and received an impressive list of honors and awards, including the 2009 Turing Award . In the same spirit as The Essential Leslie Lamport and The Essential Nancy …

The Essential Leslie Lamport Some of my favourite Leslie Lamport publications. After it was announced that Leslie Lamport had won the 2013 A.M. Turing award , the link to his list of publications found popularity on most of the tech-related sites I visit. It’s an excellent page, with a long (and growing) list of Lamport’s publications, and witty comments by the author on each one. The whole list …

Resource Management in Aurora Serverless Systems, big and small. My favorite thing about distributed systems is how they allow us to solve problems at multiple levels: single process problems, single machine problems, multi-machine problems, and large-scale cluster problems. Our new paper Resource management in Aurora Serverless 1 describes what this looks like in context of a large-scale running…

When Redundancy Actually Helps Redundancy can harm more than it helps. Just after I joined the EBS team at AWS in 2011, the service suffered a major disruption lasting more than two days to full recovery. Recently, on Twitter, Andrew Certain said : We were super dependent on having a highly available network to make the replication work, so having two NICs and a second network fabric seemed to be…

Container Loading in AWS Lambda Slap shot? Back in 2019, we started thinking about how allow Lambda customers to use container images to deploy their Lambda functions. In theory this is easy enough: a container image is an image of a filesystem, just like the zip files we already supported. The difficulty, as usual with big systems, was performance. Specifically latency. More specifically cold st…

Reading Research: A Guide for Software Engineers Don't be afraid. One thing I’m known for at work is reading research papers, and referring to results in technical conversations. People ask me if, and how, they should read papers themselves. This post is a long-form answer to that question. The intended audience is working software engineers. Why read research? I read research in one of three men…

Focus on the Good Parts Skepticism and cynicism can get in your way. Back in May, I wrote Reading Research: A Guide for Software Engineers , answering common questions I get about why and how to read research papers. In that post, I wrote about three modes of reading: solution finding , discovery , and curiosity . In subsequent conversations, I’ve realized there’s another common issue that gets i…

Aurora DSQL: Scalable, Multi-Region OLTP A paper! Our new paper, Aurora DSQL: Scalable, Multi-Region OLTP , is now available on Arxiv. I’m excited about this one: it’s a fully end-to-end look at how Aurora DSQL works, from query processing, to transactions, to replication, to the control plane. We’ve shared most of this content before in other forms, on this blog, on Marc Bowes’ Blog , Werner’s B…

Surprising Scalability of Multitenancy When most folks talk about the economics of cloud systems, their focus is on automatically scaling for long-term seasonality: changes on the order of days ( fewer people buy things at night ), weeks ( fewer people visit the resort on weekdays ), seasons, and holidays. Scaling for this kind of seasonality is useful and important, but there’s another factor th…

Optimism vs Pessimism in Distributed Systems What—Me Worry? Avoiding coordination is the one fundamental thing that allows us to build distributed systems that out-scale the performance of a single machine 1 . When we build systems that avoid coordinating, we end up building components that make assumptions about what other components are doing. This, too, is fundamental. If two components can’t …

Control Planes vs Data Planes Are there multiple things here? If you want to build a successful distributed system, one of the most important things to get right is the block diagram: what are the components, what does each of them own, and how do they communicate to other components. It’s such a basic design step that many of us don’t think about how important it is, and how difficult and expens…

Exponential Value at Linear Cost What a deal! Binary search is kind a of a magical thing. With each additional search step, the size of the haystack we can search doubles. In other words, the value of a search is exponential in the amount of effort. That’s a great deal. There are a few similar deals like that in computing, but not many. How often, in life, do you get exponential value at linear c…

What You Can Learn From Old Hard Drive Adverts The single most important trend in systems. Adverts for old computer hardware, especially hard drives, are a fun staple of computer forums and the nerdier side of the internet 1 . For example, a couple days ago, Glenn Lockwood tweeted out this old ad: At least this isn’t an ad for a HAMR drive. $10k in today’s dollars. pic.twitter.com/2h2g3Gnguw — Gl…

Spec Driven Development isn’t Waterfall Write down what you mean. After spending a few months writing (e.g. on the Kiro Blog ), and speaking (e.g. Real Python Podcast , SE Radio ) about spec-driven development, I’ve noticed a common misconception: spec driven development is a return to a waterfall style of software development. Specification driven development ( in Kiro, for example ) isn’t about…

Redundant against what? Threat modeling thinking to distributed systems. There’s basically one fundamental reason that distributed systems can achieve better availability than single-box systems: redundancy. The software, state, and other things needed to run a system are present in multiple places. When one of those places fails, the others can take over. This applies to replicated databases, lo…

The Fundamental Mechanism of Scaling It's not Paxos, unfortunately. A common misconception among people picking up distributed systems is that replication and consensus protocols—Paxos, Raft, and friends—are the tools used to build the largest and most scalable systems. It’s obviously true that these protocols are important building blocks. They’re used to build systems that offer more availabili…

Simple Simulations for System Builders Even the most basic numerical methods can lead to surprising insights. It’s no secret that I’m a big fan of formal methods. I use P and TLA+ often. I like these tools because they provide clear ways to communicate about even the trickiest protocols, and allow us to use computers to reason about the systems we’re designing before we build them 1 . These tools…

Lorenz and Little: How Much Does Your Tail Cost? Lorenz and Little sounds like hipster burger bar from 2015. It’s time for Marc’s Amateur Statistics Corner! Today: why I pay a lot of attention to tail latency when optimizing cost. I’ve written before on the importance of tail latency for customer experience (e.g. in 2026 , 2021 , and 2021 , and 2017 ). Today, I want to talk about tail latency fro…