Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They’ve also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That’s S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.

Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they’re all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don’t dispute that but what I find interesting is that it’s predictable. They tell you in advance on a monthly basis what it’s going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you’re one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you’ll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.

Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. Way back in the winter of 2017, I went to my first re:Invent, and at that re:Invent, or shortly before, they announced EFS, Elastic File System, which is basically a NetApp in the cloud, killing some of my stuffing a NetApp into us-east-1 jokes. And my initial, considered, reaction—because I’d been writing the newsletter for about six months at that
point—was, “What a piece of crap.” And the general manager of the product said, “Hey, can we meet at re:Invent?”

And showed up with, as I recall, a couple of engineers who had no neck, and I thought, “Oh, great, this is how I die.” Instead of what I expected, what happened was he asked a bunch of questions and took notes. And one by one, every issue that I had with a service—and it was lengthy—wound up getting knocked out in the next couple of years. And today, it’s one of my absolute favorite services and I use it daily. That was an early but lasting impression of how a lot of my interactions with AWS were to go. And I’m very glad today to have that former GM and now VP of Engineering at AWS, Wayne Duso here to suffer more of my slings and arrows. Wayne, thank you for joining me.

Wayne: Corey, it’s always a pleasure. Thank you.

Corey: It really was a transformative moment for me, just because I was still finding my own voice in this. Because my big fear then was, “No one’s going to read it; no one’s going to listen. I’m going to starve to death and have to get a real job and get fired some more.” And it was just about get out there at any cost. I didn’t have the reach that I did, then, and I was a lot more cynical, and in some ways, directly opposed to service launches.

I try not to do that anymore; don’t always succeed. But I never got the sense that you took any of that feedback personally. And in some cases, you set me straight when I was wrong on things, and in others, you’re not only listened, you agreed with me and took steps to make it better. This is not me shining you on. This is very much the course of EFS.

Wayne: Yeah, Corey, you know, your feedback was super important. It was early days, and we don’t pretend that we get everything right the first time. You know, we don’t. You write about how we don’t get it right the first time quite often. And it’s appreciated because, you know, as engineers, as technologists, as leaders, you’re always looking for input.

Input is one of the most valuable things we can get. You know, not often people don’t provide us input. And when I met you and you had your long scroll of input for us, for me it was a treasure trove. But it really was a treasure trove. And to this day, whether it’s you or, you know, the next you, those scrolls are still treasure troves because it means that somebody has a different view than what I currently have.

Corey: It stuck with me because suddenly it came crashing home to me that a lot of the criticisms that I was lobbying against AWS weren’t just me shouting into the void. They were being heard. And also, it really was one of those reaffirmations back then; it’s like, “Oh, yeah, making fun of Amazon Web Services; they’re a division of a company that’s however many trillions of dollars it was at the time,” and, “That’s not a person. There are no people there. It’s just a big company.”

And the dawning, creeping realization was that companies are made of people. And instead of just saying, “This is crap.” I’ve got to be able to articulate why that is. And I guess the one enduring criticism that I had about EFS that still holds true on some level, is—it has nothing to do with the capabilities of the service, but more to do with the pattern of if I’m building something to work in the cloud on day one using what is, effectively, NFS that attaches to a bunch of different instances simultaneously—or now containers or Lambdas Function—great, that feels like it’s not the direction that cloud architecture has historically gone, but now that the capability is there, that’s starting to shift, and its own right once again. So, it’s just… it was a service that I didn’t fully understand then—I’m not sure that I still do—but I definitely see the use of it and the utility of it. And by just sitting here and doing effectively nothing, it gets better with time. And if there’s a better story about cloud, I’m not sure what it is.

Wayne: You make it sound like a wine. If you, you know, if you do a proper job mixing those grapes, eventually you put in a bottle, it gets better and better as time goes on, but.

Corey: And eventually becomes vinegar, and we can list, like, three or four of those services, too, but let’s be charitable here.

Wayne: [laugh]. We’re not going to go there. So, thank you for popping the cork on this bottle of wine and drinking it on a regular basis. Files are everywhere. You know, when I came to the company in 2012, and I was handed a question, “How should we build? What should we build?”—in terms of a file offering—I asked why in the same way that I’m sure you asked why when we launched. And the truth of the matter is that doesn’t matter what type of operating system you’re on or what you’re doing files are a fundamental abstraction.

And every operating system—every language—has an interface, which is file-based. It’s a really good abstraction. And you know, objects are great abstraction blocks are great abstraction, like, they all exist for a reason. They all serve a purpose. If they didn’t, they would have gone away.

And files have, you know, been around for at least 50 years, and honestly, they’ll be around for at least another 50, and probably beyond. So, it was really important to build a capability that customers—builders—could use straight out of the tools that they use every day without having to go through any transformation. It was important.

Corey: When I say that EFS is a contender out of maybe four others for my favorite service, that people think that it’s because I love files or I love storage, and okay, fine, but that’s not the real reason; that’s sort of irrelevant. I mean, I like SSL certificates too, but ACM isn’t on that list either. What I found valuable about it, what I love about it as an exemplar service is that if I go through the console wizard to spin up an EC2 instance—which of course I do because the ultimate form of managing cloud resources is using the console and then lying about it; it’s called ClickOps—and as I go through that process, adding an EFS file system to it is built into that wizard and it just works. When you spin up an EFS volume, it automatically configures AWS Backup to begin taking backup snapshots of what’s going on in there.

And there’s a bunch of niceties built into that. Recently, at re:Invent, there were additional changes rolled out—or pre-re:Invent—rolled out to EFS that enable automatic data tiering where, “Oh, that file hasn’t been touched in a while; we’re going to move that to the infrequent access tier and charge you less for it.” And this all happens transparently in the background. It is a service that gets better without requiring active customer interaction, and you’re sort of swimming against the stream of the typical AWS service approach by talking to other service teams and integrating into them in a way that is transparent to the customer. And I’m looking at this and I’m tapping the screen going, “More like this, please.” How did you get there?

Wayne: I’m a lucky guy in many ways. You know, I use this term inside of teams on a regular basis, which is, “We stand on the shoulders of giants.” And EFS was not the first service to launch at AWS; we know that… it wasn’t SQS, it was S3. And—

Corey: We’re talking beta or general availability—

Wayne: [laugh].

Corey: Those are the two teams that will wind up fighting to the death, and I just stay out of it.

Wayne: I know. I just wanted to beat you on that one. And so many great services, whether it’s Dynamo or S3 or EBS, came before the services that I built for customers—or my teams have built for customers. And so I get to stand on their shoulders and look far out onto the horizon; and also, I get to look backwards at mistakes or issues that we may have made, you know, earlier. And if I make the same mistakes, shame on me. I should be able to ensure that what we produce takes from the lessons that are already been learned.

So, for EFS as an example, I make sure that I look at all the lessons that came from EBS or S3 or Dynamo in terms of their scale and their usability and so on and so forth and say, “How do we make sure that we’re as good if not better for our customers?” And at some point, somebody will stand on the shoulders of EFS and they’ll look forward and say, “Oh, we got to do better than what they did in their first couple of years.” And I look forward to that.

Corey: I want to be clear that your portfolio has expanded significantly. It turns out that you were the GM and then now you’re a VP of Engineering. And so what changed? “Oh, same job, just different title.” And that is very much not true.

You own a bunch of other services as well—largely storage-oriented—including the snow family, which means that you are the person to, basically, harangue into—that I get to a point being able to check that item off my lifelong bucket list of beeping the horn on a snowmobile. It’s going to happen someday; I just don’t know how or when, but I feel like you’re someone who can help me set that up.

Wayne: But I live in a part of the country where we need snowmobiles, so it behooves me. [laugh].

Corey: Oh, yes. Growing up in New England, I remember those days, too. It’s, “Okay. Don’t plow the street. That’s fine. I’ll just cross-country ski to school.”

Wayne: I have stories but we won’t go there.

Corey: Oh, yes.

Wayne: You know, Corey, in my tenure, which is now a roughly a little over nine years at AWS, I’ve had the opportunity to meet with a lot of enterprise customers, I have a very rich enterprise background base on my career, so it was very natural for me to engage cohorts of customers, storage administrators, IT administrators, application administrators, network administrators, all of which have a rich history in success in the enterprise. And in speaking with them, it became incredibly clear that there were a series of enterprise-based services that needed to be built for cloud-scale, the cloud model of consumption, that just didn’t exist. And I’m not a big fan for creating services for the sake of creating services, just like you are not a big fan of us doing that, so I wanted to make sure that everything we built was filling a need that could not be filled any other way. And in that nine years, my teams and I have built roughly ten services covering storage, edge compute, edge storage, and data services like data protection, data movement, data management. And each of these services has deep roots in those enterprise cohorts.

Corey: One thing that’s become obvious to anyone who pays attention in the space has been that when we look at the sweep of history when it comes to technology, it’s that the tide always rises. When I first started playing around with technology, firewall engineer was a specific job. Now, any network administrators expected to be able to handle that just in due course. And when you’re talking about storage, the same thing happens. Now, there have been people who spent 30 years of their career as storage admins or working on specific technologies, and now that cloud is disrupting so many aspects of this, there’s a tendency that technologists have to push back against anything that disrupts the thing that they work on because they equate their identity to the thing that they build.

And let me be very clear here. I don’t think most people are immune to this. I know I’m not. It’s one of the reasons I tend to get so irritated about things that are disruptors to the way I thought about doing things. Why do you think I hate containers so much? It’s because well, that’s something that sounds like a different paradigm, and I’m not good at that, but I’m very good at this older thing.

And letting go, like, I’ve done that a few times and changed my focus throughout my career, but it’s always been challenging to do it. When I tell the story. “Oh, yes. I saw this thing and then I pivoted.” “Yeah, it wasn’t that easy.” There was angst and pain tied to it, and the constant awareness that I’m going to become a dinosaur if I don’t change my area of focus—and by the way, I’m 26 years old at the time—it was a hard thing to do.

Storage is such a important thing for so many companies in so many environments, that it’s such a nuanced deep area, that there is significant disruption happening there. How have you found those conversations to go with the folks at your customers who probably identify themselves as, “I hate the cloud. We should build more data centers.” Which is not actually what they’re saying, but it’s how it’s expressed.

Wayne: Yeah. Evolutionary change is something that the folks that you’re talking about understand, they embrace evolutionary change; they’re curious people, they love to learn; they’re passionate about what they do, and they’re passionate about their customers. It’s the revolutionary change that is hard. And they often view the cloud consumption model versus the traditional IT consumption model of receiving equipment, setting it up, putting on the floor, so on and so forth, they see that as an evolution, where they saw cloud as a
revolution. And they don’t want to necessarily embrace such a big change.

It’s part of my job and the job of my peers to have those folks, the storage administrators, the application administrators understand that it’s not as revolutionary as it may seem. And I want to give you an example of how we’ve done that. We’ve talked at length about EFS; EFS has a sibling and it’s known as FSx. And that really stands for File System X, which is any file system. What does that really mean?

Corey: I figured that one out a—what was it? It was something like three years after FSx launched, two years, something like that—I thought FSx was some storage term I’d never heard before. And nope, I was over-complicating it. It turns out—surprise—I don’t know everything either.

Wayne: Yeah, well, we were wrestling really hard with what we should name FSx. Look, I’ll tell you that story in a second, but what FSx is, is it’s a sibling service to EFS. EFS was built to be a cloud-native set-and-forget super-simple file system on AWS. Now, with that, there are 30 years of features that we did not implement when we launched EFS because the vast majority of those aren’t needed by everyone, so we didn’t complicate the solution.

That being said, if you think about storage administrators and application administrators in the enterprise, today, they use very specific file systems and the characteristics of those file systems, whether it’s performance, or durability, or—more importantly—management APIs, Snap APIs, replication APIs, all sorts of various data management capabilities, data service capabilities. FSx is a service that was designed to bring those file systems to AWS as fully managed offerings so they could be consumed in a cloud-native fashion. You’ve seen the launch of four of those to date. The first one we launched was FSs for Windows Server so that customers that used Windows servers on-prem could lift and shift their applications their workloads to a like offering on AWS. It’s bit-compatible.

The second was Lustre, we talked to a whole bunch of customers, amazing use cases, you know, FMI that is helping have cancer treatments that are specific to individual patients, they needed to be able to run high-performance workloads on AWS. Lustre was a great solution for them, but everybody knew that Lustre was a little hard to run on-prem. It took a lot of energy, took people to keep it up and running. But we took all of that away from them and provided them a fully managed Lustre offering, which they just create the file system, load the data into the file system, and go, and they worry about nothing after that.

Those were the first two. With the success of those, we heard customers coming to us—same customers say, “Hey, listen. I got a floor full of NetApps. I would love to run my NetApp-based workloads and applications on AWS. Can you help us?”

And we partnered with NetApp—it was a very deep partnership for two years—to create a bit-compatible like-for-like, no excuses, NetApp offering on FSx. And then we quickly followed it a couple months later, with a ZFS offering, which is FSx for OpenZFS because for the folks that aren’t running NetApp, there’s a whole bunch of on-pram that are running ZFS-based file systems, and they rely on the data management APIs of that file system. So, you know, for this cohort, we took what seemed to be a revolutionary change, and we turned it into an evolutionary change. And now our job is to go speak to all of those folks and have them understand that they can make that switch without losing the 20 years, the 30 years of expertise, without losing the trust of their customers, the folks that are running on their storage because they can guarantee them that what they have today is what they will have when they move. Super important.

Corey: I think that it does come down to trust that it does come down as well to the challenge that you’re in when it comes to storage. As we look at the broad sweep of where cloud business is coming from. It’s easy to sit here and pretend that we’re all somehow—we’re all doing net new; we’re building out this thing in a garage somewhere. “I have an idea.” “What is it?” “Twitter for Pets.” “Is that a good idea?” “Absolutely not, but I just raised 200 million in seed funding, so we’re going to do it anyway.”

A lot of this stuff is existing legacy workloads. And legacy is, of course, a condescending engineering term for, “It makes money.” But that is what you have to be able to address because to move your workload to cloud is a heavy lift. It becomes borderline impossible with oh and to do that, you’re going to have to completely re-architect the entire data layer, because that thing you’re doing right now, not so much. On tap in the cloud as an FSx NetApp offering was transformative for me.

Because back when I was in data center land, NetApp was the only way I would willingly run NFS in production. And I ran production MySQL databases on top of it. Professional advice, if you’re listening to this elsewhere, don’t do that. But it can be done, and it was amazing. And WAFL remains one of my favorite file systems ever.

And I kept joking that I just wish I could get it in AWS, but they won’t let me shove a filer into us-east-1. I know because I’ve asked. Well, you went ahead and did that for me. So thanks.

Wayne: You’re welcome. It’s our pleasure. [laugh].

Corey: I really hope that is still relevant to places I worked back in 2007, but let’s face it, “This is a temporary fix,” and 20 years later, it’s still load-bearing in production. So, here we are.

Wayne: You know, you’ll never hear me use the word ‘legacy.’ And in fact, within the company, we’ll be in rooms and people use the word legacy—it’s a very popular word people love to use it—and what I often will tell them is that if an application workload is important to a customer and it is helping run their business. There is nothing legacy about it. It is current, it is real, and we need to think about it in the way they think about it, not in a way, which has us in any way deprecate its importance, the customers will deprecate its importance if that’s important to them, so our job is to embrace what they have and help them, if they so choose, to bring those workloads to AWS without change. I’m going to give you an example. I’m not sure I can use the customer’s name because I often lose track of who I can talk about in public and who I can’t. So, I [crosstalk 00:20:32]—

Corey: I long ago stopped paying attention to the details of NDAs, which sounds like a terrifying statement to make until you realize the way I do that is I just assume every conversation I have is confidential until I can affirmatively prove otherwise. And I’ll ask people sometimes like, “Hey, remember the thing you said to me? Can I quote that in public?” And they look at me strangely, and say, “It was on a podcast and we put it out to the world. Yes. Yes you can.” Oh, good.

Wayne: [laugh]. Well, in this particular case, I can tell you the country and I ca—and actually continent in this case. It was a customer down in Australia, and they ran into a situation where they needed to move out of their data centers, and they needed a place to go and AWS was already provided to them. And they moved 40 Windows-based applications to AWS over a weekend. I feel bad for the folks; hope they get a few weeks off—at least a few days off after that event.

But they were able to move almost their entire business to AWS and FSx for windows over a weekend because the experience simply was, create the file system, spin up the application, mount the file system and go. And it worked exactly as it had on-prem. Exactly as it had on-prem. When I hear stories like that, as a builder, as an engineer, as a human, I am incredibly happy. It is a day worth living when you know that you’ve helped a customer.

Corey: When you come in and look at an architecture like that, see it for the first time. And you look at like, “Hey, you’re using the cloud like it’s a data center. What’s the deal here?” And if you say that in a condescending, insulting way, they’re sitting around talking about what an amazing achievement something like that is—and let’s be clear; having done those projects, it’s an amazing achievement—but to have someone come in with no context, “Oh, you should have done this in a more cloud-native way.” It’s, “Thank you, Seymour. Yes, if we were building this stuff bespoke today, greenfield, we would have radically different constraints and radically different capabilities, but we don’t have a time machine so instead we have to move forward and we can’t just burn everything down every 18 months and start over from scratch.” There’s value there.

Wayne: Yeah. And they have the ability over time, Corey, to—I’m using this term lightly—to modernize. Because I don’t really know what that means; I’m a big fan of mid-century modern furniture, which is no longer modern. [laugh]. But it is lovely.

Corey: A glimpse of the future that didn’t happen. An alternate path, a speculative fiction expressed through furniture. I hear you.

Wayne: But what I’d like to make sure people understand is that they can move today. They can utilize these capabilities and these services today and move, and then they can take their time and how they want to evolve those applications based on the needs of their customers, based on needs of their business. I do not want to slow people down. The entire intent of the portfolio that I’ve had the privilege to build and provide to customers over these years, its intention is to enable customers to move as quickly as they want, and to then take whatever time they need to evolve that, based on your business needs. It’s really that simple.

Corey: Today’s episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that’s built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you’re defining those as, which depends probably on where you work. It’s getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that’s exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn’t eat all the data you’ve gotten on the system, it’s exactly what you’ve been looking for. Check it out today at min.io/download, and see for yourself. That’s min.io/download, and be sure to tell them that I sent you.

Corey: There’s a lot that you said that I want to dive into, but that the piece that I want to focus on specifically is you said you’ll never use the word legacy. And I’m going to challenge you on that because in one of our conversations that we had, I asked you what product you enjoyed building the most? And your answer was, “Leaders.” And that gets into a different form of legacy. And yes, I’m playing semantic games here, but it’s the question there of what are you going to be remembered for, because God willing, none of us are building technological solutions that are still going to be at least in common use in 40 years from now, but I don’t necessarily know that the same can be said of people.

Tell me more about this because you no longer oversee a product; you oversee a bunch of products. And something I learned the hard way is that when you become management and later different forms of management, you can do very little of the work yourself, if any. Your only tool is delegation, and that means you need to have the right people to delegate to which means hiring and handling leaders is effectively your IDE for lack of a better term. Talk to me about that.

Wayne: It’s a really important point. And one of the mental frameworks I put in place to myself a handful of years back, which guided me to AWS in fact, is ‘who,’ ‘how,’ and ‘what.’ Who I work with is most important to me, how I do my work come second, and what I work on comes a distant third. And the reason for that is so simple to me now—it wasn’t when I was a young engineer where ‘what’ was the most important thing on the planet; you know, what I worked on define me. And it took me years to understand that what I work on doesn’t define me, it’s who I work with and how I do that work that defines who I am. You know, a really, really lousy day with great people is still a good day. And a really great day with people you don’t want to be around is still not a great day.

Corey: I think that a lot of companies, maybe all companies, tend to get a somewhat unfair reputation once they hit a certain point of size and scale. There have been all kinds of exposés in various places about how a company X—and it doesn’t matter who X is; it can be Amazon, it can be any company people have heard of—is a terrible place to work. But I was talking to a friend at AWS recently who is coming back from parental leave, and they told me that, “So, get this. AWS changed their policy while I was out on parental leave. I have another month of it to take later in the year, and I’m really looking forward to that.” It’s like, “That’s amazing.” As opposed to the constant stories of oh, this thing is awful and this thing is terrible.

It’s very hard to see from the outside what a company is actually like. And I apply that to me. I only have the faintest glimmer of what it’s like to actually work at AWS. But talking to the people that I get to talk to and seeing how things are built, is it perfect? No. No place is. But I hear stories about different approaches to leadership and different teams, and, on some level, I become intensely grateful that I never tried to work on any of those teams because I would have given everything I was building up to have the chance to be a part of those groups, and then where would the industry be? Certainly without my snark, and that—we would all be the poorer for it.

Wayne: That’s true.

Corey: But there are always pockets of amazing things, same way there are pockets of terrible things. AWS at its—what—however many tens or hundreds of thousands of employees it has now doesn’t have a corporate culture. It has hundreds of thousands of corporate cultures because it is a team-by-team structure there. And culture—there are—from a principles perspective, has to flow from the top, but then you have management and how they run their organizations and their teams, and the blast radius attached to that is tremendous. And that’s the sort of thing that always scared the hell out of me when I was debating. Do I stop being an independent contractor—independent consultant and start hiring people here full-time?

Because the biggest blocker I had was that, yeah, if I say something wrong and get smacked off the earth by AWS, okay, great. I had it coming. Asking people to risk the next phase of their careers on me saying or doing the right thing was a hell of a responsibility. That’s the stuff that keeps me up at night. Not that I’m going to have a wrong take or something. It’s the letting down the people who are depending on me to get things right.

Wayne: Yeah. So, leadership is a, it’s a lot of fun; it’s a lot of responsibility as well. And so to your question, the thing that I will build—going back to the who, how, and what—the thing that I will build will the most important—the last thing I build will the most important—is the leaders I leave behind. Those leaders will go on for decades and build more leaders. They’ll build more products, they’ll build more businesses, they’ll do great things, but my job is to ensure that I build the right leaders that ensure that the culture that we do have at AWS—or wherever else they decide to go in the fullness of time—they will carry with them the best elements of the AWS or Amazon culture, through AWS or into other companies.

I’ll use the word: That will be my legacy. And right now, I look at doing that, not just with folks I bring into management roles, but I look at that in terms of the young engineers, the young data scientists, the young DevOps folks that we bring into the company, and say, “Where can I find incredibly passionate and curious learners, regardless of their background, regardless of where they come from, who can, in the fullness of the next five to seven years, become those leaders?” At that point, we will have a company that continues to represent the people, continue to represent the customers and the communities we serve. We will understand our customer. That to me is what it means to be customer-obsessed is to understand your customer. And you can understand your customer best by being a representative population of who they are.

Corey: One thing that we often decry in this industry is the pox of short-term thinking and over-emphasis on next quarter’s numbers. Think longer-term. But when people say that, they’re often thinking in three to five years out. You’re talking about something that spans decades and that is borderline unheard of in corporate life.

Wayne: Yeah, well. Thanks. [laugh].

Corey: [laugh]. I mean, I had to think about this stuff a fair bit myself recently because let’s face it, I have rendered myself completely unemployable. I went into this knowing it was, like, burn the boats behind me because this, for better or worse, is the last job I will ever have. And—maybe because I’m going to get killed in two months; we don’t know–but it’s—I’m very good at antagonizing people with a lot of bigger spite budget than I have—but that is how I have to think about these things. In the context of something at your scope and scale—because if your storage service or services have a bad day, so to all of your customers. That is something that weighs on the mind of every Amazonian I’ve ever spoken to, including someone who’s joined their first job out of school, two weeks ago.

It’s a culture of not fear, but awareness of the weight of responsibility. And some folks obviously carry that better than others, but it is one of those things when I start talking to people who are new Amazon employees, and I’m starting to be able to categorize them mentally how they talk about certain things of, “You’re going to go far at this company,” versus the, “Let’s see how this plays out.” You can pick up on some of these things, in the fullness of time.

Wayne: Having this level of responsibility, knowing that everything from entertainment to critical life care is dependent on the decisions you make and on the product you build and operate is a great responsibility. I can speak personally, it motivates me every day. You know, I can probably pick three days out of the near ten years I’ve been building at AWS where I just wanted the day off. [laugh]. I didn’t want that responsibility.

Now, of course, we have the leadership I just talked about that runs the services every day, who was there to make sure that on those three days that I didn’t show up, everything was going to be fine. And it, of course, was fine. But doing what we do is hard work. But I’ve never found anything more rewarding. And just speaking to a customer—a single customer—who’s had a great day, or a customer hasn’t had a great day, but you’ve done the right thing and there trust in you is strong, they’re unhappy with you, but their trust in you is strong. That is also a great day.

Corey: So, much of what happens in cloud is thankless. We started off this episode talking about how the EFS integration into other services. It is just this thing that I remark upon about how wonderful and great it is, but for most customers, like, “Okay.” They just click it and it’s great. When it’s not there, they find it obnoxious and challenging, but when it’s there, just, “Okay, this is working as it should,” and move on. So, much of the work and challenge and victories are unsung, and I try not to pass those things idly by and just take the time to appreciate the things that I see.

Because as I said, companies are made of people, and there’s a tremendous amount of innovation and improvement that’s going on constantly. I sometimes say—probably not enough—that 90, to 95% of what AWS does is excellent. That missing gap to get to 100 is both frustrating and, honestly, rife with opportunities for hilarity. So yeah, I don’t want to spend all my time talking how great all this stuff is—I should just work in marketing if that’s the case—I want to talk about what it takes to go from great to perfect, and you’re never going to get there, but that’s okay; it means I’ve going to have a job for as long as I want one.

Wayne: You know, there’s a term that we often use, and that is for our customers, we need to operate our services so that they’re indistinguishable from perfect. That’s a tall bar.

Corey: Mistakes show.

Wayne: Yeah. But it is our responsibility. And frankly, as builders and strong owners, it comes with a lot of fun. It’s really hard, but it comes with a lot of fun. Like, being able to do that… you know, this conversation we’re having right now is likely traveling over some piece of the cloud… everything that was enabled during… the pandemic, which has been very challenging for a lot of people—for everybody—for your haircut for a while it was very challenging for.

Corey: Oh, that was one of the hardest parts of the pandemic. I expect to find that the list of pandemic-related fallout on Wikipedia someday.

Wayne: [laugh]. And, you know, what we do enabled a large part of people’s personal lives, business lives, the economy to continue at some level of normalcy, which if it did not exist, it would have been a very different place. And we’re very grateful for being able to be able to do that. We’re very proud that we’ve been able to do that at the scale we do during the last couple of years. It’s taught us a lot of lessons. It’s been fantastic.

Corey: I’m very light-hearted about some of the workloads I put on AWS. I have a Last Tweet in AWS Twitter client that basically does shitposting threads in long-form. And it’s great. And if it winds up breaking, then there’s no big deal. I don’t care about that.

But I don’t have to care about that; you do have to care about that because just because I don’t take one of my workload seriously, you as AWS can never have that context. And that’s why you take every weird blip, everything, “I don’t understand this,” or, “This hasn’t lived up to my expectations on it,” as if it were a life-critical system because for some customers they are. And I have always appreciated how—and been bemused at times—by how deadly seriously AWS folks take my complaints about, “Yeah, my shitposting app isn’t posting shit quite the way I want it to.” [unintelligible 00:36:05] anything but the utmost of professionalism and respect when I’m talking about service gaps and challenges I’m having building and deploying things. And I feel a little bad at times, just because I’m making people care so seriously about things that don’t actually matter for crap. But I’ve always appreciated it.

Wayne: Our frame of reference is the millisecond and the penny. We worry about every penny you spend, and we want to make sure you—

Corey: Oh, there are times that, in some cases, for some services—I believe it was trillionths of a cent is how a couple of them have a granularity in the billing system. So, all you care about far smaller denominations than pennies.

Wayne: Well, I might be a little generous in what I’m saying right now, but for the frame of reference for the listener, you know, we don’t think about things at the quarter and the million dollars. That’s not our frame of reference. We think about things in terms of the millisecond a penny. And you know, yes, you’re right, we now think more in microseconds than we do milliseconds, and we do think in fractions of pennies, not pennies. But it’s a frame of reference.

What matters is the details. And there is no workload is unimportant. If it’s your workload, it’s just as important to us as any other workload. Even if it does poke snark at us. We’re okay with that.

Corey: And sometimes there is the idea of a memento mori, or someone yelling at the emperor that they have no clothes. The problem is in the story of the emperor not having any clothes, the kid would at least occasionally shut up once in a while, and I never seem to so
there is that part of it too.

Wayne: Springs hope eternal.

Corey: Exactly. Wayne, I want to thank you for taking so much time to speak with me today. If people want to learn more about what you’re up to, and how you view these things, where can they find you?

Wayne: Well, the two places they can find me most often, Corey, not at my desk or at a local bar; they can find me on LinkedIn, and it’s Wayne Duso. There’s only two of us on LinkedIn, so find the guy who wears glasses that has no hair, and that’s me. And the second place they can find me is on Twitter, which my handle is not obfuscated at all. It’s @wayneduso.

Corey: And we will, of course, put links to both of those in the [show notes 00:38:05]. Thank you so much for your time today. I really appreciate it.

Wayne: Corey, it’s always a pleasure. And I’m looking forward to sharing maybe a drink at that bar with you soon.

Corey: I look forward to it. I can’t wait to go back out to bars again. Oh, my God. [sigh]. Wayne Duso VP of Engineering at AWS. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice telling me that I’m completely wrong when it comes to using EFS for things that I should instead be using a better storage system that’s more cloud-native. Like Route 53 [unintelligible 00:38:48].

Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Announcer: This has been a HumblePod production. Stay humble.