Eric Pullen: I'm going to try something new. This was something that I can do very well. I think I'm, you know, uniquely suited for being able to talk about cost and articulate cost, but do it with an architecture view.

Corey Quinn: Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today spent nine long years working at AWS. Before pivoting into his current job. Eric Pullen is a cloud economist here at The Duck Bill Group. Eric, thanks for agreeing to talk to me, you know, in public.

Eric Pullen: Thanks, Corey. I'm glad to be here. And yeah, I think it's only fair that I come on your podcast since I work for you now.

Corey Quinn: Exactly. It's one of those power imbalance, forced way through. It's like, "Hey, you're being voluntold to do a thing."

Eric Pullen: I love being voluntold things. So that's great.

Sponsor: Backblaze B2B Cloud Storage helps you scale applications and deliver services globally. Egress is free up to three times the amount of data you have stored, and completely free between S3 compatible Backblaze and leading CDN and compute providers like Fastly, Cloudflare, Vulture, and CoreWeave.

Visit backblaze. com to learn more. Backblaze, Cloud Storage, Backblaze. Built better.

Corey Quinn: So where do you come from? Uh, most of the time when I talk to folks who are interested in working here, their early career, it's, it's a delicate thing to say, but cloud economists don't just spring fully formed from the forehead of some God somewhere. There, there's, there's no such thing as a junior person who is likely to succeed in this role by any angle that I can spot it from. You've been in this industry a dreadfully long time. What's your story?

Eric Pullen: So I built a data centers to start with. So I, you know, out of college, I worked for a medium sized company and built a couple of data centers, did all the sort of standards that you would expect.

And I really had to learn the sort of financial side of IT when I did that. I was a director of IT and, you know, sort of ran the department. So everything from power and cooling to networking and, Storage and all the way up to basically the OS levels where we would stop. And, and so you had to learn not only all the technology aspects, which we all learned, and you know, that's the part I was most interested in, but also I had to learn about, uh, what does CapEx mean and what does OpEx and how do we deal with that?

And that was an interesting addition to my sort of repertoire of, of, uh, skill sets, but it was also great because I got to learn not only the IT side, but also the business side. And I think that's a great learning point for a lot of people that maybe they don't get, especially if they're junior. Maybe they don't have the ability to even look at some of these things.

Like, you know, what does the budgeting process look like for a midsize or even enterprise size? I mean, those are interesting aspects to, to, uh, any IT type job.

Corey Quinn: It feels like this is a similar path to the one that I went down and everywhere I look these days, it feels like the door to do that sort of thing.

Has long since closed. You're fresh out of school, you're gonna go build data centers and become the director of the IT department. That sounds like a 90s to 2000 era story. Now you do that, the question is, oh great was the company just negligent? Like, we found a kid out back, we're gonna let them go ahead and run all the computers here.

Eric Pullen: And it feels weird now that I'm the old guy because back then it was, I was the young kid and I was doing this stuff. And I do think there is a level of that, that one, a lot of companies don't need that aspect anymore. I mean, how many companies are building data centers anymore? So even understanding or needing that technology skillset is just gone.

Corey Quinn: Folks are insisting that cloud repatriation is a big thing. Even AWS, a couple of weeks ago, made a whole stink about this, but only in response to a regulator that said, well, this is happening. Where is it happening? The, the data center company stocks are flat. Is this, is the cloud repatriation serverless?

What's the deal here?

Eric Pullen: And I look at the server repatriation and all the stuff that goes with it and people are just going crazy about it. Let's be honest, these enterprises, they move at glacial speeds and they've moved to cloud, which is great, and it took them years to do it. There is no way they're bringing that stuff back on prem in a short order.

So I think it's premature at best to say that I think a lot of those companies aren't even interested in running that again. Is that the core business? I mean, that's the thing you, I think people miss a lot of times. Like is your core business, healthcare and life sciences companies, example, is that your core business to run data centers?

No. It's to design drugs and to do drug trials. And do you even care what a data center is? No. You just need to run compute. You need a lot of it nowadays, especially with all the GenAI, but more specifically the old AI stuff that they would do. And that's great. You need that, but you don't need to understand and run those data centers.

And I think that is an art that is going away for a lot of younger people that are getting into this industry. That's why I struggle when people ask, How did you break into this? I don't know that I could tell you an answer that would be applicable for somebody that's coming out of college today, because a lot of those conditions just didn't exist.

Corey Quinn: My breakthrough job was being the network admin at Chapman University, because I, they liked my personality when I interviewed and the technical interviewer was out sick that day. They called to offer me the job 20 minutes after I left. So I crammed like hell for three months and realized that I had gotten to the level I needed to be.

And a year later, someone approached, Hey, you want to come work in corporate life? Why, why would I want that? Well, we'll give you a 50 percent raise just for showing up. Okay, I'm going to corporate life.

Eric Pullen: I'll take that every day. No, I, and I'm the same way. I mean, I kind of fell into mine as a career. I was working in IT when I was in college, and I did it at the, you know, campus computer centers, you know, trying Slackware on an old, I don't know, HP desktop of some sort and learning Unix through Next, uh, if you remember the old, uh, Nextstations and the cubes and the slabs and stuff, that's how I learned Unix and then Linux and then other things later. But, you know, I got to learn all that because it was just stuff to play with because I was working on it.

to save money for college tuition at that time, which was great. I mean, it was a great environment today. That's a little bit more, you know, esoteric. I think that.

Corey Quinn: Their are whole curricula aimed at these things too. Oh, you want to work in computers. Great. Join the 5,000 other students at this university that are doing the exact same thing and try and distinguish yourself.

Eric Pullen: It's funny that you mentioned that I actually taught a course in college once. Just out of, uh, you know, an old teacher reached out to me at the University of Lowell and said, "Hey, would you be interested in being an adjunct professor for this database course?" Which I was not huge into database, but it was an interesting thing to try.

Glad I did it. Man, that's a difficult thing to do. And also it's, It's difficult for me to articulate to them where this technology was going. This was even years ago at this point. But, you know, to be able to show people, you know, where we started and where we are today. I mean, the original database for me was, you know, Lotus 1, 2, 3, or ebase and other things.

And, you know, nowadays you've got a database for everything. So it's just kind of a crazy shift that we've seen. But I do think that, you know, when we're looking at, you know, anybody trying to get into this industry, my suggestion and the number one thing I do tell people all the time is try to gain as much knowledge as you can in a broad set of environments.

Don't limit yourself to just working in startups or just working at enterprises because each of those have positives and negatives, but you need to learn from all of those different sets to learn the different skill sets you're going to need if you want to continue to grow up in the IT world.

Corey Quinn: So after doing this for a period of time, You decided, you know what I'm going to do?

I'm going to go work for an online bookstores, computer division.

Eric Pullen: You know, it's funny. I mean, I was, I enjoyed what I did before I went to AWS, but I wanted to see what AWS was. And so in 2015, this was still early days of cloud for a lot of people. And for us, I was working at a company where we were moving things to cloud and it was great, and I kind of convinced that team that we need to go down this road after having built data centers and all that.

So I wanted to see what that was like. And ProServe had just started up inside of AWS. And so their professional services group was one where you could work remotely. And so I was an early remote worker for AWS and a lot of their positions, even at that time, were not remote friendly, but this is one that was.

And so for me, it was a great opportunity to stay where I live, which is in Kentucky and still be able to experience Amazon and learn from them. And so, uh, joined ProServe in 2015. And I, I really loved it. I got to work with some of the biggest clients, uh, you know, it was an interesting time for AWS because a lot of the tooling didn't exist that does now. I mean, there was, you know, all the stuff that we do to create all these accounts and all that stuff that didn't exist and, you know, transit gateways were not around, we were, you know, Cisco had a appliance that you could install. And it was.

Corey Quinn: We're going to do a whole lot of horrible things with EPC peering and hope for the best.

Eric Pullen: It was bad, but it was also exciting.

Corey Quinn: Come work in our cloud. An in depth knowledge of several routing protocols is required. What are you people doing over there? You really aren't going to like the answer to that.

Eric Pullen: I'm pre-VPC guy, right? I remember when we had EC2 Classic, and I was running just Linux running on a, I think it was Flask Linux at the time, and running it on a single, you know, one or two machines that had nothing but a security group protecting them, which was at the time seemed like a good idea.

But now with VPCs and all the constructs that they have, it's, it's even better and more secure, but for us, it was, it was a godsend that we even had that capability, but when I started at ProServe, the thing that I learned very quickly is there wasn't that many people that knew this stuff outside of people that worked in AWS.

So you got a community very quickly inside of the world and you had to sort of run that community and learn from all those different people very quickly because consultancies, other outside entities just didn't know how to run a cloud. And so we were doing as much teaching as we were building because you had to help all these clients just to understand what are they even going to do on this?

So, you know, and then the other aspect of that is that a lot of people missed even in the initial onset is how much is it going to cost me?

Corey Quinn: We'll worry about that part later.

Eric Pullen: Everybody was like, Oh, just make it work. And we did. And then the cost starts to come up and it's like, wow, this is expensive. Now, some companies, I will say.

I worked with many that actually said, well, it's cheaper than my on prem chargeback model that the company was using at the time. And that was an interesting conversation to have with the internal IT teams to say, look, I can get the compute for half the cost, AWS, because of this weird chargeback.

Corey Quinn: And I can get it today.

And they're not going to sit there stonewalling me for eight months, telling me no every chance they get. And it was the rise of Shadow IT.

Eric Pullen: Large enterprise that was doing chargebacks. I mean, they learned a lot through cloud migrations that all of the business units were basically saying, no, I'm not going to do this anymore.

And it was amazing to watch them have to shift. And in some cases they all shifted straight to cloud as well, because they're like, well, they can do it cheaper than me. I mean, I have all this sort of layers of complexity and maybe we just move everything to cloud. And some of those. Large enterprises did that and you heard that it reInvents and main stages and those things, which is all great.

But it was a lot of effort to get in those early days to even get people to understand what the cost sort of dimensions were around things. And right sizing was always the first thing that people looked at. But honestly, right sizing is only a piece of the puzzle. And I think that's where a lot of companies now are starting to experiment with.

Other pieces, uh, as far as cost to go.

Corey Quinn: What else did you do while you were there? Nine years is a long time to work anywhere.

Eric Pullen: And if you're know anything about AWS, you typically, if you're there for very long, you don't stay in one position. I mean, there's probably a few exceptions to that, but most people, especially in the sales org, which technically I was in the sales org inside of AWS.

You kind of move around because one, you just want to do more things, but also because you just, you need to, to sort of keep your career moving on an upward trajectory. So spent a few years doing ProServe, learned a ton. Uh, and then I got this opportunity to join the Well-Architected team. And that for me was a great experience to try to grow my brand.

Corey Quinn: Was the team itself well architected? Or was it one of those "cobbler's children have no shoes" problem?

Eric Pullen: It was an interesting group. Uh, I will say there was some of the smartest people that I worked when I first started in that team, we're, we're from all over the org, and we have all over the world.

Uh, we had team members that went to Australia and then West Coast and East Coast and the team was great. I loved working with that team. It's like a lot of positions though, it's difficult to articulate your value to the organization. When there's not hard numbers behind it, we didn't have a sales quota.

We didn't have clients that we were working with where if they use Well-Architected, they would suddenly increase. So I think we're Well-Architected. And even today, I mean, I, I still live architected and I believe in a lot of its principles, but where it's kind of gone off the rails a little bit is, is it about the concepts of Well-Architected, or is it about some tool that we built helps you with those concepts?

Cause I think it's moved towards the tool and less from the concepts.

Corey Quinn: Tool's kind of a lofty term if I'm being direct. It's a checklist. That is effectively what it is. I was curious when the whole Well-Architected Framework came out, so when I dove into it, it was, oh, this is just a formalization of the stuff everyone should be doing.

There were no surprises lurking around in there.

Eric Pullen: And it's one of those things, I will say this, for all of the faults of the tool, it was a way for people to not have Excel spreadsheets with the questions in them, which was an early days problem that we had of that, right? So, I mean, I think the tool was beneficial in that sense.

It gave you a sort of a mechanism to use that was easy to repeat. That part was great, but using metrics against the tool, I think, is a bad idea. Like, how many, you know, uh, essays are doing Well-Architected reviews? I don't know if that's a metric we should measure. If a customer doesn't want to do it, then they shouldn't do it.

But, but again, going back to the Well-Architected, the framework itself was the important part to me. And I think that's where a lot of people didn't necessarily understand that that was the key component. And for a lot of enterprises, they never looked at it in a holistic way when they look at workloads.

And so you have operation security and performance and cost, all those things. That's a different way to view it when you look at it in a totality. And every large enterprise I went into, even mid-sized companies, we start asking those questions. They didn't know how to answer the questions from one person's perspective.

We would have to bring together teams to answer all the questions to even understand one singular workload that they were doing on AWS. And I think that was the power of LoL Architect. It was the conversation. It was getting those teams together to ask those questions. The tool and all the other things that go with it were just secondary to that.

And I think that's where Well-Architected can be powerful as a base level of knowledge, but it's something that you have to do. It's not a tool that you can run. And I think that's where a lot of people got confused with Well-Architected.

Corey Quinn: I think that there's been a lot of noise about it, and whenever you have a formula like that, or a framework, there's always some folks who are wanting to take it beyond the precepts upon which it was based and into now the process becomes the reason land.

And that is where I think people start to go awry with a lot of things, honestly.

Eric Pullen: And yeah, it's true. And the other thing is, and if you know anything about the internals of AWS, you end up with competing things. Like there was also the Cloud Adoption Framework.

Corey Quinn: Oh, don't worry. You see, you get competed both by AWS outside of the company too.

Eric Pullen: It's a weird sort of system where people are encouraged to generate new instead of adding on to what already exists. And I think that's another area where it's kind of problematic for these frameworks to continue and grow. The sort of normal mode of operation is, Oh, that thing has reached a lifetime.

Let's do something new because they won't, you know, people want to show value. That's, I get why the system is built the way it is internally, but I also understand that it makes it difficult for customers that, well, is this the thing I should use or this other thing? Or what, what's going on?

Corey Quinn: This service hasn't seen a feature release in three years.

Is there someone there keeping the lights on? Or are you about to come out with a whole series of things at reInvent for it? Or Is it going to be merged into something else? Or is it just going to sort of sit there doing what it does?

Eric Pullen: And service deprecation is something very new for AWS. I mean, we're just now seeing the beginnings of that.

Corey Quinn: And I think it's a good thing.

Eric Pullen: Oh, it's a great thing.

Corey Quinn: I wish they communicated more effectively as opposed to having to like go out and find it individually. But if they can solve that, yeah, turn the stuff off that doesn't work.

Eric Pullen: And I think there's a lot of teams that wanted things to be turned off and couldn't get them turned off because.

There was a real sort of hesitation in turning things off because we didn't do that. Like that was a, not an Amazonian thing, right? We built it as kind of a contract to have services running for our clients. I think what they're learning now is it's just, you can't do that forever. I mean, there are certain things, you know, having the old cloud search platform, like why is anybody still running that?

Like, if you're not moving on to something else, then you need to be told to move on. And, you know, you give them timeframes and all that. They definitely need to articulate this better, but there's no reason to get rid of stuff that's not being used heavily.

Corey Quinn: They do, in some cases, shut it down to new users as an early step, which is great.

And I get the sense that there are still customers on these things. We're like, well, we're definitely going to be turning it off on X date. No, they're not Google, where with Google, when Google does a deprecation, it will be ripped out from under you. When AWS does it, I suspect at some point you drag your heels as a customer hard enough, they will fly a team of engineers out to help you migrate off.

And

Eric Pullen: I have actually been first party to that. And I have seen where we have had instances where we said, look, we're going to have to do upgrades on X, Y, or Z platform. Usually it's some managed service platform that we, that they offered. Uh, and those are ones where they said, look, if there was an exception, because it was a particularly important client or had a particularly important workload, they would make exceptions for that. Now, they would put all kinds of rules and sort of parameters around it, which is great, but they would do that. And I do think that's a big positive for AWS is that they're always willing to kind of bend over for customers and make sure that they're always capable of doing what they need to do to keep running their business.

Corey Quinn: So, after nine years, you decided, "You know what I should do? I'm going to go work with that guy with a big mouth who basically insults everything that the company I currently work for does and instead of working on the computers directly, we're going to go talk about the money aspect instead."

Uh, what's wrong with you?

Eric Pullen: Uh, I don't know. Uh, I actually, so, I look at it this way. I wanted to do something different. I wanted a new challenge, but I also saw inside of AWS, things are changing and it doesn't take a rocket scientist to see that things are kind of changing internally at AWS. It could be for the good, it could be for the bad.

I can't articulate yet whether it is or isn't. I can tell you though, From my perspective as an employee of AWS, at that time, it was the risk level has increased at AWS, so layoffs and return to team, return to office, all these things you hear about in the public media about AWS, that weighs on you as an employee.

That's certainly something you look at. Plus, I would just wanted to do something new. And so I had been talking with Mike actually for a while, and we conversed about, you know, different roles over time a period of time, and it was like, it just made sense now that I'm going to try something new. This was something that I could do very well.

I think I'm, you know, uniquely suited for being able to talk about cost and articulate costs, but do it with an architecture view, and I think that's what's really needed for any cloud economist is, you've got to understand the technology and what's underlying it. And for a lot of these companies, you have to understand where the technology even originated from.

And I think, you know, that's where I built those data centers. I know how that stuff used to run. I know how it runs in cloud. And I also know where you can spend money that doesn't make sense. And so for me, it was just a great opportunity and really enjoyed my time here. I think I didn't even realize how, um, I don't think I was unhappy at AWS.

I think I enjoyed what I did, but I think I'm even more excited now than I used to be. And my wife is joking with me now. She's like, you seem much happier now than you have been in a few years. And I think it's not anything other than I'm just interested in doing a fun new thing and I'm able to grow and learn new things at my own pace without having to worry about sort of the bureaucracy that is AWS.

Again, there's reasons for all that bureaucracy, but nice to not have that, I will tell you.

Corey Quinn: You're also the vanguard of a new approach that we're taking with our clients. Historically, most of our engagements have been fixed term, uh, very defined, relatively short. You're embedded full time at one of our larger customers.

Uh, 40 hours a week, and you're digging very deep into not just the architectural optimizations, but helping get some of the changes driven across the finish line, working cross functionally. And it's, it's a new experiment for us, and it's, it's going very well to the point where this is, this is not the only one like this that we now have in flight.

Imagine that. And And it was, it was fascinating to me when we were trying to figure out what this role was going to look like. Something you, you, you stumbled onto very early on in our conversations was that you, you thought beyond the big obvious services, EC2, RDS, how to optimize those things. Great.

Yeah. Everyone optimizes those things. There are tools that do it. We're not going to come up with much new ground on EC2 optimization. Yeah. We're just not, but you focus on other areas, uh, primarily, which resonates with my own hobby horses as well, as far as how I approach these things.

Eric Pullen: Yeah. And I think it's really strange that, you know, I mean, you look at the bill and, you know, we, we all should, if we're looking at AWS cloud spend, we should be looking at our overall bill.

And, you know, I just like to look at the top 20. Like, let's just take a really quick and easy look at what are you spending your money on? And what I find is every enterprise is a little bit different, but obviously RDS and EC2 and maybe S3 are the top three. That's typical. Uh, you may see CloudWatch up there, although I hope people are working towards reducing that because there's a lot of money being spent on CloudWatch that I think is a little too much being spent on it.

But, you know, either here or there on that one. But, but then you start looking down, look at the number five and six and seven on the list. Some of those can be really big drivers of cost, and people just aren't even focused on. Things like Databricks Migration Service, or Manage Kafka, MSK, or it could even be other services like, uh, maybe you're using chatbots, you know, and you're using Uh, other types of sort of, you know, ElastiSearch services, other things where maybe there's opportunities to save money there that nobody's looking at because it's just not at the top of the list and yeah, you know, maybe it's another, you know, 30K a month that you could save just by optimizing MSK, let's say. Or Dynamo is another interesting one that, you know, I've spent some time recently diving deep into Dynamo. There's a lot of cost optimizations to be done there, and the tooling for that just doesn't seem to really exist. Unlike RDS, unlike RIs, and all that stuff where you can pull these levers.

Some of these don't have many levers. So you have to go much deeper in from an architectural perspective to really understand what can we even do to save money on this? Is there even an opportunity to save money?

Corey Quinn: You have the concept of consulting skill as well, of when you see a large, a large billing item, your immediate response is not, This is broken and shouldn't be this way.

Your immediate response is to have questions about it. Okay. What is this doing? What is the business use for it? How does this interplay with other things? Is this a misconfiguration? Is it the mistake or is it underpinning an incredibly important, valuable business function and we should not look directly at it lest it stop working?

Eric Pullen: Exactly. And I think that's another area where a lot of people that work in, especially in FinOps space, but simply as an architecture team, if you're an architect on a team. You should be able to articulate to your upper management what the value of the thing is. Like somebody looks and says, you're spending 2 million bucks a month on Dynamo as an example.

That's a lot of money to spend on Dynamo. But if it's running a service that's generating 100 million dollars a month, who cares? Like your way spending under what you're bringing in, it doesn't make any sense then. Versus the other way to look at it, if you're spending a couple million dollars a month on Dynamo on a service, It's an inventory service for the pins that you have in your break room.

Uh, something's wrong. Like you, you know, there is an articulation of what the value is. Uh, and even in days from low architect that we try to help companies, you know, articulate some of the value of a workload, but specifically when it comes to, you know, looking at it from a spin off standpoint is like, one of the things that you're spending money on doesn't even make sense.

And so we look at things like database migration service. Great service from AWS. It can do CDC, so you can have all of your data on prem running to cloud all the time, and it stays in sync. That's a great service to have. It's not cheap, but it is. But are you even using it for that? Or is it a service where you installed it, you moved some data to the cloud, you moved all your services to the cloud, and you forgot about it?

So now you're paying all this money for something where, what's the perceived value of that? And so I think there's a lot of conversations when you look at Further down the bill is what are these, you know, what's the use case for this stuff? And is there a use case for it? And then having that conversation about articulating its value versus its cost.

I think that's an area where a lot of companies just don't see that yet.

Corey Quinn: Enterprise itself is a skill. And at a small company, great, you can more or less talk to one or two people that'll have the answer to any question you've got in their heads. Whereas in enterprise land, you get to go exploring.

Through a whole bunch of different folk, each have a piece of the puzzle and you've shown a an adroit ability to navigate that well.

Eric Pullen: I have a lot of experience in it, I guess, and I think that people tend to, to know that I understand what they're dealing with in their eyes. And I think that sympathy is a lot, goes a long way when you're doing these kinds of engagements where you understand that this person that you're talking to only has the ability to affect one very narrow slice of that enterprise organization.

And so talking to that person in the way that they understand things is a way that you can gain their trust enough that they can tell you what's even going on, because otherwise you're just another consultant that's wanting to change their world and is it for the better or not? Who knows? But you're, if you're able to sympathize with people and especially as they're telling you about their problems. And a lot of these FinOps conversations are they know maybe there's issues here or there. They don't know how to fix it, but they know that there is a concern there. And so you're helping them uncover that. Say, Oh, you see this cost is increasing, but it's because of a team that's maybe adjacent to you that's causing that. Well, how can we open up that conversation?

And me being sort of a third party to that is also at a huge advantage. Any consultancy you've ever worked in, I'm sure this is the case. It's like sometimes just being the guy that says, "I have this name behind me because we're the FinOps team that we're going to help you, and we're going to uncover these savings," gives you a lot of cachet with these different groups to say I want to cut across those boundaries.

I need to have conversations at different levels.

Corey Quinn: It's always been critical from my, my view of the world to be able to delve into what the technology is actually doing. There's, there's an engineering prerequisite of understanding how these systems talk to each other, what the, what the, what the constraints are.

And understanding how making a cost effective change could have detrimental effects on performance, because, you know, once you bring something down, you're not allowed to save money ever again.

Eric Pullen: And it's funny you mentioned performance, because the pillar that I led for Well-Architected was the performance efficiency pillar.

And if you read through, like, Well-Architected as an example, you know, there's all this push and pull between the different pillars, like cost and performance. Yeah, I can make it super performant, but it also is going to cost you an arm and a leg. Or when you look at reliability, it's like, I can make this the most reliable system across multiple regions and, but it's going to cost a lot of money.

And so you have to kind of recognize what's the value that. of the service you're providing so that you can articulate what the cost should be for that service. Like, again, if you have a service where it's counting the number of pins in the break room, maybe it doesn't need to be multi region, you know, but if it's counting, you know, I don't know the number of drugs that you're shipping out to pharmacies, that's an important number that should never go away.

And you may need to...

Corey Quinn: You're not spending enough on that. Go spin up some managed NAT gateways. So we get fewer questions.

Eric Pullen: Exactly. So I think it's It's always good to find out what the reason is behind spend, and sometimes that conversation has nothing to do with IT, right? This is a business problem.

You're starting to understand why they're doing what they're doing, and you may even uncover that this problem doesn't need to be solved in the way they're solving. So now you've affected an architectural change through conversations you have about why they're spending so much. And I think that's another critical piece of what we do, especially in these newer engagements that Phil is doing is, we're trying to dive much deeper into that and say, you know, it's not just enough to say you can save some money with RIs.

You have to look at why you're doing what you're doing. And it takes time to do that. It's not a, you know, one week engagement. You can find all those things out. It's, it's across the board. You have to have conversations with the organization.

Corey Quinn: Organizations generally tend to be fairly poor at articulating internally the context for a lot of these things.

You'll have some engineers spinning things up that are wildly expensive without justification, but what I always find more interesting is when I encounter folks who are well intentioned engineers, but they're treating it like it's their own personal money, where they think for some reason they can't spend more than 200 a month on their own development environment there, where it's, I assure you, you cost significantly more than the infrastructure upon which you work.

We need this working. Don't worry about the cost. You're really aiming at the wrong part of the story.

Eric Pullen: And it's funny you bring that up because I have had many occasions where I've seen both sides of that, right? I've had people that are unwilling to spend 200 on You know, some AWS service to try it out and to see if it will fix a problem.

They don't have to build a whole nother service for great or you see the other side, which is someone that is doing some minor scripting and they spun up a, you know, 12 XL instance. So run that EC2 assist because they don't care. It's not their money, and they wanted it to be fast, even though it's way overkill for whatever the thing is, they were running, and they leave it running forever because they don't want a pull shutting it down. There's a weird dichotomy you see, and I think that's probably true for their personal finances as well. Maybe they don't think about it for their own money.

Corey Quinn: Yeah, I've noticed my personal EC2 dev box has gotten larger over the years.

It's now a C7G large and it's, um, that, that's a little beefy to be running all the time. But, you know, in the, I need something there that it works. I don't have to wait for things to, uh, to complete. I'm spoiling myself.

Eric Pullen: And that's true. And spoiling yourself sometimes, like you said, is needed to save you time and effort if your job is to do building scripts or whatever.

And then the money kind of works itself out. It's when you see hundreds or thousands of those that you start to wonder about, you know, and it's, that's when you start going, well, why are you even doing this? And that's a question that you have to kind of unfurl the answer to. It could be cultural inside of a company.

It could be that they've been dictated that this is the way they do things. And so the first person on the team built a 12XL, as everybody else does too.

Corey Quinn: A stupendous number of things out there just, uh, workloads are running on a single instance size. Okay, why that? Well, the developer who was building it that day, five years ago, just pulled something out of the ether, which when you're developing is fine, but it becomes those load bearing to dos in the backlog of, oh yeah, we should go and figure out what the right scale is for this thing.

But once it was up and working, it's suddenly in production and well, that's not a high priority right now. There are features to ship.

Eric Pullen: And the other thing that I've seen at a lot of enterprises over the years is this sort of ability that the enterprise wants to have to say, we're going to have standards.

And so they want to pick a size of like, EC2, or maybe it's a higher level service like RDS or something, like, every RDS instance is going to be a 8XL. Why? They're just doing it because they want to set a standard and there's a group that sets standards in large enterprises, and they're used to doing that when they had on prem, it made a lot of sense, right?

Economies of scale, I'm going to buy the same size server or the farms that I build internally, okay, that made sense to have one standard size. It doesn't make any sense in AWS.

Corey Quinn: I will caveat that there. The time we're in it does is, are you getting a screaming deal on a highly specific private pricing agreement on this one instant size family and region?

Because if so, that shoots a hole in everything we just said. I suspect you're not, and yeah, usually when that happens, you know going in that that's the story.

Eric Pullen: And in my experience, what I've seen when working in AWS is the ones where the, you take advantage of that is weird instance types, GPUs, obviously a lot of people, but I thought with F1s.

So if you have, you know, certain industries where FPGAs are heavily used for certain tasks, and so they would get private pricing for those F1 instances, makes total sense. But that's a very sort of esoteric use case.

Corey Quinn: It's a weird choice for a bastion host, but okay, you do you.

Eric Pullen: Yeah, exactly. Whatever. Uh, but I do think that, you know, a lot of times companies are just built around standardization when they're enterprises.

And it gets to a certain point where that made sense for them. But I do think that's an area where you have to look at it and say, what is, is this making sense anymore? And it's the old days too, with standardization around database engines, as an example, right? Everybody was standardizing on Oracle or Microsoft SQL or whatever.

It's like, yeah, that's a great concept. And I get why they would do that because a lot of the heavy lifting.

Corey Quinn: Call legal. We need to figure out what the license terms are on this.

Eric Pullen: But now with open source engines and things like Aurora and, and, and MySQL and Postgres and all those, it's like, pick what makes sense and let the provider run a lot of that.

Low level infrastructure, you know, replicating data. I don't know if you remember, but I remember how difficult that was across data centers, like having an Oracle box and using GoldenGate to ship data from one place to the other, it broke all the time. And now you have multi AZ for something like RDS, you don't even have to think about it.

It just works. And, you know, that's an area where I think a lot of people, especially coming into the industry now, if they're new is. They don't even know how difficult some of these things were years ago.

Corey Quinn: I spent months on that at multiple companies, just trying to get replication between two sites that aren't that far apart working.

And it was brutal. In some cases, in the same facility, just in a different rack.

Eric Pullen: Yeah, we had a Macross facility. We did Golden Gate across two data centers, and it was a pain in the butt. They didn't even, like, think about file replication. Look at how easy something like EFS is today. Or, you know, there's even ONTAP and other things.

That's a pain in the butt to build in a datacenter, and if people haven't experienced that...

Corey Quinn: I mean, it is Well, ONTAP was the solution that you used back then. Waffle's still one of my favorite file systems. But you had to spend top dollar for NetApps to make it work.

Eric Pullen: And you still had to have all these restrictions on how the two buildings were interconnected and there's all kinds of rules around that stuff.

Whereas with AWS, you just, you know, click a button and say, I want a file system and it does it. DFS is magic. I mean, I, you know, I remember the early days of EFS when it got pre announced. It was one of the first ones that they pre announced that didn't come out because it didn't work so well from what I was told, but then later, when it did come out, and now you look at it as a service today, it's a great foundational service, and it works really well for a lot of different use cases.

It's not perfect for everyone, but if you're needing NFS, man, it's cheap and it's fast.

Corey Quinn: It's cheap, it's fast, it replicates super well. The only problems I've had with it are more or less when I have made poor life choices. Like, okay, why does it take me seven seconds to log into my box every time when I put my home directory on EFS?

Well, yeah, maybe it's time to redo my crufty decade old ZSHRC, which makes apparently 1500 calls to, oh, across the wire for that, which it never expected to do. It's maybe stop doing horseshit and see maybe it works better and wouldn't you know, it did.

Eric Pullen: You know, that's another area where I always find it kind of amazing that Amazon does eventually get around to building services on top of other foundational services.

I mean, you look at like EFS is a great service, but you also look at what uses EFS. You know, you can use it with Lambda. You can do all these interconnections, but then you even think about, I remember early days when EFS first got announced, the reason I wanted it is I had an enterprise that had FTP servers or secure FTP servers at the time, and I wanted a quick and easy way to have a centralized file system because that was a pain in the butt at the time, and EFS looked like a great option.

Well, it wasn't because of a whole bunch of restrictions that it had. But lo and behold, Amazon says, "Oh, we could build a file transfer service that frontends S3."

Corey Quinn: I requested that service when I was at BlackRock because it was, I think, my fourth company where I had built the same thing where it's, all right, you're going to use lib notify or inotify on an FTP box to make sure the transfer is done and then validate it and wait a while and make sure the connection's closed and then go ahead and throw it into S3.

Can I just pay you to make this go away? Because it's always for big enterprises. It's not for you, shopping about, sharing with your mates, but for enterprises trying to solve for enterprise workloads, it is cheap as free.

Eric Pullen: And people don't realize how tedious that process was to manage file transfer.

Corey Quinn: It always broke.

Eric Pullen: Yeah, it broke and it broke in ways you couldn't figure out. And it's just, and now having the file land in S3, which is where everyone wants it to land anyway. At the end of the day, to then kick off whatever other process. I think there's that whole aspect of things when people look at on-prem workloads that do FTP and do file manipulation and then push it through a process and a workflow.

That stuff is dead simple now in a cloud environment.

Corey Quinn: And it's an API call away, clickety clickety, and it costs pennies per hour, and you don't have to worry.

Eric Pullen: And it's so straightforward now for people, and again, this is the history of people coming to the industry today don't even understand how difficult some of this stuff was.

Now people come in and go, why would you do that when you can just push the file to S3? I would have loved to have done that over the years, but every other enterprise didn't want to touch S3 at the time.

Corey Quinn: Yeah. Go ahead and just teach that giant bank over there how to use S3 instead of FTP. It's like, do I look foolish?

I wish someone had told me.

Eric Pullen: Nobody ever wants to deal with that stuff. And I think, you know, that's technology changes. That's great. But being able to still have some of the old messages still work in a cloud friendly way is a great advantage. And I've seen that, you know, with AWS, I think that's one of the areas where they really shine is they're trying to bridge that gap between what we used to have to do and what you could do today.

And I think there's a lot of areas there that are optimization efforts that you can look at when companies are moving to cloud. So this whole repatriation and moving stuff back is like. Yeah, did you think about all the stuff we're going to have to do again?

Corey Quinn: You just moved a whole bunch of workloads into cloud, dude.

It's okay, maybe yours are better ways to do these things instead of having to have that fleet of servers to build a message cube. Have you heard of SQS?

Eric Pullen: It amazes me too, when you read some of those stories, and I've read a few of them, where it's like, did you just not try to optimize before you moved back?

There are a few cases where I could see that it makes sense. I mean, if you're just, all you're doing is, you know, I don't know, some really simple website and maybe you could get away with just one single server and, you know, you're only serving a hundred users. Okay, maybe it, maybe it doesn't make sense.

Corey Quinn: Let's be clear. It can be done. We have, this is not just knowledge of the ages. We, we still know how to run these servers at scale. It's just a question of, is that where you want to invest time and effort? It's a decision every company has to make. My assumption is, is that people are doing what's right for them.

Eric Pullen: And I don't see a lot of people talking about what that cost is. I mean, every story I've read about is like, we already had teams that could do this work. Great. What if those teams go away?

Corey Quinn: I don't know too many companies that are spending more on infrastructure than they are on people.

Eric Pullen: Agree. And it's, it's a weird, Set up to say you're going to save that much effort, you know, by moving it back on prem.

I don't know. It's just that whole repatriation stuff. It just doesn't make sense to me.

Corey Quinn: It does not. Can I just say, I'm glad that you're here. I appreciate the chance to work with you and I'm enjoying, I'm enjoying seeing how this plays out. If people want to learn more, Where's the best place for them to find you?

Eric Pullen: Well, they can find me at The Duckbill Group. And so feel free to come to duckbill.com, and you can find out more about reaching out to one of us as a cloud economist. I would be happy or Mike or Corey or any of us can reach out and, uh, find out what we can do to help you. Cause I would love to have more effort than just the one project I'm working on.

I would love to do many more of these. And I think that there's opportunities abound for different enterprises that want to engage us or even small to mid sized companies. Everyone can save money if you're in cloud. And maybe again, you can look at those non, you know, EC2 RDS type things, and maybe we can help you find areas to save money that you didn't even think about.

So.

Corey Quinn: I really want to thank you for taking the time to speak with me.

Eric Pullen: I appreciate it. Thanks Corey, I appreciate it.

Corey Quinn: Eric Pullen, Cloud Economist here at the Duckbill Group. I'm Corey Quinn, also Cloud Economist here at the Duckbill Group, and this is Screaming in the Cloud. If you enjoyed this podcast, please leave a five star review on your podcast platform of choice.

Whereas if you hated this podcast, please leave a 5 star review on your podcast platform of choice, along with an angry comment that you'll no doubt be charged way too much money for because you don't realize what that podcast platform is charging you because eh, that's someone else's problem.