Transcript

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: It’s easy to **BEEP** up on AWS. Especially when you’re managing your cloud environment on your own!
Mission Cloud un **BEEP**s your apps and servers. Whatever you need in AWS, we can do it. Head to missioncloud.com for the AWS expertise you need.

Corey: Do you wish your developers had less permanent access to AWS? Has the complexity of Amazon's reference architecture for temporary elevated access caused you to sob uncontrollably? With Sym, you can protect your cloud infrastructure with customizable, just-in-time access workflows that can be setup in minutes. By automating the access request lifecycle, Sym helps you reduce the scope of default access while keeping your developers moving quickly. Say goodbye to your cloud access woes with Sym. Go to symops.com/corey to learn more. That’s S-Y-M-O-P-S.com/corey

Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. Today my guest is a long-time listener, first-time caller. John Mille is a Principal Cloud Engineer at Sainsbury’s, which is UK-speak for ‘grocery store.’ John, thank you for joining me.

John: Hi, Corey. Thanks for having me.

Corey: So, I have to begin with, I guess, the big question that I used to run into people in San Francisco with all the time. They would work at Walmart Labs and they would mention in conversation that they work at Walmart, and people who weren’t aware that there was a labs out here figured they were a greeter at the grocery store. Do you ever wind up with people making that sort of fundamental assumption around the fact, oh, you work at Sainsbury’s as a checker or whatnot?

John: No. But it actually is one of the—if you look at one of the job descriptions from Sainsbury’s, the first thing is, why would you join a retail company to do tech? And as it turns out, tech—I mean, I think retail companies, as any other companies in the world, rely on Cloud more and more and more. And I think that one of the things that is interesting today is, if you look at the landscape of retailers, I’ve heard many times people saying, “We don’t want to go for AWS because we’re giving money to the competition.” And actually, I think AWS does a fantastic job overall giving you all the tools to actually beat them as your competition. And as it turns out, we’ve had really, really great success running a lot of our workloads on AWS for many, many years now.

Corey: On some level, if you can’t come to terms with the idea of Amazon as competition, you shouldn’t be using AWS, regardless of what industry you’re in, because their entire company strategy is yes. It’s very hard to start to even come up with industries that they don’t have some form of presence within. On some level, that’s a problem. In fact a lot of levels, that’s something of a problem.

Everyone tends to wind up viewing the world in a bunch of different ways. I like to divide companies into two groups. More or less it’s, is the AWS bill one of the top three line items at the company? And if the answer’s no, on some level, you know, that usually is an indicator that there’s a sustainable business there that, you know, both our grandparents and our grandchildren will be able to recognize, in the fullness of time. You absolutely have a business that winds up falling into that category, whereas, “Oh yeah, I fix the AWS bill,” yeah, my parents would have no idea what I do and my kids don’t have much of a better one. It feels like it’s very point-in-time type of problem. At least I hope.

Technology is not the core of what grocery stores tend to do, but I also don’t get the sense that what you’re doing is sitting there doing the back office corporate IT style of work, either. How do you use technology in the overall context of the business?

John: Well, so we use it in a very wide variety of sense. So, you obviously have everything that has to do with online shopping, orders and all of those sort of things, which obviously, especially with the drive of Covid and being everybody from home, has been a huge driver to improve our ability to deliver to customers. But certainly, I think that Sainsbury’s sees AWS as a key partner to be able to go and say we want to deliver more value. And so, there’s been a number of transformation over the years to—and one of the reasons I was hired is actually to be part of one of those transformation, where we’re going to take existing infrastructure servers that literally—I usually say to people, “Oh, are we doing an upgrade this month? Has somebody gotten their little brush to go and brush onto the hard drives to make sure that nothing is going to die?” And actually do that transformation and move over to the cloud in order to never have to really worry about whether or not they have to manage hardware and infrastructure.

Corey: It’s strange in that I never got very deep into containers until I was no longer hands-on hardware, managing things. I was more or less doing advisory work and then messing around with them. And you’d think given my proclivities historically, of being very unlucky when it comes to data, you would think that this would be great because, oh yeah, you blow away an ephemeral container? Well, that’s kind of the point. We’ll all laugh and it’ll re-instantiate itself and life goes on.

But no. Making fun of them was more or less how I tended to do approach them for the longest time until I started to see them a little bit… well I guess less as a culture, less as a religion, and more as an incredibly versatile packaging format, which is probably going to annoy the people I know who are the packaging [unintelligible 00:04:58] for Linux distributions. How do you tend to view them? And how did you start using them?

John: Right. So, that’s a great question. So historically, I was a student at, I think the school were one of the original creators of Docker were. And one of the things that you learn when you do development at the school is that, you know, containers [unintelligible 00:05:18] new invention. Docker, I think, came on the platform as the way to, you know, give everybody a great framework, a great API, to drive the deployment of containers in the world and bundle them and ship them around the world, on your laptop and somebody else’s, and help a little bit with, you know, solving the problem of it works on my laptop, but not just on the laptop properly. Maybe.

It’s obviously gone viral over the years and I really enjoy containers; I quite like containers. What I find interesting is what people are going to do with. And I think that over the last few years, we’ve seen a number of technologies such as Kubernetes and others come into the scene and say—and trying to solve people’s problem, but everybody seems to be doing, sort of, things on their own way. And historically, I started off using ECS, when it was terrible and you didn’t have security groups per containers and all of this. But over the years, you know, you learn, and AWS has improved the service quite significantly with more and more features.

And I think we are today in the place where there’s this landscape, I think, where a lot of workloads are going to be extremely ephemeral and you can go [unintelligible 00:06:28], you know, wherever you want and you have a bit—if you have a platform or workflow that you need to have working in different places, maybe Kubernetes could be an easy way to have a different sort of sets of features that allows you to move around in maybe an easier way. But that also comes with a set of drawbacks. Again, I look at using EKS, for example, and I see okay, I have to manage IAM in our back now, whereas if I used something like ECS, for the whatever the [unintelligible 00:06:56] cloud vendor of choice, I don’t have to deal with any of this. So, I think it’s finding the fine balance between how you do orchestration of containers now and what works for you and is any sustainable over the time, more than about are you going to use containers? Because the chances are, somebody is using containers.

Corey: My experiences and workflows and constraints are radically different than that of other folks because for a lot of the things I’m building, these are accounts that are I’m the only person that has access to them. It is me. So, the idea of fine-grained permissions for users from an ARBAC perspective doesn’t really factor into it. Yes, yes, in theory, I should have a lot of the systems themselves with incidents roles being managed in safe and secure ways, but in many cases, the AWS account boundary is sufficient for that, depending on what it is we’re talking about. But that changes when you start having a small team of people working with you and having to collaborate on these things.

And we do a little bit of that with some of our consulting stuff that isn’t just the shitpost stuff I build for fun. But there’s multiple levels beyond that. You are clearly in a full-blown enterprise at this point where there are a bunch of different teams working on different things, all ideally going in the same direction. And it’s easy to get stuck in the weeds of having to either go through central IT for these things, which gives rise to shadow IT every time you find a corporate credit card in the wild, or it winds up being everyone can do what they want, but then there’s no consensus, there’s no control, there’s no architectural similarity. And I’m not sure which path is worse in some respects. How do you land on it?

John: Right. So, what I’ve seen done in companies that works very well—and again, to the credit of my current company—is one of the things they’ve done really well is build a hub of people who are going to manage solely everything that has to do with accounts access, right? So, the control, IAM, Security Hub, all of those sorts of things, for you. There’s things that are mandatory that you can’t deal without, you have permissions boundary, that’s it, you have to use those things, end of story. But beyond that point, once you have access to your accounts, you’ve been given all of the access that is necessary for you to deliver application and deploy them all the way up to production without asking permission for anybody else apart from your delivery managers, potentially.

And I think from there, because there is the room to do all of this, one of the things that we’ve done within my business unit is that we’ve put in place a framework that enables developers—and when I say that it really is a question of allowing them to do everything they have to do, focus on the code, and I know it’s a little catchy [unintelligible 00:09:33] a phrase that you hear these days, but the developers really are the customers that we have. And all that we do is to try to make sure that they have a framework in place that allows them to do what they need and deploy the applications in a secure fashion. And the only way to do that for us was to build the tools for them that allows them to do all of that. And I honestly haven’t checked a single service IAM policies in a very are longtime because I know that by providing the tools to developers, they don’t have this [will 00:10:05] to go and mess with the permissions because their application suddenly doesn’t have the permissions. They just know that with the automation we’ve providing them, the application gets the access it needs and no more.

Corey: On some level, it feels like there’s a story around graduated development approach where in a dev environment you can do basically whatever you want with a big asterisk next to it. That’s the same asterisk, by the way, next to the AWS free tier. But as you start elevating things into higher environments, you start to see gating around things like who has access to what, security reviews, et cetera, et cetera, and ideally, by the time you wind up getting into production, almost no one should have access and that access that people do have winds up being heavily gated. That is, of course, the vision that folks have. In practice, reality is what happens instead of what we plan on. The idea of it works in theory, but not in production is of course, why I call my staging environment ‘theory.’ Does that tend to resonate as far as what you’ve seen in the wild?

John: Yeah. Very much so. And when I joined the company, and we put together our [standard 00:11:11] pipelines for developers to be able to do everything, the rule that I would give to my team—so I manage a small team of cloud engineers—the one rule I would say is, “We have access to prod because we need to provision resources, but when we’re going to build the pipelines for the developers, you have to build everything in such a way that the developers will only have read-only access to the production environment, and that is only to go and see their logs.” And at least try to foster this notion that developers do not need access to production, as much as possible because that avoids people going and do something they shouldn’t be doing in those production environments.

Now, as the pipeline progresses and applications get deployed to production, there are some operational capabilities that people need to have, and so in that case, what we do is we try to fine-tune what do people need to do and grant those people access to the accounts so that they can perform the jobs and I don’t have to be woken up at two in the morning. The developers are.

Corey: One thing that I think is going to be a cause of some consternation for folks—because I didn’t really think about this in any meaningful sense until I started acting as a consultant, which means you’re getting three years of experience for every year that you’re in the wild, just by virtue of the variety of environments you encounter—on some level, there’s a reasonable expectation you can have when you’re at a small, scrappy startup, that everyone involved knows where all the moving parts live. That tends to break down with scale. So, the idea of a Cloud Center of Excellence has been bandied around a lot. And personally, I hate the term because it implies the ‘Data Center of Mediocrity,’ which is a little on the nose for some people at times. So, the idea of having a sort of as a centralized tiger team that has the expertise and has the ability to go on deep dives and sort of loan themselves out to different teams seems to be a compromise between nobody knows what they’re doing and, every person involved should have an in-depth knowledge of the following list of disciplines.

For example, most folks do not need an in-depth primer on AWS billing constructs. They need about as much information fits on an index card. Do you find that having the centralized concentration of cloud knowledge on a particular team works out or do you find that effectively doing a rotating embedding story is the better answer?

John: It varies a lot, I think, because it depends on the level of curiosity of the developers quite a lot. So, I have a huge developer background. People in my team are probably more coming from ex-IT environments or this sort of operation and then it just naturally went into the cloud. And in my opinion, is fairly rare to find somebody that is actually good at doing both AWS and coding. I am by no means really, really great at coding. I code pretty much every day but I wouldn’t call myself a professional developer.

However, it does bring to my knowledge the fact that there are some good patterns and good practices that you can bring into building your applications in the cloud and some really bad ones. However, I think it’s really down to making sure that the knowledge is here within the team. If there’s a specialized team, those really need to be specialists. And I think the important thing then is to make sure that the developers and the people around you that are curious and want to ask questions know that you’re available to them to share that knowledge. Because at the end of the day, if I’m the only one with the knowledge, I’m going to be the one who is always going to be on call for this or doing that and this is no responsibility that I want. I am happy with a number of responsibilities, but not to be the only person to ever do this. I want to go on holidays from time to time.

So, at the end of the day, I suppose it really is up to what people want or expect out of their careers. I do a job that it was a passion for me since I was about 14 years old. And I’ve always been extremely curious to understand how things work, but I do draw the line that I don’t write anything else than Python these days. And if you ask me to write Java, I’ll probably change job in the flip of a second. But that’s the end of it. But I enjoy understanding how Java things work so that I can help my developers make better choices with what services in AWS to use.

Corey: On some level, it feels like there’s a, I guess, lack of the same kind of socialization that startups have sort of been somewhat guided by as far as core ethos goes, where, oh whatever I’m working on, I want to reach out to other people, and, “Hey, I’m trying to solve this problem. What is it that you have been working on that’s germane to this and how can we collaborate together?” It has nothing to do, incidentally, with the idea that, oh, big company people aren’t friendly or are dedicated or aren’t good or aren’t well-connected; none of that. But there are so many people internally that you’re spending your time focusing on and there’s so much more internal context that doesn’t necessarily map to anything outside of the company that the idea of someone off the street who just solved a particular problem in a weird way could apply to what a larger company with, you know, regulatory burdens, starts to have in mind, it becomes a little bit further afield. Do you think that that’s accurate? Do you think that there’s still a strong sense of enterprise community that I’m just potentially not seeing in various ways because I don’t work at big companies?

John: It’s a very fine line to walk. So, when I joined the company, I was made aware that there’s a lot of Terraform and Kubernetes, which I went [unintelligible 00:16:28] all the way with CloudFormation is yes. So, that was one of the changes I knew I would have. But I can move an open mind and when I looked around at, okay, what are the Terraform modules—because I used Terraform with anger for an entire year of suffering—and I thought, “Okay, well, maybe people have actually got to a point where they’ve built great modules that I can just pick up off the shelf and reuse or customize only a tiny little bit, add maybe a couple of features and that’s, it move on; it’s good enough for me.” But as it turns out, there is I think, a lot of the time a case where the need for standardization goes against the need for business to move on.

So, I think this is where you start to see silos start to being built within the company and people do their own thing and the other ones do their own. And I think it’s always a really big challenge for a large company with extremely opinionated individuals to say, “All right, we’re going to standardize on this way.” And it definitely was one of the biggest challenge that I had when I joined the company because again, big communities and Terraform place, we’re going to need to do something else. So, then it was the case of saying, “Hey, I don’t think we need Kubernetes and I definitely don’t think we need Terraform for any the things—for any of those reasons, so how about we do something a little different?”

Corey: Speaking of doing things a little bit different, you were recently featured in an AWS Open-Source Roundup newsletter that was just where you, I think, came across my desk one of the first times, has specifically around an open-source project that you built: ECS Compose-X.

So, I assume it’s like, oh, it’s like Docker Compose for ECS and also the ‘X’ implies that it is extreme, just, like, you know, snack foods at the convenience store. What does it do and where’d it come from?

John: Right. So, you said most of it, right? It literally is a question where you take a Docker Compose file and you want to deploy your services that you worked on and all of that together, and you want to deploy it to AWS. So, ECS Compose-X is a CLI tool very much like the Copilot. I think it was released about four months just before Copilots came out—so, sorry, I beat you to the ball there—but with the Docker Compose specification supported.

And again, it was really out of I needed to find a neat way to take my services and deploy them in AWS. So, Compose-X is just a CLI tool that is going to parse your Docker Compose file and create CloudFormation templates out of it. Now, the X is not very extreme or anything like that, but it’s actually coming from the [finite 00:18:59] extension fields, which is something supported in Docker Compose. And so, you can do things like x-RDS, or x-DynamoDB, which Docker Compose on your laptop will totally ignore, but ECS Compose-X however will take that into account.

And what it will do is if you need a database or a DynamoDB table, for example, in your Docker Compose file, you do [x-RDS, my database, some properties, 00:19:22]—exactly the same properties as CloudFormation, actually—and then you say, “I want this service to have access to it in read-only fashion.” And what ECS Compose-X is going to do is just understand what it has to do when—meaning creating IAM policies, opening security groups, all of that stuff, and make all of that available to the containers in one way or another.

Corey: It feels like it’s a bit of a miss for Copilot not to do this. It feels like they wanted to go off in their own direction with the way that they viewed the world—which I get; I’m not saying there’s anything inherently wrong with that. There’s a reason that I point kubernetestheeasyway.com to the ECS marketing site—but there’s so much stuff out there that is shipped or made available in other ways with a Docker Compose file, and the question of okay, how do I take this and run it in Fargate or something because I don’t want to run it locally for whatever reason, and the answer is, “That’s the neat part. You don’t.”

And it just becomes such a clear miss. There have been questions about this Since Copilot launched. There’s a GitHub issue tracking getting support for this that was last updated in September—we are currently recording this at the end of March—it just doesn’t seem to be something that’s a priority. I mean, I will say the couple of times that I’ve used Copilot myself, it was always for greenfield experiments, never for adopting something else that already existed. And that was… it just felt like a bit of a heavy lift to me of oh, you need to know from the beginning that this is the tool you’re going to use for the thing. Docker Compose is what the ecosystem has settled on a long time ago and I really am disheartened by the fact that there’s no direct ECS support for it today.

John: Yeah, and it was definitely a motivation for me because I knew that ECS CLI version 1 was going into the sunset, and there wasn’t going to be anything supporting it. And so, I just wanted to have Docker Compose because it’s familiar to developers and again, if you want to have adoption and have people use your thing, it has to be easy. And when I looked at Copilot the first time around, I was extremely excited because I thought, “Yes, thank you, Amazon for making my life easy. I don’t have to maintain this project anymore and I’m going to be able to just lift and shift, move over, and be happy about it.” But when the specification for Copilot was out and I could go for the documentation, I was equally disheartened because I was like, “Okay, not for me.”

And something very similar happened when they announced Proton. I was extremely excited by Proton. I opened a GitHub issue on the roadmap immediately to say, “Hey, are you going to support to have some of those things together or not?” And the fact that the Proton templates—I mean, again, it was, what, two, three years ago now—and I haven’t looked at Proton since, so it was a very long time now.

Corey: The beta splasher was announced in 2020 and I really haven’t seen much from it since.

John: Well, and I haven’t done anything [unintelligible 00:22:07] with it. And literally, one of the first thing did when the project came out. Because obviously, this is an open-source project that we use in Sainsbury’s, right because we deploy everything in [ECS 00:22:17] so why would I reinvent the wheel the third time? It’s been done, I might as well leverage it. But every time something on it came out, I was seeing it as the way out of nobody’s going to need me anymore—which is great—and that doesn’t create a huge potential dependency on the company for me, oh, well, we need this to, you know, keep working.

Now, it’s open-source, it’s on the license you can fork it and do whatever you want with it, so from that point of view, nobody’s going to ask me anything in the future, but from the point of view where I need to, as much as possible, use AWS native tools, or AWS-built tools, I differently wanted every time to move over to something different. But every time I tried and tiptoed with those alternative offerings, I just went back and said, “No, this [laugh] either is too new and not mature enough yet, or my tool is just better.” Right? And one of the things I’ve been doing for the past three years is look at the Docker ECS plugin, all of the issues, and I see all of the feature requests that people are asking for and just do that in my project. And some with Copilots. The only thing that Copilot does that I don’t do is tell people how to do CI/CD pipelines.

Corey: One thing you said a second ago just sort of, I guess, sent me spiraling for a second because I distinctly remember this particular painful part. You’re right, there was an ECS CLI for a long time that has since been deprecated. But we had internal tooling built around that. When there was an issue with a particular task that failed, getting logs out of it was non-trivial, so great. Here’s the magic incantation that does it.

I still haven’t found a great way to do that with the AWS v2 CLI and that feels like it’s a gap where yes, I understand, old tools go away and new ones show up, but, “Hey, I [unintelligible 00:24:05] task. Can you tell me what the logs are?” “No. Well, Copilot’s the new answer.” “Okay. Can I use this to get logs from something that isn’t Copilot?” “Oh, absolutely not.” And the future is inherently terrible as a direct result.

John: Yeah. Well, I mean, again, the [unintelligible 00:24:20]—the only thing that ECS Compose-X does is create all the templates for you so you can, you know, then just query it and know where everything has been created. And one of the things it definitely does create is all of the log groups. Because again, least-privileged permissions being something that is very dear to me, I create the log groups and just allow the services to only write in those log groups and that’s it.

Now, typically this is not a thing that I’ve thought Compose-X was going to do because that’s not its purpose. It’s not going to be an operational tool to troubleshoot all the things and this is where I think that other projects are much better suited and I would rather use them as an extension or library of the project as opposed to reinvent them. So, if you’re trying to find a tool for yourself to look at logs, I highly recommend something called ‘AWS logs,’ which is fantastic. You just say, “Hey, can you list the groups?” “Okay.” “Can you get me the groups and can I tell them on a terminal?”

And that’s it. Job done. So, as much as I enjoy building new features into the project, for example, I think that there’s a clear definition between what the project is for and what it’s not. And what it’s for is giving people CloudFormation templates they can reuse in any region and deploy their services and not necessarily deal with their operations; that’s up to them. At the end of the day, it’s really up to the user to know what they want to do with it. I’m not trying to force anybody into doing something specific.

Corey: I would agree. I think that there’s value to there’s more than one way to do it. The problem is, at some point, there’s a tipping point where you have this proliferation of different options to the point where you end up in this analysis paralysis model where you’re too busy trying to figure out what is the next clear step. And yes, that flexibility is incredibly valuable, especially when you get into, you know, large, sophisticated enterprises—ahem, ahem—but when you’re just trying to kick the tires on something new, I feel like there’s a certain lack of golden path where in the event of not having an opinion on any of these things, this is what you should do just to keep things moving forward, as opposed to here are two equal options that you can check with radio boxes and it’s not at all clear what you which does what or what the longer-term implications are. We’ve all gotten caught with the one-way doors we didn’t realize we were passing through at the time and then had to do significant technical debt repayment efforts to wind up making it right again.

I just wish that those questions would be called out, but everything else just, it doesn’t matter. If you don’t like the name of the service that you’re creating, you can change it later. Or if you can’t, maybe you should know now, so you don’t have—in my case—a DynamoDB table that is named ‘test’ running in production forever.

John: Yeah. You’re absolutely right. And again, I think it goes back to one of the biggest challenges that I had when I joined the company, which was when I said, “I think we should be using CloudFormation, I think we should be using ECS and Terraforming Kubernetes for those reasons.” And one of the reasons was, the people. Meaning we were a very small team, only five cloud engineers at the time.

And as I joined the company, they were already was three different teams using four different CI/CD tools. And they all wanted to use Kubernetes, for example, and they were all using different CI/CD—like I said, just now—different CI/CD tools. And so, the real big challenge for me was how do I pitch that simplicity is what’s going to allow us to deliver value for the business? Because at the end of the day, like you said many, many times before, the AWS bill is a question of architecture, right? And there’s a link and intricacy between the two things.

So, the only thing that really mattered for me and the team was to find a way, find the service that was going to allow to do a number of things, A, delivering value quickly, being supported over time. Because one of the things that I think people forget these days—well, one of the things I’m allergic to and one of the things that makes me spiral is what I call CV-driven tech choices where people say, “Hey, I love this great thing I read about and I think that we should use that in production. How great idea.” But really, I don’t know anything about it and is then up to somebody else to maintain it long-term.

And that goes to the other point, which is, turnover-proof is what I call it. So, making tech choices that are going to be something that people will be able to use for many, many years, there is going to be a company behind the scenes that he’s going to be able to support you as well as you go and use the service for the many, many years to go.

Corey: I really want to thank you for taking the time to speak with me today. If people want to learn more, where’s the best place for them to find you?

John: So, people can find me on LinkedIn. I’m also around on Twitter these days, although I probably about have nine followers. Well, probably shouldn’t say that [laugh] and that doesn’t matter.

Corey: It’s fine. We’ll put a link into it—we’ll put a link to that in the [show notes 00:29:02] and maybe we’ll come up with number ten. You never know. Thanks again for your time. I really appreciate it.

John: Thanks so much, Corey, for having me.

Corey: John Mille, Principal Cloud Engineer at Sainsbury’s. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment that you go to great pains to type out but then fails to post because the version of the tool you use to submit it has been deprecated without a viable replacement.

Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.