Transcript

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. Our returning guest today is Liz Rice, who remains the Chief Open Source Officer with Isovalent. But Liz, thank you for returning, suspiciously closely timed to when you have a book coming out. Welcome back.

Liz: [laugh]. Thanks so much for having me. Yeah, I’ve just—I’ve only had the physical copy of the book in my hands for less than a week. It’s called Learning eBPF. I mean, obviously, I’m very excited.

Corey: It’s an O’Reilly book; it has some form of honeybee on the front of it as best I can tell.

Liz: Yeah, I was really pleased about that. Because eBPF has a bee as its logo, so getting a [early 00:01:17] honeybee as the O’Reilly animal on the front cover of the book was pretty pleasing, yeah.

Corey: Now, this is your second O’Reilly book, is it not?

Liz: It’s my second full book. So, I’d previously written a book on Container Security. And I’ve done a few short reports for them as well. But this is the second, you know, full-on, you can buy it on Amazon kind of book, yeah.

Corey: My business partner wrote Practical Monitoring for O’Reilly and that was such an experience that he got entirely out of observability as a field and ran running to AWS bills as a result. So, my question for you is, why would anyone do that more than once?

Liz: [laugh]. I really like explaining things. And I had a really good reaction to the Container Security book. I think already, by the time I was writing that book, I was kind of interested in eBPF. And we should probably talk about what that is, but I’ll come to that in a moment.

Yeah, so I've been really interested in eBPF, for quite a while and I wanted to be able to do the same thing in terms of explaining it to people. A book gives you a lot more opportunity to go into more detail and show people examples and get them kind of hands-on than you can do in their, you know, 40-minute conference talk. So, I wanted to do that. I will say I have written myself a note to never do a full-size book while I have a full-time job because it’s a lot [laugh].

Corey: You do have a full-time job and then some. As we mentioned, you’re the Chief Open Source Officer over at Isovalent, you are on the CNCF governing board, you’re on the board of OpenUK, and you’ve done a lot of other stuff in the open-source community as well. So, I have to ask, taking all of that together, are you just allergic to things that make money? I mean, writing the book as well on top of that. I’m told you never do it for the money piece; it’s always about the love of it. But it seems like, on some level, you’re taking it to an almost ludicrous level.

Liz: Yeah, I mean, I do get paid for my day job. So, there is that [laugh]. But so, yeah—

Corey: I feel like that’s the only way to really write a book is, in turn, to wind up only to just do it for—what someone else is paying you to for doing it, viewing it as a marketing exercise. It pays dividends, but those dividends don’t, in my experience from what I’ve heard from everyone say, pay off as of royalties on book payments.

Liz: Yeah, I mean, it’s certainly, you know, not a bad thing to have that income stream, but it certainly wouldn’t make you—you know, I’m not going to retire tomorrow on the royalty stream unless this podcast has loads and loads of people to buy the book [laugh].

Corey: Exactly. And I’m always a fan of having such [unintelligible 00:03:58]. I will order it while we’re on the call right now having this conversation because I believe in supporting the things that we want to see more of in the world. So, explain to me a little bit about what it is. Whatever you talking about learning X in a title, I find that that’s often going to be much more approachable than arcane nonsense deep-dive things.

One of the O’Reilly books that changed my understanding was Linux Kernel Internals, or Understanding the Linux Kernel. Understanding was kind of a heavy lift at that point because it got very deep very quickly, but I absolutely came away understanding what was going on a lot more effectively, even though I was so slow I needed a tow rope on some of it. When you have a book that started with learning, though, I imagined it assumes starting at zero with, “What’s eBPF?” Is that directionally correct, or does it assume that you know a lot of things you don’t?

Liz: Yeah, that’s absolutely right. I mean, I think eBPF is one of these technologies that is starting to be, particularly in the cloud-native world, you know, it comes up; it’s quite a hot technology. What it actually is, so it’s an acronym, right? EBPF. That acronym is almost meaningless now.

So, it stands for extended Berkeley Packet Filter. But I feel like it does so much more than filtering, we might as well forget that altogether. And it’s just become a term, a name in its own right if you like. And what it really does is it lets you run custom programs in the kernel so you can change the way that the kernel behaves, dynamically. And that is… it’s a superpower. It’s enabled all sorts of really cool things that we can do with that superpower.

Corey: I just pre-ordered it as a paperback on Amazon and it shows me that it is now number one new release in Linux Networking and Systems Administration, so you’re welcome. I’m sure it was me that put it over the top.

Liz: Wonderful. Thank you very much. Yeah [laugh].

Corey: Of course, of course. Writing a book is one of those things that I’ve always wanted to do, but never had the patience to sit there and do it or I thought I wasn’t prolific enough, but over the holidays, this past year, my wife and business partner and a few friends all chipped in to have all of the tweets that I’d sent bound into a series of leather volumes. Apparently, I’ve tweeted over a million words. And… yeah, oh, so I have to write a book 280 characters at a time, mostly from my phone. I should tweet less was really the takeaway that I took from a lot of that.

But that wasn’t edited, that wasn’t with an overall theme or a narrative flow the way that an actual book is. It just feels like a term paper on steroids. And I hated term papers. Love reading; not one to write it.

Liz: I don’t know whether this should make it into the podcast, but it reminded me of something that happened to my brother-in-law, who’s an artist. And he put a piece of video on YouTube. And for unknowable reasons if you mistyped YouTube, and you spelt it, U-T-U-B-E, the page that you would end up at from Google search was a YouTube video and it was in fact, my brother-in-law’s video. And people weren’t expecting to see this kind of art movie about matches burning. And he just had the worst comment—like, people were so mean in the comments. And he had millions of views because people were hitting this page by accident, and he ended up—

Corey: And he made the cardinal sin of never read the comments. Never break that rule. As soon as you do that, it doesn’t go well. I do read the comments on various podcast platforms on this show because I always tell people to insulted all they want, just make sure you leave a five-star review.

Liz: Well, he ended up publishing a book with these comments, like, one comment per page, and most of them are not safe for public consumption comments, and he just called it Feedback. It was quite something [laugh].

Corey: On some level, it feels like O’Reilly books are a little insulated from the general population when it comes to terrible nonsense comments, just because they tend to be a little bit more expensive than the typical novel you’ll see in an airport bookstore, and again, even though it is approachable, Learning eBPF isn’t exactly the sort of title that gets people to think that, “Ooh, this is going to be a heck of a thriller slash page-turner with a plot.” “Well, I found the protagonist unrelatable,” is not sort of the thing you’re going to wind up seeing in the comments because people thought it was going to be something different.

Liz: I know. One day, I’m going to have to write a technical book that is also a murder mystery. I think that would be, you know, quite an achievement. But yeah, I mean, it’s definitely aimed at people who have already come across the term, want to know more, and particularly if you’re the kind of person who doesn’t want to just have a hand-wavy explanation that involves boxes and diagrams, but if, like me, you kind of want to feel the code, and you want to see how things work and you want to work through examples, then that’s the kind of person who might—I hope—enjoy working through the book and end up with a possible mental model of how eBPF works, even though it’s essentially kernel programming.

Corey: So, I keep seeing eBPF in an increasing number of areas, a bunch of observability tools, a bunch of security tools all tend to tie into it. And I’ve seen people do interesting things as far as cost analysis with it. The problem that I run into is that I’m not able to wind up deploying it universally, just because when I’m going into a client engagement, I am there in a purely advisory sense, given that I’m biasing these days for both SaaS companies and large banks, that latter category is likely going to have some problems if I say, “Oh, just take this thing and go ahead and deploy it to your entire fleet.” If they don’t have a problem with that, I have a problem with their entire business security posture. So, I don’t get to be particularly prescriptive as far as what to do with it.

But if I were running my own environment, it is pretty clear by now that I would have explored this in some significant depth. Do you find that it tends to be something that is used primarily in microservices environments? Does it effectively require Kubernetes to become useful on day one? What is the onboard path where people would sit back and say, “Ah, this problem I’m having, eBPF sounds like the solution.”

Liz: So, when we write tools that are typically going to be some sort of infrastructure, observability, security, networking tools, if we’re writing them using eBPF, we’re instrumenting the kernel. And the kernel gets involved every time our application wants to do anything interesting because whenever it wants to read or write to a file, or send receive network messages, or write something to the screen, or allocate memory, or all of these things, the kernel has to be involved. And we can use eBPF to instrument those events and do interesting things. And the kernel doesn’t care whether those processes are running in containers, under Kubernetes, just running directly on the host; all of those things are visible to eBPF.

So, in one sense, doesn’t matter. But one of the reasons why I think we’re seeing eBPF-based tools really take off in cloud-native is that you can, by applying some programming, you can link events that happened in the kernel to specific containers in specific pods in whatever namespace and, you know, get the relationship between an event and the Kubernetes objects that are involved in that event. And then that enables a whole lot of really interesting observability or security tools and it enables us to understand how network packets are flowing between different Kubernetes objects and so on. So, it’s really having this vantage point in the kernel where we can see everything and we didn’t have to change those applications in any way to be able to use eBPF to instrument them.

Corey: When I see the stories about eBPF, it seems like it’s focused primarily on networking and flow control. That’s where I’m seeing it from a security standpoint, that’s where I’m seeing it from cost allocation aspect. Because, frankly, out of the box, from a cloud provider’s perspective, Kubernetes looks like a single-tenant application with a really weird behavioral pattern, and some of that crosstalk gets very expensive. Is there a better way than either using eBPF and/or VPC flow logs to figure out what’s talking to what in the Kubernetes ecosystem, or is BPF really your first port of call?

Liz: So, I’m coming from a position of perspective of working for the company that created the Cilium networking project. And one of the reasons why I think Cilium is really powerful is because it has this visibility—it’s got a component called Hubble—that allows you to see exactly how packets are flowing between these different Kubernetes identities. So, in a Kubernetes environment, there’s not a lot of point having network flows that talk about IP addresses and ports when what you really want to know is, what’s the Kubernetes namespace, what’s the application? Defining things in terms of IP addresses makes no sense when they’re just being refreshed and renewed every time you change pods. So yeah, Kubernetes changes the requirements on networking visibility and on firewalling as well, on network policy, and that, I think, is you don’t have to use eBPF to create those tools, but eBPF is a really powerful and efficient platform for implementing those tools, as we see in Cilium.

Corey: The only competitor I found to it that gives a reasonable explanation of why random things are transferring multiple petabytes between each other in the middle of the night has been oral tradition, where I’m talking to people who’ve been around there for a while. It’s, “So, I’m seeing this weird traffic pattern at these times a day. Any idea what that might be?” And someone will usually perk up and say, “Oh, is it—” whatever job that they’re doing. Great. That gives me a direction to go in.

But especially in this era of layoffs and as environments exist for longer and longer, you have to turn into a bit of a data center archaeologist. That remains insufficient, on some level. And some level, I’m annoyed with trying to understand or needing to use tooling like this that is honestly this powerful and this customizable, and yes, on some level, this complex in order to get access to that information in a meaningful sense. But on the other, I’m glad that that option is at least there for a lot of workloads.

Liz: Yeah. I think, you know, that speaks to the power of this new generation of tooling. And the same kind of applies to security forensics, as well, where you might have an enormous stream of events, but unless you can tie those events back to specific Kubernetes identities, which you can use eBPF-based tooling to do, then how do you—the forensics job of tying back where did that event come from, what was the container that was compromised, it becomes really, really difficult. And eBPF tools—like Cilium has a sub-project called Tetragon that is really good at this kind of tying events back to the Kubernetes pod or whether we want to know what node it was running on what namespace or whatever. That’s really useful forensic information.

Corey: Talk to me a little bit about how broadly applicable it is. Because from my understanding from our last conversation, when you were on the show a year or so ago, if memory serves, one of the powerful aspects of it was very similar to what I’ve seen some of Brendan Gregg’s nonsense doing in his kind of various talks where you can effectively write custom programming on the fly and it’ll tell you exactly what it is that you need. Is this something that can be instrument once and then effectively use it for basically anything, [OTEL 00:16:11]-style, or instead, does it need to be effectively custom configured every time you want to get a different aspect of information out of it?

Liz: It can be both of those things.

Corey: “It depends.” My least favorite but probably the most accurate answer to hear.

Liz: [laugh]. But I think Brendan did a really great—he’s done many talks talking about how powerful BPF is and built lots of specific tools, but then he’s also been involved with Bpftrace, which is kind of like a language for—a high-level language for saying what it is that you want BPF to trace out for you. So, a little bit like, I don’t know, awk but for events, you know? It’s a scripting language. So, you can have this flexibility.

And with something like Bpftrace, you don’t have to get into the weeds yourself and do kernel programming, you know, in eBPF programs. But also there’s gainful employment to be had for people who are interested in that eBPF kernel programming because, you know, I think there’s just going to be a whole range of more tools to come, you know>? I think we’re, you know, we’re seeing some really powerful tools with Cilium and Pixie and [Parker 00:17:27] and Kepler and many other tools and projects that are using eBPF. But I think there’s also a whole load of more to come as people think about different ways they can apply eBPF and instrument different parts of an overall system.

Corey: We’re doing this over audio only, but behind me on my wall is one of my least favorite gifts ever to have been received by anyone. Mike, my business partner, got me a thousand-piece puzzle of the Kubernetes container landscape where—

Liz: [laugh].

Corey: This diagram is psychotic and awful and it looks like a joke, except it’s not. And building that puzzle was maddening—obviously—but beyond that, it was a real primer in just how vast the entire container slash Kubernetes slash CNCF landscape really is. So, looking at this, I found that the only reaction that was appropriate was a sense of overwhelmed awe slash frustration, I guess. It’s one of those areas where I spend a lot of time focusing on drinking from the AWS firehose because they have a lot of products and services because their product strategy is apparently, “Yes,” and they’re updating these things in a pretty consistent cadence. Mostly. And even that feels like it’s multiple full-time jobs shoved into one.

There are hundreds of companies behind these things and all of them are in areas that are incredibly complex and difficult to go diving into. EBPF is incredibly powerful, I would say ridiculously so, but it’s also fiendishly complex, at least shoulder-surfing behind people who know what they’re doing with it has been breathtaking, on some level. How do people find themselves in a situation where doing a BPF deep dive make sense for them?

Liz: Oh, that’s a great question. So, first of all, I’m thinking is there an AWS Jigsaw as well, like the CNCF landscape Jigsaw? There should be. And how many pieces would it have? [It would be very cool 00:19:28].

Corey: No, because I think the CNCF at one point hired a graphic designer and it’s unclear that AWS has done such a thing because their icons for services are, to be generous here, not great. People have flashcards that they’ve built for is what services does logo represent? Haven’t a clue, in almost every case because I don’t care in almost every case. But yeah, I’ve toyed with the idea of doing it. It’s just not something that I’d ever want to have my name attached to it, unfortunately. But yeah, I want someone to do it and someone else to build it.

Liz: Yes. Yeah, it would need to refresh every, like, five minutes, though, as they roll out a new service.

Corey: Right. Because given that it appears from the outside to be impenetrable, it’s similar to learning VI in some cases, where oh, yeah, it’s easy to get started with to do this trivial thing. Now, step two, draw the rest of the freaking owl. Same problem there. It feels off-putting just from a perspective of you must be at least this smart to proceed. How do you find people coming to it?

Liz: Yeah, there is some truth in that, in that beyond kind of Hello World, you quite quickly start having to do things with kernel data structures. And as soon as you’re looking at kernel data structures, you have to sort of understand, you know, more about the kernel. And if you change things, you need to understand the implications of those changes. So, yeah, you can rapidly say that eBPF programming is kernel programming, so why would anybody want to do it? The reason why I do it myself is not because I’m a kernel programmer; it’s because I wanted to really understand how this is working and build up a mental model of what’s happening when I attach a program to an event. And what kinds of things can I do with that program?

And that’s the sort of exploration that I think I’m trying to encourage people to do with the book. But yes, there is going to be at some point, a pretty steep learning curve that’s kernel-related but you don’t necessarily need to know everything in order to really have a decent understanding of what eBPF is, and how you might, for example—you might be interested to see what BPF programs are running on your existing system and learn why and what they might be doing and where they’re attached and what use could that be.

Corey: Falling down that, looking at the process table once upon a time was a heck of an education, one week when I didn’t have a lot to do and I didn’t like my job in those days, where, “Oh, what is this Avahi daemon that constantly running? MDNS forwarding? Who would need that?” And sure enough, that tickled something in the back of my mind when I wound up building out my networking box here on top of BSD, and oh, yeah, I want to make sure that I can still have discovery work from the IoT subnet over to whatever it is that my normal devices live. Ah, that’s what that thing always running for. Great for that one use case. Almost never needed in other cases, but awesome. Like, you fire up a Raspberry Pi. It’s, “Why are all these things running when I’m just want to have an embedded device that does exactly one thing well?” Ugh. Computers have gotten complicated.

Liz: I know. It’s like when you get those pop-ups on—well certainly on Mac, and you get pop-ups occasionally, let’s say there’s such and such a daemon wants extra permissions, and you think I’m not hitting that yes button until I understand what that daemon is. And it turns out, it’s related, something completely innocuous that you’ve actually paid for, but just under a different name. Very annoying. So, if you have some kind of instrumentation like tracing or logging or security tooling that you want to apply to all of your containers, one of the things you can use is a sidecar container approach. And in Kubernetes, that means you inject the sidecar into every single pod. And—

Corey: Yes. Of course, the answer to any Kubernetes problem appears to be have you tried running additional containers?

Liz: Well, right. And there are challenges that can come from that. And one of the reasons why you have to do that is because if you want a tool that has visibility over that container that’s inside the pod, well, your instrumentation has to also be inside the pod so that it has visibility because your pod is, by design, isolated from the host it’s running on. But with eBPF, well eBPF is in the kernel and there’s only one kernel, however many containers were running. So, there is no kind of isolation between the host and the containers at the kernel level.

So, that means if we can instrument the kernel, we don’t have to have a separate instance in every single pod. And that’s really great for all sorts of resource usage, it means you don’t have to worry about how you get those sidecars into those pods in the first place, you know that every pod is going to be instrumented if it’s instrumented in the kernel. And then for service mesh, service mesh usually uses a sidecar as a Layer 7 Proxy injected into every pod. And that actually makes for a pretty convoluted networking path for a packet to sort of go from the application, through the proxy, out to the host, back into another pod, through another proxy, into the application.

What we can do with eBPF, we still need a proxy running in userspace, but we don’t need to have one in every single pod because we can connect the networking namespaces much more efficiently. So, that was essentially the basis for sidecarless service mesh, which we did in Cilium, Istio, and now we’re using a similar sort of approach with Ambient Mesh. So that, again, you know, avoiding having the overhead of a sidecar in every pod. So that, you know, seems to be the way forward for service mesh as well as other types of instrumentation: avoiding sidecars.

Corey: On some level, avoiding things that are Kubernetes staples seems to be a best practice in a bunch of different directions. It feels like it’s an area where you start to get aligned with the idea of service meesh—yes, that’s how I pluralize the term service mesh and if people have a problem with that, please, it’s imperative you’ve not send me letters about it—but this idea of discovering where things are in a variety of ways within a cluster, where things can talk to each other, when nothing is deterministically placed, it feels like it is screaming out for something like this.

Liz: And when you think about it, Kubernetes does sort of already have that at the level of a service, you know? Services are discoverable through native Kubernetes. There’s a bunch of other capabilities that we tend to associate with service mesh like observability or encrypted traffic or retries, that kind of thing. But one of the things that we’re doing with Cilium, in general, is to say, but a lot of this is just a feature of the networking, the underlying networking capability. So, for example, we’ve got next generation mutual authentication approach, which is using SPIFFE IDs between an application pod and another application pod. So, it’s like the equivalent of mTLS.

But the certificates are actually being passed into the kernel and the encryption is happening at the kernel level. And it’s a really neat way of saying we don’t need… we don’t need to have a sidecar proxy in every pod in order to terminate those TLS connections on behalf of the application. We can have the kernel do it for us and that’s really cool.

Corey: Yeah, at some level, I find that it still feels weird—because I’m old—to have this idea of one shared kernel running a bunch of different containers. I got past that just by not requiring that [unintelligible 00:27:32] workloads need to run isolated having containers run on the same physical host. I found that, for example, running some stuff, even in my home environment for IoT stuff, things that I don’t particularly trust run inside of KVM on top of something as opposed to just running it as a container on a cluster. Almost certainly stupendous overkill for what I’m dealing with, but it’s a good practice to be in to start thinking about this. To my understanding, this is part of what AWS’s Firecracker project starts to address a bit more effectively: fast provisioning, but still being able to use different primitives as far as isolation boundaries go. But, on some level, it’s nice to not have to think about this stuff, but that’s dangerous.

Liz: [laugh]. Yeah, exactly. Firecracker is really nice way of saying, “Actually, we’re going to spin up a whole VM,” but we don’t ne—when I say ‘whole VM,’ we don’t need all of the things that you normally get in a VM. We can get rid of a ton of things and just have the essentials for running that Lambda or container service, and it becomes a really nice lightweight solution. But yes, that will have its own kernel, so unlike, you know, running multiple kernels on the same VM where—sorry, running multiple containers on the same virtual machine where they would all be sharing one kernel, with Firecracker you’ll get a kernel per instance of Firecracker.

Corey: The last question I have for you before we wind up wrapping up this episode harkens back to something you said a little bit earlier. This stuff is incredibly technically nuanced and deep. You clearly have a thorough understanding of it, but you also have what I think many people do not realize is an orthogonal skill of being able to articulate and explain those complex concepts simply an approachably, in ways that make people understand what it is you’re talking about, but also don’t feel like they’re being spoken to in a way that’s highly condescending, which is another failure mode. I think it is not particularly well understood, particularly in the engineering community, that there are—these are different skill sets that do not necessarily align congruently. Is this something you’ve always known or is this something you’ve figured out as you’ve evolved your career that, oh I have a certain flair for this?

Liz: Yeah, I definitely didn’t always know it. And I started to realize it based on feedback that people have given me about talks and articles I’d written. I think I’ve always felt that when people use jargon or they use complicated language or they, kind of, make assumptions about how things are, it quite often speaks to them not having a full understanding of what’s happening. If I want to explain something to myself, I’m going to use straightforward language to explain it to myself [laugh] so I can hold it in my head. And I think people appreciate that.

And you can get really—you know, you can get quite in-depth into something if you just start, step by step, build it up, explain everything as you go along the way. And yeah, I think people do appreciate that. And I think people, if they get lost in jargon, it doesn’t help anybody. And yeah, I very much appreciate it when people say that, you know, they saw a talk or they read something I wrote and it meant that they finally grokked whatever that concept was that that I was trying to explain. I will say at the weekend, I asked ChatGPT to explain DNS in the style of Liz Rice, and it started off, it was basically, “Hello there. I’m Liz Rice and I’m here to explain DNS in very simple terms.” I thought, “Okay.” [laugh].

Corey: Every time I think I’ve understood DNS, there’s another level to it.

Liz: I’m pretty sure there is a lot about DNS that I don’t understand, yeah. So, you know, there’s always more to learn out there.

Corey: There’s certainly is. I really want to thank you for taking time to speak with me today about what you’re up to. Where’s the best place for people to find you to learn more? And of course, to buy the book.

Liz: Yeah, so I am Liz Rice pretty much everywhere, all over the internet. There is a GitHub repo that accompanies the books that you can find that on GitHub: lizRice/learning-eBPF. So, that’s a good place to find some of the example code, and it will obviously link to where you can download the book or buy it because you can pay for it; you can also download it from Isovalent for the price of your contact details. So, there are lots of options.

Corey: Excellent. And we will, of course, put links to that in the [show notes 00:32:08]. Thank you so much for your time. It’s always great to talk to you.

Liz: It’s always a pleasure, so thanks very much for having me, Corey.

Corey: Liz Rice, Chief Open Source Officer at Isovalent. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that you have somehow discovered this episode by googling for knitting projects.

Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.