Ben Kehoe: Serverless Robots, Navigating IAM on AWS, and Creating AWS CLI Tools

About Ben

Ben Kehoe works in the field of Cloud Robotics—using the internet to enable robots to do more and better things—an area of IoT involving computation in the cloud and at the edge, Big Data, and machine learning. Approaching cloud computing from this angle, Ben focuses on developing business value rapidly through serverless (and servicefull) applications.

Ben seeks to amplify voices from dev, operations, and security to help the community shape the evolution of serverless and event-driven designs for IoT and cloud computing more broadly.

You can find him online on Twitter, LinkedIn, and his blog.

Transcript

Adam: Hey, everyone. Welcome to AWS FM, a live audio show with guests from around the AWS community. I'm your host, Adam Elmore. And today I'm joined by Ben Kehoe. Hi, Ben.

Ben: Hello. Thanks for having me.

Adam: Yeah, thanks for joining. Ben, you've been in Serverless in the AWS community prominently for years now. And I guess, my first question is for those that somehow they're living under a rock, they don't know who you are. Could you just share kind of your story, your whole career arc sort of to this point?

Ben: Sure. Yeah, it's been quite a winding road. My undergraduate is nothing related to computers. I was a physics math and technical theater.

Adam: That's an awesome combination.

Ben: Yeah. I built things for theater. And then subsequent to that, I knew I wanted to go to grad school, but I didn't know what in. So I said, "Okay, I'll work for a bit." And I found a software company that would hire me, which was Infosys, the giant Indian IT contractors. I lived in India for a bit for them and then worked with clients as just a software contractor here in the US after that. And then I went to graduate school, UC Berkeley, initially working on unmanned aerial vehicles in particular how multiple of them can work together to accomplish tasks which is a sort of different kind of cloud robotics. And then part way through that-

Adam: Yeah. Sounds high stakes.

Ben: Yeah. I switched to working on things more properly related to cloud computing and robotics, things like grasping. And then as I was getting towards graduating in 2014, I started thinking about how could you start removing... Because in robotics you have lots of people writing software and not good ways of providing that software to other people, especially seven, eight years ago. And so started to think about how could you take robotics algorithms and data and start providing those as a service.

Ben: And then in late 2014, Lambda was announced and I was like, "Oh, this sounds like something that would help with that." And so when I graduated, I joined up with iRobot. So I'm a cloud robotics research scientist at iRobot. So I kind of collect all the buzz words, cloud, Serverless, all the things.

Adam: Best word, bingo.

Ben: Yeah. And so I've been there ever since working on how we enable connected robotics. So when a robot is connected to the cloud, it can do more and better things. And earlier in my time at iRobot it was more about like, what could those things be? And that sort of shifted into, well, how do we build on the cloud side of that? And that was where Serverless came in for me, because I didn't know how to make scalable server or container based applications so using services that didn't require me to know how to do that. I really-

Adam: It's helpful. Yeah.

Ben: I still don't know. I still can't do it.

Adam: You don't need to know, you've got Serverless. So I guess your role today, like I read your title and every time I see it I forget like research in the name. I mean, I think you're just so prominent in this sort of Twitter, AWS community, and you speak to Serverless and IAM and all these things that you're so knowledgeable on, but could you explain a bit just your day job? You're doing something with robots and I don't really understand, and I want to.

Ben: Yeah. I mean, it's primarily things about how do we use AWS to enable what the business problems that we're trying to solve? Which is what Serverless is all about, is how do you focus on business problems rather than technology problems? And so things like AWS SSO are part of that. How do you enable easier and better sign onto AWS, removes classes of problems that individual organizations need to solve now it's solved on the AWS side, or when should or should we not use an SQS 502 is one of those things. So topics like that are, I think, a lot of my day job.

Adam: Yeah. Okay. So I've seen like a video of you sitting in like an office. It was like a commercial, very highly produced. Is that your actual office? There's like devices all around. It's very cool, workshop looking setting?

Ben: Yes. So back in the before times, the headquarters for iRobot is in outside of Boston in Bedford, Massachusetts. There's also an office in the US in Pasadena that also does engineering work and there are offices around the world that do other sort of various functions, sales and commercial stuff, I guess. And so that was when I worked in Bedford. And you'll actually notice, I think Richard Boyd is in that video, he's a developer advocate at AWS now who is at iRobot for a while.

Adam: Yeah. I didn't even realize that.

Ben: And I think so. I think that was, yeah. And I don't live in Boston anymore. I now live in Santa Fe, so I'm completely remote, even separate from the COVID-19 parts.

Adam: Yeah. I guess you've been in the Serverless space for so long and you've been running Serverless in production for so long. I feel like if anyone can speak to sort of how Serverless has evolved over the last five years, it would be you, could you kind of talk about how you view that arc today?

Ben: Sure. So there's the roots of Serverless go back a very long way. And there are things that there are technologies that could be called Serverless that have existed in the early 2000s, things like that. And it's not inherent, the difference between Serverless and platform as a service is fairly fuzzy. So it's not something where all of a sudden there was tech that did things that didn't exist before, but there was a level of recognition that comprehensive systems could be built using these technologies, which wasn't true before.

Ben: Before Lambda, you could be running on a platform as a service, but the scaling was still down to you. It could be scaling up, but the units were tended to be fairly coarse-grained. And so with Lambda and sort of the very transparent scaling the lack of the need to acknowledge that servers exist underneath meant that an entire system that's using API gateway, which itself wasn't a new concept. Having an API layer or a load balancing layer, whatever it is that was a service, that wasn't inherently new.

Ben: S3, DynamoDB had been around for a long time. So all the other parts were sort of already there as the compute layer that was new. So then people started trying to figure out, okay, well, how do we do this? And that was happening in sort of 2015 and 2016. And in 2016 was the first Serverless config where some people organized a warehouse in Brooklyn that had no air conditioning in the summer. And we got together and we all talked about, okay, well, none of us really know what we're doing, so what's happening?

Ben: And from there we all had high hopes that this was the immediate future. I think then Kubernetes came along and presented something that's very much the faster horse. It says, we can solve a lot of these problems that are present in server full technologies without changing the paradigm. And so people I think were attracted to that to say, "Oh, I don't need to rethink how I build things. This things solve those problems." And so it is doing helpful things, which is solving those problems for people, but it's still, there's still a gap from there to Serverless technology where you take them and say, none of those are your problem anymore. And so that set back Serverless adoption by several years.

Adam: Interesting.

Ben: And so it's, it's unusual. I think if we look back and sort of the history of technology where something has come out, the eventual future has come out earlier and then something that says, well, we can solve this for a while, and so we'll end up in a place where in 2015, 2016, we saw what 2025 looks like, but we thought it was going to be 2020.

Adam: Yeah. Interesting. Yeah, no, that sort of takes me to like listening to Simon Wardley, I feel like some of the stuff I read from him that sort of echoes that, but I didn't realize, I guess I'm like kind of thinking of things from a 2021 perspective, and I see the raging debate sort of Serverless and containers and Kubernetes. And I guess I didn't realize it goes all the way back to sort of the origins. I've just not been around long enough.

Ben: There's this also sort of logistical problem in this where, what the word container means. Like it's an overloaded term in terms of like, oh, Lambda does containers now, but it accepts container images as a packaging format. It has nothing involved with container orchestration. But when people say, oh, Lambda does containers, so it's not that different between ECS or Kubernetes and Lambda. It's like, well, no, they're using container to mean container orchestration and acquainted right there that's not the same.

Adam: So this is a random question that just came to mind, is Fargate Serverless? I don't know, maybe you've already spoken to this.

Ben: Well, yeah. So there's a bit in something like 20 16, 20 17, I wrote an article about how Serverless is a spectrum and there's no, it is Serverless or it isn't Serverless. And is it taking away con technology concerns that are not a differentiator? Yes, Fargate does that. Does it do less of that than Lambda does? Yes. So I think it's certainly more Serverless than ECS on EC2, but it's less Serverless.

Adam: But less, so it's a spectrum. Okay.

Ben: Well, so I wrote that sort of partway into this journey. And then I think in 2019 I wrote an article on how to talk about Serverless as a mindset. That it's really less about the technology itself and more about how you're approaching building things within the constraints that you have. And so it may be more advantageous for someone to move from VMs on prem, to VMs in the cloud versus moving on-prem VMs to on-prem containers. But which of those is the better and more, moves you in a more Serverless direction? Depends on which one is helping you focus better rather than-

Adam: Fargate says Serverless is all about focus.

Ben: Yeah. And that's the thing. Is, is it helping you focus on what differentiates you? And if it moves you in that direction, you are moving in the Serverless direction.

Adam: Yeah. So I had seen on the show from LIGO, and I know we talked on that show about the Serverless mindset, and I know he was inspired by you and the work you've done. In terms of the Serverless mindset, so if we think of like Serverless adoption as this sort of scale from zero to 10 or something where the technology is at a certain point, do you feel like is the mindset shift sort of lagging behind that? Have people adopted Serverless technologies and still not moved into the mindset on average? Or do you think it's hard to gauge?

Ben: Oh, I absolutely think so. I think if you look at how people view step functions, for example. Step functions removes again, a whole class of problems that you have versus writing some orchestration within a Lambda function. And people don't always view that. If they view it as, oh, well, this is something that I could use, but I'm more comfortable in code. And therefore I'm going to use a Lambda rather than an express step function that's not using the Serverless mindset.

Ben: And so I think there are people who are using Serverless technology that are not really applying the mindset. I definitely see that where people don't always think bigger than Lambda, like Lambda is not the end state of Serverless compute. There are things that we need, I wrote an article on this recently of people are saying, "Well, I want a Serverless compute that runs longer than 15 minutes." That's a completely valid request. I don't think that should be Lambda as we understand it today.

Ben: It shouldn't just be, oh, you take Lambda and you extend the timeout, the trade-offs that you need to make inside services in terms of latency or any of those things, costs, are probably going to be different for something that you have running for an hour versus something that's able to run for 50 milliseconds.

Adam: So just totally different services.

Ben: Right. So ask, so the thing we should be asking for is I have jobs that run for an hour, this is what I care about that job. I care about the completing, I care about if I can pause it in the middle, if I can address it during the execution, those kinds of things. I don't care about that for an API handler, but I care about the API handler being able to return in 50 milliseconds. I don't care about my hour long job starting within 50 milliseconds, all of that is noise.

Ben: So presenting those trade-offs and saying, "Build me something, whether it's US or any buddy, any vendor that you're asking to build this presenting, not the tactical request to extend my Lambda timeout, but saying, "Here's this thing that I want think big and solve that problem in the best way for me."

Adam: So it's focusing on your business problem just like you should as you're building out.

Ben: Yeah.

Adam: Yeah. Okay. No, that makes perfect sense. I guess, in terms of the Serverless developer experience, because it could all be easier to do the right thing. It's some future state, I would think there are steps being made. It feels like recently there's just this sort of fever pitch of new things in Serverless land. Do you have opinions on how all that's going, where we're at in terms of making the dev X better?

Ben: Yeah. I mean, so this is where we can get into a long and controversial discussion about the CDK.

Adam: Well, I love it. Let's do it. Should I not use the CDK? Ben breath started kind of convincing me that maybe I thought about it all wrong. So I want to hear more.

Ben: So there's two things. Well, let me put it this way. There are three things, I'm mildly against defining your infrastructure, your resource graph with imperative code versus declarative code. And I'm moderately against not having your resource graph definition in forced to be deterministic.

Adam: Could you explain that for the audience that probably didn't follow?

Ben: If you're running a CDK program, you can go reach out to the internet. You can use a random number generator, you can do anything that's possible in the programming language that you're writing, which means the CDK program doesn't need to produce the same output between different runs. And that's something that I'm more against than the imperative versus declarative debate. But the thing that I'm really strongly against is developer intent remaining client side.

Ben: So for me, primary flaw of the CDK is a fixable one, which is CDK sense that the notion should be that what you want is that the CDK program that you write is the thing you bring to the cloud. And the cloud understands your program as a definition, the program is not a thing that produces a definition, it is the definition. And if you do that, you gain all sorts of things. That ideally your variable names are your resources logical ideas, like why not? And that drift detection, so if you said, "Here's my CDK program, this is my definition of my cloud application that I want."

Ben: And you say, "Okay, great, I have deployed that, but somebody went and made some tweaks through the console." Tell me how it's different. The ideal state is produce for me a new CDK program that I can download, that I can do a file level diff, and it says, here's the difference between a CDK program that produces what currently exists and what you define to be your desired state and it's oh yeah, there's a new line here that modifies this definition. And so this notion that a lossy generative step that's transpiling between this imperative definition and a declarative definition is I think the thing that needs to go.

Adam: And do you think that's possible? Do you think they could... So would that involve just not turning CDK into CloudFormation? If that goes away, how does it work?

Ben: Well, it's a whole long road to that point. Like the idea that I just presented of like being able to produce a program that has a meaningful file level diff is an extraordinarily hard problem, I think, but the idea of CDK synth should start by doing less stuff. So things that constructs can do today should be representable in CloudFormation. And so that the template that comes out is closer to the higher level abstractions that are present in the program down to things like there should be a map function that allows you to replicate something multiple times.

Ben: That's not something you can do to produce a resource in every AC, that kind of a thing. If you could represent that, that's a useful thing that CDK does for developers is that kind of thing. And if that was representable in native CloudFormation, then when that happens inside the CDK, the developer writes it, the sense step doesn't actually run that loop, it just translates it into the template. So then it starts doing less and less stuff. And that puts pressure then on what the cloud formation language needs to be able to represent. Until at some point it's very close and able to just say, well, it's really just a transparent step of transpiling. And at that point, you don't need that step.

Adam: But at that point, I guess, what is the CDK offering? I guess it's an escape for me, Elmore, that you've got brackets and semi-colons, if you choose.

Ben: Yeah. This is a hot tech, but 90% of the CDK values tab completion and that's really valuable. And it's writing it in a language that people are familiar to, it's lowering that bar. And so at that point, that thing that I say, which is, I think declarative is better than imperative for one thing that in general, in a declarative language syntactical correctness is semantic correctness that it tends to be hard to write a CloudFormation template that does not do what you expect it to do, whereas that is the default state of a general programming language.

Adam: Yeah. Sure.

Ben: And so I think to me, that's a reason to go declarative, but it becomes in that very future state a personal preference thing, a thing you can have idle debates on a Friday afternoon about, rather than it being this enables one kind of developer experience and versus another kind of developer experience.

Adam: Yep. So I guess, how do you view something like SAM that's closer to the CloudFormation, just like another attraction on top of that?

Ben: Yeah. I think SAM is a better approach because it stays within CloudFormation, I think. Another near term thing that we should hope to see on the CloudFormation roadmap is CDK constructs that don't have non-local effects should be able to be registered as CloudFormation types. We have modules now in the CloudFormation registry, that's a template being able to run some code that generates a template should also be a module. And they shouldn't inherently have to be different registry types. You shouldn't have to know someone wrote this as a CDK construct versus someone wrote this as a template. You should just be able to use them. And then again, we start to get closer to, there isn't a distinction between them. There are different ways of doing the same thing.

Adam: Yeah. So do you have any opinions on some of the third party offerings? So there's others, so Serverless, Inc. has done some stuff recently that's maybe a little more on the newcomer to the cloud, maybe, I don't know, the Serverless cloud stuff. So it's sort of deriving your infrastructure from the actual application.

Ben: Yes.

Adam: And then there's others, there's... Yeah, go ahead.

Ben: And so if I talk about Pulumi, for example. So one of the features of Pulumi is the ability to in-line code in your infrastructure definition. And I feel pretty strongly that's a bad approach because the amount of ownership that custom code adds to your infrastructure is large. That's exactly what we're trying to get rid of, is to own and manage and maintain a piece of it. And when the barrier to doing that is too low and too transparent, oh, I'm connecting API gateway into DynamoDB, and it'd be kind of complicated to make it such that it's a direct integration.

Ben: So I'm just going to add a two line JavaScript function that does some transformation on it. All of a sudden you've brought in a whole new set of concerns that looks barely any different in your infrastructure definition. And so as a developer, you're going to think less about that. So in some sense for me, I don't view myself or others as entirely rational creatures. And therefore, when I think about how I want my developer experience to do, I want it to add friction in the places that I should stay away from that.

Ben: That's not the case that everything should be zero friction and I will be able to make the right choices. If you make the bad thing easy, I will make that choice because it's easy. And I think that's true of all people, that it's hard to fight against something being easy and continually. And so we should strategically choose where we want to really focus on reducing friction into the places that's where we want people to go rather than where we don't want them to go.

Adam: Yeah. Well, that's great perspective, it reminds me of like Atomic Habits, making bad habits, more difficult, making good habits easier. It's like the whole pit of success, foot guns, the whole thing. And that gives me a whole new kind of angle of looking at the CDK, because I did sort of look at it very differently before I think just thinking this is so cool. I can say grant read and all this stuff gets generated.

Ben: Yeah. Well, I think there's two things there. One is, if we talk about like grant read, the effect of that call may not be local to where it is being called. That permission may end up in a stack that's very far away from the bucket that you're operating, that you're trying to grant. And that's something that I feel like is sort of a coach smell for the CDK of like that how the developer is thinking about where those things are localized does not correspond with what it's outputting, which is another reason that we want to carry that context further forward so that it can remain more local.

Ben: The other thing is, I think there are two different audiences for the CDK. There are people who are proficient with cloud formation that can understand the output. So that, oh, this is a way for me to produce CloudFormation that I could have written myself more quickly.

Adam: Just saves time, yeah.

Ben: And those people then become advocates that the other audience, which is people who are new to AWS here that don't know how to read the CloudFormation. The CDK is sort of trying to hide from them. It aims that they don't need to understand that, but that also then means both, they don't fully necessarily know what's being deployed or understand it completely because they can't go read that template and fully understand it. But then also the notion that if they need to make the move to CloudFormation or understand CloudFormation or navigate the deployed resources, the CDK is not giving them a good off-ramp.

Ben: Tends to get talked about as an escape hatch. And I think about escape hatches in terms of submarines, go from being warm and dry and having access to being hundreds of feet under the ocean. And so you don't want to get thrown into the deep end through an escape hatch, you want something that's an off-ramp. I think Amplify has a similar approach saying, "Let's manage all this stuff for people." But they're thinking about it corresponds with how do we help them graduate from Amplify, doing things for them to them doing it for themselves?

Ben: Now the CDK doesn't need graduation in that far future. The ideal should be, oh, you don't need to change languages at any point, you can stay in the language that you've originally written and understand everything through that.

Adam: I've never thought about the term escape hatch. What are you escaping from? Just the connotation of that. Graduation sounds a lot nicer.

Ben: Yeah.

Adam: So I want to do something here, just as you were speaking, I'm thinking about things I've written and what you would say if you're over my shoulder. And so there's like the segment that AWS does, that's like the... What does it view my architecture or inside my architecture?

Ben: Yep.

Adam: So this is going to be kind of a light version of that. So I'm going to describe things I build, and I want you to tell me on a scale of zero to 10, so we'll say zero is like a... now what's a really bad... Like putting public your access keys, your secret keys in a public GitHub repo.

Ben: That's like negative infinity.

Adam: Okay.

Ben: Let's start zero, a little better than that.

Adam: Yeah. Okay. So 10 would be like navigating drones in the cloud, whatever you're doing there with robots, zero would be really, really bad. So if I have a CDK app and I like to use AppSync, I don't use Amplify that much, but I like to do AppSync stuff, graph QL stuff just makes it easier for me on the front end. So I've got an AppSync API in the CDK and I'm adding like resolvers to that API. And I'm just inlining VTL templates in string literals and TypeScript. Where does this land on the scale?

Ben: Well, I think VTL is more Serverless than a Lambda resolver, right?

Adam: Sure.

Ben: You've written it once, it works, you never touch it again. It's something you struggle with. VTL is not-

Adam: The writing it part, yeah. Sure.

Ben: And especially inlining it as a string. This is sort of on the developer experience side of saying, when we have these things that allow us to do these more Serverless forms of producing it, but we have to embed that in something that doesn't understand. If you embed a state machine definition as YAML inside a CloudFormation template, it still doesn't really understand that state machine definition separate from it just being a CloudFormation resource property. And similar is true, if you have a string that contains VTL your TypeScript parser is not going to say, "Hey, you've got an error in your VTL here."

Adam: Right. You're learning on the fly, you're deploying and you're...

Ben: And so this gets to, that's doing the better thing is harder because you have to struggle with that. Whereas if you were writing the Lambda that did all those transformations, it would feel better, it'd be more fun.

Adam: Sure.

Ben: It might even be less time to create, but your overall, your total cost of ownership is higher. And so doing the annoying thing, wrestling with the VTL for an hour and a half and finally it works, but then you never need to touch it again, is the more Serverless choice there.

Adam: Yeah. Okay. So VTL I got points for that and then the city came in.

Ben: We should be asking for things that are better than VTL for those things and we shouldn't expect to have to put it inside a Lambda to get it.

Adam: Got it. Yeah. I don't know. Sometimes I sort of do a lot of things in a vacuum and I just, I call it a wonder, where am I making it harder for myself that I'm also working sort of solo as like a freelancer not sort of on a team where I have to worry about what they think. But it's nice to get your opinion. So I guess that kind of speaks a little bit to the local testing. There's sort of like testing the cloud versus local testing.

Adam: And the reason I'm asking you this, I've asked a lot of guests this, and everyone has unanimously so far, and we're 14, 15 episodes, and everyone has said, just deploy to the cloud, don't try to emulate locally, but I have Brian LaRue on on Tuesday. And so I need to have a good argument, he's going to have the other side of the argument. I need to know how do I debate Brian, because I'm not smart enough. So tell me, what is my argument for the test only in the cloud?

Ben: Well, the argument is that your local mocking will never be able to keep up and fully replicate the cloud. And one of the biggest concerns that a developer should have with that is that when something new comes out or there's a service that isn't mocked by their developer, their dev tools, they won't be able to run their local tests. And then they're going to say, "Well, I'm going to not use that because I can't run local tests with it."

Ben: And then you're dictating your production architecture based around your dev tooling and that's backwards. And so when you choose, I'm always going to deploy, then that you can always, you can use whatever you want. I think I tend to think about local testing without making API calls. That testing my business logic is things I can do locally, testing how that actually plugs into the services is the thing that I have to deploy for.

Ben: And then that also goes to a little bit that same question that we talked about earlier in terms of long running compute being Lambda versus something else. People who say, "Oh, I want local because it's faster." When what we should be asking for is make the cloud faster, I want it to be faster. Local is an answer to that, but another answer is what we're seeing in SAM Accelerate where there are ways, and Brian's Architects has had this for a long time with its dirty deploy work and go direct.

Ben: And understanding that as a development process, it's fine if you're modifying things directly, because you can tear it all down and stand it back up. And so that we should be looking for that and ways in which that code can be more like your dev loop is closer to the cloud, it's not even true that your code has to be local. What if you could hot modify code that was running in Lambda, who knows? Without it, actually-

Adam: I mean, we have the cloud, we have the IDEs and the browser now, why not? Yeah, that makes sense.

Ben: Well, and I'm saying not even that you're editing the code using a browser, but the code, the file changes that you're making are somehow within Lambda itself.

Adam: Yeah, the run time context.

Ben: And we're also seeing separate from AWS, I mean, the Serverless cloud is working on a little bit of this and Dark had these concepts where it says you make changes in the editor and they go directly to production. You're able to directly see code flung through it. I have a lot of questions about security and data governance related to that, but it's a very interesting concept of saying that if you integrate the actual experience altogether, the distance between a keystroke and production can literally be milliseconds.

Adam: Yeah, because I know that's the argument that Brian... And I think I misrepresented even the question because I know every time I say testing and I see his responses to that, it's not about testing, it's about that local dev loop, that sort of feedback loop. But to your point, if that could all just be the same speed, but it's all running in the cloud, that's the kind of best of all the worlds.

Ben: Yeah.

Adam: And so you talked about security, you've sort of, you're an IAM expert. You are very much like a voice of security on Twitter, that's where I get all my information. So I guess how long have you been sort of security first minded, like this IAM guard?

Ben: Sure.

Adam: I mean, I think that's a little excessive, but really over the top there.

Ben: Well, because I'm not an information security person. My goal is to understand how to ask the right questions in terms of what the threat model should be, what the risk trade-offs are? I'm not good at actually judging those things. I'm good at saying, "It seems like we've got an avenue here, can someone come look at this?" And so with IAM, I'm able to understand, well, this is how the system works, but the choice of, well, what's the trade-off between providing a few more actions in this policy today for this person who isn't using it today, but might need it a month from now.

Ben: So then we wouldn't have to touch it versus having them come back to us all the time. That's a risk trade off that you have to make and for, oh, what's the likelihood that their credentials get compromised? All of those kinds of things I'm completely in the dark on those topics. So I think, well, it's always been an interest of mine. I think it goes back a little bit to that notion of not trusting myself.

Ben: A lot of developers when they perceive security, they're like, "Oh, well I'm not going to get fished." Or, "I'm not going to download malware or something like that. And therefore the security measures are not important for me to conform to." Those kinds of things. And my opinion is, "No, absolutely." Even the people who are creating phishing campaigns will sometimes click on phishing emails because you just can't...

Ben: And so once you start taking that mindset of, it's not that you don't trust people, is that people are fallible, that you need security and you need to think about it and you need to think, you need to plan for compromises to happen. Rather than saying, how do we make this never happen? It's well, let's plan for it to happen and try and keep it from happening, but when it happens that everything is not lost all at once.

Ben: And so then with, and then of course the other thing is, getting credentials for IAM roles on AWS has always been a headache. And so everybody's got, because you don't want to use IAM users because you don't want long-term credentials. And so you want to federate an IDP like ADFS or Okta or paying or whatever into your roles through assume role with YAML. And you go that route and there is no standard way of doing that integration.

Ben: So everybody's got a different custom script that their organization wrote or there's Okta AWS and AWS Fault can sometimes do it. there's all these tools and there's a thousand of them and most of them are custom and none of them work well programmatically, all of those things. So the native SSL comes along and says, "Hey, we can actually encapsulate the communication with the IVP entirely browser side only behind the AWS API."

Ben: And now there's a programmatic way of getting credentials. And that also includes moving the management of AWS permissions from the IDP side to the AWS side. And one thing that I often talk about with AWS teams is thinking through their personas and what if they're not friends. And so if, when you make a change to your AWS permissions and you have to go to the people who own Ping in your organization to go change it and you're not on good terms, that's going to be a headache.

Ben: But with AWS as a setup, all you're asking for is, "Hey, can you create a group?" And then the permissions of that group are managed over on your side. That's nicer, that's easier. And so the separation of concerns is also better with AWS SSO. And then on top of that, it enables infrastructure as code for AWS permissions. So it really ushers in so many things that are beneficial and that more than anything is what has led me further into the IAM space. I was thinking about that, thinking about tag based access control, those things have put me on a path to understand IAM better.

Adam: Yeah. So I just had the thought, as you were speaking, I don't think the CDK supports SSO credentials or at least it didn't last I checked.

Ben: This is not the CDKs problem. Sort of is the JavaScript SDK V2 does not have AWS SSO support. JavaScript SDK V3 does, but the CDK and other tools are built on V2. And so it's not really the CDKs fault that they didn't support it.

Adam: Yeah, I didn't realize that. Okay.

Ben: Now they do need to migrate to V3 at some point, that's a big project.

Adam: Oh, sure.

Ben: It'd be better if the SDK V2 just added it. Unfortunately, and I didn't realize this until recently, the JavaScript SDK V2 supports credential processes, but it only supports them if they're defined in the credentials file, not in the config file.

Adam: Really?

Ben: Absolutely baffling. And what it means is that the approach that I took with aws-sso-util which is that it puts a credential process in there that can backfill for lack of support. It puts that in config because that's the right place for it, but it doesn't work with the JavaScript SDK V2. So those people aren't getting that benefit. So I need to look if I can make it write that into the credentials file as well.

Adam: Yeah. And I wanted to talk about that a little bit. You've written several sort of AWS CLI tools, smoothing rough edges. Any of those you want to call out?

Ben: That should be necessary.

Adam: They're all just like stop gaps until AWS incorporates stuff.

Ben: So aws-sso-util is generic thing for all the places that I find the tooling for AWS SSO to be inadequate. So that includes for CICD your ability to define it in CloudFormation the high cardinality of assignments that you need. You don't want to define those one by one. For a login that integrates the SSO login, like the CLI requires you to specify a profile, but the profile doesn't matter, it's just needing that your start URL on that.

Ben: So those kinds of things just smoothing that over, assume-role-lib is another one of those things where you can't in some languages, the JavaScript SDK V2 included you can programmatically assume a role. So you define a config in code that says, "Based on the credentials I already have, I want to assume this role." And then that's just a normal config that you can use with the SDK. The Python SDK but a three does not do that, you can always do it through your config file. You can define a profile and your config file that has a role to assume based on some other credentials. And so I wrote a thing that allows you to do that in Python. So it's a lot of things like that.

Adam: It's filling in gaps.

Ben: Yes. I'm hoping there's an issue open on Boto that's programmatic role assumptions should just be a part of the CLI. A lot of the things that aws-sso-util does should just be part of the AWS CLI. I don't want to have to write these.

Adam: Yeah. Eventually you don't have to maintain them. You can just delete those repos, and that's a little safe.

Ben: Yeah, that's a Serverless way. It's, oh, I can throw this card away. Fantastic. I am not attached to any of it.

Adam: Yeah. I love it. And we did have a question here. So Dax Rad, who's been on the show. He said, cost of ownership is talked about how much custom code you write versus using AWS native concepts. Why is an AWS native concept different than importing a library example of validation library versus using scheme validation and API gateway?

Ben: Yes, because that validation library has updates and you are responsible for pulling in figuring out when those updates exist, rebuilding your code, making sure it all works. Whereas schema validation and API gateway, which currently is only present in rest APIs and does not work properly with the Lambda proxy integration in rest APIs and is not present in the API. So there's ways to go before a schema validation is all there on the API gateway front.

Ben: The contract is there, you've defined the schema, making sure that it actually works and doesn't have any vulnerabilities, everything, all of that is AWS's responsibility and completely transparent to you. This is also why I'm sort of split on or sort of torn about compiled languages for Lambda. So I like Go, it's a great language but I am responsible for upgrading that binary that runs in Lambda because it's statically compiled with Python.

Ben: I'm not responsible for patch versions of the Python runtime, that's AWS's responsibility. So using a compiled language pushes more of that responsibility onto me. How much that matters, especially with a language like Go, unclear, but it is a thing that we don't talk enough about. So it's that thing of, if you can not make changes to it and it gets better, that's a more Serverless way of doing things.

Adam: Sure. Yep. Okay. So thank you, Dax, for the question. I do want to get to a couple of games here. We've got two of them today. We've got the Innovators for Amazon and then there's the IAM actions real or fake. So we're going to play both of them back to back. So there's a minute on the clock, Ben.

Ben: Well, and so the baseline that people should understand is that the theory behind it is services branded Amazon should be usable on their own and services that have AWS they're supposed to be sort of supporting services that only makes sense in the AWS context. It's not used consistently and literally no customer cares. Marketing spends a lot of time on this and enforcing it and making sure it's all correct. Literally, no customer has ever looked at it and it's up on the difference, let alone made some choice based on it. But that being said I see it enough that I can probably, I think I have a good knowledge.

Adam: Yeah. There's some recognition there. Linda Viva had on the show, she had researched and learned that. And she told me before the show, I didn't know that, I don't think it helped her just thinking about, is this a standalone service? Okay, here we go. So a minute on the clock. You're ready, Ben?

Ben: Yep.

Adam: All right. WAF.

Ben: AWS.

Adam: Certificate Manager.

Ben: I think it's Amazon.

Adam: Monitor on.

Ben: AWS.

Adam: Proton.

Ben: Proton is AWS for sure.

Adam: Service Catalog.

Ben: Service catalog is AWS.

Adam: Location Service.

Ben: Amazon.

Adam: RoboMaker.

Ben: It's AWS.

Adam: Pinpoint.

Ben: Okay. Go ahead.

Adam: Pinpoint.

Ben: Pinpoint is Amazon.

Adam: Honeycode.

Ben: That's the hardest one. I think it's, under the logic it's Amazon, but I'm going to go AWS.

Adam: Oh, shoot.

Ben: Oh, no, that's fine.

Adam: Outflow.

Ben: Outflow is Amazon.

Adam: You got it.

Ben: Yup. Monitor on I had no idea.

Adam: Yeah, I honestly didn't know that was a service. So I learn a new service every time I play this game.

Ben: You should insert a non-existent service into this.

Adam: Oh, there you go. Yeah, just see if anyone catches it. Okay. And then the second game. So IAM real or fake. I'm going to read off IAM actions that are either made up or real. And you're going to have to tell me if they're real or fake.

Ben: The hardest one of this is going to be anything in systems manager, because there's the number of AWS services and the number of API calls and systems manager are they compete for? Which one is ahead at any given moment?

Adam: There's a lot. I don't think I have any here on systems manager for you, so you lucked out. Okay. So we're going to do a minute on the clock for this one as well. You're ready, Ben?

Ben: Yes.

Adam: Okay. Cloud nine, validate, delegate.

Ben: Fake.

Adam: CodeCommit, get-blob.

Ben: Is real.

Adam: Device farm, stop upload.

Ben: That sounds fake.

Adam: KMS, delete imported key material.

Ben: Delete. I know important key material is a part of KMS but I'm going to say real.

Adam: OpsWorks described stack parameters.

Ben: That sounds real.

Adam: Organizations delete account.

Ben: Oh, that should be there, but it's not there.

Adam: RDS create DB instance.

Ben: Real.

Adam: S3 put metrics configuration.

Ben: Real.

Adam: SMS publish.

Ben: Yeah, delete account is of course the API that we all want that does not exist.

Adam: It's not there.

Ben: And so there's no IAM permission that could be associated with it.

Adam: Thanks to all the guests to join the live show. We do this every week. We're doing the show live on Twitter spaces. So joining the Twitter space, if you'd like to listen in and ask questions for the guests to answer on the show, it's a good time. Also, I'd love to see you on the discord channel. You can join me and the guests from the show, we're talking about how to improve the show, we're talking about AWS.

Adam: Yeah, it's just a great community that's sort of forming around AWS FM. Also, if you listen to the show on a podcast platform like Apple Podcasts or Spotify, I'd love it if you'd leave a rating and even a review. That's great feedback for me, it makes me feel really good to see them. So I'd appreciate if you could leave a review. Ben, it's been so good. I admired you from afar for a really long time. So thank you so much for taking the time.

Ben: I appreciate it. Thanks for having me, it's been great.