Boris Tane: Building SaaS on AWS and Serverless Observability

About Boris

Boris is the founder of Baselime, an observability solution for serverless architectures. Boris started his career as an aerospace engineer before joining the startup eco-system in London, UK. He is devoted to helping teams to adopt serverless without living in constant pain, not knowing what exactly their code does in production.

Boris hangs out on Twitter where he'd love to chat (DMs are open)!

Transcript

Adam: Hey everyone. Welcome to AWS FM, the live audio show with guests from around AWS community. I'm your host, Adam Elmore. And today I'm joined by Boris Tane. Hi Boris.

Boris: Hi. How are you doing?

Adam: I'm doing well. How are you?

Boris: I'm very good. Thank you. It's a rainy day in London, today so, I mean, typical.

Adam: That's normal in London. It's actually really nice in the Midwestern United States today. We've had kind of some heat the last couple of days that is delaying fall just a little bit longer. So Boris, I followed you for a while on Twitter. You're someone that sort of bridges two worlds for me. So I follow a lot of the kind of indie, like SaaS building maker types. That's kind of a whole community on Twitter. And then I also follow obviously a lot of AWS folks, you're sort of in the intersection of that Venn diagram, you're kind of living, bridging that gap for me. And I'd like to just start kind of with your story, if you could just tell me kind of a bit about your background, getting into building things, getting into AWS, just kind of start there.

Boris: Yeah. So yeah, I actually, I'm an aerospace engineer, so I studied physics and aerospace engineering. And I started my career in space, spent a few years as an aerodynamics consultant, but I've been writing code since I'm a kid because I wanted to make video games back in the Crash Bandicoot times. I wanted to make people believe [crosstalk 00:01:32] what I wanted. And yeah, after a few years in aero, I'd been building internal tools. I actually started in Azure interestingly and then joined a startup, which was a 100% on AWS. And that's when I started digging deep into the AWS ecosystem. Typical architecture when I started containers on EC2 and then ECS. And one day I saw this article about, you can do everything without having to spin up a single server. What world is this?

Boris: And yeah, that's how I then started playing around with serverless Lambda. That was when 2018 sometime there, I started playing around with Lambdas and slowly, getting more into the ecosystem and then, oh, there's this thing called SQS. You don't need to have your broker, interesting. Oh, there's this thing called SNS. Oh, and then S3. Nice, nice. Then slowly over the past three to four years, I've just been in the ecosystem and yeah, I joined the Indie Hacker Movement. I've been building little apps on the side, but during the pandemic, that's when at the very beginning, that's when I discovered the Indie Hacker community.

Boris: I joined and it was fantastic. It's one of the most welcoming communities on the web and yeah, that's when I started being a little bit more serious about the apps that I was building and yeah, I think it's one thing I'm on the Indie Hacker community that if you want to do something when you're a small team or alone, go for a SaaS, don't go for a social network. That's when I started digging more into the SaaS world and yeah, I've been building a couple of SaaS since then. And I'm building something, which is a little bigger than what I've been building before.

Adam: Yeah. And I want to talk about that. So Baselime, you just announced that you're working on Baselime, which is targeted at folks building things on AWS, it sort of solves the observability problem. Could you talk a bit just about Baselime, kind of what you're building and maybe more broadly about observability and serverless?

Boris: Yes. So doing serverless for this number of years now on AWS, it's a problem that you actually don't know what's going on in your systems when you're doing microservices architecture, normal, typical quotation microservices architectures with Docker and search. You still have that problem but when you move to serverless, it goes from a 10 X problem to a hundred X problem. It just explodes because you now have this many moving parts into your architecture. And I had faced it in my day-to-day work within companies, but within companies, we always had something. It always had a Datadog, which was kind of helping. We always had the [crosstalk 00:04:47], which was kind of helping. We could always dig into the CloudWatch logs, which were kind of helping. But building my last project, I mean, I'm still running it bookmark, the bookmarks' manager thing.

Boris: I had two weeks when people couldn't sign up on my app and I didn't know. I simply didn't know. It took someone to message me saying, "Hey, I can't sign up on your app." And then I was like [crosstalk 00:05:18]. And then after, I don't know, one hour or so of looking at where the problem was coming from, it was one change in one Lambda, which impacted another one, but it wasn't throwing an error. So nothing was shouting at me, "Hey, there is an error going on here." And that's when I thought, "Someone should do something about it."

Adam: And that someone should be you.

Boris: I thought that looking at solutions that are out there, there are a few solutions for monitoring of serverless architectures. You have the Lumigo, the Dashbird, all of those players. And also very recently, I think you interviewed them. What's the name of the app?

Adam: Cloudash?

Boris: Cloudash, exactly. That's a bit of application that is basically helping you with your CloudWatch logs basically. And there are also players doing one step ahead of that. Somebody like Honeycomb.io, for example, or Lightstep, they are literally going to the, "Okay, we want you to actually be able to dig into the data into your systems." They're not doing for it service specifically, they're doing it for general architectures. But the serverless world is a little different from the Docker world, the container world where timeouts like big problem. You don't necessarily have such a big problem with timeouts when you're dealing with a typical container architecture. And after looking at all these players in the market, I thought, "Okay, there is a place where you're actually doing that observability, that real observability for serverless." And it's quite hard for any team to adopt it right now. And that's when I decided, "You know what? I'm going to do it."

Adam: Yeah. Yeah. Well, I told the Cloudash founders, I mean, I think anybody working right now on this problem, just trying to improve sort of a developer's life on a serverless application, I think, it's time well spent because there is still a lot to be done in this space to kind of improve on the developer experience. So it's really exciting to me every time I see anybody kind of starting to tackle sort of dev tools that space and I know venture capitalists are super excited about that space right now. So that's good if you go that route. So I guess you've kind of got a track record of building SaaS applications on AWS. Could you speak a little bit just in terms of advantages that you see building on AWS versus a lot of the Indie Hacker crowd is sort of using much higher level services? They're hosting on Render or Supabase or whatever. How do you view kind of building SaaS applications on AWS and sort of strengths, weaknesses, that sort of thing?

Boris: Yeah. If you are doing indie hacking, which means you are probably alone trying to get to a few thousands MRR, I'm running my app for free, right? It's literally free. I'm not hitting any of the threshold to then start paying. Whereas if I was on DigitalOcean or any of the higher-level platform as a service providers, I will be at least at $100 a month right now. Right? So it completely free. And also it allows, at least for me, it allows me to build things that are much, much more resilient. I don't have to worry about is DynamoDB going to scale properly, right? That almost a given. I don't have to worry about, "Okay, I have to worry a little bit about is my Lambda going to scale appropriately?" But I don't have that many users. So technically speaking, I don't have to worry about that.

Boris: So yeah, definitely if you have the skillset because I think there is also that question of, should I do it when I don't know what I'm getting into? Serverless on AWS has a lot of captchas. You have to read [crosstalk 00:09:35] you have to... There are things that I discovered just reading tweets, right? It's not clear in the documentation that this is how things work, but when you really look at documentation, you understand you tweet by someone who has done it before, you understand that if you want to get the API gateway logs in CloudWatch, you need to do this, this, this, and that.

Boris: So if you want something out real quick that is going to provide value to your customers and your users, and you don't necessarily have that serverless experience. Yes, it's probably better to do it on a different stack where your original, what you're doing. But if you already have that knowledge, oh, you're keen to learn, right? The beauty of the AWS community and the serverless community, in particular, is that it's also welcoming is just as welcoming as the Indie Hacker community. Everybody is ready to answer questions that you might have, everybody is ready to help. And there is already a lot of content out there to help you build an application if you're trying to do it solo. Right? So, definitely do it. I encourage everyone. And my personal bias is that serverless is going to be the future of software infrastructure. So yes, you can try to spin up a [inaudible 00:10:59] cluster, blah, blah, blah, blah, blah. Why do that when you can run a command and have a few functions running in 10 seconds? So yeah.

Adam: Yeah, yeah. I see the arguments, a lot of folks in the Indie Hacker community seem to resonate with the thought that you should stick with what you know, don't sort of take on new technologies when you're also trying to build a new business. I think I understand those arguments, but at the same time, I think there's so many technologies within AWS that are underutilized. And I think they are a great fit in that scenario where you're scaling from zero, you're sort of going from, I have no customers to I'm trying to build up a customer base, but all the while you're experimenting, you're not really incurring costs. You have all these tools at your disposal and services like DynamoDB. I mean, I think that I really wish the whole broader kind of Indie Hacker, web dev community could understand some of the strengths and trade-offs with DynamoDB, I think because it's kind of within AWS land and they don't necessarily have experience with AWS. They don't really get that exposure and to understand what it's capable of. I think it's sort of a cornerstone of that serverless advantage.

Boris: Yeah, definitely. And it gives you the, at least me, gives me the possibility to try things out and shift without having to rewrite a lot of things, right? If you are changing the access buttons to one thing, okay, everybody says you need to design your access patterns for your DynamoDB table beforehand. That's true. That's what you should be doing. But if you need to change it, but you have what? Less than a thousand records in the database, it's not a big deal. You can rewrite it. Right? So, whereas if you have a complete ORM on the MySQL Database, now you have to run this migration. You can lose data.

Boris: On DynamoDB, you don't have to worry about backups, right? It's one line of CloudFormation and it's done. Whereas everywhere else, you need to write custom scripts to read the database every evening and then write it to S3 or whatever cloud cold storage that you're using. I think, yes, people want to experiment and the software world is moving really fast. Of course, you can build a profitable application on a single PHP file, but do you really want to be doing that? Probably not.

Adam: Yeah. So I have a few more questions I want to go down the kind of AWS SaaS route, but you said something earlier about the Indie Hacker community and that's something I'd love to just pick your brain a bit. I sort of only have the capacity, I guess, for one community on the internet. So I've sort of resonated with Twitter and with where people are building in public on Twitter, but I've never been able to, I think I'm signed up on Indie Hacker, the forums and everything. I've just never had sort of the ability to juggle those two communities. Is it something you've been able to do? Are you active sort of both places at the same time?

Boris: I used to for the past four or five months, I think I've been in the same position where you are where there's just not enough bandwidth to be able to be on Twitter and then be on Reddit and then be on the Indie Hackers forum. And then answering questions in Stack Overflow is just too much, right? I think most recently I've "focused" on the mall, it's where I hang out. I already have a few friends on Twitter, already interacting with a lot of people there on a daily basis. So I just stick to Twitter now.

Adam: Yeah. When you interact with folks in the more Indie Hacker circles, people that are just makers, do you lobby for... Are you ever trying to convince them to kind of come over and test the waters with AWS if they don't have that exposure?

Boris: Yes, definitely. It's a difficult battle because as I said, there are a lot of very successful Indie Hackers who are running on PHP, on Rails in a single big server and it's definitely valid, right? If you want to build a business fast and you already have the skills on Rails, probably you want to stick to that. But if you're curious and you want to explore, definitely join the servers train because the train has already left the station and it's going to the moon. So definitely. And I like it when in Indie Hacker DMs me saying, "Hey, I tried this Lambda that they used is quite good. What do you think about this? What do you think about that?" And I'm all happy to answer those questions.

Adam: Yeah. So do you have any sort of tips or anything that you've run across when sort of interacting with these folks, things that you would tell people who are just getting started in AWS maybe, but they have that background of building products? Anything you've learned in your experience that you would sort of distill and handoff?

Boris: I think if you are coming from a traditional server world and you want to adopt serverless, start by having your cron jobs on the Lambda function. I think that's the easiest way to get in. Usually, you don't have to move a lot of stuff. And then that gives you straightaway the power of having a Lambda, right? You don't have to go into a computer and manually write the cron jobs. No, it's literally a line. If you're using the serverless framework is one line, right? Then it happens every day or whatever interval that you set up. And I think, at least for me, that was the hook because when I started, what I did was what most people do, which is taking an extra server, having a wrapper around it, and put it into a Lambda. And I was like, "Yeah, this is good, but it doesn't really do anything different." Right? Okay. I can deploy relatively easily. I don't have to pay when other don't have any requests, but startup time is five seconds.

Adam: Not ideally on demand.

Boris: Exactly. But when I started putting cron jobs, I think the first thing that I did, which was interesting for me was a bot. It tweet a book telling me if the London on the ground was on time or too late or whatever for each [crosstalk 00:17:51]. And that was the cron job, calling the API for transport for London every hour and tweeting if there was a problem. And then doing that into a typical server architecture would have taken me at the time, at least if all weekend. And that was three hours and I was done. So that's when I realized, "Okay, this thing is powerful and I need to dig deeper into it." And here we are today, a few years later, I'm building a solution to help people adopting it.

Adam: Yeah, because I mean, that's one of the first things I think if someone's really trying to start a small bootstrapped business and they want to find issues in their serverless application before their customers, that's non-trivial. So I really think the Baselime to the world, they need to exist. And I wish you luck kind of pushing that forward because I know I've sort of built things, small things, and I'm generally leaning towards serverless, just having that insight. And I've worked on teams where it's not serverless even with traditional applications, we're finding out from customers when there's issues. That's sort of par I guess when it comes to observability these days and it's really not great.

Adam: We want to find those issues before our customers do. Yeah. So I guess, have you heard, this is something I wanted to ask you about just knowing you're in the space and there aren't that many people, I feel like, building SaaS on AWS, at least publicly. Are you familiar with... Well, there's the Well-Architected Lens, are you familiar with the SaaS? I'm sorry, the Well-Architected tool, but the SaaS Lens they've recently in the last year, I feel like they've launched this sort of whole learning platform on AWS, around building SaaS. Is that something you're familiar with? You've had any exposure to that?

Boris: Actually, no. I'm learning and that's great. There's always something new to learn on the AWS world. On the Well-Architected, I don't know, it's a service on AWS from memory. I think at one of the companies I worked at, we played around with it a little bit and it gave us just so many areas that we said this is not-

Adam: It's like noise. Yeah.

Boris: This is not the right thing for us right now.

Adam: Yeah. No, yeah.

Boris: And I think that's one of the problems of such tools where they tell us, I mean, obviously, your architecture is going to be crap, right? If you are [inaudible 00:20:27] operating with real users, you're not going to have a perfect architecture. There are going to be areas, which are literal garbage that you've been carrying around for a few years now. And when you click on a Well-Architected thing, which analyzes your entire stack and gives you this four pages report saying that you need to rewrite everything, the natural behavior is to just be like, "Not for today, we are going to see this later."

Boris: And what has happened between now and that later is you're going to be at worse. So I think those tools, at least in my opinion, should give piecemeal information that are actionable right now, rather than this huge report, which is not helping anyone. If you're adopting those tools early in your journey. So when you start every week or every month, you run it to make sure that you're not deviating from industry standards. Yes, you are going to be on track. But if you do that after four years of operation, yeah, it's not really going to help you.

Adam: Yeah. Man, I'm so glad I asked you, that's such a better response than... I forgot that you're a practitioner. You're building things and that's the real world answer I needed, which is sometimes it's easy to get kind of absorbed in all the best practices or whatever, everything AWS puts out there. And that was just a service that I've wondered, who's using this in the real world? Because in my mind, the SaaS Lens, I would think it's sort of for Greenfield people building out stuff for the first time, but now that you kind of responded, I hear it. It just sounds like a bit much for that sort of bootstrapped mindset where you're not so concerned about whether your architecture is checking the literal hundreds of boxes you're concerned with, are your customers happy? Are they buying your thing? Yeah. Refreshing to hear that. So I'm glad I asked.

Boris: Yeah.

Adam: Yeah. So you, you kind of said earlier and I wanted to hit on this, serverless really does... It sort of opens up whole new types of applications you can build. I know that's a feeling I've had sort of having experience in AWS and dabbling in the maker kind of mentality like building things, putting them out there quickly. It feels like you have tools at your disposal when you have these skills with serverless and with cloud technologies that you can kind of go out and think of applications that maybe other people would be really hard to build on those other platforms. So I don't know if I'm asking a question so much as just kind of reiterating what you're saying, which is some of the pro... There's trade-offs with everything.

Adam: And I think with serverless, it's sort of if you take the time to learn these technologies, to learn the sort of... It's not that many... Everyone talks about how many services on AWS, 200 plus services. But with serverless, you're talking about a dozen that you really need to kind of interact with regularly. If you take the time to develop those skills, it sort of opens up all these possibilities that you can as a sort of business person, someone sort of entrepreneurial, trying to build things. You have this set of tools and these new types of businesses you can build.

Boris: Yeah, definitely. It makes things, at least in my opinion, way easier, right? Little things like sending emails, right? That's one thing that every single application does. You don't want that to be synchronous. You don't want that to be every time someone clicks on, sign up on your app, the email sending process is run before they get the response, right? You want that to happen later. If you're building a monolithic application, I think that's going to be relatively hard because you have to build all these communications versus internally into your app. Whereas if you're in the serverless world, you just write the user into a DynamoDB table, have a stream that is going to send an event that goes into EventBridge. EventBridge sends it to another Lambda. The Lambda sends the email. It sounds complicated, but you actually don't write any code between those steps. It's CloudFormation like...

Adam: You're just gluing the bits together.

Boris: Exactly. And now your user gets the response in 100 milliseconds instead of two seconds if you were to call your email service manually. I mean, not manually like in the process. And now, if you want to change your email service, it's a complete, separate thing that is leaving in a different repo event, right? You don't have to worry that if I change this, is that going to impact anything else? Itself enclosed within the Lambda. Okay. If you have more complex flows and you put them into step functions. Yes, if you change one, maybe it's going to change the rest. I mean, do have an impact on the [inaudible 00:25:36] But that's the knowledge that I think we need to have in order to build those applications way faster and at a lower cost.

Adam: Yeah. Yeah. I feel like serverless is, I'm sure I'm not the first person to say this or think this. It's like the no-code movement. It feels very parallel like they're on parallel tracks. It's really about writing less stuff, less proprietary stuff that you have to maintain. No-code may have a different approach, but it feels very much these two communities have something in common.

Boris: I never thought of it that way. But now that you say it, am I a no-code developer?

Adam: Are you?

Adam: I mean, that's always the goal now. I feel like building stuff out with serverless now, it used to be like, "Don't spin up an EC2 Instance." And I heard from the Cloudash founders, they work it steady and they sort of limit, they just shut off EC2 with SCPs. So the whole account level, they're not allowed to launch the EC2 Instances, which I think is awesome. But that used to be the goal. Now the goal sort of is how can I build this whole thing without writing a single custom Lambda? Can I do this all with just bolting together services that have direct integrations? And every time AWS launches some new feature that integrates two services that you kind of knew eventually it would be integrated. It's just that feeling of, "Oh, I get to go back through three repos and just cut out all this custom code."

Adam: We're just constantly pulling back on what we have to maintain, which is very empowering. Especially from a bootstrap standpoint where you're really critical with the time that you spend on anything. Yeah. I think it's great to have folks like you in the community that are sort of bridging these two worlds because AWS people aren't necessarily into bootstrapping small businesses or sort of these indie hacked startups. And Indie Hackers, clearly we've talked about aren't necessarily interested in AWS or lower-level things like that. But I think there's a harmony there that's worth exploring. I'd like to get a hot take from you, Boris. It can be AWS-related, it could be just technology more broadly, something that's really it's good. It's juicy. Somebody who's going to be very upset when you say it [crosstalk 00:27:54] angry and they're going to flame us on Twitter.

Boris: Let me see who is in the audience before, I can see Vic. Okay. Well, hot take, this is something I've wanted to tweet for a few months now. And I've been [crosstalk 00:28:12] restraining myself from it. Your unit tests are useless. If you're writing Lambdas on either AWS, the vast majority of the unit tests that you write are useless because the test, this bit of code in isolation and that bit of code never, ever exist in isolation. The whole point is that it exists within your infrastructure. So if you have a function, most of the time, the code that we write is not complex. We don't write like I was an aerospace engineer before, the code writing then was natively more complex than what we do now. It's a couple of e-statements, maybe a loop there. And then you have to await the promise to do something.

Boris: [crosstalk 00:29:07] It's not rocket science. So most of the time our unit tests are mocking S3, mocking SQS, mocking SNS. So at the end of the day, the [Burg 00:29:21] is going to come from your Lambda, not having the permission to write to SQS, not the fact that they didn't properly write the function to do it. So what you should be trying to test is that within the ecosystem that your service is, or your Lambda lives in, it works properly. Not that in isolation it works properly.

Boris: So yeah, that's why I'm saying in this new world, your unit tests, the vast majority of them are useless. You don't waste time writing it tests to test with four marks, to test that you wrote in the database [inaudible 00:29:59] SQS. I mean, I'm sure that someone was screwed up at some point, but that's way more unlikely than having the wrong permissions than having the wrong queue passed in cloud formation or whatever you're using to glue together your services. It's way more likely that your... What's the name? The polling time in your SQS queue is too high or too low. That's way more likely to happen than you-

Adam: Like a visibility time-out issue.

Boris: Exactly. It's way more likely to happen than someone screwing up a promise. So, write unit tests when they are necessary, but trying to get 100% coverage full of unit tests, it's a waste of time.

Adam: Yeah. So I must have the same hot take because I haven't written unit test for years. I'm probably just lazy. I have a library that I don't want you to test there either. I should probably do that. On the serverless side, I totally get it. The only time I find myself and I hate mocking out database resources feels awful. But the only time I do find myself writing a unit test is I just need to run this code to make sure I'm doing the right thing. I know what I'm working through here, but that's pretty rare. So you would say integration tests test the thing in AWS to make sure you're getting ahead of those sort of permission errors and things like that.

Boris: Yep. Yep. [crosstalk 00:31:35]

Adam: Can I get you to just say like... No, you know what? I'll just edit out the unit. So you're just saying tests are bad. That's going to be [inaudible 00:31:41] Boris saying, "Test's no good. Automated testings for losers." I was really honestly hoping you would say something about web3 because my Twitter timeline, that's all it is. It's either for or against web3, it just took over the last 48 hours.

Boris: Yeah. I can't really have a hot take on web3 because I don't know much about it. But from what I understand, people are trying to build a decentralized network on a decentralized network to replace a decentralized network. And [crosstalk 00:32:24] problem somewhere there.

Adam: Yeah, I mean, I really can't say anything about web3 without sounding like a super old person.

Boris: Exactly. Yeah. Maybe it just means that we don't have the necessary knowledge to actually have an opinion. And there are just so many businesses to build in web tools still that, "Okay, if you want to be at the forefront, go web3, but you still have a lot to do in web2." Right? I don't think I know anyone in the real world, like not on Twitter who knows anything about web3. Right?

Adam: Yeah. It's a very Twitter [crosstalk 00:33:04]. Yeah. I mean, for me, that's where I get... I'm sure that everyone else is getting it. If they're in the space, it's from Discord and whatever, like dApps or whatever they're making where they're talking to each other. But for me, yeah, it's very contained to Twitter, which is nice because if I just don't want to read the words, web3, any given day, I just don't have open Twitter. But yeah. That's so much of my life, I think living... So you live in London, you grew up in London like you've always lived in London?

Boris: No, actually I'm from Cameroon in Central Africa. I grew up there till I was 20. I came here for my studies and yeah, I've been here for the past, what? Eight, nine years now?

Adam: Yeah. Okay. And is it you're in London or do you live like... I don't know anything about London. Are you outside of... Are you in the city, I guess?

Boris: No, I don't live in the city. That is too expensive. I live in what you would call the suburbs. I don't think we call it the suburbs here. Yeah. Yeah, it's not London, London, but it's train distance from London.

Adam: So I live in a very rural sort of farmland area in the United States in the Midwest. I mean the city that Springfield that I'm closest to is a hundred thousand people, but basically nobody... If I said web3 on the street, it just doesn't even register as a concept they've heard of. But it's interesting stuff. It's hard not to be sort of to notice it and to want to kind of learn more.

Boris: Yeah. I think there is something there just that I don't know anything about it. And just from looking at it from the outside, it feels like just the hype machine, every six months VC Twitter is going to come up with some-

Adam: And I think that's what I'm trying to figure out is where on the sort of hype cycle that's well-documented, where are we with web3 specifically or crypto more broadly? Because I feel like web3 is kind of even like a subset of broader blockchain and crypto, it kind of means something. Web3 is its own concept. But I felt like the hype cycle was sort of years ago hitting peak and then we've kind of come down, but now I don't know, this feels... My Twitter feed, it's all that's in my Twitter feed.

Adam: So I feel like if I'm on Twitter, talking on a space, it doesn't feel out of place to be just talking about web3, because [crosstalk 00:35:43] you're talking about on Twitter. My wife's super into Enneagram. I don't know if you're familiar with personality tests, but I'm an Enneagram type nine. So I like everything to be at peace. I like everybody to be good. And my Twitter is just full of people not good right now. They're all [crosstalk 00:36:00] about each other. And I just want them to all be at peace, no more sort of uncomfortable conflict, please.

Boris: Well, I think it's going to get better when more people do their research, right? When I have a few hours, I will do a bit of research and understand exactly what this is about. If it's something that is super interesting, I'm definitely going to encourage people in it. It just happens that right now it's as there are too many businesses to build in web tools, let's do that. And then let's leave, at least for me, I will leave the pioneers in web3. But one thing else, when you talked about hot takes, I just remembered one exactly your staging environment is useless.

Adam: Oh, thank you. I really don't want to have one.

Boris: [inaudible 00:37:01] as well, because it's simply is not a representation of your production environment. If you're doing serverless, maybe you can get close. But if you're at a company, which is at least five years old with a lot of code that was written by a lot of people who already left the company and they were not doing infrastructure as code from day one. Yeah, your staging environment or dev environment, whatever you want to call it is never going to look like exactly your production environment. And organizations, at least the ones that I've spoken with, waste so much time and resources trying to keep those two somewhat similar.

Boris: And it gives them the false sense of security that if it's working, staging is going to be working in prod, when that is not true, because they are not the same environment. So yeah, if you're spending a lot of time keeping a staging environment close to prod, I think you should ditch it and invest in having your logs in production in order. Right? Being able to know what's going on in production rather than trying to have a separate environment, which is kind of the same, but not really where you do your testing, it's a waste of time. That's my opinion.

Adam: Do you do any sort of testing in production? That's something I really want to know more about so I'm going to ask every guest I have until I get an answer.

Boris: So testing in production, what do we call testing in production? That's the main question for me, because if you're talking-

Adam: Just posting around in the UI like if you're just logging in and just testing it, I guess that's testing in production.

Boris: Yeah. If that's what we're calling testing in production, definitely. And everyone should be doing that. Everyone, that's why... Because I wasn't doing it for two weeks, nobody was able to sign up on my app and I didn't know. Everyone should have... If you're running a business or you're in an engineering team, at least once a week, someone should do the core workflow in the app from zero and note down every bottleneck. Because every single time you're going to notice that, "Hey, the onboarding is not that good." Or when you click on this button for the first time, it throws an error the second time it works. But if all you're doing is testing in dev or in staging where usually what happens in staging is sales team is sharing staging. Customer support is sharing staging as well. Whatever other, like the CEO is sharing staging as well because he's in those parties, showing it to his VC friends, your staging environment is not real. It's a smokescreen that is hiding the problems that have been in prod.

Adam: And then if no one's testing it, there's no one's ever seeing production, but your customers then yeah, you have no visibility.

Boris: Exactly, exactly. What a lot of people consider testing in prod will be doing the chaos engineering and all those super complicated things. I have-

Adam: Stuff that like Netflix, they've got time for that. They've got the people. You just mess everything up in production.

Boris: Exactly. I've never been in an organization big enough to be doing those kinds of experiments. Stress testing, definitely. I think that's something that when you start to be [paid 00:40:53], you should be doing, just making sure that if for whatever reason, Kylie Jenner tweets your link, you can at least keep running before everything goes to shit. But yeah, I think it's important to at least once a week, just go through the call flow of your app and see where things are going wrong.

Adam: Yeah. That makes sense. I think just talking with you, I can't help but think of the different sort of circles on Twitter. And it's so easy when someone says something like someone I follow because they're big into... Maybe they work at Netflix and they're embedded in that world where things are very different. They say something like you should test in production. I read that and go, "Ah, I guess I should." I respect that person, but not everybody's doing the same things. That's sort of a bold statement. We're not all doing the same things on the internet. But I guess it's easy to follow a lot of different types of people and to hear a lot of different advice and sort of think it just generally applies to the things you're doing day-to-day and that's not necessarily the case.

Boris: Yeah, yeah, yeah. I don't know. That chaos engineering thing, if you're running, if you're alone running an app with 2,000 users. Just building the infrastructure of that is a waste of your time. Right? You should be focusing on things that your customers actually care about. When you get to, I don't know, 10 million customers logging into your app every single day, it becomes more important because now you have so many potential points of failure that okay, they're going to break everything and see what comes out of it.

Adam: Yeah. So I guess the key is just having that filter and knowing that you know your kind of use case or your situation and you can run everything you read through that. And just following the right people, being in the right circles of other indie hackers that are building small bootstrap businesses and thinking like they do, even if you kind of have a different angle on it. You're building on AWS. Yeah. [crosstalk 00:43:14] Go ahead.

Boris: One thing in Lambda, which is also pretty important, is to grow with your audience, right? Now I'm following and interacting with way more AWS and observability people on the internet. And that's simply because I'm not building a bookmarking tool anymore. Right? When I started, it was a bookmarking tool for myself, just to save my bookmarks and be able to retrieve them. If I had 10 users, I was happy. Right? Now, I'm actually going for something which is much more bigger. So it's important that in that journey, first of all, the people that, at least for me, that I started being active on Twitter with, we grow together. Right? We are evolving at the very similar pace. And then you also want to aspire to discovering what people that are at a different stage at doing and are thinking, knowing also that, that doesn't necessarily apply to you today. Right? It's good, at least for me, to know that eventually when I get 10 million users, okay, I will start doing chaos engineering. I don't need to know exactly how it works, but if I understand it conceptually, it's good for now.

Adam: Yeah. Yeah. Good advice. And I think people just getting started in tech, I think that's one of the biggest challenges probably is just knowing... There's a lot of opinionated people and there's a lot of different opinions on Twitter. Being able to sift through that and kind of navigate your way to success, I think that's a big challenge. Well, I've got my hot take. I think anything else [crosstalk 00:45:06].

Adam: I'm totally going to take it out of content. This is going to be cut up and extracted into something awesome. It's going to involve web3, I'm sure. So it's been so good-

Boris: Yeah, go ahead. I was saying if you're writing unit tests in web3 for your staging environment, you are on a million-dollar idea.

Adam: I got three. That's right. I got three hot takes from you. You just melted them all together for me in one little soundbite. That's awesome. Well, it's been so good, Boris. I think, yeah. You've put up with my just kind of trying to figure out what I'm doing here, being on one of the very first shows. Greatly appreciate it. You're somebody I've admired for a long time on Twitter. It's great to finally kind of talk face to face. Thanks everyone on the Twitter space that joined. One thing I've found in starting an audio show, the hardest thing in the world is to get off of it. I don't know. I guess that's everyone has a recorded a thing that they just say every single time.

Adam: I don't have that yet. So they're going to listen to these is going to just kind of listen to me meander at the end and be like, "Okay, I'm going to stop it now [crosstalk 00:46:13] button because I don't really know what to do." So you guys can all just leave now. Save me the trouble of trying to end this space. Thank you again, Boris and thanks everyone for joining.

Boris: Thank you for having me. Awesome. Thank you very much. Thanks everyone listening. Yeah. Catch me on Twitter.

Adam: Oh, yeah, yeah. Catch Boris on Twitter. Check out Baselime, sign up for the newsletter. Yeah. We'll make sure all that stuff's in the show notes as well.

Boris: Awesome.

Adam: All right. Thanks, Boris.

Boris: Cool. Bye.