281. Developers Are Reinventing the Train - The Dunning-Kruger Effect

You have probably heard of the Dunning-Kruger effect, but do you know what it is and why it is important to understand as a developer? And how does that relate to the reinvention of trains? These are the questions we will answer in today's episode of Dev Questions.

Website: https://www.DevForge.com/ 
Ask Your Question: https://suggestions.iamtimcorey.com/
Sign Up to Get More Great Developer Content in Your Inbox: https://signup.iamtimcorey.com/

Transcript

Copy Transcript Copied

SPEAKER_01 0:00

Software developers keep reinventing the train. And the reason why comes down to the Dunning Kruger effect. In today's episode of Dev Questions, we're going to talk about how to avoid reinventing the train, how to identify when you're falling under the negative implications of a Dunning Kruger effect, and how to use this knowledge to further your career.

SPEAKER_00 0:30

Welcome to the Dev Questions podcast with Tim Corey.

SPEAKER_01 0:33

Software development is more than just writing code. So let's talk about the rest of it. Specifically, let's talk about the Dunning Kruger effect and how it impacts software developers. Now, if you don't know about the Dying Kruger effect, basically the conclusion is that when you don't know anything or much about a topic, then you overestimate how much you know. And once you learn more about it, you actually find out that you know less than you thought you did, and it actually decreases your confidence in how much you know to the point where sometimes even experts will undervalue how much they know or under-report that because they feel like there's so much more to learn. So that's the kind of the nutshell of it. But let's kind of give an illustration of it. There's a couple of different illustrations I've seen recently that have sparked good conversations. Now, the underlying issue is somewhat of a Dunning-Kruger effect in both of these cases, but it's also the conversations around it. So the first illustration is back in 2018-2019, the boring company that's part of the Elon Musk empire, um, decided to create an underground loop in Vegas. And the idea was that they would create this underground subway system, but it's not a subway. And what would happen is it would take people in pods of 16 people or so from like the airport to downtown Vegas, which is about, I believe, 10 miles. And it would do so at 150 miles an hour, making it a rapid transition from the airport into downtown Vegas and back. Cool idea, uh cool concept. And never really came about. I mean, we now have cars, physical Tesla vehicles, driving 30 miles an hour, um, driving people between points. Um, so it's not really the same thing. But that started raising some ideas and some people talking about what are some ways to make this more efficient. Because 16 people at a time, that's not very much. And when you're talking about an airport, maybe a plane lands. Well, a plane is going to hold 150, 200 people who probably all want to go downtown at the same time. So now you're talking about you need, what, 10 cars just to handle one plane load and planes landing all the time? So that need you need to have more cars, right? Or because they're going so fast, it's it's fine. But when they're going so fast, you have to have space between them so you don't run into each other, right? And you have to have ramp up time and slow down time. And you know, each pod has to have some sort of like motor or engine or something to propel it. So now you've got to have each individual pod that you're creating to have all this stuff. And you have as backup, you say, well, how do we make this better? How do we and this is where the the um idea of people kind of talking through this comes in where they start to say, well, what if you had pods that were bigger, maybe 32 people instead of 16? You're like, well, yeah, but that kind of is kind of weird and it may not work around turns or something like that. So what if we just had two pods that we connect together? And then you wouldn't have to have the motor in each pod, you could have just the motor in one, which makes a second pod cheaper. And what if you could put multiple of those together? That way the speed up and slow down time is for all of them at once, and then you can handle a bigger influx of people, and before you know it, what you invented is the train, right? Um in Europe, there was something similar where they were looking at the idea of ships coming into port. And the problem is port cities get very clogged up with all these containers being offloaded and then transported throughout Europe. And European roads are not like American roads because they're a little more uh smaller, a little more uh compact. And so it's harder to get these containers all over Europe. And so the idea was instead of having one port where all the containers coming out gets all clogged, what if you came into one port, but then you had a system that would send certain containers to subports, inland ports, that you could then distribute from there. It makes it closer to the target and it allows you to reduce the congestion at those major ports. And so that was the idea behind a new way of doing things. And the the thought was: well, we want to put those containers into a system that's automated. We want to put them on this like car or container for the container that would allow them to move very quickly through a tube. And in fact, they're going to remove most of the air from the tube, make it almost a vacuum so it moves even faster. The problem is to get these containers on these cars up to or down to pressure so they can get into the system, they had to have like this airlock system. And that's going to cause a clog because we're doing one container at a time and a ship is unloading thousands of containers. That takes a long time. So what if we connected multiple pods together? And before you know it, you've invented the train. And so that's the kind of thought process that goes into a lot of these new inventions, new ideas, and you know, futuristic capsules and low to no pressure vacuum tubes that move things very quickly. You start kind of inventing all these things, and even the thought process with people, we start talking through how can you do this better and be more efficient, more futuristic. And the end result is the lowly train moves more people and more goods faster, even though it goes very slow sometimes, than all these fancy ways of doing things. So why is that? It's because you didn't understand the true problem. You had an outsider's perspective, you came in all this flashy stuff, but you never understood the true problem and the true logistics of it. And so you reinvented the wheel, or you in this case reinvented the train. You reinvented a way of doing things because you didn't have a good perspective. You over, you um thought you were set you were too smart. You thought you were smarter and thought you knew more about the problem than you actually did. And once you get into the weeds of the actual problem, you go, wow, that's that's more complicated than I thought. And the end result is you come to the same conclusion that many, many engineers have come to before you. Now, yes, there are ways of modernizing these things and making these things better, but you have to know what came before and know the decisions that went into it, because that will inform your decisions. Trains aren't flashy solutions. They aren't new and innovative, but they are the near perfect solution for most transportation needs. And when you're thinking about modernizing the way we do things, you should probably start with the proven method and figure if you can iterate and make it a little better. In software development, we have something similar. We create complex, flashy, new ways of doing things. But at the end of the day, we're creating CRUD apps. Practically everything is a CRUD app. And if you don't know, CRUD stands for create, read, update, delete. Basically, we are working with data in every single application we've built. And almost every situation can be solved with a three-tier application that runs on one machine. And that's gonna blow your mind. You're gonna think, no, we can't, we gotta have all this extra stuff. Let's talk about the seven myths that suck people in. Myth number one is you know, scaling is flashy, scaling is necessary. We need to have scale. Make your app more performant instead. That's almost always the solution. In fact, most monoliths that are three-tier monoliths working on one machine can scale to millions of users if you do it right. Now, if you create a poorly performing application because you don't know what you're doing, well, then yes, you're gonna need to have scaling in different ways. But scaling is almost never the first place you should go. The first place you should go is making your app more performant, and you'll find that you almost don't need that, those other scaling things because of the fact that you made your application more performant. Number two, new database types. We need to have new database types and new types of storing data and new ways of storing data. You know what? Your existing SQL Server that you're currently running is probably more than adequate. In fact, again, it can probably handle millions of records per second if you do it right. If you do it wrong, yeah, you're gonna have bad performance. You're gonna need to think about different ways of storing your data. But it almost always comes down to the simple old relational database, probably fine. Which is why things like MySQL rule the world. Because it's simple, it just works. And if you do it right, it scales to practically infinity. And yet people are looking at, well, we need to have new database types and new ways of doing things. I'm not saying AS is bad. You can iterate on the process, but again, don't reinvent the train. Look at the train as a starting point and go from there. When it comes to your database, don't look to replace your database with something new and flashy that you've never really used well and in depth. Instead, look to make your current database better and more efficient, and then look to see if you can add bits and pieces to improve it even further. Number three, uh, the latest JavaScript import framework is the most important one. We need to switch to the latest JavaScript framework. If we don't, well, it's just not good. You know what? You probably only need a server-side app with a little bit of interactivity. You probably don't even need a client-side application. You don't need to have a spa. You probably just need to have a little bit of interactivity on a couple of pages and the Raspberry Server side. You could probably get away with running PHP and a little bit of JavaScript. And that might be mind-blowing. But you know what? Most of the web runs on PHP. Yes, if you're a C sharp developer, maybe PHP is not the right choice for you. But if you're a C sharp developer, that could be MVC. Now, if you don't already have an MVC application, I'd recommend a Blazor server side app, server-side rendered, not the Blazor server as in the client-side connectivity. Just do the Blazor server-side rendered side, which is basically the same as MVC, only more modern. But again, it's just a server-side app with a little bit of interactivity where you need it. That's probably all you need. Number four, AI is going to revolutionize apps. We need to put AI in all of our applications. Probably not. Probably not the way you're using it. In fact, the way that most people use AI, it's just slowing your app down. So, yes, there are valuable things to use AI in. Yes, embed AI is probably a way forward in the future for improving your app, for adding features to your app, but it's not the foundation of your app. It's not something you should be front and center in your application. It's just not. You're probably just slowing your app down. Myth number five, we need a multi-cloud vendor neutral solution. No, you don't. What you need is good tested backups and a disaster plan. You may say, wait, wait, what? Even if you have a multi-cloud, vendor neutral solution where you have it deployed to Amazon and to AWS or and to Azure and maybe also to you know Google Cloud or something else, or even locally, it doesn't matter. You still need to have a disaster plan. You still need to have good backups. You still need to test those backups. So you're not going to get out of any of that work. But at the end of the day, probably what you need is just that good tested backup set solution with a disaster plan. You don't need a multi-cloud solution. You might not even need a cloud solution. You might just need a server somewhere. And speaking of which, myth number six, we need to implement microservices on Kubernetes. You probably don't. You probably need a partial server for your monolith. That's probably a much more efficient solution. You know what? With all these things, what we're talking about is added complexity, added ways of doing things. Yes, they're flashy. Yes, they're new. Yes, they're, you know, what we've been sold by people that say, hey, this, we have to do this. This is a new way of doing things. But the reality is when you look at what you're actually doing, you're creating a CRUD application. You probably just need a monolith, and you probably just need to run it on one server. You probably don't need a dedicated server, even, because your scale is not where you where you estimate it could go. And you know what? Someday it could go there. But probably not today. You can run it on a partial server as a monolith. So adding in microservices in Kubernetes just adds overhead. Now, again, I'm not saying never do these things, but you have to know why you're moving off of a standard train. You have to know why you're moving off of a standard CRUD application. You have to know why you're actually getting added value that outweighs the added cost. Too often people think that it's just value, not there's no cost. There's always a cost, and in fact, that cost is very, very high for a lot of the things we're talking about. Myth number seven, we need, I'm gonna step on some toes here, we need domain-driven design, clean architecture, and 18 more design patterns. You probably need about 15 classes and well-written code. You don't need to have all the complexity in most applications. In most of your applications, you probably don't need to have anywhere near that much code. And again, you're adding all that overhead. You never counted the cost for that. You just looked at the benefits. Well, when we do this, we could have all this, you know, replaceability all the rest. Are you using it? Are you going to use it? Have you tried this? Have you used it in production for 10, 15 years? See, this is the problem is that you've learned these things in theory, maybe even a little bit of practice, but have you actually supported it long term for any of these things? Are you looking at this from a novice perspective, or are you looking at this from an expert's perspective? That's the important bit. You can build a working modern application using tools that have been around for 20 years. In fact, a lot of the applications you use today are running on systems, on back ends, on even the front ends that are 10 to 20 years old. So it can be done. They can be very powerful, they can still be effective using the old system, the old ways of doing things. The problem is that it's not sexy. It's proposing a simple train instead of a high-speed bullet capsule going hundreds of miles an hour in near vacuum. The latter sounds a whole lot more fun. The thing is, it's a whole lot more expensive, harder to maintain, harder to work with, and in fact, won't be as performant as a simple train plotting along. So when you're building applications, you need to think about that. And part of this comes down to having experience. So why does this happen? Because developers haven't actually gone through this process multiple times. They haven't lived the consequences of their decisions for long enough. They haven't seen their grand ideas and theories on software development actually lead. The funny Dunning Kruger study was that incompetent, and that's a harsh word, but incompetent individuals fail to recognize their own lack of skill. They don't know, they don't have the skills. Essentially, if you don't know enough about a topic, you think you know more than you do. This is why people think that they could fight and win against a bear or a moose. That's not happening. This is why large portions of men who have never flown a plane feel confident they could land an airliner if ever called upon. In software development, that leads to people designing over-architected mega applications and then underestimating how long it will take to complete them. This is why there are so many passionate arguments on lines about which argument or database is the best. So this is why people are talking about, you know, this is best or that's best, or this language, or that database. They're talking about trying to optimize microsecond differences. This is why so many developers get mad at the job market because they know they're overqualified and yet no one hires them. That might hit a little heart close to home. Just like with the innovative engineers who ended up reinventing the train eventually, developers can spend a lot of time and energy eventually working their way back to a simple credit application. So what's the takeaway from this? Well, number one, keep things simple. Remember that you don't have to architect everything. And kind of along with that, and we'll call this number 1.1 or maybe number two, is learn from the past. Remember that just because it was built 20 years ago doesn't mean we should throw it all away. Just because it was built 20 years ago doesn't mean there aren't lessons learned there. In fact, there's a lot of lessons learned. I see too many people that look at an existing application that's messy, very messy. Most applications are once they've gone through 5, 10, 15 years of life. But they look at that application and say, you know what? The best thing to do is probably just to delete this whole thing, start over from scratch, rebuild it from the ground up. We know all we need to know. We're gonna build something so much better. And I've seen it done. I've seen people try to do it. And what happens is they end up making a more complicated solution. Things break down, it takes forever to build, and the end result is still a messy system that doesn't even address all the things the old one did because they didn't learn, didn't know all the things the old one did, or didn't learn those lessons the hard way instead of relearning them the hard way, and you create a mess. That's not the way of going about it. You want to learn from the past. Keep things simple, look at the old systems and learn from them and then iterate on them. Don't just try to reinvent the wheel. Number three, learn more about your topic before changing how you do things. So when you look at an existing system, you say, this was dumb. I've done this. You look at a system and go, the guy who didn't for me was an idiot. He was stupid. You find out later yourself. But you know, that person, they made some bad choices. I've had this in my house where I look at some of the things that have been done, I'm like, man, this was dumb. I he should have done something different. And the more I get into it, the more I learn about it, the more I start working with myself, the more I realize, you know what? That was a simple solution. That actually worked. Yeah. It felt like a workaround, but it's probably the best solution here. The more I learned about it, the more I realized that the person before me wasn't nearly the idiot I thought they were. Okay? Learn more with the topic before you start changing things. Number four, build multiple practice applications with new systems to learn their value. In all of this, I hope you don't take away that I'm saying don't ever do new things. I am not. I advocate for doing new things. I advocate for moving to new versions of.NET. I advocate for using Aspire and other new things. However, I do so within limits. Okay? I don't just throw out everything that's old. I don't just say, hey, you know what? Now that's.NET 10, let's throw in everything.NET 9. In fact, the C sharp master course, which I am going to be redoing soon, but even so, right now it teaches.NET framework and various versions of.NET. So it has some.NET Core in there,.NET Core 3.1. It has some.NET 6. It has some.NET 8 in there as well as.NET Framework. And the reason why is because you shouldn't throw out the old systems. You might be working on them. And so you should know how to work with them. You shouldn't just go to the latest and greatest things. But that doesn't mean that you shouldn't learn about those new systems. It does mean, though, that you don't just learn in theory. Learn in theory by watching somebody else and going, oh, that's a great idea. That's a great new way of doing things. That feels good. It feels like you understand something. The problem is that you don't know the pitfalls. You don't know the practical real world hands-on lessons that you need in order to really use something like that in the real world. You don't know where you're going to fall down. And so by building practice applications, you're going to find some of those. You're going to find out, ooh, you know what? Yeah, that's a problem. You know, I'm doing this course on Unity, the game developer master course. And I'm working through, you know, we're working through the various things like adding new ass art assets. And I'm not an artist, so we're going to go find art assets. And you'll look at art packs and go, man, that's perfect. That's exactly what I want. And it looks great. But then when you go to use it, you almost always find that it's missing something. It's not quite what you need. It's and all those things you learn from practice, from putting your hands on it, from actually building something with it. And when you do that, you realize, oh, this is what I need to look for. This is how I need to evaluate better. The same thing is true for building software applications. When you're building applications, you need to actually build applications to learn their value. So it's not just enough to watch somebody else do it or do a couple small practice things. Build something that's actually somewhat real, even though it's small. Number five, look past the hype and the flash to find the true added value. So there are so many things out there that have hype and flash. I mean, you probably cringe when you heard me say AI because you've heard that term so many times. And quite frankly, most of the times you've heard it are probably hype. And when you look at any new technology, you're going to hear people talk about it that they're talking about because they are trying to get you to use it. They're trying to hype it up. They're trying to say, hey, try this. And that's not necessarily a bad thing, but you need to figure out what's true, what's real, what the real added value for you is. Because maybe it's a lot of added value for a certain demographic and you don't fall in that demographic. So look past the hype, look past the flash, look past the really cool demos that do really amazing things, and try to actually use it. Figure it out. Try to figure out where the actual value is for you, where the thing falls down a bit, and figure out what's going to be good for you and what's going to be something that, you know what, you're just going to let that pass by. Maybe you don't use Kubernetes because it doesn't add enough value to you. Whatever that is, figure those things out. So as a software developer, knowing how to identify the fact that you may be falling under the Dunning-Kruger effect where you may be looking at a situation and overestimating your own knowledge of it is really important. Getting in there, getting your hands on, figuring it out, and figuring out how much you don't know is extremely valuable. Because once you do that, you can start to identify where that thing might have value for you. Because once you find all those potholes and figure out the things it can't do and figure out where it actually can provide you some value, then you can start looking at that, evaluating that with your other applications and maybe iterate and make things a little better. Don't just throw everything out and start over again. When you do that, you are going to incur a huge penalty and you're not going to add a lot of value to your companies. Instead, figure out how you can iterate and improve, but don't take your eye off what works. Don't abandon the train because of a flashy new topic. Figure out how to make a better train. All right. Thanks for listening. And as always, I am Tim Corey.

SPEAKER_00 26:51

Thank you for joining us for this episode of Dev Questions. When you're ready to learn to think and code like a professional developer, head over to IamtimCorey.com and enroll in a course.