324. What Should We Do When We Have Had a Data Breach

We just had a data breach. Now what? What should we do to protect ourselves after data was accessed or stolen? What should we change in our application after a security incident? These are the questions we will answer in today's episode of DevQuestions.

Website: https://www.DevForge.com/ 
Ask Your Question: https://suggestions.iamtimcorey.com/
Sign Up to Get More Great Developer Content in Your Inbox: https://signup.iamtimcorey.com/

Transcript

Copy Transcript Copied

SPEAKER_01 0:00

You just experience a data breach. As a developer, what should you do next? Now I'm not talking about handling the breach itself, but there's an important set of tasks that developers should do post data breach to further secure your systems and ensure that your code is not causing a future breach. And if you haven't experienced a data breach yet, this will be a really good proactive set of steps that you can take to secure your systems. So let's talk about it in today's episode of DevQuestions.

SPEAKER_00 0:36

Welcome to the DevQuesti podcast with Tim Corey.

SPEAKER_01 0:40

Software development is more than just writing code. So let's talk about the rest of it. Specifically, let's talk about what to do after a data breach. Let me be clear again, this is not advice about the actual breach itself. In the case of a breach, talk to the relevant authorities, employ an expert to help you, and let the affected parties know. But as a software developer, you do have next steps. Steps that you should be taking to prevent a breach in the first place, but also that are much more popular with management after a breach. Use this time when people are aware of a danger to do the most good for your application and your company. And if you haven't had data breach yet, still follow us advice if you can. Now, the boss may say, yes, that's great advice, but this isn't a priority right now. If you do get that message, well, at least you have it in writing that these are the things that you suggested. That way, if there is a breach later in the future, you can say, okay, now we've had this breach, let's go back to what I suggested and do those steps. That way, not only have you said, here's a plan for what to do now, but also you kind of subtly pointed out, I talked about this. And then you can further secure your application. Security, it's hard, right? It's something that doesn't show up in new features. It's not something that's that's flashy and great to see. But at the same time, it's really important. So by going through this list and saying, hey, let's do these things, being proactive, you get the opportunity to at least put it in front of your boss. And maybe they say yes. And if they say, no, you know what, that feature is more important than securing your application, well, that's fine. That's their prerogative, but now you have it in writing for when it does come up later. Okay, so let's say you've had a breach or you're preparing for a breach. Either way, what are the things you should do? Number one, inspect every place where you're gathering user input. So too often people trust users. It just don't do it. Never trust the user. But it's very easy. Maybe you have just a comment field somewhere and you say, leave a comment here. Well, are you making sure that that data is stripped out of any type of escape characters or other ways to bypass security? Or are you just putting that in the database? Don't just put it in the database. Make sure you clean the data first. Inspect every place where you're gaining or gathering user input. Make sure that every single one of them you are stripping out anything that could be an escape character. And you might say, well, Tim, that's that's that list is huge. Well, then what you do is you just say, these are the things we allow. And don't allow angle brackets or you know, double dashes or whatever the case may be that might escape out data. And then also make sure that when you're using that information, that you also store it safely. There are tools that will help you put it in the database safely. So, for example, uh, when it comes to store procedures, or you're using Dapper in general, you can use parameterized SQL. So you're not just saying, hey, let's build this string based upon user input. Don't do that. That's SQL injection 101. Don't do that. Instead, use parameters. That way, even if there is bad data that gets through, you are protected against that. Now, you don't want to let bad data through to that point, but this is defense in depth. Don't just have one wall, have multiple. Now, that does mean that you need to think about the data that's in your database and not always treat it as safe. Because maybe it got through the first layer and went into the database and it didn't have a second layer at that time. Maybe someone edited the database manually. Maybe there is an app out there that's writing to the database that doesn't have a safety and security in place. Well, this is where, first of all, layers help because if you had a layer closer to the database, like even the store procedures themselves, that would protect you by catching it at that point. But still don't trust data and database just because it's in your database. Make sure that you're checking that data too. And that comes number two, inspect every place you get data from that's an untrusted source. I don't trust most of my own database because it came from users initially. So even that can be an untrusted source. But then if you pull data from an API, if you pull it from another application in some way, if you have any type of data transfer between applications, don't trust those. Because what if they get hacked? Well, then they're gonna be inside your walls. And they can then send unsafe data over to you and then breach you. And it's a cascade effect. Instead, have defense in depth. Make sure that you put walls around your things so that even if somebody else is compromised or another app is compromised, that you are not compromised as well. So inspect every place you get data from that's an untrusted source, and then figure out how to secure that data or then turn it into trusted data. Number three, shrink down the permissions of every connection string. So talking to a database is very, very common. That's you have to do that. But too often we have connection strings that are admin level. This should not be. Now it depends on your database and your data source where you're using what you can cannot secure. But let's talk about Microsoft SQL. That's a pretty popular database option. With Microsoft SQL, the easiest thing to do is just give system administrative permissions to your credentials, and then everything works. Don't do that. Never do that. But I know it's easy and your application will work. And if you give it too strict of permissions, it might break. That's the thing. You want it to break. You want to say, hey, this is too restrictive, let's loosen it up a bit. Rather than saying, hey, this works, we think it's secure enough. Make sure you lock it down. This is why I personally love store procedures. And I know that some people go, oh, that's that's old technology. It's not. But one of the things it allows you to do is say the connection string I'm using only has the ability to execute these store procedures. It cannot read tables, it cannot list the tables, it cannot drop tables, it cannot create entries, it cannot do SQL injection and do anything bad because of the fact that we're locked down to only executing certain store procedures. Which means that even if a person grabbed the connection string and got the username and password, they still can couldn't do anything more than the application can do. That's locking down your connection string. But if you say, well, it's the entity framework connection string and it has to be able to create tables and drop tables, that's an admin. So if you're using the same connection string, that's a problem. You can use a different connection string, one that can only read certain tables or do certain things. Lock down every connection string possible. If it's APIs, try to get a read-only API as opposed to a read-write API if you don't need to write. All these things are least possible permissions. That's what you're looking for. Not the most possible permissions, not the easiest to use, the least possible permissions. Lock this down. Number four, audit who has access to secure data or systems and reduce that list as much as possible. If you have a web server, that web server, let's say it runs on-prem, and so it's it's gonna in a server room somewhere. Well, that server is going to have unencrypted access to connection strings. It's gonna know how it talked to APIs, it's gonna know whatever your database credentials are, and it's gonna have that information unsecured. Even if you encrypt all that information, it has to decrypt it in order to use it. So in its memory, it's unencrypted. So if someone has access to physically access that room, well, they might have access to get your unsecured passwords. So secure those rooms, secure access to your systems. If it's Azure, don't let everybody have an Azure admin access. Give them only the rights that they need. And the same is true for your data. Not everybody needs to even be able to see the production database. Personally, I don't like any of my developers to have any access to anything production, which may be mind-blowing. How do you debug things? Well, the way we do it is we take the production database every night, we back it up, we clean out the sensitive data, and then we create a new dev database from that. Well, now we have as close as possible to production to recreate almost everything. And the same is true for debugging when it comes to connecting to servers and systems. I have a pre-production site, I have a staging site, I have a dev site that are as close as possible to the production site so that we should be able to recreate things that happen in production, in development, or in pre-prod, or in staging. So we never have to have access to production. So that as few people as possible have access to that secure data. This reduces your overall exposure level as an organization. Number five, if you've had a data breach, or if it's been a long time, rotate all your keys and passwords, all your API keys, all your database passwords, everything. And you may say, well, I don't even know where all those are. There's a red flag. Figure it out, document it, make sure you know every single security entry, every single password, every single API, what it's used for, by who, who has access to it, when it was last changed, and how to change it. These should all be procedures. And I know we get busy, we don't do it. This is a great opportunity to change that. Put some security in place so that when it comes to rotating keys and passwords, it's just easy to do. When it's hard to do, here's what happens. Maybe you have an employee who goes rogue or you know gets frustrated at the organization and they end up being fired. So security comes by, sends it to HR. HR says, hey, we're gonna frog march you out of here. You can't have access to anything. We've locked down your passwords, all this, all this stuff, right? But if your security keys for your database or your Azure information, or whatever these API keys, whatever these other things are, if they're hard to rotate, well, all of a sudden now you have an exposure window where you have a disgruntled worker who has had access to all these things, who might have written it down, or might still remember a password who could do things that you don't want done. This happens a lot. So make sure it's very easy to change these things. Here's another thing. Maybe you have a fellow coworker that you love. They're great, they're awesome, and they just get a better opportunity. Maybe they're moving out of the country, maybe they're you know moving to a different state and they found a great job somewhere else. There's no hard feelings at all. Everyone's happy, you have a party, et cetera. They leave. What's the first thing you should do? Change all the passwords. Now, is this because you hate them? No. Is this because you don't trust them? No. What this is, is a security measure. It protects them and it protects you. I have been in situations where we trusted the person, the person had no ill intent, but they still had access to a system they should not have had access to. And they wanted to help someone out. So they accessed the system they should not have had access to, and they did things that were no longer the correct policy. And because they didn't know, because they'd left. Well, now they have done something that caused a problem that blows back on them. They get blamed for something, they get in trouble for something that really what it was was just an all-around bad situation. They shouldn't have had access. They should have tried it and go, oh, that's right. I'm no longer an employee there. I can't access that. So they don't feel bad anymore telling a person they're trying to help, hey, I can't help you anymore. So they don't feel bad. And then you don't have to track down problems. You don't have this whole big issue because they did help somebody in the wrong way. So rotate all your keys and passwords every time a person leaves on a routine schedule, and definitely anytime you have any kind of disgruntled person. That's number five. Number six, reduce the amount of data you retain. Companies tend to be hoarders when it comes to data. They store everything. And developers, we kind of encourage this. Hey, you know what? We might need this, we might need this. And before you know it, you have these massive databases where you're storing things indefinitely. When we used to store things in filing cabinets, it was a little easier because at some point we're like, we have too much stuff. And so we create a system and say, hey, every three years this gets archived. Every seven years, these get deleted. Every 10 years, this was what happened, this is what happens. Because we knew we didn't have the physical space. But when it's digital, it's a little easier to say, well, we could store this forever. We don't need to have the order history with address and other sensitive personal identifiable information of a person who hasn't been an employee for or hasn't been a user for a decade, right? There has to be some type of scenario where you say, you know what, we're going to archive off this data. It doesn't mean you delete the data. It might just mean that you put it into a different place that doesn't have direct access very often. And it might take some type of background job to retrieve that information. So maybe that person has been gone for a decade, maybe come back a year later. Well, you could have a system that says, okay, hey, we're going to retrieve your old information, but it might take a day or two. And where it might take 15 minutes, whatever the case may be. But you have a background job that says, hey, let's go out to the repository, let's find the archive data, let's look through it for this person, let's repopulate the information, et cetera. But that way you're not storing on your server all the sensitive information that if you did have a data breach, all of it goes. You have to let people for the past decade know that you lost their information. Instead, try to retain as little as possible. You don't need every bit of data. Here's a key way to see if you need it. Do you use it? Do you use it? Do you actually use it? Not just a display, do you actually use it for something? What you could do, for example, yes, my Amazon history, my purchase history, yeah, that's something that is probably important because I can go back and say, hey, when did I last purchase this? And it may have been six years ago. But my search history, no, that's not important. Get rid of it after 30 days or 60 days, because at that point it's no longer relevant. And there's other things as well, what I watched, all the things that it's not high-level important data. So figure out how to identify what's important, what's not. Even if you say, you know what, you watched this movie, but we didn't track how far into it you watched. We didn't track where you left off because it's been a year. So you tracked that yes, you watched it. That's just one bit in the database. Whereas where you want, you know, how long you watched it for, where you left off, et cetera, that's just information you don't need to keep. It is additional storage. Okay. Number seven, check every boundary. So if you have two applications and they talk to each other, make sure that communication is secure. Whether it be using HTTPS, whether it be credentials, whether it be some type of authentication, encryption system, whatever the case may be, check between the systems. This is where people get in, is in the boundaries. Where they sit between two things, maybe they listen for that communication between each other, or whether they can impersonate one thing to talk to another thing, check every boundary. Which is why microservices are the worst in this type of situation. Because there's lots of boundaries. There might be thousands of different ways to connect to all these different systems. That can be a massive security nightmare. So be very careful. You might say, well, our microservices are all internal. Guess what? So are your users, and not all of those should you trust. Not all of your admins should you trust. So let's make sure you check every boundary to make sure that you're not leaking information, that you're not leaving a hole open for someone to exploit. And if you do find, hey, we didn't do this, look at every other boundary to see if that same thing was occurred in other boundaries as well. Number eight, check your logs for sensitive data. Logging is a good thing. Logging helps you track down errors. Having good logs that really give you a sense of what was going on before an issue occurred, during the issue, and after the issue, really, really helpful. And too often what you'll find is when you're trying to track down an issue, the logs weren't in detail enough. And that's frustrating. But your logs are a place to accidentally send out sensitive information. So let's say you have an exception where a user is filling a form out and they fill out this data, including maybe their phone number, their social security number, something else that's sensitive. And they fill out the whole thing out, but they put some kind of data in there that crashes the application. It shouldn't, but let's say it does. And your log catches that exception and stores all the data that's in the form. Well, that might include the phone number, the person's name, the person's address, their social security number, or other sensitive information. Now that's in the log files, not just in your database. Now your log files probably have a different level of security than your database does. Maybe your entire development team has access to your logs where almost none of your developers have access to production data. So all of a sudden, you've exported this secure information to a less secure informate place. So that's a problem. Check your logs, make sure you're not capturing sensit information. And if you are, delete it, figure out how to get rid of it, obfuscate it, you know, swap it out. So when you're capturing things that are sensitive, maybe put stars in their place. Whatever the case may be, make sure that you clean the data in your logs, clean it out of anything sensitive. Number nine, update what you log to make it easier to catch problems. So, yeah, sometimes that means more log statements, but realistically, going back to what we talked about with reducing the data, what do you actually use of your logs? Too often people say, Well, I need to catch every single problem and I need to make sure I can track all these things. And you have massive logs that you never look at, or the only thing you look at is a graph where you say, Oh, the number of exceptions has gone down recently, or whatever the case may be. That's not actually helpful, and you're you're catching way too much stuff, which means you don't look at anything. It's better to have three log entries that are valuable than to have 300 where 30 of them might be valuable. Because you might not ever find those 30. But you'll see all three if there's only three. So make sure that you log what is important and you make sure that you're making it easy. Easier to actually catch problems. That probably means reducing a lot of your logging. It also might mean adding some logging too. That's for sure. But it probably means reducing a lot of your logging. Number 10, update your documentation on how to secure data. This means when new developers come in, they know exactly how to scrub information that comes from the user. They know how to check the boundaries for any type of data loss. They know how to make sure they protect certain fields from going into the log files. All these things are important to document. So you know very clearly how to do these things so that maybe your system is secure today, but it might not be tomorrow when the new developer works on a new system. You want to make sure that you're secure today and in the future. So there's my top 10 list. And you know what? That's not a comprehensive list. There's definitely more things you could do after a data breach depending on your circumstances. But this list is going to get you well on your way to securing your applications. The two biggest things that endanger your applications are obvious mistakes and compromise employees. So obvious mistakes, it's, you know what, we allowed users to just put whatever they want and we didn't check to make sure it's valid data. Obvious mistakes is we forgot to put HTTPS on the communication between those two microservices. Obvious mistakes is where you use the SA password, the system administrator password, on your connection string. These are all obvious mistakes. And then compromise employees might mean that yes, they're being blackmailed to do something. That's probably not the case normally. It may be that they're trying to be helpful and going places they shouldn't go. It may mean that they don't realize the danger and they just do something. It may mean that they're disgruntled and want to hurt you. But these are the two biggest areas of security holes in your organization. It's not about the this dark hacker that's sitting in the back room that's got his whole team working on nuanced ways of getting in. You may have that, but that's not the most common thing. The most common thing is that you make mistakes and leak data because of it. You make mistakes and it make it easy for a kid who's just learned how to do a basic SQL injection to access your system and access your data. Like these basic mistakes are a big part of data breaches. So this list is going to help you eliminate both of those issues and it's going to make it harder for you to be breached. Now, no system is breach-proof, but that's no excuse for making it easy. Make sure that you protect your system as much as possible. Thanks for listening. As always, I am Tim Corey.

SPEAKER_00 25:12

Thank you for joining us for this episode of Dev Questions. When you're ready to learn to think and code like a professional developer, head over to IamtimCorey.com and enroll in a course.