Christa: Efficient, effective digital forensics and incident response involves not just the right tools and processes, but also the ability to share insights and collaborate on work. Here with us on the Forensic Focus Podcast to talk about it this week are Mason Toups, a DFIR analyst with Solis Security, and Emre Tinaztepe, founder and CEO of Binalyze. I’m your podcast host, Christa Miller. Welcome Mason, and welcome back Emre.
Emre: Hi Christa.
Christa: Mason, I’d like to start this podcast off with a question that calls on your expertise. Going back to the basics of digital forensics and incident response, what for you are the essential elements of a proper digital forensics investigation for any organisation?
Mason: So, first of all it’s identification. Identifying what the issue, cause or purpose of the investigation is. What I often like to do is really listen to the client and listen to what they say. I’ll even ask them non-technical questions, ask about their lifestyle. Because I believe sometimes the answer as to how an incident happened, or where the root cause is, lies in what someone has said, rather than in the data you might be trying to collect.
Christa: Very interesting. Can you give an example?
Mason: Yea. So for example, let’s say someone has a 2500 endpoint environment, and the client goes and tells you…he’ll tell you how he doesn’t understand, he has every single security measure known to mankind, but he doesn’t understand how they keep getting in. And then you might talk to him and you listen to him, and he says to you…when you ask about his lifestyle or any recent things, he might tell you, “oh you know, I’ve had this weird bank activity going on”, or something like that. And then you’re like “OK, all right”. And it was just a passing question on any recent personal things he had to deal with. So that type of information could clue me into saying, “well, maybe he has a phishing problem, or has some malware on his personal computer, or something like that, and that’s maybe contributing to the incident.”
Emre: That’s really smart, Mason. I’ve never thought of it.
Mason: Yeah, I became a lot more of a people person over the pandemic!
Christa: I want to say, haven’t we all? But I feel like some of us went one way, and maybe some of us went the other way and became less of a people person, I don’t know. But I think that’s even more critical though, right? Because I feel like the way that they did business changed during the pandemic, with people going a lot more…I feel like that kind of changed the threat model for a lot of organisations.
Mason: For example, if you talk to them (if you talk to your client or your requester), and they say to you, “over the pandemic I had to figure out how to allow my employees to remotely access computers from my house”. And I’m like, “so you had an IT or MSP do that?” They’re like, “no I looked it up over the internet”, and I’m like “oh!” And usually the first thing that any beginner to networking does is try to do port forwarding, like kids do to have, like, a Minecraft server or something like that. I’m like “oh, OK!” Maybe they did some port forwarding for RDP opening [inaudible].
Sometimes the answers for the root cause of your incident lie in what you get out of people, rather than just the data you can collect. And in terms of time efficiency, sometimes it’s better to (most of the time) see what you can figure out from the person themselves, rather than be a thousand or something machines in, and then realise that, “oh, I had this one time…” and you’re like, “why didn’t you tell me that earlier?”
Christa: Yeah, I can imagine that would, if anything, help you figure out what data to collect, doing that.
Mason: Yea, absolutely. Well you kind of have to ask questions like, “hey, where is your important data at?” Because the attackers, they’re gonna think the same thing, they’re gonna think, “oh, this environment is really big, how can I be efficient with my time so that I can move on to the next guy and pop him?” Because they’re thinking about it also like a business, right?
Christa: So after identification what other elements do you feel are essential?
Mason: Well at my experience here at Solis, frequently we run in and the environment is burning. All the time! And then you have analysts, but you can’t get to the analysis. First you have to make sure the incident is contained. If you don’t contain the incident first and then you just wait for machines to come in, and you just do it the normal way, then while you’re doing the processing of data, not ensuring containment, the attacker could be at this moment, every second, be pulling out a MB or a GB worth of files out of the environment. So that’s why verifying containment is usually my next step.
Then you have the collection and the preservation of data, and now that you’ve preserved the environment, you can collect and preserve that data without worry of interference from the threat actor. Because these days, again, for criminals this works as a business. You reveal your hand, or the cards that are in your hand, before you have a chance to block them off from seeing the cards that are in your hand. And they might go, “oh, well we see this tool getting dropped down” or, “oh, we see they’re doing these things now, so let me use (the most frequent thing they use) a Cobalt Strike to drop like a Terra backdoor, or an RMM tool, or a remote managed monitoring tool”. So that you know when antivirus or whatever gets dropped on there, they don’t see anything and I can maintain my access, or, “let me go and just cancel all these mass collection jobs that are happening”. So containment first is key before you start doing any collection.
Christa: OK.
Mason: And then…
Christa: Yeah, no, go ahead.
Mason: I was gonna say, and then for analysis: the focus on analysis is well, “you’ve contained it”, but often that containment can lead to business downtime. So when you’re doing your analysis at first, your first focus is on finding the indicators of compromise and finding out how they got in. So you need to find those two things very first before then you can focus on necessarily all of the lateral movement, all the files you touch. First you’ve got to focus on the IOCs, so that they can start having something blocked, and then the: “how do they get in?” So that you can remove some of your containment steps that might be leading to a business interruption.
And then, of course, documenting down those indicators of compromise and the root cause very early on, just in case (let’s say, you’re working with another investigator, or you have to step away to go take care of another client or requester), that the analyst doesn’t have to go and do everything you did all over again, they can look at the notes and go, “OK, I see that he has the root cause and the indicators, and the containment steps have taken place, so now all I need to worry about is just taking this collection of systems and start processing”.
So documentation after analysis is the next important step. And then of course, you’ve got your usual presentation, which is showcasing those facts. But sometimes that cycle repeats, or sometimes you go: analysis then straight to presentation, then analysis, presentation. Because sometimes the client might have to update the board, or the people above them on what’s going on, and so you might have to quickly deliver some of those answers before you’re able to sit and feel very comfortable with everything right in front of you. Like, sometimes, you have to answer specific questions first so that they have something to tell somebody that’s above them.
Christa: Right. Well I imagine also to communicate with, I mean, really all of the stakeholders in their organisation, right? Including legal and risk, right? So my next question is for both of you.
Emre, I’ll start with you: going back to the time when you were doing forensic analysis work yourself, how much do you think the digital forensics industry has transformed between then and now, and in what ways?
Emre: Thank you, Christa. When I started it was around five years ago into the visual forensics. Before that I was heavily engaged with the malware analysis. And so that’s why I was actually trying to apply the knowledge I acquired in malware analysis to the forensics. And the reason was, at that time, the traditional forensic methodology was…most of the time was not sufficient. Because industry was moving into memory forensics, which was actually kind of an answer to that insufficiency.
But again, it faced the same problems of the traditional forensics, just like the case 30 years ago when we had 4 MB of disk space, it was something feasible. But nowadays you can easily buy 20 TB of storage, and it makes it quite insufficient.
So memory forensics was facing the same, even though it was popular. The way I was involved in high profile cases, I was mostly, like, asking for a RAM image. So, “send me the RAM image”, because that was the quick and easy way for me to find the answer: how it happened, or what type of malicious software is running there. But then that was exactly the time it started to be insufficient as well. Because RAM sizes were quite big, so it was not like the times when we had, like, 4 GB of RAM, which was easy to investigate. And again the lack of the tools for investigating them in a bulletproof way. I mean, most of the time the biggest problem…it’s 80% works, but 20% it doesn’t. And especially after the changes in the operating system, these tools started to break.
And when I look at it now, people are persuaded that a quick way of starting an investigation, quick way of understanding what happened on a machine, is not only possible with disk images, RAM images: there are easy ways. And I think the current mindset is already changed, because previously when I started around five years ago (when I completely moved to digital forensics, let’s say), the mindset was still…I call it traditional forensic remnants oriented, because people were still thinking about, “should we take it to court, how can we do it?”
Because in RAM, or in volatile investigations, it’s not the case. I mean, 99% of the time you don’t have anything related to the court. Like as Mason pointed out, it’s on fire so you need to get back to normal. You need to, like, save your data, or prevent further damage.
So today people are persuaded that there is no court anymore, except some cases. Like it can be an internal investigation, or IP theft, something. But most of the time, especially when we are having almost a new breach every week, a global incident, that is affecting all the industries. So I think the awareness is highly increased now, compared to five years ago, and it’s again, due to the fact that there’s a global affecting cyber security incident.
And also, what I observe is the confidence into the security products we are using right now is much less than it used to be compared to five years ago. So we appreciate that there will be a breach, and we should be ready for that, and I think that’s the main reason we are talking about cyber resilience, forensic readiness, those type of terms are getting quite popular.
Christa: Mason: compare and contrast. Would you concur, or have you seen different industry changes?
Mason: I would say yes. From my perspective, the way that forensics has changed is just…you really have to learn about what your opponents are doing. Like, when you go to school for forensics (if you’ve ever gone), you would usually focus on the technical details and everything, but maybe what a lot of people don’t realise is that there’s psychology in that. You need to understand what your opponent is going to do next, in order for you to understand what you need to do next.
And to that goal, you have to be more decisive, or quick to make good decisions faster than your opponent. You have to always be, after every case, you always have to be…even if it’s 1%, it’s going through and saying, “hey, you know we did this wrong, did this wrong, but we did these things right”, so that you can formulate your next plan of how you’re going to tackle the next incident. And you have to keep doing it, because that’s what the attackers are doing.
Christa: Yes.
Emre: And also something to add, Christa: I don’t remember hearing the word APT that often five years ago. Because that was not the case, I mean, what was popular was some groups of people or maybe individuals, they were mostly financially oriented. So I think the increase in the APT groups and the organised cyber crime, let’s say, is making people think about assuming breach more.
Christa: So that brings us to the present day. We’ve talked a little bit about what’s changed. So Mason, when you’re leading investigations, coordinating them, bringing them to a resolution and then reporting on security incidents is very dynamic work. It requires being fast and agile at all times. How do you achieve that?
Mason: People, process, technology: the three things that are important to always be thinking about when you’re trying to improve anything moving forward. You want…in order to be able to quickly move and be agile, you have to have the right people with the right skills. In order to fully utilise those people’s skills and talents, you have to have the right processes in place. And then to scale the talent and skills that you have available with the processes that you have, you need to have technology.
Because you can’t have, like, 100 investigators that you hire doing manual labour and only doing one case at a time. You need that to scale, you need to be able to have, let’s say, four or five analysts that can take on two or three cases at a time, and have their talents and skills be scalable on those incidents with the processes and procedures in place.
Christa: So, what are the challenges that you encounter across those three pillars?
Mason: Honestly right now, it’s all three. The attackers…
Christa: Fair!
Mason: It’s really not just one. You have where, like, most of America is just like, “oh, we’re going online, or adopting technology”. It even goes down to just a simple owner who owns, like, a coffee shop or two, who might have never been using really technology at all, they just might have had a website or whatever. But now they have this certain infrastructure set up so that they can remotely handle business.
And there is a shortage of people that…there seems to be less of a shortage of people that can set them up with the technology, but there is definitely a shortage in the people that can secure, or actually are security-minded enough to know, “OK, I can set up the technology, but I also know that the way I need to set this technology up in a way that’s secure”. And there’s a lack of people that lack that last statement that I said. Knowing how to not just stand up technology, but knowing how make it so it’s secure, and not make it a high risk to have that then leads to an impact to business.
Christa: Right.
Mason: Yea, go ahead.
Christa: Oh, no I was just thinking that it’s quite a bit different from setting up a Minecraft server for your kid!
Mason: Yeah totally different! Hey it’s a good start but…
Christa: Right, and sorry I think I interrupted you, please continue.
Mason: No it’s OK. And then, in terms of processes: again the attackers (I’m going to talk about the ransomware threat actors), those guys have like well over a million dollars in budget. Like each small group has their own nest egg of just an unbelievable amount of income, versus how much income they would have had a couple years ago, versus the security (or the blue team) has much less of a budget than those guys do. And so what that means is, in terms of challenges, is you have to have processes…you have to constantly change your processes in order to adapt to what the threat actors are doing. Because they’re doing it all the time.
I mean, I don’t know if you’ve heard of it, but there’s the Conti ransomware, right? They have their own playbook that almost all of the other ransomware actors are following through with and using. So from a forensic perspective, if you have 10-30 cases where they’re going to copy and do the same thing, then it’s just like, “oh, well fine then I’m going to adopt…I’m going to adapt my process that says we’re going to look for these exact things and if they show up, they’re going to get blocked!” But then if you’re….when you get a good success rate going within like a month, then they’re like, “oh this is not working!” They’ll tell their other friends that, “oh, you know, this adversary is a little bit too much”, you know. They treat it like a fun challenge more than a, “holy crap, we might be killing somebody”.
And then the next month, they would have changed everything. And so then you have to change what you collect, then you have to change what you’re going to process, or look at, or block, or do. So that’s a real challenge with processes, because right now especially, I think globally, we incident responders, we have so much work coming in, you have to…you’re trying to find somebody that will take on the work. Because it’s so much, and then you have to keep changing your processes and procedures in order to adapt it to the next new thing that the threat actors do. Because they do it quick. They do it quick! Let’s say, people keep using EDR, they’re like, “oh they’re using EDR and AV now. Well then, I guess I’m gonna start…since I have millions of dollars, let me start handing out some investment money for people who work on exploits for those particular devices.”
And then you have technology: around technology there’s usually a bureaucratic process around what technology are you going to use, and where you’re going to use it, and how you’re going to use it, and also what budget you’re going to have for it. Because as organised individuals that want to do things right, that’s what we’re going to do. But for the threat actors, you’re just like, “oh, you know, they’re blocking all of our Russian IP addresses,” and then in less than a month, they’re going, “OK, we’re going to switch to United States IP addresses, or Canadian IP addresses. Now when they employ those region blocks, the only way those are going to be effective is if they’re willing to cut off their line of business and just be able to do no work.”
Christa: Right.
Mason: So they have the…because of how funded they are they’re able to do that. They can just snap their fingers and decide they’re going to go ahead and switch technology, or how they’re going to use that technology. Versus the blue team, it’s usually a much longer process.
Christa: Right. So Emre, as you listen to all of these challenges, in your opinion what is the best way given the seriousness of the situation and the work that practitioners are doing every day to overcome the challenges, and what can industry leaders do to help them out?
Emre: Actually, like I agree with all the items as Mason pointed out. He’s on the front line, so he’s facing it almost every day. Sometimes a few times a day, Mason, correct me if I’m wrong.
Mason: Oh yes!
Christa: I can only imagine.
Mason: They’ll even drop messages, they’ll be phone calling the client, they’ll be like, “I am in here, and I’m not going anywhere until I get my money man!”
Emre: So when I think of the problems, every incident response is like putting the investigators into an escape room, so it’s just like that. And we are asking them to solve the puzzle, and for them to solve that puzzle, they need to look at every single corner, like investigating everything. And when they find the way out, it’s not always the end, because there’s another room that they need to solve again, and it basically becomes like a labyrinth.
So when you start an investigation (most of the time, this is what I hear from our clients, like a lot)…so they start with one asset, and then they end up with thirty, sometimes hundreds of assets, that they need to investigate. And as Mason already pointed out, people start asking questions and…because they want to know how it happened, what are the assets that should be contained, so they have less damage. So not only the industry leaders, I think the whole industry, we should understand this and take actions for simplifying their jobs, because those are the guys who are dealing with this.
And again, it all boils down to the budget, so I think we need to invest more into the technologies that will bring…that will enable us to have more visibility for our environment, and unfortunately our understanding of security is formed around receiving alerts. I think this needs to change as well.
But for this first decision makers should be changing their mindset. I really feel a huge need in the industry that we should assume breach, and start employing the continuous compromise assessment and continuous auditing as a practice, because as we already know, in order to identify which…they’re still spending 200 days, and if we had continuous practises that becomes a part of everyday job of the instant responders (both in MSSPs or in security operation centers), I think we can be much more guarded against these type of threats.
So it’s just a way of patrolling our assets, at least the critical ones. Every few weeks, or sometimes a month. And for this as also Mason mentioned, there’s a lack of talent in the global (and this is called talent wars), and the only way to solve this problem is automation, so we should be investing more into automation. And again it boils down to the budgeting as well, because the more automation you have is less errors, less dependency of people. So I think the automation is the force multiplier for modern cyber crime investigations, and we should be like supporting people on this.
Christa: So in one of your interviews (because I want to go back to a point you just made about the communication aspect), one of your previous interviews with Forensic Focus you mentioned that Binalyze is shifting focus towards sharing and collaboration to advance the DFIR community further. For that, you plan to release Binalyze Hub, a real-time IOC sharing platform powered by the community. How would Hub be beneficial in particular for the recent Log4j exploit?
Emre: Great question. It was a great, actually, indicator of the lack of communication in the industry. Unfortunately our industry lacks communication, and I don’t remember in the last 15 years a customer asking us as a vendor, if we are using a library or not. And again, this was a great example of making people talk to each other (in this case like vendors and customers).
And Hub is actually an effort, not only for this type of communication but for uniting the incident response community. This can be individuals, this can be vendors, like customers: so whoever that takes part in the incident response efforts. And it’s just like the previous version of development. So we used to have stack overflow where we were finding snippets of code, now we have GitHub for that in which we can collaborate. So we see Hub as a way of like sharing incident IOCs, so that way we won’t be required to dig in Twitter for finding IOCs.
And Log4j was a great example of this: it’s been more than a week, and people are still searching for IOCs. This shouldn’t be the way, especially in this era, this shouldn’t be the way of sharing IOCs. Of course there are some…a lot of CTI providers (sabotage intelligence providers), that they are sharing. But in my opinion this lacks the action aspect, because we are sharing information, but the problem is this information is under utilised.
So what we aim with Binalyze Hub is, we want to make this a community effort. So anyone…this can include CTI providers, this can include professionals, experts that are doing this on a daily basis, so everyone should be able to share this. And the question is like, how will we validate this? Because anyone can share IOCs, so there should be some level of confidence. And we have some solutions for fixing this.
So Hub is actually our answer to solve this problem. Because in our opinion, sharing IOCs, especially in an era that APT groups are even more organised, and compared to us, and they even have more budget, as we already discussed. So we should be more organised, we should be more collaborative. Hub is an answer to this that, hopefully, will make a change in this industry.
And what we are aiming with this (the initial version of the product), is we want to integrate this process into the enterprise forensics, so that way whenever there’s a new IOC we want to start the investigation automatically, and we want to have the compromise assessment, again automatically. And if there are any actions that needs to be taken (of course, based on the level of confidence of the IOC share), the action should be taken automatically as well. So this shouldn’t be manual, this should be fully automated, and we’ll do our best to create the first version very soon.
Christa: Sounds good, sounds really interesting. Mason do you have anything to add about either sharing collaboration, or exploits?
Mason: So what I’ve seen in our community is that…what’s happened is the threat actors are looking at the same sources that we do. For instance, if I was to drop IOCs on Forensic Focus, they’re going to look at it and they go, “oh, he knows our infrastructure, we got to change tactics now.” So I think part of the sharing problem now really with these specific…these ransomware actors, people have become more, like, holding it to the chest.
Because, again every time…if they figure out you know about what they’re doing, and you know exactly where their infrastructure is, then they’re going to change it, and then your cost for the next incident might be insignificant, or significantly higher the next time around, because you’ve taught them, or you’ve told them what you’re going to do or how you’re going to do it. So from people who share IOCs, they’re either like, “well I don’t want to share IOCs until the case is kind of done,” or you know, it’s been a month or whatever, and then they’d be like, “oh, they’ve probably changed it already, now I’ll share the IOC.”
Christa: At which point it’s too late!
Mason: Yea. With IOCs, it’s very important that if you’re going to utilise or operationalize those, that you’re able to use them very quickly. Like, if I see an IOC, and it’s a good IOC, I need to be able to have it implemented or being operationalized within a day, otherwise you might immediately just see a degradation in the usefulness of that IOC, and so…
Emre: That’s not an IOC anymore, so that’s the biggest problem. And based on what Mason said, Christa, this should be just like real-time messaging applications. So, I’m able to share information with my friends, like text them, but at the same time, I’m able to create a group, and only chat inside that group.
So this will be, like, collaborative. So you’ll be deciding. We are planning to have this free, so you’ll be deciding to create a public channel that anyone can see, anyone can subscribe, but at the same time, you’ll be able to create private channels that you want to collaborate with your colleagues. So as an example, multiple MSSPs can collaborate, or multiple banks, like security operation centers can collaborate. So there are a number of aspects that are quite similar to the real-time messaging.
Mason: And that’s kind of really where it is now. Now it’s, “we will absolutely not publish this public”, but if someone calls, or we have this private group between these X companies will go, “OK, we’ll share between us so that we can make sure that the client stays safe.” Because we definitely know if we put it publicly, they’re just gonna…the threat actors are gonna crawl, or if you stick it in VirusTotal they’re gonna crawl, they’re gonna have their 30,000 licence key that not a lot of people have. And they’re gonna go, “oh, he found our C2, we need to change it.”
Emre: Mason, I’m curious how many [inaudible] are subscribed to CTI provider, like a paid licence?
Mason: I doubt it, I bet they’re stealing it from somebody else!
Christa: So, Emre, how does Hub prevent that from happening? I guess, is it just limited to your customers, or how does that work?
Emre: It’s exactly like real-time messaging. So how we envision it, is exactly like real-time messaging. You can create a public Telegram channel, that anyone can join. In this case, if you’re an expert that you want to share this information…because there are a lot of people that do this as a passion. So they are creating like URLs, they are sharing it on GitHub.
But the problem is, as an example GitHub is not designed for sharing IOC. Neither Twitter. So the problem we see…as an example, I want to be sure that the IOC I receive does not cause a false positive. In the Twitter case, this is not possible, because I need to run it in a repository of files. Only after that, I can make sure that…at least to some extent, so if you can run it on millions of files, then it gives you some level of confidence. But that’s not the case.
So how we envision it, is being able to create (this is your decision), so you can make a public channel, you can create a private group. This can be the people you trust in the industry, or this can be old colleagues you used to work with, or some other companies. So it’s completely up to you. That’s why we don’t see any risk of sharing IOCs.
But I completely understand, and this is most frequently asked by the customers because. Because I agree with Mason, no one wants to share (especially the customers), no one wants to share IOCs, because they are scared of the fact that it will be also seen by the attackers.
Christa: Yea. Well I’m going to close it there. Mason and Emre, thank you again for joining us on the Forensic Focus podcast.
Emre: Thanks a lot, Christa.
Mason: Appreciate it.
Christa: It’s a pleasure. Thanks also to our listeners. You’ll be able to find this recording and transcription along with more articles, information and forums at www.forensicfocus.com stay. Safe and well.
Correction: An earlier version of this transcript misidentified Mason Toups’ employer as Sola Security. The company name is Solis Security.
