Alex Desmond: Welcome, everyone, to another Forensic Focus. It’s evening for myself, morning for Si, and we’re joined by Chris Doman who is also in the UK, a nice bright early morning for him, as well. He’s from… I should have asked how I pronounce this before I started the intro, but is it Cado?
Chris Doman: I don’t care how you pronounce it, but I pronounce it Cado.
Alex Desmond: Cado, okay.
Chris Doman: Whatever is fine.
Alex Desmond: Yeah, from Australia, we’d butcher it anyway.
Si Biles: I think I’d assume the long A, as well. Yeah.
Alex Desmond: I’m just like, “Cado.” Yeah, can get the accent there. Welcome, Chris. Thanks for joining us on the podcast today.
Chris Doman: Thanks for having me, glad to be here.
Alex Desmond: We always like to start with a bit of an introduction to yourself and how you ended up where you are, so maybe you could just give us a brief background of where you are now and how you got there.
Chris Doman: Yeah, sure. Where I’m now, is at Cado Security, cloud forensics and incident software provider. Co-founded the company about four years ago with James, he’s also Australian, so good Aussie-UK backgrounds there. Before this, I was at AlienVault, if you use their threat intelligence products, I did a lot of software engineering on that.
Before that, classic big 4 DFIR, I was at PWC for about four or five years, doing all the fun of going after to clients when they had problems. I got into cyber security through the DC3 competition. I don’t know if you remember, but the US DoD used to do this really cool, fantastic forensics competition, a really good thing. I did well on that, and off the back of that I got my first job.
Si Biles: Oh, wow.
Alex Desmond: Yeah, that’s really cool. I actually only know DC3, because I did one of their courses, not from the competition. I’m not sure whether they… Maybe they still do it, but I just remember doing a course with them. No? Okay.
Chris Doman: They ran it for about 10 years. It was really good, it had 100 different questions, there’d be an exercise, some coding, some forensics, some unsolved problems you had to research, so it was a really great way of getting new people in the industry without having to pay anything, too, so it’s really good equalizer.
Alex Desmond: Yeah. Nice, nice.
Si Biles: Excellent. We’ll see if they’re still around and put the link in, because if it’s good enough to get you your first job, I think that’s something that we should probably share with the audience.
Before that, did you come from a technical background, or are you a previous ballet dancer? How did you come to this?
Chris Doman: I definitely cannot dance. Standard computer science degree, I was hacking around, writing code and stuff at a young age, so I went through that. And then, went off into a different kind of marketing career for a couple of years after uni, just because I was so sick of computers. And then came back to my first love, so had a brief segue there.
Si Biles: Co-founding a company, that’s a brave step in the world. What put the two of you together in the way that made you risk your livelihood by deciding this was a good idea?
Chris Doman: It’s definitely a scary thing to do. We started the company the first week of COVID. When all that stuff was in the news, and then James and I looked at each other like, “Are we still doing this?”
I was like, “Yeah, I guess so.” It ended up working out pretty good timing actually, because people needed to do remote cloud stuff quite a lot, but definitely, it is an interesting experience starting a company, growing it to now about 50 people, so fairly fast growth in four years.
Alex Desmond: Yeah, wow.
Si Biles: That is impressive, yeah.
Alex Desmond: When you did start, it was just you two together and that was it? Yeah.
Chris Doman: Yeah. James was my boss at PWC, he was running a lot of the, building up the UK IR practice, and before that, he was an ASD in Australia responding to incidents there.
Alex Desmond: Oh, yeah. Okay. That’s really cool.
Chris Doman: It means a lot of people in Australia know him, but they met him for the first time when they had a big incident, and it’s like, “Oh, I met you that time, didn’t I?” He has a lot of those when i meet people at Australian companies.
Alex Desmond: I have so many unfortunate friends that way, and they’re always like, “We hope we never see you at this job again.”
Chris Doman: Yeah.
Si Biles: It’s when you walk into a room and they start twitching nervously, just out of a Pavlovian response, that you need to start worrying.
Alex Desmond: Yeah, yeah. They’re twitching that stress leave application in.
Chris Doman: I think the Aussie government’s done a good job, though.
Si Biles: Something I’m slightly jealous of you guys for, that the people you work with, you can turn into friends. As a criminal defense forensic expert, generally speaking, I don’t want to know the people I work with particularly well. Of course, they’re all innocent, but anyway.
Alex Desmond: Let’s talk about when you first started the company and moving into COVID, that would’ve been interesting. Did you have quite a good market looking for that kind of stuff straight out of the gate?
Chris Doman: Yeah, we looked at doing a few different things when we started. We tested some ideas by putting out some blog posts saying, “If it existed would you have it?” The best feedback we got was around collecting forensic evidence of cloud environments like AWS, that kind of stuff. We put a couple of free tools to collect EC2 virtual machines, this kind of things, and we got good feedbacks around that. And then, we also released another free tool to collect more triage evidence, a bit like KAPE, similar concept, which is a good time with COVID because suddenly everyone was working remote, so we had a lot of uptake of those free tools. And then we ran with it and commercialized off that, off the back of that.
Alex Desmond: Okay. Do you still run any free tools, any open source stuff, or is it you’ve commercialized it all now?
Chris Doman: We changed it, so now we have a Community Edition.
Alex Desmond: Oh, sweet.
Chris Doman: Cado Community deploys into your cloud environment, so we don’t keep anyone’s data, we don’t want to do SaaS and stuff with forensics, for obvious reasons. And then, people can deploy that and it’ll process evidence. It won’t collect it though, but you can use other tools like Velociraptor, or KAPE, or something, and send it into Community Edition, and that will work.
Alex Desmond: That’s really, really cool. Yeah, still having that Community Edition.
Si Biles: You say that you don’t ship the collection tools with the Community Edition, that’s fair enough. So, you do have a wide range in the commercial product of collection tools, as well?
Chris Doman: Yeah, and that’s often the value of why people want to get the product in the first place. [inaudible 00:06:38] for things like containerized environments. Kubernetes, there’s five or six different ways of getting data from Kubernetes, Docker, containerd Systems, and the one you have to go for depends entirely on your security set up and all these other various things. So, that’s a hard problem to solve and that’s why that’s the core commercial product there. We get a lot of asks around that kind of stuff.
Si Biles: It is entirely your developed products, because nobody else has a tool for doing this?
Chris Doman: Yeah.
Si Biles: Or there may be other commercial tools for doing this, but there’s no open source solution for doing this, is probably the direction I’m heading in.
Chris Doman: I think, for so much like cloud forensics, there is the option of chaining together 20 different open source tools, which can work depending on who you are and what you want. But yeah, we had to write a bunch of our own stuff. We also had to do things like rewrite the Kubernetes API, just to speed things up, because it depends on how fast you want data. So, there’s lots of really interesting hard problems to solve, when you’re messing around with all these kinds of UAPIs. Which has made it quite fun for the engineering team to work it out that you have to do this stuff, but there are a few ways of getting this data that you don’t just have to use us.
Si Biles: So, purely from an open source perspective, Kubernetes is open source I believe, isn’t it? So, are you up streaming your rewrites, and your patches, and things like that?
Chris Doman: Some of it. I mean, for example, on the parsing side, we were using some of the plaster library, we upstreamed a couple of changes there, so we like to do open source where we can, A lot of it wouldn’t really make sense to you, either. I mean a lot of its specific for the way that we deploy and the way they all chain together, so some of that, I guess.
Si Biles: No, I just happen to think it’s really cool that companies give back into the community, so I’ll pick you up for that wonderful thing that you’re doing, and it is people like you that make things better from a real-world perspective, especially for forensic analysts. So, that’s a really cool thing to do, so thank you.
Chris Doman: Yeah, I mean, a lot of it is about, I mean, the mission is a bit of a cliche now, I haven’t talked about the mission, but the wider mission is to help people respond to incidents in the cloud, and that goes beyond open sourcing a few tools. There’s some training and stuff where we ship out for the community, as well, so try and make sure that everyone can improve, too.
Alex Desmond: I think before we started recording, in my mind I had cloud forensics as in you are pushing that service into the cloud, but you’re doing forensics for cloud services, so people would already have a cloud environment. So, I guess the initial compliance questions that I had might not necessarily apply, because they would’ve, may or may not, but they would’ve already solved it. But maybe you can talk about some of the interesting challenges you have, because I know we mentioned criminal cases with court admissibility, before we jumped on the call, but do you come across any of those kinds of things or is it more your clients are commercial, and they have a cloud environment and they’re going to want to do DFIR?
Chris Doman: Yeah, generally it’s more on the cybersecurity triage use cases. There’s a crypto miner in some environment, then you investigate it. But court admissibility does come up, particularly with the bigger banks and bigger healthcare providers we work with. If they’re healthcare under HIPAA, they have to do investigations, which makes it an interesting case.
For us, something that helps a lot is the fact that we deploy into their environments. And obviously, in the cloud you can choose what regions it is into. So if you’re storing a forensic data, you might end up putting it in Germany, if it’s like German data, for example, those kinds of things give you some more options.
And going to your earlier point, we do also do things like, you can collect a classic forensic disk image, upload to the cloud, then use our product to analyze it, too. It’s not the core use case, but we do have customers doing that, and that goes through more around some of those classic questions.
Alex Desmond: Do you have it as well with, do you have endpoint collected, or is the main product the parsing, and if they were running a hybrid environment, you would expect them to roll out something like Velociraptor or KAPE, and then push all that data into your platform?
Chris Doman: It’s very open, so there’s 1,000 ways of getting data in. So, we do collect data, and that’s the core value for a lot of those customers. Then we also process it and present it in a standard modern web UI. So you can collect with GRR or KAPE, and then we’ll happily process that.
Also, other kinds of commercials like Tanium or those kinds of EDRs, like SentinelOne and CrowdStrike. Actually, we can push out our one-off agent, collect that data and then suck it up the cloud. So, there’s lots of different ways of getting our data and sticking it all in one place. That’s part of the technical challenges, too, because you have to standardize that data, which it could be from a floppy disk image, it could be from a Kubernetes container. There’s so many things that could be from.
Alex Desmond: Yeah.
Si Biles: Yeah, normalizing data is a fascinating logical problem at the best of times, so yeah, the larger your potential data set sources, the harder it gets.
I mean obviously, there’s two models that you can go about for doing cloud, well, there’s probably more than two models, but essentially you’ve got cloud to cloud, or cloud to download and then process locally. Why did you choose cloud to cloud? And what are the advantages of doing it this way, versus just pulling it, and then running it through a standard forensic desktop thing?
Chris Doman: Yeah, I mean it depends a lot on what your architecture looks like, and how many cases you’re doing, how important speed is. Obviously, I’ve got my preference given that we went down this route. But some of the clear ones is you’re keeping the data in the cloud so everything’s a lot faster. You can go through a couple of clicks, get a copy of 100 EC2 virtual machines, or something like that, and that will just process through really quickly.
There’s a potentially lower CapEx cost, too. So you’re starting and stopping the actual processing, as well. So, you don’t need to have a bunch of servers continuously running off in a forensic lab somewhere. You can start and stop these, and then so long as you’re doing that in a sensible way, that works out a lot cheaper.
You also can avoid egress fees, although that’s not as much an issue as most people think it is. So AWS will charge you money if you download out of AWS, because they want you to keep the data in there. The same with Azure and Google are actually getting rid of some of that stuff, which is very good of Google. But it also means if it’s a really large amount of data, simply downloading to your laptop can take a long time. It can cost quite a lot of money, too.
And then, there’s some weird stuff around the way that the security works in the cloud with IAM and things like that, working cross account, between multiple accounts or multiple cloud providers. That this is the only approach that really works in terms of getting that data, because unless you’re actually deployed in that cloud, getting access to those resources is very difficult otherwise.
Alex Desmond: So, I noticed on the website, as well, with the product, the audit logs and investigation, so you’ve got, “Records comprehensive audit logs.” So, that’s for the investigators when they’re using it, is that the types of audit logs that you’re talking about?
Chris Doman: So, that was a requirement from a lot of banks, as you can imagine, where they need to confirm who had access to what data. Also lock down, which investigators can access which projects. So potentially, some investigations would be, you’ll know about this more than I do, more sensitive than others, in which case, you need make sure only some people access it, make sure that they’re doing appropriate things there, as well. It’s not something I love, the idea that you can be recording every analyst action, it’s something that you can turn on or off, but it’s just required for some requirements.
Alex Desmond: Yeah, yeah. I guess I could see this being used broader than cybersecurity incidents, like financial fraud, that kind of thing. And if you’ve got analysts working on that, as well, you want to be tracking that audit track. And also adds to court admissibility, if you can prove what an analyst has done then you can report a bit better.
Chris Doman: Yeah, exactly. And it’s all part of that. And also, I think particularly some of the requirements around things like insider threat, where what if they potentially even know the person there? So again, it is not part of our core use case, given more cybersecurity focus, but given it is a fairly open forensics platform, that’s what we’re using it for.
Alex Desmond: I think that’s really good, because I think that’s something that we’ll see as a trend. I know within Australia we’ve got, I think, Si, we’ve spoken about it before, but the SOCI Act within Australia, which requires critical infrastructure to have insider threat programs. So, I think across the world, we’ll be seeing this more and more where cybersecurity and that insider threat joins with the tools anyway. So good to see that, and having a feature before you really need it for all customers, I think it’ll just creep in to everyone.
Chris Doman: It’s all overlapping, too. Yeah.
Si Biles: Yeah, I mean from my perspective, we’d make extensive notes about everything that we do anyway, anything that automates that process saves me ink. So, I don’t find that necessarily a particularly off-putting concept.
You mentioned about scaling, actually, which is obviously, it’s one of the key, well no, it’s the key defining feature of the cloud, isn’t it? Is that you don’t want something, you turn it off, you do need it, you turn it on. You need more RAM, you download the more RAM, and instantaneously it’s there. Does your product contain some automation for that?
Chris Doman: Yeah, yeah. Again, quite an interesting problem to solve, to work out how to do that in the most efficient way. So, we’ll work out what kind of data you’re processing, how much of it is there, and when we make a plan of what workers to start. So, we have these workers which are other systems to start and stop. And then, we’ll chain them and layer them together, too.
So, there’s a lot of interesting problems there, about doing it the most efficient way possible, because maybe you want to have one worker per evidence item. Maybe you want to have one worker do multiple evidence items sequentially. And they also depend on the customer and their limits around how many workers they want to spin up for cost reasons, too.
So yeah, there’s a lot of logic in there. It’s an interesting one, if you like playing around with infrastructure, it’s a lot of fun. And then working out what the optimum setting is too, because obviously some of these virtual machines, or other worker types, they’ll have fast SSD disks, you have to work out the fastest way to use those and make sure the data’s being on the right kind of disks, too. So yeah, we do that and it is an interesting problem.
Si Biles: Fantastic. That is really cool. And in that regard, I should have structured these questions way better before we started. How are you delivering the service? I mean, this is actually probably a fundamental question we should have asked at the beginning is, are you selling SaaS? We talked about the open source version, that’s obviously a different kettle of fish, but I come to you, I go, “I run a large international bank. Help.” Where do we go from there? What’s this actually look like in the real world?
Chris Doman: So, the way this works is that we deploy into their environments. So, in AWS it’d be a cloud formation or Terraform, so infrastructure as code, similar. Terraform and Azure, Google Cloud, GovCloud, announced next week, so you heard it here first. And yeah, it all deploys as infrastructure, via a simple template into the customer’s environment. That makes a lot of things so much easier.
For one thing, when we were two people, I don’t think people were trusted with their forensic evidence, which is sensible. We’ve now gone through all the fund certification processes, but it makes it so much easier. We are looking at supporting a SaaS model, particularly for more mid-size businesses where they’re not evolved to be able to handle their own infrastructure, that’s still kind of TBC. But this all deploys into customer environments, which helps, too, because some of the really regulated customers, they don’t actually have that much in the way of cloud security tooling often. Which is kind of scary, but it’s because of those requirements. So they’re not allowed to deploy these cloud security tools, which is bent, because they’re all SaaS and they simply won’t allow them to do the things they need to do.
Si Biles: So, when you are deploying this, you are working hand in hand with the customer to develop these programmatic definitions of future deployment of equipment. So, you said that they want to keep costs down, therefore they’re only going to spin up one work note, and that’s something you are working with them to do, or is it something that they have to define themselves?
Chris Doman: I mean, they can ask us for advice, but it’s pretty simple, there’s just a setting in there. You can say minimum maximum size, you specify that there. And then there’s also default settings around things like the life cycle of storage. So, often you want to have this evidence preserved permanently, maybe in your NKZ-01, or more modern, ORD format.
And then, there’s all the ways in the cloud around how you can keep those costs down, where maybe after 30 days they automatically switch to cheaper storage, so long-term storage, and all these kinds of optimizations. So, by default we do all that, but then there’s some customization people can make to, and depending on the country and industry, they may have some particular requirements there.
Si Biles: So, I mean, that then sort of leads to the question of, and it’s a question we talk about a lot here, is what’s the training provision that you guys have for making sure that administrators aren’t spending their entire bank’s profit margin on AWS things, because they’ve set it wrong?
Chris Doman: Well, in terms of training and things like that, I mean, we go through and we have an admin session. We’ll say, “Here are the kind of things you’d be setting up. Here’s the best practices.” And then, there’s some safety catches too, to make sure that you can’t do anything silly like spin up 1,000 workers, or leave one on for a month, that’s not possible, it stops and informs you at that point.
Si Biles: Cool. And from the end user perspective, what’s your training program like?
Chris Doman: So, it’s pretty fun. So, we go through a classic Linux crypto mining scenario, so you have some detects into the cloud provider. You go through, they hacked into an Apache server that they installed. It’s always XMRig, if you’re doing investigations in the cloud.
Si Biles: It is.
Chris Doman: Yeah, it is, I know, 90% of the time. Those are the more fun ones out there too, but that’s the most common one. And then also, we have a more classic Windows case, where someone’s gone through a few systems and used power to deploy malware, to move laterally, that kind of classic stuff. But primarily, it is all Linux investigations most of the time on the cloud side, so that’s what the training reflects.
Alex Desmond: Do you have, I know you’ve got a small team there as well, but is there any support? So say a customer who’s trying to do an investigation, do you have any investigation support, or you just have maybe a trusted partner that you would just refer them to?
Chris Doman: So we’ve got a team of IR experts where they help us to do the work they would do manually, so we can automate that, they will provide advice sometimes. But yeah, generally working with partners though, so we work with a lot of IR providers and MSSPs, and then we’ll do that kind of sharing. Yeah.
Alex Desmond: Yeah, nice. And then, I had a question around, so your platform ingests a lot of data, not necessarily collects it. How do you guys go if it’s phone data, or an image of an iPad, or something like that? Is that something that the platform can handle? Noting that a lot of, I can think of a company in Australia that is purely all Mac, and they do a lot of work on their phones, so I can imagine that being a thing.
Chris Doman: Yeah, so people have put through images of mobile phones, it’s not something that I would suggest, we’re not built for that. I mean, there are fantastic tooling out there, like Magnet and Celebrate.
And then, we have support for things like plists, and other standard forensic artifacts, just because we try to support all the artifacts, it’s not like a default use case. We collect from Macs where we can do our one-off collection, like a triage collection. And that’s important, because particularly the tech companies, they’re all Mac, so that’s required for that.
And then also in the cloud, too, often customers actually have Macs running in the cloud, because that’s where they’ll have some build environments, so we need to make sure we support that, as well.
Alex Desmond: Yeah, nice. Even the triage would be great, because I am imagining you’ve got all this cloud data and you want to have timestamps for some of the stuff that’s happened on the endpoints. So, to lead in, rather than having a switch between, say a Magnet, or something like that, and then the single pane of glass that everyone likes, is always good to have.
Chris Doman: That works really well for some of the, in particular, the tech investigations we’re seeing customers use it for, where a developers’ laptop is compromised, then they then pivot to the cloud and they do some malicious stuff, and then you’ve got a combination there of the cloud provider logs, it might be like guard duty detections are more likely, cloud trail telling what APIs are called. And then you have some of the actual resources, so malicious containers or incidents. Plus also, the developers got a laptop and then seeing, “All right, the crap was stolen here, then this happened, then AWS picks up here.” And they’re always interesting when it’s like a hybrid investigation.
Alex Desmond: Yeah, it’s funny you mentioned developer, because exactly what I was thinking of. I was thinking of the Okta breach from the developer. And the fact that they would’ve had to link cloud authentication logs with the attacker and then try and find out where the creds are gone.
Chris Doman: I mean, they’re interesting ones, because it’s a whole new set of attacks, and I think people are still catching up there, and just the number of them is significant, too. So yeah, there’s a lot of breaches like that going on right now. Lots of stuff, too, where people are doing things like they’re emailing cloud formation templates to tech companies. So infrastructure’s code, so you deploy a virtual machine with your template, which does it, but they’re emailing people saying there from a cloud provider saying, “Install the security updates.” It’s classic phishing, but in the cloud, it spins up malicious stuff in their environment, it’s crazy. So there’s all these new sets of attacks, which are quite interesting to see.
Alex Desmond: That’s amazing.
Si Biles: Fantastically brilliant.
Alex Desmond: I watched a talk at Besides London on how people aren’t paying attention to their hybrid boundaries, between their on-prem and cloud environment. And they don’t treat their keys as securely as they treat their passwords, because it’s just all new, and there’s so much out there, and the whole IAM is really hard to wrap your head around.
Chris Doman: And it’s essentially, if you haven’t got the authentication security correct, it is like a flat network, because you can have a breach here instantaneously in seconds, 1,000 other accounts are breached. That couldn’t really happen in the same scale as the old days. Plus, people can actually spin up new servers and charge your company money. You couldn’t really do that, an attacker couldn’t go and install a server in the old days, probably, and then suddenly start billing you all that money. Whereas now that can totally happen. So the impact of breaches is…
Si Biles: Adds insult to injury, isn’t it? Paying for your own breach.
Chris Doman: Yeah, I know, and the cloud suppliers are nice, sometimes they give you a refund, but it depends on how it happened.
Alex Desmond: Yeah. Yeah, that’s true. To put you on the spot, and we might edit this out if you don’t have it, but is there any cool investigation story that you could, or cool feature that you’ve had to implement that you can share?
Chris Doman: Yeah, there’s been a couple. So one of the early ones that we worked on with a customer is manufacturing. So, it was interesting, because even though it’s not a core cloud use case, which is normally what we talk about, quite high impact. So, it was very important manufacturing stuff, leave it that. But then they had to quickly identify about 100 different systems, which one still had this back door after the ransomware had done some pretty nasty stuff.
So, they could collect 100 systems with us, across a dispersed geographical area. And then, through that they found this act was still hiding by persistence in the Windows registry in a couple of systems, so they had a back door packed in. Turned out they’ve been there for quite some time, too, which the EDR hadn’t picked up, because EDR is not forensic, so they missed that. So that was a really interesting one, just because of the impact of that, and then the scale. And because early on, too, we had to make sure we could scale to support processing hunt systems really fast, so we had to make a couple of tweaks for them there. So that was a really interesting one.
And then, yeah, I don’t know, I might be thinking of a couple more examples, it kind of depends.
Alex Desmond: Yeah, that one one was cool to listen to.
Si Biles: So, on an ongoing basis, I mean obviously, you’re doing development to manage the fact that AWS, and Azure, and Google, are all constantly updating and changing their stuff, but what’s your actual release cycle like? Are you agile and push stuff out, or do you have a one drop a quarter or how does it pan?
Chris Doman: Yeah, I mean, so we release two or three times a week. So John, our Director of Engineering, when he joined, early on, the first thing he said is, “Right, we’re going to kill the DORA metrics.” Which is basically around how often you release. It’s probably the strongest signal of a strong engineering team. So yeah, two or three times a week. We put a ton of time into automated testing, which is the only way you can do that, a stable way. So yeah, we just have an insane number of tests, data sets, as well, and that’s how we achieve that. So yeah, two or three times a week will we normally do updates.
Si Biles: And how does that push out to a client? Is it they choose to update, or will they just automatically reach out and pull it in, or they choose to automatically reach out and pull it in? Actually, and the other question, before you start, though, is generally speaking, how much effort is it to do, because it’s not going to rebuild the entire system. It’s a patch.
Chris Doman: Yeah, I mean, that’s actually quite an interesting one, because it changed early on. I thought naively, “People want auto updates. Everyone wants auto updates.” But absolutely not the case, particularly, imagine you’re in a forensic investigation and someone pushes out an update to you. You do not want that. Particularly, again, most sophisticated customers, they will want to have control over how they do these updates.
I mean, under the hood is actually pretty simple. We have a data disk, so we spin up a new system and switch the disk over with a bunch of checks to make sure it’s going to work before we do that. So, that’s how it works under the hood.
With the customer, basically click a button, it does the whole thing, takes 15 minutes, pretty fast. Or particularly, if they’re tech companies, they’ll manage everything through infrastructure’s code. So, they’ll have their Terraform code, normally in a system, they just update the base image in that. So one line change and they then push that, because that’s how they like to manage their infrastructure, they don’t want to use a button, they want to use code to manage any updates.
Si Biles: Cool.
Alex Desmond: So, talking about the development cycle, what’s exciting that you could share on the roadmap for you guys? Any exciting features that you can talk about that are coming out, or?
Chris Doman: Yeah, I mean, I’ll see if the marketing team hates me now for talking about it too early. But yeah, we’ve got a bunch of tech partnerships with some large cloud and EDR vendors we are about to announce probably over the next couple of weeks, so that’s pretty exciting. Some large IR providers, too. So that’s some stuff we’ll announce soon. They’ve been using our stuff a while, but we’re going to go in public on that. But in terms of the actual product, we’ve done infrastructure as a service for a very long time. So AWS is your Google cloud, but there’s some fun stuff going around SaaS, too. So many breaches are things like Office 365, email breaches. You do all this stuff, so you’ve seen a bunch of these, so there’s some stuff coming there pretty soon.
Alex Desmond: I was just reading before this, the top security predictions for 2024 and SaaS was on there. So is that the focus on certain SaaS, majorly used ones? So, what pops to mind is Atlassian is the major one that a lot of people work with, or is it going to be trying more generic of SaaS?
Chris Doman: I mean at first pretty big focus around back fraud, that kind of stuff. Businesses being compromised, because we’re working with a lot of MSSPs now, and some of their workloads are, like 80% of the investigations they do right now are just email compromise. So, not the most technically interesting ones sometimes, but very high impact and particularly for smaller businesses, that can be really, really damaging.
Si Biles: The last couple that I’ve worked on, ours have been email-based, and without appropriate tooling, it’s a nightmare.
Chris Doman: They don’t make it easy either, right? There’s undocumented APIs, there’s like 20 different APIs, all which give you slightly different things. So yeah, we’re hoping we can help there a bit.
Alex Desmond: Yeah, I always love that as an investigator being like, “I don’t know how to read these logs.” I’ll go to the company’s website and their website is nowhere near what their logs look like, and you’re like, “Great.” Time to test.
Chris Doman: And the samples don’t work.
Alex Desmond: Yeah, time to test to figure out what these logs actually do.
Chris Doman: Do you remember there was a bit of a furor about five years ago, because Microsoft had this undocumented API, but only certain IR persons knew how to use it? So, CrowdStrike was banned to do business email compromise cases, but most of the other small providers couldn’t do it, because they didn’t know about that secret API. And then, someone shared how to get it, which is great, and then Microsoft turned the API off, because it wasn’t designed to be used, so it’s interesting problems.
Alex Desmond: Yeah, that’s always fun when you find that kind of stuff.
Si Biles: Do you actually, from a developer perspective, how are you finding the cloud providers on their, Google being the one that is typically the people who just turn stuff off randomly and break everything for other people because they don’t care, how have you actually found it, developing with this as your major business line? How painful is it?
Chris Doman: Yeah, it’s been pretty good overall. I mean, as a start, they give you a bunch of fee credits, because they’re going to lock you in, but they’ve all been pretty friendly to work with. Biggest problem, I won’t name which cloud provider it was, they basically shut off access to start any virtual machines for everyone in the region, because they gave it to one large customer that had an urgent requirement. So, the idea of the cloud is infinitely scalable isn’t true, because one very important customer had a very urgent need to start a lot of machines, and then we had to then work around and make sure all of our test pipelines still worked.
Alex Desmond: Was that when Palworld was released? That’s not this recent thing, is it?
Chris Doman: No. What’s Palworld? What’s that?
Alex Desmond: It’s a huge game in the moment, but they essentially took up all the game servers within a whole bunch of regions.
Chris Doman: It’s not infinitely scalable, that’s the thing.
Si Biles: Well, no this is it.
Chris Doman: Actually, quite recently [inaudible 00:34:23].
Si Biles: So somebody, somewhere, actually has a room full of servers, and it’s not infinite. So, yeah.
Chris Doman: We’ve also breached…
Alex Desmond: It’s that T-shirt of, “The cloud is just someone else’s computer.”
Si Biles: Yeah, a friend of mine…
Chris Doman: It has to sit somewhere still.
Alex Desmond: Yeah.
Si Biles: Yeah. It is actually, I’m going to say, when I was working back in security more, it was like just replace the word cloud with somebody else’s computer and see if you’re still happy with the rest of the sentence. Your data is in somebody else’s computer. Yeah, well. So yeah, totally understand that.
Chris Doman: I still believe in the premise of the cloud. I mean, I would say that, because of what I do, but it definitely is, there are limits there though that the cloud providers could be more honest about.
Si Biles: Yeah, but you wouldn’t buy it if you knew that.
Chris Doman: That’s true.
Si Biles: Especially, if you’re a large government suddenly realizing that you can’t spin up extra machines because somebody with more money than you is doing it, is not something you want to advertise.
Chris Doman: You don’t see that in the larger regions, but in the smaller regions you see that more often. We’ve also breached the limits for some of our test sets before for certain resources, too, and they’ve had to make code changes on their end to make sure we could do up that stuff for testing, too. So its always a good sign when you started to hit the limits.
Si Biles: Yeah, yeah, certainly impressive. It’s something I was talking to my daughter about this morning, because I used to work, a long, long time ago, I used to work at the Rutherford Appleton Laboratories in the UK where we do our particle physics experiments and stuff. And we were running fraction of a second long experiments, and then getting five petabytes of data off the back of it, trying to figure out how on earth we manage this data set for science. But, yeah.
Chris Doman: Yeah, that’d be a fun problem.
Si Biles: It is always fun. Yeah. So, big tape robots at the time we didn’t have this big enough.
Chris Doman: How did the network work? I guess it was very local, right? You weren’t shipping that data?
Si Biles: Well, you say that, but no, we were pushing stuff backwards and forwards to CERN in Geneva. So we were, I mean, it was just slow. Do an experiment and then six weeks later we’d have all of the data, or some of the data, over to work on. Yeah, amazing stuff.
Alex Desmond: So, back onto to your product, I was just browsing through some of the features that you guys had. It’s main use cases, instant response, digital forensics, do you find it being used as a threat hunting tool, as well as, do you have integrations where you might ship alerts, or certain data, to scenes for the benefit of a SOCK, as well? How does that play into the whole infrastructure and environment?
Chris Doman: Yeah, I mean, we’re seeing that as a trend more and more. So I mean, we built this originally as more like a reactive tool, so you get a detection, we’ll go collect the data, analyze it, maybe automatically, particularly if it’s a container, they disappear every 30 minutes. If it’s an auto scaling group, they disappear. But we are seeing that more for proactive threat hunting now, which is quite interesting.
And yeah, exactly, it does get shipped off into their scene, or the data lake. So, we’ll export all this data, and it’s a lot of it, because we parse out every zip inside every file, inside every log, or whatever, etc. And then, yeah, then people do their own threat hunt queries. We’ve also had people do things like when Log4Shell, when that vulnerability, obviously in use quite a bit.
Alex Desmond: Everyone was searching for it, yeah.
Chris Doman: So, some people using our product to look for that in their cloud, because you can add YARA rules, or other things, too, for detections. And because there’s not an impact, which is the fun side of the cloud, too, is you make a snapshot of a system, it doesn’t even know you’ve done it, whereas in the old days you couldn’t do that. They’re doing that to scan key systems for the bomb.
Alex Desmond: Do you find, from my experience being in incident response and consulting, as well, do you find people using your tool for collecting a data inventory, or even an endpoint inventory?
Chris Doman: I mean, we were looking at that with a couple of customers, but because we go so deep, the breadth isn’t such a great fit. We have customers deploying us through other EDR agents, and then they use the EDR agent to see what the inventory is, and then we’ll go and suck up certain things to check what that system actually hits, because they don’t always know, and that kind of thing, so I guess we’re kind of the triage investigation tool there.
Alex Desmond: Yeah, okay. It’s always an interesting thing to talk to vendors about this, and even people who are trying to solve it, because I honestly don’t think, it’s in every plan and every framework is have an inventory, but I don’t think anyone has a good idea of how to do that well, and make it manageable and searchable, and easy to update. It’s just an Excel spreadsheet at the end of the day, is usually what I find.
Chris Doman: Yeah, and the fun thing about the cloud is that it changes every 10 minutes. So, as soon as you have that data, it is already expired. So, that’s why the cloud providers themselves haven’t got great inventories. Azure probably has the best, it has a built-in resource manager, but it still can’t show you everything. And AWS is trying to ship some stuff there, but it’s not very good, frankly.
Alex Desmond: And I can imagine, as soon as they start trying to do that, then it generates an insane amount of logs, and you can’t store that many logs that’s feasible to search through.
Chris Doman: Yeah, exactly. If you want to know what that is, if there’s a container that lasts five minutes and you’re spinning up 10,000 containers every five minutes, then shutting them down, by the time you have that data, it’s no longer valid anyway. And yeah, exactly, tons of data storage costs, so it’s an interesting problem that I look forward to seeing someone else solve.
Alex Desmond: Me too. Then we can get them on the podcast and they can explain how it is not an Excel spreadsheet.
Chris Doman: Yeah, well I guess it is on prem, right? Or it can be on prem, so it’s better than nothing.
Si Biles: Yeah, I’d sympathize, because it’s the first thing in the ISO 27001 Handbook, isn’t it? It’s like, “Have an inventory of your assets.” Yeah, not going to happen.
Chris Doman: Yeah, that doesn’t make sense as the concept in the cloud necessarily. Key assets probably, probably makes sense.
Alex Desmond: I’ve probably got one more question, because I found your blog resources very interesting to read, and how you simplify chain of custody. So is that just the logs that we were talking about before? Because I’m reading here, it’s got autonomous chain of custody, multi-cloud support, centralized preservation, and standard compliance, they’re the four points that it hits on. So, is there anything other than the auditing of investigator logs? And how else do you handle chain of custody, for evidence itself?
Chris Doman: Yeah, and with an expert witness on the line, I’ll be very careful what I say here. So, we do the standard stuff where we’re collecting the virtual machines, but we’ll hash the disk and SHA-256, and the other stuff, even before and at the end. And then, there’s standard tools requiring those disks, and then we record a log of who collected that data.
That isn’t possible with every resource type though. If it’s something like a container, there’s basically two ways of getting containers. One is you can actually get a snapshot, in which case we hash the full disk before and after, we parse out that data. But sometimes you have to get this data live, there’s just no other way, in which case, we’ll log what we can, but there are limits there.
And then, something else we do, as of about a month ago, is that we essentially preserve everything. So, this is a common customer ask in finance is that if they collect a system with our tool from on-premise laptops, from Google, Azure, AWS, they can then make sure the copy of that system all ends up in one place. So, one storage bucket in one cloud provider, it all gets copied across. And then, they have that there for as long as they want. And that’s part of an all-in-one forensics preservation piece that a lot of people are asking for.
Alex Desmond: Have you had any feedback from seeing any of those be challenged in court, and how successful that’s been? Or is that you just haven’t had visibility, because maybe customers have just taken that over and run with it themselves?
Chris Doman: We haven’t, and honestly, most of the things that we’re doing here will not see court. It’s cybersecurity breaches, ransomware jobs. I mean, there’ll be a court element, but this evidence is not going to be used to prosecute someone necessarily. It’ll be used for investigations. It’ll be shared with parties, though, so there might be a secondary IR firm confirming our findings, that kind of stuff, we see that a lot. But often, it won’t normally go to court, not this kind of thing. If we were doing mobile phone forensics, that would be a very common thing, but that’s not the road we went down.
Si Biles: In the regard of storing this, I mean are you encrypting it as part of your product, or is it just the encryption of the platform that’s being used that the customer decides to dictate?
Chris Doman: That’s the nice thing about cloud, encryption is actually super easy. So, they basically have four different core encryption options in each cloud provider. They can just pick one, and that depends on where the keys are stored, what encryption options, et cetera. So, that’s all done by the cloud provider natively, so we didn’t have to do much of that ourselves, which is good.
Si Biles: Excellent, good stuff. Are you coming to any of the conferences this year and putting out a stand?
Chris Doman: We’ll be at InfoSec. We do quite a lot with SANS, as well. So probably doing a couple of SANS conferences, as well. We normally do a couple of talks around cloud forensics at those things, too. So a few of those. Don’t think we’re doing Forensic Expo, though. I’ll mention that to Meg on the team, maybe we will then/
Si Biles: All right, well I mean if you do, I look forward to it. Forensic Focus will be there and it would be great to catch up with you in person, and come and look at the stand and stuff.
Chris Doman: I mean that would be good, it’s probably something we should be doing anyway, actually. And it’d be great to see you for coffee, I know you’ll both be in London soon, so maybe catch you around.
Si Biles: Yeah, no, that’d be brilliant. Well, again, thank you very much for joining us today. It has been fun and it is fascinating to hear about groundbreaking products put together in the last couple of years under strenuous circumstances and fantastic growth, and it sounds like you’re doing really well, so I’m excited to see where this is going to go in the future.
Thank you very much for joining us, listeners. You can find Forensic Focus on all places where you can find good podcasts, like Spotify, Apple Podcasts, YouTube, and also on our own website. There will be a transcript of this, which is a completely pointless statement, because if you’re reading the transcripts you know that, and if you’ve listened this far, you haven’t needed the transcript. But anyway, that will also be on the website, as will any useful links that we have discussed during this. I really must script these out at some point in future. Oh, dear.
Alex Desmond: No, it’s the spice of life.
Si Biles: Yeah, absolutely. Thank you very much, Chris. It’s been an absolute pleasure and we hope we’ll speak with you again in the future about some new inventions and new things, and how terrible Microsoft Azure or Google Cloud is, in terms of keeping their APIs stable, so it’ll be great.
Chris Doman: I look forward to it. Thank you.
Si Biles: Cheers, Desi.
Alex Desmond: Thanks all.
