Dr. Bradley Schatz presents a roadmap to what’s changed in digital forensics at DFRWS EU 2019.Bradley: OK, good afternoon. Thank you all for coming to this session. My name is Bradley Schatz, and today I would like to talk with you about a subject that I’ve spent the last decade or so focusing on, and that’s pretty much forensic acquisition: something that we, as forensic practitioners, all overlook to some degree and think is a done story.
We all know what a forensic image is, and there’s really not much new to learn in them. But a fair bit’s been going on in the meantime, so I thought I’d talk with you today about what’s changed in regard to forensic image formats, and the interplay between forensic imaging and the wider aspect of performing forensics in general.
I spend my time running an independent forensic company; we’ve just turned a decade this year, which is a nice thing. I have, for the best part of the last decade, spent my time doing civil litigation, criminal defence and prosecution work, as an independent computer forensic expert.
At the same time, I’ve continued to do research in the area off the back of the PhD I did starting in 2003. My continued interest in research has led me to continue to be heavily involved with DFRWS, where I’m the chair of the US conference this year; I’m helping organise the Australian version of DFRWS, which is going to be 2020; so if any of you are looking for a good excuse to get to Australia, it will be a good opportunity – which is where I’m from, by the way.
So some of the things I’ve been researching over the last decade or so: the focus of my PhD was in representation, in bringing together disparate sources of forensic evidence and fusing them together. A lot of the research I’ve done on forensic imaging since has actually been influenced by that work I did during my PhD. Along the way, I got the Volatility framework working and usable on Vista and Windows 7 back in 2010, and you might find some source code that I’ve written in Autopsy.
But most of what I’ve been doing for the last decade or so has been AFF4, so that’s the Advanced Forensic Format version 4.
What I wanted to start with today was to talk about what the widespread challenges are that we’ve got in regard to forensic imaging in general; and then break into some of the more focused bits of research that I’ve done. Some of them you’ll see in the productisation work I’ve been doing in Evimetry, which is based on AFF4, so I’ll be talking a little bit about that as we go. But I’ll be touching on subjects like SSD, NVMe, full-disk encryption, logical imaging, and dealing with locked devices and forensic jailbreaking, as we go through; the common thread being forensic acquisition.
To my mind, the firmest theory that we’ve got around what we do with acquisition is in the physical area. We’re all pretty comfortable dealing with E01s and RAW; we’re pretty confident about the idea, and the abstract idea, of what a forensic image is – being a linear bitstream copy that’s complete, that’s protected by a hash. That’s pretty much it.
The problem that we’ve got, though, with forensic images in general these days, is that they’re not particularly expressive in terms of how we can represent wider evidential aspects of the hard drives that we’re copying. So for example, if we’re reusing an E01 to represent volatile memory, how do we represent the discontinuities that exist in volatile memory? Unlike a disk, where you go from the first block to the end and there’s evidence in every block, in volatile memory there’s big holes everywhere. So how do we represent those? We don’t really have a good way of representing those in a raw image or an E01.
Similarly, we don’t have a good way of representing the streams, or the parts of an image, that are a DCO or an HPA. In newer versions of E01s, we might be able to some degree to specify which parts of the image might correspond to a read error, but not in a particularly interoperable way.
So what we’ve had over time with our physical image formats is a format that represents that stream image that’s got the hash over it; but then a lot of the information that is surrounding it, that is of forensic relevance, like smart data etc., tends to be outside of the scope of forensic tools; outside of interoperability, etc. And that all comes down to, really we don’t have very good metadata storage in the forensic image.
Another issue with E01s and raw is that they’re slow, which I’ll get onto as well.
A lot of the work I’ve done with the Advanced Forensic Format version 4 has been around these speed issues; it’s been around dealing with these metadata issues. How do we come up with a representation that lets us store as much different evidence as we can in one evidence container?
If we step forward, the next thing that we focused on in forensics was logical imaging: the idea of storing many bitstreams that are protected by hashes, but usually those only being the size of a file, so it’s a much smaller granularity we’re dealing with.
As we’ve gone up the stack – we’ve gone from the physical to the logical – we’ve gotten to a point where there’s even less interoperability between the tools. So these days, if we look across the mobile forensics landscape and the disk forensics landscape, there’s really not a unifying format there that’s generally around, so people do tend to spend a lot of time converting from one format to another.
And when we do that, we lose data. So we might see L01s, AD1s; in the mobile space we’ll see TGZs and ZIPs fairly regularly. Every time that we have a new format like this, we lose on a chance to have interoperability.
And in general, when we adopt these types of formats, we’re usually losing some sort of file system metadata behind. One that I’ve found in the mobile space, that I’ve really found quite crucial in a number of matters, has been the birth time of a file. We don’t generally tend to see that preserved.
In the logical space, I’ve been doing some work recently on defining a refinement to the AFF4 that supports logical images, and I’m going to be presenting that at DFRWS USA later this year. We’ve got a Python implementation of that standard actually published on the AFF4 Github; I’ll go onto that in a little bit.
Looking to the future: I think really we need to move to an area where we’re starting to go more granular again. We’ve gone from physical to logical; I think really we need to start talking about sub-file imaging, or imaging data structures specifically. This hasn’t really been an issue in the past, when we’ve had access on disk to containers like EDBs and PSTs: these databases that contain emails, or database records.
But as we move to the cloud, we don’t have access to the container. We’re having to go through APIs that are still giving us access to the records, but the question then becomes: if I’m pulling these records down out of a cloud source, how am I going to store them in a way that is forensically sound, and is interoperable between tools?
The main one that I’m thinking of at the moment would be something like Office 365, where we’ve got access to the records underlying emails, but really the most convenient way of conserving those at the moment would be to store them in an MSG, or to create a new PST. My question is, moving forward, should we be thinking about something that stores records? It’s convenient in the case of 365 that we’ve got a storage option that exactly matches, but what if we’re talking about the Twitter API? We’re talking about the LinkedIn API, or any of the other hundreds of APIs that we could be dealing with?
The problem here of taking the approach that we’ve had with AFF4 to trying to shoehorn sub-file storage – we don’t even have a name to use for it at the moment – is that once you get over a certain level, the metadata information that we’re using for AFF4 starts to become a little bit cumbersome: it takes too much time to load, time to search. So that’s kind of a future issue.
Going back to the work that I’ve done on physical acquisition: I started out with a question a few years ago, looking at the methodology of computer forensics, and thought about, why is it that we’re taking so long to get from the acquisition to the information point?
And the initial question I asked was, why can’t we just make acquisition go faster? What’s slowing us down?
It turns out that we need to consider that question on a couple of sides. If we look at these results here, we can see what sort of speeds we’re getting with our generations of storage. The key point that we’ve got here is that new generation storage is very fast. So whereas we’re dealing with gigabit ethernet going at 100Mb/S, and it’s roughly as fast as a 1TB drive spinning disk a couple of years ago. Spinning disks might go at 200Mb/S these days, in general. That’s still very slow.
Once we get to NVMe and SSD, we start getting these 100s, 1000s, etc. And this goes equally for high-end storage like RAIDs, etc.
When we start trying to apply existing forensic imaging technology, like E01s and raw – E01s specifically – to fast storage, we find that we’re actually limited first by the implementation, and then secondly by where we’re actually going to store the information.
The main issue with E01 is that it uses the deflate algorithm to do compression. And the deflate algorithm is actually extremely expensive CPU-wise. So for example, on an 8-core i7, that’s a couple of years old, the maximum speed that we can get is 255Mb/S. Not a particularly big deal if you’re dealing with a hard drive that can only pump data at 200Mb/s, i.e. one spinning disk; but the minute that you start dealing with a RAID that’s got 10TB, that can start pumping data at 1,000Mb/s; or an NVMe that can pump data at 3,500Mb/s – albeit that it’s all zeroes, generally you might get them going at about 1,500Mb/s with real data – you’re losing an opportunity there to actually get into the field and out of the field; to actually get your equipment freed up so that you can actually do something else.
So to solve this particular problem, I ended up falling back on the AFF4 research – that’s the Advanced Forensic Format version 4 research – that I did with Michael Cohen, who you might know from Recall, and what was his other Google work? Anyway. Also, with Simpson Garfinkel who invented AFF.
We identified AFF as a next-generation forensic storage container in 2009. Didn’t really do much with it until I came along in 2005 and tried to solve this former problem.
What we did with AFF4 was say, why don’t we add a virtualisation layer to it that lets us, instead of just storing linear blocks of compressed data and a hash, why don’t we add a virtualisation layer to it firstly, that lets us glue data into all sorts of different shapes? Now in terms of the virtualisation layer in AFF4, what it lets us do is interesting things like, take blocks of zeroes and represent them symbolically. Much like in Unix we’ve got a dev0, in AFF4 we’ve got this synthetic block stream.
So we can say in a really concise form; in a map… another way of thinking of it is like a virtual memory, to some degree. We’ve got this map; it represents the address space of the source hard drive. We can take areas of that source hard drive and then say, that’s actually represented by this virtual block stream of all zeroes, or a virtual block stream of all FFs, which you find in, say, nand. Or you can point a particular block at a compressed block that we’re storing in the compressed block stream.
What this means is that we can actually not have to store runs of zeroes, that’s one advantage. When you’re acquiring using E01s, and you’ve got a really large hard drive that’s a spinning disk, and half of it’s full of all zeroes, you’re still actually having to run all of those zeroes through the deflate compression algorithm, which takes a lot of time and CPU resources, and it slows you down. Or if you’re using raw, which is even worse, you’re just spending your time copying zeroes back and forward, again a very slow thing to do. This lets us have very significant compression of sparse, or predominantly unallocated, zeroed-out sectors.
So AFF4, I think the biggest advance that came along with AFF4 was this definition of the virtual blockstream. The other thing that we integrated in it which has proved to be very useful is a metadata scheme that allowed us to represent arbitrary metadata. So if a new type of evidence storage comes along and you want to represent something new, you can always just create your own representation to go in it and express it, and standardise on it later on.
The metadata storage scheme that we used is actually based on RDF. We used a serialisation called Turtle, but it’s the same representation that’s being adopted today by the CASE community, so I think there’s some really interesting opportunities there for CASE to work with AFF4 moving forward.
How fast can we go? What are the interesting effects around using NVMe and SSD?
This graph here is interesting, I wanted to show you this. This is a graph of three different acquisitions that we’ve done of a 1TB NVMe drive. The first acquisition we’ve done is where the drive was completely empty. And you can see in the red here that it’s taken us five minutes to acquire a terabyte, and we’ve gone pretty much almost 3,500Mb.s.
An interesting point to this is that when the drive is empty, it’s all zeroes. So going back to what I was saying earlier about not compressing the zeroes, and storing them in a really efficient way, it’s taking very little CPU effort to store all of the 1TB-worth of zeroes in our image, and actually the image, when we store it on the disk, is kilobytes in size: it’s super small, because it’s all virtual.
What we’ve then done is, we’ve formatted the drive with NTFS – we’ve installed Windows on the drive, it’s been formatted NTFS – and we’ve filled the drive with the govdocs corpus to the point where it’s about 40% full, and then we’ve reacquired it.
You’ll notice here that we’ve been going at more than 1,000Mb/s. So the challenge here, with this acquisition, is that for us to even go at 1,000Mb/s when we’re doing our acquisition, we have to have something we can store the image on that can go at that speed. So recalling that a hard drive – a regular spinning disk – only stores information at 200Mb/s, we’re going to be limited to 200Mb/s if we were just storing to one drive.
So what we’ve done with AFF4 is, because we’ve got that virtualisation layer in it – because we’ve got that map layer – we can actually use that to store information outside of one container and refer to another container. So we can do things like have a RAID-like application within the forensic image itself, and use the combined bandwidth of multiple hard drives to get to the speeds that we need.
So in this particular instance here, to get to the speed that we need I’ve had to image in parallel to four SSDs, each of which had 50 to 400Mb/s worth of bandwidth. Then we’ve got the 1,000Mb/s worth of bandwidth to actually keep up.
When you look at the green line, you can see that the speed that we’re going changes a fair bit, and that really changes based on the entropy of the data that we’re acquiring. So for this bit here, we’re going through all of the Windows operating files and the govdocs corpus. We get our worst performance here – I think we might have thrown in an encrypted file, as well – I think our compression performs worst on encrypted data, so that’s why there’s a drop here. Then we get to the end of the drive again, where it’s all zeroes, and suddenly we start going at that very high speed again.
I can’t confirm this, but I suspect that with these speeds we’re getting up here at the top, in the implementation of these SSDs we’re dealing with some kind of flash translation layer. I suspect that if all of these zeroes that this drive is pumping were all actually stored on the flash, we wouldn’t be going this fast. I don’t think the flash has the bandwidth to actually read all of those zeroes from the flash and then pump them out. I suspect that they’re actually being – similar to how we’re storing all of the runs of zeroes – I suspect that they’re doing a similar thing internally, in the flash translation layer of the drive.
Finally, we’ve filled the drive up to 95% utilised – that’s the blue line here – and we’ve managed to basically go at about 1200 or so Mb/s, to get the full thing acquired.
So to explain in a little bit more detail what I was saying before about how we get that aggregate IO, really all we’ve done is we’ve taken the virtual blockstream that we’ve defined in the AFF4; we’ve got our compressed block streams, where we’re storing the data, and what we do is, when we read a block we compress it; we hash it; and we send it to the first container that’s available. So in this case, we’d send it to here – let’s say it’s the MBR, the start of the disk. We’d store the data of the MBR here, we’d put a map entry in here that says LBA0 is stored in here. We’d have a corresponding map over here that says for LBA0, go over here. And then we just keep spraying them back and forward to keep the data flow going.
In working my way through all of these problems to get data flowing as well as I could, we went through a lot of hardware. This is a bit of a summary of what kind of speeds we could get out of various USB3 bridges that we’ve stuck over the time that we’ve been playing with this stuff.
Largely, all of it’s been done with a Samsung 850 PRO SSD, which is a SATA SSD. We’ve also been using a Samsung T5, which uses mSATA internally. But interesting point here: if you’re using USB3 as an interconnect, or as a bridge, to store data or to read data, it could actually be a bottleneck in your forensic workflow. So if you’re doing things like imaging SSDs, that could be bottleneck.
Old versions of USB3, that could be USB3.0 back in 2015, 2014: the microcontroller in the USB bridge was actually a problem. It was slowing things down to 200Mb/s even though we had a hard drive that was capable of pumping data at 500Mb/s.
UASP came along. That effectively doubled what we were able to get, getting us to about 400Mb/s. If we’re using write-blockers like Tableaus – this is some testing we did last year – they tend to slow us down too, to about 270Mb/s.
So another aspect that we looked at was the other part of the workflow, that says acquisition comes before analysis. So the question we asked was, why? Why can’t we do both at the same time? And it turns out that the reason we can’t do both at the same time is the linear physical forensic image is what stops us. If we shift away from doing a linear image to doing a non-linear image – which is something that the virtualisation layer of AFF4 allows us to do – then we can start doing things like having the analysis drive our path forward while we do acquisition.
So for example, if we’ve got a source hard drive here, we can choose the path that we fly over the hard drive as we’re doing our analysis. So we can read the MBR, we can go and read the GPT, then knowing where the volumes are we can read file system metadata; knowing where file system metadata is we can know the topography of the drive, and we can choose which files we want to acquire in any order that we want.
We can influence that by profile – by saying we want all the high-value picture documents first, followed by the word documents, followed by all of allocated, followed by all of unallocated. At the same time, we can inject live analysis into that and we can share it as a virtual file system or a virtual file, and start having a look at that with EnCase or your regular toolset.
But you need a non-linear image in order to be able to do that. You need to, as you are reading from the drive, be storing the blocks as you actually go. Otherwise you just end up spending all your time re-reading from the drive; you have the different processes competing with each other.
So in this instance, what you can see here is we’ve got the MBR being stored here. We read the GPR next, it’s being stored here but it’s still in the virtual address space pointing back to here, and so on.
One thing I won’t go into is the effect of hashing on this scheme. If you really want to dig into AFF4 and how we handle hashing, we take care of all of that. We’re storing block hashes for every tree that we read. We’ve got a Merkel tree base construction that still gives us a single hash that protects everything. That’s all taken care of – it was published back in 2015.
Moving on. Using that approach of acquiring at the same time as we’re analysing, what I’ve been demoing outside here, and will continue to do so, has been an implementation of that. What I’m showing here is some test results of an old workflow that was being used in a lab, where they had a central high-end storage using gigabit ethernet as a transport, using Windows file sharing. What they were doing was using X-Ways, which is a great tool, one of the fastest E01 imagers I know. They were using X-Ways to image a drive, to a Windows file share on a very large san, over 1gig e.
And this is how long it was taking for this 1TB drive to acquire, and then to verify, then they were doing some processing to get to the end. If you upped the transport to being 10GB ethernet, for this particular bit they actually made it go a little bit longer, I don’t know why, but it did speed it up considerably, and the processing stayed pretty similar.
Using our implementation that allows us to do the acquisition and the processing at the same time, basically we were able to do the acquisition in this amount of time, the verification in this amount of time, and then doing that in parallel.
One of the upsides of using AFF4 – one of the ways that we solved that heavyweight compression problem – was to use a very lightweight compression called NAPI. It’s basically meant that, rather than the verification and the compression being a CPU-dominant process, it’s actually shifted it to an io-dominant. So given a really fast image storage pool in the centre of your network, you can throw as many CPUs at it as you want and it tends to mean that you can verify it extremely quickly. So that’s why verification of that image took so short.
So applying these techniques to full disk encryption I found kind of interesting. Back when SSDs were launched I went out and spent a ridiculous amount of money and threw one into my laptop at the time, and then BitLockered it, and then over the next six months my laptop got slower and slower and slower, and then it failed.
At that point in time, there was no conversation about the idea of overprovisioning of SSDs, of flash. Nor was there any concept of the idea of trim. So I had an SSD that didn’t do trim, and I ran that thing probably 90% full for six months – I basically burned it out.
And when we look at a full disk volume, what we see if we’re looking at the file level view of it, generally we see our files; sparse areas that are generally empty; when we look, we’ll see them, they’ll show up as encrypted, as high-entropy data. Our regular data in the system will show up as regular files.
If we look down at the actual hard drive, we kind of see the inverse, to some degree. Where we’ve got files, there’s high entropy data. But on SSDs these days, where we’ve got the empty areas, these are the areas of the file system that are unallocated, they’ll actually show up down here in the physical layer, as an SSD or an NVMe, actually empty: as all zeroes.
It didn’t work that way with that original drive that it was dealing with. What’s happened in the meantime is that the creators of full disk encryption – BitLocker and FileVault – they’re actually clever enough now to take the unallocated areas of the encrypted volume and then turn them into a trim demand when they go down to the actual physical layer. So that then allows the drive itself to know, OK, this bit isn’t being used anymore; we recycle it; we get a more efficient drive; the drive lasts longer.
The upshot of this… I’ve got a picture here… if you look at your raw image drive, the areas that are empty are all zeroes.
Interesting point here is that, if you do an image of the decrypted volume, the unallocated areas of your drive will actually show up with this kind of pattern over here. And this is because the areas that are all zeroes are all still going back up through the AES encryption, and that’s what it gets turned into, zeroes.
So the question then is: which do we acquire? Because, going back to what I was saying earlier, if you’ve got an encrypted volume and half of it’s empty from the perspective of the file system, you’ll get a situation where that half-empty section is down at this layer going to be all zeroes; but up at this layer, the entry layer is going to be all high-entropy data.
So what you get is, if you’ve got a 1TB SSD that’s encrypted and 50% full, if you do it down at this layer you’ll end up with an image that’s about 500MB; but if you do an image up at this layer of the decrypted version you’ll have an image that’s about 1000MB; about a terabyte, because of this transformation of the sparse areas back into high entropy data.
So which do we acquire? Physical? It’s a good question. If we get just the physical, what if we didn’t get the right key? We’ve got a smaller image, but we can’t use it.
I’m really not here to give you any recommendations; I’m mainly drawing this out because I found it interesting how full-disk encryption interacts with trim. Personally, I think that it would be quite useful to integrate storage of the relevant keys for FileVault or BitLocker in the forensic image format itself, so that when we do the imaging we verify whether or not we can decrypt the volume, and then we take an image of the lower volume – the physical – but actually have the keys alongside it. It may, though, be a safer thing – and I would generally take the choice of doing both, but that’s going to take quite a while in the field. It’s a bit of a balancing question there.
Logical imaging. Going back to the problems we were talking about earlier with logical imaging: we were recently asked to formalise a proposal for AFF4 for storing logical images. So with the WinPmem volatile memory imager, Michael Cohen had built into it the ability to (1) do an acquisition of RAM of Windows; but at the same time he’d implemented a partial implementation of storing a logical image. So what he was doing was storing things like the DLLs that were mapped in; the drivers that were mapped in, etc.
So what I’ve been trying to achieve here with the logical imaging portion of AFF4 is to create a logical image format that can store not only the file streams that we’re interested in, and the file names and the file paths that we’re interested in, but also any metadata that’s associated with those.
So one thing you may not be aware of with AFF4 is that it’s based on the zip64 compression format, so in its lowest layers it’s actually a zip file. All of those other abstractions that I was talking about are built on files stored within zip64. What we’re trying to achieve here is the ability to create an AFF4 volume that is used to store a bunch of logical files and be able to open up that zip file and give that zip file to a non-forensic person, and then still be able to open up the zip file with 7zip or WinRar or whatever, and be able to look through it and still see that they can click through a folder called AFF4-L and see a Unicode-named file, or see another zip file and extract it, or see a text file and click on it and read it. That way, assisting with providing evidence to non-forensic people.
Here’s an example using 7Zip of one of the logical images that we’ve created. As you can see from the prior slide that I presented, you can see the zips that have been stored, you can see the Unicode file name. We’re still working through how to make sure that these file path mappings that we can see here are actually fully interoperable based on operating systems. You’ll notice the double slash here that’s in the zip file; that’s not particularly good for extracting out into a file system, but it does mirror and account for the naming of Windows UNC network paths, etc. Kind of like the file:// URL that you would see on web browsers.
So we’ve implemented that and published the implementation on Github. There’s a paper coming out that I’m going to be presenting later this year at DFRWS USA. On top of this, we’ve done some work for NIST, who are doing work with the NSRL where they’re trying to deal with software as a service. So they’re doing snapshots of STEAM games on a daily basis.
They’re very large; take a lot of storage; so they were looking for a logical image that they could use to store those games on a regular basis, but they also wanted the ability to have it deduplicated. So what that could also contain is an implementation of a deduplicated logical forensic image implementation using AFF4 that compresses the different versions of the files that they’re ingesting into the image at about a 50% reuse.
And the interesting bit about the work that we’ve done with AFF4 logical and the deduplication part that we’re doing: all of this doesn’t actually need us to change any of the fundamentals of the AFF4 format that we developed back in 2009. It’s really mainly changes in the conventions of how we’re using it and we’re inventing new names for things, etc. So we haven’t had to change it all that much in the meantime.
Where I can see this logical work being useful moving forward is as we start to do more work in the cloud, with cloud APIs, trying to preserve the output of those into something. So I’m putting forward AFF4-L as something to address logical imaging as we move forward into the cloud and as something to interoperate with.
It’s my opinion that we’ve gone from a world 15 years ago where computer forensics – and I think Simpson Garfinkel referred to it as the ‘golden age’ of computer forensics – where we were relying on really standard interfaces to our storage, to get access to it. I think we’re moving to a world now where exploitation is going to be a big part of forensics: dealing with interfaces that are locking up data.
One of the bugbears for me as a practitioner is dealing with iPhones. Many of you would be law enforcement, I imagine; you might have access to things like GrayKey, or the equivalent from another provider. But if you’ve been in practice for the last five years or so, you’ll have noticed that we’re regularly being locked out of iPhones. The backups that we’re getting out of them don’t contain important bits of evidence like email; we don’t get the well files out of the SQLite uses to write, there’s always some great data in there of deleted records; and then there’s just swathes of the file system that we just don’t get as well in backups.
So in the government space, people are getting help from people like GrayKey. In my experience, in the civil space I’ve not been able to get people to unlock iPhones for me, so that’s led me over the last number of years to be turning to using jailbreaking to get into the phone to get the data that I want.
That regularly involves people giving me an iPhone and me saying, well OK, that’s going to go and sit in my evidence locker for six months or so until the jailbreak has come up that will deal with your version of iPhone, at which point I then need to download a jailbreak from an uncertain source – an uncertain provenance; acquire myself a phone that’s the same operating system version as the phone I’ve had sitting around for six months; test the effects of the jailbreak on the phone that I’m doing my testing with. In a number of instances it’s involved also, as I’ve run the jailbreak on the phone, it reaching out to an unknown website to download more information onto the phone, the provenance of which I don’t know. It’s a real can of worms, and I don’t know… in a civil context, personally I’m comfortable with doing it but I don’t like it, and it’s certainly not where we want to go with forensics.
In a criminal case, yeah, I can’t imagine anyone doing it. Anyone in criminal prosecution or defence use jailbreaks? No? Anyone use jailbreaks?
I think ultimately the issues that we’ve got with jailbreaks – well actually, I’m skipping ahead.
What questions do we want to answer with jailbreaking? Well a big one we’ve got is, has my phone been jailbroken? Has my phone been compromised by someone? Can I get my deleted text messages back? What time was a voice message first recorded? And then generally, what other things do we want to get out of them? The deleted data recovery, and other information. There’s a lot of useful stuff there when you start digging in.
Where are we at currently with the concept of jailbreaking in forensic sat the moment? It’s still a relatively young field at least in the iOS space. Elcomsoft have done a little bit of work there, they’re suggesting the use of particular jailbreaks on particular versions of iOS followed by their software toolkit to extract out the information. Sarah Edwards has been doing some work in this space as well, with particular jailbreaks. And I think it’s actually good that we have forensic people considering the use of jailbreaks and looking at what the effects are, and proposing methodologies for the use of it. I’m not here to say today that we shouldn’t be using them – I think it’s going to be necessary in the future.
What I am here to say is that I think we really need to think about where our jailbreaks are coming from and we need to evaluate what effects they are going to have on the phones that we’re dealing with, with reference, for example, to the ACPO principles.
So what does jailbreaking at the moment on iOS look like? Generally it involves downloading a jailbreak from the internet. To download the jailbreak from the internet you need to know what version of operating system you’re using, what version of iOS you’re running on the phone. Also, it helps to know what CPU is running on the phone as well.
Install and run the jailbreak on the phone: well, you’re going to want to test that first, you wouldn’t just want to run it blindly. Generally, the next step is to install SSHD, get an SSH server on the phone using Cydia. Cydia is a package manager that’s used on jailbroken phones to download all sorts of software. This includes things like SSHD.
And then ultimately it ends up using SSH to get a shell on the phone, and then execute some code on the phone, or use SSH to get some stuff off it, use SCP.
So how do jailbreaks work? This diagram here is taken to some degree based on Jonathan Levin’s *OS Internals volume 3. Good read, I can recommend it. Generally the jailbreaks we are using today are deployed as a single iOS app which you load onto the phone using a method called sideloading. There’s a tool called Impactor which you can use to take your downloaded jailbreak and load it onto the phone. Then you’ve got the app on your phone, and you click the app on your phone and you run it.
When you run it, the first thing it needs to do is… we’re running it here. First thing it needs to do is escape Apple’s sandboxing. Apple have been doing a lot of work to harden the environment inside iOS, applying things like sandboxing. They’ve got an entitlements model as well, that also limits what running codes can do, which APIs it can call itself, etc.
So first it needs to escape the sandbox. It needs to gain high privilege, needs to stop being a user process and be running as root, for example. And beyond running as root, it needs to have entitlements to do other things. Then generally it needs to have some other way to read kernel memory and patch the kernel, to do a number of those things. Very complex things.
Ultimately, once it’s got god mode on the phone – it’s got the ability to run as root – then generally it’s going to remount the file system as read/write, so big changes there; extract a bunch of binaries out into the file system; allow people to run Cydia; install packages, etc.; install services; modify services on the phone; patch processes; inject code into a number of the processes on the phone to allow things to continue to work.
Now this varies significantly from jailbreak to jailbreak, and is constantly changing with every new version of iOS: they’re having to jump through new hoops.
So how do you tell if an iPhone is jailbroken? One of the signs would be that you can get SSH shell to it. A number of jailbreaks will actually bind a bash shell to just a single TCP port so that you can netcat to it, and then you’ll get a shell that way. Other jailbreaks might take the AFC conduit, they might get root privileges so it’s no longer sandbox and you can get a view of the full file system. I’m not really aware of any particular place where anyone from a forensic perspective has gone through and enumerated the signs that you might have from the outside, the black box perspective: figure out whether or not an iPhone has been compromised. There are some pieces of software out there in the forensic space that are trying to address that.
Looking at it once you’ve got access to the phone, though, where are these jailbreaks changing things? Generally they’ll install Cydia, that’ll be under Applications. You’ll generally find a lot of Unix commands, that’ll be in a user bin, so you might find bash there, etc. A lot of those things will get put into there. Additionally, Cydia will put things into these paths. Plists around, indicating that Cydia was installed, provisioning profiles are used, etc. There’s a lot of changes.
Here’s an example of a provisioning profile that was left behind on my phone when I did a jailbreak of it on November 11 last year.
What are the risks of jailbreaking? One that I’ve highlighted as I’ve been speaking earlier is that issue of the uncertain provenance of the jailbreak and the third-party libraries that come with them. So there’s a bit of a supply chain issue there, in that at least at the moment – and this has been for a couple of years – there’s been a very active open source jailbreak development scene thanks to Ian Bier at Google, he kind of opened that area up.
There’s a number of people working on jailbreaks; they tend to share a lot of code, share a lot of binaries, and those tend to be flowing through that supply chain and showing up in different jailbreaks.
Another risk that we’ve got is that some jailbreaks – one of the primary focuses that sent me off into looking in this direction was that I was asked by someone whether or not their phone had been compromised, but there was only one jailbreak for that version of the phone that I was looking at, and I couldn’t really use that jailbreak to get access to the phone to tell whether it had been jailbroken, because that would have overwritten all of the traces of the former jailbreak. So that’s a risk.
Arguments re. forensic soundness: depending upon who you’re dealing with, people may wave their hands around a lot and generally talk about how you’re violating the integrity of the evidence. Obviously what we’re wanting to do here is understand exactly how we’re changing the evidence.
Earlier versions of jailbreaks used to do some interesting things in terms of file system layout, and if you looked at all the timestamps of all the files on a partition you’d find that there had been significant timestamp overriding that corresponded to the copying of a large swathe of the operating system from one partition to another, and you’d lose a lot there in doing the jailbreaking. That used to be called stashing.
Then I have seen some mention of people doing partition resizing as well, in doing jailbreaking.
So with all of this in mind for the work that I did, I wanted to build a prototype to test whether or not this iPad that I had had actually been jailbroken before. What I identified as the goals for a forensic jailbreak was that it should minimise the changes to a file system; it shouldn’t overwrite traces left by existing jailbreaks; it shouldn’t collide with any of the TCP ports which were typically used by jailbreaks to open up netcat sessions or to open up SSH sessions; and it certainly shouldn’t remount the root file system as read-write.
So the theory that I came up with was to take an existing jailbreak and limit the amount of binaries that were going to be installed on the machine, and use the SSH server – part of the jailbreak – to do all of the ifile copying and enumeration that I wanted to get out of the phone
Basically what I did was approach a jailbreak author who had worked on this particular version of the iOS operating system and CPU combination, and wI worked with him to take his existing closed source publicly provided jailbreak and modify it so that the jailbreak would minimise its effect on the machine. And what we came up with was an executable that did the regular bit of jailbreaking and extracted out dropbear, which is one of the SSH servers that’s used in jailbreaking, and we then mounted a disc image on the phone that contained a number of other bits and pieces that we needed. The reason we did that was that then we only had one file that was being put onto the machine, rather than a whole bunch of files. That was one way of limiting the number of iNodes that we were modifying. We shared the bash shell on port 44 and then we spawned dropbear on port 22.
We used a new name for the app that contained the jailbreak. It did its extraction of bash and dropbear to private/var/temp, and we stored the bootstrap of the image under, you can see there phoenixshell.app, bootstrap.dmg.
So what we used to do that was iproxy, which is a bit of software that you use to let you do TCP connections into an iPhone that you’ve got connected by a USB, so basically what we’re doing is redirecting port 44 on the iPhone to port 4444 on your local machine, and then we can just netcat into localhost 4444 and hey presto, we’ve got our root shell, which the jailbreak has created by running a bash process with root privileges.
And then we can use SSH – manually – and we can use tar and we can use stat to get the metadata and do compression, etc. Now tar isn’t installed on the iPhone, neither is stat.
So having done that, one of the bits of metadata that we weren’t getting for the files was the file born timestamp. So we also wanted to get that with the image that we were creating. So what I ended up doing was creating a small Python program that used an implementation of SSH called paramiko. I took the AFF4 logical implementation that I was referring to earlier and used the SCP – the secure copy – part of paramiko to basically enumerate all of the files on the phone starting from the root, and then get all the file metadata from each of those, storing them as AFF4 metadata, and then downloading each of the files and storing them as a logical image in the AFF4 volume.
Like I said, I wasn’t getting the born time, so I also got a copy of the stat command, which was in another iOS distribution of software; uploaded that using a unique name, so I wasn’t overwriting anything, and executing the stat command for each of the files that I wanted, to get that extra bit of metadata that I wanted, for the born time. And created a forensic image.
So here’s a portion of the AFF4 image that I created using unzip, because it’s a zip 64, to see what’s in there. You can see here we’ve got a logical image that has got a number of the contents of the user bin directory. Information.turtle, that’s an artifact of AFF4 where we store all of our metadata, but all of this stuff here down to here, again, are logical images, and these two, again, are AFF4 artifacts.
And you can see here an extract of the metadata that we’re storing in AFF4 in RDF, in the information.turtle file; and this is basically the root path, you can see that we’ve got a portion of the fsevents dlog, which if you’re not familiar with iOS is kind of similar to the usn log in Windows, quite useful for figuring out what changes have occurred on the iPhone in its recent history.
And then we’ve got timestamps related to the birth time; we’ve got hash, linear 1 and mdb imagery; we’ve got hashes of the stored file; we’ve got the last access, last written; the original filename stored there as well; and a record changed, and the file size.
So what are the limitations of this approach to jailbreaking? Firstly, it needs quite a complex jailbreak to run the SSH server. When a jailbreak works, usually you’re executing one particular process on the iPhone at very limited privileges, and it does all of those things that I was talking about earlier with the goal of running as root and gaining the ability to spawn new processes.
That actually involves going quite a few further miles to achieve the ability to spawn an SSH to remount the root file system; to spawn a bash shell and re-root it to a TCP-IP connection. So if you can discard those things and just keep the jailbreaking related to getting enough privileges to access the root filesystem, then your life gets a whole lot easier.
The second limitation is the uncertain operation in the presence of a still-running jailbreak. I don’t have any solid answers around what happens when you run one jailbreak over another jailbreak; I’ve seen some horror stories out there, bricked phones etc. That’s an open question.
If we’re talking about changes to the file system, this approach I’ve just described has easily halved the amount of changes we have made to the file system. We;ve got source code for everything that’s happened, so through reading source code we can understand whether or not there are any lurking other bits of code that shouldn’t have been running; and we know exactly what changes are being made, so it fits quite nicely with the ACPO principle of knowing what changes you’re making to evidence if you have to make changes on evidence.
The current jailbreaking approach that we’re working on is to take a forensic agent, which was running as a single process, and do all of the jailbreaking within the forensic agent, so that all we need to do is sideload the forensic agent onto the phone, and that has a conduit in it which gives you logical imaging support over TCP; and just include the key components of the jailbreak that give you the privileges to access all the roots systems of the phone and read all of the files on the phone. And not jump through all those extra hoops around gaining more entitlements around rerouting read/write files, etc.
So we’ve done this – with Evimetry we have a live agent, so we’ve [indecipherable] our live agent to this and implemented this for, I think it’s iOS version 11.2.1, or something like that, and we’re playing with that at the moment.
So that’s what I wanted to talk about today.
