Anyone got a bot to...
 
Notifications
Clear all

Anyone got a bot to find deleted truecrypt container header?

46 Posts
7 Users
0 Likes
7,399 Views
(@4144414d)
Posts: 33
Eminent Member
 

That is clear now, still if you could describe how exactly you managed to create the graphic, it would be useful (notwithstanding the buiilt-in autodetect capabilities of your nice script).

Added those comments now, thank you. If you have any other comments or corrections let me know. You can even do a pull request if you are feeling adventurous, although that is a lot of effort. I was thinking of linking to an existing tutorial about calculating and visualising the entropy if you happen to know of a good existing one?

I've not written the guide for this one but if you fancy a challenge here's the next image. It is a FAT volume that once contained a container, it was reformatted FAT, and a number of pictures copied on to it. This overwrote the header at the start of the container. The password is 'password' again.

https://raw.githubusercontent.com/4144414D/pytruecrypt/gh-pages/resources/example2.zip

I need to make pytruecrypt a pip package so it's easier to install though.

 
Posted : 07/03/2018 7:47 am
(@loonaluna)
Posts: 33
Eminent Member
Topic starter
 

Hey thanks for your tutorials. I've installed it all and got it running and it does exactly what it shows in the tutorial, with your example file. I have a chunk of material I need to look into though, so high and low entropy is not an option for me, so I think I need to do a brute force instead of a chain. I know how to extract this chunk into a smaller file, but I'll still have to do a brute force of each sector (or hex kb of this sector or whatever it's called), until I cover the entirety of the free space that I suspect is involved here. Do I do this with dump.py instead of python.py instead, and if so how?

 
Posted : 07/03/2018 7:49 pm
(@4144414d)
Posts: 33
Eminent Member
 

I know how to extract this chunk into a smaller file, but I'll still have to do a brute force of each sector.

If you can extract this space into a smaller file you can still run hunt.py against it to find, hopefully, your missing TrueCrypt header. If you do what jaclaz suggested in one of your other threads and extract a few gb around where the sections of unallocated space are.

This thread https://www.forensicfocus.com/Forums/viewtopic/t=16377/postdays=0/postorder=asc/start=7/

The high/low entropy all happens without any effort from you and it's what I would run first for sure. It'll work on sections of the image just as well as the whole image.

You can try the brute force option but it will be extremely slow. Try it on that 30mb test file first and you'll see how slow it is.

You can just run hunt.py against the whole 2TB image, but I feel like it may crash as it's currently holding all the entropy information in memory. The entropy calculations will take an age too.

Given that you think your missing containers are well over 2GB the hunt.py command would be

hunt.py <your image> <your password> –chain=4000000

EDIT Also a quick note, if your password has a space in it this won't work. I'll need to update the script.

 
Posted : 07/03/2018 8:13 pm
(@loonaluna)
Posts: 33
Eminent Member
Topic starter
 

Ah ok I misread the chain concept as something that differentiated suspiciously obscure space typical of truecrypt from typically boring text files with little entropy. That's why I put a test container I made through the same script, didn't get much of anything at all, and thought I needed a brute instead. But of course I hadn't changed chain=256 to chain=4 million, so that's why the headers didn't show up probably. By the way, what is the reason why chain=4000000 when the file is 2GB?

 
Posted : 07/03/2018 10:43 pm
(@4144414d)
Posts: 33
Eminent Member
 

Ah ok I misread the chain concept as something that differentiated suspiciously obscure space typical of truecrypt from typically boring text files with little entropy.

No, you are completely correct. That is exactly what it does. Hunt tries to locate headers based on the differences between high/low entropy.

That's why I put a test container I made through the same script, didn't get much of anything at all, and thought I needed a brute instead.

A couple of things that are probably causing this.

1) hunt will create a file called 'ent.pickle' once it's calculated the entropy for a file. That's because calculating the entropy takes quite a while, and if we might try a few passes at the same file it doesn't make sense to recalculate the entropy each time.

However, if you move from one file to another and you've stayed in the same folder it will reuse the ent.pickle but for the wrong file. So instead of calculating the entropy it simply says 'Loading source entropy from ent.pickle'. Not the most obvious warning message I admit. If you delete ent.pickle and try again see how it goes.

I personally like to keep each of the images I'm working on in different folders, that way the extracted headers, ent files, and the results logs don't get mixed up.

2) The script isn't designed to run against containers on their own but 'hunts' (hence the silly name) for containers inside disk images. I actually think it'll crash on a full container. It might see the entropy starting at sector 0 but by default, it tries to look 8 sectors either side of where it thinks a header will be and sectors -1, -2, -3 etc do not exist and when it tries to read them it would crash.

I'll make changes to the script to make this more obvious and update the guide when I've finished the write up for example 2.

For now, try the brute force method on what you've extracted but only let it run few a minutes or so. If that doesn't get the header you are better off with the chain method.

But of course I hadn't changed chain=256 to chain=4 million, so that's why the headers didn't show up probably. By the way, what is the reason why chain=4000000 when the file is 2GB?

chain=256 is going to find basically any TrueCrypt header but has the most chance of getting confused with other random files on the disk. That is because 256 sectors are the size of the header, which is about 128kb. It's likely you'll have other random files that are larger than that which hunt will then waste time on trying to decrypt them.

chain=4000000 is looking for a minimum of 4000000 high entropy sectors in a row before it'll start trying to decrypt things. 4000000 * 512 = 2048000000 bytes, which is around the 2gb mark. You are less likely to have other files of that size that are as random as a TrueCrypt container so hopefully, hunt will find the result quicker this way.

Hence the ent.pickle file. If you manage to parse the full 2TB file you could try with a very big chain number and keep lowering it if you don't find anything. The ent.pickle file saves you from recalculating the entropy for each attempt, which saves a lot of time.

 
Posted : 08/03/2018 7:15 am
(@loonaluna)
Posts: 33
Eminent Member
Topic starter
 

2) The script isn't designed to run against containers on their own but 'hunts' (hence the silly name) for containers inside disk images. I actually think it'll crash on a full container. It might see the entropy starting at sector 0 but by default, it tries to look 8 sectors either side of where it thinks a header will be and sectors -1, -2, -3 etc do not exist and when it tries to read them it would crash.

When it does brute force it doesn't look for sectors -1,-2,-3, only when it's chain?

Btw, how did you create such a small 30mb image (on windows hopefully)? I only know how to put a container into something like a flash drive, but that's 7gb size.

For now, try the brute force method on what you've extracted but only let it run few a minutes or so. If that doesn't get the header you are better off with the chain method.

Why only a few minutes?

Is there any way to run multiple instances of this program, to make the search faster? Like, install various pythons and put pytruecrypt on each one xD, or is there only one cmd per OS?

chain=256 is going to find basically any TrueCrypt header but has the most chance of getting confused with other random files on the disk. That is because 256 sectors are the size of the header, which is about 128kb. It's likely you'll have other random files that are larger than that which hunt will then waste time on trying to decrypt them.

chain=4000000 is looking for a minimum of 4000000 high entropy sectors in a row before it'll start trying to decrypt things. 4000000 * 512 = 2048000000 bytes, which is around the 2gb mark. You are less likely to have other files of that size that are as random as a TrueCrypt container so hopefully, hunt will find the result quicker this way.

Hence the ent.pickle file. If you manage to parse the full 2TB file you could try with a very big chain number and keep lowering it if you don't find anything. The ent.pickle file saves you from recalculating the entropy for each attempt, which saves a lot of time.

I put a test container inside a FAT32 flash drive, made a raw image of the flash drive, but only because winhex allows me to do that with what it understands is a drive. It ran through at about a rate of 5gb per day with the brute method and found the container's header. But the chain method just wouldn't work no matter what size. With the chain method, first I got a memory error and the follow times I tried, iirc varying the size of the chain=x, I went on to get EOFErrors. The size of the pickle file was about 400mb so maybe that had something to do with it? And the container was sitting in one of the sectors near the beginning of the flash drive.

 
Posted : 09/03/2018 6:56 pm
(@4144414d)
Posts: 33
Eminent Member
 

When it does brute force it doesn't look for sectors -1,-2,-3, only when it's chain?

Brute force never does, the chain method looks around places where it thinks a header might be. If it thinks that this is right near the start or the end of an image it might try to read sectors that do not exist. Its a bug I should probably fix.

It's line 149. search_size = 8 e.g. check 8 sectors before and after where we think a header might be.

Btw, how did you create such a small 30mb image (on windows hopefully)? I only know how to put a container into something like a flash drive, but that's 7gb size.

They are actually TrueCrypt containers themselves. I made a 30mb container, formatted it so that it was zeroed out, then put the 10mb containers in them.

Why only a few minutes?

If you get lucky and the header is right near the start, which will only take a few mins. Otherwise, as you've seen its super slow.

Is there any way to run multiple instances of this program, to make the search faster? Like, install various pythons and put pytruecrypt on each one xD, or is there only one cmd per OS?

You could do this but it would need changing some code, or run multiple instances against different parts of an image.

I could make the code multi-threaded which will use more cores… but multiprocessing is a right faff. I'll do it one day but I don't have the time right now.

But the chain method just wouldn't work no matter what size.

Well, that's rubbish. Could you create an entropy listing like Jaclaz suggested? Maybe with https://github.com/dupgit/entropie setting the size to 512? Or send me the ent.pickle file to look at. I can take a look at the peeks and see if I can work out why the chain method is making the wrong choice.

(If you can send me the full 7GB file image I could check that too if you like and see whats going on)

With the chain method, first I got a memory error and the follow times I tried, iirc varying the size of the chain=x, I went on to get EOFErrors. The size of the pickle file was about 400mb so maybe that had something to do with it? But the chain method just wouldn't work no matter what size.

Yeah, I was worried this might happen. hunt currently tries to keep all the entropy information in ram at the same time. It's not really needed and is just lazy programming on my part. I will work on an update that deals with this better.

 
Posted : 10/03/2018 10:51 am
(@loonaluna)
Posts: 33
Eminent Member
Topic starter
 

Is there any way to run multiple instances of this program, to make the search faster? Like, install various pythons and put pytruecrypt on each one xD, or is there only one cmd per OS?

You could do this but it would need changing some code, or run multiple instances against different parts of an image.

I could make the code multi-threaded which will use more cores… but multiprocessing is a right faff. I'll do it one day but I don't have the time right now.

No need thanks. Anyone can do this on their own by enabling multithreading and more cmd instances. I didn't know cmd could be open more than once when I asked.

But the chain method just wouldn't work no matter what size.

Well, that's rubbish. Could you create an entropy listing like Jaclaz suggested? Maybe with https://github.com/dupgit/entropie setting the size to 512? Or send me the ent.pickle file to look at. I can take a look at the peeks and see if I can work out why the chain method is making the wrong choice.

(If you can send me the full 7GB file image I could check that too if you like and see whats going on)

I don't know what was up with that 7gb file but seeing as it was just a test container and a test file, I think we'd be better off if I don't send it to you for the time being as it would be very cumbersome. It's probably the pickle file that gets too large to be read, or something like that.

OK, so here's the thing, and I'm wondering if you or jaclaz has the answer. This weekend I found out that by typing three times the password of one of the containers into the first big chunk I isolated on the hard drive, at the third attempt the password works. But it comes with a warning, saying the headers are damaged. Indeed, the news isn't good and without being an expert, it looks like fragmentation. Whether or not I fix the headers of course, it's unreadable but says it's an 80gb file. But the chunk is only 52,7gb. Does this mean it saw the backup header because it was exactly 131072 bytes from the end of the file, as dantz said once along time ago in wilderssecurity that's where the backup headers were stored in truecrypt containers?

Btw, fixing the headers with truecrypt volume tools doesn't seem to rewrite the file beginning, as it still can't be spotted in –brute run with pytruecrypt. Maybe this is because the header can only be spotted if it's in a coherent file where the header is where the backup header says it is, and this isn't the case?

Anyway, so this chunk when run through entropie.py, only has one entropy number. Is that what you mean by entropy listing? Would finding other chunks with the same entropy mean adding them all up together could reach 80gb? Could the assumption be made that the first chunk is missing and that it could be sitting in one big chunk elsewhere, and that it could be added, and if so, can this chunk only be in a previous sector or can it be in sectors below? The second big chunk that I was hoping would be the second container gives 5 different entropy numbers btw, whereas this first chunk gives one result. Can this result be written on a pickle.ent file to get the chain method to work on such a big drive or is that not how it works?

 
Posted : 12/03/2018 4:54 pm
jaclaz
(@jaclaz)
Posts: 5133
Illustrious Member
 

More doubts than answers, unfortunately.

You stated (here or elsewhere) that you found two suitable "chunks" one around 52 Gb and one around 74 gb.

Then, notwithstanding that at the moment the nice script by 4144414D didn't found anything (yet) you - evidently by sheer luck - isolated EXACTLY the right chunk (so exactly that actually Truecrypt accepts - albeit on third attempt only - the "right password") ? .

I cannot see 😯 (but I am not an expert on Truecrypt specifically) how inserting the "right" password three times (as opposed to one, two being a good compromise wink ) can work.

The result you report seems like telling us that *somehow* the Truecrypt at first and second attempts tries to decrypt the header (that either isn't there or it is not in the right offset in the mapped chunk) then, on third attempt, *somehow* finds the footer instead[1]?

BUT, even if it somehow accepts the password, it states that the volume is around 80 GB, whilst the chunk is only 52 GB?

If this is the case, logically it should mean that *somehow* the chunk is missing around 38 GB at the beginning.

jaclaz

[1] by "footer" I mean the backup of the header at S-131072

 
Posted : 12/03/2018 7:50 pm
(@loonaluna)
Posts: 33
Eminent Member
Topic starter
 

More doubts than answers, unfortunately.

You stated (here or elsewhere) that you found two suitable "chunks" one around 52 Gb and one around 74 gb.

Then, notwithstanding that at the moment the nice script by 4144414D didn't found anything (yet) you - evidently by sheer luck - isolated EXACTLY the right chunk (so exactly that actually Truecrypt accepts - albeit on third attempt only - the "right password") ? .

I cannot see 😯 (but I am not an expert on Truecrypt specifically) how inserting the "right" password three times (as opposed to one, two being a good compromise wink ) can work.

The result you report seems like telling us that *somehow* the Truecrypt at first and second attempts tries to decrypt the header (that either isn't there or it is not in the right offset in the mapped chunk) then, on third attempt, *somehow* finds the footer instead[1]?

BUT, even if it somehow accepts the password, it states that the volume is around 80 GB, whilst the chunk is only 52 GB?

If this is the case, logically it should mean that *somehow* the chunk is missing around 38 GB at the beginning.

jaclaz

[1] by "footer" I mean the backup of the header at S-131072

Yeah apparently this is something truecrypt does when the header is damaged for some reason, I read it in another forum happen there too. You may not be an expert in truecrypt (and maybe 4144 is), but do you know if when a file is saved it can have its first parts stored in later sectors and last parts stored in earlier sectors, or if file saving always goes for the first sectors first?

 
Posted : 13/03/2018 2:27 am
Page 2 / 5
Share: