±Forensic Focus Partners

Become an advertising partner

±Your Account


Username
Password

Forgotten password/username?

Site Members:

New Today: 2 Overall: 36296
New Yesterday: 6 Visitors: 241

±Follow Forensic Focus

Forensic Focus Facebook PageForensic Focus on TwitterForensic Focus LinkedIn GroupForensic Focus YouTube Channel

RSS feeds: News Forums Articles

±Latest Articles

±Latest Videos

±Latest Jobs

Anyone got a bot to find deleted truecrypt container header?

Computer forensics discussion. Please ensure that your post is not better suited to one of the forums below (if it is, please post it there instead!)
Reply to topicReply to topic Printer Friendly Page
Forum FAQSearchView unanswered posts
Page Previous  1, 2, 3, 4, 5, 6  Next 
  

4144414D
Member
 

Re: Anyone got a bot to find deleted truecrypt container hea

Post Posted: Mar 08, 18 07:15

- loonaluna
Ah ok I misread the chain concept as something that differentiated suspiciously obscure space typical of truecrypt from typically boring text files with little entropy.


No, you are completely correct. That is exactly what it does. Hunt tries to locate headers based on the differences between high/low entropy.

- loonaluna
That's why I put a test container I made through the same script, didn't get much of anything at all, and thought I needed a brute instead.


A couple of things that are probably causing this.

1) hunt will create a file called 'ent.pickle' once it's calculated the entropy for a file. That's because calculating the entropy takes quite a while, and if we might try a few passes at the same file it doesn't make sense to recalculate the entropy each time.

However, if you move from one file to another and you've stayed in the same folder it will reuse the ent.pickle but for the wrong file. So instead of calculating the entropy it simply says 'Loading source entropy from ent.pickle'. Not the most obvious warning message I admit. If you delete ent.pickle and try again see how it goes.

I personally like to keep each of the images I'm working on in different folders, that way the extracted headers, ent files, and the results logs don't get mixed up.

2) The script isn't designed to run against containers on their own but 'hunts' (hence the silly name) for containers inside disk images. I actually think it'll crash on a full container. It might see the entropy starting at sector 0 but by default, it tries to look 8 sectors either side of where it thinks a header will be and sectors -1, -2, -3 etc do not exist and when it tries to read them it would crash.

I'll make changes to the script to make this more obvious and update the guide when I've finished the write up for example 2.

For now, try the brute force method on what you've extracted but only let it run few a minutes or so. If that doesn't get the header you are better off with the chain method.

- loonaluna
But of course I hadn't changed chain=256 to chain=4 million, so that's why the headers didn't show up probably. By the way, what is the reason why chain=4000000 when the file is 2GB?


chain=256 is going to find basically any TrueCrypt header but has the most chance of getting confused with other random files on the disk. That is because 256 sectors are the size of the header, which is about 128kb. It's likely you'll have other random files that are larger than that which hunt will then waste time on trying to decrypt them.

chain=4000000 is looking for a minimum of 4000000 high entropy sectors in a row before it'll start trying to decrypt things. 4000000 * 512 = 2048000000 bytes, which is around the 2gb mark. You are less likely to have other files of that size that are as random as a TrueCrypt container so hopefully, hunt will find the result quicker this way.

Hence the ent.pickle file. If you manage to parse the full 2TB file you could try with a very big chain number and keep lowering it if you don't find anything. The ent.pickle file saves you from recalculating the entropy for each attempt, which saves a lot of time.  
 
  

loonaluna
Member
 

Re: Anyone got a bot to find deleted truecrypt container hea

Post Posted: Mar 09, 18 18:56

- 4144414D


2) The script isn't designed to run against containers on their own but 'hunts' (hence the silly name) for containers inside disk images. I actually think it'll crash on a full container. It might see the entropy starting at sector 0 but by default, it tries to look 8 sectors either side of where it thinks a header will be and sectors -1, -2, -3 etc do not exist and when it tries to read them it would crash.


When it does brute force it doesn't look for sectors -1,-2,-3, only when it's chain?

Btw, how did you create such a small 30mb image (on windows hopefully)? I only know how to put a container into something like a flash drive, but that's 7gb size.



- 4144414D

For now, try the brute force method on what you've extracted but only let it run few a minutes or so. If that doesn't get the header you are better off with the chain method.



Why only a few minutes?

Is there any way to run multiple instances of this program, to make the search faster? Like, install various pythons and put pytruecrypt on each one xD, or is there only one cmd per OS?


- 4144414D

chain=256 is going to find basically any TrueCrypt header but has the most chance of getting confused with other random files on the disk. That is because 256 sectors are the size of the header, which is about 128kb. It's likely you'll have other random files that are larger than that which hunt will then waste time on trying to decrypt them.

chain=4000000 is looking for a minimum of 4000000 high entropy sectors in a row before it'll start trying to decrypt things. 4000000 * 512 = 2048000000 bytes, which is around the 2gb mark. You are less likely to have other files of that size that are as random as a TrueCrypt container so hopefully, hunt will find the result quicker this way.

Hence the ent.pickle file. If you manage to parse the full 2TB file you could try with a very big chain number and keep lowering it if you don't find anything. The ent.pickle file saves you from recalculating the entropy for each attempt, which saves a lot of time.


I put a test container inside a FAT32 flash drive, made a raw image of the flash drive, but only because winhex allows me to do that with what it understands is a drive. It ran through at about a rate of 5gb per day with the brute method and found the container's header. But the chain method just wouldn't work no matter what size. With the chain method, first I got a memory error and the follow times I tried, iirc varying the size of the chain=x, I went on to get EOFErrors. The size of the pickle file was about 400mb so maybe that had something to do with it? And the container was sitting in one of the sectors near the beginning of the flash drive.  
 
  

4144414D
Member
 

Re: Anyone got a bot to find deleted truecrypt container hea

Post Posted: Mar 10, 18 10:51

- loonaluna

When it does brute force it doesn't look for sectors -1,-2,-3, only when it's chain?


Brute force never does, the chain method looks around places where it thinks a header might be. If it thinks that this is right near the start or the end of an image it might try to read sectors that do not exist. Its a bug I should probably fix.

It's line 149. search_size = 8 e.g. check 8 sectors before and after where we think a header might be.


- loonaluna

Btw, how did you create such a small 30mb image (on windows hopefully)? I only know how to put a container into something like a flash drive, but that's 7gb size.


They are actually TrueCrypt containers themselves. I made a 30mb container, formatted it so that it was zeroed out, then put the 10mb containers in them.


- loonaluna

Why only a few minutes?


If you get lucky and the header is right near the start, which will only take a few mins. Otherwise, as you've seen its super slow.



- loonaluna

Is there any way to run multiple instances of this program, to make the search faster? Like, install various pythons and put pytruecrypt on each one xD, or is there only one cmd per OS?


You could do this but it would need changing some code, or run multiple instances against different parts of an image.

I could make the code multi-threaded which will use more cores... but multiprocessing is a right faff. I'll do it one day but I don't have the time right now.



- loonaluna

But the chain method just wouldn't work no matter what size.


Well, that's rubbish. Could you create an entropy listing like Jaclaz suggested? Maybe with github.com/dupgit/entropie setting the size to 512? Or send me the ent.pickle file to look at. I can take a look at the peeks and see if I can work out why the chain method is making the wrong choice.

(If you can send me the full 7GB file image I could check that too if you like and see whats going on)

- loonaluna

With the chain method, first I got a memory error and the follow times I tried, iirc varying the size of the chain=x, I went on to get EOFErrors. The size of the pickle file was about 400mb so maybe that had something to do with it? But the chain method just wouldn't work no matter what size.


Yeah, I was worried this might happen. hunt currently tries to keep all the entropy information in ram at the same time. It's not really needed and is just lazy programming on my part. I will work on an update that deals with this better.  
 
  

loonaluna
Member
 

Re: Anyone got a bot to find deleted truecrypt container hea

Post Posted: Mar 12, 18 16:54

- 4144414D

- loonaluna

Is there any way to run multiple instances of this program, to make the search faster? Like, install various pythons and put pytruecrypt on each one xD, or is there only one cmd per OS?


You could do this but it would need changing some code, or run multiple instances against different parts of an image.

I could make the code multi-threaded which will use more cores... but multiprocessing is a right faff. I'll do it one day but I don't have the time right now.


No need thanks. Anyone can do this on their own by enabling multithreading and more cmd instances. I didn't know cmd could be open more than once when I asked.


- 4144414D

- loonaluna

But the chain method just wouldn't work no matter what size.


Well, that's rubbish. Could you create an entropy listing like Jaclaz suggested? Maybe with github.com/dupgit/entropie setting the size to 512? Or send me the ent.pickle file to look at. I can take a look at the peeks and see if I can work out why the chain method is making the wrong choice.

(If you can send me the full 7GB file image I could check that too if you like and see whats going on)


I don't know what was up with that 7gb file but seeing as it was just a test container and a test file, I think we'd be better off if I don't send it to you for the time being as it would be very cumbersome. It's probably the pickle file that gets too large to be read, or something like that.



OK, so here's the thing, and I'm wondering if you or jaclaz has the answer. This weekend I found out that by typing three times the password of one of the containers into the first big chunk I isolated on the hard drive, at the third attempt the password works. But it comes with a warning, saying the headers are damaged. Indeed, the news isn't good and without being an expert, it looks like fragmentation. Whether or not I fix the headers of course, it's unreadable but says it's an 80gb file. But the chunk is only 52,7gb. Does this mean it saw the backup header because it was exactly 131072 bytes from the end of the file, as dantz said once along time ago in wilderssecurity that's where the backup headers were stored in truecrypt containers?

Btw, fixing the headers with truecrypt volume tools doesn't seem to rewrite the file beginning, as it still can't be spotted in --brute run with pytruecrypt. Maybe this is because the header can only be spotted if it's in a coherent file where the header is where the backup header says it is, and this isn't the case?


Anyway, so this chunk when run through entropie.py, only has one entropy number. Is that what you mean by entropy listing? Would finding other chunks with the same entropy mean adding them all up together could reach 80gb? Could the assumption be made that the first chunk is missing and that it could be sitting in one big chunk elsewhere, and that it could be added, and if so, can this chunk only be in a previous sector or can it be in sectors below? The second big chunk that I was hoping would be the second container gives 5 different entropy numbers btw, whereas this first chunk gives one result. Can this result be written on a pickle.ent file to get the chain method to work on such a big drive or is that not how it works?  
 
  

jaclaz
Senior Member
 

Re: Anyone got a bot to find deleted truecrypt container hea

Post Posted: Mar 12, 18 19:50

More doubts than answers, unfortunately.

You stated (here or elsewhere) that you found two suitable "chunks" one around 52 Gb and one around 74 gb.

Then, notwithstanding that at the moment the nice script by 4144414D didn't found anything (yet) you - evidently by sheer luck - isolated EXACTLY the right chunk (so exactly that actually Truecrypt accepts - albeit on third attempt only - the "right password") Confused .

I cannot see Shocked (but I am not an expert on Truecrypt specifically) how inserting the "right" password three times (as opposed to one, two being a good compromise Wink ) can work.

The result you report seems like telling us that *somehow* the Truecrypt at first and second attempts tries to decrypt the header (that either isn't there or it is not in the right offset in the mapped chunk) then, on third attempt, *somehow* finds the footer instead[1]?

BUT, even if it somehow accepts the password, it states that the volume is around 80 GB, whilst the chunk is only 52 GB?

If this is the case, logically it should mean that *somehow* the chunk is missing around 38 GB at the beginning.

jaclaz


[1] by "footer" I mean the backup of the header at S-131072
_________________
- In theory there is no difference between theory and practice, but in practice there is. - 
 
  

loonaluna
Member
 

Re: Anyone got a bot to find deleted truecrypt container hea

Post Posted: Mar 13, 18 02:27

- jaclaz
More doubts than answers, unfortunately.

You stated (here or elsewhere) that you found two suitable "chunks" one around 52 Gb and one around 74 gb.

Then, notwithstanding that at the moment the nice script by 4144414D didn't found anything (yet) you - evidently by sheer luck - isolated EXACTLY the right chunk (so exactly that actually Truecrypt accepts - albeit on third attempt only - the "right password") Confused .

I cannot see Shocked (but I am not an expert on Truecrypt specifically) how inserting the "right" password three times (as opposed to one, two being a good compromise Wink ) can work.

The result you report seems like telling us that *somehow* the Truecrypt at first and second attempts tries to decrypt the header (that either isn't there or it is not in the right offset in the mapped chunk) then, on third attempt, *somehow* finds the footer instead[1]?

BUT, even if it somehow accepts the password, it states that the volume is around 80 GB, whilst the chunk is only 52 GB?

If this is the case, logically it should mean that *somehow* the chunk is missing around 38 GB at the beginning.

jaclaz

[1] by "footer" I mean the backup of the header at S-131072



Yeah apparently this is something truecrypt does when the header is damaged for some reason, I read it in another forum happen there too. You may not be an expert in truecrypt (and maybe 4144 is), but do you know if when a file is saved it can have its first parts stored in later sectors and last parts stored in earlier sectors, or if file saving always goes for the first sectors first?  
 
  

jaclaz
Senior Member
 

Re: Anyone got a bot to find deleted truecrypt container hea

Post Posted: Mar 13, 18 11:17

- loonaluna

Yeah apparently this is something truecrypt does when the header is damaged for some reason, I read it in another forum happen there too. You may not be an expert in truecrypt (and maybe 4144 is), but do you know if when a file is saved it can have its first parts stored in later sectors and last parts stored in earlier sectors, or if file saving always goes for the first sectors first?


There is no "fixed rule", it depends on the available space, level of fragmentation, etc., etc., BUT - generally speaking - data is written from beginning to end (if you prefer from left to right) and when one creates a (large) container/file, he/she does so at a time the underlying file system has been recently defragged and has plenty of space available, so it is possible (but unlikely) that the file was saved with a beginning part at a later address.

But the point I was trying to make was another one.

The Truecrypt (at third attempt) must access *something* (either an actual header - even if partially corrupted - or a backup of the header - again even if partially corrupted).

IF what is accessed is NOT the real header, THEN it must be the header backup. [1]

This latter is by definition placed @S-131072 on Truecrypt 7.

So, what happens if you try the python tool on that sector (or on the area around that sector)?

jaclaz

[1] the alternative being that it accesses a sector full of "digital garbage" and attempts to interpret it nonetheless Shocked
_________________
- In theory there is no difference between theory and practice, but in practice there is. - 
 

Page 3 of 6
Page Previous  1, 2, 3, 4, 5, 6  Next