How to speed up EnCase Searches?

My workstations have 8 cores so I split my searches into 8 individual searches. In your case I would do searches of 12 or 13 keywords at a time. This usually max's out the processors as there are 8 threads (1 per search) running. It takes as long as it takes thereafter, you can't ask for more than 100% processing 😉

I'll often run Process Explorer at the same time just to check that EnCase is working as hard as it possibly can.

My understanding is that the next version of EnCase will be better able to use multi-core machines so you won't have to use work-arounds like this.

You might want to consider indexing your case if you are searching so intensively. The pain is all at the indexing stage then.

Paul

Limiting the things EnCase has to search through would be a good starting point and an easy/free one too. Have you used the NSRL hash sets to ignore all known files (Windows system files, Microsoft Office, etc). Also, is it necessary to search through unallocated/unpartitioned space?

I do not know how enacse searches, so this comment may be 'off target'.

With any search, the slowest part will be disk access. My concern about running multiple instances of a search program is that each search will be looking at different sections of the disk. This will a lot of disk thrashing, and with 8 instances running, it could also mean each sector being read 8 times. The machine may look busy, but I would be concerned that this sign of hardwork may be missleading.

An index does sound the best solution, or software that will search for multiple strings in a single disk pass.

Newer versions of EnCase support the building of inverted indexes (a la FTK), which makes ad hoc searches much faster (at the expense of the time spent indexing).

You can also filter out known files through the use of hashes to identify system files that do not need to be searched. The NSRL hash set is particularly good for identifying system files (though you can hash your own system if it is faster). Eliminating known files can reduce search times considerably.

EnCase's indexing still has issues and I've found it is better to run it over selected files or use the script's condition(s) to eliminate certain types of files rather than run it over the entire disk. If there is no reason to index unallocated, exe, mp3, video files, etc, why spend the time? Also, the query index conditions run one keyword at a time so after indexing you have to enter a keyword, bookmark the results or export or whatever and then move on to the next. Doing that 5 times - no biggie. Doing that 100 times will get very tedious.

Given the amount of keywords you are looking to run you may want to consider mounting the image and using someting like dtSearch. Or, without mounting the image, Microforensics' Mercury product. But all of this is predicated on first identifying what data needs to be searched. Sometimes the long road is the best option. Whatever you decide, choose your keywords wisely.

I do not know how enacse searches, so this comment may be 'off target'.

With any search, the slowest part will be disk access. My concern about running multiple instances of a search program is that each search will be looking at different sections of the disk. This will a lot of disk thrashing, and with 8 instances running, it could also mean each sector being read 8 times. The machine may look busy, but I would be concerned that this sign of hardwork may be missleading.

This makes sense intuitively, but is not actually the case when running large keyword sets and/or grep terms. It's easy to see in the task manager, especially if you display graphing of kernel times. You'll see 1 core pegged to 100%.

Jon

Jon, are you referring to multiple keywords, rather than multiple search instances or multiple instances of encase with search operations. Surely the multiple copies of encase will each attempt to search the disk independently, and result in slower performance as mscotgrove is describing? (Its for the same reason I run all my searches of a disk in one process/instance of EnCase as he describes, so it just has to read through the disk in one pass, rather than 8 processes competing for reading data)
Of course, willing to be told there's a better way ) (in the absence of time to test p )

Using conditions, signature analysis, and hashing to reduce the number of files to be searched is a sound suggestion. If you don't want results from unallocated clusters in your final set, why search unallocated in the first place? And so on…

Second, avoid the spurious usage of grep operators. I've seen folks develop "standardized" grep fragments that they use on almost every keyword. Don't do this; you'll slow your searches down to a crawl, especially if they're used to prefix your keywords. Use grep only to account for necessary variances or to reduce a significant number of false positives. The fact is that adding grep operators to search terms slows the search down significantly.

If you only need to concern yourself with document formats, exporting them out and using another tool is a sound suggestion.

As for configuration changes, since the process is CPU bound, mucking with your hardware won't help too much. Dividing the search up into a few parallel searches, as balzanto suggests, can help, though it's a pain to set up. Binding EnCase to a core may help a little.

good luck,

Jon

Jon, are you referring to multiple keywords, rather than multiple search instances or multiple instances of encase with search operations. Surely the multiple copies of encase will each attempt to search the disk independently, and result in slower performance as mscotgrove is describing?

Rich,

What I've found is that EnCase slows down the more grep keywords you search for (if all your keywords are fixed strings, then it'll be fast), enough so that your search will be CPU bound–this means it will benefit from parallelization. As inefficient as it may sound to be reading redundantly from disk, you'll get benefit from running multiple searches simultaneously. You can easily verify whether this holds true for your search and your system by seeing if EnCase pegs your CPU… if it does, you are not I/O bound.

I prefer letting EnCase do one thing at a time, no matter how long it takes. But if the goal is to improve performance of searches, Tony's suggestion should help decrease wall-clock time. I would run the searches from within the same instance/process of EnCase, to minimize context switching and maximize disk caching (in addition to the system cache, EnCase has some block caching as well).

Jon