Performing live response

Hi Harlen

For me I find that 'live' work tends to be in response to a specific incident, data loss, hack attack, suspected trojan, and tends to be more oriented around a 'live' network with a target machine that cannot be powered down for operational/business reasons. Perhaps a mail server has been compromised or a person is thought to be acting illegally.

Where possible I use the Helix distro in its Windows mode for dumping memory etc to another machine via netcat. If full forensic conditions are not needed I've written a batch file which runs from a USB key to dump memory, logs, security settings, system info etc back onto the key. (One of the tools is yours actually but I can't remember which one and dont have it with me to check!). FTK imager is also on the latest version for image aquisition

I've never carried out a full forensic investigation of a live system without powering down though.

If you are interested I'll repost with a full list of the tools I run when I get back to the office.

Nick

Nick,

"…I use the Helix distro in its Windows mode for dumping memory etc…"

What do you do with the memory dump, and what constitutes "etc" (ie, what else do you grab?

"If you are interested I'll repost with a full list of the tools I run when I get back to the office."

That's be cool, thanks.

Harlan

In a majority of cases I don't collect live information. The utilities that are currently in use for windows responses –WFT, Helix, FSP all modify parts of the system that IMO, shouldn't be modified. However, I just read Chris Brown's book(you know..the prodiscover owner) and I'm considering changing my methods on systems with more than 2 gb of ram.( I might lower it to 1gb but we'll see).

In one testing (don't take this as gospel, it was a simple inctrl test, run twice –I plan on doing more)
The modifications made by every free piece of software are in a key location –The prefetch directory. With helix, there are something like 24 registry modifications made as well. WFT and FSP hit the prefetch directory as well although I haven't tested these for what they truly modify. Why is the prefetch directory so important to me? When someone compromises a windows box, they typically load a toolset that runs a number of utilities including but not limited to -fport,net,net1,netstat,ftp and many others.
When the free tools run from the 'live response' utilities, they overwrite these prefetch files, destroying very very important evidence. In my book, that's a no no.
And, by the time I get called on to the scene, unless something is happening at that time (which it typically isn't due to IR triage that isolates a box), the volatile evidence that could be used as 'best' evidence is long gone.

If Mr. Brown wants to send me a copy of PD IR I'd try it. )

I'd almost rather see something coded in say..perl that collects this information without overwriting the prefetch files on the disk, and displacing the memory required to hash files,run exe's etc.

That said, there is nothing unique that I try to collect from the system in those cases where I do live collection. I follow the order of volatility, collect the data, and with memory dumps, typically look for strings containing information that will lead me elsewhere or those that contain passwords. The more I think about it, I'd like to get a tool together that takes the output of pslist.exe and runs userdump.exe against each pid.

While I see the value in collecting live information, I haven't found a tool that is the least intrusive.

"…modify parts of the system that IMO, shouldn't be modified…"

Such as?

"…in a key location –The prefetch directory."

That's true for anything run on a live XP system. Why is that an issue?

"…With helix, there are something like 24 registry modifications made as well. "

Which keys?

As I said, any tool that's run on XP will hit the Prefetch directory…assuming that the contents of the Prefetch directory haven't reached their limit.

"When the free tools run from the 'live response' utilities, they overwrite these prefetch files, destroying very very important evidence. In my book, that's a no no."

Agreed…you're destroying potentially useful data. Know an eaasy fix for that? When you run the FSP, rename the executables that you run with the FRU.

"I'd almost rather see something coded in say..perl that collects this information without overwriting the prefetch files on the disk, and displacing the memory required to hash files,run exe's etc."

Well, I'd say that if you're using the FSP, you've already got a tool that won't overwrite prefetch files. As far as not displacing memory…I can't think of a live tool that won't do that.

"I'd like to get a tool together that takes the output of pslist.exe and runs userdump.exe against each pid."

What does userdump.exe do? Are you talking about dumping the memory contents of each PID? Pmdump.exe will do this. I wouldn't use pslist.exe, though…it doesn't provide enough info. Use tlist.exe, and just get the PIDs by parsing the output. Or, use WMI to get the PIDs.

Re intrusiveness. If you know and can explain the changes that are made, wouldn't that be enough?

Harlan

This issue of intrusiveness brings to mind another example…

You're walking home one night, and you hear a moan in an alley. Investigating, you find that someone's been stabbed. You call 911, and the paramedics arrive, stabilize the guy, and take him to the hospital…removing him from the scene. At the hospital, a surgeon operates on him. Even if the victim expires, the cops are still able to collect evidence, and even prosecute the attacker(s).

So…investigating a live XP system, you plop in your CD of tools and run them, documenting the time (if you're using the FSP, this is done for you). Therefore, knowing the tools you're running, you can justify any .pf files or Registry modifications that are made. Also, testing your tools ahead of time will tell you what changes you can expect to occur…and this testing will serve as part of your documentation.

Given this…why *not* do a live response?

Harlan

I ran a few tests last night and couldn't get a consistent read from Helix's FRED script –the output from one such test is below. I need to run it a few more times and I'd also like to conduct memory capture differences(dd the memory, run the tools, dd the memory again and diff them).

Renaming the exe's is a good idea, however in a 'canned' tool like Helix or WFT, most people probably haven't renamed them.

Userdump provides the same functionality of pmdump, but a process dump captured by userdump can be processed by dumpcheck. I haven't run dumpcheck against a pmdump output yet.

On to intrusiveness…
Using your stabbing analogy, of course the first responder gets there and attempts to save the persons life. However, if in the process they contaminate the crime scene, any evidence collected is useless.

One example that comes to mind was when a state police investigator trapsed through a crime scene before it was secured and his boot prints covered other, presumably crime scene related prints. The defense attorney ripped him a new one.

So, in a live response situation, if your tool stomps all over the volatile data it doesn't matter if you can explain what changes were made by the tool, if the changes caused loss of evidence, then odds are the entire live response would be thrown out.

For instance, while running a live memory capture today against one of my honeypots that had been compromised, dd failed unexpectedly after copying 8k. An argument there would probably be, "well how can we trust ANY of this other evidence if your tools failed?"

Reasons for not doing a live response
Too far after the fact.

Most of the evidence needed is static on the disk or captured via pcap. Referring back to Brown's book..the hacker defender example given for live captures..the password for hxdf is stored in a file, so the memory dump isn't needed for that purpose.

Not enough current documentation on what the live tools actually modify –It's extremely time consuming to attempt to test against every operating system you might run in to.

Like I said, I understand the importance of the data that can be gathered in a live response, but it's currently not in my playbook unless I can get to the scene asap. Our SOP for first responders –after triage(usually consisting of network based isolation) is to pull the plug.

I need a more compelling argument to formalize a procedure for live captures. Is there any case law where evidence captured from a live response has been used?

one report from inctrl after running Helix FRED.

Installation Report fred-nc
Generated by InCtrl5, version 1.0.0.0
Install program D\ir\fred-nc.bat | D\ir\bin\nc.exe 192.168.197.131 3000
1/5/2006 1207 AM

————————————————————
Registry
********

Keys ignored 0
—————
* (none)

Keys added 18
————–
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ESENT\Process\nbtstat
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ESENT\Process\nbtstat\DEBUG
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\RFC1156Agent
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\RFC1156Agent\CurrentVersion
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\RFC1156Agent\CurrentVersion\Parameters
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\Root\LEGACY_WMIAPSRV
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\Root\LEGACY_WMIAPSRV\0000
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\Root\LEGACY_WMIAPSRV\0000\Control
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\WmiApSrv\Enum
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Control\DeviceClasses\{6994AD04-93EF-11D0-A3CC-00A0C9223196}\##?#PCI#VEN_1274&DEV_1371&SUBSYS_13711274&REV_02#3&61AAA01&0&90#{6994ad04-93ef-11d0-a3cc-00a0c9223196}\#Wave\Device Parameters\
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Control\Lsa\SspiCache\
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Control\Lsa\SspiCache\
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Control\Lsa\SspiCache\
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Services\lanmanserver\Shares\
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\Root\LEGACY_WMIAPSRV
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\Root\LEGACY_WMIAPSRV\0000
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\Root\LEGACY_WMIAPSRV\0000\Control
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WmiApSrv\Enum

Keys deleted 4
—————
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Control\DeviceClasses\{6994AD04-93EF-11D0-A3CC-00A0C9223196}\##?#PCI#VEN_1274&DEV_1371&SUBSYS_13711274&REV_02#3&61AAA01&0&90#{6994ad04-93ef-11d0-a3cc-00a0c9223196}\#Wave\Device Parameters\B
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Control\Lsa\SspiCache\0
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Control\Lsa\SspiCache\0
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Control\Lsa\SspiCache\0

Values added 29
—————-
HKEY_CURRENT_USER\Software\Microsoft\Windows\ShellNoRoam\MUICache "D\ir\fred-nc.bat"
Type REG_SZ
Data fred-nc
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ESENT\Process\nbtstat\DEBUG "Trace Level"
Type REG_SZ
Data
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\RFC1156Agent\CurrentVersion\Parameters "TrapPollTimeMilliSecs"
Type REG_DWORD
Data 98, 3A, 00, 00
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\Root\LEGACY_WMIAPSRV "NextInstance"
Type REG_DWORD
Data 01, 00, 00, 00
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\Root\LEGACY_WMIAPSRV\0000 "Class"
Type REG_SZ
Data LegacyDriver
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\Root\LEGACY_WMIAPSRV\0000 "ClassGUID"
Type REG_SZ
Data {8ECC055D-047F-11D1-A537-0000F8753ED1}
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\Root\LEGACY_WMIAPSRV\0000 "ConfigFlags"
Type REG_DWORD
Data 00, 00, 00, 00
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\Root\LEGACY_WMIAPSRV\0000 "DeviceDesc"
Type REG_SZ
Data WMI Performance Adapter
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\Root\LEGACY_WMIAPSRV\0000 "Legacy"
Type REG_DWORD
Data 01, 00, 00, 00
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\Root\LEGACY_WMIAPSRV\0000 "Service"
Type REG_SZ
Data WmiApSrv
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\Root\LEGACY_WMIAPSRV\0000\Control "*NewlyCreated*"
Type REG_DWORD
Data 00, 00, 00, 00
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\Root\LEGACY_WMIAPSRV\0000\Control "ActiveService"
Type REG_SZ
Data WmiApSrv
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\RemoteAccess\Performance "Error Count"
Type REG_DWORD
Data 02, 00, 00, 00
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\WmiApSrv\Enum "0"
Type REG_SZ
Data Root\LEGACY_WMIAPSRV\0000
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\WmiApSrv\Enum "Count"
Type REG_DWORD
Data 01, 00, 00, 00
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\WmiApSrv\Enum "NextInstance"
Type REG_DWORD
Data 01, 00, 00, 00
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\Root\LEGACY_WMIAPSRV "NextInstance"
Type REG_DWORD
Data 01, 00, 00, 00
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\Root\LEGACY_WMIAPSRV\0000 "Class"
Type REG_SZ
Data LegacyDriver
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\Root\LEGACY_WMIAPSRV\0000 "ClassGUID"
Type REG_SZ
Data {8ECC055D-047F-11D1-A537-0000F8753ED1}
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\Root\LEGACY_WMIAPSRV\0000 "ConfigFlags"
Type REG_DWORD
Data 00, 00, 00, 00
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\Root\LEGACY_WMIAPSRV\0000 "DeviceDesc"
Type REG_SZ
Data WMI Performance Adapter
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\Root\LEGACY_WMIAPSRV\0000 "Legacy"
Type REG_DWORD
Data 01, 00, 00, 00
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\Root\LEGACY_WMIAPSRV\0000 "Service"
Type REG_SZ
Data WmiApSrv
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\Root\LEGACY_WMIAPSRV\0000\Control "*NewlyCreated*"
Type REG_DWORD
Data 00, 00, 00, 00
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\Root\LEGACY_WMIAPSRV\0000\Control "ActiveService"
Type REG_SZ
Data WmiApSrv
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RemoteAccess\Performance "Error Count"
Type REG_DWORD
Data 02, 00, 00, 00
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WmiApSrv\Enum "0"
Type REG_SZ
Data Root\LEGACY_WMIAPSRV\0000
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WmiApSrv\Enum "Count"
Type REG_DWORD
Data 01, 00, 00, 00
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WmiApSrv\Enum "NextInstance"
Type REG_DWORD
Data 01, 00, 00, 00

Values changed 6
—————–
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography\RNG "Seed"
Old type REG_BINARY
New type REG_BINARY
Old data 39, 47, 28, CD, 9D, 0C, C1, 8C, E6, 81, 3A, B6, 4D, 38, E5, EF, 3F, DC, 11, BE, 03, 41, 7B, BE, 06, B7, D4, 6E, 20, 43, DF, E3, 15, F9, 77, CA, 42, D6, 2E, AE, 21, 2D, 52, 62, B7, FE, A7, C0, BF, 42, C0, 05, 9F, 5A, ED, 8A, FA, 90, 16, 79, 1E, 4F, 2E, 96, 5B, 45, 0D, D9, DE, AA, DD, 3C, 4E, 3E, 06, 20, 6D, 52, F8, 62
New data 5B, 34, B6, 54, 13, C9, B9, 62, D6, 2B, BF, 7D, 56, 65, 96, 6D, 03, F7, A7, 4F, 3A, 76, 1C, 01, E2, 01, 81, 4D, F0, 51, 4F, 9D, B5, 41, C3, C6, 57, 36, E1, 9F, 72, 6F, 64, 9E, 3D, 0F, 10, 8B, 07, 7E, 61, FE, F0, 05, F4, F2, B5, 26, F8, 1B, DA, A1, B5, 97, 13, 53, 6B, 59, 0A, B6, D9, 31, 60, 09, 75, 8B, B2, CA, BE, 30
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WBEM\PROVIDERS\Performance "Performance Refreshed"
Old type REG_DWORD
New type REG_DWORD
Old data 01, 00, 00, 00
New data 00, 00, 00, 00
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Prefetcher "TracesProcessed"
Old type REG_DWORD
New type REG_DWORD
Old data 5E, 00, 00, 00
New data 7A, 00, 00, 00
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Prefetcher "TracesSuccessful"
Old type REG_DWORD
New type REG_DWORD
Old data 45, 00, 00, 00
New data 48, 00, 00, 00
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\ServiceCurrent "(Default)"
Old type REG_DWORD
New type REG_DWORD
Old data 0E, 00, 00, 00
New data 0F, 00, 00, 00
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\ServiceCurrent "(Default)"
Old type REG_DWORD
New type REG_DWORD
Old data 0E, 00, 00, 00
New data 0F, 00, 00, 00
————————————————————
Disk contents
*************

Drives tracked 1
—————–
* c\

Folders added 5
—————-
c\Documents and Settings\forens\Application Data\Microsoft\Crypto
c\Documents and Settings\forens\Application Data\Microsoft\Crypto\RSA
c\Documents and Settings\forens\Application Data\Microsoft\Crypto\RSA\S-1-5-21-1409082233-1450960922-839522115-1003
c\Documents and Settings\forens\Application Data\Microsoft\Protect
c\Documents and Settings\forens\Application Data\Microsoft\Protect\S-1-5-21-1409082233-1450960922-839522115-1003

Files added 4
————–
c\Documents and Settings\forens\Application Data\Microsoft\Crypto\RSA\S-1-5-21-1409082233-1450960922-839522115-1003\91f0bb26de1ad5ccd985d6616a71fdfd_b76ba155-6c8f-4ed7-887b-5ad639f2b5fd
Date 1/5/2006 1205 AM
Size 1,715 bytes
c\Documents and Settings\forens\Application Data\Microsoft\Protect\CREDHIST
Date 1/5/2006 1205 AM
Size 24 bytes
c\Documents and Settings\forens\Application Data\Microsoft\Protect\S-1-5-21-1409082233-1450960922-839522115-1003\82a790ae-8ecb-4ca7-9d2f-067c48a2cd54
Date 1/5/2006 1205 AM
Size 388 bytes
c\Documents and Settings\forens\Application Data\Microsoft\Protect\S-1-5-21-1409082233-1450960922-839522115-1003\Preferred
Date 1/5/2006 1205 AM
Size 24 bytes

Files changed 9
—————-
c\Documents and Settings\forens\NTUSER.DAT.LOG
Old date 1/5/2006 1205 AM
New date 1/5/2006 1205 AM
Old size 1,024 bytes
New size 1,024 bytes
c\WINDOWS\Prefetch\CMD.EXE-087B4001.pf
Old date 1/5/2006 1201 AM
New date 1/5/2006 1205 AM
Old size 11,450 bytes
New size 11,900 bytes
c\WINDOWS\Prefetch\WMIPRVSE.EXE-28F301A9.pf
Old date 1/4/2006 1155 PM
New date 1/5/2006 1205 AM
Old size 41,518 bytes
New size 41,120 bytes
c\WINDOWS\system32\wpa.dbl
Old date 1/2/2006 423 PM
New date 1/5/2006 1205 AM
Old size 2,206 bytes
New size 2,206 bytes
c\WINDOWS\system32\config\software.LOG
Old date 1/5/2006 1204 AM
New date 1/5/2006 1205 AM
Old size 1,024 bytes
New size 1,024 bytes
c\WINDOWS\system32\config\system.LOG
Old date 1/4/2006 1159 PM
New date 1/5/2006 1205 AM
Old size 1,024 bytes
New size 1,024 bytes
c\WINDOWS\system32\wbem\Logs\FrameWork.log
Old date 12/14/2005 524 PM
New date 1/5/2006 1205 AM
Old size 2,885 bytes
New size 3,145 bytes
c\WINDOWS\system32\wbem\Logs\wbemess.log
Old date 1/4/2006 1158 PM
New date 1/5/2006 1205 AM
Old size 49,501 bytes
New size 49,693 bytes
c\WINDOWS\system32\wbem\Logs\wmiprov.log
Old date 1/2/2006 434 PM
New date 1/5/2006 1205 AM
Old size 655 bytes
New size 948 bytes
————————————————————
INI file
********

Ini files tracked 4
——————–
* C\boot.ini
* c\windows\control.ini
* c\windows\system.ini
* c\windows\win.ini
————————————————————
Text file
*********

Text files tracked 2
———————
* c\windows\system32\autoexec.nt
* c\windows\system32\config.nt
————————————————————
InCtrl5, Copyright © 2000 by Ziff Davis Media, Inc.
Written by Neil J. Rubenking
First published in PC Magazine, December 5, 2000.

Harlen, sorry I havn't posted the tools list, I'm working out on site and getting home late each night.

Just in response to Hogfly's comments; in a corporate setting where LE are not involved, what the client will accept can be very different. Perhaps they are concerned that a person is passing confidential data out of the company for example. If I come along and say 'OK chaps just going to shut down your companies mail server for 1/2 a day' they will freak. Often server based investigation where the client refuses down time, does not leave you with many options.

Hogflys comment on case law. I'm not sure about here in the UK. I've done maybe 50 live investigations and none have ever ended up in Court or Tribunal, as evidence found has often been used to quietly dismiss a person as few businesses want the 'noise' of a public case.

I think if a case did go to Court a good defence expert could cast enough doubt over changes that 'could' have been made, as to make it tricky. You would almost need to test the tools in an indentical enviroment and track the changes before going on site for the live examination. Impractical and virtually impossible in the real world.

Cheers

Nick

Nick,

The corporate comments are understood and appreciated, but what happens if there is a wrongful termination suit? The live evidence used for the firing will be called in to question. That's the main reason I'm asking for case law on this. If the current cases are allowing live evidence then I can see myself rewriting the SOP.

Re server based investigation. I have my own opinions on this and I'll share both sides of my thought process. If an environment is forensically ready[1], downtime isn't a concern because the environment can handle the downtime. ie, load balancing/redundancy. Ideal? Yes. Unreal? most of the time. So that leads us to when the client refuses downtime. In those situations you do what you can, and it would seem that a live response is all you can do. The question remains though. Is that enough to build a case?

[1] www.ijde.org/docs/04_winter_v2i3_art2.pdf

Hogfly,

It seems now that your technical reasons for not grabbing volatile data were being used to support your opinion regarding lack of case law.

I'm making an effort to determine if there is any available case law out there (US only, sorry)…I'll let you know what I find out.

Thanks,

Harlan