±Partners and Sponsors

±Your Account


Nickname
Password


Forgotten password/username?


Membership:
New Today: 0
New Yesterday: 2
Overall: 27137
Visitors: 59

±Follow Forensic Focus

Join our LinkedIn group

Subscribe to news

Subscribe to forums

Subscribe to blog

Subscribe to tweets

Moving large amounts of data within the network

Computer forensics discussion. Please ensure that your post is not better suited to one of the forums below (if it is, please post it there instead!)
Reply to topicReply to topic Printer Friendly Page
Forum FAQSearchView unanswered posts
Go to page 1, 2, 3, 4  Next 
  

Moving large amounts of data within the network

Post Posted: Tue Jun 17, 2014 9:55 pm

I'm wondering what software solutions people use within their own network/lab to move around large data sets?

At the moment I'm using a simple copy&paste type method to move large amounts of data, and while relatively reliable it means I'm doubling up and sometimes have to redo the process if one of the computers crashes or the USB caddy drops out.

I'm looking at backup solutions, FTP software or any software really that operates sort of like Tera Copy, but that is designed for network data transfer.

Speed and reliability are key, and it must run on Windows 7 and be GUI based. I have neither the time nor the inclination to mess around with Linux options as I need it to work and be simple for non tech's to use as well.

The software would ideally meet the following:
MUST run on Windows 7
Setup a shared location on the server (will be a RAID 5 array)
Have the ability to send files to that network share either via some GUI or right click menu option on the client computers
Have the ability to monitor the progress
Have the ability to either auto or manually retry any failed files
Have the ability to report why a file transfer may have failed (loss of connection, long file name etc)
Have the ability to pause/resume a process
Have the ability to continue a process in case of unexpected power loss or crash

Any thoughts?  

Adam10541
Senior Member
 
 
  

Re: Moving large amounts of data within the network

Post Posted: Wed Jun 18, 2014 12:55 am

Some of what you want in standard in Windows 8 - eg pause a copy when you have multiple steams copying. Also a simple performance chart of the copy.

I often copy disk images(500GB to 2TB) and they just take time. I will interested to hear if there are better programs available

The most import aspect when copying large amounts of data is to make sure you do not 'thrash' drive by trying to read or write to different areas of the disk at the same time. With multiple copies to/from a single drive, performance is very very poor. (SSDs will probably cope better, but not have 3 or 4 TB capacity with out being RAIDED)
_________________
Michael Cotgrove
www.cnwrecovery.com
cnwrecovery.blogspot.com/ 

mscotgrove
Senior Member
 
 
  

Re: Moving large amounts of data within the network

Post Posted: Wed Jun 18, 2014 1:56 am

I've been talking to a close friend who works in IT and he had the amusing comment that if I find the "holy grail" of data transfer software to let him know as he'd be interested Rolling Eyes

There appears to be a few Linux apps out there dedicated to this I may have naively assumed that there would be something like this for Windows.

The best I've found so far (apart from good old Windows) is Comodo backup which allows what they call a 'simple file copy' mode which basically just copies the files to the location of your choice.

I've done some testing on small data sets to confirm it works as expected, so the next test I guess will be a real world comparison on a large data set to see if it's actually saving me any time.....the only problem is finding time to do that Shocked

I may have to lower my expectations, I guess I'm just not overly keen on Windows as if something fails or I lose power it's difficult to pin down exactly why a copy failed, not to mention if I select 1TB of data for copy Windows sits there for a good 20 mins or longer just 'finding files' before it even starts the copy process, this causes a red film to wash down my eyes and exposes episodes of wild rage.....not healthy. I'm sure there is a good reason why Windows needs to 'discover' the files before it transfers them but it still bugs the hell out of me Evil or Very Mad  

Adam10541
Senior Member
 
 
  

Re: Moving large amounts of data within the network

Post Posted: Wed Jun 18, 2014 4:17 am

I believe that it depends also of the "nature" of the "data set".

If I recall correctly some tests made some time ago a same app that dealt fine (fast) with large files, then had issues (slower) with zillions of small files and viceversa.

But Robocopy (MS) is usually fine and there are GUI's for it.
Or Richcopy (still MS):
technet.microsoft.com/...light.aspx

There is this tool Qcopy which was "designed" for transfer over Network (but never tested it):
www.flit.com.au/web/co...iles/qcopy

What I personally use "normally" for "backup" is an old software that has proved over the years very handy and reliable, but that may not completely fit your bill and that cannot say how (and if) it will work fine under Windows 7, Samedir:
web.archive.org/web/20...1102950594
You will have to test it as the documentation is a bit scarce, and is not the easiest to set it the way one wants it, but has quite afew nice features that I didn't find on other programs.

jaclaz
_________________
- In theory there is no difference between theory and practice, but in practice there is. - 

jaclaz
Senior Member
 
 
  

Re: Moving large amounts of data within the network

Post Posted: Wed Jun 18, 2014 6:47 am

Your requirements are pretty lofty. Let us know if you find such a program.

I just use robocopy. It's built in to Windows 7, and any examiner should be able to learn to use it with little trouble. It's slow with many very small files--just like I expect any other tool to be--but with the 2+ GB image files I'm usually transferring, it runs at near wire speed.

I used to also use NUIX Evidence Mover, but when it started crashing when copying the root of a drive, I stopped. Evidence Mover is slower than other tools because it's copying, then hashing the source, then hashing the destination.

A typical robocopy command:
Code:
robocopy /E <source dir> <destination dir> /LOG:<log file location>

Note: if you're copying from the root of an NTFS volume, use /XD to exclude the Recycle Bin, System Volume Information, and any other system directories. Windows doesn't like copying those files to a new location.

There's no pause and resume.
It will automatically retry on failures up to 1,000,000 times by default (can be modified with /R: and /W:).
Can run from any location, but since it's there by default in Windows 7, why bother.
No GUI.
It will output what file it's copying currently, which means you can sometimes tell how much time is left, but there's no explicit monitoring. Doing explicit monitoring would slow things down.
Log will show failed files.
Will skip files that already exist, which is pretty much the same as a resume process.  

Bulldawg
Senior Member
 
 
  

Re: Moving large amounts of data within the network

Post Posted: Wed Jun 18, 2014 7:33 am

I have used Teracopy, which performs a CRC check during transfer. I also like that if certain files to not transfer successfully, you can easily restart those files. Teracopy can be set to replace the standard Windows copy, so that it is used each time. I have had instances where Teracopy has caught a CRC error. When moving evidence files, it may still be prudent to verify the hash after copied.

I have also used Richcopy (the GUI for Robocopy). There are plenty more options, so I only use this in certain cases and when I wish to select multiple files or directories in different locations.

Both run fine on Windows 7.  

inspectaneck
Senior Member
 
 
  

Re: Moving large amounts of data within the network

Post Posted: Wed Jun 18, 2014 8:43 am

- inspectaneck

I have also used Richcopy (the GUI for Robocopy).

Just for the record, Richcopy is NOT a GUI for Robocopy.

It is a separate (though similar in functions/scopes) program that has also BTW it's own set of command line options.

A (IMHO good) GUI for Robocopy is CopyriteXP:
copyrite.dyndns.biz/

jaclaz
_________________
- In theory there is no difference between theory and practice, but in practice there is. - 

jaclaz
Senior Member
 
 
Reply to topicReply to topic

Share this forum topic to encourage more replies



Page 1 of 4
Go to page 1, 2, 3, 4  Next