Lto Tapes Read errors

Posted by: @ma-algendy

• Most of the tapes suffer from unrecoverable read errors when trying to restore them after a while. I tried different drives to read the tapes but I reach the same result. 

If you haven unrecoverable read error, the problem is with high probability on the tape, not in your drivers. The device you use returns 'can't read this block', just as a disk may return 'can't read requested sector'.

You may be able to clean the drive -- but I assume you already have done the obvious things. You may be able tweak read parameters, such as tape speed -- check the documentation of the tape device. You may be able to change to a different device with potentially different read characteristics.  But you are unlikely to change anything fundamental by changing the driver.

Consider the physical tapes: have they been stored as they should? Have they be handled as they should?  Has the tape device been maintained as it should Have they been replaced as they should?  If they have, ... you have a best guess where the problem actually may be.  But if they haven't ... it's like trying to read punched tape, when that punched tape contains tear-outs: it can't be done.

As for skipping read errors ... that's where tech platform knowledge comes in: can your device do that? Do you need to reconfigure it manually?  or do you need to have some special reading software to do that?  Talk to the manufacturer tech support.

You may need to mention the actual tape platform you are using for better answers.

Getting locked into a 'I can do this myself' position might lead to additional wear and tear on tapes you probably don't want to damage any further. You should probably talk to someone in the data recovery business, and who know the types of tapes and equipment.

Tape drives have a lot of built in error correction, and also very comprehensive data retry routines.  If a tape hits an error it will have many attempts at reading the problem block before it calls it a day.

Sometimes, having encountered a failed block the drive will treat it as end of media and not read any further.  You can sometimes skip to blocks after the failed one, or to the next file mark.

A tape is made up of blocks, and file marks, and partitions.  Typically all blocks are the same size, but this is not always the case. Thus trying to create a tape image with a  DD type command will not always work.  LTFS uses two partitions. One has the data, the other is the data directory.

If you manage to create an image with skipped data blocks, you will then need to also have software to read this partial image and extract your required files.  Modern tape formats can be very complex - it looks as if this format will backup multiple streams at the same time.  Reading will require demultiplexing this data stream.

As with disk drives, you have tapes that sound as if they are failing - every time you try and read them there is a danger you may cause more damage.

If the data is important, a bit like the others, I'd stop "playing around" with it, and more seriously consider a data recovery company. I've not used them myself (although had colleagues who used to work for them) but I believe Kroll Ontrack are reputable / highly regarded in this field.

@athulin

Thanks for your reply.

I'm pretty sure that the problem is In the tapes not the drive. As only some tapes are experiencing these problems. However, the numbers are increasing although all these tapes are brand new, fist used and all of the are stored in dry cooled environment away from any electric field.

I was using LTO3 before 6 and 8 for years . I want to say that I did not see that amount of problems. 

I think this related to the tape vendor it self. While using lto 3 I used to use Sony tapes. Starting from lto 6 and 7, I started to use HP and Fujitsu. 

I also tried to use different drives from different vendors and I got the same error at the same sector in the tapes. So, trying to modify the read speed is interesting but I think each drive would at least has a different read speed already.

@mscotgrove

Really appreciate your valuable information.

Yes this is the case, after many read attempts. The result is that all the restore job is failing and no data is retrieved if this error was at the start of the tape. I have to try to restore each file separately, which of course not reliable at all. 

The tapes are made with Archiware P5 in most cases as we are using this software since 2009 without having any issues.i tried to contact them regarding this issue many times but the don't have much to say and the software has not much to do with the problem. 

So finding a tool to make a raw image of the tape with skipping the error is a way to solve the problem. 

I read about this software :

Kernel for Tape Data Recovery, I tried to use it but I couldn't get anything back from it and no one back there is concerned about reply my emails. 

Also, I read that cloning the tape to another one using a duplicator may solve the problem but I didn't try that yet. 

But the most important question is, does the tape vendor be the reason for all of this issue? Or why we are facing all these problems after start migrate to lto6 and change the tapes vendor. 

@Rich2005

I could not agree more, I have read alot of reviews regarding their work and they are my first choice if I need to recover important data. However, most of these tapes has another copy. But I really need to know the cause of this problem and find a solution for it. 

Posted by: @ma-algendy

@Rich2005

I could not agree more, I have read alot of reviews regarding their work and they are my first choice if I need to recover important data. However, most of these tapes has another copy. But I really need to know the cause of this problem and find a solution for it. 

I would say it's likely not just you. There's lots of stuff out there saying tapes are great and really reliable....yet whenever you get surveys of people with actual tape backups the results seem to show lots of them having problems in the end:

https://www.acronis.com/en-us/blog/posts/acronis-and-redmond-magazine-survey-tape-unreliable-cloud-backup-rise

https://www.quora.com/What-percentage-of-restores-from-a-tape-drive-fail

Now I don't know whether this is down to wear and tear of the tape drive, the tape media, environmental factors, improper usage, etc, but those numbers are very high across multiple surveys it would seem to me.

We switched to disk backups a long time ago, and whilst not completely reliable, or the cheapest, we've (touch wood) not had a problem yet, and it's far quicker / easier for me to just get a drive out of the safe, and start copying over on e-sata.

Of course that may not be practical for your situation.

Posted by: @rich2005

There's lots of stuff out there saying tapes are great and really reliable....yet whenever you get surveys of people with actual tape backups the results seem to show lots of them having problems in the end:

Count me in.

In my little experience I always found the experience troublesome (and I am not the only one here):

https://www.forensicfocus.com/forums/postid/6551956/

and  BTW I personally would put the tape backups in the category of "unreliable nonsense that IT stuff use to be able to tick the "backup done" box in their list of things to do".

I also think they do have a template to the effect of:

Dear xxxxxxxx,
today[1] we have tried accessing the backup tape duly made on xx/xx/xxxx[2], but unfortunately, no matter what we tried, all day and with several methods[3], we couldn't extract the copy of the data you asked for.

We could attempt hiring a specialized data recovery firm, however the cost would probably be in excess of several zillions[4] US dollars, so if the file is really-really necessary we will need an explicit authorization for the expense directly from the CEO .

We are sorry, but nothing more can be done with our current budget/hardware/software/resources[5].

Have a nice day,
Your friendly IT stuff

jaclaz

Notes for the layman:
[1] today represents usually a date several days/a few weeks from the request
[2] we did tick the box
[3] "all day" means here a couple hours and "several methods" means "thrice, using two different tape readers, and once even re-retensioning the tape"
[4] the higher the amount the better
[5] we need more money

I would like to contradict the 'tapes are unreliable' statement.  At one time that was very true and in the 80s and 90s we had 10s of different formats and drives, and standards.  The use of tapes was also much broader, from home users to business of any size.

The current use of tape is now almost entirely LTO, and also typically large enterprise.  Modern LTO tapes, properly stored are reliable.  The other evolution is the number of backup programs.  We now have about a hand full of programs for probably 95% of all backups. In the 1990s, there were dozens. 

Where is does leave us is the range of tapes, say from 1990 to 2010 which can still be dubious, and not reliable to read.  I think Mahmoud's tape probably fit that category, definitely the LTO-3, though I would have hoped no problems with the LTO-6

By chance, today I have been working on a 20 year old DAT DDS-4.  5GB of data saved but one failed block to be overcome.

@mscotgrove

I can't think about the reliability of the LTO backup today. I still think that it is cost effective and very reliable as well.

I started using lto backup at 2009. On that time I was asked to solve a problem with the LTO3 device with one software the company was using that day. They were facing problems with restoring data from tapes. After a search and trial and error, I found a software that was just fitting our needs and it solved all our problems.

Through a couple of years and specifically in 2011 we were able to digitize 75% of our company video library and transfer it to lto Tapes instead of digital beta cams ones. With the eruption of the 2011 revolution in Egypt and as our company was in the middle of Tahrir Square. We transferred 125 tape containing 75% of our library in a few small boxes and we could successfully restored all of them afterwards.

What I need to say is that the problem is usually in something different from one to another. You may have problems in the software, another one has a problem with the drive and other with the tapes . 

As we started to use lto6 after using LTO3 for years. We started to use a tape from a vendor which brings alot of problems to us. we discovered after checking their serial numbers that they where faked not original.

Now we are using the same software we were using since we started and we are making double backups. One on lto6 using the software and other with lto8 using Ltfs.

So, making a double backup even using the same tape generation was the key to make us sure about the backup and I think this the best approach even if you are doing backups using HDd but this will be more expensive. 

I completely understand that there is something I'm missing that causes these errors and till I find it I will still count on LTO as my favorite backup technique. 

Posted by: @mscotgrove

I would like to contradict the 'tapes are unreliable' statement. 

I believe that statements made were more like:
1) the reliability of tapes is not constant and can greatly vary (by make/type of drive, make/type of media, exact storage conditions and *what not*)
2) the expected/perceived reliability of tapes is higher than real life one
3) there is no valid data (set aside personal anecdata, armchair experts opinions and results of (dubious) surveys) about the actual reliability in real world operation.

Almost *any* study, article or whitepaper around is either baseless (too few samples/queer experimental conditions, etc.) or heavily biased (because it is made by people that actually produce tapes and drives or by the people that produce/sell alternative solutions).

Typically:
a. tape backups are the second best thing in the world after the invention of ice cream (say the tape drives and tape manufacturers)  
b. tape backups are an unreliable obsolete approach (say the hard disk manufacturers and cloud backup providers) 

In addition, there is a sort of sampling error even in the reported/documented cases.

I mean, you have 120 (say LTO) tapes (let us assume of the best technology/make and used and stored properly) covering (still say) your last ten years of backups (1 tape per month).

In most (that is 90% or 95%) cases you (fortunately) will never need to access them.

The one time you need to access one, let's say tape #37, you can recover all the data just fine.

The reasoning might then be "I took a random tape and it worked, hence I can assume that all tapes are fine, tapes are reliable".

 On the other hand, another time you need to access tape #57 and no matter what you try and do, you cannot retrieve the data.

The reasoning then might then be "I took a random tape and it didn't work, hence I can assume that all tapes are crap, tapes are not reliable". 

But in reality you took two times a 1/120 chance, once you won, once you lost, that's life.

If you had attempted recovering data from ALL the 120 tapes, and (say) 119 had no problems or (the other way round) 119 were not recoverable, then you could draw the line between reliable and unreliable.

I believe (and I may well be wrong) that it is more probable that out of 120 tapes, 110 will read fine (tapes are "reliable[1]") but - due to Murphy's Law - the data you need are in the 10 that fail.

jaclaz

[1] And here we will need to define "reliable", is 110/120=91.67% "reliable"?

Or is it 119/120=99.16%?

Or is it 120/120=100.00%? (I would call this latter "perfection")

P.S.: The board filters out the word "queer" (i.e. q u e e r ) that last time I checked wasn't a "bad" world in English, (BTW used by G.K.Chesterton in the title of one of his books):
https://www.gutenberg.org/files/1696/1696-h/1696-h.htm