New Linux Tool - HDDSuperTool

maximus

Member
Hddscripttool has now become HDDSuperTool. Actually that was the first name I had for it, but I did not think it was worthy of having the word “super” in it when first released. HDDSuperTool is a scriptable tool for accessing hard drives. It is still closed source, but the free version is considered freeware (proprietary version to come in the future with more goodies). It now comes with more scripts that can do more of the things that MHDD can do, including scanning, smart data, and basic imaging. Since it is provided as an executable, it can be run from a Linux Live CD without installing. It is created in Ubuntu (Debian based), but also seems to work on other distributions such as Fedora. As long as the version of Linux supports the needed functions, it is likely to work.

HDDSuperTool can be found at:
http://www.sdcomputingservice.com

There are also a couple exciting improvements over the original hddscripttool.

First, the advanced scripting has been opened up to be free. This allows the scripts to be written in a way that can be menu driven. The scripts I am providing with it are menu driven, but can also be run separate from the command line if desired. Along with the scripting menu ability, it now has the ability to list available drives with model and serial numbers and allow choosing the drive from a menu. Please feel free to suggest additional functions you would like to see added.

Second, HDDSuperTool now has the ability to perform direct port IO. This means it can see and work with drives that Linux or other operating systems are not able to. Direct IO only works in IDE mode for SATA drives, AHCI is not supported. Some computers may not properly support IDE mode. I happen to have one of those computers, and Linux sees all SATA drives as AHCI even though it is switched in the BIOS, where MHDD is able to see them. On the plus side, I also have computers without IDE connectors on the motherboard (as many newer ones do not have this), and have installed JMicron IDE PCI adapter cards. MHDD is unable to see those, but HDDSuperTool can. When using direct IO mode, it is highly recommended to apply power to the drive AFTER Linux boots up. That way the OS doesn’t see the drive and therefore does not try to access it. Please see the warnings in the manual about direct IO before using it.

An additional bonus is that it can work with USB attached ATA drives in the normal passthrough mode. However, you are at the mercy of the USB to ATA adapter as to what commands are supported, and how it responds to those commands.

Please use HDDSuperTool with caution. It is a very powerful tool. It is also very flexible since it is script based. This means anyone that wants to learn the scripting language can write their own functions. I have documented the language as best I could. It also means that if for some reason I am no longer able/willing to support it, new scripts can still be written for it. As long as the core program is as bug free as possible, it could live on for some time. Move over, MHDD! HDDSuperTool is coming out to play!

PS
Wonder why I think it is now worthy of the word “Super”? Direct IO can do some amazing things. I have seen the potential to be able to work with and even clone drives that something like ddrescue would struggle with or totally fail. There is great potential to be able to work with some drives that would normally require an expensive hardware tool. So maybe you can look forward to a new style cloning utility in the future, although that option will not be free (but likely very affordable). Much work is needed before that will be ready…
 

maximus

Member
pclab":6svlwhgi said:
I have to test it.
How does it manage the imaging?
Right now the imaging is script based. And it is very basic, and only reads in 256 sector blocks (unless you edit the script). And when using direct IO you currently only have PIO read ability. At this time is it meant more to be a utility, and something to test with. When I add true cloning ability it will be internal and not script based.
 

maximus

Member
So the biggest interest right now seems to be the imaging ability using direct IO. I am very far away from implementing a proper cloning option (I want to have something that can compare to ddrescue). However, since it can be done at a somewhat basic level using the scripting, is that an interest right now? If it could perform a basic imaging of a drive that would normally not do well with ddrescue, even though it was linear with no fancy skipping algorithm, is that still desired? Would it be worth paying for the direct IO DMA access?

I could do a bit more with the scripted imaging. It would still only write to an image file, as it is not able to clone to another drive. No matter what, when using the scripting to image a drive, it could never truly be as fast as being a dedicated function. The more complicated the script gets, the slower it gets. This would be more noticeable on slower computers of course. I will say that the basic imaging right now is mostly comparable to ddrescue in speed on a fast computer when using options that match the operation.

Right now it only copies 256 sector blocks, so any error is the size of 256 sectors. This number can be easily changed to be lower, but setting it too low would cause great slowdown. There are a couple possible methods to better this.

First possible method: When an error is encountered, it could then switch to reading sector by sector in that error block. This would slow things down, but would not require a resume file other than to know the current position. The recovery would be done in one pass.

Second possible method: Use two passes. Perform a first pass and log all the error blocks in a binary file. That binary file could be written to after every error, or after a set amount of time like every 30 seconds. I am not sure how much that would slow it down. Then perform a second pass based on reading the binary file and processing it to get the bad blocks. This method is much more complicated, and while it may perform a first pass faster than doing method 1, it would likely add to the overall time due to the additional scripting. Plus it requires additional files, where method 1 would possibly only require one small file if automatic resume capability was desired.

From a scripting point of view, method 1 would be much easier for me to implement. Method 2 could be problematic and slow. I really never intended the scripting to be a good method of cloning a drive, but it can be done.

So what do you think? I still need to make some modifications to my licensing ability to be ready for actual trial and paid versions. But if there is interest, I will focus on that so I can get a trial version out to a few select people for testing so it can be seen if this is worthwhile right now as it is.
 

Jared

Administrator
Staff member
I think method 2 would be better as it's less likely to kill the drive part way through the imaging. Even more than just block size a good imaging tool also needs to be able to manage the read timeouts. In the first pass these should be very low like 150ms, then increased in subsequent passes. When imaging drives here with bad sectors I usually do 3-4 passes increasing the read timeout each time in increments something like this:

Pass 1: 150ms, 256 sector block size
Pass 2: 300ms, 32 sector block size
Pass 3: 450ms, 8 sector block size
Pass 4: 850ms, 1 sector block size

Not sure if it'd be possible to script that, but would impress me if it could.
 

maximus

Member
The scripting is very powerful. Could it do that? Actually, maybe, but I am not sure about about the overhead. I will have to think about that. And that could be a lot of work to put into making a script when I could be spending that time on the real thing. But I guess it comes down to being able to produce something productive in the least amount of time.
 

maximus

Member
FYI

Ddrescue 1.19 has an awesome algorithm. I actually helped a bit with that. With the correct settings, it can really get the most good data fast. If you choose good settings you can skip out of a bad head fast. I want my cloning tool to do this also, but that is way to complicated for a script. Just so you understand what I mean when I say I want to make a good cloning tool.
 

LarrySabo

Member
Being a novice, I liked the Atola Bandura's imaging strategy, where it let me choose the number of passes to make (indicative of the condition of the drive) and it used what it thought would be the most effective time-outs and number of sectors to skip for each pass. I've asked for a similar option for the DFL-DDP, but was told each drive is different so the settings for each need to be entered for each case. It would have been nice to have editable default settings for each category, so after a cursory scan for bad blocks, I could choose a category, the DDP would follow the strategy, and I could get on with other work. I have not had much success in finding out what strategies others use, so I'm interested in what strategies others follow. Incorporating such an approach into the HDDSuperTool would be a great idea, I think, if it's feasible.
View attachment Atola Bandura imaging strategy.jpg
 

Jared

Administrator
Staff member
I don't think like that chart. A timeout of 20 seconds on some drives will kill the heads long before you get a decent amount of data. 60 seconds is just excessive. My experience is that for most drives (WD & Seagate at least) if a single sector doesn't read in 850-1000ms, it's never going to read.
 

Jared

Administrator
Staff member
maximus":2qwfnjl5 said:
FYI

Ddrescue 1.19 has an awesome algorithm. I actually helped a bit with that. With the correct settings, it can really get the most good data fast. If you choose good settings you can skip out of a bad head fast. I want my cloning tool to do this also, but that is way to complicated for a script. Just so you understand what I mean when I say I want to make a good cloning tool.

The only thing it can't do is soft/hard reset a drive when it becomes unstable. I'm hoping this new tool will have that capability.
 
Top