A comparison of current disk archival tools
This is a compilation and comparison of my experience with various archival tools and methods.
(Broken in to multiple posts because the forums software barfs on the length. I didn't intend for this to turn in to a book!
)
There have been a number of other archival tools that have come and gone. The Catweasel and DiskFerret were once promising products that were intended for reading and writing archival data using a "modern" computer. But those are hard to come by, so I have not had the pleasure of working with them.
Kryoflux and SuperCard Pro:
Kryoflux: http://www.kryoflux.com/
SuperCard Pro: http://www.cbmstuff.com/proddetail.php?prod=SCP
The Kryoflux and SuperCard Pro are conceptually similar devices. They both use a programmable CPU on a small USB-attached circuit board to read the raw digital flux transition output from a standard IBM PC style (Shugart interface) floppy disk drive.
Although they do mostly the same thing, they are geared towards slightly different uses. The SuprerCard Pro is primarily geared towards duplicating copy protected disks, and the Kryoflux fancies itself as a software preservation tool.
It is possible to use these for data recovery and interoperating with vintage/legacy hardware. However this is not the focus of either device, and some "features" can get in the way of this.
Unlike a USB floppy drive, you can not just drag and drop files directly to or from a drive with these tools. They work more like a CD-ROM burner, where you prepare an entire image in advance and then write it at once, or read an image all at once. Neither of these tools even deal with file systems at all, as they are designed to read, write, and copy all disk formats - of which there are many thousands.
(Broken in to multiple posts because the forums software barfs on the length. I didn't intend for this to turn in to a book!
)There have been a number of other archival tools that have come and gone. The Catweasel and DiskFerret were once promising products that were intended for reading and writing archival data using a "modern" computer. But those are hard to come by, so I have not had the pleasure of working with them.
Kryoflux and SuperCard Pro:
Kryoflux: http://www.kryoflux.com/
SuperCard Pro: http://www.cbmstuff.com/proddetail.php?prod=SCP
The Kryoflux and SuperCard Pro are conceptually similar devices. They both use a programmable CPU on a small USB-attached circuit board to read the raw digital flux transition output from a standard IBM PC style (Shugart interface) floppy disk drive.
Although they do mostly the same thing, they are geared towards slightly different uses. The SuprerCard Pro is primarily geared towards duplicating copy protected disks, and the Kryoflux fancies itself as a software preservation tool.
It is possible to use these for data recovery and interoperating with vintage/legacy hardware. However this is not the focus of either device, and some "features" can get in the way of this.
Unlike a USB floppy drive, you can not just drag and drop files directly to or from a drive with these tools. They work more like a CD-ROM burner, where you prepare an entire image in advance and then write it at once, or read an image all at once. Neither of these tools even deal with file systems at all, as they are designed to read, write, and copy all disk formats - of which there are many thousands.
Comments
The Supercard Pro (created by Jim Drew who formerly worked for Cental Point) has a good set of track analysis and disk utilities, and it supports writing/copying disks from the main user interface. There is no command-line tool.
The Kryoflux main application is a Java program. (With a user interface that would be right at home next to Netscape Navigator 7.0) The main application has no track analysis tools and does not support writing back to a disk. To write, you must use their DTC command line tool.
*Working with disks*
Both devices normally save their data in to "raw" flux stream files. This is essentially a direct signal recording of your floppy drive's Data Out line. Before you can *DO* anything with that data, besides writing it back to disk, you must first decode the data in to an image file.
The Kryoflux software can interactively decode standard (IBM/ISO) FM, MFM, Apple DOS 3.2, 3.3 CGR, Apple 3.5" 400K/800K VBR GCR, Amiga, and various Commodore CGR formats in to unstructured sector images. It can only encode and write Amiga .adf, .g64, and and Kryoflux IPF images. Because it interactively decodes while it reads the disk, it can retry failed reads. A "preservation" stream can be created at the same time as a decoding image, ensuring that the stream image contains valid good sectors. Third party tools, such as HxC, can convert between Kryoflux stream and other image formats.
At present, the SuperCard Pro can only directly decode to .g64 and .adf and write from .g64 images. Converting other image formats to or from the SCP flux stream format is deferred to the HxC software tool.
So, to put it simply, both are a bit of a pain for interoperating with existing vintage equipment, but it can be done. Third party tools such as the HxC disk tool and the PCE emulator tools can help you encode/decode a much, much wider variety of disk and image formats. There are still a few encoding formats that are not yet recognized or supported by these tools, such as the Apple variable bit rate 400K/800K 3.5" disks, but in time these will probably get added.
*Recovering data*
The Kryoflux and SuperCard Pro have one very strong advantage that can aid data recovery: You can archive the entire disk in a single pass and analyze the contents later. This is useful for disks that are falling apart - you may only get one chance to read them. Some other disk archiving tools may keep the disks spinning for long periods of time as they try to analyze sector layout.
However, as mentioned before, these tools are not really designed for data recovery. There are a number of issues to be aware of.
- When decoding a stream in to a sector image, the Kryoflux analyzes the underlying signal. If the signal under a sector does does not look perfect, the Kryoflux will treat the sector as completely unreadable. This can happen even when a real floppy disk controller will successfully read the sector. The reasoning for this is that Kryoflux is a "preservation" tool, and only good quality images should be preserved.
- Similarly, the Kryoflux's decoding frequently interprets marginal tracks as being completely blank. In this case it will not even retry the read operation. This is partially because it does not truly know your disk's sector layout geometry and assumes erroneous tracks are copy protection.
Fortunately in both cases you can use third party tools or a real floppy controller to often successfully read the sector or track. Personally, I have gotten so tired of the Kryoflux's "you no can haz data!" attitude, I usually don't even bother to use their tools for anything other than saving the stream file.
In contrast, the SuperCard Pro dosn't even have the ability to decode streams (except for a couple of formats) and relies on third party tools anyway.
- Another interesting issue to watch out for is that using the Kryoflux or SCP, you read an entire track at once. In some circumstances, reading a single sector at a time can be more successful. While very uncommon, I have actually run in to this. On one disk, every revolution would have sector A readable and B as bad or A as bad an B as readable - never both readable. The Kryoflux preservation stream will store multiple revolutions, however its sector decoder insists that all sectors be readable in a single revolution. The PCE emulator disk tools will actually let you select which revolution from a stream file to decode, so I could manually pick out a good read for each sector from multiple revolutions. This is not an issue for PC floppy disk controllers as they read one sector at a time.
- Kryoflux will refuse to read the back of a "flippy" disk if there is not a second index hole. It will insist that you use a special "flippy modded" drive to read both sides at the same time. (A normal double sided drive can't quite do that because the tracks on the second side have a slightly different positioning) The justification for this is that factory original "flippy" application disks were usually mass produced using such modified drives, and is therefore the pedantically correct way to read them back. It also needs the index to determine your disk's exact rotational speed.
The Supercard Pro does not have this limitation.
Other options, if you are not trying to preserve original media, include punching a second index hole with a hole-punch through the jacket (not the cookie!), or removing the disk cookie and placing it backwards in another jacket.
A seperate issue is that most 1.2mb floppy drives have "smarts" built in that prevent the drive from ever entering the Ready state if no index hole is present.
The Kryoflux does not support hard-sectored disks.
- Another overly pedantic issue is that some pre-defined Kryoflux and SuperCard pro formats insist that you use a 1.2MB 96TPI drive. The most annoying part is that neither tell you this - They ignore your settings where you have TOLD it that you are using a 360K 48TPI drive, it doesn't warn you, and half way through reading you hear a loud THUNK THUNK THUNK as your drive starts banging its head. The justification for this is that factory original disks for some platforms can use a "half track" copy protection scheme and such disks were mastered using 96TPI drives.
When you know copy protection is not involved (such as writing a system boot disk), it would be easier and more reliable to just use a 48TPI drive. In some cases the usual third party tools can help work around this bullshit. Thankfully IBM PC applications can not use that kind of copy protection.
If you use a 96tpi drive to write a disk for use in a 48tpi drive you MUST first thoroughly degauss the disk.
- Another interesting issue is that Apple Prodos/Macintosh 3.5" 400K/800K floppy disks use a variable bitrate. Apple's drives accomplish this by speeding up or slowing down the drive motor. IBM PC 3.5" drives don't do that, but the Kryoflux and SuperCard pro can read and duplicate these disks by maintaining the bitrates as seen at 300RPM. While that "works", standard 3.5" floppy drives are only designed for a single bitrate per density, and some can have troubles with that.
- When duplicating disks for use in legacy/vintage systems, it is important to understand that both the SuperCard Pro and Kryoflux copies much "noise". They sample data at a very detailed level that gets very close to the analog nature of the media. Similar to copying an audio or video cassette tape, the copy will not be as clear as the original. If you made a copy of a copy of a copy of a copy, it would eventually become unreadable.
But since the content is digital, it can and should (where possible) be reduced to the lowest common denominator. For example, non-copy protected PC disk images should generally be reduced to a 360K/1.2MB/720K/1.4MB sector image or an ImageDisk image. This does require that you are familiar with content of the disk.
- Finally, if you are concerned with authenticity, the Kryoflux writes a "signature" back to duplicated disks.
There is some political tension between the Kryoflux and SuperCard Pro teams. Neither product is "open source", and each claims to have its own secret sauce. Although there seem to be more people behind the Kryoflux, both boil down to personal hobby projects.
I prefer to judge by technical capabilities, and I currently find each lacking in important areas.
Although long out of production, another tool that is well established for software archival is the Central Point Deluxe Option Board/Transcopy. This is an ISA card that acts as a pass through device for your system's floppy drives. These ocasionally pop up on eBay and usually sell for around $100. It is worth mentioning here for comparison as is it is considered somewhat of a gold standard for disk duplication and still sought after today.
Similar to the Kryoflux and Supercard pro it deals with the signal directly from the floppy drive, but uses a low sampling rate equivalent to disk's data rate. In some ways, this is closer to the kinds of techniques that commercial disk duplicators used.
Transcopy can read/write images of and duplicate low density FM, MFM, and CGR encoded floppy disks. The later versions also support reading, writing images of, and duplicating 400K/800K Apple 3.5" floppy disks. It can duplicate most copy protection schemes used on the IBM PC. Due to pressure from software vendors of the day, several copy protection methods, however, are recognized and intentionally not duplicated.
The Transcopy software does not have the ability to decode its image files in to sector images or to create Transcopy files from sector images. So like the Kryoflux and SuperCard Pro, it is not ideal for data recovery or interoperating with legacy/vintage systems.
Real FDC
There are some advantages to reading and writing data with a real floppy disk controller chip. Real FDCs read a sector at a time, as mentioned earlier, and can ocasionally have more success on a damaged disk.
Instead of poking directly at your floppy drive's raw flux data in/out signals, the IBM PC uses a floppy disk microcontroller chip. With a microcontoller chip, your CPU can simply tell this chip "here is a block of data, now go write it", while the CPU goes off and does other things. This has the advantage of being efficient in multitasking environments, and avoids system design timing dependencies.
The primary limitation of using a genuine floppy disk controller chip is that it can only read and write the low-level signal encoding methods and sector formats for which it was specifically designed.
There were many manufacturers of these microcontoller chips, and many other systems used similar chips. A few examples are the TI-99/4A, TRS-80, and most CP/M systems.
IBM PC or clones can obviously read and write any software disk designed for IBM PC systems. However since other systems, as mentioned above, used similar compatible controller chips, it can also read and and write disks for many of these systems.
When dealing with non-PC disks, there are a number of issues to be aware of.
First of all, these kinds of microcontollers can not read or write "GCR" encoding schemes used by Apple and Commodore computers. They only support "MFM" and sometimes "FM" encoding shemes.
In the context of floppy disks, "FM" (Frequency Modulation) and "MFM" (Modified Frequency Modulation) not only refer to the signal encoding of data, but also a standard low-level sector layout. This layout was originally designed by IBM for their mainframe disks, and passed along to 8", 5.25" and 3.5" disks. Floppy disk controllers that implement this standard can technically interoperate with each other at a sector level, even if the machines they are used in use different file systems.
When selecting a floppy disk controller for interoprating with other vintage/legacy systems, it is important to note features it supports. Specifically, you will want to know:
- Does it support high and low density?
- Does it support FM encoding?
- Does it support 128 Byte MFM sectors?
The IBM PC was designed only to use MFM encoding and 512-byte sectors. Chips from other vendors often maintained the internal ability to deal with these earlier standards. As such, this support is very hit-and-miss.
You can test your controller's compatibility with the ImageDisk TESTFDC tool:
http://www.classiccmp.org/dunfield/img/index.htm
On old list of tested chips is here:
http://www.classiccmp.org/dunfield/img54306/pcfdc.txt
The best floppy disk controllers are generally the National Semiconductor DP8473, often found on Adaptec AHA-1542B, AHA-1542CF, and AHA-1542CP SCSI controllers, ALi M5135 and M1543 SouthBridge motherboard chipsets, and the NS306 Super I/O motherboard chip.
It is often easier to add a fully capable FDC to a computer than it is to find one built in to a motherboard. As long as your PC has an ISA slot, and the ability to disable any integrated FDC, you can add a third party floppy disk controller card. Most ~Pentum, K6, and Athlon motherboards will have a BIOS option to disable the integrated motherboard FDC. Earlier systems may or may not provide such an option.
A common and well supported card for this job is the Adaptec AHA-1542 series. This includes the AHA-1542B, AHA-1542CF, and AHA-1542CP models.
There is a catch, with a simple solution, to be aware of.
Around the time Adaptec released these cards, Intel released their 82077AA-1, which breaks the ability to use 128 byte sectors in MFM mode. Although uncommon, some disk formats do use this. As Adaptec produced their cards, they indiscriminately used either Intel 82077AA-1 chips or National Semiconductor DP8473 chips.
So when buying one of these Adaptec cards, you must visually inspect the FDC chip. Although a sticker partially obscures the part number, if you can see a small swirly "N" logo poking out, then you are good. If you see a large "i" logo, then it is trash.
There is more discussion and some pictures here: http://www.vintage-computer.com/vcforum ... AP-adapter
A minor issue with real FDCs is that they have a slight "blind spot". They are unable to read anything for a brief moment after the index pulse occurs. Normally that is not an issue. Except for Atari disk formats, which don't use the index pulse. In this case sectors may cross the index pulse boundary and fall in this "blind spot". An interesting workaround is just to cover the index hole with a post-it note. That works in most 360K drives, but not in most 1.2mb drives. Most 1.2mb drives have various "smarts" in them that requires detection of an index pulse before the drive itself enters a "ready" state.
A similar issue is that some earlier pre-PC disk controllers could start writing the beginning of their track just a hair too soon. This results in the first sector falling partially in to the above "blind spot" when read by a PC. A common workaround for this is to slightly slow down the rotation rate of the floppy drive.
As far as I know there are no new motherboards being produced with real FDC chips.
A review of some of the last few motherboards with real FDCs is here: http://www.vintage-computer.com/vcforum ... quot-tests
At the moment, whole computer systems can commonly be found at "recyclers", garage sales, etc.
Here is a quick guide to selecting machines for disk archiving and acting as a "tweener" between newer and much older systems.
The typical ideal "tweener" machine would:
-Have any Pentium, K6, or Athlon era CPU
-Have BIOS support for *two* real, internal floppy drives.
-Have Ethernet Networking (easy to add)
-Have Windows 98SE as the primary OS for easy DOS access (ME/2000/XP are more difficult)
-Have USB ports for flash drives if you don't use networking.
-Have at least one ISA slot (depending on how you might expand it)
-Ideally the FDC should support FM encoding, but that is rather uncommon and hard to tell just by looking.
-The motherboard should use a coin cell CMOS battery instead of a Dallas or Odin integrated clock/battery chip.
These usually do not come with 5.25" drives, but you can grab a "360K" drive or "1.2mb" off of eBay. A lot of sellers want big bucks, but if you are patient, they can come up for $10-$30. Alternately, keep in mind it is often possible to attach a regular 1.44mb floppy drive to a vintage machine (Such as an IBM PC 5150) and use it at least as a 720K drive.
These systems are becoming less common, so grab them up!
So now that you have a real FDC, what do you use to copy disks?
Copy II PC+Snatchit:
http://retro.icequake.net/dob/files/bleuge/copy2pc.600/
Copy II PC from Central Point Software is probably the best tool for duplicating copy protected disks using a real floppy disk controller. It requires no special hardware. To avoid direct competition with the Central Point Delux Option Board, Copy II PC does not save images to a hard disk - it only writes them directly to another disk. However the companion "Snatchit" program enables saving images to hard disk and reading them back from hard disk. The downside is Snatchit's .cp2 file format is not considered much of a standard.
Teledisk:
http://retro.icequake.net/dob/files/ble ... edisk.215/
There are large numbers of Teledisk .TD0 archives out there, but its usage has been superseded by Imagedisk. Teledisk is a sector archiver that can detect and read any sector sizes and track layouts in either MFM or FM encodings. It does not handle copy protection, except for simplistic types.
ImageDisk:
http://www.classiccmp.org/dunfield/img/index.htm
Like Teledisk, ImageDisk is a sector archiver tool can detect and read any sector sizes and track layouts in either MFM or FM encodings. It also does not handle copy protection, except for simplistic types.
Here are some previous notes on how to convert raw sector images to ImageDisk format: viewtopic.php?t=6931
WinImage:
http://www.winimage.com/winimage.htm
A Windows-based sector copier. It can only handle standard 360K, 720K, 1.2MB, 1.44MB, and DMF disks. It also acts as a file manager, enabling file extraction from DOS disks. It does not handle DOS 1.x formatted disks, or disk for other OSes. See viewtopic.php?f=36&t=7875 for a workaround.
USB floppy drives:
USB Floppy drives are cheap lobotomized junk, but worth mentioning here since you can still buy them new.
USB floppy drives can be used with DOS if your PCs BIOS supports it. Under DOS, USB floppy drives appear as a BIOS compatible device. Tools such as ImageDisk or copy protected applications that require direct access to a floppy disk controller chip, will not function. Some functions such as formatting are ignored.
Some only support 1.44mb disks. Others support 720K and Japanese "Mode 3" disks. They will not work with disks that use non-standard or non-DOS sector geometry.
720K Support in a random specific model can be hit-or-miss. Manufacturers rarely advertise their capibilities.
Previous testing determined that of the two variety sold by Microcenter, the Bytecc drives DID NOT support 720K.
http://www.microcenter.com/product/2839 ... Disk_Drive
And a non-namebrand supports 720K and mode 3. Although it is possible they might switch models.
http://www.microcenter.com/product/4206 ... ve_-_Black
Tools for working with Flux images:
Since the Kryoflux tools leave a bit to be desired, and the SCP has almost no decoding functionality, it is often necessary to use third party stream processing tools.
HxC Disk Tool
http://hxc2001.free.fr/floppy_drive_emu ... l#download
The HxC disk tool is an image converter program designed to accompany the HxC floppy disk emulator hardware. However, it may be used on its own.
Most notably, the HxC disk tool can convert between many sector or emulator formats and Kryoflux, SuperCard Pro, or Transcopy streams.
It also presents all information graphically, which can aid understanding the layout of odd disks.
HxC is open source, and actively developed. Both a stable and beta testing version are available.
With HxC track editor's disk view, you can visually see much of the jitter and "noise" preserved in Kryoflux and SuperCard Pro stream files.
An example of a typical floppy disk with preserved sector "jitter" and noise:
Keep in mind that when copying disks with the Kryoflux or SuperCard Pro stream formats, that this jitter and noise is duplicated on to your disk.
And some disks are just trippy
PCE tools
http://www.hampa.ch/pub/pce/pre/pce-201 ... -win32.zip (Current version as of June 2017 - when it changes, browse for newer "pfi-win32" versions through http://www.hampa.ch/pub/pce/pre/ )
The "PCE Tools" are a set of three command-line tools that accompany the PCE emulator, The latest versions have added support for SuperCard Pro SCP files although the help does not mention them.
PFI.EXE - For converting and modifying "Flux" stream files
PRI.EXE - For converting and modifying PCE "Raw" (Decoded FM/MFM/GCR) disk files
PSI.EXE - For converting and modifying PCE Sector disk files.
You can use these to decode the content of Kryoflux or SuperCard Pro stream files. They are command-line only but offer more flexibility and scriptability than the HxC disk tools. They use different decoding algorithms and can often succeed where the Kryoflux DTC tool fails.
At present the PCE tools can not convert sector images back in to Kryoflux or SCP streams.
PCE and the PCE tools are open source.
An example of decoding a double-density MFM Kryoflux image, outputting to a sector image, and creating a text file with the details of what it found:
Other drive types
The above does not take in to consideration if your disks need an odd drive type. There were a variety of drive types, such as 3.25" (not a typo), 100TPI drives, and 8" drives. Those with "Sugart" style interface connectors may be attached, with the right cabling and adapters, to a Kryoflux, SuperCard Pro, or a "Tweener" PC and used with the above tools.
Here is a guide for attaching an 8" floppy drive: http://www.classiccmp.org/dunfield/img54306/cnct.htm
Conclusions
No, it's not simple :P Each of the above tools has it's own advantages and disadvantages. Thanks to the increased interest in "retrocomputing" and archival, there is increased interest in these tools.
Thanks, SomeGuy!
There is only one problem with this guide, it doesn't make a point of not using USB drives. They are NOT good for archiving in any form.
But everything else is great about it. I have no idea why no one has replied!
Although USB 3.5" drives that don't support 720k should be smashed to bits and publicly shamed.
Hi,
I don't know this before.
Can you tell me more information ?
What "signature" ? Where it is written on a floppy ?
If I want to write then redump to fix my broken disks, this seems to be a "BIG" problem !
Is it possible to disable this ? Or I need a hacked version of dtc ?
Best regard,
coolhaken
There may be a workaround for that, but I am not sure.
I have heard this can cause serious problems on 8" drives that only have 77 tracks, as the head will hit the stop and bounce back, overwriting data.
Remember, the Kryoflux folks fancy their device as a "preservation" tool. It didn't even support writing origionally. The idea is once you write data to a factory original disk, it is no longer "authentic". Plus flux-level copies may contain unintended flux "noise" that will get written back. This can be very important if copy protection is involved.
The SuperCard Pro does not have this limitation.
Of course, keep in mind that writing back to a disk will likely fail, or even make a bigger mess, if there is physical damage on the disk. On the other hand, if the original user just reformatted it to store a backup of their CGA goat porn... :P
If you are new to all of this, I would strongly recommend setting aside any bad original disks at least until you get more experience.
2015? All this time and we are STILL waiting on an update to the Kryoflux software.
Edit: Never mind, created some new images and edited the above posts.