Max fragment size / induced fragmentation

[RickH]

Hi.  I'm afraid this is going to sound like blasphemy, but my idea is that a very small amount of judiciously applied fragmentation could actually be pretty useful and beneficial, when dealing with really huge files.

I've been using your program for a while now, and while I generally love it, I have noticed one nagging issue.  In these days of terabyte drives, the only reasonable way to back up a big hard drive is to another hard drive.  I have a couple of systems, and each is set to back up the other's drives.  The backup program puts all the files for a given backup operation into one giant (50-100+ GB) snapshot database file, which then grows daily with further incremental backup runs.  I also have quite a few multi-GB DVD images which have been copied to disk for faster & quieter access (and ironically, so they can be backed up, adding to the backup database size).

When I defrag my drives, these huge files play havoc with the speed and efficiency of the defrag & optimization process.  The program ends up spending by far the majority of its time just defragging and moving these huge files around, long after the rest of the disk is finished.

It seems to me that trying to keep these really huge files in one piece is really counter-productive.  The performance loss inflicted by a few fragment breaks is minuscule, and probably even unmeasurable, when dealing with a series of fragments hundreds of megabytes or longer in size.  When it takes over 15 minutes just to copy the file, who could possibly care about the few extra milliseconds caused by a few fragment breaks and extra head seeks?  The drive head is almost certainly going to be moving around occasionally due to simultaneous access to other files anyway.

Also, even if these huge files can eventually be defragmented, they mess up the optimization algorithm with huge awkward chunks of data that don't fit anywhere reasonable and just get in the way, fragmenting the free space area needlessly and making it that much harder to fit things together efficiently.

So exclude those files, you say?  There's already a switch for that?  If they're such a pain, just don't optimize them?  But the problem there is that they then do get ridiculously fragmented over time; after several weeks of backups, an unoptimized backup database can end up with hundreds of fragments, scattered all over the drive in awkward places, breaking up all the space and generally getting in the way of optimizing the other files.

I'd like to suggest that what's needed is a way to go ahead and defrag & optimize these huge files, but only to the extent that it actually helps performance, not to the point where you're welding big swatches of your hard drive clusters into unmanageable monolithic blocks.

I was thinking it wouldn't be too hard to add a "maximum fragment size" switch, which would allow setting the largest fragment size to be used on the drive, normally to something like 1 GB.  As GetFragments() builds the fragment list for the file, it could keep track of the clusters mapped so far, and when it crosses the max fragment size, artificially induce a fragment break at that spot by computing the appropriate intermediate LCN and NextVCN values, and adding an extra NewFragment struct in the linked list.

Next, TreeInsert() (or a wrapper for it), when seeing that the file being inserted is longer than the max fragment size, could clone the file item into multiple instances, each containing one max fragment size's worth of fragment list items, and inserted separately into the tree.

From that point on, defragging and optimization would continue as normal, but the huge files would be treated like a bunch of separate 1 GB files, which could be defragged into whole 1 GB pieces and fitted conveniently together with other files when optimizing.

Huge database files like my backups that are mostly changed by appending would be especially efficient this way, since only the tail end would need to be defragged and/or moved on subsequent runs.  Most of the 1 GB fragment pseudo-files created would be defragged and optimized once, and then be allowed to sit quietly from then on.

So, waddaya think?  Pointless?  Silly? Promising? Needs work?

[jeroen]

Thank you very much for sharing your idea, I appreciate it! Microsoft agrees with you. The build-in Vista defragger will not defrag segments that are bigger than 64 megabyte. In version 4 of JkDefrag I plan to add an option to do a "chunked" defragmentation, as you describe, that will leave fragments of a specified size alone and only join up smaller fragments. It will not be in the first version, sometime later.

[RickH]

That's very interesting about the Vista defragger; I didn't know that.  I wonder how they decided on a size of 64 MB though?  That seems too small, I would have thought you'd want the fragments to be as big as possible while still being reasonably easy to move around, maybe 1 GB or at least several hundred MB.

Anyway, I'm glad you're planning an option for leaving large fragments alone and only joining smaller ones.  That will at least avoid making the problem of too-large fragments worse by making huge ones out of reasonable size ones.  But that's only half the battle.  If my hard drive is already nicely defragmented when I create or copy a huge file (which it usually is, thanks to using your great utility regularly), then the new file is likely to start out with huge multi-GB fragments already.  I think it's also important to actively break up huge multi-GB fragments into a more reasonable size, as well as not making new huge ones.

Unless perhaps Vista already avoids making such huge fragments in the first place?  Maybe this problem will solve itself as XP is eventually replaced.  Even if so, I may have to make myself a modified version of your program to use until then, or at least until your version 4 is ready.  And thank you for making your app open source so I can do so.

[Joel Low]

I'm not 100% sure how they came up with 64MB, but I do think that it is because most modern harddisks do NOT hit a burst transfer rate of more than 40+MB/s (not the interface between the motherboard/harddisk, but the harddisk technology itself), the average I observe to be no more than 18-20MB/s. Reading a 64MB fragment would take ~3s, and after than a head seek of ~10ms would be negligible.

Joel

P/s the figures I give is not the rated harddisk figures, but rather the speed observed by the OS

[jeroen]

I think it's also important to actively break up huge multi-GB fragments into a more reasonable size, as well as not making new huge ones.

I'm not sure if I agree with that. I can see that leaving big fragments alone will speed up defragging, but I don't see any advantage in actively fragmenting big files.

[RickH]

Because it then helps tremendously with optimization and free space consolidation.  My idea is basically that it's better to defrag a 100 GB file as if it were 100 separate 1-GB files instead of as a single giant file.  You can move the 1 GB pieces around much more flexibly, allowing you to get them closer to the fast end of the disk, and to merge the free space at the slow end more effectively.  Huge files are time consuming to move and difficult to pack together; since there's no performance benefit to leaving them intact as huge files, why not treat them as a set of big but reasonable size files instead?  A single 50 GB fragment gives no advantages, and causes problems.

By breaking the huge files up into manageable pieces, you can smoothly fit all the files together into nice solid zones without large gaps.  Microsoft's goal with not merging large fragments seems to be mostly to speed up the defrag process by avoiding useless work, which makes sense.  I'm not so much trying to speed up the defrag, as to make sure that the end result is a nice well-packed and solidly optimized disk whether there are huge files present or not.

One way it would speed up defrag runs is you'd rarely need to try and move the entire huge file later for any reason.  Most of the big fragments would settle quietly into zone 3 and stay there without moving, and future optimization runs would generally only affect a small part of the file.

[jeroen]

Ah, I see what you mean now. Thanks for explaining it. I have put the idea in my hat.

[Waveweasel]

Sorry to interject, but how do I merge an allocated partition with an unallocated partition... Is this a good working command (not sure): JkDefrag.exe -a 5 -d 2 -q c: d:

[poutnik]

Your question looks little strange.
JKDefrag cannot move any data between different partitions,
only within them.

The provided command will just sequentially process partitions c: and d: