Defragmenting large file

[torsten]

JkDefrag is defragmenting most files but now I have one big file that can not be defragmented.
Every run of the program takes a very long time because the fragments are moved to other clusters ervery time but never defragmented.
I think there is enough space available for defragmentation so what might be the problem?

Here is the first part of the log where you can see the details for the fragmented "file1.bin":

11:23:56 JkDefrag v3.24
11:23:56 Date: 2007/09/04
11:23:56 NtfsDisableLastAccessUpdate is inactive, using LastAccessTime for SpaceHogs.
11:23:56 Starting JkDefrag for 'z:'
11:23:56 Phase 1: Analyze
11:24:02 Phase 2: Fixup
11:26:58 Zone 1: Fast Optimize
11:26:58 Zone 2: Fast Optimize
11:26:58 Zone 3: Fast Optimize
11:26:58 Finished.
11:26:58 - Total disk space: 27595780096 bytes (25.70 gigabytes), 6737251 clusters
11:26:58 - Bytes per cluster: 4096 bytes
11:26:58 - Number of files: 18386
11:26:58 - Number of directories: 1141
11:26:58 - Total size of analyzed items: 24234418176 bytes, 5916606 clusters
11:26:58 - Number of fragmented items: 1, 0.01% of all items
11:26:58 - Total size of fragmented items: 1162158080 bytes, 283730 clusters, 4.80% of all items, 4.21% of disk
11:26:58 - Free disk space: 2133643264 bytes, 520909 clusters, 7.73% of disk
11:26:58 - Number of gaps: 340
11:26:58 - Number of small gaps: 274, 80.59% of all gaps
11:26:58 - Size of small gaps: 7467008 bytes, 1823 clusters, 0.35% of free disk space
11:26:58 - Number of big gaps: 66 (19.41% of all gaps)
11:26:58 - Size of big gaps: 2126176256 bytes, 519086 clusters, 99.65% of free disk space
11:26:58 - Average gap size: 1532.09 clusters
11:26:58 - Biggest gap: 684904448 bytes, 167213 clusters, 32.10% of free disk space
11:26:58 These items could not be moved:
11:26:58   Fragments       Bytes  Clusters Name
11:26:58           1    67108864     16384 z:\$LogFile
11:26:58           1        2720         1 z:\$MFT::$BITMAP
11:26:58           2    22233088      5428 z:\$MFT
11:26:58           1        4096         1 z:\$MFTMirr
11:26:58           1        4172         2 z:\.::$SECURITY_DESCRIPTOR
11:26:58           1      842160       206 z:\$Bitmap
11:26:58   --------- ----------- --------- -----
11:26:58           7    90195100     22022 Total
11:26:58 These items are still fragmented:
11:26:58   Fragments       Bytes  Clusters Name
11:26:58           2    22233088      5428 z:\$MFT
11:26:58           5  1218794478    283730 z:\APPS\file1.bin
11:26:58   --------- ----------- --------- -----
11:26:58           7  1241027566    289158 Total
11:26:58 The 25 largest items on disk:
11:26:58   Fragments       Bytes  Clusters Name
11:26:58           5  1218794478    283730 z:\APPS\file1.bin
11:26:58           1  1238368256    202546 z:\GAMES\555.BIN
11:26:58           1   533877888    129975 z:\GAMES\123.BIN
11:26:58           1   570236016    129639 z:\APPS\file2.bin
11:26:58           1   518436776    126572 z:\VID\CIMG0016.AVI
...

[cf]

I think there is enough space available for defragmentation so what might be the problem?

...
11:26:58 - Biggest gap: 684904448 bytes, 167213 clusters, 32.10% of free disk space
...
11:26:58   Fragments       Bytes  Clusters Name
11:26:58           5  1218794478    283730 z:\APPS\file1.bin
...

JkDefrag needs a gap that is big enough for the whole file (which is about  1.13GB).
But, as you see in your log, the biggest gap on your disk is 650MB. So, JkDefrag can only try to make the file less fragmented and so it moves quite big parts of the file around every time you start it, because there are new gaps that are big enough to join at least some of the fragments together (please correct me, if necessary).

[torsten]

Ok, that makes sense, but how can I defragment the MFT? There is enough space free on the disk but it seems the fragmented MFT is reducing the gaps.
Any idea to solve this?

[torsten]

I moved the big file to another partition to see the max gap on the drive.
It is about 2.3 GB. So it should be possible to defragment the file.
The defragmenter does not need a gap greater than its size.

Example:
 1. I have largest gap of 2.3 GB on z:
 2. I copy a file of 1.2 GB to Z:
 3. Now I have a fragmented file of 1.2 GB and may be largest gap of 0.6 GB

The defragmenter can move cluster by cluster to the beginning of the gap and of couse the framented clusters are free after moving. Do You understand?
There is no need to have a gap bigger than the file in theory.
Only the gap wich includes free space AND fragmented data must be big enough.

Here a simple graph before fragmentation:

6  FFFFFFF FFFFFF FFFFFFFFFFFFF
5  FFF FFFFFFF FFFFFFF FFFFF FF
4  FFFFFFFFFFFFFFF
3  
2  UUUUUUUUUUUUUUUUUUUUU
1  UUUUUUUUUUUUUUUUUUUUU

U is unfragmented, F is fragmented and empty is free space.

The defragmenter can move each F to line 3 and then to 4, 5 and 6.
And if there is an F from the end on the fragmented file on line 4 where it wants to move other F, the defragmenter can move this blocking F to free space in line 5 or 6.

Is this only possible in theory or also in practice?

[WHRoeder]

This discussion is exactly why DIRMS defragger defrag on the fly was written.  Yet JKDefrag is much better at eliminating the small gaps.  

But according to the information, it takes windows some time to free up clusters after a move.  Is this the reason that the sorted options move a GB worth of files out of the way instead of only enough to move the first file?

If you can't get it defragged, try the command line version of DIRMS

[jeroen]

The defragmenter does not need a gap greater than its size.

You are right, but JkDefrag is not smart enough to defrag like that. My advise is to forget about it, your file now has only 5 fragments and is only a fraction of a fraction of a percent slower that when it would be completely defragged.

how can I defragment the MFT?

At the moment JkDefrag cannot defrag the MFT. It's on my list. Try one of the commerical defraggers.....

[torsten]

Thanks jeroen and WHRoeder,

I moved about two GB of small files to another partition and then the defragmentation of the big files was possible. After that, I moved the small files back and now all files are defragmented.
Ok, it's not the most comfortable way but it works.

I didn't want to have the file defragmented because on every run  JkDefrag tried to defrag and moved the fragmented clusters to other positions (wich takes a very long time). Now a scheduled defrag call on all my partitions only takes about one or two minutes.

Great Defragmenter, this JkDefrag!

[robert]

Jeroen,

I would like to reiterate this issue, the problem isn't that jkdefrag cannot defrag a huge file the problem is that it believes that it can. It appears that it will often (always?) move a big fragmented file even if it ends up with more fragments.

In one case this slows down a quick defrag to the point where contig is faster. :-(

[jeroen]

It appears that it will often (always?) move a big fragmented file even if it ends up with more fragments.

That would be a very stupid thing to do for a defragger, wouldn't you say? No. JkDefrag will only move a file if the move reduces the number of fragments. See the Defragment() subroutine in the JkDefragLib.cpp source.

[RJARRRPCGP]

Looks like a bug was found. A boo-boo.

[robert]

It appears that it will often (always?) move a big fragmented file even if it ends up with more fragments.

That would be a very stupid thing to do for a defragger, wouldn't you say? No. JkDefrag will only move a file if the move reduces the number of fragments. See the Defragment() subroutine in the JkDefragLib.cpp source.

Yup, it was. Well, I don't have the example filesystem anymore (and it was a lot of versions ago) because I defraged it by careful use of -a6 and -a5. I do have another filesystem that looks similar and it's not 'flip-flopping'  so I'll assume that bit of code fixed it, thanks.

Anyway my current problem is around 10000 files like this:
21:01:06           1        3584         7 C:\WINNT\Installer:$I30:$BITMAP\e880e5.mst

that won't defragment but I'm trying version 3.25 tonight. After that I'm rather interested in what POS actually created these files!

[robert]

Anyway my current problem is around 10000 files like this:
21:01:06           1        3584         7 C:\WINNT\Installer:$I30:$BITMAP\e880e5.mst

that won't defragment but I'm trying version 3.25 tonight. After that I'm rather interested in what POS actually created these files!

Looking good, this one's fixed too.