MyDefrag fragments written files around freespace as it runs... sometimes!

[Falcon4]

Oh man, this is frustrating. Sometimes, when I run MyDefrag on a computer, it'll run and run and run, and for some reason it'll start crapping-out strings of yellow for no apparent reason! Files that were written by MyDefrag, with inexplicable spots of free space breaking the file into fragments - the bigger the file, the more fragments!

Have a look at this screenshot (attached).

Also included is my own homebrew mod of SlowOptimize (didn't create it for distribution, so the original comment's still there). It seems to be working, so I don't think it's the script, is it? It's intended to rewrite the entire drive and require a good amount of freespace. It's also intended to provide the exact defragmentation strategy I'd been desiring for very many years as I'd seen defragmenters blindly group totally unrelated files on the disk and do nothing for performance... hopefully this script, and strategy, will do what I'm thinking of!

Anyway, what's with those holes? Even with a standard SlowOptimize strategy, I've seen it crunch away on a computer and fragment it all to hell with freespace "holes"... that HDD had some bad sectors but _my_ laptop (the one I wrote this for) is totally OK...

[Falcon4]

Maybe this will help provide some insight into its behavior. Here's the log for moving $mft (and trying to defragment it)...

Code:

19:42:47     Starting FileBoolean.
19:42:47     Finished FileBoolean.
19:42:47     Starting FileActions.
19:42:47     Calculating the end of the zone.
19:42:47       Number of clusters in selected items: 28960
19:42:47       Iteration 1: 2 unmovable clusters between 0 and 28960, ZoneEnd = 28962
19:42:47       Iteration 2: 2 unmovable clusters between 0 and 28962, ZoneEnd = 28962
19:42:47       New ZoneEnd = 28962
19:42:47     Starting to sort by Name.
19:42:47     Starting to create sorted list.
19:42:47     Finished creating sorted list.
19:42:47       I want to place Inode=0 ('C:\$MFT') at Lcn=0, 28960 clusters.
19:42:47         Testing if Inode=0 ('C:\$MFT') is already at LCN=0
19:42:47           Skipping unmovable data at LCN=0, LCN is now 2
19:42:47           Item at LCN=2 is movable.
19:42:47           No, it is at LCN=38135.
19:42:47         I need 28960 free clusters at LCN=0, calculating until where I have to vacate.
19:42:47           There are 28955 empty clusters plus 3 movable clusters between 0 and 28960.
19:42:47           Looking for the first gap after Lcn=28960.
19:42:47             Gap found: Lcn=28960, Size=9175
19:42:48           There are 9175 empty clusters plus 0 movable clusters between 28960 and 38135.
19:42:48           Status: vacating from 0 to 38135 will result in 38133 free clusters.
19:42:48           Result: I have to vacate until 38135.
19:42:48           Vacating movable items between LCN=0 and 38135.
19:42:48             I want to vacate Inode=1308 ('C:\WINDOWS\bootstat.dat') at Lcn=2, 1 clusters.
19:42:48             Looking for the first gap of 1 clusters after Lcn=38135, or the largest gap.
19:42:48               Gap found: Lcn=107414, Size=33412
19:42:48             Forcing Inode=1308 ('C:\WINDOWS\bootstat.dat') to gaps starting at LCN=107414
19:42:48               Moving Inode=1308 ('C:\WINDOWS\bootstat.dat')
19:42:48                 Moving 1 clusters to Lcn=107414.
19:42:48                 API is moving 1 clusters at offset=0 to Lcn=107414.
19:42:48                 Getting cluster bitmap:
19:42:48                   Extent: Lcn=107414, Vcn=0, NextVcn=1
19:42:48                 Testing fragmentation from 0 to 1 (1 clusters).
19:42:48                   No, not fragmented
19:42:48             I want to vacate Inode=5 ('C:\.') at Lcn=1848, 2 clusters.
19:42:48             Forcing Inode=5 ('C:\.') to gaps starting at LCN=107415
19:42:48               Moving Inode=5 ('C:\.')
19:42:48                 Could not open: [123] The filename, directory name, or volume label syntax is incorrect.
19:42:48                 Changing the item into 'not movable'.
19:42:48                 Calculating the end of the zone.
19:42:48                   Number of clusters in selected items: 28960
19:42:48                   Iteration 1: 4 unmovable clusters between 0 and 28960, ZoneEnd = 28964
19:42:48                   Iteration 2: 4 unmovable clusters between 0 and 28964, ZoneEnd = 28964
19:42:48                   New ZoneEnd = 28964
19:42:48             Finished vacating.
19:42:48         Looking for the first gap after Lcn=0.
19:42:48           Gap found: Lcn=2, Size=1846
19:42:48         Forcing Inode=0 ('C:\$MFT') to gaps starting at LCN=2
19:42:48           Moving Inode=0 ('C:\$MFT')
19:42:48             Moving 1846 clusters to Lcn=2.
19:42:48             API is moving 1846 clusters at offset=4 to Lcn=2.
19:42:48             Getting cluster bitmap:
19:42:48               Extent: Lcn=786432, Vcn=0, NextVcn=4
19:42:48               Extent: Lcn=2, Vcn=4, NextVcn=1850
19:42:48               Extent: Lcn=39981, Vcn=1850, NextVcn=28964
19:42:48               Hiding the first 4 clusters of: C:\$MFT
19:42:48                 Extent: Lcn=2, Vcn=0, NextVcn=1846
19:42:48                 Extent: Lcn=39981, Vcn=1846, NextVcn=28960
19:42:48             Testing fragmentation from 0 to 1846 (1846 clusters).
19:42:48               No, not fragmented
19:42:48           Item partially placed, 27114 clusters more to do
19:42:48           Looking for the first gap after Lcn=1848.
19:42:48             Gap found: Lcn=1850, Size=38131
19:42:48           Moving Inode=0 ('C:\$MFT')
19:42:48             Moving 27114 clusters at offset=1846 to Lcn=1850.
19:42:48             API is moving 27114 clusters at offset=1850 to Lcn=1850.
19:42:53             Getting cluster bitmap:
19:42:53               Extent: Lcn=786432, Vcn=0, NextVcn=4
19:42:53               Extent: Lcn=2, Vcn=4, NextVcn=1850
19:42:53               Extent: Lcn=1850, Vcn=1850, NextVcn=28964
19:42:53               Hiding the first 4 clusters of: C:\$MFT
19:42:53                 Extent: Lcn=2, Vcn=0, NextVcn=1846
19:42:53                 Extent: Lcn=1850, Vcn=1846, NextVcn=28960
19:42:53             Testing fragmentation from 1846 to 28960 (27114 clusters).
19:42:53               No, not fragmented
19:42:53         Testing if Inode=0 ('C:\$MFT') is already at LCN=0
19:42:53           Skipping unmovable data at LCN=0, LCN is now 2
19:42:53           Skipping item's data at LCN=2, LCN is now 1848
19:42:53           Skipping unmovable data at LCN=1848, LCN is now 1850
19:42:54           Skipping item's data at LCN=1850, LCN is now 28964
19:42:54           Yes, although there are some unmovable items between the LCN and the item.
19:42:54       Finished sorting by Name.
19:42:54     ZoneBegin is now 28964.
19:42:54     Updating list of unmovable files at: I:\Apps\MyDefrag\MyDefrag.dat
19:42:54     Loading file into memory: I:\Apps\MyDefrag\MyDefrag.dat
19:42:54       Finished updating list of unmovable items.
19:42:54     Finished FileActions.

edit: Aha! It's caused by SortByName(Ascending)... SortByName actually breaks up files over areas it says there's an unmovable file! So if an unmovable file is in the way, it will actually break up the file and put it there anyway. Aargh. I modified my script to use Defragment() and FastFill() instead of SortByName, and voila! All blue.

Strange, though... it is actually breaking it up over what appears to be freespace. Could that be caused by "hiding virtual clusters..." of the NTFS metadata?

[jimbo]

From what I have seen here, when it splits across "free space" it is actually a file that has been written after the initial analyse phase, so the space is occupied on disk, but MyDefrag has no way of knowing what by. It correctly doesn't use the space, but leaves it shown as empty.

[Falcon4]

Yeah, I found it was the logfile writing that was causing a lot of grief. The debug log file was on the same volume, so it was sort of tripping over itself! When I ran MyDefrag a second time, the holes in the map were filled (but I didn't know where they were anymore). I started MyDefrag from another drive (Flash stick) and disabled debug logging, and it ran better, but still a little clumsy (due to the forced sorting by name). Defragged using FastFill() and it worked flawlessly with zero leftover fragmented files.

[poutnik]

Splitting files across inmovables is intentionally feature of all sorting optimizations
in both Jkdefrag and MyDefrag, so it is not a bug.

You need not agree with it, but it is Jeroen's decision.

[Falcon4]

Splitting files across inmovables is intentionally feature of all sorting optimizations
in both Jkdefrag and MyDefrag, so it is not a bug.

You need not agree with it, but it is Jeroen's decision.

What an odd comment to add, and I sure sense a bit of hostility... I said it was splitting over what appeared to be free space. Read the thread before jumping on my case?

[poutnik]

What an odd comment to add, and I sure sense a bit of hostility... I said it was splitting over what appeared to be free space. Read the thread before jumping on my case?

Do not look for hostility, Falcon4, where it is not, unless you want to find it.
So do I could find it in your reply, if I want.

And do not blame people for doing, what they dont.

edit: Aha! It's caused by SortByName(Ascending)... SortByName actually breaks up files over areas it says there's an unmovable file! So if an unmovable file is in the way, it will actually break up the file and put it there anyway. Aargh. I modified my script to use Defragment() and FastFill() instead of SortByName, and voila! All blue.

Strange, though... it is actually breaking it up over what appears to be freespace. Could that be caused by "hiding virtual clusters..." of the NTFS metadata?

[jeroen]

inexplicable spots of free space breaking the file into fragments

I agree that the program is performing mysteriously and not as expected. It's only the very first beta.... I will see what I can do about the black gaps. Jimbo is correct, these areas are actually files that have been created after MyDefrag has finished analysis, such as the MyDefrag logfile. They should be colored/listed as unmovable unknown data, not as black empty space. Wrapping files around unmovable data, and thereby fragmenting that data, is normal and standard behavior. The fragments are aligned on disk and there is virtually no performance penalty. It looks ugly, but it's better than vacating more and more diskspace until there is a gap big enough for the entire file.

[Falcon4]

Regarding fragmentation, does NTFS actually load all the information about file fragments into memory before reading the file data? On heavily fragmented (>3000 fragment) DVD-data files, the data was pretty much in a continuous (yet broken) line, and yet the HDD was constantly thrashing to read the continuous file. Could it be that it has to go back to the MFT (...?) to read the additional fragments of a fragmented file?

[jeroen]

does NTFS actually load all the information about file fragments into memory before reading the file data?

I don't know, but I would expect so. Even if it doesn't then the disk cache should prevent trashing. That many fragments will cause a lot of head movement in and by itself. Or perhaps there is something else happening, such as writing to a logfile. If you really want to know then perhaps you can find out with the free Microsoft DiskMon utility.

[Mr Noodle]

If it's still on your to-do list as low priority, my apologies for bring this back up, but it's something I've been noticing for a while, and it's still the same story in beta 7, although a quick roll of the scroll wheel on the mouse fills it in even though it still comes up as empty space on the description when you hover over it.

[jeroen]

it's still the same story in beta 7

Beta 7 still writes a big debug logfile, it is tripping over itself... You can avoid it with for example the "-d 2" commandline parameter, or by adding a line "debug(3)" to your "Settings.MyD" file.

[Falcon4]

For a few betas it was working good (before full-blown debug logging was re-enabled by default...), but now it trips all over itself again, by design

Isn't there another way that debug logging can be done by default, without writing an ever-growing file? Perhaps have it write a 50mb log file "container", then run the program and have it fill that file in? That way MyDefrag itself would be aware of its own log file, and it wouldn't be fragmented a thousand ways on the drive as MyDefrag moves data around...

[jeroen]

Isn't there another way that debug logging can be done by default, without writing an ever-growing file?

Thanks for sharing your idea, I appreciate it! I will see what I can do...

[Mr Noodle]

inexplicable spots of free space breaking the file into fragments

They should be colored/listed as unmovable unknown data, not as black empty space.

That's what I was referring to. I have now put in the debug(3) into the settings script, but what I meant by  my post was that where new files have been created or free space has been taken up after Mydefrag has analysed the disk, it still comes up as free space, as opposed to red/unmovable, unless you scroll up or down with the mouse, when it then fills in the gap with blue, but it's still listed as free space when you hover the mouse over it.