Vacate files to the end of the drive, instead of end of zone

[Falcon4]

Okay, I've posted a number of things about the way MyDefrag ends up "shuffling" files from zone to zone when it needs to vacate them. But they're all a little... well... complicated. It'd involve having to add code to "predict" about where a file would end up on the drive.

But I've got an easier solution. Just friggin' move the vacated files to the end of the drive, from the end back to the top. Also, add in full defragmentation on top of that, and I swear to god MyDefrag will see a 500% speed boost. Right now it's doing so much "shuffling" it's downright depressing to watch. Having it vacate to the end of the drive would guarantee that it would only need to move each vacated file a maximum of 2 times, instead of the zone->zone->zone->destination that it's currently doing. Especially noted for the large (>100mb) files that end up placed at the end of the drive in zone 7. If it's currently in the beginning of the drive, it'll get shuffled at least 7 times! Add in the old functionality of fully defragmenting a part of a vacated file, and life would be so much nicer.

Perhaps that can be added as a per-zone option, so the script can determine where files get vacated to, at least as a temporary measure until MyDefrag gets to a point where it can "predictively vacate" files. Anything, anything at all would be better than having MyDefrag shuffle everything around and around and around, and choke every time it encounters a fragment of a 10,000-fragment compressed file. Seriously, take a 1gb video file, compress it with NTFS, then tell MyDefrag to defrag that drive. Bricks will be shat

[theantagonizer]

The user guide warns for good reason against NTFS compression of large files as serious performance issues may result. I do not use NTFS compression or the indexing service.

[Falcon4]

The user guide warns for good reason against NTFS compression of large files as serious performance issues may result. I do not use NTFS compression or the indexing service.

Good on you. But Windows does, when people run Disk Cleanup and "compress old files". So that's kinda irrelevant.

[theantagonizer]

Kind of irrelevant except for the part about the user guide warning that serious performance issues may arise with NTFS compression. I suppose what you mean is irrelevant is your point about NTFS compressed 1gb movie files and MyD choking which makes my response to it irrelevant as well. Perhaps I just dont like the word choke... Or Brick shat

I think we all agree we want MyD to do less shuffling and be faster and the files moving from zone to zone is not the "optimal" solution. Putting the vacated files at the end has drawbacks and has been discussed. I have seen your posts about improving the situation and there is many good ideas being thrown around. Mine is dont use NTFS compression or windows disk cleanup. You will see a 500000% improvement of MyD speed if you have a lot of fragmented 1gig movie files that you compress with disk cleanup...

 

[Falcon4]

Except that I don't compress them, and I don't use Windows' crappy disk cleanup. I clean up other peoples' trashed hard drives with a series of tools (some I even wrote myself) and MyDefrag, again with a script I wrote.

Also, to say that using NTFS compression causes "serious performance issues" is really a pretty dumb statement. Windows itself uses NTFS compression in various places and there is _ABSOLUTELY ZERO_ performance "penalty" for compressing large, unused files. Even if you do use those files, the performance "penalty" is nearly zero. The only performance penalty is when MyDefrag bashes its head against a wall trying to move 16kb virtual clusters at a time.

ANYWAY. Back on topic. Have you got anything to actually say about the topic itself? Moving vacated files to the end of the drive instead of shuffling? You say it's "been discussed" but I haven't seen any discussion about it...

[poutnik]

....... Moving vacated files to the end of the drive instead of shuffling?
You say it's "been discussed" but I haven't seen any discussion about it...

I would say more efficient is redesign of Mydefrag zone processing to estimate zone positions in advance
and placing such a file to its zone area. (supposing using fastfill, sorting need different approach).

[Falcon4]

Bumping this. *headdesk*

Still a highly irritating problem without even as much as a workaround. I first played around with the idea of taking the "spacehogs" first and moving them all to the end of the drive (which is typically empty on a lot of drives, or at least a lot less cluttered than the start of the drive)... but... if I make a no-vacate gap to the end of the drive, the whole thing implodes on itself - plus, MaxNextZoneBegin (I think?) makes enough room for the rest of the unplaced files - that is, the entire data of the drive. So it wouldn't even work if I wanted it to.

The problem is that MyDefrag is fundamentally flawed - it can only process a drive in a linear fashion, and when it vacates files it can't think ahead. It's very, very linear and that causes a huge overhead in defrag/shuffling time. It really needs to be able to think more than one zone ahead - hell, even thinking past the end of the current zone would be a bonus.

So, yes, this is the very, very top of my wishlist for the next version. Some sort of way to tell MyDefrag WHERE to vacate files to....

p.s.: Yes, I still use MyDefrag several times a day
edit: Attached... MyDefrag in action!

[BloodySword]

Very antique systems, you have ^^.

[jeroen]

Just friggin' move the vacated files to the end of the drive, from the end back to the top.

Thanks for sharing your idea, I appreciate it. But the end of the drive is not only the slowest part of the disk, it would also mean that the harddisk heads have to move a long way for each and every file that is being vacated. I am sorry, but your idea would make MyDefrag slower, not faster.

p.s. MyDefrag does not always vacate files to just above the end of the current zone. It sometimes vacates to just above the end of the last zone. And try to make as few zones in your scripts as possible. Vacating files will then help to fill up gaps in the next zone, making things faster.

[Falcon4]

I just saw the new MoveToEndOfDisk fileaction in 4.2.8... THANKS! That should allow me to move Spacehogs out of the way in the first action, so it doesn't keep tripping over them zone after zone after zone (move above zone 1... move out of gap... move above zone 2... move out of gap... move out of zone 3....etc.....place in zone 7). IMO that'll dramatically improve MyDefrag performance.

It's better, if the actual placement can't be predicted (scan all zones first then assign each file to a "zone" and vacate them near their final area), then moving then once to the end then once to the final zone is better than moving over and over and over. And since MyDefrag scans the diskmap for each file move, the less number of total file-move operations it needs to do, the better. I think it's all around a great idea and I'm very much looking forward to implementing it later today!

[ff_mfg]

It's better, if the actual placement can't be predicted (scan all zones first then assign each file to a "zone" and vacate them near their final area),

I sometimes wonder if it's still possible to do it DOS SpeedDisk way, or the huge amount of calculations or the size of the dataset would gobble up even modern machines. If someone doesn't remember how it was back then... the basic idea is to move the file exactly where it should go (ideally with 1 operation). So disk analyzed, all fragments analyzed and determined where what goes with minimum hassle (it would take more than 1 move for some fragments if free space is low). I realize this is a bad approach to working with mounted volumes, but as we want to speed up optimization process, sometimes stopping file io and locking volume for maintenance is an option. Then the long analyze might pay off. But I just don't know enough about NTFS and WinDefrag API to even determine if this approach is realistic...

[jeroen]

There are some practical problems there. For one thing, DOS is single tasking, Windows is multitasking. So there are unmovable files in use by other processes. Taking a disk offline is not acceptable to me. Also, I remember that DOS SpeedDisk could sometimes take a LONG time to analyze/precalculate, minutes. And that on a small harddisk disk of only a few megabytes.

[ff_mfg]

Also, I remember that DOS SpeedDisk could sometimes take a LONG time to analyze/precalculate, minutes. And that on a small harddisk disk of only a few megabytes.

Well, that's why I'm not sure it's realistic now. If calculation time grows nonlinear with dataset size for this task, then it's definitely not. If closer to linear, than with modern processors and enough memory it might pay off. Few minutes of calculation and mostly 1 move for every fragment needed to be moved would be quite faster than shuffling a lot of data a number of times.

Taking a disk offline is not acceptable to me.

That, on the other hand, puts the kibosh on the idea

Anyway, the idea with placement estimation is quite good too. I'd like to see that someday.

[Thaliur]

Maybe there is an easier solution to avoid shuffling, one that's already built into MyDefrag (and seems to be completely bug-free now): Add "DoNotVacate" to all of your gaps.
That way, if a file is in one of the desired gaps, it will stay there until it is moved by a zome command, which will really speed up things since MyDefrag doesn't have to push probably fragmented files from zone to zone while working on a disk.

This is especially useful on iPod harddrives, since those are reeeeeeeeally slow (as all Mac hard drives I saw so far, probably someone's taking saving energy serious).

[BloodySword]

Mac users are really slow, too. They don't need fast hard drives .