Idea for lessening head wear maybe? help performance too?

[ToolmakerSteve]

Jeroen,

I recently had a problem (while experimenting with a commercial defrag app I had downloaded), where a drive overheated during defrag, causing a write failure. This got me pondering how to minimize drive activity.

In this particular situation, I think there was a lot of head movement of small fragments, back and forth between original and new position.

So I am wondering: can system be told to allocate most of RAM as disk cache, and then MyDefrag pre-read a bunch of tracks in order (say 1 GB on a 2 GB PC), would FSCTL_MOVE_FILE then use that cached info, so not have to seek to read it?

If so, then after that giant read, by doing the FSCTL_MOVE_FILE's in order of destination sector, would be minimal seeking needed...

Just an occasional seek to write directory and $MFT info, and to skip over data that isn't moving.

What do you think?
~Steve

[Falcon4]

If the function actually does make use of the file cache, then oh my god I would absolutely love to see this implemented. It bugs me so much to see defraggers taking file move operations in such tiny pieces on systems with huge gigs of RAM available to it. Please please tell me this is possible! *drool*

edit: My simplified interpretation would be to simply "dummy read" the file (read the file to null), immediately prior to the move operation. Since the system disk cache is ever-changing and always caches the most recent read (as long as the file being read does not exceed the available space for cache - perhaps a limit of pre-reading size should be used), the next operation - moving the file - which consists of reading then writing, should just take its reads from the cache instead. Then it would only spend its time writing the data, then updating the MFT with the new location at the end. Ah yeah, that would be a welcome - and safe - improvement! 

[Steve Coffee]

I think this is a limitation of the Microsoft defrag API.  It will only accept one file at a time and does not appear to try to coalesce any writes behind the scenes.  Even if the file read made use of the cache (which I think it does), the drive would still need to write the relocated file data, update the pointer and then write something into the NTFS log.  All of that would have to actually be written to the disk (not write cached) before the API would consider the move complete.  Only after all those writes complete would it start on the next file.

[jeroen]

can system be told to allocate most of RAM as disk cache

Steve Coffee is right. The Windows defragmentation API will only move 1 file at a time. Falcon4's idea is very interesting and I have made a note to look at it in the future.

[ToolmakerSteve]

Cool - thanks for considering this!

Falcon: nice simple variant 

Steve C: thanks for clarifying what Defrag API does. I can see why Microsoft took that conservative approach, given the potential for a disk error or system crash during defrag to randomly destroy some file.

Jereon, if you implement Falcon's idea at some point, I'm still interested in situations where there are many small files being moved a significant distance. So if the basic concept works, I'd love an optional parameter to specify a number of bytes to try to pre-read; Defrag would keep reading files to null until it couldn't read another one without exceeding this total size. Then it would do the move commands for those files.

I'm thinking this would be useful when doing a total re-sort of files. As long as the parameter was a lot smaller than the system page cache, the data shouldn't get flushed from cache, because the most recent ones would be the ones we are about to use.

---------------------------------
How I want to organize a disk:
I want to try three zones: one for recently accessed files (which I'd like to also sort by path name if possible), second sorted by path name but not spacehog, third spacehog.


==> Hmm that just made me realize that when sorting, I'd like an option that only moves files if they need defragmenting, or if they have changed zone since the last such sort. <==

That is, I don't want re-sorting to end up moving almost every file on the disk, just because a few have moved into or out of recently accessed.

==> IDEA: Move unfragmented file only if beyond a parameterized distance from its ideal location. <==

That is, if I've separated the zones by 1% on the previous sort, then on the next sort I can tell whether it has "changed zones" by whether its current position is at least 1% away from where the new sort script is telling it to go.

I'm thinking here of something inbetween the current Fast Optimize and a full Sort: How to get most of the benefit of a Sort, with only a little incremental work if not much has changed.

[Steve Coffee]

Take a look at the scripting information at mydefrag.com.  Most of what you just mentioned can be handled by modifying the default scripts. 

Several ideas similar to yours related to improving performance for sorted zones have been discussed in the forums.  They are all on hold for now as Jeroen works on stability issues for the first release.  You might try searching through the forums and extending or improving what's there.  It's ultimately up to Jeroen, but if we can come to a consensus on some of these items they might carry more weight when Jeroen is choosing what to implement next.

[ToolmakerSteve]

Take a look at the scripting information at mydefrag.com.  Most of what you just mentioned can be handled by modifying the default scripts. 

Yes.  I am using the defaults to make a sample script that will come as close as possible to what I want.

I had somewhat misunderstood what FastFill does when combined with zones; now that I have a better grasp of how to use zones I think it does essentially what I was asking for re moving data into the zone that was out of it.

=> The case where it doesn't quite do what I would like is when sorting by path. Details follow <=

Basically, I don't ever want to do a full "slow optimize" sort to a large disk, because I don't trust that much disk activity.

Eventually I'll beg for a clever way to do an "imperfect slow optimize" (a more complex algorithm that would accept some gaps and other imperfections, in order to not move everything just because a few files have changed).

But I realized that I could get a "good enough" solution if the following were possible:

Consider the scenario where a whole new folder has been added to a drive. Plus various other file changes mixed in with it.

Consider the case where that folder is not in the zone I want it to go to. In my specific case, I have a temporary area for new files in the fast area at front of drive, BEFORE the dir + MFT. Now I want the new files to be moved to their proper zone. I'd like the folder to "move as a unit" -- end up together in the zone after the optimize.

I could get that without doing a full sort, if I could do a "sort but only move if more than 1% from target location".

=> So the idea is to apply the specified sort when deciding what order to "fill" the files in. But then do something like FastFill, rather than doing a normal slow sort that moves all files. <=

So if the following new files had been created in random order:
dirA/file1
dirB/file2
dirA/file2
dirA/file1

The order gaps would be filled is:
dirA/file1
dirA/file2
dirB/file1
dirB/file2

I haven't thought about how this would interact with different size gaps; I was thinking of the large gap at the end of the files in the zone, so that all the files in a dir would end up together.

=> Maybe the solution would be an option that would treat all files in a (leaf) dir as if they were part of one giant file, refusing to put them into a smaller gap. <=

Yes, if I had that as an option for FastFill, it would accomplish my goal!!

UPDATE:
Sometimes one wants several levels of dir to be close together. So a full SortByName is a good starting point by this. But since the idea is a FastFill compromise, it is fine to drop each "leaf dir full of files" into whatever gap-in-the-zone it fits in. If this is done in SortByName order, most of the time neighboring leaf dirs will end up near each other. Good enough for a quick optimize.

~TMSteve

[jeroen]

sort but only move if more than 1% from target location

Thank you very much for sharing your idea, I appreciate it! I haven't decided yet on a "partial" sort algorithm, but I will put your idea in the hat and will think about it. In the meantime perhaps you can use the "FastUpdate.MyD" script. It is designed for daily maintenance after a full sort.

[ToolmakerSteve]

In the meantime perhaps you can use the "FastUpdate.MyD" script. It is designed for daily maintenance after a full sort.

Yes, I am happy with either FastUpdate or FastOptimize (modified to my zones) -- already MUCH more to my liking than any other defrag/optimizer's options.

This zoned, scripting driven approach is incredible.