Revolutionary type of sorting

[Esprit]

Since directories are the most accessed files on disk they should be in the centre of gravity of files. Time of reading directory is time of moving of heads, time of rotation delay and time of reading data. But directories are very small so the last is insignificant. Directories can be placed anywhere indeed. But the best place is in the middle of all files to lower the time of moving of heads. One of possibilities is before MFT because MFT grows only in one direction and MFT is also often accessed (time stamps and rights). Zone 2 could be from the start of the disk to zone 1, then MFT, then reserved space and then the rest of zone 2 and finally zone 3.

[Lundholm]

Since directories are the most accessed files on disk they should be in the centre of gravity of files.

More revolutionary than that. Directories should be in the center of zone 2, which should be sorted with most frequently used files in the center, and least frequently used at the edges.

The combined zone 1/2 package should be placed at the beginning of the disk. Zone 3 doesn't matter.

I wonder if this will be possible in version 4?

[jeroen]

I agree, the directories and the MFT should be at the centre of gravity of files, allowing for the speed differential across the disk. JkDefrag does not do that yet simply because I have not yet got around to programming it yet. Also the MFT can only be moved by the Microsoft defragmentation API on Vista, not 2000/XP/2003.

[Esprit]

Since directories are the most accessed files on disk they should be in the centre of gravity of files.

More revolutionary than that. Directories should be in the center of zone 2, which should be sorted with most frequently used files in the center, and least frequently used at the edges.

That sounds great and it's worthy of implementation but this gonna be tough.

[Lundholm]

That sounds great and it's worthy of implementation but this gonna be tough.

The most frequently used stuff is not easy - Jeroen uses most recently - but if you can design something good, then go ahead.

I have thought about the free space, and I suppose it is still a good idea to have some free workspace in the bull's eye like this: directories - surrounded by free space - surrounded by the sorted zone 2.

[Esprit]

That sounds great and it's worthy of implementation but this gonna be tough.

I have thought about the free space, and I suppose it is still a good idea to have some free workspace in the bull's eye like this: directories - surrounded by free space - surrounded by the sorted zone 2.

I take the same view.

[Ophilar]

Hi,

I think that if there could be a way to determine the most frequently used files and puttin them on the fastest area on the disk, the improvement in performance would be more notifiable. Especially for someone like me who defrags about every 10 days.
Maybe it will be possible with the scripting language in v4 as mentioned in some posts.

As a refinement to this attitude I think the sorting should consider the last access time of the most accessed files.
It may be that files were accessed many times a long time ago, and are now not of any use. This refinement would keep the currently most accessed files at the most rapid access.

Moreover, but this one is a very big problem - sorting the files by the relations between them. I mean that in sorting the connections between loading of files be made (as by data-mining) and they would be put together on the disk. It will consume a lot of time, so this would probably should be as a different option - but then, most accessed system and program files are probably not being moved around and deleted a lot so it should suffice.

Thanks for a great program indeed!!

Ron

[rbscholtus]

More revolutionary than that. Directories should be in the center of zone 2, which should be sorted with most frequently used files in the center, and least frequently used at the edges.

Nice, but when these files grow, they immediately fragment to... the outside of zone 1+2. Hence the disk would become very slow very quickly.
I fully agree though with putting MFT and directories together.

[Lundholm]

Nice, but when these files grow, they immediately fragment to... the outside of zone 1+2. Hence the disk would become very slow very quickly.

This is a possible issue, yes, and that's why I wrote reply #4 above. 

[Esprit]

The most frequently used stuff is not easy - Jeroen uses most recently - but if you can design something good, then go ahead.

There's only needed a database of files. Each running of JkDefrag would update this database with access time of files and current time. Then some smart algorithm will take all access times of all files and it will estimate frequency of use of each file. Of course JkDefrag.exe will win , but this would be quite better than only last access. Moreover there could be computed data dependecies which will move files together if they are accessed after each other quite frequently. The database doesn't need to store all files but files from zone 2.

[Lundholm]

There's only needed a database of files. Each running of JkDefrag would update this database with access time of files and current time. Then some smart algorithm will take all access times of all files and it will estimate frequency of use of each file.

Yes, this could work.
Collecting several sets of access times could give a basis for a statistical analysis. Maybe there is some textbook on the correlation between most recent and most frequent?

I suggest that this is done in a separate analysis program, which should be run at random times during idle periods. The defragger should only read the analysis file and place the files according to the calculated sequence.

[jeroen]

There's only needed a database of files. Each running of JkDefrag would update this database with access time of files and current time.

It's a great idea, but having Windows update/maintain the last access times is very slow. Vista does not do it by default, and many users turn it off on XP (the "disablelastaccess" setting, with the "fsutil" command).

The best solution - I think - is a service with a small device driver. It can keep the statistics in memory and only write them to disk every now and then. Writing a good device driver is difficult enough, but the real challenge will be using the measurement data to optimize the disk. It may sound simple enough, just sorting the most frequently used files together, but there's a little more to it than that.

[Lundholm]

It's a great idea, but having Windows update/maintain the last access times is very slow. Vista does not do it by default, and many users turn it off on XP (the "disablelastaccess" setting, with the "fsutil" command).

The best solution - I think - is a service with a small device driver. It can keep the statistics in memory and only write them to disk every now and then. Writing a good device driver is difficult enough, but the real challenge will be using the measurement data to optimize the disk. It may sound simple enough, just sorting the most frequently used files together, but there's a little more to it than that.

I read somewhere that the file attributes are heavily cached in memory, and are only written to disk once in a while, so the last access time feature is only significant during heavy disk access like virus scans.

Putting the analysis in a driver is the best solution of course, but is it necessary?

[Lundholm]

The database doesn't need to store all files but files from zone 2.

Actually, this analysis also makes it possible to re-introduce some of Jeroen's spacehogs. A lot of files will always show up at the end of the list with similar access times. These are files that are only accessed at virus scan and such. They should be classified as spacehogs, but Jeroen's 30 day limit will never be reached, exactly because of virus scans, etc.

[schitzn]

There's only needed a database of files. Each running of JkDefrag would update this database with access time of files and current time.

It's a great idea, but having Windows update/maintain the last access times is very slow. Vista does not do it by default, and many users turn it off on XP (the "disablelastaccess" setting, with the "fsutil" command).

The best solution - I think - is a service with a small device driver. It can keep the statistics in memory and only write them to disk every now and then. Writing a good device driver is difficult enough, but the real challenge will be using the measurement data to optimize the disk. It may sound simple enough, just sorting the most frequently used files together, but there's a little more to it than that.

Sure, a device driver to hook file access api's, and then skim out processes such as virus scanners etc.   Consume a memory database of these file touches and then write them to disk when processing is at an idle.  However, is their likely to be a penalty in memory consumption and processing time which outweighs the benefit of the gain in disk performance?