Revolutionary type of sorting

[Ophilar]

Sure, a device driver to hook file access api's, and then skim out processes such as virus scanners etc.   Consume a memory database of these file touches and then write them to disk when processing is at an idle.  However, is their likely to be a penalty in memory consumption and processing time which outweighs the benefit of the gain in disk performance?

Maybe the driver doesn't have to run all the time? As, starting with boot and running for an hour, or running every odd hour for 15 minutes, with diminishing frequency - collecting data at first every day, and then twice a week, maybe.

Shouldn't folders be treated the same as files in a database? These [more often/last] accessed should be quicker than others?

[tOM Trottier]

This is hardly a revolutionary way. Apple ProDos did it - on floppies....

I think that if there could be a way to determine the most frequently used files and puttin them on the fastest area on the disk, the improvement in performance would be more notifiable. Especially for someone like me who defrags about every 10 days.
...

Some insights:

1. You don't have to get the files in the right place each time - you can migrate the files slowly to their optimal place, in successive runs over time.

2. By placement, you are reducing the head movement to get to the data. This matters more for smaller files than big ones, usually. Big files you might as well move far away since head movement is just a minor part of the total access time.

3. The faster part of the disk should be used for the biggest active files to reduce loading time.

4. The slower part of the disk should be used for for files which are seldom accessed.

5. The middle of the disk is the ideal place for frequently accessed shorter files, like directories, pagefile, MFT,  since it is only 1/2 disk away from anywhere, and usually much less, and read time is insignificant.

So I'd suggest this change in behaviour for JKdefrag:
a) Use area 1 for big popular files (ie, have recent modification date, match a default spec list(programs, OS parts, ... ), loaded from start
b) Use area 2 for directories and small files and place it about 1/3 of the disk from the start, with all the free space on either side.
c) Use area 3 for big inactive files (load them starting at the end and working forward)
d) Each defrag, adjust area 1 and 3 to be approximately equal in size.

Could Performance Logs be used to determine:
 - commonly accessed programs/files
 - seldom accessed programs/files

tOM

[jeroen]

Some insights:

Thanks for sharing your insights, I appreciate it! Version 4 of JkDefrag will have a small scripting language that amongst other things will make it possible to define as many zone's as you want, select which files go into which zone zone, and choose an optimization per zone. I agree with you that the MFT and directories are best placed somewhat inward of the disk, called the sweet spot. I'm not sure about placing the biggest active files at the beginning of the disk, though. But you will be able to finetune to your hearts content with v4 of JkDefrag.

[Eric2]

Some insights:

Tom, sounds an interesting idea! thanks. I'd like to try it.

3. The faster part of the disk should be used for the biggest active files to reduce loading time.

Here I would like specifying myself the big files to be here (outlook pst, ost files, image database like picasa database,...) rather than leaving JKDefrag finding them itself.

4. The slower part of the disk should be used for for files which are seldom accessed.

I would also add big video files there, basically big files which are not in the faster part.

5. The middle of the disk is the ideal place for frequently accessed shorter files, like directories, pagefile, MFT,  since it is only 1/2 disk away from anywhere, and usually much less, and read time is insignificant.

well the MFT file and MFT reserved zone  location is not really a choice under XP, so I would say directories just before the MFT file and the frequently accessed shorter files just after the MFT reserved zone.  I am not sure about the pagefile.. I would probably define its size to be small or no pagefile at all if I have enough ram, its location maybe before the directories if possible..

[Lundholm]

Version 4 of JkDefrag will have a small scripting language that amongst other things will make it possible to define as many zone's as you want, select which files go into which zone zone, and choose an optimization per zone. I agree with you that the MFT and directories are best placed somewhat inward of the disk, called the sweet spot. I'm not sure about placing the biggest active files at the beginning of the disk, though. But you will be able to finetune to your hearts content with v4 of JkDefrag.

Version4 will transform this forum from a theoretical discussion club into an open war between scripts. 
It will be interesting, but I am not sure that life will be easier.

[tOM Trottier]

Version 4 of JkDefrag will have a small scripting language that amongst other things will make it possible to define as many zone's as you want, select which files go into which zone zone, and choose an optimization per zone. I agree with you that the MFT and directories are best placed somewhat inward of the disk, called the sweet spot. I'm not sure about placing the biggest active files at the beginning of the disk, though. But you will be able to finetune to your hearts content with v4 of JkDefrag.

Version4 will transform this forum from a theoretical discussion club into an open war between scripts. 
It will be interesting, but I am not sure that life will be easier.

Then the "shoulds" become "coulds"  - and everyone can try different methods, see how they work, and share, rather than just making suggestions.

tOM

[Lundholm]

Then the "shoulds" become "coulds"  - and everyone can try different methods, see how they work, and share, rather than just making suggestions.

Yes, and the designers with less than perfect scripts will feel that their systems become very slow, and they will hear a lot of disk activity. They will see their disk heads crash, and they will blame Jeroen, as always. And the experts will smile - discreetly, of course. 

[Ophilar]

Survival of the fittest?

[poutnik]

I read somewhere that the file attributes are heavily cached in memory, and are only written to disk once in a while, so the last access time feature is only significant during heavy disk access like virus scans.

Putting the analysis in a driver is the best solution of course, but is it necessary?

I have read recently last access timestamps for files are hold completely in memory, and reported from memory.
Only if they differs either 30 or 60 minutes - not sure about exact number - from known disk written values,
then they are written down.

It was commented it would otherwise turn all read operations into write operations.

[poutnik]

.............. These are files that are only accessed at virus scan and such. They should be classified as spacehogs, but Jeroen's 30 day limit will never be reached, exactly because of virus scans, etc.

It depends on particular software.
Once I had found article about proper managing last access timestamp and made Avast antivirus authors not to modify scanned files.
( Or it could be AVG used previously, but Avast did not modify it )