SortByName relaxation

[Frunobulax]

Hi,

as a C++ developer I have to work with a huge number of files -- a typical branch has 50.000-100.000 files, and the compiler will read 1.000-2.000 headers to compile a single file. Naturally "simple" defragmentation will not work in this case because classic defragmentation will still leave all those files scattered across the disc (or rather the used space). A "sort by filename" is the defragmentation of choice.

However, since every compile process creates and replaces a lot of object files, the HD will fragment extremely fast. Running a "sort by filename" takes very long time because it reorganises everything, but at the moment that's the only thing I can do.

Would it be possible to run a "consolidation" process that treats a directory just as a large file and tries to move all files in this directory instead of a complete reordering? Clearly some user input would be required to identify the directories that should be grouped. One could identify specific directories, or set a depth - say all directories that are X levels below root are consolidated. For a standard windows system one could group files starting in the second directory, so all files in an installation foo in "c:\program files\foo" would be grouped.

tv

[BloodySword]

That will be not effective, though. I think you should buy a SSD hard drive. This little buddies are VERY FAST on Writing and Reading with extremly randomly LBA locations.
Or you should simply choose an HDD wich has very good access times and you should work with these files in the very first partition of this HDD to make sure, that these files
are on the beginning of the drive and therefore the access will be significant faster.

For your purpose, defragmenting for optimizing such huge and complex amount of files and dirs is not a good idea. PErmanent defragmenting is very time intensive and at the end it is not very effective because the defragmentation takes more time than the time wich is safed for compiling your big projects. A faster HDD or even an SSD (Solid State Disc) would solve your performance losings. 


Edit:
If you still want to do such optimizations: MyDefrag is able to create zones. With DirectoryPath you can select all these files of a path to sort them by name. You can combine multiple paths to one zone by using the OR operator. SortByName will then sort the files of the dirs in one zone and brings them together. But I still think that this is not a good idea and would tend to consume more defragmentation time than time wich will be saved on compiling.

[jeroen]

Would it be possible to run a "consolidation" process that treats a directory just as a large file and tries to move all files in this directory instead of a complete reordering?

Not exactly. What you could do is make a MyDefrag script that puts all the files of a branch in a separate zone. You can then optimize and defragment with fastfill, much faster than sortby.

[BloodySword]

A cheaper way of speeding this up would be: *rataplan* A RAMDisk. You should first test how big the data amount of the temporary obj files actually is on your biggest project. Do not look at the file sizes, just look on the file sizes ON DRIVE (with cluster tips, and other overhead.) No add ca. 33% to this value and you have a save size of your
RAMDrive. So now you must have at least the double of the RAMDirve size free available RAM to work in a stable environement.

Now you can configure your compiler to use the RAMDrive as temporary directory for storing the obj files. After linking you can delete the files out of the RAMDisc if the linker didn't this automalicy.

Alhtough RAM-Modules have not a good price/GB ratio, this solution would be even cheaper if you must upgrade your RAM.

RAMDisk driver

[Frunobulax]

A cheaper way of speeding this up would be: *rataplan* A RAMDisk.

I do work on a ramdisk on the main branch (at least I store all compiler-generated files there, the sources are naturally on a hard disk). But there is only so much space.

As far as SSDs are concerned, yes, they would be nice. And I've tried to convince our IT for half a year that we need some of these buggers, so with any luck we should get some in 2011.

The speedup due to defragmentation (sort by name) is in the range between 25% and 33%, that's significant.

tv

[BloodySword]

The speed up of a defragmentation might be significant. But the effort to get this is higher and the time takes longer as the speed-up will last. So the costs of defragmenting can be higher than if you simply upgrade to SSDs.

If you still want to defrag you can test it:

Select all .cpp .h and other files and sort them by name. Then if fragmentation occurs, use fast fill with shuffling. And every month or week, repeat defragmentation with SortByName. This could get speed and the effort is lower.

[jeroen]

The speedup due to defragmentation (sort by name) is in the range between 25% and 33%, that's significant.

Time saved multiplied by salary per hour equals a nice donation?

[Frunobulax]

Well, if we manage to run defragmentation overnight then maybe we can work something out :-)
We are fighting hefty performance problems, see "Very slow performance with huge files" in the bugs section.

[jeroen]

See the How can I run MyDefrag automatically every day? page for instructions.

[Frunobulax]

See the How can I run MyDefrag automatically every day? page for instructions.

This will tell me how to reduce the MyDefrag runtime from an estimated 100 hours to 10 hours or so? ;-)

tv

[Bibby]

When I first read the topic title my initial thought was relaxing the SortBy method by adding a further parameter such as the following:

FileSelect
  ....
FileActions
  # Sort filenames in ascending order and only move the next consecutive filename if more than 1% of volume away from the previous file
  SortByName(Ascending, 1%)
FileEnd

So for example if b.jpg is over 60MB in distance (on a 6GB volume) from a.jpg then it will move it as normal - otherwise if it was less than 60MB distance it can leave it in place allowing the SortBy method to work faster.

I suppose the only problem with this would be that b.jpg would then be classed as unmovable otherwise it may be moved out of the way later on in the same run on that volume.

[cg]

That's a pretty cool idea.  I'd just add that it would still need to move the file if it were in a different zone but within the 60 megs... You still want your zones correct.

[jeroen]

This will tell me how to reduce the MyDefrag runtime from an estimated 100 hours to 10 hours or so? ;-)

Yes, if you use the Daily script instead of the Monthly script.