Maximize RAM use

[Ivan]

What about option to maximize memory use?

Some advanced file managers has move's feature "read files to RAM buffer, then write to disk". It's great improve speed at some cases.

Option "Use all free RAM to cache": Read maximum files to RAM, then write-at-once without seek.

[poutnik]

It is good for file managers ( e.g. total commander has similar feature for moving/copying).
but MyDefrag does not move files at all. It just asks windows defragmentation API to move the file.
This API, serving to almost all defragmenters, is for them something like black box.

[Ivan]

Thanks!

[Falcon4]

However, this can still be done. If MyDefrag can be made to "pre-read" a group of files before "telling" the defrag API to move a file, Windows may cache the file in memory, preventing it from needing to constantly switch between reading and writing while it moves a file (read a few KB, write a few KB, so on and so forth). If it would pre-read the next, say, 50mb of files, then "move" them, the defrag API should be able to read the file data from disk cache and simply make it a "write" operation. Since the data move operation would still be done in the same way, it would make the process a lot quicker while keeping it just as safe as ever.

[jeroen]

If MyDefrag can be made to "pre-read" a group of files

I'm not sure if it was you or somebody else that suggested it before, but this idea is already on my wishlist. Thanks anway, it's an interesting idea.

[Falcon4]

Yeah, that was me Just wanted to remind these folk that it's possible!

Thanks for having it on your wishlist

[BloodySword]

Mhmmmmmm.... I don't think that the Defrag API is using the cache... 

[antikythera]

i personally hope the API doesn't use the cache but seeing as files are not removed from their original location until they have been completely written in the new one it should be *cough* safe enough *cough*.

I may be barking up the wrong tree but if you tell MyDefrag to read all the files it can into memory and then instruct the the defrag API, wouldn't the API need to access some of that memory as well? to defrag in this manner could force possibly excess usage of the pagefile and have a negative effect and slow the process down as the HDD would be accessed more.

[BloodySword]

I would simply hope that there will be an Update wich uses a blocksize of 10 MB, like TeraCopy does. This would speed up defragmentation not only a bit, it will speed up defragmentation such rapidly that it makes fun!!

[Kasuha]

I am pretty sure the API uses much bigger buffer. The only thing is, one call to the API can move only one file's clusters and it returns only after the operation has finished, so moving small files seems like it's unbuffered.

[BloodySword]

No! The API does NOT use a big buffer. I can hear the drive rumbling while defrag a 8 GB file. Is this file small?

[Kasuha]

For me defragmenting files in large chunks is pretty fast and quiet. But it might depend on OS version, disk internal cache and filesystem.
Defragmenting FAT volumes was always pain, even on XP. And yes, before I converted my disk to NTFS (two years ago?) it was always rumbling while defragging. I guess Microsoft decided to put only minimum necessary effort to anything about FAT filesystem, including defragmentation API.

[BloodySword]

Strange. For me it is really slow and rumbling on Vista SP2 and NTFS Partitions.

[Falcon4]

Raugh, people get this so confused it's no surprise it hasn't been implemented yet.

To cache the files before moving them would be nothing more than reading more files before writing them. It would just involve MyDefrag reading a group of sequential files (perhaps color these... grey?), then move them as normal. Perhaps the level of read-ahead could be adjustable to make the best use of free RAM.

It's been proven countless times that Windows immediately caches the most recent file-read into free RAM, pushing out old data from the disk cache as new reads occur. If you have 2gb RAM, for example, and only 800mb in use, then read a 600mb AVI movie (perhaps to scan it), there will be no more hard drive activity on subsequent passes, as the entire file is already read into the disk cache - unused system RAM. Key point: caching files will have NO EFFECT on pagefile usage. The only potential downside is if the disk cache size decreases (due to opening programs, etc.) and your read-ahead was set too high, the entire effect would be lost since the most recent file (and the first to be moved after pre-reading) was already pushed out of cache. That would mean that, well... we'd get the same (dismal) performance we get now, for that loop.

As for "oh my god what about IF THE DATA CHANGES??!"... um, dude. If Windows didn't know how to cache data that changes, your computer would be a disaster. It would do nothing special, nothing different, than what Windows already does for every other file on your PC. Writes are (almost) never cached, for more than a few milliseconds. If the data changes, it's immediately written, and the cached copy in RAM gets updated. This is all stuff that Windows already does behind the scenes.

All MyDefrag would need to do is pre-read a group of files, before having them moved by Windows. That's all. Just the same "safety" as ever. It just throws another step into the process - dummy-reading the files. All it'd do is open the file, read its contents to thin air (null), then close the file. Yes, MyDefrag would read the file into null, not its own memory cache. Windows would handle the caching since the read was made. Then, the defrag API wouldn't have to make a thousand trips back and forth, since "for some reason...", the file was already cached when it went to perform the read!

[Bibby]

Also, as Falcon suggests, there is confusion how Windows API works. The API must complete it's current task that is issued by MyDefrag before the next instruction is passed to the API. As far as I can tell Windows does not cache the writing of files during these types of low level file operation.

Therefore it would have to be programmed into MyDefrag to read-ahead and cache for speed improvements.