Leave gapS for constantly growsing Data fileS

[irneb]

I've been looking for a "decent" defragger for years now. This seems to have the ability to at least be customized. Before I start learning yet another "language" I'd like to know if someone's already made such:

I've got this "unique" situation (quotations to mark sarcasm). Most of my data files have this pesky tendency to constantly "grow". So each time they're "defragged" and "placed" into a "data" section with "one gap" at the end of this entire section ... they all look neat and tidy! Then I start working on them, and within a week nearly 50% of them have at least 2 fragments. Why does this happen you ask? Well, logically as one single file grows there's now no space directly after it ... so the paragraph you typed into the Word document now becomes a new fragment of that file.

Sorry, for that. Here's my question: How can I get all files modified (say) since a month ago to be placed in such a way as to have a gap directly following each. The gap should be something like 20% of that file's size, or preferably larger as the file is newer. I don't mean a gap for growth after the datafile group, I mean a gap after EACH file in the datafile group.

Thus far the ONLY way I get this to work is to NOT use any Defragger at all, but simply have a CMD script file to move these files off to another HDD, then move them back. The NTFS seems to have such a "fragmentation-avoidance" feature already built-in. This because if I then analyze the disc I notice each file does have a "small" gap following it - same idea as with the Linux ext3/4 file system which I've never defragged in 2 years, yet if I analyze it there's 0.02% fragmented. Though it seems the NTFS is too lazy in this, if it can't find a spot for the file "soon enough" it goes and fragments the dam thing. The ext FS would even start moving other smaller files to make space for a larger - i.e. it's an "on-the-fly" de-fragmentation at worst, or at best a fragmentation avoidance feature. But seeing as this is M$, we're stuck with having to defrag our discs "manually". Though it would be nice to see a defragging program which actually works as it should, hopefully this one can be customized to do just that.

And then "just for fun" I've got a file which gets moved up and down the disc ad-infinitum ... yes By MyDefrag. Each of the "monthly" / "weekly" scripts keep on moving this file (even though it's not fragmented). It's a 150GB VDI file for a VirtualBox hard-drive, it's set to not grow to avoid any fragmentation. But each time MyDefrag runs it moves it to the front, then to the back of the disc, then back to the front, then ... You can imagine just how long that takes! This file is on my "data" HDD, so preferably it should just be moved to somewhere close to the front of the disc and LEFT there! For YEARS!

[irneb]

I've been looking through the script system. But all the functions relating to gaps are geared towards making a gap at the end of the zone. So I only see this happening by creating a new script each time through an external program, setting a zone for each file with a gap at its end. IMO this seems excessive. Here's a more explanatory comment on how I'd wish the zone calcs to happen:

Code:

# Zone1: Arranges all system files and programs as close to the start of the disc as possible.
#        Most recently accessed moved closer to the front. Omit any programs not accessed in the last 6 months.
#        Programs created in the last month to be placed at the back of this zone.
#        No gaps between these files, a gap of 10% of the disc volume at the end of this zone.
# Zone2: All other files accessed in the last 6 months, but not modified for 6 months to be placed in a
#        contiguous block with a 5% gap at the end
# Zone3: All files modified in the last 6 months placed in a non-contiguous block. Gaps between files to be
#        calculated by ((1 / (Today'sDate - ModifiedDate)) * 0.2 * FileSize).
# Zone4: All other files neither created/modified/accessed in the last 6 months to be moved to the end of the disc

AFAICT only Zones 1, 2 & 4 are currently possible. Zone 3 (i.e. recently modified files) is either extremely difficult (even with an external program) or totally impossible. Oh well, back to my old method!

[Darlis]

Creating a gap behind files is not possible with MyDefrag (and any other defragmenter I know of). And there is a reason as to why this is so: The NTFS driver doesn't care about these gaps. NTFS is optimized for performance, so it writes files there where currently I/O operations are going on.

To do what you want, the program itself would need to have the logic to place new data directly behind the existing file or use a driver that catches write operations and redirects them (like Diskeeper does). Note that most file types (like "Office" files) are usually completely rewritten into a new file when saved, not appended to existing ones.

Also see Request new AddGapPerFile( regexp, gapsize ) action and Placing space after files.

And then "just for fun" I've got a file which gets moved up and down the disc ad-infinitum

If there is a gap before the file, where no other file can be placed into, MyDefrag will move this file away and fill up the the gap with other files. That's the WithShuffling option for FastFill that is used in the weekly scripts.

[irneb]

Yep, that's what I was afraid of. As for files being re-written, that's possibly true for most programs - not for the ones I use though (AutoCAD and Revit) which don't re-write the entire file (which can be anything from 1MB to 500MB or more in size). Rather they change and append like a database file.

I was just hoping there some way of getting Winblows to actually keep itself organized. Perhaps I should just run a linux server to have my "data disc" on - just need a bit more speed through that network cable though! The 1GB/s isn't as fast as a SATA2/3 connection.

I've tried Diskeeper as well, it doesn't do the job either! And it slows the PC down worse than any CPU hogging AV I've tried.

About the fun file , no, there's no gap whatsoever before it. As I've stated, I run MyDefrag over-night on that disc using the Data Disc Monthly (end-tasked everything I could in Task Manager so there's as little as possible going on as well). Then the following morning I run it again (just to check-see) ... lo-and-behold it starts moving that 150GB file from where it placed it not 5 hours ago (without anything touching it or any other files on that entire disc, besides MyD ) to somewhere after the last file on the disc. Then it decides to leave it there and move other files into the spot where it used to reside. If I then leave it to complete, that file is again moved to somewhere at the middle/end of that group of files. OK, so now I'd expect it's finally found its spot! Now, I'll run MyD again (same script) and it moves the other files away just so it can place that 150GB file back to the place it had it before (i.e. at the start of the disc)!

This seems extremely strange. I've looked at that script. It only has 3 zones: MFT, Directories and then all files sorted by name. The name certainly didn't change. I definitely didn't delete/rename/add any file whatsoever (unless I've sleepwalked  ). But at least I can figure out a way to keep such files in a separate zone or omitting them from any movement! That's one nice thing about MyDefrag as opposed to most others I've tried!

Oh well ... seems I've got my answer about trying to avoid immediate fragmentation after defragging: No, there's NO WAY! Thanks anyhow.  I think I'll keep my system disc defragged regularly, though few things change on it except perhaps settings files. The data disc will have to be done a different way, I'm still open to suggestions on that - wish I could just run ACad/Revit on linux and be done with this failure to perform LooseDows.

[sbrads]

And then "just for fun" I've got a file which gets moved up and down the disc ad-infinitum ... yes By MyDefrag. Each of the "monthly" / "weekly" scripts keep on moving this file (even though it's not fragmented). It's a 150GB VDI file for a VirtualBox hard-drive, it's set to not grow to avoid any fragmentation. But each time MyDefrag runs it moves it to the front, then to the back of the disc, then back to the front, then ... You can imagine just how long that takes! This file is on my "data" HDD, so preferably it should just be moved to somewhere close to the front of the disc and LEFT there! For YEARS!

Any mileage in using a 150Gb partition for this single file?

[jeroen]

I've got a file which gets moved up and down the disc ad-infinitum ...

It can sometimes happen, for several reasons. To see exactly what is happening you could look in the debug logfile. Perhaps you can avoid the problem by modifying the script and placing the file in a zone all by itself.

[irneb]

Any mileage in using a 150Gb partition for this single file?

That's probably the better solution for that problem! Then with some fiddling I could even have a dual-boot instead of a VM, though I like the idea of having the VM "appear" to work as if it's part of Win7 (difficult to get XP to work as dual-boot AND vm from the same partition - have done so previously but with Ubuntu as the guest/dual-boot with XP as host). It's just not as "simple" to setup the VM to use a partition instead of the virtual disc file, and could be dangerous as the host OS is still able to use that same partition while the Guest OS is running on it. Will probably also need something like DriveImage to convert the V-HDD-file into a "real" HDD-partition, don't particularly want to reinstall the XP and other programs (which WILL NOT run on Win7) on that again (yet  ).

@jeroen: Yes, that's definitely why MyD is better than most other defraggers out there. Customizable, at least to a point. But this is a special case, unlike my problem with files becoming fragmented the minute I start working on them.

My original problem appears that it's more about MS's filesystems than defraggers. NTFS (but especially FAT) is just not cut out to use with these types of files (and IMO any files whatsoever). And unfortunately I can only get ext2 running decently on Windows - not the best other FS out there. It would have been nice to have either ext4 or ZFS running for these files. Hang on a sec, I'll try installing a NetBSD VM and mapping a HDD or Partition to that VM - that way I can have it running inside Win7 and then share it's FS as a virtual-server to win (the NetBSD because it comes standard with ZFS and probably most efficient *nix RAM/CPU-wise, it's usually what's installed on NAS boxes). And since it's a VM-Network on the same PC the speed shouldn't be much different from a normal SATA connection.

Uhmm, will have to figure out something here. I haven't shared a folder out from a guest OS before, only from the host OS into the guest. And no, I can't have Win7 as the guest - due to the CPU and GPU hungry programs it's not an option.

[Kasuha]

And then "just for fun" I've got a file which gets moved up and down the disc ad-infinitum ... yes By MyDefrag. Each of the "monthly" / "weekly" scripts keep on moving this file (even though it's not fragmented). It's a 150GB VDI file for a VirtualBox hard-drive, it's set to not grow to avoid any fragmentation. But each time MyDefrag runs it moves it to the front, then to the back of the disc, then back to the front, then ...

It's possible that you sometimes use that file soon after you started your PC and sometimes you only use it after long time - so the file sometimes gets to the layout.ini (and gets optimized to the beginning of the disk) and sometimes drops out (and gets optimized to spacehogs). Standard scripts really love to do this with files and I hated it a lot.

I've fixed this for myself by narrowing the boot zone file selection - I have one zone instead of standard two and its fileboolean looks like this:

    ImportListFromProgramHints("NTOSBOOT*.pf")

instead of standard

    ImportListFromBootOptimize()
... and in next zone
    ImportListFromProgramHints("*.pf")

I might be losing some performance on not doing optimizations for all .pf files but I don't feel any problems and the system boots up nicely.

Another option is just to exclude this file from that zone manually or to exclude any files over some reasonable size from these two zones.

[irneb]

Yes, that is what I was getting to as well about MyD being a better defrag than most. Thanks for the alternative script though! But I think I'm going to follow sbrads suggestion of moving that VM file to a partition of its own. Either leaving it as a VM or simply cloning the image onto the partition itself - still need to figure out which I'd prefer doing. The cloning might allow for a better performing dual-boot, but then running as VM could be dangerous (if not impossible).

BTW, I'm not too well versed with how a HDD's addressing works on the platters. And where exactly is the fastest access positions of the platters? The outside edges / the middle / the inner portion? I'm just guessing here, but isn't that the "true" idea about optimizing constantly accessed files to the "fastest" spot on the disc? Is the "start" of the disc say the bottom most platter or is the start spear across each platter from inside to out (or visa-versa or some other arrangement), or is there differences between different HDDs? Anyone know for sure "why" the start of the disc is considered "faster" than the rest? Perhaps this question should be in another thread though.

And then one that's totally off topic (and just a silly idea I got in my head): I've had the idea to mount a HDD/Partition as a folder of the system disc (apparently that is possible with NTFS, haven't tried yet). This would be perfect for something like the Users/Documents and Settings folder. That way you wouldn't need a C and a D drive, but when you have files in your Documents folder they actually reside on the data disc / partition (as they're "supposed" to). But then you don't need to fiddle with windows' numerous settings to get it to actually "work" like that. Anyone tried MyD on such an arrangement? Would it still work? Or does MyD need the drive letters to defrag a disc/partition? Or is this just a silly idea?

[Darlis]

And where exactly is the fastest access positions of the platters?

The fastest part is the beginning of the disk. Physically that is the outer part of the platters. By default, MyDefrag displays the beginning of the disk at the bottom of the diskmap, other defragmenters at the top. You can download HDTune and run the benchmark. You'll see how the read speed will decrease towards the end of the disk.

[irneb]

Thanks! That's succinct enough!

[alttbm]

yoou might try my new script
http://www.mydefrag.com/forum/index.php?topic=4916.msg28079#msg28079
version2