FastFill(DoNotVacate) and improvements to FastFill(WithShuffling)

[Kasuha]

These two suggestions are something I'm thinking about for some time already, still not perfectly sure if it's good idea to post here - especially because I guess it could be hard to implement with current MyDefrag architecture.

FastFill(DoNotVacate):

At present, the current zone space is vacated during the optimization process (SortBy or any of Fill commands) and filled files are always placed to already cleared gaps. The idea behind the suggestion is, do not vacate files from inside the zone unless FastFill really plans to place a zone file there. For example:

11aaabbcc2222--3333

(numbers are zone files, letters are non-zone files)
in this case, files aaa and bb get vacated to make room for 3333 but cc stays in place.

FastFill would need to calculate the gap size not from real gap on the disk, but instead as a space between zone files. Then it would calculate files needed to fill the gap (perfectly or best match) and only when the needed area is known, vacate the needed space and fill zone files in there.

The idea here is
- to prevent vacating files if it is not necessary and leave the job up to the appropriate zone's MoveUpToZone command
- to allow overlapping zones

Improvement to FastFill(WithShuffling):

This idea is of similar type but applies also to the zone files. In short, do not vacate zone files and extend the gap, instead just "imagine" the gap has already been extended and try to calculate filling for it without actually extending it. Only after the filling is found, start moving files and always only move file out when it needs to be moved out to make room for new file.

There's more to this, though. The set of files used to fill the gap can be reduced - any files "vacated" to extend the gap should not be included into the calculation except they're fragmented or followed by another gap, otherwise placing that file into the gap just shifts the gap and does not improve its filling. When the simulation finds that the gap size exceeded size of remaining files, FastFill should give up and go just for either best fit or MoveDownFill. It is especially this step that I believe would make this command much more efficient on end of the spacehogs zone.

I know that making this change would mean MyDefrag may "freeze" for some time because of doing repeated calculations without moving any files, but again, the total time spent on optimizing the disk can only get shorter, not longer, as it currently does the same calculation except it also moves files during that.

[Bibby]

I too have need for FastFill(DoNotVacate) functionality! 

I have been trying to figure out how to fastfill some small files while keeping larger files in place. If I were to do for example.....

#Large file zone
Fileselect
  Size(50000000,0)
FileActions
FileEnd

#Small file zone
Fileselect
  Size(0,1000000)
FileActions
  Fastfill()
FileEnd

.... then this would cause the Fastfill() to start the zone at the end the large file zone. I just want to fill in the gaps in the large zone with the small files.

[Darlis]

You have to set the position of the second zone to the beginning of the first zone:

SetVariable(ZoneBeginning, ZoneBegin)
#Large file zone
Fileselect
  Size(50000000,0)
FileActions
FileEnd

MakeGap(ZoneBeginning, DoNotVacate)
#Small file zone
Fileselect
  Size(0,1000000)
FileActions
  Fastfill()
FileEnd

[Bibby]

You have to set the position of the second zone to the beginning of the first zone:

Cool!

Although this does feel more like a workaround rather than a function I won't grumble!!!

[Darlis]

Yes it is a workaround, because there is a bug with the ZoneBegin variable. If it gets fixed, it would look like this:
#Large file zone
Fileselect
  Size(50000000,0)
FileActions
  AddGap(ZoneBegin)
FileEnd

#Small file zone
Fileselect
  Size(0,1000000)
FileActions
  Fastfill()
FileEnd

[poutnik]

You have to set the position of the second zone to the beginning of the first zone:

SetVariable(ZoneBeginning, ZoneBegin)
#Large file zone
Fileselect
  Size(50000000,0)
FileActions
FileEnd

MakeGap(ZoneBeginning, DoNotVacate)
#Small file zone
Fileselect
  Size(0,1000000)
FileActions
  Fastfill()
FileEnd

I am afraid such a script would vacate non zone1 files from zone1,
before performing no fileactions for zone1 files and returning zone2 files back.

I have decided to verify it here in a job on single partition disk.
It started to act i have described.

[Darlis]

I am afraid such a script would vacate non zone1 files from zone1,
before performing no fileactions for zone1 files and returning zone2 files back.

You want to say that zone 1 will move files even if there is no FileAction specified? This shouldn't happen and a test on my data partition proves this.

Note that these zones together do not select all files! If the second zone is processed, files in size between 1000000 and 50000000 bytes are being vacated.

[poutnik]

I am afraid such a script would vacate non zone1 files from zone1,
before performing no fileactions for zone1 files and returning zone2 files back.

You want to say that zone 1 will move files even if there is no FileAction specified? This shouldn't happen and a test on my data partition proves this.

Note that these zones together do not select all files! If the second zone is processed, files in size between 1000000 and 50000000 bytes are being vacated.

I have noticed they do not select all files.

But you are right. Now it seems confirmed by another trial. I know i am not always true - some weak moment    .

I got mistaken probably by fast switching from 1st to 2nd zone. Looking up there was displayed zone 1, but when looking down to graph, it was in fact already processing zone 2.

[poutnik]

These two suggestions are something I'm thinking about for some time already, still not perfectly sure if it's good idea to post here - especially because I guess it could be hard to implement with current MyDefrag architecture.

FastFill(DoNotVacate):
Improvement to FastFill(WithShuffling):

I have not previously commented them - I think it would be very good idea. We will see what Jeroen thinks about.

I was thinking about similar idea, called early vacating ( as Fastfill often does ) or late vacating, as you are proposing.

Late vacating would show its advantage e.g. if 2 zones have mixed up files ( by change of script, change of file status  or by disk activity ). 

Early vacating of the lower zone can easily fill all gaps in the upper zone. Then it starts putting upper zone files to above zone gaps, while files from lower zone are still sitting in the upper zone. After removing lower zone files from the upper zones these above zone files are moved back to zone.

Note to MoveUpToZone - it seems to me it makes very aggressive vacating of upper zones files. If there are more subsequent zones, and the first zone applies MoveUpToZone, it will unconditionally vacate all upper zone files. It could be also called MoveUpFromZone.