FastFill(WithShuffling) very slow

[Kasuha]

Today I was in the following situation while filling a zone using FastFill(WithShuffling) algorithm:

<end of disk>
<bunch of big files (each bigger than the "big" gap)>
<another gap>
<bunch of small files (all together smaller than the "big" gap)>
<big gap of about 5 GB>
<already sorted files>
<start of disk>

What was happening was, MyDefrag was moving these small files away one after another, pausing for about one second after each file. I believe during that second it was trying to fill the gap exactly using current files but didn't succeed.

Except that it was slow, there was no problem. After it removed all small and some of the big files and extended the gap enough, it moved files in.

But it was very slow.

So my idea is, to give FastFill(WithShuffling) some limits after which it will stop trying to fill the gap exactly and will change to a different strategy. These limits might be:
- number of files moved away (give up after, say, 20 moves)
- size of the gap (give up if the gap is bigger than biggest file in the zone; give up if the gap is bigger than 1 GB)

The different strategy might be the modified MoveDownFill() command which will move file down if it fits into the gap or if the gap is bigger than move chunk size, and move file away if it doesn't.
If the file is moved away, FastFill(WithShuffling) may try again to fill the gap exactly. After moving file down, it will only retry if the gap was merged with another gap this way (otherwise the gap size is the same so there is no fit, too). As soon as it finds perfect fit and moves to another gap, it may restart with basic strategy.

[jeroen]

Thanks for sharing your idea, I appreciate it.

[Cushy]

Currently on 4.3.1 I observe some similar behaviour. Sometimes during FastFill(WithShuffling) phase while moving out the files that do not belong to the current zone MyDefrag starts to wholly consume one of my CPU's cores. During these CPU-hungry times MyDefrag does not move the files that are in the way out as a whole but instead it moves them out by some 1-32 clusters at a time. Each of these move-outs takes around 1 sec and it takes a lot of time for MyDefrag to recover from this behaviour.

[BloodySword]

Your files are probably compressed or encrypted. It's a API problem.

[Cushy]

Your files are probably compressed or encrypted. It's a API problem.

No, they are not.

[BloodySword]

A sparse file is also a compressed file. Are this sparsed ones?

[Cushy]

Are this sparsed ones?

No, the files that were moved out extremely slowly are ordinary mp3s.

[jeroen]

When enlarging a gap for FastFill(WithShuffling), the program should move away as much of the file above the gap as possible, not just 1-32 clusters. Perhaps there are only small gaps available? If not then please make a debug logfile, see I have a problem! for instructions.

p.s. BloodySword: the compressed file problem that you are thinking of has been solved.

[BloodySword]

Yes I know that this problem was solved, but I wonder why a hand full of people have such serious performance problems, while Me and all my neighbourhood does not have any problems with MyDefrag. It runs very fast and is 100% reliable.

[jeroen]

Me and all my neighbourhood does not have any problems with MyDefrag. It runs very fast and is 100% reliable.

Thanks for the compliment, I appreciate it. Yes, I wish all computers were the same. I suspect the most common reason for slow performance is low memory. The computer will then start swapping, making MyDefrag extremely slow. Also note that people generally look for a defragmenter when the computer already has a performance problem, so they are using MyDefrag on a slow computer to begin with.

[BloodySword]

As mentioned in some threads ago, I only have slow performance with mydefrag, when there is no near free gap for sorting. Perhaps the API then makes a bad job. Is there any way to pinpoint this problem to see if there is anything you can do to speed it up, like the problem with compressed files some time ago?