Defragment() is pretty time-consuming

[Cushy]

Some weird behaviour is observed during the Defragment() phase. It seems that every time after moving out some file MyDefrag starts consuming 100% of one core for about a second. It goes like this:
...
1. Move out some file or part of fragmented file.
2. Think about something for a sec.
3. Move out next file or part of fragmented file.
4. Think about something for a sec.
...
n. Defragment & move in some big file.
n+1. Move out some file or part of fragmented file.
n+2. Think about something for a sec.
...

According to debuglog it seems that this "thinking about something for a sec" is actually "looking for a gap at the end of the disk".
On one of my volumes I have about 249000 files. Suppose that only 1/5th of them are getting moved during Defragment() phase (although if you haven't done defrag for a long time more than half or even almost all of the files are getting moved). That would amount to 830 minutes during which MyDefrag is only thinking what to do during Defragment() phase. Add to this the time during which MyDefrag is only thinking during FastFill() phase (http://www.mydefrag.com/forum/index.php?topic=2926.0). Add to this the time during which the actual moving of the files occurs. And we get a very sad picture.

[BloodySword]

It is very silly to complain about slowlyness while omitting important system specs...
Perhaps you are running Mydefrag on a Pentium II or something... Please inform us about your system specs first!
I don't have such problems and I have a lot of files on my system drive thanks to vista. And there is no such behaviour for me.
System specs: Core 2 Duo @2 GHz, 3GB DDR2-RAM, Vista SP2 32-Bit, HDD: Hitachi 2,5 Inch TravelStar, System Type: Notebook 18 Inch.

[Cushy]

It is very silly to complain about slowlyness while omitting important system specs...

Well I don't think so. Because none of the defragmentation utilities should consume 100% of one core every time it moves out some file even on Pentium II.

Please inform us about your system specs first!

I have Q6700 running under Windows XP SP3.

[BloodySword]

Hmm...

I can not understand why so much people are complaining slowness on MyDefrag while saying "other defragmenters work well"... I've installed MyDefrag on varous systems. A P4 3,06 GHz HT with 2GB RAM and XP, an older AMD system with Win 7, a Core 2 Duo Notebook with Win 7 64 Bit, my Notebook... And everything is fast... Except for my external USB 2.0 drive. The drive is slow, and USB, too. But it is not a big pain. It's okay and I understand it.

Make shure that you stop your antiviurs, stop uneccessary services and processes while defragmenting. It might be that antoher process is tracking the move of the file, then checks something on the file  and consumes much CPU, while MyDefrag is hanging then... It could be your antiviurs, but it could also be suspicious backup software, I won't call names ;o).

What software is running on the system?

[quanthero]

Cushy,
I run MyDefrag on a 5-year-old Toshiba laptop with Pentium M @ 2GHz and 2GB of RAM. I almost never see high CPU usage, and MyDefrag generally tends to finish faster than most other defragmenters (with the exception of PerfectDisk). So I was wondering if something else is happening to your system. Could it be a virus scanner (Kaspersky and Norton are known to mess with defrag) or, perhaps, some kind of file indexing / database software?

[Cushy]

I have auto-scan of my AV turned off. I don't have database or whatever heavy software running on my system. And I don't understand why somebody in this case would want to ask me about the software running on my system while there's Task Manager or Process Explorer which pretty much tells you what loads your system.

And here is what is written about MoveToEndOfDisk() which apparently uses the same code as Defragment() & FastFill() do to move out the files that are in the way:

This action is relatively slow, best to be used for big files only. It's because the Microsoft defragmentation API is not very efficient in finding the last gap suitable for a file.

So it seems that the slow code is there & it is executed every time MyDefrag needs to find a gap for the file which needs to be moved out of the way.

And for those of you who say that MyDefrag is fast I have some questions:
How often do you look into Task Manager/Process Explorer during the times MyDefrag runs?
How many files do you have on your volumes?

[BloodySword]

The only thing I get higher CPU usage is while sorting directories and there is no gap at the end of the directory zone. See attached picture. So it seems there is a problem in calculation for the next gap, if the gap is far, far away from the current location. The partition has a size of 182 GB. Zoom is 2x.

[jeroen]

Some weird behaviour is observed during the Defragment() phase.

The behavior that you describe only happens in certain specific situations, most notably when there is not enough room on the harddisk to move files out of the way. It does not happen always, as you seem to assume. You can avoid it by using Defragment(Fast) instead of Defragment().

[Cushy]

The behavior that you describe only happens in certain specific situations, most notably when there is not enough room on the harddisk to move files out of the way.

On my 900GB volume with about 19500 files I have around 29GB free. The size of the largest file on this volume is 8GB so I think 29GB is enough. The following script was applied to this volume:

Code:

FileSelect
FullPath("D:\Downloads","*")
or FullPath("D:\Temp","*")
or FullPath("D:\To Burn","*")
FileActions
Defragment()
FastFill(WithShuffling)
FileEnd

MakeGap(MaxNextZoneBegin,DoNotVacate)

FileSelect
All
FileActions
Defragment()
FastFill(WithShuffling)
MoveUpToZone()
FileEnd

While MyDefrag was working on the beginning of the last zone in the above script I paid attention to how it works & noticed that between moving out some files MyDefrag consumed 25% of CPU (100% of one core). See the first attachment. And that was happening with every file that was moved out. When MyDefrag was moving in some small files with MoveUpToZone() at the end of the disk it was consuming 100% of one core constantly. See the second attachment (this screenshot shows the different situation but this is how it looked in case of MoveUpToZone() too). The third attachment shows what was happening on a 160GB volume with around 249000 files & with 12.9GB free space while MyDefrag's FastFill was enlarging the gap for some big file. So these "certain specific situations" occured pretty often.

You can avoid it by using Defragment(Fast)

That would leave some files fragmented. This is something that defragmentation utility should not allow to happen.

[BloodySword]

I have also around 200 000 files on a partition wich is 96 GB big and I don't have this problem... I don't get it what's going on on your machine.

[jeroen]

On my 900GB volume with about 19500 files I have around 29GB free. The size of the largest file on this volume is 8GB so I think 29GB is enough.

I see in your script that you are using MakeGap(MaxNextZoneBegin). Your 29GB free space is below that point and is not used. The program has to make do with whatever small gaps happen to be after that point. In other words, your script is a recipe for slow processing. Perhaps you should consider using the MoveToEndOfDisk file action instead.

[Cushy]

I see in your script that you are using MakeGap(MaxNextZoneBegin). Your 29GB free space is below that point and is not used. The program has to make do with whatever small gaps happen to be after that point.

Well first of all it was MakeGap(MaxNextZoneBegin,DoNotVacate). Most of the free space was after the beginning of the last zone. So it could be used & was used but it was pretty CPU-intensive.

Secondly, Jeroen, could you post the whole FindGap() in its current state here?

Maybe the size of the output buffer (65536+8+8 bytes) a pointer to which & the size of which are passed to the DeviceIoControl() in FindGap() is too small.

[jeroen]

Secondly, Jeroen, could you post the whole FindGap() in its current state here?

Sorry, no.

[BloodySword]

MyDefrag is closed sources for some reasons. We have to accept this.

[Cushy]

Sorry, no.

Well then will you investigate the situation about the size of the output buffer for DeviceIoControl() in FindGap()?

Right now it seems that the further the zone is to the end of the disk the slower FindGap() becomes. Apparently this has something to do with StartingLcn passed in BitmapParam to DeviceIoControl(). Isn't it quicker to get the whole volume bitmap at once (for 900GB volume it's around 28MB) thus minimizing the number of calls to DeviceIoControl() which probably becomes slower with the increase of StartingLcn?