Ah hah!  These files are big AND I'm using NTFS compression (where those files are located).  That would explain it!   I've set the files to uncompress, which will take approximately 30GB more HD space, but I can afford that space.

I will let you know how the defrag goes after the uncompress is complete (that is going to take a long time).

MyDefrag runs EXTREMELY slow on my 250GB HD.  I have no other issues with the HD, so I don't believe it's bad.

While defragging, the hd usage light blinks VERY intermittently, so I don't think there's heavy disk IO.  While MyDefrag is running, it uses 50% of the CPU (100% of one core, as I have a dual-core system).  I have obtained a debug log of the activity, and it shows that it takes about 15 seconds to move 3-4 clusters.

MyDefrag starts out running pretty fast, and then appears to slow down over time during the defragmentation run.  The only thing I can think that's different from most hard drives is that there are a LOT of files on that drive.  I have attached a snippet from the debug log, and then the contents of the normal log below.  I can provide the entire debug log, if you have somewhere you want me to put it.

Thanks for your time making MyDefrag.

Example:
10:12:15             I want to move 3 clusters at offset=18768 of item 'D:\work\builds\fast\v5intg\backup\audit_operation.xml' (Inode=1070041).
10:12:15               Moving 'D:\work\builds\fast\v5intg\backup\audit_operation.xml' (Inode=1070041)
10:12:15                 Moving 3 clusters at offset=18768 to Lcn=12204828.
10:12:15                 Getting fragment list for: D:\work\builds\fast\v5intg\backup\audit_operation.xml
10:12:34             I want to move 3 clusters at offset=18771 of item 'D:\work\builds\fast\v5intg\backup\audit_operation.xml' (Inode=1070041).
10:12:34               Moving 'D:\work\builds\fast\v5intg\backup\audit_operation.xml' (Inode=1070041)
10:12:34                 Moving 3 clusters at offset=18771 to Lcn=12204831.
10:12:34                 Getting fragment list for: D:\work\builds\fast\v5intg\backup\audit_operation.xml
10:12:54             I want to move 4 clusters at offset=19172 of item 'D:\work\builds\fast\v5intg\backup\z_approval_event.xml' (Inode=150882).
10:12:54               Moving 'D:\work\builds\fast\v5intg\backup\z_approval_event.xml' (Inode=150882)
10:12:54                 Moving 4 clusters at offset=19172 to Lcn=12204834.
10:12:54                 Getting fragment list for: D:\work\builds\fast\v5intg\backup\z_approval_event.xml
10:12:58             I want to move 3 clusters at offset=18774 of item 'D:\work\builds\fast\v5intg\backup\audit_operation.xml' (Inode=1070041).
10:12:58               Moving 'D:\work\builds\fast\v5intg\backup\audit_operation.xml' (Inode=1070041)
10:12:58                 Moving 3 clusters at offset=18774 to Lcn=12204838.
10:12:58                 Getting fragment list for: D:\work\builds\fast\v5intg\backup\audit_operation.xml



(Note, this log may not be from the same run as the run above)
Date:                       2009/12/13 10:47:09
Script:                     Optimize Monthly
MyDefrag version:           MyDefrag v4.2.6 for X64
Commandline:                "C:\Program Files\MyDefrag v4.2.6\MyDefrag.exe" -d 4
Working directory:          C:\Program Files\MyDefrag v4.2.6

###############################################################################
Processing volume:    D:
###############################################################################

Total disk space:       250,048,507,904 bytes
Volume type:          NTFS

Before:
  Unfragmented items:   101,452,169,216 bytes         1,072,270 items
  Fragmented items:      27,465,461,760 bytes            21,841 items
  Gaps:                 121,130,860,544 bytes           543,644 gaps
  Average gap:                  221,184 bytes
  Median gap:                    49,152 bytes
  Biggest gap:           53,010,440,192 bytes
  Average end-begin distance:    8,365,189.0870 clusters

is there a reason that the *.wma files are not in that list too since they are also music files?

JkDefrag needs a fair amount of time to compare all the files on a disk with all the SpaceHogs masks ("Applying Exclude and SpaceHogs masks...."). To keep the time within reasonable limits I have to keep the list small and cannot add all possible audio file formats. Only the more popular. If I enter mp3 into Google I get 289.000.000 hits, for wma only 3.640.000. On my computer I have 12510 mp3 files and 144 wma files. I'm sorry, but wma is simply not popular enough. If there are many votes on this forum for adding wma then I will be happy to add it, but you are the first to ask.

Jeroen:
If I optimized the routine for spacehogs, would you be willing to add a lot of file formats?  Is it just a performance issue?

The performance of determining which files are spacehogs should be O(n), where n is the number of files.  Putting the files into a HashSet and doing a lookup based on the extension should be an O(1) operation.

I'm not sure how you're currently doing it, but if this is the concern, I'll whip up a solution for you that runs much faster (it is, after all, open source).  It seems odd with modern processors that we're concerned about keeping the number of file extensions that we support to a minimum.

Is this something you'd like?

I agree with other people that wmv and wma should be considered spacehogs.

Lothsahn

09:52:24         646   394116336 indicates to me that there were no large gaps to defrag files.  therefor it couldn't move files out of the way for the sort (it will not fragment files while making the gap.)  therefor it was spending a lot of time trying to fill small holes and fragmenting file into those holes.


I think this is the problem.  Other defrags fail--I think I need more free space, it doesn't shuffle files around as I'd expect.



As for the high cpu usage--it's not compressed files or antiviral.  It only happens when I do -a 7.  -a 3 uses like 4-5% cpu usage.  There must be some performance issue with -a 7.

Jeroen:

I'm so very sorry--I forgot that Microsoft had released a buggy update to Windows (Vista).  I should have specified my OS.  I'm using Windows XP Professional 32-bit edition.

Also wanted to say that I've been using JKDefrag for a while (but not option -a 7), and it's so far worked very well for me, so I'm very happy for your software.

Thanks again,
Lothsahn

I would argue against including .wav.  Small wav files, used frequently by windows or files should be fast to access.  Wav files that people are using for music or video editing or some purpose are going to be huge, as wav files grow in size quickly.  Therefore, they'll be covered by the following requirement:
Files bigger than 50 megabytes

Therefore, I think the current scheme is fine for .wav files.

My system:
AMD Athlon 3000+
C: Individual 120 GB HD (Windows is here)
D: 195GB Partition of a Raid 1 of 2*250GB WD HD's  (23GB free)
E: Remaining partition for the raid array.

All drives are NTFS.

I have a number of compressed folders on D:, and some on C:

I have full permissions to the entire contents of C:, D:, and E:.

I have no known viruses or spyware--I am an extremely experienced technical user, and have no reason to believe anything on this computer is broken.

Prior to running JKDefrag, I've verified that except for 2 or 3 files, all files on D: are NOT in use as I've shutdown programs and verified file handles with SysInternals Process Explorer.  Therefore, I would expect nearly all files to be accessible.

I ran the following command:
JKDefrag -a 7 -d 3 -l c:\JKDefrag.log d:


Problems:

1) I found a number of files that could not be moved.  I don't understand why they could not be moved--in fact, I picked a random file (RASETUP.exe) and was able to move it to a different folder.  I at no time had Red alert running during the defrag.  I have included the "could not be moved" portion of my log below.  None of these files showed as "in use" based on the file handle search from Sysinternals process explorer.


2) The defrag reports that it finished, but I don't think it has.  I see the following in the log:
09:50:35 d:\games\Halo\MAPS\bitmaps.map
  Moving 2 clusters from 39512825 to 40034734, VCN=68288.
09:52:24 Finished.

When I left the house (at 9:30 AM), it reported that it was 13% (Zone 3) complete.  This defrag has been running since 10PM last night, so I would not expect the other 70% to complete within 10 minutes.  When I restart JKDefrag, I see free space scattered around the disk, and the disk is full all the way to the edge, which is the slow part that shouldn't be filled.  It looks anything but what I expect.  There are gaps far in excess of the 1% (2.5GB) in the middle of the drive, so I can't expect that it's the 2 "buffer" zones.  Plus, the log file indicates it stopped on d:\games\Halo\MAPS\bitmaps.map, but there are tons of folders after this one, alphabetically.  I don't know if JKDefrag moves the files alphabetically, but if it does, it's missed a very large portion of my drive.


3) JKDefrag in -a mode (on any drive in my system) eats all of the CPU.  I expect it to place heavy disk access, but there is something in the software that is causing extreme CPU consumption.  The entire system is sluggish and slow, but these issues resolve after I lower the priority to "below Normal".  For comparison, Windows Disk Defragmenter takes 1-2% of the CPU. 


4) I cannot run JKDefrag against c: and d: at the same time.  Once #3 is resolved, I would love to do this, since they are two separate physical disks, so the IO access on each shouldn't interfere with each other.


Please let me know if you need any other parts of the log.


Log:
09:52:24 These items could not be moved:
09:52:24   Fragments       Bytes  Clusters Name
09:52:24           1    67108864     16384 d:\$LogFile
09:52:24          10       37504        10 d:\$MFT::$BITMAP
09:52:24           1   307183616     74996 d:\$MFT
09:52:24           2        8192         2 d:\.
09:52:24           2   116707106     28457 d:\drivers\soyo USB bluetooth\widcomm\BTW5.0.1.1200.zip
09:52:24          12   644491987    114219 d:\games\Atari\Axis & Allies\Data.rwd
09:52:24           2   149828360     36492 d:\games\Battlefield 2\mods\bf2\Common_client.zip
09:52:24           2      131200        15 d:\games\Battlefield 2\mods\bf2\Levels\Dragon_Valley\Info\loadmap.png
09:52:24           2      131200        17 d:\games\Battlefield 2\mods\bf2\Levels\FuShe_Pass\Info\loadmap.png
09:52:24           2      131200        16 d:\games\Battlefield 2\mods\bf2\Levels\Gulf_of_Oman\Info\loadmap.png
09:52:24           2      131200        15 d:\games\Battlefield 2\mods\bf2\Levels\kubra_dam\Info\loadmap.png
09:52:24           2      131200        13 d:\games\Battlefield 2\mods\bf2\Levels\Operation_Clean_Sweep\Info\loadmap.png
09:52:24           2      131200        14 d:\games\Battlefield 2\mods\bf2\Levels\Strike_at_Karkand\Info\loadmap.png
09:52:24           3   641042884    156226 d:\games\Battlefield 2\mods\bf2\Objects_client.zip
09:52:24           2      131072         9 d:\games\bleem\ff7(2).MCD
09:52:24           2      131072        11 d:\games\bleem\ff7(4).MCD
09:52:24           2      131072        11 d:\games\bleem\ff7(5).MCD
09:52:24           2      131072        11 d:\games\bleem\ff7(6).MCD
09:52:24           2      131072        11 d:\games\bleem\ff7(7).MCD
09:52:24           2      131312         8 d:\games\Civilization III\Civ3PTW\Scenarios\Antartida S.bic
09:52:24           2      128471         9 d:\games\Civilization III\Civ3PTW\Scenarios\Double Your Pleasure\Text\Pediaicons.txt
09:52:24           2      256040        14 d:\games\Civilization III\Civ3PTW\Scenarios\RingWorld (Large) v2.bic
09:52:24           2    86251508     21056 d:\games\comedy\movies\teamwizzy\TW-NFEpisodeOne.wmv
09:52:24           1        4096         1 d:\$MFTMirr
09:52:24           1     6399896      1563 d:\$Bitmap
09:52:24           3   357481664     87275 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm) II\RA2\movies01.mix
09:52:24          21   304170688     74260 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm) II\RA2\movies02.mix
09:52:24         646   394116336     96215 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm) II\RA2\movmd03.mix
09:52:24           3    18666588      3698 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm) II\RA2\multi.mix
09:52:24           2    31264268      6088 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm) II\RA2\multimd.mix
09:52:24           2       90112        16 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm) II\RA2\patchget.dat
09:52:24           3      184320        30 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm) II\RA2\patchgetmd.dat
09:52:24           2      171520        31 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm) II\RA2\PATCHW32.DLL
09:52:24           2   281888480     47194 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm) II\RA2\ra2.mix
09:52:24           2     1314623       239 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm) II\RA2\RA2MD.exe
09:52:24           2   204527696     34826 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm) II\RA2\ra2md.mix
09:52:24          16    76862662     18713 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm) II\RA2\theme.mix
09:52:24          26    46859102     11426 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm) II\RA2\thememd.mix
09:52:24           2     6181704      1387 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm) II\RA2\wdt.mix
09:52:24           3     1277757       231 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm) II\RA2\YURI.exe
09:52:24          34     3070092       749 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\2nd_hand.aud
09:52:24           2     2941132       717 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\araziod.aud
09:52:24          43     3178252       775 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\backstab.aud
09:52:24          31     2386955       579 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\bog.aud
09:52:24           3     1161235       174 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\EDIT.DAT
09:52:24           3      869888       117 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\EDWIN.EXE
09:52:24           2      458242        96 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\EXPAND.MIX
09:52:24           2      469922        99 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\EXPAND2.MIX
09:52:24           2     2476183       379 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\GAME.DAT
09:52:24           2      261425        31 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\HMIDRV.386
09:52:24         117  1331073921    324645 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\MAIN.MIX
09:52:24           2      142336        26 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\PATCHW32.DLL
09:52:24           3     2375680       363 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\ra95.dat
09:52:24           2      108498        19 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\RASETUP.EXE
09:52:24           3      244874        47 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\README.EXE
09:52:24           2     1440056       266 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\RED ALERT\ALLIED.BMP
09:52:24           3     1439824       349 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\RED ALERT\ALLIED.WAV
09:52:24           9     1440056       314 d:\games\Command & Conquer The First Decade\Command & Conquer Red Alert(tm)\RED ALERT\Ant.BMP

When JKDefrag finishes, it just has a black screen.

It'd be nice to show the final results of the defrag, instead of a black screen, with the word "finished".

Not sure if this would be hard.  Seems easy from my perspective, but sometimes things aren't as they seem.

By the way, awesome program.

Lothsahn