Program is moving small files far, one cluster at a time...

[ecgn]

I am currently defragmenting a drive that has been very fragmented, and near the end, there are some files that take a very long time to get to the right place on the disc. (The full optimisation method is being employed.)

The problem files all seem to be just one cluster in size, and when JkDefrag moves them, it moves them one cluster at a time, i.e. from X to X+1, from X+1 to X+2, from X+2 to X+3 and so on...

When the first file "behaved" like this, I stopped the application, moved the file to another drive, and moved it back again. This allowed JkD to continue to process other files.

Sometimes, it is one file being moved like this, but other times it seems the application is working with two or three files in a group, but still only moving them a very short distance at a time.

Do you have any idea why it behaves like this? The two files currently being moved are compressed, but I think I have seen files that are in uncompressed directories behave in this way as well.

Finally, thank you very much for a great tool! My computer really needed it!  

//ecgn

[jeroen]

Which version of JkDefrag are you using? It sounds like a problem I have solved several versions back.

[ecgn]

I'm running version 3.7. Just downloaded it yesterday.

[ecgn]

I have a series of screenshots of what looks like a small green worm moving to the right and upwards, one line at a time, if that is of any use.

Are there any logs I can get for you that may help you analyse the problem?

[jeroen]

Perhaps you can run the program with the "-d 6" commandline option for me, and send me the logfile? You can stop the program after it has looped for a bit, that should be enough info in the logfile. The logfile will be huge. JkDefrag writes all messages to a logfile called "JkDefrag.log", or "JkDefragCmd.log", or "JkDefragScreenSaver.log", depending on which version you are using. You can inspect the logfile if you want, it's a flat-text file.

The green worm moving to the right and upwards sounds like JkDefrag is trying to enlarge a gap. I wonder why it is looping, there must be something special going on.

[ecgn]

I "solved" the problem by applying the following workaround:
 * Move three large files (250Mb, 650Mb and 1.8Gb to another drive)
 * Let the application run overnight

I will, however, pay close attention to how the application behaves after I move the files back this weekend.

Generally, when a drive is heavily fragmented, the best way seems to be to move the as much stuff as possible off the drive, do a full defragmentation, move the stuff back, and possibly do another defragmentation run.

Thanks for your time, and thanks again for writing this application!

[jeroen]

Perhaps the program wasn't looping, but was simply trying to enlarge a gap for a big file?

[ecgn]

That may very well be the case, but why was it enlarging it by one cluster at a time?
It would have made more sense to determine where the smaller file should be moved to and move it there directly, not just shifting it to the right.

[jeroen]

It would have made more sense to determine where the smaller file should be moved to and move it there directly, not just shifting it to the right.

JkDefrag tries to perfectly fill all gaps (in full optimization mode). If it cannot then it will move the file at the end of the gap away (to the first available higher gap) and tries fitting again. In your case there happened to be a lot of small files there. The program doesn't pre-fit, that is, it doesn't determine beforehand what the smallest possible gap is to perfect fit a combination of files from above the gap. In your case that would have been better, but in most other cases it would make the program far to slow because of the number of permutations it would have to calculate. In other words, I agree with you that it would make more sense, but alas, computers have no sense.

[mheintz]

I am experiencing this same issue with version 3.7.  It seems that JkDefrag is attempting to expand the gap by moving a few clusters of data out a couple of clusters at a time, then checking to see if the gap is large enough for the file... this continues until the gap is large enough.  

May I suggest that if the file you are attempting to squeeze into the gap is larger than the block you are moving, that you move the block to the beginning of the gap instead of pushing it out a couple of clusters?

[alys]

JkDefrag tries to perfectly fill all gaps (in full optimization mode). If it cannot then it will move the file at the end of the gap away (to the first available higher gap) and tries fitting again.

if i not mistaken - it is good if the first available higher gap is out of cluster numbers range [n,n+N] where n is a cluster of gap start, and N is a length of a file, you try to insert into this gap. it is clear that you should move clusters to enlarge gap out of the "file image".
but I can be mistaken, because do not understand completely the strategy of your files optimization.

[jeroen]

May I suggest that if the file you are attempting to squeeze into the gap is larger than the block you are moving, that you move the block to the beginning of the gap instead of pushing it out a couple of clusters?

The net result would be that the gap moves upward and remains the same size until it joins up with a gap further along. I used this strategy in an earlier version, and it works, but it is very inefficient. It usually causes huge series of files being moved down one by one. It's much more efficient to only move the last file in the series, which is what the current strategy tries to do.

[jeroen]

do not understand completely the strategy of your files optimization.

The strategy repeatedly enlarges a gap until it finds a combination of files that will perfectly fill the (new) gap. It does not try to enlarge a gap so it can fit a particular file. Does that answer your question?

[alys]

The strategy repeatedly enlarges a gap until it finds a combination of files that will perfectly fill the (new) gap. It does not try to enlarge a gap so it can fit a particular file. Does that answer your question?

I also watched a mentioned "worm". This artefact dramatically slows down a job execution. I have the latest version of your defragmenter.
This is not an infinite loop, because when jap reaches the appropriate size, it is filled by some file and worm disappears.

It seems that some special case takes place here.
and i have some ideas.
I think that here the programm is trying to move large file to new place, but gaps are small and gap chosen for enlargement have small (for example - 1 cluster) files at the tail.

000000xxxxx000000XXX
here 0 - is a free claster
x - single cluster file, 5 files are here.
leading 6 zeroes is a jap chosen for enlargement.
ending Xs - are some files, not meaningfull.
in this illustration we have 12 free clusters and can put here some 12 cluster(or less) file.
check what your strategy will do in this case.
it seems that strategy gets leading "small x" file and puts it after ending "small x" file. and repeats it. the worm appears. worm creeps till reaching XXX files

[jeroen]

This is not an infinite loop, because when jap reaches the appropriate size, it is filled by some file and worm disappears.

Yes.

I think that here the programm is trying to move large file to new place

No. The program will look at all the files above a gap, and tries to find a combination of files that will perfectly fill the gap. It does not try to move a particular file.

I think the worm will be solved by adding a special case to the strategy. When the program wants to enlarge a gap it should check the size of all the files above the gap until the next gap. If less than the size of the gap, then the program should move the files down to the begin of the gap, instead of enlarging the gap by moving the files upwards.