Esprit's mods

[Esprit]

This thread continues in what was started at http://www.kessels.com/forum/index.php?topic=911.0.

I started to work on modification which treats files only with some fragment smaller than 5000 clusters as fragmented. This includes fast defragmentation (just treating fragments smaller then 5000 cl.). But there was an issue with defragmenting files. Because I removed virtual fragments and those laying next to each other thus this version needs much less RAM space (about 2/3) but now defragmentation of files doesn't work because Win API sometimes needs to move each fragment separately (especially those splitted by virtual fragments). Solution is to rewrite whole move function. This will take a time...

[Lexar]

I have been waiting for you to create this thread here.

I use your latest mod of JkDefragLib.cpp to make a JkDefragCmd.exe optimized for background defragmentation of drives busily read from and written to, while I use the original JkDefrag.exe for periodical full defragmentation.

[Esprit]

Here is simple formula for computing good fragment size:

fragment_size = access_time * linear_read_speed / wasted_time

for example we got flash memory with linear speed 16.5 MB/s, access time 0.5 ms and we don't to waste time more than 5 % against fully defragmented file so

0.0005 * 16.5 / 0.05 = 0,165 MB.

So either flash memories need defragmentation but only for fragments much less than 1 MB (HDD's need 100 times bigger fragments).

[jeroen]

fragment_size = access_time * linear_read_speed / wasted_time

Interesting! Here are a few thoughts:
- The "linear_read_speed" (throughput) on harddisks is not constant. It varies depending on the position on the disk.
- Fragmentation for some files is more important than for other files. Fragments in a huge logfile are not important at all, but I think databases and movies should always be completely defragmented.
- Not defragmenting a file can cause other files to be placed at a slower position on disk than they need to be. For example a big fragmented spacehog at the beginning of the disk will prevent the moving of more important files to the beginning of the disk.

[Esprit]

- The "linear_read_speed" (throughput) on harddisks is not constant. It varies depending on the position on the disk.

I know. It's just an approximation. Either access time is not constant. But for this calculation we can use the worst numbers.

- Fragmentation for some files is more important than for other files. Fragments in a huge logfile are not important at all, but I think databases and movies should always be completely defragmented.

Movies don't need to be completely defragmented because you watch them or copy them. Both is sequential reading so a movement of head once in a while is not painful.

- Not defragmenting a file can cause other files to be placed at a slower position on disk than they need to be. For example a big fragmented spacehog at the beginning of the disk will prevent the moving of more important files to the beginning of the disk.

I talk just about defragmentation not about optimization. A large spacehog that has just 1 piece at the beggining of disk maybe doesn't need defragmentation but definitely moving that fragment is a must.

[Lexar]

Just hit upon an idea...

Dear Esprit, in your latest mod, FragmentGoodSize is defined as 5000, meaning that on a drive made up of 4KB clusters, if every fragment of a file is larger than 20MB, the file isn't treated in the defragmentation phase.

From the above, I guess the following can be said. If a 100MB file is split into two fragments, each one being 50MB in size, IsFragmented() returns a NO. However, if a 100MB file is split into one fragment of 90MB and the other of 10MB, IsFragmented() returns a YES, though 90+10 doesn't seem to be any worse than 50+50.

Instead of measuring the degree of fragmentation based on the size of the smallest fragment, you could calculate the average fragment size and compare it with the threshold set by FragmentGoodSize. How about incrementing a counter variable each time the program moves on to Next on the chain of the recursive struct and after that dividing Item->Bytes by the counter value?

By the by, Jeroen, thanks to the detailed comments on your beautifully coded sources, even a beginner like me seems to be able to start out figuring out little by little what JkDefrag is really doing! I think I've learned more from 100 lines of your codes than from 100 pages of an ordinary, commonplace C++ textbook (which I gave up reading, anyway). Thank you very much.

[jeroen]

Movies don't need to be completely defragmented because you watch them or copy them. Both is sequential reading so a movement of head once in a while is not painful.

They were just examples. The point I was trying to make is that your formula cannot be applied universally to all files. It's all right for some files, but not for other files. By the way, I disagree about fragmented movies. Playback of fragmented movies can cause very annoying pauses and glitches, especially on old or busy computers.

[Esprit]

However, if a 100MB file is split into one fragment of 90MB and the other of 10MB, IsFragmented() returns a YES, though 90+10 doesn't seem to be any worse than 50+50.

Last fragment is not treated so in this cases it should return NO.

Instead of measuring the degree of fragmentation based on the size of the smallest fragment, you could calculate the average fragment size and compare it with the threshold set by FragmentGoodSize.

It's all about quality of service (QoS). We need to prevent worst case. While first half could be contiguous second half still can be so much fragmented so HDD cannot deliver desired throughput. When the smallest fragment is big enough then desired throughput is guaranteed.
Try burn some fragmented file on DVD without buffer underun protection. Would you use your type of fragmentation decision before burning?

[Esprit]

By the way, I disagree about fragmented movies. Playback of fragmented movies can cause very annoying pauses and glitches, especially on old or busy computers.

Old computers that are unable to process movies in the play speed of 105% are not used for playing movies at all. Because the smallest fragment is big enough read speed cannot drop too much. Moreover the problem of old computers are CPU speed not disk.

[jeroen]

Old computers that are unable to process movies in the play speed of 105% are not used for playing movies at all. Because the smallest fragment is big enough read speed cannot drop too much. Moreover the problem of old computers are CPU speed not disk.

Not true. I am a 46 year old computer nerd and I have worked on every incarnation of PC since the very first IBM pc that was on sale. Please believe me when I say that the bottleneck for video playback on old computers is disk, not CPU. The speed of the CPU is important, of course, and also the video card, but the disk is the bottleneck. If the fragment boundary just happens to be at the exact same place as a buffering boundary then your scheme would not cause a problem, but otherwise it will.

[Esprit]

Old computers that are unable to process movies in the play speed of 105% are not used for playing movies at all. Because the smallest fragment is big enough read speed cannot drop too much. Moreover the problem of old computers are CPU speed not disk.

Not true. I am a 46 year old computer nerd and I have worked on every incarnation of PC since the very first IBM pc that was on sale. Please believe me when I say that the bottleneck for video playback on old computers is disk, not CPU. The speed of the CPU is important, of course, and also the video card, but the disk is the bottleneck. If the fragment boundary just happens to be at the exact same place as a buffering boundary then your scheme would not cause a problem, but otherwise it will.

I've never met a PC which was unable to play a (compressed) movie due to slow disk access. Only such problem I know is DV video on a CD. However there is only slowdown of 5% at most and every good player tries to keep input buffer full. Moreover the access time will increase by 20 ms which is mostly half of duration of one frame. And this will happen just once in a while.

[Lexar]

It's all about quality of service (QoS). We need to prevent worst case. While first half could be contiguous second half still can be so much fragmented so HDD cannot deliver desired throughput. When the smallest fragment is big enough then desired throughput is guaranteed.

I see your point.

Then, how about keeping the fragment size above 20MB at average and above 5MB at minimum? 5MB/s is 40Mb/s, which is way above 24Mb/s, an MPEG2 bitrate satisfactory to most movie directors.

[Esprit]

Then, how about keeping the fragment size above 20MB at average and above 5MB at minimum? 5MB/s is 40Mb/s, which is way above 24Mb/s, an MPEG2 bitrate satisfactory to most movie directors.

Movie directors don't care about fragmentation. I set 20 MB because it comes from ordinary disk speed, 5 MB could be quite small and average fragmentation doesn't help us at all as well as average number of fragments per file. For movie partition it's normally higher but from system partition it is expected to be less than 1,01. The goal is to set fragment size to some good size and make all fragments of this size (except last one fragment of a file).

Better algorithm could be when we take size and position of fragment into an account. Smaller fragment could be in the beginig of the disk because the read speed is higher. When moved to the end it needs to be merged with neighborhood fragment.

[ghorgh]

I've never met a PC which was unable to play a (compressed) movie due to slow disk access. Only such problem I know is DV video on a CD. However there is only slowdown of 5% at most and every good player tries to keep input buffer full. Moreover the access time will increase by 20 ms which is mostly half of duration of one frame. And this will happen just once in a while.

I don't know where is the bottleneck but I had a 700MB movie compressed in XviD which was on 9427 fragments (!). And there wasn't any troubleshooting seeing it with any video player I used to.
In the other hand, I had a 3GB VMWare image (.vmdk) on 12590 fragments, and it become impossible to run VMWare (after more than two hours torturing my HD I had no choice than physicaly shuting off my computer as my keyboard, my mouse and any programs became unusable)

[Lexar]

A 3GB file in 12590 fragments means the average fragment size is about 249KB. That's too much fragmentation for a virtual machine hive. In my idea, the number of the fragments of that file needs to be reduced to about 650 at most, for the virtual machine to show any decent performance.