Intro Download and install Frequently Asked Questions Tips and tricks

Homepage







© J.C. Kessels 2009
MyDefrag Forum
October 23, 2014, 04:01:46 am *
Welcome, Guest. Please login or register.

Login with username, password and session length
News:
 
   Home   Help Search Login Register  
Pages: [1] 2
  Print  
Author Topic: How to drastically improve performance as result of defrag  (Read 15129 times)
cris
JkDefrag Supporter
***
Posts: 12


View Profile
« on: January 03, 2007, 05:54:19 am »

Hi,
I want to point out to characteristic that I noticed in the best defrag (best = best performance after defragmentation) I have ever found.
A few years ago I was using Norton Defrag.
When it optimized the disk it moved the directories at the beginng of the disk and then it moved the files in alphabetiacal order folder by folder from the root tree. It was not just a matter of reorganizing the folder names, it rewrote ALL the files. Ex: if it a folder has files 1.txt, 2.txt and 3.txt, after the defrag the contain of files 1,2,3 were on contiguos clusters  on the disk.
The process was quite slow but very effective.

There is a good reason why this speed up your computer. What slow down the disk is not just the fragmentation of a file but, above all, how spread are the files that a process use on the disk.
Usually a program access files (.exe, .dll and data files) in its folder and the subtree of its main directory. Other files that are accessed are the registry files and DLLs in the Windows/system folder.
Ordering the files how I described usually (if the folder is not too big) concentrates the files used by a process in the same area of the disk. This reduces the movements of the heads of the HD resulting in better performances.

This is the reason why the first times you run a defrag utility usually decrease the performance of the first applications you installed on your PC: at the end of the optimization process file that were near on the disk can be far even if they are undefraged.

Another good idea is to alter the alphabetical order an move thefiles in the Windows folder and its subfolder at the beginning of the disks. Windows contains the files that are more often used by all the applications.

I don't know why the major part of the current defrag utilities have not this feature but I wonder if you can implement this way to optimize the disk as an optition of your nice program.

If you like it but you have some concerns with the implemention I can help.

Have a great 2007,
Cris  Cool
Logged
cris
JkDefrag Supporter
***
Posts: 12


View Profile
« Reply #1 on: January 10, 2007, 04:55:36 am »

Jeroen,
I expected at least comment from you  :wink:

Cris
Logged
jeroen
Administrator
JkDefrag Hero
*****
Posts: 7232



View Profile WWW
« Reply #2 on: January 10, 2007, 03:50:07 pm »

Sorry for not replying before, somehow your posting must have slipped my attention!

I agree with you that ordering the files on disk is very important. At the moment JkDefrag only defragments and optimizes, and does not order the files because my first goal was to make it as fast as possible for daily use. But I am certainly planning to add optimization modes that will order the files.

There are many strategies to ordering files. The alphabetical ordering that you describe is one, and is a very good strategy. Others are for example ordering by last access date or last modification date, ordering by size, or ordering by boot access sequence, like Windows Vista does. You can find some discussions about sorting methods in other postings on this forum.

Thank you for your ideas and your help, it's certainly appreciated! It will take me some time to implement something, though. Ordering a harddisk is not quite as simple as it sounds, and that's why most free tools don't do it.
Logged
marko
Newbie
*
Posts: 2


View Profile
« Reply #3 on: January 29, 2007, 08:22:14 am »

I liked the old Norton defragger.  That brings back memories, good memories in fact.

One should also consider that the NTFS disk structure is a lot different than FAT32 or FAT.  I'm not sure how much of a performance boost is gained versus the amount of time required to order files alphabetically.  I'm guessing that we would waste more time defragging than we would gain back from any boost in peformance?

I'm not a programmer, so I'm not sure.  I think that JKDefrag is amazingly fast and effective the way that it is.
Logged
loopdemack
JkDefrag Supporter
***
Posts: 14


View Profile
« Reply #4 on: February 08, 2007, 01:26:57 am »

I absolutely agree with the cris,

I will give you an example,
if you have one folder  which has 12000 files inside (in the root of that folder), if you use any defrag optimization which is based on the modification time( I will characterized it as optimization 1), or if its based just on the simple defrag sorting the files randomly (optimization 2) or if its sorted alphabetically (optimization 3),
and when you want to enter in that defragged folder with the explorer or with the total commander,
for the first defrag optimization you will need a 32 seconds just  to enter in that folder (after cashing that folder you will enter almost instantly any other time you try to enter, here I just talk about entering the first time),
for the second optimization you will need 19,5 seconds,
and for the third optimization you will need only 9,2 seconds.

Situation is more extreme the first,  if you try to access that folder (I explained which folder), from the network, if you use only lame default MTU size such is 1500, you will need  
134 sec. for the first optimization, 51 sec. for the second optimization, and 31 seconds for the third optimization, if you tweak MTU perfectly for your network devices, (like in my case MTU 1460),
you will need 44,5 sec. for the first optimization, 27 sec. for the second optimization, and just 14 sec for the third type of the defrag optimization.
You will say why are you talking about network, mtu and such thing, I can only say its a real time usage, in a real case. I just want to compare optimization for the windows and system folder toward any other folder.

In this example we see that alphabetical optimization is ultimate optimization for every day usage.
In my opinion, optimization based on modification time should be used only for a few files and the system folders, and it absolutely ridicules for the all drives defragment method.


Second thing is combined optimization, like selecting one method for a some folders and other for the other drives or other folder.

Third thigh should be the temporary storage for any defrag optimization.
Why anyone didn't try to use temp storage on other drive just to speed up the defrag time, like if I have 2 or more drives , files should be copied to the second drive and then transfer back at the optimized position on the defragged drive.
I'm sure that in the such way the defrag time should be almost halved.
Logged
jeroen
Administrator
JkDefrag Hero
*****
Posts: 7232



View Profile WWW
« Reply #5 on: February 08, 2007, 07:18:35 am »

Quote from: "loopdemack"
I will give you an example

Is your disk NTFS, or FAT32, or something else? Also, I'm not quite sure if I understand your measuring method. Listing a directory does not access the data of the files, so where are the timing differences coming from that you have found? It cannot be optimization, because optimizing a harddidk only concerns itself with the data of the files, not with the data in the MFT (Master File Table) or the FAT (File Allocation Table).

Quote from: "loopdemack"
files should be copied to the second drive and then transfer back at the optimized position on the defragged drive.

I think your idea would work very well if both disks are SCSI harddisks, because SCSI can simultaneously read data drom one harddisk and write data to another. But not if the second disk is actually another volume on the first harddisk, or if both disks are IDE or SATA harddisks (only 1 IDE/SATA harddisk in a computer can be active at any time).

Secondly, and perhaps more importantly, JkDefrag does not move files by itself. It sends commands to the Microsoft defragmentation API to move files, just like virtually all other defraggers in existence. And the API cannot use a second disk for temporary storage. It does not use temporary storage at all, it only has a command to move the data of a file from one location to another, within the same disk.
Logged
loopdemack
JkDefrag Supporter
***
Posts: 14


View Profile
« Reply #6 on: February 08, 2007, 01:47:12 pm »

My drive system is NTFS of course.
-------------

Listing of the folder trough the explorer, total commander, or with the any other file manager is just one obvious method which is showing drastic differences between different optimization methods.
Of course if you turn off icons you will get almost instant access in the worst case you will have just a few seconds(2-5 with the worst optimization).
But remember you should count only the first enter in the folder, because every following enter will be cached and almost instant.

Files are been accessed because of the icons,
if the files are placed sequently one to another, alphabetically, file manager could read those files without the degrading the speed because it doesn't have to move drive head to a different location on the drive surface, like,
files from the one folder, would been scattered trough the drive surface if you use optimization 1, based on the modification time.

--------------
Yes I know that you use Microsoft defragmentation API, and because of that its bloody slow.
Physically when you use IDE hard drive all the bad will pop-out because its just an IDE technology, and if you use only one drive, and if you use defragment on that one drive, that's it, case close, till the defrag is done.

If you measured the performance of the moving files on the single drive you will see that its around 12000-27000kb/s effectively, and the I/O is almost  maxed because of the drive is using both read/write, if you use two modern IDE drives pata/sata, and if you transfer files from one to another and back, it would be from 28000-62000 kb/s but the most important thing is that you will have almost the 50% free I/O, for other things you are doing on your system.
Logged
jeroen
Administrator
JkDefrag Hero
*****
Posts: 7232



View Profile WWW
« Reply #7 on: February 08, 2007, 04:51:27 pm »

Quote from: "loopdemack"
Files are been accessed because of the icons

Ah yes, the icons. I didn't think of them. Yes, I can see now why listing directories will be faster if the files are optimized in alphabetical order on disk. Well, like I said before, I agree that alphabetical ordering is a very good optimization strategy and I definitely want to implement it in a future version of JkDefrag.

Quote from: "loopdemack"
you will have almost the 50% free I/O

I can see why moving a file from one disk to another would be faster than moving a file to another location on the same disk, because the harddisk heads don't have to move back and forth. But where is the 50% free I/O coming from?
Logged
loopdemack
JkDefrag Supporter
***
Posts: 14


View Profile
« Reply #8 on: February 08, 2007, 07:25:44 pm »

Theoretically, when you just copy the files from one to another drive in the time, files are been cached into the free memory, you can see traces of files in a free memory with any mem debugger. That fact lowers the I/O busyness for good %.
If you defrag system drive, and you have already busy drive with opened system dll's and page file, and with active defrag on that system drive, its a absolute killer, because there is no available drive transfer for any disk usage or there is no left I/O for any other serious activities.
As you can see, in this picture guys at the raxco are trying with the latest Perfect disk to compensate I/O when there is a high neediness of the I/O, but nothing could change the truth.
That Microsoft API is old and needs to be upgraded.



I would like the day when IDE group would consider to implement integrated ram just for caching purpose, like SCSI has eons ago.
Logged
jeroen
Administrator
JkDefrag Hero
*****
Posts: 7232



View Profile WWW
« Reply #9 on: February 09, 2007, 06:18:26 am »

Quote from: "loopdemack"
files are been cached into the free memory

So your earlier statement about 50% free I/O was based on a direct disk to disk copy, inside the harddisk controllers, without the data flowing through the CPU and the main memory? Yeah, I can see the advantages, but that is a technology far beyond my reach. I am just a single guy working from home, not a big company with lot's of programmers and resources.

Quote from: "loopdemack"
the latest Perfect disk to compensate I/O

Dynamic throttling based on activity is on my list, but building that into JkDefrag is a lot more difficult than the current "-s" option. Which by the way works very well, I think.
Logged
cris
JkDefrag Supporter
***
Posts: 12


View Profile
« Reply #10 on: February 12, 2007, 09:46:48 am »

Hi Jeroen,
Just implement the alphabetiacl order. It's not just a matter of icons.
If you need a hand I can tell you a few trick about how implementing it quickly and realiable. Are you familiar with the Standard Template Library?

It will be also useful to have a tree view where we can dinamically move the files using drag and drop. It will be so possible to arrange the file order how we want before running the defrag.

Another feature that will help to increase the performances would be to scan the uninstall information of the applications and keep together the files of each app. Unfortunatelly system drivers must be differently managed.

Regarding the time constraint:
Have you ever think about put the source on sourceforge or sometihng like that?
People can help you to implement JKDefrag.

Cris
Logged
jeroen
Administrator
JkDefrag Hero
*****
Posts: 7232



View Profile WWW
« Reply #11 on: February 12, 2007, 07:46:16 pm »

Quote from: "cris"
Just implement the alphabetiacl order.

I said I will, but JkDefrag is a hobby and I will do it when I get around to it. At the moment I am working on directly reading the MFT on NTFS disks, which promises a huge speed increase in the analysis phase, and the ability to defrag certain hidden system files.

Quote from: "cris"
Are you familiar with the Standard Template Library?

No. I try to keep away as far as possible from OOP programming.

Quote from: "cris"
It will be also useful to have a tree view where we can dinamically move the files using drag and drop.

Interesting idea! I have made a note, it would be nice to have something like that in a "pro" version.

Quote from: "cris"
Another feature that will help to increase the performances would be to scan the uninstall information of the applications and keep together the files of each app.

Yet another interesting idea! Can you tell me how to do such a scan, on all Windows versions?

Quote from: "cris"
Have you ever think about put the source on sourceforge or sometihng like that?

No, I have a very strong dislike for CVS systems. I'm a single programmer and do not like working in a group.
Logged
cquinn
JkDefrag Hero
*****
Posts: 81


View Profile WWW
« Reply #12 on: February 12, 2007, 07:50:52 pm »

Since the source code is available, might it not be helpful to present an example of this implementation that is compatible with the existing code that Jeroen has written?  

If you do include this implementation, I think it should be done as an optional step (and assume that you were planning it that way).  

There is a point to moving Windows files to the fastest part of a single disk to optimize access, but Windows itself tries to cache the more frequently used code in memory, to avoid the need to go to disk; and the user also has other files and applications that they might feel should have priority in accessing once the OS has finished loading.

One should also keep in mind that there is a significant difference in a single Hard Drive, Software RAID, Hardware RAID, JBOD, NAS, and SAN volumes.   The ideas for defragmentation and file optimization that used to work for a single drive, may no longer be valid for the different types of storage that are available to the user, and why it is important that the defrag program makes use of the proper APIs.   Moving all the files to "the front" of a volume may not be moving them to the fastest part of the volume itself, because of the way the file table layout, file system allocation, drive controller, etc... all manage the storage resource.  

The fact that JKDefrag emphasizes the use of the Microsoft defragmentation API is one of the primary reasons I started to use it.    Also, the companies that claim to have the fastest defrag programs on the market also still make use of the standard APIs.  They may make use of more pre-analysis and background processing to get their defragging done in a more efficient manner, but they still do so in proper order.  The main reason we don't continue to use some of the older Norton or PCTools utilities is because they tended to apply their own algorithms to the defrag process, which did not always take into account other processes that the OS might be trying to manage on the file system, which resulted in breaking files and file systems in fairly interesting ways back in the day too.  

Even IDE technology has advanced to the point that you cannot guarantee that defragging on only one drive is a secure process unless the defrag app is forcibly locking down all other access to the device until the defrag is done, something the OS itself is not going to allow.  Especially when dealing with features that have been borrowed from SCSI.

The major (commercial) defrag utilities do have features like this, but they probably have them named under some sort of marketing terms that do not make it as obvious what they are doing (or to sell it as some feature that the competition does not have).
Logged
cris
JkDefrag Supporter
***
Posts: 12


View Profile
« Reply #13 on: February 13, 2007, 06:44:09 am »

Quote from: "cquinn"
Since the source code is available, might it not be helpful to present an example of this implementation that is compatible with the existing code that Jeroen has written?


I am sorry, but I don't have ANY source code of the JKDefrag. The source code that is avaliable is about using the .dll that is not open source.

Quote from: "cquinn"
If you do include this implementation, I think it should be done as an optional step (and assume that you were planning it that way).  

There is a point to moving Windows files to the fastest part of a single disk to optimize access, but Windows itself tries to cache the more frequently used code in memory, to avoid the need to go to disk; and the user also has other files and applications that they might feel should have priority in accessing once the OS has finished loading.


Well, there are good theorical reason why you said is uncorrect but  the practice is enough. Try running a defrag program that use the alphabetical order (as Norton Speed Disk and another one. You will notice the difference.

Quote from: "cquinn"
One should also keep in mind that there is a significant difference in a single Hard Drive, Software RAID, Hardware RAID, JBOD, NAS, and SAN volumes.   The ideas for defragmentation and file optimization that used to work for a single drive, may no longer be valid for the different types of storage that are available to the user, and why it is important that the defrag program makes use of the proper APIs.   Moving all the files to "the front" of a volume may not be moving them to the fastest part of the volume itself, because of the way the file table layout, file system allocation, drive controller, etc... all manage the storage resource.  

The fact that JKDefrag emphasizes the use of the Microsoft defragmentation API is one of the primary reasons I started to use it.    Also, the companies that claim to have the fastest defrag programs on the market also still make use of the standard APIs.  They may make use of more pre-analysis and background processing to get their defragging done in a more efficient manner, but they still do so in proper order.  The main reason we don't continue to use some of the older Norton or PCTools utilities is because they tended to apply their own algorithms to the defrag process, which did not always take into account other processes that the OS might be trying to manage on the file system, which resulted in breaking files and file systems in fairly interesting ways back in the day too.


Personally I have never had any problem with Norton. The older Norton and PCTools for Dos simply don't work on the new large HDs and on NTFS and I doubt you can run them in win32.
However my point was not making the defrag process faster but make the PC faster after using the defrag. I've never questioned using the standard Defrag API.

Quote from: "cquinn"
Even IDE technology has advanced to the point that you cannot guarantee that defragging on only one drive is a secure process unless the defrag app is forcibly locking down all other access to the device until the defrag is done, something the OS itself is not going to allow.  Especially when dealing with features that have been borrowed from SCSI.

The major (commercial) defrag utilities do have features like this, but they probably have them named under some sort of marketing terms that do not make it as obvious what they are doing (or to sell it as some feature that the competition does not have).


This is quite inaccurate.

Cris
Logged
cris
JkDefrag Supporter
***
Posts: 12


View Profile
« Reply #14 on: February 13, 2007, 07:14:59 am »

Quote from: "jeroen"

Quote from: "cris"
Are you familiar with the Standard Template Library?

No. I try to keep away as far as possible from OOP programming.


I don't see the reason to keep away from OO. Nearly all the industry has adopted it. It makes the software more manageble, maintanable, readable, expandable, ect.
If there are bottlenecks you can address them specifically after profiling the code.
However I really doubt that for a defrag program the problem of performance will come from OOP. The access to disk is at least 50 times slower than any OOP overhead or calcualtion you can have in the code. The challenge is reduce the number of accesses  to the disk. And it's easier to implement some optimization with an hash_map or a map in the STL than trying the same thing with plain structured programming (at least 20 times faster considering the debug time).

Quote from: "jeroen"

Quote from: "cris"
It will be also useful to have a tree view where we can dinamically move the files using drag and drop.

Interesting idea! I have made a note, it would be nice to have something like that in a "pro" version.

Quote from: "cris"
Another feature that will help to increase the performances would be to scan the uninstall information of the applications and keep together the files of each app.

Yet another interesting idea! Can you tell me how to do such a scan, on all Windows versions?

Quote from: "cris"
Have you ever think about put the source on sourceforge or sometihng like that?

No, I have a very strong dislike for CVS systems. I'm a single programmer and do not like working in a group.



I see... your ultimate purpouse is to make some money from this.
But think a moment about the price of the well-known and well-supported programs and a price that you should set to be competitive.
Think how many people would continue using your software if they have to pay.
I think that you can hardly do more than $1000...
Think about the time spent on this project. Could you make $2 a hour of it?
I personally think that there is pleanty of defrag paywares out there that we don't need another made by a single programmer.
If you make JKdefrag opensource at least you will have the fame to have started a good project and people could contribute to make it better than some professional software.

Anyway, thanks for the effort you put in JKefrag and for having ketp it free till now,
Cris
Logged
Pages: [1] 2
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.5 | SMF © 2006-2008, Simple Machines LLC Valid XHTML 1.0! Valid CSS!