Intro Download and install Frequently Asked Questions Tips and tricks

Homepage







© J.C. Kessels 2009
MyDefrag Forum
August 22, 2014, 02:43:23 pm *
Welcome, Guest. Please login or register.

Login with username, password and session length
News:
 
   Home   Help Search Login Register  
Pages: 1 [2] 3 4
  Print  
Author Topic: Multiplatform! JKdefrag on Linux?  (Read 43290 times)
jeroen
Administrator
JkDefrag Hero
*****
Posts: 7220



View Profile WWW
« Reply #15 on: May 18, 2007, 12:58:53 pm »

I am a little tired of this "discussion". I don't know if Linux has a fragmentation problem or not, but I suspect it does and so far I haven't heard anything proving that it doesn't. I'd be interested to hear from knowledgeable people with real hard facts.
Logged
lulu135
JkDefrag Junior
**
Posts: 6


View Profile
« Reply #16 on: June 06, 2007, 09:58:12 pm »

Hello..  jumping into the discussion.  It's annoying when people repeat the same party line over and over again when there's no basis for believing it other than speculation and wishful thinking.

I have a Linux box that I use like a HD-Tivo among other things, with tens-of-gigabyte files being created and deleted quite frequently.  Sometimes the filesystem has gotten quite full, which is a known bad case for fragmentation.  So, after reading this thread I decided to run a test on the /home partitition using the filefrag utility.   There are 155013 total files in the partition.  Of these, 69000 of them are at least 4k (meaning they don't fit in a single cluster).  Only 8188 files are >64k.  The partition has 2351 files that are fragmented.  574 of them have at least 10 fragments.  290 of them have at least 100 fragments.  122 files have over 10000 fragments, and 13 files have over 100000 fragments.

So please I never want to hear anyone say "Linux doesn't need defragmentation" again.   Maybe _your_ box doesn't need it, but my workload has created a highly fragmented filesystem.

It's a standard ext3 filesystem by the way.
Logged
jeroen
Administrator
JkDefrag Hero
*****
Posts: 7220



View Profile WWW
« Reply #17 on: June 06, 2007, 10:30:36 pm »

Thank you very much for chipping in, I appreciate it! The "filefrag" utility is new to me, I didn't know about that. Great! Your numbers are an eye-opener. I've never seen a Windows machine with even a single file having more than 100000 fragments, and you have 13? Wauw! You have confirmed what I suspected; Linux needs a defragger.
Logged
lulu135
JkDefrag Junior
**
Posts: 6


View Profile
« Reply #18 on: June 07, 2007, 01:41:55 am »

Quote from: "jeroen"
Thank you very much for chipping in, I appreciate it! The "filefrag" utility is new to me, I didn't know about that. Great! Your numbers are an eye-opener. I've never seen a Windows machine with even a single file having more than 100000 fragments, and you have 13? Wauw! You have confirmed what I suspected; Linux needs a defragger.


The 13 files with over 100000 fragments are all in a P2P download folder.  This makes sense because the program initially creates a sparse file of the appropriate size and then writes pieces of the file to disk in a random order as the data is downloaded.  There is a new allocation of space each time a piece is written to disk, because that is how sparse files work.
Logged
Soichiro
JkDefrag Senior
****
Posts: 34


View Profile
« Reply #19 on: June 07, 2007, 01:48:47 am »

Are those files complete or only partially downloaded? If they're incomplete, I wouldn't count them as files, but if they're complete, that means whoever designed that program needs to get rid of that massive fragmentation problem and linux needs to get a defragmenter.
Logged
lulu135
JkDefrag Junior
**
Posts: 6


View Profile
« Reply #20 on: June 07, 2007, 02:54:37 am »

The files are complete.  I would argue that it isn't really the responsibility of an application designer to worry about filesystem fragmentation.  (actually, the program has an option to pre-allocate all space for the files, but then it takes a long time to start the download.)
Logged
Soichiro
JkDefrag Senior
****
Posts: 34


View Profile
« Reply #21 on: June 07, 2007, 09:25:52 am »

Well, I do acknowledege that it is mostly the responsibility of the file system to handle fragmentation, but in this case the blame also falls upon the programmers, since there is so much fragmentation created by this file download/creation method, and since better methods exist, such as the option to allocate space for the file when the download begins (which would eliminate, or if free space is low, at least reduce fragmentation greatly). However, I do agree that Linux needs a good defragmentation program, since nearly all users will undoubtedly run into some amount of fragmentation at the fault of the file system. The ones that exist are either still in development or exclusive to one or two file systems.
Logged
robert
JkDefrag Supporter
***
Posts: 17


View Profile
« Reply #22 on: June 11, 2007, 09:40:27 am »

Most of the measures of fragmentation are completely bogus.
The important values are
(1) How many blocks can you read before you have to seek  and
(2) How far do you have to seek (short seeks are faster)

In line with this I think 'bytes per fragment' or 'fragments per megabyte' would be a better measure even though they don't take into account seek times.

But back to Linux, to help the first problem Linux 'reserves' chunks of about 512k for every file, this space isn't allocated to the file but it will not normally be used by other files.

For the second one Linux simply tries very hard to keep parts of a file in the same general area of the disk.

When these two work as normal my disk throughput reduces from a raw 50Mb/s to about 30Mb/s on a large file. IMHO this is not a problem so linux doesn't need a defragmenter.

However,  If you've been paying attention you'll notice that even the smallest file is allocated or has reserved 512k. This can be a problem and is why IMO linux does need a disk optimiser that can compact small files that will never use the reservation and reorder files based on usage patterns (like application startup and shutdown).

An optimiser would also allow features like 'dynamic partitioning' (like the old norton dos defragmenter), that jkdefrag is just starting to get, where the sysadmin can control exactly where files should be allocated on the drive either directly or relative to others.

Anyway enough of this; jkdefrag is a windows defragmenter/optimiser and windows does need it; I have see half a million fragments, or about 40k per fragment, on a large NTFS file written in one run!
Logged
jeroen
Administrator
JkDefrag Hero
*****
Posts: 7220



View Profile WWW
« Reply #23 on: June 11, 2007, 12:09:20 pm »

Quote from: "robert"
I think 'bytes per fragment' or 'fragments per megabyte' would be a better measure

If a single file has more than ca. 50 megabytes per fragment (depending on the speed of the harddisk) then defragmentation of that file will not make much of a difference. But do not make the mistake of calculating the "average bytes per fragment" for an entire harddisk, and then saying the entire harddisk does not need to be defragmented.

Quote from: "robert"
Linux simply tries very hard to keep parts of a file in the same general area of the disk.

Linux has many filesystems and not all are fragmentation resistant. Your arguments apply to a few Linux filesystems, such as the very popular EXT3, but that does not mean that Linux cannot use a defragmenter. If only for the other filesystems.

By the way, Windows/NTFS also has strategies to resist fragmentation. The performance is about the same as for Linux/EXT3, or so I'm told.

Quote from: "robert"
my disk throughput reduces from a raw 50Mb/s to about 30Mb/s on a large file. IMHO this is not a problem so linux doesn't need a defragmenter.

I am totally amazed that you shrug your shoulders at a 66% speed reduction. You are not alone, though. I get this from more Linux users.
Logged
robert
JkDefrag Supporter
***
Posts: 17


View Profile
« Reply #24 on: June 11, 2007, 09:10:38 pm »

Quote
I get this from more Linux users.


I would suspect you would get this from a lot of users in all.  Most people are not going to care about these benchmarks unless something has made them a problem; eg "it used to be a lot better than this" and that's probably the point; despite what we've been told about NT's anti-fragmentation properties there doesn't seem to be a limit to how bad the fragmentation will get in normal use with with NT. OTOH an abused three year old linux ext3 partition is still performing 'good enough' with cp(1) as a defragmenter for p2p. It still serves files to the network faster than windows could on the same hardware.

As for other filesystems, to put it bluntly, why should they care? The primary aim of the the non-native filesystem drivers is to exchange data with other operating systems, speed is just an option, getting there intact is the goal.

Anyway to sum up ... lies, damned lies, and statistics.  Cheesy

PS: /me waiting with bated breath for the jkdefrag scripting language.  Cool
Logged
jeroen
Administrator
JkDefrag Hero
*****
Posts: 7220



View Profile WWW
« Reply #25 on: June 12, 2007, 10:50:53 am »

Quote from: "robert"
despite what we've been told about NT's anti-fragmentation properties there doesn't seem to be a limit to how bad the fragmentation will get in normal use with with NT.

Wow! No limit? Files that are not fragmented will not get fragmented. And NT is completely antiquated, you should compare with Windows 2000 or 2003.

Quote from: "robert"
It still serves files to the network faster than windows could on the same hardware.

I don't want to start a "which is faster" war here, but I suspect this is a myth, just like linux not needing a defragger is a myth. In the early days of windows it was true, but Windows has improved tremendously. Take a look at this comparison: Microsoft Windows Server 2003 vs. Linux Competitive File Server Performance Comparison. It was payed for by Microsoft, so it may be biased, but still an impressive advantage for Windows.
Logged
Soichiro
JkDefrag Senior
****
Posts: 34


View Profile
« Reply #26 on: June 13, 2007, 09:52:25 am »

Well, nothing's going to get accomplished by debating about how much the file systems fragment, since we already know all file systems get fragmented. Since it doesn't look like anyone else is going to, I guess I'll start experimenting with porting jkDefrag to Linux, since I have some C++ knowledge and access to a computer to test it on.

This probably won't end well... :roll:  But at least worst case scenario is I have to format a few partitions with no important data on them.  Tongue
Logged
jeroen
Administrator
JkDefrag Hero
*****
Posts: 7220



View Profile WWW
« Reply #27 on: June 13, 2007, 11:05:41 am »

JkDefrag relies on the Microsoft defragmentation api, which has just three calls:

- FSCTL_GET_RETRIEVAL_POINTERS to get a list of all the fragments of a file.
- FSCTL_GET_VOLUME_BITMAP to get a bitmap of the disk, one bit per cluster, 0 is free, 1 is in use.
- FSCTL_MOVE_FILE to move a file from wherever it is to a specific disk location. Fragments are automatically joined or created.

Click one of the calls to see the manual page on the Micorosft website. If you can build a Linux version of these three calls then JkDefrag can be ported.
Logged
Soichiro
JkDefrag Senior
****
Posts: 34


View Profile
« Reply #28 on: June 14, 2007, 08:56:54 am »

It turns out it won't be easy. The only way to port it to Linux would be to rewrite the functions completely, or to somehow get Microsoft to release the source code for their libraries, but I don't think that'll happen. It's obviously possible to code this, since Microsoft did it once, but it's going to be pretty difficult. To begin, the way Microsoft calls these functions doesn't make any sense at all. How it actually works is beyond me. I might be able to create some functions that produce the same result, although the way they obtain it would be different and would require significantly modifying the code inside jkDefrag.

In other words, it's going to take a while. Sad
Logged
jeroen
Administrator
JkDefrag Hero
*****
Posts: 7220



View Profile WWW
« Reply #29 on: June 14, 2007, 09:40:36 am »

I agree. I'm not worried about changes in JkDefrag, but the big problem will be the third function, to move a file to a given spot on harddisk. I think it requires special hooks deep inside the filesystem drivers to make sure that data cannot get lost or corrupted, on a running system without using locking.
Logged
Pages: 1 [2] 3 4
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.5 | SMF © 2006-2008, Simple Machines LLC Valid XHTML 1.0! Valid CSS!