Multiplatform! JKdefrag on Linux?

[Justin Goldberg]

Could you see if there is a way to make command-line version of JKDefrag work on Linux/BSD with ntfs-3G driver?

JkDefrag is based on the Microsoft defragmentation API. As far as I know that API has never been ported and is simply not available on Linux.

There are ways to use the windows ntfs.sys driver in linux and bsd. I remember seeing it on slashdot years ago.

google

[reivilo]

Here is an interesting link on a project of ntfs defrag from within linux(as part of ntfsprogs):
http://wiki.linux-ntfs.org/doku.php?id=ntfsdefrag

[jeroen]

Here is an interesting link on a project of ntfs defrag from within linux(as part of ntfsprogs):
http://wiki.linux-ntfs.org/doku.php?id=ntfsdefrag

Thanks for the link, I appreciate it! I have taken a look and it sounds very interesting. Their claims are very ambitious and grandiose, mysterious and premature. They will have quite a job making them true. I'll believe it when I see it. Meanwhile JkDefrag is up and running and getting better with every release...

[reivilo]

As you can see on the documentation of ntfprogs, the bitmap is a file on the disk ($bitmap) just like $mft, $mftmirr, etc
http://www.linux-ntfs.org/content/view/104/43/#file_bitmap

So using the library of ntfsprogs it would be easy to emulate the second function.

For the first one(get the location of a file on the disk), i guess you would simply have to read the file entry in $mft.

[jeroen]

As you can see on the documentation of ntfprogs ....

I think Soichiro is working on a generic solution for all filesystems available on Linux, not just NTFS. Reading the raw NTFS data on disk is allright for showing a map of the disk and calculating which files must be moved and where they should go, but not for the actual moving of a file. I don't know when changes to the NTFS data are actually committed to disk, I imagine there are moments when the data on disk is out of sync, stale and old.

[reivilo]

Ok, but i just don't understand how you could come up with a generic solution using low-level ioctl calls?
Linux generic filesystem handling uses VFS, a layer, that calls the specific filesystem functions from e2fsprogs(ext2 and ext3 FS), reiserfsprogs(ReiserFS) or ntfsprogs(NTFS), which does not , as far as i know, provide support for listing free clusters, or other low-level functions.

[Soichiro]

As you can see on the documentation of ntfprogs, the bitmap is a file on the disk ($bitmap) just like $mft, $mftmirr, etc
http://www.linux-ntfs.org/content/view/104/43/#file_bitmap

So using the library of ntfsprogs it would be easy to emulate the second function.

For the first one(get the location of a file on the disk), i guess you would simply have to read the file entry in $mft.

File systems in Linux are set up similarly. There is a bitmap stored in the supernode (equivalent to the MFT, in a way), and each file has an inode which stores the location on the disk of each fragment. This means the only part that would really be difficult to program is moving the file.

[dennis]

There is hope to get JKdefrag on linux

http://www.linuxinsight.com/ols2007-ext4-online-defragmentation.html

If there are the same functions available on linux as there are on windows, porting may be not so difficult after all.

Dennis

[jeroen]

http://www.linuxinsight.com/ols2007-ext4-online-defragmentation.html

Thanks for the link, I appreciate it! It was an interesting read. The document is a proposal for 5 defragging functions to be added to Ext4. Strangely enough the document is totally stum about optimization, which is far more important than defragmentation. There is a lot of talk about "free space fragmentation", but that's not the same thing.

The Windows defragmentation api has just 3 functions. Two of those could be simulated with two functions from the 5 Linux functions. The third Windows function is to find out which clusters are free on the disk, and cannot be simulated with the Linux functions. Perhaps it's already available elsewhere in the kernel, I seem to remember somebody saying something about that earlier in this thread..

It seems to me that the proposal is a bit overengineered. Some of the functions are not needed and can be dropped. And one of the 5 functions is actually dangerous and should definitely not be made available to a user-mode defragmentation program.

It's encouraging that the Linux community is finally getting of it's high horse and admitting that fragmentation is a problem, even on Linux. But instead of trying to reinvent the wheel thay should take their queue from the Windows api. It is simple and straightforward, extremely stable and foolproof, and has a proven track record for both defragmentation and optimization.

[Lecter]

I think you guys missed the main point and turned this into an usual rhetoric about best OS based on very few references with no OS programmer in them.  JK is probably the only real guy on the level, and he knows too few on Unix-like systems.  

I am here to put that he wrote this in the NTFS-3G site too, which would give JK real references in how best unix-like OS are managing NTFS without knowing anything about how NTFS works in depht.  JK himself made this based on Windows NTFS managing, and until not only a cracking window NTFS writer was made for Linux, and only one paying MS to use its stuff on NTFS are using that.  Therefore JK should go see NTFS-3G forums to look what they can do in NTFS, he could ask them features that would allow them to be compatible.  This would be a temporary solution to NTFS-3G for the fragmentation problem, being Unix based they will keep working on it until it doesnt frament anymore (unless in special critic conditions).  Defragmenting for Unix is like trying to remove a sticky Virus from a computer without reformating its disk, for them its both errors that shouldnt happens in the first place.  

The reason of existence of NTFS readers/writers in unix-likes is almost only due to Linux's being versatile and not centered developping made by Windows users who wanted a less cracking OS release on TV.  With pure GPL OS and FS. (NTFS aint open like the UFS).  There is probably a hack way to make work jkdefrag with the cracking NTFS.sys combining GPL OS-FS-programmers with Windows crackers. Or the pay way of

*************************************************************
http://forum.ntfs-3g.org/viewtopic.php?t=665&highlight=defragmenter
Note that few people wanna write ntfs in linux, and less wanna build Defragmenter on it yet.  Also since jkdefrag is GPL... It wont be long to Linuxize it if they really wants it (unless there ia a terrible lack of community on it.    
http://forums.whirlpool.net.au/forum-replies-archive.cfm/443682.html
That site talk about ext3 servers.
As to give JK a good idea about Unix FS and their performance and need to be defragged:
http://forums.whirlpool.net.au/forum-replies-archive.cfm/443682.html
With 30% free disk space, Unix FS wont really fragment anything,
also using different FS with diferent advantages....
to compare youll need to see this:
http://en.wikipedia.org/wiki/Defragmentation#Filesystems
Shake is the best defrag to bypass any FS.
*************************************************************

Just to know, JK doesnt you got a developper corner forum?

I think someone should manage a global GPL FS readers/writers/utilities forums to help GPL FS projects to communicate together to really improve FS managing, until now GPL FS managing aint very good in comparison to the private ones, even when the private ones open their FS codes (unlike NTFS which is still kinda secret).  Also it would be the best way for FS utilities developpers to harass FS developpers about ressources.

[Lecter]

http://forum.ntfs-3g.org/viewtopic.php?t=613
http://forum.ntfs-3g.org/viewtopic.php?p=1333#1333

*************************************************************
skaza
lead developper said:
*************************************************************
Ok. Though only 25% of the ntfsinfo output managed to arrive, I can see that the file consists of about 7700 fragments. On a completely fresh, new volume this number would be 1 or maybe 2.

PerfectDisk or O&O Defrag would probably help but not the Windows built-in one, nor Diskeeper and neither all the free ones (they all use the same Windows code which is not efficient enough).

I also think I know the reason for the high fragmentation and will think about the solution for the short (months) and long term (years) as well. Thanks.
************************************************************
Just because PerfectDisk make spaces to prevent fragmentation -.-
I like the way he discarded the mighty DiskKeeper.
But the thread here show real fragmentation of NTFS with Linux, and the problematic of really big ntfs disk using.  

Geez why using NTFS on such big file system, should use XFS for that it would barely frag and you can defrag quickly from the xfs webpage.  

Whats really great is that slowy NTFS-3G will go faster allowing less performant defraggers, and that the users wont endure the top cost defrag long, since this is GPL...

[Lecter]

>The open source ntfs resizer (ntfsresize) I wrote 5 years ago did already
>partial file and disk space level defragmentation more efficiently and
>sophisticatedly as a side-effect than the ones which use the Windows defrag
>API (e.g. jkdefrag). It wouldn't be a big deal to make it a good
>defragmenter.
>
>Hovever there is no real need for it (there are many free ones), and I also
>don't have the time to do unfortunately.
>
>Anyway, thanks for the email.
>
>   Szaka

http://man.linux-ntfs.org/ntfsresize.8.html

If you wanna take a look at it.

[ABasketOfPups]

I'm going to go out on a limb and say the reason that Linux built up this "doesn't need defrag" legend is partially because of how Linux used to be setup.

Nowadays, a fair number of installations have only one major partition, but in the past there were a LOT.  You were encouraged to have the root partition, which would be small and filled with system files, and the /usr partition, which would have your libraries and apps.  The /tmp partition which is where temporary files go by default in Linux, and /home for your "desktop" files.  And the swap partition, for virtual memory.

You break up Windows similarly, into partitions for Windows itself, one for the paging file, one for application install and one for application data, and one for user's desktops, and you'll see similar, much-longer-before-fragging because the contention for contiguous space is greatly reduced.

It's a royal pain, of course, to set up partitions this way.  If you're setting up a server you tend to know what kind of parititioning you'll need, but a user machine? That'll be doing all sorts of things? It's just too hard to guess.