Optimisation for Flash/SSD media

[fjw]

As you well know most defragmenters are intended for use only on spinning hard drives, and are not suited to Flash media.  Hard drives have a very different access pattern.  In addition, in some circumstances, though not on most modern SSDs, Flash has a limited number of write cycles, making defragmentation undesirable.

I was thinking it would be good to have a defragmenter that is optimised for Flash devices.  Hopefully the information below is of interest not only to you but to people interested in Flash based drives.

Comparison between Flash devices and hard drives:

Proximity:

On a hard drive, files that are located near each other on the disk reduce seek time because the RW head travels less distance between them.  In Flash, seeks are just as fast anywhere on the drive.  Therefore, the concept of 'zones' is meaningless in Flash - seeking to directories or frequently accessed files can happen just as fast if they are scattered all over the drive as if they are within 1% of the drive of each other.  Optimising for Flash would not involve 'zones' or sorting.

In fact, almost all modern Flash drives employ an internal remapping of blocks for wear-levelling purposes.  This means that the location of the data on the drive is completely meaningless as it only refers to a logical concept, at least at the block level (Flash uses large erase blocks of 128KB to 4MB).

Random read access:

Random read access is an order of magnitude faster on Flash drives than hard disk drives, a fact that is often quoted as a reason that Flash drives don't need defragmenting.  This much is true; defragmenting a file will not noticeably affect reading of it, especially on a more expensive SSD (as opposed to a USB flash memory, or SD card) where random read latency is 0.3ms or less (compared to 10-15ms for typical hard drives).

Random write access:

Random write latency on Flash media unfortunately suffers.  In some cases, very heavily.  Whereas a hard drive can typically write to more than 100 random 4kb chunks per second (<10ms latency), on a Flash drive it can be much slower, with as much as 250ms (a quarter of a second) latency on MLC SSDs.  SLC SSDs are usually faster than this, but most still do not match hard drives for random write latency and are still typically slower than hard disk drives.

This means that fragmentation during writes is more important on Flash drives, even more so than hard drives.  Unfortunately, it is not possible to have much control over where the operating system writes, with a file system optimised for hard disk drives like NTFS.  Writing new files places the files where-ever Windows thinks they ought to go, which may be informed by where Windows thinks the R/W heads are on a physical disk.

New and outrageously expensive drives do not suffer this random write problem, because an internal controller remaps all writes so that they are sequential in regard to the block (similar to a log-structured file system), even if the OS sees them as happening on different parts of the disk.  In the consumer space, the only ones available now that do this are the Intel SSDs, which will still set you back four figures.  Coming soon are the ExtremeFFS based drives from Sandisk, expected in early 2009, which as of today are only a press release.

For optimum performance on a typical SSD, we must therefore ensure that writing is done sequentially, as much as possible.  We don't have much control over where the OS will create new files, except by ensuring that the free space on the drive is as contiguous as possible, to maximise the chance that writes of new files (for example, cache files) are done sequentially.

(Other options to speed up a Flash disk exist, such as using an alternative file system, see "Managed Flash Technology", or "JFFS2" on Linux, but these are outside the scope of what I'm writing).

Proposed optimisation:

Moving files to the start of the disk is meaningless, as is defragmenting existing files.  The main thing that needs optimisation is free space, which should be made as contiguous as possible.  Small areas of free space under 4MB (a block size on some larger Flash SSDs) should be plugged however possible, remembering that it is fine to cause other files to become fragmented.

Modern drives can handle 50GB of writes per day and still last for over 10 years, but that said, people are still concerned over subjecting their SSDs to undue wear caused by unnecessary erase/write cycles.  Therefore, the optimisation should avoid moving files larger than a certain amount to achieve its goals.  We needn't try to make the free space into one big contiguous area (which will be largely impossible anyway, due to unmovable files).  So we can stop once an area of free space is, say, 4MB or larger.

Summary:

If nothing else, it is important enough to remind people that defragmenting a Flash based drive using the traditional methods is not helpful, and only causes unnecessary wear.

[jeroen]

As you well know most defragmenters are intended for use only on spinning hard drives, and are not suited to Flash media.

Thank you very much for all the information, I appreciate it! Quoted from the JkDefrag frequently asked questions, the "Will JkDefrag reduce the lifespan of my flash memory disk?" question: "Yes, all defragmentation/optimization will reduce the lifespan of flash memory disks (USB memory sticks, Solid State Disks (SSD), and such). Flash memory will get faster if defragmented, but does not gain much from optimization. My advise therefore is to use "-a 2" only, and only incidentally (not every day). JkDefrag does not usually process flash memory disks by default, only if you specify the drive letter on the JkDefrag commandline (for example "jkdefrag h:"), because they are usually seen by Windows as removable volumes. "

defragmenting a file will not noticeably affect reading of it

Users here on the forum have reported that defragmenting their flash drive DOES noticeably affect reading speed. It is true that read latency is very low on flash drives as compared to harddisks, but there is also some latency inside Windows itself, presumably caused by having to look in the MFT to find the location of the next fragment.

This means that fragmentation during writes is more important on Flash drives, even more so than hard drives.

Interesting, I did not know that.

The main thing that needs optimisation is free space, which should be made as contiguous as possible.

The current default "-a 3" optimization in JkDefrag already optimizes free space. It fills gaps with files from above, which results in less gaps and bigger gaps. However, it does much more work than would be necessary for a free space optimization only, because it also moves everything to the beginning of the disk. I see now that a special free space optimization would be useful for Flash drives, with a configurable maximum gap size. I will put it on the list for future implementation.

[fjw]

I agree with the FAQ that of all the current methods, -a 2 is probably "best" for Flash disks currently, but mostly in that it's the "least inappropriate" - which is certainly not a criticism of a defragger, which by their nature are optimised for hard disks.

The random read latency varies a lot between cheap and expensive Flash media, even though it is normally under 1ms.  For example, $400-$1000 SSDs often have read latency of something like 0.05ms to 0.1ms, whereas cheaper Flash such as a USB memory drive or SD card may have 1ms.  Compared to hard disks which have as high as 14ms sometimes, the differences between Flash drives seems insignificant.  However, with a fast drive like an Mtron Mobi 3000, even if a file is as fragmented as possible, as in, every single 4kb cluster is in a completely random location of the drive, the read rate is still going to be around 50% of the maximum sequential read rate.  A cheap SSD or something like an SD card would suffer more, but still, only from extreme fragmentation.  In real world, a heavily fragmented drive would be nowhere near that fragmented.  Therefore, depending on the drive, fragmenting files will either have no noticeable effect or a very small noticeable effect (edit: for subsequent read operations).

The effect of random write latency is going to be an order of magnitude more noticeable on a Flash based drive than any effect caused by reading of fragmented files.  It shows a fundamental incompatibility between NTFS, a file system heavily optimised for hard disks where small random writes have little penalty, and SSDs.  As long as we are writing hundreds of small files to the drive it's going to be slower on a Flash drive (without next-gen linear write re-mapping, or "Enterprise" drives such as from Memoright).  But if we are writing large files, a defragger can be of some limited help: by ensuring that free space is somewhat defragmented, at least at the block level (Flash erase blocks can be 128KB to 4MB in size).

Some may be interested to read this article, which shows how this random write latency can really slow things down on an SSD, in this case one of the consumer MLC based SSDs with pretty slow writes, which is not atypical of affordable MLC based drives.
http://www.alternativerecursion.info/?p=106

[jeroen]

Users here on the forum have reported that defragmenting their flash drive DOES noticeably affect reading speed, as I said before. I agree with you that the speed increase cannot be explained by Flash drive latency. Can you think of a theory that explains the speed increase? My theory is that there is an additional and substantial latency inside Windows itself.

[lh]

My guess reads speeding up would be from the commands issued by the sata controller.

Lets say you have two blocks A and B.

In example 1 they are right next to each other.  AB.
You would get 1 command from the controller of 'get 2 blocks starting here'.

Now in example 2 the are separated by some distance.  ACB
You would get 2 commands from the controller of 'get 1 blocks starting here' then 'get 1 blocks starting here'.

There is also a bunch of other 'ack/nak' junk that goes along with that.  With 2 blocks you get twice as much 'chatter' going on.  This same thing happens on a spinning drive.  It just is not as noticeable as the latency for the drive itself is so high which ends up 'hiding' some of that.

I agree that pushing everything (and defraged) to the end for a flash drive would probably be best.  As NTFS will 'tend' to start at the front of the drive when writing new items.  This would give NTFS a better chance of getting a 'clear' area to write into.  Eventually after a few runs 'older' items will end up towards the end of the drive.

Probably something like 'move what you can into the areas from the end zone where it fills it perfectly'.  Then crunch it all into one contig area.  You probably would start at the end of the drive and work backwards.

Now the only downside to this I think is you 'might' be fighting against the wear leveler and actually making the lifetime of the drive shorter.

Long term the way NTFS works and the way Windows 'slow grows' its log files will need to change.  Which is out of our control

[JimF]

Maybe this is old news to most people by now, and it just reinforces the comments made above, but Diskeeper has introduced a new HyperFast technology for consolidating free space in SSDs:
http://www.diskeeperblog.com/archives/2008/12/hyperfast_is_al.html

It is said by the gurus on the OCZ forum to actually work:
http://www.ocztechnologyforum.com/forum/showthread.php?p=339763#post339763

I have also seen favorable comments on other forums, though not everyone achieves such impressive results. 
http://theovalich.wordpress.com/2009/01/28/diskeeper-releases-ssd-optimization-utility/

Most defragmenters also defragment files, which is undesirable as it increases the wear on the SSD, when all you want to do is consolidate the free space.  It would be great if JKDefrag included such an option.

[quanthero]

As far as I understand, HyperFast technology is about consolidating free space. If this is true, JKDefrag -a 5 will achieve this goal without too many file movements, and there will not be much wear.

[jeroen]

Version 4 of JkDefrag (called MyDefrag) has a special script for Flash memory. For more information see Using MyDefrag. The script will not only consolidate (defragment) free space, but will also defragment files.