Defrag suggestions for Fraps video capture

[Raffriff]

Hello, I'm active (unpaid!!) on another forum dealing with PC screen capture of movies with Fraps. There's a long-standing issue with occasional "freezing" or hesitation (caused by 1-10 seconds of high disk activity by *something* other than Fraps). The reason I'm posting here is that I think there may be a fragmentation issue - not of the capture files themselves, but of the MFT (?):

Here's the report that had me wondering about this:

1 more thing I've noticed about this problem:
There are 2 things I can do to make the freezing problem become more common.
1) Record with Fraps everyday. If I do that, the freezing will slowly become more and more common.
2) Delete the Fraps videos I don't want to keep. If I delete the videos and then go back to recording some
  more, I notice that the freezing happens much more often after I deleted the videos.

If, for a couple of weeks, I don't record with Fraps and I don't delete the recordings I don't want, then the
freezing goes away and it slowly comes back only after I start doing either of the 2 activities.

...does this look like MFT becoming fragmented / full, or like anything else you recognize? Note Fraps generates dozens of 4GB files when recording, many of which will be 'bad takes'  and will be deleted fairly quickly.

The files are usually written at nearly the maximum data rate the disk can handle - that is, the user usually cranks up the quality and resolution until performance problems show up. (That could be a reason for the problem right there - I'm guessing the disk write cache might be overwhelmed periodically)

The user making the report above states the drive is defragmented (with MS defrag) weekly; MFT is 91MB on a 2TB drive and is at 100% usage(?); fsutil shows mftzone = 0 (auto?); free space is at 50%. Besides the video files, only seldom-used files such as backups are on the drive. I *assume* System Restore, antivirus, search indexing etc is disabled, as this is standard procedure and this user seems very knowledgeable.

I'm trying to get a Process Monitor log to examine, but so far this is all I know...

A side question: is FAT32 better for this kind of thing, do you think?

Any advice / thoughts are appreciated. Hope I'm not too far off topic!

[Darlis]

I doubt that it's the fragmentation of the MFT. It's probably the fragmentation of the free space. Fraps preallocates a 4 GB file, so the space has to be marked as occupied. The higher the fragmentation of the free space the more fragmented the file will be. And adding all the fragments to the MFT takes some time.

Using a dedicated partition (formatted with a large cluster size) just for recording, that is cleaned after every recording session should solve this problem. If you want to use MyDefrag, you can also move every file to the (slow) end of the disk (except the MFT).

[Raffriff]

Thank you Darlis. Yes I agree about the dedicated partition and non-fragmented free space.
I would add: use a dedicated hard drive; dedicate the first/fastest 30% or so for Fraps,
and the rest can be used for backup etc.

I came up with a defrag script for this situation:
this assumes we want as much free space at the start of the partition, we want
the MFT located at around 30% of the disk size, and any non-capture files should
be moved to the end, thus maximizing contiguous free space, as you say.

First run takes a while but subsequent runs are very fast.

Don't know if this will help the "freezing" problem, but I think it should eliminate
disk fragmentation as a factor.
 
Fraps-01.MyD 

Code:

# MyDefrag v4.0 script for Fraps partitions
#
# L Bigelow / raffriff 14-Nov-2011.
# adapted from existing scripts from mydefrag.com, as noted.

Title('Fraps partition 0.1')
Description('
Process a Fraps capture partition (dedicated to video capture;
mostly free space, but with a *few* files we want to keep for awhile): 
place the MFT and the directories at 30% into the disk;
move all except "raw" and "work" folders to end of disk.
Assumes "raw" (capture) files change rapidly; "work" (rough edit)
files change *fairly* rapidly; and speed unimportant for all other files.
For daily use - not a full defrag routine.
')

/* Write the header to the logfile. See the "Settings.MyD" file for the
   definition of the "LogHeader" string. */
WriteLogfile("MyDefrag.log","LogHeader")

/* Select and process the volumes one by one. */
VolumeSelect
  CommandlineVolumes()
VolumeActions

  /* Write the "before" statistics to the logfile. See the "Settings.MyD" file
     for the definition of the "LogBefore" string. */
  AppendLogfile("MyDefrag.log","LogBefore")

  /* -------- SystemDiskDaily.MyD -------- */

  /* Place files at 30% into the disk. */
  MakeGap(RoundUp(VolumeSize * 0.3,VolumeSize * 0.01))

  /* Zone 1: Place the MFT and some other special NTFS files. */
  FileSelect
    SelectNtfsSystemFiles(yes)
  FileActions
    PlaceNtfsSystemFiles(Ascending,MftSize * 0.1)
  FileEnd

  /* Zone 2: Directories. */
  FileSelect
    Directory(yes)
  FileActions
    SortByName(Ascending)
  FileEnd

  /* -------- ..\Example Scripts\MoveToEndOfDisk.MyD -------- */

  /* Place your normal zones here. The zone with files at the end of the
     disk must be the last zone. */
  /* .... */

  /* Move all remaining files to the end of the disk. */
  FileSelect  all
              & not (
                DirectoryName("raw")
                or DirectoryName("work")
                or DirectoryName("System Volume Information")
                )
  FileActions
    MoveToEndOfDisk()
    /* Defragment(Fast) */
    /* FastFill() */
  FileEnd

  /* Write the "after" statistics to the logfile. See the "Settings.MyD" file
     for the definition of the "LogAfter" string. */
  AppendLogfile("MyDefrag.log","LogAfter")

VolumeEnd

/* Write the footer to the logfile. See the "Settings.MyD" file for the
   definition of the "LogFooter" string. */
AppendLogfile("MyDefrag.log","LogFooter")

...I also made a script to fill the partition with as many 4GB files as possible,
to get a feel for the time required. On my machine the first 3/4 of available free space
fills up very fast; after that, there's a delay for each new file of 1 or 2 seconds
(assuming partition is defragmented; if not it can go much more slowly)

It's a script for general research, and also might be good for remote troubleshooting.
 
Note contig.exe attempts to create contiguous files (as the name says) but will create
a fragmented file if it has to; the slowdown appears to come when the fragmenting starts.

prereq: contig.exe from systernals
  (install on %path%)
usage: at a command prompt, type "cscript contig-fill.vbs d"
  where "d" is a drive letter (required)

contig-fill.vbs

Code:

Option Explicit

'** fill the disk with as many contiguous temp files as possible, then delete them
'
'@version 1.0 L Bigelow 16-Nov-2011

Dim gFileSysObj  'As Scripting.FileSystemObject
Dim gShellObj    'As Wscript.Shell

'******************************
'
'[[VBA -- uncomment this block for VB/VBA use (ie, for debugging in an IDE)
'Public Sub Main(args)
']]
    StatMsg "****************************************************"
    StatMsg "** contig-fill: fill the disk with as many contiguous"
    StatMsg "   temp files as possible, then delete them"
    StatMsg ""
    StatMsg "NOTE: systernal Config.exe must be installed on %PATH%"
    StatMsg "     http://technet.microsoft.com/en-us/sysinternals/bb897428"
    StatMsg ""
    StatMsg "usage: contig-fill <drive letter> [#] [/c#] [/nodelete]"
    StatMsg "     [#] = 1 to 4 [default 4] // file size in GB"
    StatMsg "     [/c#] // max number of files to create"
    StatMsg "     [/nodelete] // if present, do not delete temp files"
    StatMsg "****************************************************"
    StatMsg ""

'[[VBA
'    Dim countArgs: countArgs = UBound(args) + 1
'    If (Err) Then
'        StatMsg "  missing arguments"
'        QuitApp False
'        Exit Sub
'    End If
'][VBS -- uncomment this block for VBScript
    Dim countArgs: countArgs = WScript.Arguments.Count
    If (countArgs = 0) Then
        StatMsg "  missing arguments"
        QuitApp False
    End If
    Dim args(): ReDim args(countArgs - 1)
    Dim pArg
    For pArg = 0 To (countArgs - 1)
        args(pArg) = WScript.Arguments(pArg)
    Next
']]
    On Error Resume Next

    Dim sRoot:    sRoot = Left(args(0), 1) & ":\"
    Dim fSize:    fSize = 0
    Dim iMax:     iMax = 0
    Dim nodelete: nodelete = False

    For pArg = 1 To (countArgs - 1)

        If (args(pArg) = "/nodelete") Then     
            StatMsg "  /nodelete = true"
            nodelete = True
       
        ElseIf ((Left(args(pArg), 2) = "/c") And (iMax = 0)) Then
            iMax = CLng(Mid(args(pArg), 3))
            If (Err) Then
                StatMsg "  /c argument is not a valid number"
                QuitApp False
            Else
                StatMsg "  max file count = " & CStr(iMax)
            End If

        ElseIf ((CLng(args(pArg)) > 0) And (fSize = 0)) Then
            fSize = CLng(args(pArg))
            If (Err) Then
                StatMsg "  size argument is not a valid number"
                QuitApp False
            ElseIf ((fSize < 1) Or (fSize > 4)) Then
                StatMsg "  size argument is out of range"
                QuitApp False
            End If
        Else
            StatMsg "  your argument is invalid"
            QuitApp False
        End If
    Next

    If (fSize = 1) Then
        StatMsg "  temp file size = 1GB"
        fSize = "1073741822" '1GB - 2 bytes
    ElseIf (fSize = 2) Then
        StatMsg "  temp file size = 2GB"
        fSize = "2147483646" '2GB - 2 bytes
    ElseIf (fSize = 3) Then
        StatMsg "  temp file size = 3GB"
        fSize = "3221225470" '3GB - 2 bytes
    Else
        StatMsg "  temp file size = 4GB"
        fSize = "4294967294" '4GB - 2 bytes; max supported by contig.exe ??
    End If

    Set gFileSysObj = CreateObject("Scripting.FileSystemObject")
    If (Err) Then
        StatMsg "  error 702: " & Err.Description
        QuitApp False
    End If

    Set gShellObj = CreateObject("WScript.Shell")

    If (gFileSysObj.DriveExists(args(0)) = False) Then
        StatMsg "  error: drive """ & args(0) & """ not found"
        QuitApp False
    End If

    Dim sPath    'As String
    Dim errLvl   'As Long
    Dim i:      i = 0
   
    Dim TMPFILE: TMPFILE = "$contig"
    Dim LOGFILE: LOGFILE = gShellObj.ExpandEnvironmentStrings("%temp%") & "\contig-fill.log"
   
    If (gFileSysObj.FileExists(LOGFILE)) Then
        gFileSysObj.DeleteFile LOGFILE
    End If

    StatMsg "  creating contig files in " & sRoot & " ..."

    Do     
        i = i + 1
        If ((i > iMax) And (iMax > 0)) Then
            Exit Do
        End If

        sPath = sRoot & TMPFILE & CStr(i) & ".tmp"
       
        If (gFileSysObj.FileExists(sPath)) Then
            gFileSysObj.DeleteFile sPath
        End If

        Dim startTime: startTime = Now

        errLvl = gShellObj.Run( _
            "cmd /c contig -v -n " & sPath & " " & fSize & " >> " & LOGFILE, 7, True)
       
        StatMsg "  created " & sPath & " in " & _
                       FormatNumber(86400 * (Now - startTime), 0, True, True, False) & " sec"

        If (Err) Then
            StatMsg "  error: " & Err.Description
            Exit Do
        ElseIf (errLvl) Then
            'StatMsg "  errorlevel: " & errLvl
            Exit Do
        End If   
    Loop

    If Not (nodelete) Then
        StatMsg "  deleting temp files..."
        gShellObj.Run "cmd /c del /q " & sRoot & TMPFILE & "*.tmp", 7, True
    End If

    StatMsg "...done"
    QuitApp True

'End Sub

'********************************
'
Sub QuitApp(ByVal success)

'    If (success) Then
'        'continue
'    Else
''[[VBA
''Debug.Assert (False) 'step
'']]VBA
'       StatMsg "  error: " & Err.Description
'    End If

'[[VBA
'    End
'][VBS
   WScript.Quit
']]
End Sub

'********************************
'** show/log message
'
Sub StatMsg(ByVal msg)

    On Error Resume Next
'[[VBA
'    Debug.Print msg
'][VBS
    WScript.Echo msg
']]
    Err.Clear

End Sub

Any thought or suggestions welcome. Do you agree with my placement of the
MFT at 30% of volume? Do I need to worry about its' being moved *back* the
the front if other defrag scripts are run? If so, maybe I should stick to common
practice....

[Darlis]

Do you agree with my placement of the MFT at 30% of volume?

Since the MFT is frequently accessed, it should be where the most reads/writes occur.

Do I need to worry about its' being moved *back* the the front if other defrag scripts are run? If so, maybe I should stick to common practice....

If other script won't put the MFT at 30% of the disk or won't ignore it, they will of course move it to somewhere else.
But since most scripts will move all files to the beginning of the disk (again), the MFT is your least problem.
You have to use compatible scripts to avoid unnecessary data movement.

[Raffriff]

But since most scripts will move all files to the beginning of the disk (again), the MFT is your least problem.

oh, right
Anyway, it turns out there's no significant change in HDD benchmarks with either the above script, or "data disk daily" (on a short-stroke partition), so I will stick with the standard defrag scripts, together with occasional reformatting as you advise. Thanks for your replies!

[Raffriff]

Hi, for any Fraps users or other people doing video capture, I'd like to share my updated scripts. They're working very well for me. Using these, together with using a dedicated, "short-stroke" partition, and allowing plenty of free space, has improved the sequential write performance of my drive by almost 20 percent.

Nothing special about them, except for the handling of any folders named "raw" or "work". There's a normal version and a "quick" version, but the normal one is pretty quick and leaves more contiguous free space.

It's assumed the Recycle Bin is disabled for this partition.

EDIT - updated with suggestions from Darlis

fraps04.myd
EDIT - if you're new to MyDefrag and you don't know what to do with this code:
you save it as "fraps04.myd" under the MyDefrag installation folder in "Scripts".
(If that's too much trouble, just use the built in "Consolidate free space" script.)

Code:

# MyDefrag v4.0 script for Fraps partitions
#
# L Bigelow / raffriff
# ver 0.3 02-Dec-2011
# ver 0.4 06-Dec-2011

Title('Fraps 0.4')
Description('
Process a partition dedicated to video capture:
mostly free space, but with a few auxiliary files, ie project settings,
etc. This routine attempts to create a single contiguous area of free space,
while being as quick as possible.

Places the MFT and the directories at start of the disk.

Performs a quick defragment on any folders named "raw" or "work".
These are files that change frequently.

Moves contents of all other folders to the end of the disk.
These are files that change infrequently; NOTE there should not be too many
of this type of file in your capture partition.
')

/* Write the header to the logfile. See the "Settings.MyD" file for the
   definition of the "LogHeader" string. */
WriteLogfile("Fraps.log","LogHeader")

/* Select and process the volumes one by one. */
VolumeSelect
  CommandlineVolumes()
VolumeActions

  /* Write the "before" statistics to the logfile. See the "Settings.MyD" file
     for the definition of the "LogBefore" string. */
  AppendLogfile("Fraps.log","LogBefore")

  /* Place files at 30% into the disk. (NOT USED) */
  /* MakeGap(RoundUp(VolumeSize * 0.3,VolumeSize * 0.01)) */

  /* Zone 1: Place the MFT and any special NTFS files. */
  FileSelect
    SelectNtfsSystemFiles(yes)
  FileActions
    PlaceNtfsSystemFiles(Ascending,MftSize * 0.1)
  FileEnd

  /* Zone 2: Directories. */
  FileSelect
    Directory(yes)
  FileActions
    SortByName(Ascending)
  FileEnd

  /* Place your normal zones here. The zone with files at the end of the
     disk must be the last zone. */

  MakeGap(RoundUp(ZoneBegin,VolumeSize * 0.001))

  FileSelect (
              DirectoryName("raw")
              or DirectoryName("work")
              )
  FileActions
    Defragment(ChunkSize(100))
    FastFill(WithShuffling) // WithShuffling ver 0.4
  FileEnd

  /* Move all remaining files to the end of the disk. */
  FileSelect  all
              & not (
                DirectoryName("raw")
                or DirectoryName("work")
                or DirectoryName("System Volume Information")
              )
  FileActions
    /* Defragment(ChunkSize(100)) // removed ver 0.4 */
    MoveToEndOfDisk()
  FileEnd

  /* Write the "after" statistics to the logfile. See the "Settings.MyD" file
     for the definition of the "LogAfter" string. */
  AppendLogfile("Fraps.log","LogAfter")

VolumeEnd

/* Write the footer to the logfile. See the "Settings.MyD" file for the
   definition of the "LogFooter" string. */
AppendLogfile("Fraps.log","LogFooter")

fraps04-quick.myd

Code:

# MyDefrag v4.0 script for Fraps partitions
#
# L Bigelow / raffriff
# ver 0.3 02-Dec-2011
# ver 0.4 06-Dec-2011

Title('Fraps 0.4 quick')
Description('
Process a partition dedicated to video capture:
mostly free space, but with a few auxiliary files, ie project settings,
etc. This routine attempts to create a single contiguous area of free space,
while being as quick as possible.

Places the MFT and the directories at start of the disk.

Ignores any folders named "raw" or "work".
These are files that change frequently.

Moves contents of all other folders to the end of the disk.
These are files that change infrequently; NOTE there should not be too many
of this type of file in your capture partition.
')

/* Write the header to the logfile. See the "Settings.MyD" file for the
   definition of the "LogHeader" string. */
WriteLogfile("Fraps.log","LogHeader")

/* Select and process the volumes one by one. */
VolumeSelect
  CommandlineVolumes()
VolumeActions

  /* Write the "before" statistics to the logfile. See the "Settings.MyD" file
     for the definition of the "LogBefore" string. */
  AppendLogfile("Fraps.log","LogBefore")

  /* Place files at 30% into the disk. (NOT USED) */
  /* MakeGap(RoundUp(VolumeSize * 0.3,VolumeSize * 0.01)) */

  /* Zone 1: Place the MFT and any special NTFS files. */
  FileSelect
    SelectNtfsSystemFiles(yes)
  FileActions
    PlaceNtfsSystemFiles(Ascending,MftSize * 0.1)
  FileEnd

  /* Zone 2: Directories. */
  FileSelect
    Directory(yes)
  FileActions
    SortByName(Ascending)
  FileEnd

  /* Place your normal zones here. The zone with files at the end of the
     disk must be the last zone. */

  /*** NOT USED IN QUICK VERSION ***
  MakeGap(RoundUp(ZoneBegin,VolumeSize * 0.001))

  FileSelect (
              DirectoryName("raw")
              or DirectoryName("work")
              )
  FileActions
    Defragment(ChunkSize(100))
    FastFill(WithShuffling) // WithShuffling ver 0.4
  FileEnd
  */

  /* Move all remaining files to the end of the disk. */
  FileSelect  all
              & not (
                DirectoryName("raw")
                or DirectoryName("work")
                or DirectoryName("System Volume Information")
              )
  FileActions
    /* Defragment(ChunkSize(100)) // removed ver 0.4 */
    MoveToEndOfDisk()
  FileEnd

  /* Write the "after" statistics to the logfile. See the "Settings.MyD" file
     for the definition of the "LogAfter" string. */
  AppendLogfile("Fraps.log","LogAfter")

VolumeEnd

/* Write the footer to the logfile. See the "Settings.MyD" file for the
   definition of the "LogFooter" string. */
AppendLogfile("Fraps.log","LogFooter")

[Darlis]

Note that MoveToEndOfDisk() completely defragments files, so Defragment(ChunkSize(100)) is just doing unnecessary extra work.

For the "slow" script you might want to use FastFill(WithShuffling) to get rid of small gaps.

[Raffriff]

Thank you for your advice! I will revise the scripts. I don't care about tiny gaps, certainly anything under 100MB would be useless when recording; but yes, for the "slow" script I guess FastFill(WithShuffling) would yield more contiguous free space.

EDIT 14-Dec: The standard "Consolidate free space" script also works well and is quite fast.