I've been using IOBit for over a month with "Defrag System Directories" checked, no problems at all.
I'm not sure this is the reason for the checkbox or not, but in theory, you shouldn't need to defrag system directories. System directories are just that... directories that hold your system files. Since your system files should only be changed when you are installing an update or a service pack, theoretically they never need to be defragged except after an update or a service pack install. Realistically though, software (both legimate and malware) have a tendency to write to and replace files in the system directories. This is what causes many system instabilities. (For example, after testing their product extensively, the makers of ProgramA decide that Important.dll version 2.2 works best with ProgramA and therefore include it in the software's install program. When ProgramA is installed, Important.dll version 2.2 overwrites version 1.5 that ProgramB and ProgramC require to run because they're older applications that don't work with the new features of Important.dll version 2.2. When ProgramB is launched, it gives an error and blue screens the computer. ProgramC is not as old as ProgramB, so when it's launched, there is no error. However, the version incompatibilities cause ProgramC to use more RAM than it normally does and it doesn't release it when it's finished. This eventually causes the system to run out of memory and crash. The makers of ProgramA don't care about this because they don't make ProgramB or ProgramC and ProgramA is working fine.)
Wow. That may be the most complicated example I've ever come up with. Sorry if that made you more confused than before you read it.