Custom session in floating window mode requires XQuartz to be installed on OS X. You can install it by opening Applications => Utilities => X11 and by following the instructions shown there. More details can be found here: http://support.apple.com/en-is/HT201341
