[XFS] Don't update i_size for directories and special files