Optimize symlink/directory detection