server: Keep a file descriptor open to the config directory to make sure we don't...