Accelerate clear_skip_worktree_from_present_files() by caching