packed_ref_store: get rid of the `ref_cache` entirely