Allocate cached_refs objects dynamically