Properly keep track of the base of relative relocations