Add a consistent hash function to util/hash
Summary:
Add a consistent hash function to util/hash. A consistent hash maps a
key to some set of buckets (like any hash), but has the added property
that changing the number of buckets tends to only minimally disturb
the mapping. That is, most keys will continue to map to the same
bucket they did previously. This is opposed to normal hash functions,
where any change tends to radically change the output (by design).
Use an algorithm from "A Fast, Minimal Memory, Consistent Hash
Algorithm" by John Lamping and Eric Veach. This algorithm is compact
and relatively quick. It also doesn't require data structures scaling
with the number of buckets (like many older consistent hashes like
rendezvous hashing).
Consistent hashing is useful with extern-worker because you want to
batch work together in ways that is robust against new data being
added/removed across runs.
Reviewed By: edwinsmith
Differential Revision:
D34746411
fbshipit-source-id:
79beb1c9015afba385e87830f9f44070c39ce3f9