sink, source: Fix setting the latency offset when the sink/source is unlinked.