Store the buffer handle directly in the source and buffer queue list