Simplify removing the context handle from the device's context array