char: remove the right fd been watched in qemu_chr_fe_set_handlers()