gsocketlistener: Don't double unref address