server: Wait for connection threads to exit before unloading plugins.
A frequent cause of crashes on shutdown happens when the quit signal
has been received and the main thread and other threads race each
other at shutdown. As in the typical stack trace below what happens
is that the main thread unloads the plugins while they are still being
used by one of the connection threads, causing the connection thread
to segfault after calling dlclosed functions.
To avoid this simply count the number of connection threads we create,
and at exit in the main thread wait until this count drops to 0 before
we actually start unloading anything.
Thread 2 (Thread 0x7f8fb054ea40 (LWP
3105233)):
#0 _asn1_set_down (down=0x5633dd26c010, node=0x5633dd26be70) at parser_aux.h:116
#1 asn1_delete_structure2 (structure=structure@entry=0x7f8fb0dd3fa0 <_gnutls_pkix1_asn>, flags=flags@entry=0) at structure.c:328
#2 0x00007f8fb06e702b in asn1_delete_structure (structure=structure@entry=0x7f8fb0dd3fa0 <_gnutls_pkix1_asn>) at structure.c:293
#3 0x00007f8fb0c66544 in _gnutls_global_deinit (destructor=1) at global.c:419
#4 0x00007f8fb0e0d13b in _dl_fini () at dl-fini.c:138
#5 0x00007f8fb0a0ae87 in __run_exit_handlers (status=0, listp=0x7f8fb0b8e578 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#6 0x00007f8fb0a0b040 in __GI_exit (status=<optimized out>) at exit.c:139
#7 0x00005633d47626ee in main (argc=<optimized out>, argv=0x7ffecd6115d8) at main.c:705
Thread 1 (Thread 0x7f8fb054d700 (LWP
3105257)):
#0 backend_finalize (b=<optimized out>, conn=conn@entry=0x5633dd287c40) at backend.c:227
#1 0x00005633d476579f in handle_single_connection (sockin=<optimized out>, sockout=<optimized out>) at connections.c:222
#2 0x00005633d476e9e9 in start_thread (datav=0x5633dd269620) at sockets.c:351
#3 0x00007f8fb0b9e4e2 in start_thread (arg=<optimized out>) at pthread_create.c:479
#4 0x00007f8fb0acd643 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95