demux: add sanity checks to packet allocation functions

demux: add sanity checks to packet allocation functions

Change new_demux_packet() and resize_demux_packet() length parameter
type from int to size_t and add a check to abort() if the size is over
1 GB. This should make integer overflow problems leading to memory
corruption in demuxers less likely; and aborting should be no worse
than insane memory consumption. Also make the functions abort() if the
actual allocation fails instead of trying to continue with a
zero-sized buffer.