mp4: handle memory error and cleanup