Only set the needed HRTF coefficients