Department Publications

Separation of simultaneously active multiple speakers is a difficult task in environments with strong reverberation and many background noise sources. This paper uses the relative transfer matrix (ReTM), a generalization of the relative transfer function of a room, to propose a simple yet effective approach for separating concurrent speakers using noisy multichannel microphone recordings. The method is built upon the covariance subtraction to enable practical applicability to mixtures of independent sound sources. The proposed method (i) allows multiple speech and background noise sources, (ii) includes reverberation, (iii) does not need the knowledge of the locations of speech and noise sources nor microphone locations and their relative geometry, and (iv) uses a relatively small segment of recordings for training. We illustrate the speaker separation performance through a simulation study, particularly under …

Voice communication is essential in post-disaster scenarios where users may be unable to type due to injury, stress, or environmental constraints. While Meshtastic provides a decentralized LoRa-based mesh network for long-range messaging, it does not natively support voice and is limited by low data rates and strict regulatory duty-cycle requirements. This paper investigates two methods for enabling voice over such networks: waveform-based audio packetization and a semantic speech-to-text (STT)/text-to-speech (TTS) approach. Preliminary experimental evaluations demonstrate that waveform-based methods, even when using low-bitrate codecs like Codec2, are impractical for urgent communication; delivering a 3-second voice message requires hundreds of packets and several minutes due to ransmission latency and duty-cycle constraints. In contrast, the proposed offline STT/TTS method transcribes speech locally, transmits it as a standard text payload, typically in a single packet and reconstructs the audio at the receiver. This semantic approach reduces delivery time from minutes to seconds, while maintaining high reliability and regulatory compliance. The study concludes that semantic communication is the most viable solution for voice interaction in bandwidth-constrained LoRa mesh environments.

The Relative Transfer Matrix (ReTM), recently introduced as a generalization of the relative transfer function for multiple receivers and sources, shows promising performance when applied to speech enhancement in noisy environments. Estimating the ReTM of sound sources by exploiting the covariance matrices of multichannel recordings is highly beneficial for practical applications and, to date, remains the only proposed approach. This paper investigates deep learning-based ReTM estimation. We propose three novel supervised learning frameworks using time and short-time frequency transform domain convolutional networks, and a Long Short-Term Memory-based recurrent neural network. Experimental results demonstrate that the proposed models achieve more accurate estimation of the ReTM using five objective metrics compared to the covariance-based method. We also show the effectiveness of the proposed frameworks for speech enhancement, achieving performance on par with the baseline method.

Isolating a desired speech signal in noisy multi-talker conversational scenarios is a key requirement for augmented reality (AR) wearable microphone array systems. In this work, a binaural target speaker extraction (TSE) framework, termed BiTSE, is proposed. It leverages both spatial and temporal cues, specifically the direction-of-arrival (DoA) of the target speaker and corresponding voice activity information, to guide the extraction process. Built upon a binaural signal denoising architecture, our model integrates three key enhancements: (i) a DoA-aware attention mechanism using cyclic positional embeddings, (ii) a timestamp-based masking strategy that utilizes speaker activity to suppress non-target segments, and (iii) a novel two-stage loss optimization strategy that first trains the model for robust denoising and then fine-tunes it to improve perceptual quality. Evaluations on the SPeech Enhancement for Augmented Reality (SPEAR) challenge dataset demonstrate that the proposed BiTSE consistently improves upon conventional approaches, leading to enhanced signal fidelity and perceptual quality.

Drone audio broadcasting enables rapid, flexible speech communication in scenarios such as search and rescue, disaster response, and public safety by projecting speech from onboard loudspeakers. However, speech played from drones is contaminated by its own intense, time-varying rotor noise, while increasing playback level is often undesirable as it raises concerns about privacy and noise pollution. Speech intelligibility improvement methods could be used to improve the intelligibility of speech under noise without increasing its sound level. To date, most methods have been evaluated using conventional maskers such as speech shaped noise or babble noise, rather than fluctuating drone noise. This study presents a subjective listening test with native New Zealand English listeners, that investigates the performance of four modern speech intelligibility improvement methods under varying input signal-to-noise ratio (SNRs) with two recorded drone noise conditions. The results show that the spectral shaping and dynamic range compression (SSDRC) and its variant consistently achieved better performance yielding up to a 72% intelligibility score improvement over unmodified speech at an input SNR of-16 dB, with diminishing gain at higher input SNR levels. Performance difference between drone types was are while the ranking of the evaluated methods was consistent across both conditions.

With growing applications of drones, there is a demand for drone audition functionality that allows microphone arrays to be embedded into or attached to the body of drones for sound source localization and signal enhancement. In practical acoustical environments, the performance of drone audition is degraded due to the free-field sound propagation assumption. In this paper, we propose (i) a method for modeling the scattering and diffraction of a drone body beyond free-field propagation as a Drone-Related Transfer Function (DRTF), and (ii) a practical method to estimate the DRTF coefficients using circular loudspeaker arrays. The method incorporates the acoustic reciprocity principle and a finite order spherical harmonic expansion by utilizing a discrete set of measurements by multiple loudspeaker placements on circular arrays. We validate our DRTF model and parameterization method on a known rigid sphere …

The Relative Transfer Matrix (ReTM), recently introduced as a generalization of the relative transfer function for multiple receivers and sources, shows promising performance when applied to speech enhancement and speaker separation in noisy environments. Blindly estimating the ReTM of sound sources by exploiting the covariance matrices of multichannel recordings is highly beneficial for practical applications. In this paper, we use covariance subtraction to present a flexible and practically viable method for estimating the ReTM for a select set of independent sound sources. To show the versatility of the method, we validated it through a speaker separation application under reverberant conditions. Separation performance is evaluated at low signal-to-noise ratio levels in comparison with existing ReTM-based and relative transfer function-based estimators, in both simulated and real-life environments.

This review outlines the current status and proposes future directions for using auditory brainstem responses (ABRs) to continuous speech as a tool in hearing aid assessment. In current clinical practice, ABRs measured via electroencephalography (EEG) in response to short stimuli, averaged across repetitions, are a cornerstone of objective hearing screening and evaluation. Recent advancements have demonstrated that ABRs can be estimated from EEG recorded during continuous speech listening by using linear models which map the auditory speech features, extracted using auditory models, to continuous EEG activity. This approach enables the simultaneous investigation of multiple auditory pathway stages using complex, ecologically valid sounds; advancing potential future hearing assessments that can be seamlessly integrated into daily life. Current findings indicate that responses similar to click-ABRs …

In this paper, we consider the problem of recovering desired sound source signals from on-board microphone recordings on a noisy drone. Enhancement of source signal degraded by drone noise is considered to be a difficult task due to the strong noise generated from its motors and propellers causing an extremely low signal-to-drone noise ratio (S D ‾ NR). We propose a solution (i) by combining the widely known multichannel Wiener filter (MWF) to remove drone noise from microphone recordings, and (ii) further reduction of residual noise using a Gaussian mixture model (GMM) based dual-stage parametric Wiener filter (WF). The method exploits known statistics of motor current-specific drone noise. This combination of techniques to the specific context of signal enhancement for drone audition is applicable to irregular microphone arrays embedded on a drone enabling realistic integration to most drones. We …

Perception of sounds and speech involves structures in the auditory brainstem that rapidly process ongoing auditory stimuli. The role of these structures in speech processing can be investigated by measuring their electrical activity using scalp-mounted electrodes. However, typical analysis methods involve averaging neural responses to many short repetitive stimuli that bear little relevance to daily listening environments. Recently, subcortical responses to more ecologically relevant continuous speech were detected using linear encoding models. These methods estimate the temporal response function (TRF), which is a regression model that minimises the error between the measured neural signal and a predictor derived from the stimulus. Using predictors that model the highly non-linear peripheral auditory system may improve linear TRF estimation accuracy and peak detection. Here, we compare predictors from both simple and complex peripheral auditory models for estimating brainstem TRFs on electroencephalography (EEG) data from 24 participants listening to continuous speech. We also investigate the data length required for estimating subcortical TRFs, and find that around 12 minutes of data is sufficient for clear wave V peaks (>3 dB SNR) to be seen in nearly all participants. Interestingly, predictors derived from simple filterbank-based models of the peripheral auditory system yield TRF wave V peak SNRs that are not significantly different from those estimated using a complex model of the auditory nerve, provided that the nonlinear effects of adaptation in the auditory system are appropriately modelled. Crucially, computing predictors …

This paper presents several machine learning methods and highlights the most effective one for detecting elephant rumbles in infrasonic seismic signals. The design and implementation of electronic circuitry to amplify, filter, and digitize the seismic signals captured through geophones are presented. The process converts seismic rumbles to a spectrogram and the existing methods of spectrogram feature extraction and appropriate machine learning algorithms are compared on their merit for automatic seismic rumble identification. A novel method of denoising the spectrum that leads to enhanced accuracy in identifying seismic rumbles is presented. It is experimentally found that the combination of the Mel-frequency cepstral coefficient (MFCC) feature extraction method and the ridge classifier machine learning algorithm give the highest accuracy of 97% in detecting infrasonic elephant rumbles hidden in seismic signals. The trained machine learning algorithm can run quite efficiently on general-purpose embedded hardware such as a Raspberry Pi, hence the method provides a cost-effective and scalable platform to develop a tool to remotely localize elephants, which would help mitigate the human–elephant conflict.

The auditory brainstem response (ABR) is a valuable clinical tool for objective hearing assessment, which is conventionally detected by averaging neural responses to thousands of short stimuli. Progressing beyond these unnatural stimuli, brainstem responses to continuous speech presented via earphones have been recently detected using linear temporal response functions (TRFs). Here, we extend earlier studies by measuring subcortical responses to continuous speech presented in the sound-field, and assess the amount of data needed to estimate brainstem TRFs. Electroencephalography (EEG) was recorded from 24 normal hearing participants while they listened to clicks and stories presented via earphones and loudspeakers. Subcortical TRFs were computed after accounting for non-linear processing in the auditory periphery by either stimulus rectification or an auditory nerve model. Our results …

The Relative Transfer Matrix (ReTM), recently introduced as a generalization of the relative transfer function for multiple sources and multiple microphones, shows promising performance when applied to speech denoising and speaker separation in a noisy reverberant room. This work utilizes the ReTM to propose a novel framework for drone noise suppression. Difficulties in noise cancellation for drone audition applications arise due to its self-generated noise that causes an adverse noisy environment of low Signal-to-Drone Noise Ratio (SDNR) levels. In this paper, we divide the drone on-board microphones into two multichannel groups to approximately estimate the drone noise from one group to the other with known drone noise ReTM for denoising. We demonstrate the ReTM spatial mapping ability in both indoor and outdoor experiments using hovering drone on-board microphone recordings with low average …

The auditory brainstem response (ABR) is a measure of subcortical activity in response to auditory stimuli. The wave V peak of the ABR depends on the stimulus intensity level, and has been widely used for clinical hearing assessment. Conventional methods estimate the ABR average electroencephalography (EEG) responses to short unnatural stimuli such as clicks. Recent work has moved toward more ecologically relevant continuous speech stimuli using linear deconvolution models called temporal response functions (TRFs). Investigating whether the TRF waveform changes with stimulus intensity is a crucial step toward the use of natural speech stimuli for hearing assessments involving subcortical responses. Here, we develop methods to estimate level-dependent subcortical TRFs using EEG data collected from 21 participants listening to continuous speech presented at 4 different intensity levels. We find that …

Estimating the Relative Transfer Functions (ReTFs) of multiple speakers in a noisy reverberant environment are beneficial to signal processing applications of acoustic scene analysis, denoising, dereverberation, signal enhancement, and separation. Whilst most audio applications exploit the covariance matrix structure of multichannel recordings, estimating ReTF in the presence of simultaneously active multiple speakers is still considered to be a challenging problem. In this paper, we propose a novel method for estimating the ReTF using the relative transfer matrix (ReTM) of multiple microphones for multi-talker scenarios, which is suitable for noisy reverberant rooms. The method is based on the noise-only ReTM, and calculates the covariance matrices of (i) the first speaker and noise, and (ii) both speakers and noise at two microphone groups to reconstruct the ReTF of the second speaker. We demonstrate the …

With the recent emergence of virtual reality, augmented reality, and mixed-reality technologies, considerable attention has been received to integrate spatial audio recording capabilities into these devices. Extracting higher order spherical harmonics from the recordings is not always possible due to complex scattering effects from the arbitrary-shaped devices, which motivates the need for modeling the acoustic properties beyond free-field convention. In this paper, we propose modeling the wearable device-related transfer function (WDRTF) that incorporates scattering, and diffraction from the device and wearer’s body to an incident soundfield. We adopt the spherical wave model to translate the measurements into microphone alignment, by exploiting both magnitude and phase corrections. We use a discrete set of measurements by multiple circular arrays for efficiently estimating the WDRTF coefficients. We illustrate …

Multichannel Wiener filter (MWF), commonly used as a spatial filter for noise reduction, exhibits promising performance due to its robustness in multi-channel setups. Additionally, the preservation of spatial cues at the MWF is usually a favorable property that contributes to sound source localization. Previous works have shown that the MWF can be used in drone noise suppression for acoustic sensing tasks in drone audition. In this paper, we analyze the preservation of spatial cues at the MWF from on-board microphone recordings on a drone at very low signal-to-drone noise ratio (SDNR) levels. The analysis builds upon the relative transfer function (ReTF), and explores the spatial information of the desired sound sources such as human speakers and birds through experimental recordings from simulated and real-life environments. We also demonstrate the effectiveness of the preservation of spatial cues at the MWF in an acoustic localization application

The auditory brainstem response (ABR) is a measure of subcortical activity in response to auditory stimuli. The wave V peak of the ABR depends on stimulus intensity level, and has been widely used for clinical hearing assessment. Conventional methods to estimate the ABR average electroencephalography (EEG) responses to short unnatural stimuli such as clicks. Recent work has moved towards more ecologically relevant continuous speech stimuli using linear deconvolution models called Temporal Response Functions (TRFs). Investigating whether the TRF waveform changes with stimulus intensity is a crucial step towards the use of natural speech stimuli for hearing assessments involving subcortical responses. Here, we develop methods to estimate level-dependent subcortical TRFs using EEG data collected from 21 participants listening to continuous speech presented at 4 different intensity levels. We find that level-dependent changes can be detected in the wave V peak of the subcortical TRF for almost all participants, and are consistent with level-dependent changes in click-ABR wave V. We also investigate the most suitable peripheral auditory model to generate predictors for level-dependent subcortical TRFs and find that simple gammatone filterbanks perform the best. Additionally, around 6 minutes of data may be sufficient for detecting level-dependent effects and wave V peaks above the noise floor for speech segments with higher intensity. Finally, we show a proof-of-concept that level dependent subcortical TRFs can be detected even for the inherent intensity fluctuations in natural continuous speech. Visual abstract Significance …

Making drones talk is the next step in Human-Unmanned Aerial Vehicle interactions by installing an on-board loudspeaker array. However, due to the drone being loud, it creates an adverse noisy environment of very low signal-to-drone noise ratio (SDNR) levels (less than -10 dB) making it difficult to project a perceptual soundfield to a desired location on the ground. In this paper, we present a preliminary study on developing a sparsity-based interior response model of a commercial loudspeaker that can be used to modify the speech signals before playback through the loudspeaker. We incorporate the ground reflections into the incident response model of the real loudspeaker to simulate the outdoor environment. We validate the proposed model using soundfield recording over a larger region for low frequencies up to 114 Hz.

This thesis explores the acoustic signal processing on drones which can enhance the drone audition functions for search and rescue missions, wildlife monitoring, and video recording for media and filming industries. With the recent emergence of drone-based applications and services, microphone arrays are beginning to be mounted to drones. However, audio capturing using drone-mounted microphone arrays leads to very low signal-to-drone noise ratio (SDNR) recordings (below negative 10 dB) due to the drone-generated motor and propeller noise signals. Hence, this causes a highly adverse noisy environment and degrades the quality and intelligibility of the recorded audio signals. Therefore, there is a growing interest in deriving new signal processing algorithms for the future deployment of the drone audition. To this end, we develop novel methods for spatial soundfield modelling, and spatial audio processing using a microphone array embedded into the body of drones for sound source localisation and audio signal enhancement. To a lesser extent, drone noise characterisation is also studied. We propose modelling of the drone noise harmonics as current to noise transfer function by taking the significant harmonics of the drone noise signature. The model is derived from the experimental measurements of a single motor propeller combination, and validated using extensive error analysis. A cross-correlation-based DOA estimation method with drone noise angular spectrum subtraction is derived for acoustic source localisation. Our approach is to exploit the underlying spatial characteristics of drone noise to mitigate the noise emission by …

With the recent emergence of virtual reality, augmented reality, and mixed-reality technologies, considerable attention has been received to integrate spatial audio recording capabilities into these devices. Extracting higher order spherical harmonics from the recordings is not always possible due to complex scattering effects from the arbitrary-shaped devices, which motivates the need for modeling the acoustic properties beyond free-field convention. In this paper, we propose modeling the wearable device-related transfer function (WDRTF) that incorporates scattering, and diffraction from the device and wearer's body to an incident soundfield. We adopt the spherical wave model to translate the measurements into microphone alignment, by exploiting both magnitude and phase corrections. We use a discrete set of measurements by multiple circular arrays for efficiently estimating the WDRTF coefficients. We illustrate the WDRTF modeling procedure from a head-worn microphone array and obtain accurate reconstruction performance over frequency and 3D directions.

The Relative Transfer Matrix (ReTM), recently introduced as a generalization of the relative transfer function for multiple sources and multiple microphones, shows promising performance when applied to speech denoising and speaker separation in a noisy reverberant room. This work utilizes the ReTM to propose a novel framework for drone noise suppression. Difficulties in noise cancellation for drone audition applications arise due to its self-generated noise that causes an adverse noisy environment of low Signal-to-Drone Noise Ratio (SDNR) levels. In this paper, we divide the drone on-board microphones into two multichannel groups to approximately estimate the drone noise from one group to the other with known drone noise ReTM for denoising. We demonstrate the ReTM spatial mapping ability in both indoor and outdoor experiments using hovering drone on-board microphone recordings with low average magnitude spectrum error. Finally, we validate the method in a real-life environment for source signal enhancement over different SDNR conditions and offer both improved speech intelligibility and signal-to-distortion ratio.

Neural speech tracking has advanced our understanding of how our brains rapidly map an acoustic speech signal onto linguistic representations and ultimately meaning. It remains unclear, however, how speech intelligibility is related to the corresponding neural responses. Many studies addressing this question vary the level of intelligibility by manipulating the acoustic waveform, but this makes it difficult to cleanly disentangle the effects of intelligibility from underlying acoustical confounds. Here, using magnetoencephalography recordings, we study neural measures of speech intelligibility by manipulating intelligibility while keeping the acoustics strictly unchanged. Acoustically identical degraded speech stimuli (three-band noise-vocoded, ~20 s duration) are presented twice, but the second presentation is preceded by the original (nondegraded) version of the speech. This intermediate priming, which generates a …

Auditory cortical responses to speech obtained by magnetoencephalography (MEG) show robust speech tracking to the speaker's fundamental frequency in the high-gamma band (70–200 Hz), but little is currently known about whether such responses depend on the focus of selective attention. In this study 22 human subjects listened to concurrent, fixed-rate, speech from male and female speakers, and were asked to selectively attend to one speaker at a time, while their neural responses were recorded with MEG. The male speaker's pitch range coincided with the lower range of the high-gamma band, whereas the female speaker's higher pitch range had much less overlap, and only at the upper end of the high-gamma band. Neural responses were analyzed using the temporal response function (TRF) framework. As expected, the responses demonstrate robust speech tracking of the fundamental frequency in the high-gamma band, but only to the male's speech, with a peak latency of ~40 ms. Critically, the response magnitude depends on selective attention: the response to the male speech is significantly greater when male speech is attended than when it is not attended, under acoustically identical conditions. This is a clear demonstration that even very early cortical auditory responses are influenced by top-down, cognitive, neural processing mechanisms.