Department Publications

Separation of simultaneously active multiple speakers is a difficult task in environments with strong reverberation and many background noise sources. This paper uses the relative transfer matrix (ReTM), a generalization of the relative transfer function of a room, to propose a simple yet effective approach for separating concurrent speakers using noisy multichannel microphone recordings. The method is built upon the covariance subtraction to enable practical applicability to mixtures of independent sound sources. The proposed method (i) allows multiple speech and background noise sources, (ii) includes reverberation, (iii) does not need the knowledge of the locations of speech and noise sources nor microphone locations and their relative geometry, and (iv) uses a relatively small segment of recordings for training. We illustrate the speaker separation performance through a simulation study, particularly under …

Voice communication is essential in post-disaster scenarios where users may be unable to type due to injury, stress, or environmental constraints. While Meshtastic provides a decentralized LoRa-based mesh network for long-range messaging, it does not natively support voice and is limited by low data rates and strict regulatory duty-cycle requirements. This paper investigates two methods for enabling voice over such networks: waveform-based audio packetization and a semantic speech-to-text (STT)/text-to-speech (TTS) approach. Preliminary experimental evaluations demonstrate that waveform-based methods, even when using low-bitrate codecs like Codec2, are impractical for urgent communication; delivering a 3-second voice message requires hundreds of packets and several minutes due to ransmission latency and duty-cycle constraints. In contrast, the proposed offline STT/TTS method transcribes speech locally, transmits it as a standard text payload, typically in a single packet and reconstructs the audio at the receiver. This semantic approach reduces delivery time from minutes to seconds, while maintaining high reliability and regulatory compliance. The study concludes that semantic communication is the most viable solution for voice interaction in bandwidth-constrained LoRa mesh environments.

The Relative Transfer Matrix (ReTM), recently introduced as a generalization of the relative transfer function for multiple receivers and sources, shows promising performance when applied to speech enhancement in noisy environments. Estimating the ReTM of sound sources by exploiting the covariance matrices of multichannel recordings is highly beneficial for practical applications and, to date, remains the only proposed approach. This paper investigates deep learning-based ReTM estimation. We propose three novel supervised learning frameworks using time and short-time frequency transform domain convolutional networks, and a Long Short-Term Memory-based recurrent neural network. Experimental results demonstrate that the proposed models achieve more accurate estimation of the ReTM using five objective metrics compared to the covariance-based method. We also show the effectiveness of the proposed frameworks for speech enhancement, achieving performance on par with the baseline method.

Isolating a desired speech signal in noisy multi-talker conversational scenarios is a key requirement for augmented reality (AR) wearable microphone array systems. In this work, a binaural target speaker extraction (TSE) framework, termed BiTSE, is proposed. It leverages both spatial and temporal cues, specifically the direction-of-arrival (DoA) of the target speaker and corresponding voice activity information, to guide the extraction process. Built upon a binaural signal denoising architecture, our model integrates three key enhancements: (i) a DoA-aware attention mechanism using cyclic positional embeddings, (ii) a timestamp-based masking strategy that utilizes speaker activity to suppress non-target segments, and (iii) a novel two-stage loss optimization strategy that first trains the model for robust denoising and then fine-tunes it to improve perceptual quality. Evaluations on the SPeech Enhancement for Augmented Reality (SPEAR) challenge dataset demonstrate that the proposed BiTSE consistently improves upon conventional approaches, leading to enhanced signal fidelity and perceptual quality.

Drone audio broadcasting enables rapid, flexible speech communication in scenarios such as search and rescue, disaster response, and public safety by projecting speech from onboard loudspeakers. However, speech played from drones is contaminated by its own intense, time-varying rotor noise, while increasing playback level is often undesirable as it raises concerns about privacy and noise pollution. Speech intelligibility improvement methods could be used to improve the intelligibility of speech under noise without increasing its sound level. To date, most methods have been evaluated using conventional maskers such as speech shaped noise or babble noise, rather than fluctuating drone noise. This study presents a subjective listening test with native New Zealand English listeners, that investigates the performance of four modern speech intelligibility improvement methods under varying input signal-to-noise ratio (SNRs) with two recorded drone noise conditions. The results show that the spectral shaping and dynamic range compression (SSDRC) and its variant consistently achieved better performance yielding up to a 72% intelligibility score improvement over unmodified speech at an input SNR of-16 dB, with diminishing gain at higher input SNR levels. Performance difference between drone types was are while the ranking of the evaluated methods was consistent across both conditions.