MERLIN: Modern methods for the restoration of lost information in digital signals

General Information 

Bilateral project funded by the Austrian Science Fund (FWF) and the Czech Science Foundation (GACR)

Principal Investigator: Nicki Holighaus (Institut für Schallforschung, ARI, ÖAW)

Co-Principal Investigator: Pavel Rajmic (Signal Processing Laboratory, SPLab, Brno University of Technology)

Project Team (ARI) : Andres Marafioti, Rene Repp, Shristi Rajbamshi (from 12/2020)

Local Collaborators (ARI): Peter Balazs, Piotr Majdak, Günther Koliander, Georg Tauböck, Thibaud Necciari, Zdenek Prusa

Duration: 01.06.2017 – 31.05.2021 (ARI) and 01.01.2017 – 31.12.2019 (SPLab)

Project Description

Locally degraded or lost information is frequently encountered in signal processing. Some prime examples include corrupted time segments in damaged old audio recordings, missing data in compressed audio or lost blocks of time-frequency coefficients, also referred to as packet loss during transmission, e.g. in VoIP. In these scenarios, information is lost or unreliable in large connected regions in time, frequency or time-frequency domains. Hence classical denoising or declicking methods that treat the whole signal or only isolated samples, respectively, cannot be applied. Automatic procedures to recover lost segments of signal data in either domain have seen increased attention in recent years and are often collectively referred to as inpainting.

However, the methods developed so far assume simplistic signal models that fail to capture the characteristic structures of speech and music data. Thus, the core objective of MERLIN is the development of novel and innovative inpainting methods through the combination of:

1. modern adapted time-frequency representations,
2. appropriate signal models obtained from prior information and the reliable signal segments,
3. state-of-the-art signal processing and machine learning, and
4. the consideration of perceptual indicators. 

The team at ARI is concerned in particular with

• developing the mathematical theory, implementation and application of adapted time-frequency representations, in particular invertible representations inspired by auditory perception
• phase processing and phaseless reconstruction for time-frequency representations
• neural audio generation in the time-frequency domain and audio inpainting with neural networks

Further Information

Project webpage: http://merlin.utko.feec.vutbr.cz/

Publications: http://merlin.utko.feec.vutbr.cz/research-and-publications/

Keywords

adapted time-frequency representations, frame theory, perceptual signal processing, time-frequency processing, generative neural networks, audio inpainting, phaseless reconstruction, neural audio generation