Spatial computing of sound fields in virtual environment

Abstract

Over the past decades, computer graphics researchers have put enormous effort into rendering realistic visual scenes by simulating light transport. With the high-level goal of creating realistic immersive experiences in virtual worlds, physically plausible sound is a critical piece and remains to be explored. Humans’ experience when perceiving sound is spatially varying and scene dependent, e.g. whether a point sound source is occluded or not with respect to listener will lead to a different perceived sound. Simulating sound propagation is the key to reproducing such effects but it differs from light transport simulation in visual rendering due to the importance of diffraction effects. By using physical simulations, the grand goal of this thesis is to provide auditory cues that respect the influence of the virtual environment. We address this problem by precomputing an expensive simulation of sound wave propagation through a voxelized 3D scene and encoding perceptually important acoustic parameters per voxel from the simulation data, which enables efficient real-time sound rendering at run-time. All methods proposed are immediately practical with potential applications in AR/VR and gaming. Our first contribution is proposing a framework that simulates ambient sound propagation in a preprocessing stage and reconstructs ambient sound efficiently at render time. By modelling spatio-temporally incoherent ambient sound source appropriately in numerical simulation, a streaming encoder captures the loudness and directivity per listener position compactly in spherical harmonics coefficients. The encoded coefficients are further coupled with Head-related transfer function (HRTF) data, rendering physically plausible binaural ambient sound at run-time. We then observe that in most ambient sound scenarios, the sound texture perceived varies in space. For example, near a water stream, crisp water bubble sounds are audible with transient details, whereas it becomes closer to a randomized colored noise in far field. A more compelling example is a babbling crowd, in which individual speech is recognizable next to a person but not when far away from the crowd. The intuition is that the perceived ambient sound is a random collection of similar micro sound events and the variation of atomic sound events’ temporal density and their distribution of amplitudes leads to different sound textures. We propose a simple ambient sound texture representation in terms of an event density function (EDF). By modelling micro sound events directly in the precomputed simulation phase, EDF is compactly encoded. At run-time, sound is rendered by real-time granular synthesis, resulting in a spatially varying sound texture that enhances the experience of ambient sound in a virtual environment.