Computer graphics techniques enable the creation of rich digital content that can react to the external environment in a physically realistic manner. Advances in digital data acquisition and portable display devices carry the promise of extending the capability of these techniques beyond animation to revolutionary new use cases, such as medical diagnosis and treatment, computational design and fabrication, and online education. To unleash the full potential of these methods, however, there is a need for computational algorithms and data structures that allow for high-fidelity simulations in interactive settings. A particularly interesting and challenging aspect of this problem is that of organizing computation on modern hardware platforms that are becoming increasingly heterogeneous, i.e., workstations equipped with several bandwidth-optimized accelerator cards. In this talk, I will detail the steps I have taken towards addressing this challenge. In particular, I will present a data structure that exploits the virtual memory management system to efficiently store and process multiple data channels on highly irregular voxelized domains with over a billion degrees of freedom, I will describe a numerical solver that benefits from the high memory and compute bandwidth of GPU accelerators even for problem sizes that are too large to fit entirely on GPU memory, and I will briefly summarize methods for simulating complex multi-material interactions with dynamic objects.