All quantum systems are open to some extent, i.e. they interact with their environment. In this thesis, we develop novel techniques to control these system-bath interactions and then demonstrate through experiments their significant influence on system properties and dynamics. We develop a novel imaging technique in the context of ultracold lattice gases. This imaging technique allows us to tune the rate at which the atoms are measured (which can be thought of as an interaction with the electromagnetic radiation environment) over several orders of magnitude, without concomitant heating or loss of the atoms. Using this technical ability, we show that in the weak measurement limit, the atoms undergo unabated quantum evolution, i.e. they freely tunnel around the lattice, whereas as the measurement strength is increased, the tunnelling gets suppressed, the coherence is lost, and the atoms approach a classical limit of slower diffusion; demonstrating the influence of the degree of system-bath interactions on the system's dynamics. Moreover, the dissipation of open systems also allows for the realization of driven-dissipative phase transitions. We demonstrate and characterize such a phase transition in a system of ultrahigh-Q optomechanical Silicon Nitride membrane resonators, and then employ it to study the influence of system-bath interactions on criticality and phase transitions. In particular, we develop an active feedback protocol that allows us to change not only the strength of the resonators' interactions with the bath but also the very nature of their interactions (non-Markovian vs Markovian). We experimentally demonstrate that these can markedly influence the criticality of the driven-dissipative phase transition through measurements of critical and scaling exponents, which significantly change with changing system-bath interactions. Furthermore, we also demonstrate that the very phases that the system supports can be influenced by the interactions – a class of non-Markovian interactions is shown to effect a phase, a nonequilibrium steady state, that has no analog in the Markovian case. Lastly, we consider a couple of applications of these resonator systems to enhance force-sensing capabilities. We also discuss the future prospects of such control techniques and other extensions of the works presented in this thesis for gaining further insights into the influence of system-bath interactions on system properties and behavior. [1] Nondestructive imaging of an ultracold lattice gas, Y. S. Patil, S. Chakram, L. M. Aycock, and M. Vengalattore, Physical Review A, 90, 033422 (2014) [2] Measurement-induced localization of an ultracold lattice gas, Y. S. Patil, S. Chakram, and M. Vengalattore, Physical Review Letters 115, 140402 (2015) [3] Thermomechanical Two-Mode Squeezing in an Ultrahigh-Q Membrane Resonator, Y. S. Patil, S. Chakram, L. Chang, and M. Vengalattore, Physical Review Letters 115, 017202 (2015) [4] Critical behavior of a driven dissipative system: Universality beyond the Markovian regime, Y. S. Patil, H. F. H. Cheung, T. Villazon, A. G. Date, A. Polkovnikov, A. Chandran, and M. Vengalattore [5] Emergent dynamical order and time translation symmetry breaking due to non-Markovian system-bath interactions, Y. S. Patil, H. F. H. Cheung, and M. Vengalattore [6] Back-action evading measurements of two quadratures using a parametric coupling, Y. S. Patil, S. Chakram, and M. Vengalattore [7] Multimode phononic correlations in a nondegenerate parametric amplifier, S. Chakram, Y. S. Patil, and M. Vengalattore, New Journal of Physics 17, 063018 (2015) [8] Emergent phases and novel critical behavior in a non-Markovian open quantum system, H. F. H. Cheung, Y. S. Patil, and M. Vengalattore, Physical Review A (2018) [9] Demonstration of enhanced force sensitivity using a transient squeezing protocol, H. F. H. Cheung, Y. S. Patil, L. Chang, S. Chakram, and M. Vengalattore [10] Dissipation in ultrahigh quality factor SiN membrane resonators, S. Chakram, Y. S. Patil, L. Chang, and M. Vengalattore, Physical Review Letters 112, 127201 (2014)