Extreme weather events and rising sea level have largely raised attention to shore protection. As a natural means against ocean waves, coastal vegetation on dissipating incoming wave energy has been extensively studied via mathematical/numerical modeling and laboratory/field observations. The hydrodynamics and sediment transport within vegetated areas have also received an increasing interest. However, the vegetation effects on damping waves have not been fully understood and are awaiting further investigation. In addition, the model forests in most of the existing literature are impractical and still far from real situations. Most importantly, a reliable approach to predict wave attenuation by coastal vegetation is still demanded. In this dissertation, accordingly, the theoretical development in Mei et al. (2011, 2014) is extended to study water waves propagating through a heterogeneous coastal forest of arbitrary shape. Arrays of rigid and surface-piercing cylinders are used to model the trees. Assuming that the incident wavelength is much longer than the characteristic cylinder spacing (and cylinder diameter), the multi-scale perturbation theory of homogenization (Mei and Vernescu 2010) is applied to separate the micro-scale flow problem within a unit cell from the macro-scale wave dynamics throughout the entire vegetated area. Driven by the macro-scale (wavelength-scale) pressure gradients, the flow motion within a unit cell, which can have one or a few cylinders inside, is obtained by solving the micro-scale boundary-value-problem. The cell-averaged equations governing the macro-scale wave dynamics are derived with consideration of the cell effects, which would be parameterized into the complex coefficients. Employing the boundary integral equation method, the macro-scale wave dynamics is solved numerically in which a coastal forest can be composed of multiple patches of arbitrary shape. Each patch can be further divided into several subzones based on different properties, e.g. vegetation sizes, planting arrangements and porosity. Each subzone is then considered as a homogeneous region where a constant bulk value of eddy viscosity can be yielded. The integral formulations for a forest subzone and the open water region are both provided. The matching conditions along the boundaries are also presented. Two types of small-amplitude waves are first considered. To model wind waves, the theoretical model for periodic waves of relatively short wavelength in Mei et al. (2014) is extended from two-dimensional to three-dimensional applications. The constant bulk value of eddy viscosity for each homogenous forest subzone is determined by invoking the balance of time-averaged energy dissipation rate and the averaged rate of work done by wave forces on cylinders within the specific subzone. Morison-type formula (Morison et al. 1950) is used to model the wave forces, in which the drag coefficient formula is constructed based on the experimental data in Hu et al. (2014). A computing program is established based on the present approach and can be used for solving wave dynamics with proper inputs. In the interest of understanding the capability of coastal vegetation to dissipate long waves, a transient wave of small amplitude is studied by extending the theoretical model in Mei et al. (2011). Fourier transform is employed to solve the transient problem. The constant bulk value of eddy viscosity for each homogenous forest subzone is determined by modifying the empirical formula in Mei et al. (2011). A similar computing program is also developed for solving the wave amplitude spectrum. Once the wave amplitude spectrum is obtained, the wave solution can be found by inverse Fourier transform. For simplicity, only the incident waves with a soliton-like shape are considered. Three special forest configurations with experimental data are used to check the present models for both types of waves. An infinitely long forest belt with a finite width, where the semi-analytical solutions are provided, is first used to study the wave damping inside the forest. For periodic waves, the model results of normal incidences are compared with Hu et al. (2014)’s experimental measurements. Obliquely incident waves are also discussed. For transient long waves, the experiments reported in Mei et al. (2011) are re-examined. Very good agreements with experimental data are found in both conditions. A single circular patch is then used as a special configuration to study a forest region with finite extent. The semi-analytical solutions for periodic and transient problems are respectively presented, in which negligible differences are found in comparison with the numerical results. A set of experiments was conducted at the University of Cantabria for model validation. Good agreements are found in the data-model comparisons. For transient waves, the effects of wave parameters and different arrangements of cylinders on wave damping are also discussed. The numerical model is then further explored by using a forest region consisting of multiple circular patches as used in Maza (2015). Good agreements are also observed between the simulated free surface elevations and the experimental measurements. The effectiveness of different forest configurations on wave attenuation is discussed as well. The capability of the linear models to predict wave attenuation by coastal forests has been presented for both types of waves. However, the limitation of the nonlinearity of incident waves is also confirmed by comparison with experimental data. The last part of this dissertation aims to study the possible nonlinear effects by including the convection terms in the momentum equations. Considering shallow-water incident waves of simple harmonic, higher harmonic components are expected to grow within the vegetated area and be radiated out to the open water region. Applying multi-scale perturbation technique, the micro- and macro-scale equations for each harmonic are derived. The cell problem for micro-scale flow motion, driven by the macro-scale pressure gradient, becomes a nonlinear boundary-value-problem while the macro-scale problem for wave dynamics remains linear. A modified pressure correction method is employed to solve the nonlinear cell problem. Using a forest belt as a special case, the comparisons of numerical results obtained by linear and nonlinear models are presented. The numerical results are also compared with the experimental data reported in Mei et al. (2011). Varying nonlinearity is further tested and the effects of different parameters on wave solutions are discussed as well.