The higher-order-mode (HOM) fiber has recently received great attention due to its special optical properties. The most significant one is its capability of propagating different modes, which is now being utilized to increase data transmission capacity in telecommunication through mode-division multiplexing. Furthermore, the LP02 mode of the fiber can be designed to provide dispersion characteristics dramatically different from conventional step-index single-mode fibers (SMFs), such as anomalous dispersion below 1300 nm. This feature has also been widely utilized in a number of applications, including dispersion compensation, high energy pulsedelivery, and nonlinear wavelength conversion. In all the existing applications of the HOM fibers, however, coupling between different guided modes is not desirable, and thus avoided. In order to minimize mode coupling induced by waveguide perturbation, HOM fibers are designed such that the modes have very different effective refractive indices (neff) at the wavelengths of operation. Consequently, optical effects involve multiple modes, especially multimode nonlinear effects, in the HOM fiber have rarely been explored. This work focuses on the nonlinear wavelength conversion effects in HOM fibers, including soliton self-frequency shift (SSFS), Cerenkov radiation, and four- wave mixing (FWM), in both single-mode and multimode schemes. We experimentally demonstrate four nonlinear effects with HOM fibers: 1) SSFS below 800 nm and efficient Cerenkov radiation in the vicinity of 850 nm; 2) high-energy soliton generation at 1080 nm; 3) Intermodal FWM in an all-fiber laser system; 4) Intermodal Cerenkov radiation. In addition, we have developed a convenient dispersion measurement technique specifically tailored for HOM fiber. The results of these experimental demonstrations may lead to potential applications in both biomedical imaging and telecommunication.