We observe clear evidence of adiabatic passage between photon populations via a four-wave mixing process, implemented through a dispersion sweep arranged by a core diameter taper of an optical fiber. Photonic rapid adiabatic passage through the cubic electric susceptibility thus opens precise control of frequency translation between broadband light fields to all common optical media. Areas of potential impact include optical fiber and on-chip waveguide platforms for quantum information, ultrafast spectroscopy and metrology, and extreme light-matter interaction science.