It has been argued that the experimentally observed limitation of heat transport through boron nitride nanotubes is due to intershell scattering rather than to inefficient heat transfer to inner shells. Using an atomistic Green’s function calculation, we present evidence that on the contrary, intershell or any other type of scattering along the nanotubes is not the main limiting mechanism, and heat conduction restricted to a few layers is responsible for the low thermal conductivities experimentally measured. Our results also indicate that anharmonic scattering in boron nitride is relatively weak, which might lead to considerably larger thermal conductivity for well-contacted nanotubes than previously reported.