Japanese rice farmers manage many small-scale paddy fields in mountainous regions. This situation challenges farmers’ decision-making regarding crop management depending on each field’s characteristics. Fertilizer inputs balance rice productivity and grain quality while enhancing the farm’s profits. This current research attempted to characterize the effect of field-specific fertilization rates and soil characteristics on rice yield and quality via on-farm experiments using a farmer’s yield monitor in 168 fields over three years. The Bayesian approach was employed to model the relationships among fertilizer, soil characteristics, and underlying unquantified factors, such as a farmer’s management block and climatic season. The results indicate that yield variability is mostly explained by the farmer’s management block rather than by fertilizer and soil characteristics. Therefore, lower fertilization rates could contribute to improving cost efficiency. However, a better understanding of yield- and quality-limiting factors is necessary for each farmer’s management block to optimize crop management.