As development and humanitarian agencies increasingly advance the objective of 'building resilience', three resilience measurement methods have come into especially widespread use: the Resilience Indicators for Measurement and Analysis approach developed by FAO, the multi-dimensional index approach developed by TANGO International, and the probabilistic approach of Cissé and Barrett. We compare performance across those three methods using nationally representative panel data from Ethiopia and Niger. We find that the three measures exhibit significantly different distributions and orderings among households, and they vary significantly in the households they identify as resilient or least resilient. All three measures exhibit only modest out-of-sample predictive accuracy, generating many false negatives and false positives relative to the food security outcome measure whose resilience they are meant to reflect. It remains unclear what these measures capture and what value they add beyond more established wellbeing measures such as the food consumption score or real expenditures. Significant room exists for improvement in resilience measurement to better guide and evaluate development resilience interventions.