Reconsidering Ranking-and-Selection Guarantees

Other Titles


This dissertation deals with the various statistical guarantees delivered by ranking-and-selection (R&S) procedures: a class of methods designed for the problem of selecting the best from among a finite number of simulated systems. Examples of such guarantees include ensuring that an optimal or near-optimal system is selected with high probability or that the expected performance gap between the selected system and the optimal system is below a specified threshold. We explore three fundamental issues concerning R&S guarantees that are of practical and theoretical interest to the simulation community. First, we discuss the shortcomings of the popular indifference-zone-inspired guarantee on the probability of correct selection (PCS) and argue that delivering a guarantee on the probability of good selection (PGS) is a more justifiable goal. We present an overview of the PGS guarantee and examine numerous techniques for proving the PGS guarantee, including sufficient conditions under which R&S procedures that deliver the IZ-inspired PCS guarantee also deliver the PGS guarantee. Second, we study Bayesian R&S guarantees, contrasting them with their frequentist counterparts and investigating the practical implications of this distinction. R&S procedures deliver Bayesian guarantees by terminating when a posterior quantity of interest - e.g., the posterior PCS or PGS - crosses some threshold. We develop several methods for improving the computational efficiency of checking this stopping rule when there are a large number of systems and demonstrate their effectiveness compared to existing approaches. Third, we study R&S guarantees for the setting in which a R&S procedure is run after a simulation-optimization search. We show that for searches that use the observed performance of explored systems to identify new systems, the simulation replications are conditionally dependent given the sequence of returned systems. We demonstrate that reusing replications taken during a search as input to a R&S procedure can result in an empirical PCS or PGS below the guaranteed threshold. Based on these negative findings, we call into question the guarantees of established R&S procedures that reuse search data.

Journal / Series

Volume & Issue



Date Issued




Statistics; Ranking and Selection; Industrial engineering; simulation optimization; Operations research


Effective Date

Expiration Date




Union Local


Number of Workers

Committee Chair

Henderson, Shane G.

Committee Co-Chair

Committee Member

Frazier, Peter
Bitar, Eilyan Yamen

Degree Discipline

Operations Research and Information Engineering

Degree Name

Ph.D., Operations Research and Information Engineering

Degree Level

Doctor of Philosophy

Related Version

Related DOI

Related To

Related Part

Based on Related Item

Has Other Format(s)

Part of Related Item

Related To

Related Publication(s)

Link(s) to Related Publication(s)


Link(s) to Reference(s)

Previously Published As

Government Document




Other Identifiers


Attribution 4.0 International


dissertation or thesis

Accessibility Feature

Accessibility Hazard

Accessibility Summary

Link(s) to Catalog Record