Cornell University
Library
Cornell UniversityLibrary

eCommons

Help
Log In(current)
  1. Home
  2. Cornell University Graduate School
  3. Cornell Theses and Dissertations
  4. Reconsidering Ranking-and-Selection Guarantees

Reconsidering Ranking-and-Selection Guarantees

File(s)
Eckman_cornellgrad_0058F_11459.pdf (809.8 KB)
Permanent Link(s)
https://doi.org/10.7298/0js0-9q94
https://hdl.handle.net/1813/67385
Collections
Cornell Theses and Dissertations
Author
Eckman, David
Abstract

This dissertation deals with the various statistical guarantees delivered by ranking-and-selection (R&S) procedures: a class of methods designed for the problem of selecting the best from among a finite number of simulated systems. Examples of such guarantees include ensuring that an optimal or near-optimal system is selected with high probability or that the expected performance gap between the selected system and the optimal system is below a specified threshold. We explore three fundamental issues concerning R&S guarantees that are of practical and theoretical interest to the simulation community. First, we discuss the shortcomings of the popular indifference-zone-inspired guarantee on the probability of correct selection (PCS) and argue that delivering a guarantee on the probability of good selection (PGS) is a more justifiable goal. We present an overview of the PGS guarantee and examine numerous techniques for proving the PGS guarantee, including sufficient conditions under which R&S procedures that deliver the IZ-inspired PCS guarantee also deliver the PGS guarantee. Second, we study Bayesian R&S guarantees, contrasting them with their frequentist counterparts and investigating the practical implications of this distinction. R&S procedures deliver Bayesian guarantees by terminating when a posterior quantity of interest - e.g., the posterior PCS or PGS - crosses some threshold. We develop several methods for improving the computational efficiency of checking this stopping rule when there are a large number of systems and demonstrate their effectiveness compared to existing approaches. Third, we study R&S guarantees for the setting in which a R&S procedure is run after a simulation-optimization search. We show that for searches that use the observed performance of explored systems to identify new systems, the simulation replications are conditionally dependent given the sequence of returned systems. We demonstrate that reusing replications taken during a search as input to a R&S procedure can result in an empirical PCS or PGS below the guaranteed threshold. Based on these negative findings, we call into question the guarantees of established R&S procedures that reuse search data.

Date Issued
2019-05-30
Keywords
Statistics
•
Ranking and Selection
•
Industrial engineering
•
simulation optimization
•
Operations research
Committee Chair
Henderson, Shane G.
Committee Member
Frazier, Peter
Bitar, Eilyan Yamen
Degree Discipline
Operations Research and Information Engineering
Degree Name
Ph.D., Operations Research and Information Engineering
Degree Level
Doctor of Philosophy
Rights
Attribution 4.0 International
Rights URI
https://creativecommons.org/licenses/by/4.0/
Type
dissertation or thesis

Site Statistics | Help

About eCommons | Policies | Terms of use | Contact Us

copyright © 2002-2026 Cornell University Library | Privacy | Web Accessibility Assistance