Evaluating Injection Laryngoplasty Skills Using a Foundation Model: A Feasibility Study
Access to this document is restricted. Some items have been embargoed at the request of the author, but will be made publicly available after the "No Access Until" date.
During the embargo period, you may request access to the item by clicking the link to the restricted file(s) and completing the request form. If we have contact information for a Cornell author, we will contact the author and request permission to provide access. If we do not have contact information for a Cornell author, or the author denies or does not respond to our inquiry, we will not be able to provide access. For more information, review our policies for restricted content.
OBJECTIVES: To evaluate the construct validity of a commercially available multimodal foundation model (Google Gemini 2.5 Pro) in assessing simulated injection laryngoplasty. METHODS: Thirty video recordings of simulated injection laryngoplasty procedures were stratified by operator experience (10 novice, 10 intermediate, and 10 expert participants). Videos were evaluated by Gemini 2.5 Pro using two prompt engineering strategies: zero-shot (rubric-based, no examples) and few-shot (rubric plus examples). Performance was compared against operator training level (ground truth). Model reliability and stability were assessed through 90 repeated inference trials. RESULTS: Under a zero-shot strategy, the model failed to discriminate between skill levels (Spearman's ρ = 0.12, p = 0.52). Conversely, few-shot prompting demonstrated a strong, positive correlation with operator experience (Spearman's ρ = 0.66, p = 0.0002) and successfully stratified skill levels (Kruskal-Wallis H = 12.4, p = 0.002). Pairwise analysis confirmed the few-shot model significantly differentiated experts from both novices (p = 0.002) and intermediates (p = 0.026). Additionally, few-shot prompting significantly improved precision, reducing mean absolute error by nearly half (0.74-0.41, p = 0.04). Reliability analysis revealed 100% ordinal consistency (75.6% exact match stability), indicating the model varied under identical conditions, but did not commit any gross classification errors. CONCLUSION: General-purpose multimodal models lack the intrinsic surgical judgment necessary to assess procedural skill. However, resource-efficient few-shot prompting successfully calibrates the model to distinguish expert from trainee performance. While promising as a scalable assessment tool, current models exhibit inherent variability that requires mitigation, such as averaging repeat model evaluations. LEVEL OF EVIDENCE: N/A.