Prostate cancer is the most frequently diagnosed malignancy in men. Commonly used androgen deprivation therapies lead to the development of highly aggressive castration-resistant neuroendocrine prostate carcinomas (NEPC). NEPCs carry frequent alterations in genes such as MYCN, PTEN, TP53, and RB1. Recent studies showed that the plasticity of mouse prostate tumor cells may lead to de-novo neuroendocrine (NE) differentiation. Such observations are supported by mouse models based on the prostate epithelium-specific loss of Pten and Rb1, especially in conjunction with MYCN overexpression. At the same time, some NEPCs may develop from cells showing features of luminal and neuroendocrine cell differentiation, such as in the case of Trp53 and Rb1 inactivation. Using single-cell transcriptomic analysis, we show that NEPC cells deficient for Trp53 and Rb1 have distinct and similar cell subpopulations compared to NEPC associated with Pten and Rb1 loss and MYCN overexpression. This work provides the basis for future studies to understand how the differences in target cell populations may determine the mechanisms driving the initiation and clinical behavior of NEPCs. Determining the cellular heterogeneity of NEPCs should also allow the stratification of human NEPC into clinically distinct groups and contribute to the development of more effective NEPC treatment strategies.