Large genomic datasets abound with unreported relatives, making them a prime candidate for identifying latent pedigree structures. However, the most commonly applied methods for identifying relatives infer only degrees of relatedness between pairs, leaving the specifics of their family structure unaddressed. Where methods do attempt to resolve specific pedigree relationships, they typically do not attempt to discern the sexes of the untyped relatives they infer to have connected a pair. We developed CREST, a method that can accurately infer the sex of the intermediate parent for half-sibling and grandparent-grandchild relationships using only the identity-by-descent (IBD) segments the pair shares together with sex-specific genetic maps. When tested on real relatives in the Generation Scotland dataset, CREST labeled the parent sexes of more than 99% of pairs correctly. Importantly, CREST makes no use of sex chromosome data. The X chromosome would be a supremely useful source of information for this classification, but even if high quality X chromosome genotypes were available, it is not clear a priori that IBD detection methods designed for the autosomes would function as expected on the X. To examine this question, we investigated the quality of IBD calling on the X chromosome using seven different detection methods, and compared the results to calls made on chromosome 10. These comparisons revealed that there are subtle but substantial differences in IBD calling between the two chromosomes. These differences remain even when downsampling the chromosome 10 marker density to match that of the X, suggesting a more complex explanation. We contend that IBD calling on the X presents unique challenges and requires additional considerations to ensure its results are properly interpreted in connection with those from autosomal data.