Obtaining effective polymer representations for quantitative structure propertyprediction is a challenging task, but deep learning techniques has progressed remarkably in learning such contextual representations from unlabeled polymeric data. In the field of cheminformatics, significant advances were made in Deep Learning Architecture Transformers, but development in effective polymer representations through Transformers in the field of Polymer Informatics is limited. In my thesis, I highlight the performance of Transformer learned representations in relation to prior techniques i.e. both handcrafted and deep learning-based approaches. Next, I detail how increasing the size of the Transformer affects the downstream task performance on the Glass Transition Temperature dataset. Finally, I identify several challenges and discuss promising future research directions.