We introduce growth diagrams arising from the geometry of the affine Grassmannian for $GL_m$. These affine growth diagrams are in bijection with the $c_{\vec\lambda}$ many components of the polygon space Poly($\vec\lambda$) for $\vec\lambda$ a sequence of minuscule weights and $c_{\vec\lambda}$ the Littlewood--Richardson coefficient. Unlike Fomin growth diagrams, they are infinite periodic on a staircase shape, and each vertex is labeled by a dominant weight of $GL_m$. Letting $m$ go to infinity, a dominant weight can be viewed as a pair of partitions, and we recover the RSK correspondence and Fomin growth diagrams within affine growth diagrams. The main combinatorial tool used in the proofs is the $n$-hive of Knutson--Tao--Woodward. The local growth rule satisfied by the diagrams previously appeared in van Leeuwen's work on Littelmann paths, so our results can be viewed as a geometric interpretation of this combinatorial rule. Similar diagrams appeared in the work of Speyer on osculating flags.