We introduce growth diagrams arising from the geometry of the affine Grassmannian for $GLm$. These affine growth diagrams are in bijection with the $c\lambda \to$ many components of the polygon space Poly($\lambda \to$) for $\lambda \to$ a sequence of minuscule weights and $c\lambda \to$ the Littlewood--Richardson coefficient. Unlike Fomin growth diagrams, they are infinite periodic on a staircase shape, and each vertex is labeled by a dominant weight of $GLm$. Letting $m$ go to infinity, a dominant weight can be viewed as a pair of partitions, and we recover the RSK correspondence and Fomin growth diagrams within affine growth diagrams. The main combinatorial tool used in the proofs is the $n$-hive of Knutson--Tao--Woodward. The local growth rule satisfied by the diagrams previously appeared in van Leeuwen's work on Littelmann paths, so our results can be viewed as a geometric interpretation of this combinatorial rule. Similar diagrams appeared in the work of Speyer on osculating flags.