With the slowdown of Moore’s Law and the end of Dennard Scaling, computer architects have embraced specialization as the main way forward for continuing performance and efficiency growth previously made through traditional technology scaling. Specialization comes in several forms including application specific specialization, domain specific specialization, and parallel pattern specific specialization. This emergence of hardware specialization has pushed the chip industry towards integrating a sea of heterogeneous specialized hardware units, each with its own specialized program abstraction, into a single system on chip (SoC). However, given certain area, power, and budget constraints, there is limited room for the number of specialized hardware units possibly integrated into an SoC. One viable solution is to unify multiple kinds of specialization under the same program abstraction and in the same hardware (e.g., GPGPUs). This unifying approach essentially lowers area costs by trading off the optimality of program abstractions and hardware implementations for individual program patterns. In this thesis, I explore another specialization approach called evolutionary specialization that supports multiple types of specialization in the same hardware. The evolutionary specialization refers to starting from an optimal abstraction and micro-architecture for one program pattern and gradually adding a minimal set of hardware changes to the existing micro-architecture to support additional program patterns without changing their optimal abstractions. The thesis makes a case for the evolutionary specialization through two novel architectures: big.VLITTLE and SparseZipper. The big.VLITTLE architecture evolves a multi-little-core system to efficiently support both single-program multiple-data (SPMD) and single-instruction multiple-data (SIMD) program pat- terns. The SparseZipper architecture minimally extends a modern matrix architecture specialized for a dense general matrix multiplication (GEMM) pattern to support a sparse GEMM pattern.