In emerging systems, CPUs and memory are integrated into active disks, controllers, and network interconnects. Query processing on these new multiprocessor systems must consider the heterogeneity of resources among the components. This leads to the more general problem of how to deal with performance heterogeneity in parallel database systems. We study database query processing techniques that increase the leverage of heterogeneous resources. We show that the traditional algorithms used in shared-nothing parallel databases fail to utilize non-uniform resources. Uniform resource usage across non-uniform components leads to resource bottlenecks. We describe a class of new execution techniques that balance the usage of system resources using non-uniform intra-operator parallelism. We show that these techniques improve performance on heterogeneous architectures by allowing trade-offs between the various resources. Traditional techniques are subsumed as a special case for symmetric architectures. We show a formal model that maps out the new execution space of alternative processing techniques. A simplified cost model allows analytic performance evaluation of the alternative techniques. The proposed new execution paradigm is an extension of the classical dataflow paradigm.