Understanding the interaction between people from images is yet a challenging task in recent years. Several approaches attempted to address this challenge by different proposed models and compact descriptors encoding the consistency between people's spatial-temporal body features or their overall activities. Among all human body features, the head pose provides a distinct description of an individual's attention and is considered a key feature to interpret social interactions. In this thesis, I present a novel approach to infer interaction among a group of people in a group conversation scenario using the Markov Random Field model. A novel interaction feature is proposed to represent the transactional segment using the head pose. Furthermore, I extend the approach to infer the hierarchical structure of a group with the contextual information, which improves the leading method in the analysis of interactions among people and detect the emergent leader of the group. The qualitative result shows the effectiveness of interaction inference using a state-of-art dataset.