The benefits of autonomous vehicles (AVs) not only depend on the maturity of technologies, but also on how AVs behave and interact with their peers and human-driven vehicles (HVs). Similar to many other systems, individual and collective dynamics of traffic flow are not always aligned with each other (for instance, aggressive driving may benefit an individual driver but disrupts the overall traffic). It is therefore imperative to consider behavior design for AVs such that the benefits of AVs can be realized at both individual and collective levels, especially when centralized control is absent and all involved agents are self-interested. This research explored behavior designs for AVs in mixed autonomy traffic using a game-theoretic approach. In this new framework, we defined agent utilities and casted interactions of heterogeneous agents (i.e., AVs and HVs) as a collective bargaining game. We analytically characterized the equilibria of mixed autonomy traffic, including equilibria types and conditions when each type of equilibria can be reached. We found that in general, mixed autonomy traffic may reach two types of Nash equilibria, namely one-pipe and two-pipe equilibrium. Depending on traffic regimes, mixed autonomy traffic can always reach one of the equilibria that is Pareto-efficient, and this is known as the collective rationality of self-interested agents. Based on the theoretical characterization, we proposed a class of lane use policies that determine the capacity allocation between AVs and HVs. We then developed a computing algorithm to construct the equilibria numerically and conducted simulation experiments to investigate different AV behavior scenarios when they interact with HVs. We showed that with the proposed lane use policy, mixed autonomy traffic can always reach collective rationality and attain uniform speed and flow improvements over non-Pareto-efficient equilbria. We also showed a capacity drop phenomenon for HVs in mixed autonomy traffic when exclusive AV lanes are introduced, which pinpointed a potential subtlety of similar lane policies.