The state machine approach is a general method for achieving fault tolerance and implementing decentralized control in distributed systems. This paper reviews the approach and identifies abstractions needed for coordinating ensembles of state machines. Implementations of these abstractions for two different failure models -Byzantine and fail-stop-are discussed. The state machine approach is illustrated by programming several examples. Optimization and system reconfiguration techniques are explained.