Priority effects arise when the history of species arrival influences local species interactions, thereby affecting the composition of ecological communities. The outcome of some priority effects may be more difficult to predict than others, but this possibility remains to be fully investigated. Here, we provide a graph-based, non-parametric, theoretical framework to understand the classification of priority effects and the predictability of multi-species communities. We show that we can classify priority effects by decomposing them into four basic dynamical sources—the number of alternative stable states, the number of alternative transient paths, the length of composition cycles, and the interaction between alternative stable states and composition cycles. Although the number of alternative stable states has received most of the attention, we show that the other three sources can contribute more to the predictability of community assembly, especially in small communities. We discuss how this theoretical framework can guide new experimental studies.