State machines have been given a poor reputation in high reliability design. The reason for this is a concern over state machines entering unknown or undefined states. In an attempt to code complex processes without using directly implemented state machines, many designers have resorted to high-level confusing processes that essentially create a state machine without the structure. This type of design reduces the clarity of the function. This attempt at the evasion of state machine problems results in code which is far more complex and far more difficult to understand than a state machine implementation would be. The proper course of action for complex multi-state design is to design a state machine with the intent of producing readable code.
State machine design is highly subjective as the designer must try to balance the complexity with the function. Often, a large complex set of steps can be generalized into several less complex sets of steps. In these cases it makes sense to split the design into several autonomous state machines with a high level 'task master' state machine to sequence them. State machines, or any process should not be so long and complex that someone can not digest and understand the function in a single pass through the code. A function of 3 pages is a fair limit @ 100 lines per page. 2 pages would be better. I have seen some so poorly written that they span over 3,000 lines of HDL in a single process, and can be rewritten in under 300 lines with the same functionality. Processes should be singular and self contained, meaning do not do EVERYTHING in one process nor split a single task into multiple processes. As example: a serializer process with a shift register may include parity generation as this is a shift function, but should not include the data selection or word generation logic. A final word on state machines is determinate. SMs with multiple exit paths from states, and internal loop paths should be critically examined to remove the possible embedded lower level state machines within them. Most states should have only 3 exit paths: pass, fail, and recovery from illegal entry/reset.
Some basic suggestions for coding state machines (VHDL)
Use enumerated types for state definitions rather than constants
Use a single sequential process for current -> next state transitions and state outputs (registered).
Use a single combinatorial process for next state and output logic.
Use a case statement in the combinatorial process for the current state.
Limit the number of nested IF and CASE statements.
Choose IF when selection needs priority encoding.
Choose CASE when selection is only one possible (ie counter has only one possible value at a time).
Use default assignments BEFORE the CASE selection to assign idle signals rather than reassigning in each CASE's "when" block.
Don't use multiple IF blocks with same conditions.
Don't perform multiple signal reassignment in successive IF blocks.
Balance fewest number of states with simplest state transitions.
Partition big tasks into LOGICAL sub-tasks.
