Mainnet halted at the last block of the epoch due to an Unjail message. If an Unjail message is sent at the end of the epoch, two modules attempt validator set updates in their EndBlocker.
The x/staking and x/epoching modules were producing validator set updates in their respective EndBlocker .
Detailed description of the bug:
x/staking module’s EndBlocker execution (ref). The x/staking functionality is instead used via the x/epoching module.x/staking module in the EndBlocker list (ref).x/capability module was removed from the EndBlocker execution (instead of the x/staking module).x/epoching and x/staking module EndBlock functions were active and able to modify the validator set. This behaviour is explicitly prevented by Cosmos-SDK and leads to a panic (ref), which halts the chain.x/epoching module modified the validator set. Unfortunately, a MsgUnjail was also processed at the epoch boundary, causing x/staking to also modify the validator set. This led to a panic and chain halt.