Mainnet halted at the last block of the epoch due to an Unjail message. If an Unjail message is sent at the end of the epoch, two modules attempt validator set updates in their EndBlocker.
The x/staking
and x/epoching
modules were producing validator set updates in their respective EndBlocker
.
Detailed description of the bug:
x/staking
module’s EndBlocker execution (ref). The x/staking functionality is instead used via the x/epoching
module.x/staking
module in the EndBlocker list (ref).x/capability
module was removed from the EndBlocker execution (instead of the x/staking
module).x/epoching
and x/staking module EndBlock functions were active and able to modify the validator set. This behaviour is explicitly prevented by Cosmos-SDK and leads to a panic (ref), which halts the chain.x/epoching
module modified the validator set. Unfortunately, a MsgUnjail was also processed at the epoch boundary, causing x/staking to also modify the validator set. This led to a panic and chain halt.