The Solana Network faced a significant slowdown in block production after a planned upgrade, adversely impacting transactions and forcing validators to downgrade their software to restore performance.
The network freeze continued over the weekend as validators were preparing a second restart of the network.
The Latest Solana Outage
The Solana Network faced yet another network slowdown following an upgrade initiated in the validator software. The incident, which took place on the 25th of February, resulted in a significant disruption in transactions on the network. As a result, validators scrambled to downgrade the validator software in an attempt to restore performance on the network. The issue began at 6:00 am UTC when the network upgraded the validator software to 1.14. With the network stuttering, validators were forced to downgrade the network back to version 1.13 to restore network performance. Solana’s Compass website noted,
“The network experienced a significant slowdown in block production that coincided with an upgrade to validator software. Engineers are still conducting a root cause analysis.”
However, the downgrade failed to restore normal operations on the Solana network, forcing validators to attempt a restart of the network on 1.13.6. The network announced the plans for a restart on Twitter, stating,
“The Solana network is currently restarting after an issue during the upgrade from 1.13 to 1.14 that slowed block finalization. Once validators with 80% of stake have restarted, the network will resume.”
The Root Of The Issue
According to the information available, the problem is linked to an upgrade from version 1.13 to version 1.14, leading to a slowdown in block validation. In order to restart the network, 80% of active validators are required to resume operations.
“As more validators complete their restart, this number will rise in line with the amount of stake they have delegated: this means larger validators such as CEX have an outsized impact on restart times.”
Solana validators remained in discussions regarding the incident over the weekend, with infrastructure provider Chorus One noting that the incident demonstrated the level of decentralization on the network.
“Without all these debates, we would be back up in an hour. But, every decision along the way – whether to downgrade, whether to restart, when to switch from the downgrade approach to the restart approach – is debated. Voting happens. We end up taking 8-10 hours to recover instead of 1.”
Freeze Drags On
The network freeze on Solana spilled over into the weekend, as it continued on Saturday, with validators preparing a second restart attempt to restore services to users. By evening, validators concluded that the best way to move forward would be to synchronize a restart and fork the chain. However, the first attempt had to be abandoned as validators discovered they had picked the wrong starting point for the restart, further prolonging the outage.
A Near Complete Shutdown
The issue, which began as a simple slowdown in processing transactions, spiraled into a near-complete shutdown of any type of activity on Solana, with block production halted, leading to transactions not being processed or validated. As a result, on-chain crypto assets have been rendered immovable until the network comes back online. Key players in the Solana ecosystem were still looking to identify the issue over the weekend, with one theory suggesting that a “fat block” messed up the mechanics of the blockchain.
Not The First Network Outage
This is not the first time Solana has had a problem with network outages. In September 2021, the network was hit by intermittent instability due to what the protocol’s Twitter handle described as “resource exhaustion.” The network was in the news again in December 2021 after it emerged that technical difficulties and congestion of the network led to another slowdown in the network. Just a month later, in January 2022, a 48-hour outage wreaked havoc on the Solana network, forcing some users who had taken loans to liquidate their holdings.
Disclaimer: This article is provided for informational purposes only. It is not offered or intended to be used as legal, tax, investment, financial, or other advice.