From CoinShares Research Blog by Max Shannon

The Berkley Package Filter Issue

Recently Solana experienced its first outage in over 11 months. An infinite loop was propagated across the network’s validators due to a faulty implementation inside of the core software via the Berkley Packet Filter which is the mechanism to deploy, upgrade and execute programs on Solana. Block production halted, impeding the chain’s progress (in layman’s terms, it went offline). Validators then manually verified, inside of a Google Sheet, a snapshot of the last valid block. For the chain to restart, at least 80% of validators began running the patched software for blocks to be added to the chain.

The Aftermath

The impact of this incident could have been mitigated with a more diversified client base beyond Solana Labs. Other clients such as Firedancer, Agave, and Sig clients are all currently in development to solve this issue. Despite the setback, SOL prices have rebounded over 20% from post-outage lows, as of 19 February 2024, demonstrating that Solana is still the crypto darling given the very subdued negative price action immediately after the outage. Albeit price action had a tail wind combined with the recent bullish bitcoin sentiment pushing the broader crypto market higher.

JITO, an MEV client which aims to extract the most value from ordering transactions inside of blocks, has Bundles which allow sequential and atomic transaction execution — otherwise known as maximal extractable value (MEV). These have been increasing steadily in numbers, around 79% since the outage, having dropped around 17% on the day because of the downtime.

Looking Forward

A subsequent post-mortem analysis revealed that the protocol’s downtime had less to do with the scalability and performance, which had previously plagued the protocol in previous outages. Nevertheless, it is evident from the JUP airdrop that there remains room for improvement given the level of failed transactions negatively affecting user experience. While airdrops may seem like a vacuous use case, the event tested the protocol, gauging its ability to handle high throughput while maintaining chain stability. Implementing multidimensional fees and scheduler upgrades has been discussed to better manage high periods of demand, despite the vast improvements made already.

Currently, block producers (validators) schedule and execute transactions inside one piece of hardware. This has resulted in mixing of transactions in the pipeline because there is no mempool in Solana, as there is in Ethereum or Bitcoin, for example. Ideally, upgrading the scheduler should separate it from the multi threaded processing for execution as scheduling is only single threaded. This development should improve user experience by reducing MEV and polish up block inclusion.

Further, Solana’s fee mechanism currently implements a base fee, which everyone pays, and a priority fee, which is variable depending on the immediate need for block inclusion. Local fee markets modulate demand in the sense that one hotspot of high demand, such as a DEX with lots of activity, will not affect the gas price for users in a less active area of the protocol. Multidimensional fees go a step further by not only increasing network throughput during periods of high demand but also improving the user experience by decoupling unrelated transactions and providing a predictable cost of transaction inclusion.

Summary

While the Berkley Packet Filter incident caused ~5 hours of disruptions, Solana demonstrated investor confidence while price action rebounded. Efforts to diversify the client base and address issues such as MEV and transaction management are crucial for enhancing the protocol’s robustness and user experience.