In a traditional blockchain, every node stores the entire state (account balances, smart contract code, etc.) and performs all of the work associated with processing every transaction in the chain. This is analogous to having a single worker build an entire car.
From manufacturing to CPU design, pipelining is a common technique for dramatically scaling up productivity. Flow applies pipelining to blockchains by separating the jobs of a validator node into four different roles: Collection, Consensus, Execution, and Verification. This separation of labor between nodes is vertical (across the different validation stages for each transaction) rather than horizontal (across different transactions, as with sharding).
In other words, every validator node still participates in the validation of every transaction, but they do so only at one of the stages of validation. They can therefore specialize for — and greatly increase the efficiency of — their particular stage of focus.
This allows Flow to scale to thousands of times higher throughput and lower cost while maintaining a shared execution environment for all operations on the network. In database terms, smart contracts and user accounts on Flow can always interact with each other in one atomic, consistent, isolated, and durable (ACID) transaction. This ensures good user experience and full composability, letting developers easily build on each other’s work.
Problems with Sharding
Most proposals aim to improve the scalability of blockchains by fragmenting them into inter-connected networks: commonly shards, although sidechains have the same issues. These approaches remove serializability (“ACID”) guarantees common in database systems.
Loss of ACID guarantees makes building an app that needs to access data across fragments far more difficult and error-prone. Interactions between smart contracts become very complicated, and even individual large-scale applications would have to resort to complex mechanics to scale across shards due to latency issues and higher transaction failure rates. The combination dramatically limits the kinds of applications possible on the network as well as their network effects. Sharding effectively saddles the hardest part of scaling the blockchain onto application developers rather than solving it at the protocol level.
A simple user action (purchasing a hat for a CryptoKitty using a stablecoin like TUSD) can take twelve transactions and seven blocks on a sharded blockchain. In an unsharded, ACID-compliant environment like Flow, the same action, and many more complex than it, can be handled by one atomic transaction in a single block.
Worse than the additional time and cost is the increased attack surface and complexity: it will be much harder to design, test, and harden the smart contract code on a sharded blockchain.
Separating Consensus from Compute
The core insight that led to architecture of Flow is that we can separate non-deterministic processes from deterministic ones and assign each to different types of nodes based on their technical capabilities to dramatically increase the blockchain throughput and solve several user- and developer experience problems with existing networks at the same time. Our realization was that tasks within a blockchain can be divided into two types:
- Non-deterministic (or “subjective”) tasks, such as determining the presence and order of transactions in the blockchain;
- Deterministic (or “objective”) tasks, such as computing the result of those ordered transactions once it has been determined.
Non-deterministic tasks require a coordinated consensus process (like Proof of Work or Proof of Stake). Deterministic tasks, on the other hand, always have a single, objectively-correct outcome. The critical insight behind Flow’s architecture was that the single biggest bottleneck to blockchain performance is the deterministic task of executing transactions after they’ve already been included into a block, and not the subjective process that requires consensus, i.e. the formation of the block itself. This insight is outlined in our first technical paper: Separating Consensus and Compute.
Flow Multi-Role Architecture
Flow pipelines the work of a blockchain miner or validator across four different roles that all require staking; a separation of concerns that significantly reduces redundant effort:
- Consensus Nodes decide the presence and order of transactions on the blockchain;
- Verification Nodes are responsible for keeping the Execution Nodes in check;
- Execution Nodes perform the computation associated with each transaction;
- Collection Nodes enhance network connectivity and data availability for dapps.
Flow is designed such that even a single honest node, of any role, can punish and trigger recovery from invalid data introduced by dishonest Collection or Execution Nodes.
Consensus and Verification Nodes together are the foundation of security in the Flow network and leverage cryptoeconomic incentives to hold the rest of the network accountable. These validators can optimize for security and decentralization: the roles of Consensus and Verification are streamlined to allow high levels of participation, even by individuals with consumer-grade hardware running on home internet connections. Consensus nodes run a variant of HotStuff, one of the most proven proof of stake algorithms.
Execution and Collection Nodes, on the other hand, do work that is fully deterministic – making them less vulnerable to attack. The work of these nodes is also verified and held accountable by the other node types. These node roles can therefore safely optimize for security and scalability, allowing the network to scale. Operating these nodes requires dedicated server hardware in a professionally managed data center.
Specialized Proofs of Confidential Knowledge (SPoCKs)
Specialized Proofs of Confidential Knowledge (SPoCKs) are a new cryptographic technique developed by the Flow team and formally defined in our Technical Papers. SPoCKs allow any number of provers to demonstrate to a third-party observer that they each have access to the same confidential knowledge. These proofs are non-interactive and don’t reveal the confidential knowledge. Each prover’s SPoCK is specialized to them, and can’t be copied or forged by any other prover.