Workflow
Domain Instantiation & Upgrades
When a new domain is instantiated, the consensus runtime generates a genesis ER hash using the genesis_config
and domain_config
for that domain.
The consensus chain will initiate a “balance” for this domain to track how many SSC were transferred in and out of this domain to make sure a domain cannot create SSC it did not already have via fees, XDM transfers or storage fee deposits. This balance does not include staked SSC as staking is tracked on the consensus chain directly.
genesis_receipt_hash
is derived using a host function call passing the required genesis domain_config
details mentioned below. The host function will construct the DomainConfig
using the domain runtime and create the genesis ER by deriving the genesis_state_root
, presented here, and returning the genesis_receipt_hash
. This ER will not have any intermediate state roots, but only the final state root, as Genesis ER is considered special.
Operators who want to run this domain will create a genesis block using the GenesisConfig
placed on the consensus chain during domain instantiation and construct the genesis ER used when submitting the next block’s ER thereby building a block tree.
Domain Instance Genesis Block Generation
When the domain is first instantiated in the consensus runtime, a genesis_state_root
is derived using a host function with domain_config
as input, the genesis_state_root
will be further used to drive a genesis ER which will not have any intermediate state roots, only the final state root, as Genesis ER is considered special.
The genesis_state_root
is required to be unique among all domain instances and consistent with the genesis_state_root
generated by the domain instance node on the client side. The host function and the domain instance node derive genesis_state_root
by generating a genesis block with RuntimeGenesisConfig
as input, thus, to achieve uniqueness and consistency, the RuntimeGenesisConfig
needs to be unique and consistent.
For all domain runtime types, RuntimeGenesisConfig
should include the pallet genesis config of:
- system pallet
SystemConfig { code }
for it to be a validRuntimeGenesisConfig
- domain-id pallet
DomainIdConfig { domain_id }
for it to be unique as it includes a uniquedomain_id
Domain Instance Node Bootstrap
After the operator instantiated a domain instance at the consensus chain by submitting an instantiate_domain
extrinsic and waiting until the extrinsic is finalized (past the CONFIRMATION_DEPTH_K
consensus blocks), the operator can use the resulting domain instance domain_id
and created_at
(block height) arguments to the subspace-node
binary to run a domain instance node for the instantiated domain.
The domain instance node has two modes: bootstrap mode and sync mode. In the bootstrap mode, the node must bootstrap the domain instance chain by itself based on the domain registry state at the consensus chain. The node can sync the chain from other domain instance nodes in sync mode.
In bootstrap mode, there is an embedded consensus node and a bootstrapper. After the bootstrap is finished, a domain node will replace the bootstrapper.
The bootstrapper listens to the consensus node block import event, skips the block before created_at
, after the block at created_at
is imported, which contains the instantiate_domain
extrinsic of the domain instance, using runtime API with domain_id
to get the domain instance’s domain_config
and the runtime_obj
state.
Uses the domain_config
and runtime_obj.domain_runtime_code
to generate a RuntimeGenesisConfig
in the same way as the host function, and uses that to construct sc_service::Configuration
(which includes the chain_spec
) then use runtime_obj.runtime_type
to determine and build the desired domain service with sc_service::Configuration
as input. After this step the domain service is up and the bootstrap is finished.
Domain Genesis Config
The domain_config
contains:
domain_name
: user-defined name for this domain.runtime_id
: domain runtime type that exists inRuntimeRegistry
.domain_id
: identifier assigned to an instance of the domain.- specific configuration items, such as:
bundle_slot_probability
: the number of successful bundle expected to be produced in a slot (active slots coefficient); defines the expected number of bundles in the consensus block. Must be> 0
and<= 1
, recommended value1
.max_bundle_size
: the max bundle size for this domain; may not exceed the system-wideMaxDomainBlockSize
limit; The average domain block size is then expected to be belowbundle_slot_probability * max_bundle_size / SLOT_PROBABILITY
on average.max_bundle_weight
: the max bundle weight for this domain; may not exceed the system-wideMaxDomainBlockWeight
limit; The average domain block weight is then expected to be belowbundle_slot_probability * max_bundle_weight / SLOT_PROBABILITY
on average.
allowlist
: list of addresses allowed to run operators on this domaininitial_balances
: list of initial balances on domain accounts- Any further genesis config details can be included as required and be passed down. These specific genesis details ensure the
genesis_state_root
is unique for each instantiated domain and thereby makinggenesis_er_hash
unique across different instances of the same domain runtime.
Domain Runtime Upgrades
When a domain runtime is updated using upgrade_domain_runtime
, the new runtime will come into effect at a future consensus chain block, specifically, the block at which the extrinsic upgrade_domain_runtime
was executed successfully and DomainRuntimeUpgradeDelay
blocks have passed since. When that future block height arrives, the consensus chain considers the new runtime to be the latest runtime and adds a digest log to indicate the upgrade to all domain operators.
Since every operator runs the consensus chain, they will include the new runtime as part of the next domain block taken from the consensus chain, since they see the digest log in the consensus block header.
There are some scenarios where new runtime may introduce the new host APIs that newer clients will use during any stage of bundle production and block import. If the operators still use the older clients, they won’t be able to proceed and the clients are supposed to panic due to the usage of missing host APIs in the new runtime. Hence, every operator would be forced to update the client in this case. The upgrade process involves running the latest client in place of the older client. While it's not strictly necessary, it would be beneficial to automatically signal the outdated client to operators later.
Bundle Producer Election
For each time slot, each operator denoted with operator_id
participates in the slot leader election for the domain domain_id
they are staking on to determine whether they are eligible to produce a bundle in this slot, as follows:
- Initialization
- Get the
global_challenge
for this slot from the Proof-of-Time chain. - Retrieve
secret_key
from keystore.
- Get the
- VRF
- Make
transcript
for the VRF from theglobal_challenge
and VRF label for thisdomain_id
. - Generate a VRF signature by applying the VRF to the
global_challenge
and the operator's private key asvrf_signature = vrf_sign(secret_key, transcript)
. The VRF signature contains avrf_signature.proof
, which can be used by others to verify that the VRFvrf_signature.output
was correctly generated without knowing the operator’s private key.
- Make
- Threshold Check
- Compute the
threshold
based on the operatorsoperator_stake = current_total_stake
inOperators
registry for this domain proportionally to thetotal_domain_stake = current_total_stake
of all operators of this domain instake_summary
of theDomainRegistry
asthreshold = MAX * (operator_stake / total_domain_stake) * bundle_slot_probability
-
Example
If
threshold
is stored inu128
, thenMAX
is . If the operator has of total stake in this domain, according to the formula above they should check whether their VRF output numeric values is below .
-
- Check whether the VRF
vrf_signature.output
for a slot is strictly below () thethreshold
as integers. - If it is, the operator is a slot leader for that slot and can produce a bundle. They should generate a
ProofOfElection
. - If it isn’t, they skip this slot
- Compute the
Domain Bundle Production
If, for this time slot, this operator was successfully elected a slot leader, they can produce a bundle (as defined) as follows:
-
Take the
ProofOfElection
of the slot leader. -
Fetch the
ExecutionReceipt
for the last block executed locally from the domain client and attach it to the bundle header. ThisExecutionReceipt
must be based on the longest branch of the consensus chain, although it may not point to the tip of the chain, as this depends on when the last bundle for this domain was included in a consensus chain block. If there was a fraud detected at the same height, the locally producedExecutionReceipt
(if valid) will be replacing the fraudulent one in the BlockTree. -
Attach the full
execution_trace
to the givenExecutionReceipt
-
Grab all extrinsics within the specified range
tx_range
and attach to the body.If there is no extrinsic, the operator will skip producing a bundle and the following steps (TODO when challenge period is redefined in consensus blocks.)
(Currently) If there is no extrinsic, the operator will further look into the block tree
- If all domain blocks in the challenge period are empty blocks then the operator will skip producing an empty bundle thus skipping the following steps
- If there's a non-empty domain block in the challenge period, the operator will continue the following step to produce an empty bundle to derive the non-empty domain block out of the challenge period (accelerate confirmation time)
-
Compute the
bundle_extrinsics_root
and attach to the header. -
Compute the
bundle_size
andestimated_bundle_weight
. -
Note the storage fees to be paid to the consensus block author as per Bundle Storage Fees
-
Sign the bundle header.
-
Build the bundle header as described.
-
Broadcast the full bundle over the consensus network.
Transaction Selection for Bundle Production
When an operator is elected to produce a bundle, they must select transactions to be included in that bundle according to their transaction pool range (TX_RANGE
), as follows:
- Compute
slot_vrf_hash
for this slot ashash(vrf_signature.output)
- Identify txs for inclusion into the bundle for this slot by looking into the transaction pool and identify all transactions whose senders
account_id
(as integer), is within the range asbidirectional_distance(slot_vrf_hash, public_key_hash) <= TX_RANGE/2
The operator may only include as many transactions within this range as fit within the bundle max_bundle_weight
and max_bundle_size
limits for this domain.
Initial Domain Bundle Verification by Consensus Nodes
All consensus nodes will perform the following verification when a new bundle is received over the network. All valid bundles are added to the local extrinsics pool and propagated to all peers on the network. Any invalid bundles are not added to the pool (no fraud proofs for invalid bundles received, only fraud proofs for invalid bundles that are included in a block).
- Ensure
HeadDomainNumber - HeadReceiptNumber = 1
otherwisesubmit_receipt
is expected instead ofsubmit_bundle
- Verify the
domain_id
is in theDomainRegistry
. - Verify the
ProofOfElection
for this domain and operator.- Ensure the
slot_number
is no older than the slot of the blockcurrent_block_number - BundleLongevity
. - Verify the
slot_number
and theproof_of_time
is correctly computed. - Verify the
vrf_signature
based on the operator signing key and the global challenge that is derived from theslot_number
and theproof_of_time
. - Verify the
vrf_signature
is below the threshold that is derived from theoperator_stake / total_domain_stake
and thebundle_slot_probability
.
- Ensure the
- Verify the bundle header
signature
for the registered domain operator. - Ensure the bundle does not exceed the bundle
max_bundle_weight
andmax_bundle_size
limits for this domain. - Ensure the bundle is well-formed:
- Verify the
execution_trace_root
is correctly computed for theexecution_trace
. - Verify the
bundle_extrinsics_root
is correctly computed for all included extrinsics. - Verify the
bundle_size
andestimated_bundle_weight
were correctly computed for the bundle body.
- Verify the
- Ensure the
ExecutionReceipt
builds on the head of currentBlockTree
for this domain.- Verify
domain_block_number
is equal to:HeadReceiptNumber + 1
if this is the first bundle of the domain in the block- or
HeadReceiptNumber
sinceHeadReceiptNumber
is increased by the previous bundle in the block
- Verify the
consensus_block_hash
exists at the specifiedconsenus_block_height
on the consensus client. - Based on
parent_domain_receipt_hash
, verify theparent_domain_block
exists at the specifiedparent_domain_height
within theBlockTree
on the operator client. If the ER is beyond theBlockTreePruningDepth
it is too old and will simply be ignored. - Verify all
block_extrinsics_roots
exist within theexecution_inbox
of theparent_domain_block
.
- Verify
Bundle Equivocation
A dishonest operator may produce multiple bundles on the same slot with the same proof-of-election. Similar to how consensus block equivocation is addressed, consensus chain nodes perform a check to determine if a bundle has been equivocated when verifying its validity. If an equivocation is detected, then this bundle is invalid, and is not included in the block.
Consensus Block Verification
On receipt of a new consensus block, each consensus node now needs to check to ensure all included bundle headers were pre-validated locally. If they see a new bundle, they will request and run validation. If that bundle is invalid or any previously invalidated bundles are included in the farmer block, it is simply discarded and ignored.
Bundle Header Application
On execution of a new consensus block, all included bundles will be applied to the state of pallet_domains
as each included bundle header will call submit_bundle
.
For each new bundle, each consensus node will:
- Extract the
ExecutionReceipt
- Retrieve the
parent_domain_block
from theBlockTree
and conditionally update the tree. If the parent does not exist within the tree, thisExecutionReceipt
has just expired (rare event) and is simply ignored. - If this is a new ER, we will extend the
BlockTree
. If no fraud has occurred, it will extend the tip of the longest branch.- Add a new layer to the tree, inserting the ER as the first entry as a new
DomainBlock
. - Add the
bundle_extrinsics_root
to theexecution_inbox
- Add the
operator_id
to theoperator_ids
who submitted this ER - Apply XDM coin transfers to the domain’s balance
- Apply all operator fees from the ER for the domain block for which the challenge period has now passed:
- The compute fees should be divided equally between all operators in the
operator_ids
field for the parentDomainBlock
in theBlockTree
. - The compute fees are automatically staked and subtracted from domain’s balance.
- The storage fees should be refunded to the bundle authors of bundles included in the confirmed block. These should be applied individually to their
current_epoch_reward
in theOperatorPool
- The compute fees should be divided equally between all operators in the
- Add a new layer to the tree, inserting the ER as the first entry as a new
- Otherwise, reject the receipt that tries to create new branch in the block tree. If fraud has occurred, a new branch will not be created. Instead, the system requires the submission of a fraud proof to prune the fraudulent ER at the specific height before any new ER can be submitted at that height. Fraud verification is not handled here as the consensus node cannot determine which (or all)
ExecutionReceipt
is actually fraudulent at this level. It is implied that an honest operator will eventually submit a fraud proof to address the issue before submitting new ER. If the fraud proof for the receipt already present in the block tree has already been seen, then it's operators are marked as pending slashing and the new receipt will create a new head as described in step 3. - If this ER has already been seen, we will be confirming an existing entry within the block tree. Retrieve the existing
DomainBlock
from theBlockTree
-
If this is the tip of
BlockTree
- Add the
bundle_extrinsics_root
to theexecution_inbox
- Add the
operator_id
to theoperator_ids
- Add the
-
If this is not the tip of the block tree and we have a stale ER, it is directly rejected and not included in the BlockTree at all.
-
- If any domain block reached
BlockTreePruningDepth
, then we confirm it:- refund the bundle storage fees;
- distribute the operator rewards;
- mark as pending slash the operators whose bundles this receipt marked as invalid;
- if
StakeEpochDuration
has passed, do epoch transition.
- Slash any operators (and their nominators) who are pending slash, but not more than
MAX_NOMINATORS_TO_SLASH
at a time. - Accept this bundle as successful and awaiting execution on the domain.
Domain Epoch Transition
A domain staking epoch is an interval of blocks during which staking distribution remains the same. It is important to ensure a correct and provable @Bundle Producer Election that is not influenced short-term by deposits, withdrawals, and earned fees. This StakeEpochDuration period is currently set to 100 blocks, or roughly 10 minutes. The end of each epoch triggers a series of events to transition to the next epoch. These events include:
- allocation of fees earned for the blocks confirmed (older than BlockTreePruningDepth) during the epoch,
- deposits and withdrawals of stake,
- operator registrations and deregistrations,
- recalculation of stake distribution for the slot leader VRF election.
An epoch transition occurs after every StakeEpochDuration
blocks (or when forced in a force_staking_epoch_transition(domain_id)
extrinsic) after all new bundles in the last block have been executed. During the domain epoch transition, we do the following steps:
-
Re-stake operators’ nomination taxes on their rewards
- Each operator will get a cut of
nomination_tax * current_epoch_fees
of all rewards issued to their pool as pernomination_tax
specified in the operator’s config. - The operator’s cut will be automatically re-staked to the operator’s nomination as a deposit. Operator’s
shares
,current_total_shares
andcurrent_total_stake
will be updated with the corresponding deposit later when deposits are processed. - The
current_epoch_fees
is temporarily updated tocurrent_epoch_fees*(1-nomination_tax)
for the rest of the calculations during the epoch transitions. It will be reset to 0 for the new epoch.
- Each operator will get a cut of
-
If there are any operators pending slash, remove their stake from the VRF election for the next epoch.
-
Finalize domain’s staking summary.
For each operator operating on the next epoch (existing and new operators), do the following:
-
Update the stake with received fees
total_stake = current_total_stake + current_epoch_fees
-
OperatorEpochSharePrice
storage is updated with new share price (which excludes the collected nomination tax).share_price = (current_total_stake + current_epoch_fees) / total_shares
-
Compute how much to reduce the stake corresponding to all
withdrawals_in_epoch
unstaked sharestotal_stake=total_stake-withdrawals_in_epoch/share_price
-
Compute how much to increase the stake corresponding to all
deposits_in_epoch
astotal_stake=total_stake+deposits_in_epoch*share_price
-
Set
current_total_stake
andcurrent_total_shares
to newly computed values anddeposits_in_epoch
,withdrawals_in_epoch
andcurrent_epoch_fees
to 0.
-
As soon as the end of the epoch transition is finalized, the next epoch begins.
Domain Block Production
The domain block is deterministically driven from the consensus block and always follows the fork choice of the consensus chain.
The operator subscribes to the consensus block import notification, and for each imported consensus block the operator tries to build a domain block of defined structure by constructing the following components:
Block Body
- Extract the bundles of the domain, which the operator registered, from the consensus block and extract the extrinsics from the bundles.
- If there is no bundle, skip producing domain block for this consensus block
- Extrinsics will be ordered as described in Cryptographic sortition for Extrinsic ordering
- The resulting extrinsics will be used as the block body
Extrinsics Root
Merkle tree root of the extrinsics
State Root
- Execute the block body by following the instructions mentioned at Domain Block Execution on the Operator Node
- The state root after execution will be used as the state root in the block header
Parent Block
The parent block should be the last domain block that drives from the same branch as the incoming consensus block following the consensus chain Fork Choice Rule
-
Example
Consensus chain:
.. → b1 → b2 → b3 → b4
,b4
is the incoming consensus block that the operator trying to drive a domain block, whileb2
andb3
didn’t drive a domain block due to not bundle contains inside them, the last domain block of this branch is the one driving fromb1
thus it will be used as parent block of the domain block driving fromb4
Cryptographic sortition for Extrinsic ordering
- Deduplicate extrinsics.
- Group the signed extrinsics by sender
account_id
, and unsigned extrinsics into a separate group. - Use the consensus chain
Randomness
derived from PoT asextrinsics_shuffling_seed
. - Shuffle the grouped extrinsics using the Fisher–Yates algorithm based on the
extrinsics_shuffling_seed
. This generates an unbiased and deterministic permutation, while relative ordering for the transactions for the same sender does not change.
-
Example
Before grouping:
(Alice, 1), (-, 1), (Bob, 1), (Bob, 2), (Alice,2), (Charlie, 1), (Alice,3), (Charlie, 2), (-,2), (-,3)
(-)
for unsignedAfter grouping:
(Alice, 1), (Alice,2), (Alice,3), (Bob, 1), (Bob, 2), (Charlie, 1), (Charlie, 2), (-, 1), (-,2), (-,3)
After shuffle:
(-, 1), (Charlie, 1), (Alice, 1), (Bob, 1), (Alice,2), (-,2), (Charlie, 2), (Alice,3), (-,3), (Bob, 2)
Fork Choice Rule
The consensus chain uses the heaviest chain rule (i.e., smallest solution range) if forks have the same weight go with the longest one, while the domain chain always follows the fork choice of the consensus chain regardless of whether the domain fork is the longest fork or not.
Consensus chain (assume every block has the same weight):
b2 → b3 → b4
/
→ a1 → a2 → a3 → a4 → a5
Given domain chain, the consensus block a3
and a4
don’t contain bundles thus there is no domain block driving from them:
domain_b2 → domain_b3 → domain_b4
/
→ domain_a1 → domain_a2 → domain_a5
The best fork of the consensus chain is fork A as it is the longest fork, and the domain chain always follows the fork choice of the consensus chain thus its best fork is also fork A even though it is not the longest fork.
Domain Block Execution on the Operator Node
The main distinction between domain block execution and normal Substrate block execution lies in the calculation and collection of the storage root after completing each execution phase (InitializeBlock
, ApplyExtrinsic
, FinalizeBlock
). The storage roots collected during the process create an execution trace. This trace is then utilized to identify precise computational discrepancies within the fraud proof.
In Substrate, there is a trait Hooks
that each pallet can implement to execute some logic during the block execution, the related hooks here are on_initialize
and on_finalize
. A block with extrinsics in Substrate is primarily executed as follows:
-
Initialization
initialize_block(header)
- Execute the
on_runtime_upgrade
hooks if the runtime has been upgraded - Initialize System module pallet
- Execute the
on_initialize
hook of all non-system pallets
After executing
initialize_block
, we calculate the first state root as - Execute the
-
Execute the extrinsics one by one using
apply_extrinsic(uxt)
method- Apply extrinsic 0 ⇒
- Apply extrinsic 1 ⇒
- …
- Apply extrinsic -1 ⇒
After executing each extrinsic, we calculate the state root as
-
Finalization
finalize_block()
- Execute
on_idle
hook if there are still some weights remaining - Execute the
on_finalize
hook of all non-system pallets - Finalize system pallet
After executing
finalize_block()
, we calculate the state root as . - Execute
Therefore, the execution trace for a block with extrinsics is
Domain Sudo
Domains have a modified pallet to provide sudo call. The Sudo is triggerred from the Consensus chain and then executed in the Domain block.
pallet_domain_sudo
has a inherent extrinsic that is created and imported into Domain block if the Consensus block from which Domain block is created
from contains a Sudo Call for the targetted Domain. Only one sudo call is allowed per domain block. Multiple Call can be achieved using pallet_utility::BatchAll
.
Flow to execute a Sudo call on Domain.
- Sudo on Consensus chain will submit an encoded unsigned domain extrinsic to
pallet_domains::Call::send_domain_sudo_call
- If the targetted domain has the
pallet_domain_sudo
enabled, then the encoded call is stored.- Note: This storage is cleared on Consensus chain when there is a Successful bundle submission from the Domain.
- When domain operators are deriving a Domain Block from a given Consensus block, they check
pallet_domains::domain_sudo_call(domain_id)
if there is any sudo call. - If so, they will inject this Domain sudo Call as an Inherent extrinsic and executes the Domain block.
- Note:
pallet_domain_sudo
executed the provided the encoded domain call withRoot
origin.
- Note:
Since pallet_domain_sudo
provides an Unsigned extrinsic, if this extrinsic is manually constructed and included in the Domain Block, it will trigger
FraudProof::InherentExtrinsic
from the honest operators.
This inherent extrinsic also affects the FraudProof::InvalidDomainExtrinsicRoot
if any malicious operator does not include this inherent while importing Domain block.
Honest operators will submit above FraudProof variant with all the necessary storage proofs to construct the Domain Extrinsic root.
Lagging operator protection
In a distributed system, it is inevitable that some nodes may be lagging (e.g. due to network partition or slow hardware). This is critical for the operator node because when it produces a bundle, it needs to verify and guarantee all the extrinsic included in the bundle is valid. When the domain node tries to derive a new domain block from the bundles included in the consensus block, it will also verify all the extrinsic against the latest domain block (i.e. the parent domain block of the new block), if there is any invalid extrinsic found the whole bundle will be marked as invalid and the operator who produced this bundle will be slashed.
For a lagging operator, it is possible that when producing a bundle it verifies the extrinsic against an old domain block, and the extrinsic turns out to be invalid when other domain nodes verify it against the latest domain block, as a result, an honest but lagging operator will be slashed.
To protect the lagging operator, the consensus node when verifying the bundle will check the bundle contains an execution receipt that is derived from the latest domain block, which means the producer is not lagging, if it is not then the bundle will not be included in the consensus block so the operator won't be slashed.
The consensus node when performing this check also needs to ensure the execution receipt is extending the previous head receipt, this means if there is a gap between the latest domain block (i.e. HeadDomainNumber
) and the latest receipt on chain (i.e. HeadReceiptNumber
), which usually happen after a fraud proof is accepted and bad receipts were pruned, any bundle will be rejected. In this case, the operator needs to produce submit_receipt
to fill up the gap and after that they can resume producing submit_bundle
.
Domain Freeze, Unfreeze, and Prune Execution Receipt by Consensus Sudo
Generally, malicious activity from a domain operator is handled through Fraud proofs where honest operators verify the bundles before importing domain block and submit Fraud proof targeted at given bad Execution receipt. In a particular case where Fraud proof could not be verified automatically or included in the Consensus block, the bad ER never gets pruned and given enough time, it may even go out of challenge period.
In order to safe-guard against such an attack, Sudo on Consensus has the ability to Freeze, Unfreeze, and prune execution receipt of a domain.
pallet_domains::Call::freeze_domain(domain_id)
pallet_domains::Call::unfreeze_domain(domain_id)
pallet_domains::Call::prune_execution_receipt(domain_id, bad_er_hash)
The prune execution receipt dispatch makes an assumption that Sudo has validated the invalidity of the bad ER by verifying it offchain and/or through social consensus if the Governance is at play.
Note: Domain must be frozen to stop accepting new bundles before pruning a execution receipt.