Workflow

Domain Instantiation & Upgrades

When a new domain is instantiated, the consensus runtime generates a genesis ER hash using the genesis_config and domain_config for that domain.

The consensus chain will initiate a “balance” for this domain to track how many SSC were transferred in and out of this domain to make sure a domain cannot create SSC it did not already have via fees, XDM transfers or storage fee deposits. This balance does not include staked SSC as staking is tracked on the consensus chain directly.

genesis_receipt_hash is derived using a host function call passing the required genesis domain_config details mentioned below. The host function will construct the DomainConfig using the domain runtime and create the genesis ER by deriving the genesis_state_root, presented here, and returning the genesis_receipt_hash. This ER will not have any intermediate state roots, but only the final state root, as Genesis ER is considered special.

Operators who want to run this domain will create a genesis block using the GenesisConfig placed on the consensus chain during domain instantiation and construct the genesis ER used when submitting the next block’s ER thereby building a block tree.

Domain Instance Genesis Block Generation

When the domain is first instantiated in the consensus runtime, a genesis_state_root is derived using a host function with domain_config as input, the genesis_state_root will be further used to drive a genesis ER which will not have any intermediate state roots, only the final state root, as Genesis ER is considered special.

The genesis_state_root is required to be unique among all domain instances and consistent with the genesis_state_root generated by the domain instance node on the client side. The host function and the domain instance node derive genesis_state_root by generating a genesis block with RuntimeGenesisConfig as input, thus, to achieve uniqueness and consistency, the RuntimeGenesisConfig needs to be unique and consistent.

For all domain runtime types, RuntimeGenesisConfig should include the pallet genesis config of:

• system pallet SystemConfig { code } for it to be a valid RuntimeGenesisConfig
• domain-id pallet DomainIdConfig { domain_id } for it to be unique as it includes a unique domain_id

Domain Instance Node Bootstrap

After the operator instantiated a domain instance at the consensus chain by submitting an instantiate_domain extrinsic and waiting until the extrinsic is finalized (past the CONFIRMATION_DEPTH_K consensus blocks), the operator can use the resulting domain instance domain_id and created_at (block height) arguments to the subspace-node binary to run a domain instance node for the instantiated domain.

The domain instance node has two modes: bootstrap mode and sync mode. In the bootstrap mode, the node must bootstrap the domain instance chain by itself based on the domain registry state at the consensus chain. The node can sync the chain from other domain instance nodes in sync mode.

In bootstrap mode, there is an embedded consensus node and a bootstrapper. After the bootstrap is finished, a domain node will replace the bootstrapper.

The bootstrapper listens to the consensus node block import event, skips the block before created_at, after the block at created_at is imported, which contains the instantiate_domain extrinsic of the domain instance, using runtime API with domain_id to get the domain instance’s domain_config and the runtime_obj state.

Uses the domain_config and runtime_obj.domain_runtime_code to generate a RuntimeGenesisConfig in the same way as the host function, and uses that to construct sc_service::Configuration (which includes the chain_spec) then use runtime_obj.runtime_type to determine and build the desired domain service with sc_service::Configuration as input. After this step the domain service is up and the bootstrap is finished.

Domain Genesis Config

The domain_config contains:

1. domain_name: user-defined name for this domain.
2. runtime_id: domain runtime type that exists in RuntimeRegistry.
3. domain_id: identifier assigned to an instance of the domain.
4. specific configuration items, such as:
   • max_block_size: the max block size for this domain; may not exceed the system-wide MaxDomainBlockSize limit; used to compute bundle size limit.
   • max_block_weight: the max block weight for this domain; may not exceed the system-wide MaxDomainBlockWeight limit; used to compute bundle weight limit.
   • bundle_slot_probability: the probability of successful bundle in a slot (active slots coefficient); defines the expected bundle production rate, which must be > 0 .
5. allowlist: list of addresses allowed to run operators on this domain
6. initial_balances: list of initial balances on domain accounts
7. Any further genesis config details can be included as required and be passed down. These specific genesis details ensure the genesis_state_root is unique for each instantiated domain and thereby making genesis_er_hash unique across different instances of the same domain runtime.

Domain Runtime Upgrades

When a domain runtime is updated using upgrade_domain_runtime, the new runtime will come into effect at a future consensus chain block, specifically, the block at which the extrinsic upgrade_domain_runtime was executed successfully and DomainRuntimeUpgradeDelay blocks have passed since. When that future block height arrives, the consensus chain considers the new runtime to be the latest runtime and adds a digest log to indicate the upgrade to all domain operators.

Since every operator runs the consensus chain, they will include the new runtime as part of the next domain block taken from the consensus chain, since they see the digest log in the consensus block header.

There are some scenarios where new runtime may introduce the new host APIs that newer clients will use during any stage of bundle production and block import. If the operators still use the older clients, they won’t be able to proceed and the clients are supposed to panic due to the usage of missing host APIs in the new runtime. Hence, every operator would be forced to update the client in this case. The upgrade process involves running the latest client in place of the older client. While it's not strictly necessary, it would be beneficial to automatically signal the outdated client to operators later.

Bundle Producer Election

For each time slot, each operator denoted with operator_id participates in the slot leader election for the domain domain_id they are staking on to determine whether they are eligible to produce a bundle in this slot, as follows:

1. Initialization
   1. Get the global_challenge for this slot from the Proof-of-Time chain.
   2. Retrieve secret_key from keystore.
2. VRF
   1. Make transcript for the VRF from the global_challenge and VRF label for this domain_id.
   2. Generate a VRF signature by applying the VRF to the global_challenge and the operator's private key as vrf_signature = vrf_sign(secret_key, transcript). The VRF signature contains a vrf_signature.proof, which can be used by others to verify that the VRF vrf_signature.output was correctly generated without knowing the operator’s private key.
3. Threshold Check
   1. Compute the threshold based on the operators operator_stake = current_total_stake in Operators registry for this domain proportionally to the total_domain_stake = current_total_stake of all operators of this domain in stake_summary of the DomainRegistry as threshold = MAX * (operator_stake / total_domain_stake) * target_bundles_per_slot

      • Example

        If threshold is stored in u128, then MAX is $2^{128}-1$. If the operator has $1/10$ of total stake in this domain, according to the formula above they should check whether their VRF output numeric values is below $2^{128}/10$.

   2. Check whether the VRF vrf_signature.output for a slot is strictly below ($\<$) the threshold as integers.
   3. If it is, the operator is a slot leader for that slot and can produce a bundle. They should generate a ProofOfElection.
   4. If it isn’t, they skip this slot

Domain Bundle Production

If, for this time slot, this operator was successfully elected a slot leader, they can produce a bundle (as defined) as follows:

1. Take the ProofOfElection of the slot leader.

2. Fetch the ExecutionReceipt for the last block executed locally from the domain client and attach it to the bundle header. This ExecutionReceipt must be based on the longest branch of the consensus chain, although it may not point to the tip of the chain, as this depends on when the last bundle for this domain was included in a consensus chain block. If there was a fraud detected at the same height, the locally produced ExecutionReceipt (if valid) will be replacing the fraudulent one in the BlockTree.

3. Attach the full execution_trace to the given ExecutionReceipt

4. Grab all extrinsics within the specified range tx_range and attach to the body.

   If there is no extrinsic, the operator will skip producing a bundle and the following steps (TODO when challenge period is redefined in consensus blocks.)

   (Currently) If there is no extrinsic, the operator will further look into the block tree

   1. If all domain blocks in the challenge period are empty blocks then the operator will skip producing an empty bundle thus skipping the following steps
   2. If there's a non-empty domain block in the challenge period, the operator will continue the following step to produce an empty bundle to derive the non-empty domain block out of the challenge period (accelerate confirmation time)

5. Compute the bundle_extrinsics_root and attach to the header.

6. Compute the bundle_size and estimated_bundle_weight.

7. Note the storage fees to be paid to the consensus block author as per Bundle Storage Fees

8. Sign the bundle header.

9. Build the bundle header as described.

10. Broadcast the full bundle over the consensus network.

Transaction Selection for Bundle Production

When an operator is elected to produce a bundle, they must select transactions to be included in that bundle according to their transaction pool range (TX_RANGE), as follows:

1. Compute slot_vrf_hash for this slot as hash(vrf_signature.output)
2. Identify txs for inclusion into the bundle for this slot by looking into the transaction pool and identify all transactions whose senders account_id (as integer), is within the range as bidirectional_distance(slot_vrf_hash, public_key_hash) <= TX_RANGE/2

The operator may only include as many transactions within this range as fit within the bundle max_bundle_weight and max_bundle_size limits for this domain.

Initial Domain Bundle Verification by Consensus Nodes

All consensus nodes will perform the following verification when a new bundle is received over the network. All valid bundles are added to the local extrinsics pool and propagated to all peers on the network. Any invalid bundles are not added to the pool (no fraud proofs for invalid bundles received, only fraud proofs for invalid bundles that are included in a block).

1. Ensure HeadDomainNumber - HeadReceiptNumber = 1 otherwise submit_receipt is expected instead of submit_bundle
2. Verify the domain_id is in the DomainRegistry.
3. Verify the ProofOfElection for this domain and operator.
   1. Ensure the slot_number is no older than the slot of the block current_block_number - BundleLongevity.
   2. Verify the slot_number and the proof_of_time is correctly computed.
   3. Verify the vrf_signature based on the operator signing key and the global challenge that is derived from the slot_number and the proof_of_time.
   4. Verify the vrf_signature is below the threshold that is derived from the operator_stake / total_domain_stake and the bundle_slot_probability.
4. Verify the bundle header signature for the registered domain operator.
5. Ensure the bundle does not exceed the bundle max_bundle_weight and max_bundle_size limits for this domain.
6. Ensure the bundle is well-formed:
   1. Verify the execution_trace_root is correctly computed for the execution_trace.
   2. Verify the bundle_extrinsics_root is correctly computed for all included extrinsics.
   3. Verify the bundle_size and estimated_bundle_weight were correctly computed for the bundle body.
7. Ensure the ExecutionReceipt builds on the head of current BlockTree for this domain.
   1. Verify domain_block_number is equal to:
      • HeadReceiptNumber + 1 if this is the first bundle of the domain in the block
      • or HeadReceiptNumber since HeadReceiptNumber is increased by the previous bundle in the block
   2. Verify the consensus_block_hash exists at the specified consenus_block_height on the consensus client.
   3. Based on parent_domain_receipt_hash, verify the parent_domain_block exists at the specified parent_domain_height within the BlockTree on the operator client. If the ER is beyond the BlockTreePruningDepth it is too old and will simply be ignored.
   4. Verify all block_extrinsics_roots exist within the execution_inbox of the parent_domain_block.

Bundle Equivocation

A dishonest operator may produce multiple bundles on the same slot with the same proof-of-election. Similar to how consensus block equivocation is addressed, consensus chain nodes perform a check to determine if a bundle has been equivocated when verifying its validity. If an equivocation is detected, then this bundle is invalid, and is not included in the block.

Consensus Block Verification

On receipt of a new consensus block, each consensus node now needs to check to ensure all included bundle headers were pre-validated locally. If they see a new bundle, they will request and run validation. If that bundle is invalid or any previously invalidated bundles are included in the farmer block, it is simply discarded and ignored.

Bundle Header Application

On execution of a new consensus block, all included bundles will be applied to the state of pallet_domains as each included bundle header will call submit_bundle.

For each new bundle, each consensus node will:

1. Extract the ExecutionReceipt
2. Retrieve the parent_domain_block from the BlockTree and conditionally update the tree. If the parent does not exist within the tree, this ExecutionReceipt has just expired (rare event) and is simply ignored.
3. If this is a new ER, we will extend the BlockTree. If no fraud has occurred, it will extend the tip of the longest branch.
   1. Add a new layer to the tree, inserting the ER as the first entry as a new DomainBlock.
   2. Add the bundle_extrinsics_root to the execution_inbox
   3. Add the operator_id to the operator_ids who submitted this ER
   4. Apply XDM coin transfers to the domain’s balance
   5. Apply all operator fees from the ER for the domain block for which the challenge period has now passed:
      • The compute fees should be divided equally between all operators in the operator_ids field for the parent DomainBlock in the BlockTree.
      • The compute fees are automatically staked and subtracted from domain’s balance.
      • The storage fees should be refunded to the bundle authors of bundles included in the confirmed block. These should be applied individually to their current_epoch_reward in the OperatorPool
4. Otherwise, reject the receipt that tries to create new branch in the block tree. If fraud has occurred, a new branch will not be created. Instead, the system requires the submission of a fraud proof to prune the fraudulent ER at the specific height before any new ER can be submitted at that height. Fraud verification is not handled here as the consensus node cannot determine which (or all) ExecutionReceipt is actually fraudulent at this level. It is implied that an honest operator will eventually submit a fraud proof to address the issue before submitting new ER. If the fraud proof for the receipt already present in the block tree has already been seen, then it's operators are marked as pending slashing and the new receipt will create a new head as described in step 3.
5. If this ER has already been seen, we will be confirming an existing entry within the block tree. Retrieve the existing DomainBlock from the BlockTree

   1. If this is the tip of BlockTree

      • Add the bundle_extrinsics_root to the execution_inbox
      • Add the operator_id to the operator_ids

   2. If this is not the tip of the block tree and we have a stale ER, it is directly rejected and not included in the BlockTree at all.
6. If any domain block reached BlockTreePruningDepth, then we confirm it:
   1. refund the bundle storage fees;
   2. distribute the operator rewards;
   3. mark as pending slash the operators whose bundles this receipt marked as invalid;
   4. if StakeEpochDuration has passed, do epoch transition.
7. Slash any operators (and their nominators) who are pending slash, but not more than MAX_NOMINATORS_TO_SLASH at a time.
8. Accept this bundle as successful and awaiting execution on the domain.

Domain Epoch Transition

A domain staking epoch is an interval of blocks during which staking distribution remains the same. It is important to ensure a correct and provable @Bundle Producer Election that is not influenced short-term by deposits, withdrawals, and earned fees. This StakeEpochDuration period is currently set to 100 blocks, or roughly 10 minutes. The end of each epoch triggers a series of events to transition to the next epoch. These events include:

• allocation of fees earned for the blocks confirmed (older than BlockTreePruningDepth) during the epoch,
• deposits and withdrawals of stake,
• operator registrations and deregistrations,
• recalculation of stake distribution for the slot leader VRF election.

An epoch transition occurs after every StakeEpochDuration blocks (or when forced in a force_staking_epoch_transition(domain_id)extrinsic) after all new bundles in the last block have been executed. During the domain epoch transition, we do the following steps:

1. Re-stake operators’ nomination taxes on their rewards

   • Each operator will get a cut of nomination_tax * current_epoch_fees of all rewards issued to their pool as per nomination_tax specified in the operator’s config.
   • The operator’s cut will be automatically re-staked to the operator’s nomination as a deposit. Operator’s shares, current_total_shares and current_total_stake will be updated with the corresponding deposit later when deposits are processed.
   • The current_epoch_fees is temporarily updated to current_epoch_fees*(1-nomination_tax) for the rest of the calculations during the epoch transitions. It will be reset to 0 for the new epoch.

2. If there are any operators pending slash, remove their stake from the VRF election for the next epoch.

3. Finalize domain’s staking summary.

   For each operator operating on the next epoch (existing and new operators), do the following:

   1. Update the stake with received fees total_stake = current_total_stake + current_epoch_fees

   2. OperatorEpochSharePrice storage is updated with new share price (which excludes the collected nomination tax).

      share_price = (current_total_stake + current_epoch_fees) / total_shares

   3. Compute how much to reduce the stake corresponding to all withdrawals_in_epoch unstaked shares total_stake=total_stake-withdrawals_in_epoch/share_price

   4. Compute how much to increase the stake corresponding to all deposits_in_epoch as total_stake=total_stake+deposits_in_epoch*share_price

   5. Set current_total_stake and current_total_shares to newly computed values and deposits_in_epoch, withdrawals_in_epoch and current_epoch_fees to 0.

As soon as the end of the epoch transition is finalized, the next epoch begins.

Domain Block Production

The domain block is deterministically driven from the consensus block and always follows the fork choice of the consensus chain.

The operator subscribes to the consensus block import notification, and for each imported consensus block the operator tries to build a domain block of defined structure by constructing the following components:

Block Body

1. Extract the bundles of the domain, which the operator registered, from the consensus block and extract the extrinsics from the bundles.
   1. If there is no bundle, skip producing domain block for this consensus block
2. Extrinsics will be ordered as described in Cryptographic sortition for Extrinsic ordering
3. The resulting extrinsics will be used as the block body

Extrinsics Root

Merkle tree root of the extrinsics

State Root

1. Execute the block body by following the instructions mentioned at Domain Block Execution on the Operator Node
2. The state root after execution will be used as the state root in the block header

Parent Block

The parent block should be the last domain block that drives from the same branch as the incoming consensus block following the consensus chain Fork Choice Rule

• Example

  Consensus chain: .. → b1 → b2 → b3 → b4 , b4 is the incoming consensus block that the operator trying to drive a domain block, while b2 and b3 didn’t drive a domain block due to not bundle contains inside them, the last domain block of this branch is the one driving from b1 thus it will be used as parent block of the domain block driving from b4

Cryptographic sortition for Extrinsic ordering

1. Deduplicate extrinsics.
2. Group the signed extrinsics by sender account_id, and unsigned extrinsics into a separate group.
3. Use the consensus chain Randomness derived from PoT as extrinsics_shuffling_seed.
4. Shuffle the grouped extrinsics using the Fisher–Yates algorithm based on the extrinsics_shuffling_seed. This generates an unbiased and deterministic permutation, while relative ordering for the transactions for the same sender does not change.

• Example

  Before grouping:(Alice, 1), (-, 1), (Bob, 1), (Bob, 2), (Alice,2), (Charlie, 1), (Alice,3), (Charlie, 2), (-,2), (-,3) (-) for unsigned

  After grouping: (Alice, 1), (Alice,2), (Alice,3), (Bob, 1), (Bob, 2), (Charlie, 1), (Charlie, 2), (-, 1), (-,2), (-,3)

  After shuffle: (-, 1), (Charlie, 1), (Alice, 1), (Bob, 1), (Alice,2), (-,2), (Charlie, 2), (Alice,3), (-,3), (Bob, 2)

Fork Choice Rule

The consensus chain uses the heaviest chain rule (i.e., smallest solution range) if forks have the same weight go with the longest one, while the domain chain always follows the fork choice of the consensus chain regardless of whether the domain fork is the longest fork or not.

Consensus chain (assume every block has the same weight):

     b2 → b3 → b4
    /
→ a1 → a2 → a3 → a4 → a5 

Given domain chain, the consensus block a3 and a4 don’t contain bundles thus there is no domain block driving from them:

            domain_b2 → domain_b3 → domain_b4
          /
→ domain_a1 → domain_a2 → domain_a5

The best fork of the consensus chain is fork A as it is the longest fork, and the domain chain always follows the fork choice of the consensus chain thus its best fork is also fork A even though it is not the longest fork.

Domain Block Execution on the Operator Node

The main distinction between domain block execution and normal Substrate block execution lies in the calculation and collection of the storage root after completing each execution phase (InitializeBlock, ApplyExtrinsic, FinalizeBlock). The storage roots collected during the process create an execution trace. This trace is then utilized to identify precise computational discrepancies within the fraud proof.

In Substrate, there is a trait Hooks that each pallet can implement to execute some logic during the block execution, the related hooks here are on_initialize and on_finalize. A block with $n$ extrinsics in Substrate is primarily executed as follows:

1. Initialization initialize_block(header)

   1. Execute the on_runtime_upgrade hooks if the runtime has been upgraded
   2. Initialize System module pallet
   3. Execute the on_initialize hook of all non-system pallets

   After executing initialize_block, we calculate the first state root as $Root_{0}$

2. Execute the $n$ extrinsics one by one using apply_extrinsic(uxt) method

   1. Apply extrinsic 0 ⇒ $Root_{1}$
   2. Apply extrinsic 1 ⇒ $Root_2$
   3. …
   4. Apply extrinsic $n$-1 ⇒ $Root_{n}$

   After executing each extrinsic, we calculate the state root as $Root_{\{1, 2, .., n\}}$

3. Finalization finalize_block()

   1. Execute on_idle hook if there are still some weights remaining
   2. Execute the on_finalize hook of all non-system pallets
   3. Finalize system pallet

   After executing finalize_block(), we calculate the state root as $Root_{ n+1 }$.

Therefore, the execution trace for a block with $n$ extrinsics is $[Root_{0}, Root_{1}, …, Root_{n+1}]$

Domain Sudo

Domains have a modified pallet to provide sudo call. The Sudo is triggerred from the Consensus chain and then executed in the Domain block. pallet_domain_sudo has a inherent extrinsic that is created and imported into Domain block if the Consensus block from which Domain block is created from contains a Sudo Call for the targetted Domain. Only one sudo call is allowed per domain block. Multiple Call can be achieved using pallet_utility::BatchAll.

Flow to execute a Sudo call on Domain.

• Sudo on Consensus chain will submit an encoded unsigned domain extrinsic to pallet_domains::Call::send_domain_sudo_call
• If the targetted domain has the pallet_domain_sudo enabled, then the encoded call is stored.
  • Note: This storage is cleared on Consensus chain when there is a Successful bundle submission from the Domain.
• When domain operators are deriving a Domain Block from a given Consensus block, they check pallet_domains::domain_sudo_call(domain_id) if there is any sudo call.
• If so, they will inject this Domain sudo Call as an Inherent extrinsic and executes the Domain block.
  • Note: pallet_domain_sudo executed the provided the encoded domain call with Root origin.

Since pallet_domain_sudo provides an Unsigned extrinsic, if this extrinsic is manually constructed and included in the Domain Block, it will trigger FraudProof::InherentExtrinsic from the honest operators.

This inherent extrinsic also affects the FraudProof::InvalidDomainExtrinsicRoot if any malicious operator does not include this inherent while importing Domain block. Honest operators will submit above FraudProof variant with all the necessary storage proofs to construct the Domain Extrinsic root.

Lagging operator protection

In a distributed system, it is inevitable that some nodes may be lagging (e.g. due to network partition or slow hardware). This is critical for the operator node because when it produces a bundle, it needs to verify and guarantee all the extrinsic included in the bundle is valid. When the domain node tries to derive a new domain block from the bundles included in the consensus block, it will also verify all the extrinsic against the latest domain block (i.e. the parent domain block of the new block), if there is any invalid extrinsic found the whole bundle will be marked as invalid and the operator who produced this bundle will be slashed.

For a lagging operator, it is possible that when producing a bundle it verifies the extrinsic against an old domain block, and the extrinsic turns out to be invalid when other domain nodes verify it against the latest domain block, as a result, an honest but lagging operator will be slashed.

To protect the lagging operator, the consensus node when verifying the bundle will check the bundle contains an execution receipt that is derived from the latest domain block, which means the producer is not lagging, if it is not then the bundle will not be included in the consensus block so the operator won't be slashed.

The consensus node when performing this check also needs to ensure the execution receipt is extending the previous head receipt, this means if there is a gap between the latest domain block (i.e. HeadDomainNumber) and the latest receipt on chain (i.e. HeadReceiptNumber), which usually happen after a fraud proof is accepted and bad receipts were pruned, any bundle will be rejected. In this case, the operator needs to produce submit_receipt to fill up the gap and after that they can resume producing submit_bundle.

Domain Freeze, Unfreeze, and Prune Execution Receipt by Consensus Sudo

Generally, malicious activity from a domain operator is handled through Fraud proofs where honest operators verify the bundles before importing domain block and submit Fraud proof targeted at given bad Execution receipt. In a particular case where Fraud proof could not be verified automatically or included in the Consensus block, the bad ER never gets pruned and given enough time, it may even go out of challenge period.

In order to safe-guard against such an attack, Sudo on Consensus has the ability to Freeze, Unfreeze, and prune execution receipt of a domain.

pallet_domains::Call::freeze_domain(domain_id)
pallet_domains::Call::unfreeze_domain(domain_id)
pallet_domains::Call::prune_execution_receipt(domain_id, bad_er_hash)

The prune execution receipt dispatch makes an assumption that Sudo has validated the invalidity of the bad ER by verifying it offchain and/or through social consensus if the Governance is at play.

Note: Domain must be frozen to stop accepting new bundles before pruning a execution receipt.