Skip to main content

Cross-Domain Messaging (XDM)

This document describes the current messaging protocol between domains in a trusted code environment (permissioned runtime instantiation). This protocol describes messaging between the consensus chain and any domain and between two domains.

Primitives​

Chain​

A chain is a blockchain within the Subspace Network. A chain is identified as the Consensus chain or a Domain with a DomainID

    pub enum ChainId {
        Consensus,
        Domain(DomainId),
    }

Domain​

A Domain is a blockchain with some application modules. These applications act as senders and receivers of the messages using messaging protocol. A unique identifier identifies each application. Domain operators execute transactions bundled by other operators of the same domain when a new block is available.

Trusted third party​

A trusted third party from the point of view of the domains is the consensus chain, the farmer network for block production. Domains use consensus chain to submit transaction bundles, verify the Message proofs of domain_a on domain_b and submit fraud proofs.

Channel​

A Channel is a bi-directional connection between two domains. Channel connection is established when the src_chain_id initiates the channel connection open message, and dst_chain_id responds with either approval or rejection. Once a connection is open, sending messages back and forth is possible. A Channel would be open until a maximum number of messages are sent. This could be configured or defaulted to the maximum possible value of the Nonce type.

There is a deposit to open a channel between domains. Deposit should be high enough to discourage and make it economically inefficient to DDOS channel initiation connections between domains.

A channel can be closed on either end by the channel owner. Once closed, the channel will stop sending and receiving any further messages. The Relayer will communicate to the other domain to close the channel and clean up.

A channel can be in one of the following states (State):

  • Initiated: When a channel is initiated but has not received acknowledgment from the other domain.
  • Open: When a bi-directional channel is open to relay messages between domains.
  • Closed: When the channel is closed between the domains and stops receiving and sending new messages to the other domain.

A Channel is defined as follows

    struct Channel {
    	// Unique channel identifier within DomainID namespace
    	channel_id: ChannelID,
    	// State of the channel
    	state: State,
    	// Next valid Inbox nonce
    	next_inbox_nonce: Nonce,
    	// Next valid Outbox nonce
    	next_outbox_nonce: Nonce,
        // Latest outbox message nonce for which response was received from dst_chain.
    	latest_response_received_message_nonce: Nonce,
    	// Fee Model for this channel
    	fee: FeeModel,
    	// Max number of messages to be in outbox at a given time on both domains.
    	max_outgoing_messages: u32,
    	/// Owner of the channel
   	    /// Owner maybe None if the channel was initiated on the other chain.
	    maybe_owner: Option<AccountId>,
    }

Channel Inbox​

All the incoming messages to the domain are validated and added to a pool before processing. If specific message arrived earlier than a previous message, it is stored until the previous message(s) is processed in the order of Nonce.

Channel Outbox​

All messages originated from src_chain_id to a dst_chain_id will be added to this queue in the runtime state.

Messages stay in the outbox of src_chain_id until the domain block of src_chain_id containing the originating extrinsic is out of the challenge period or reached archiving depth (if src_chain_id is consensus chain).

There is also a notion of back pressure by limiting maximum number of messages queued max_outgoing_messages to outbox of src_chain_id. So when dst_chain_id doesn’t send any message responses, this should throttle the outbox until normal operation. Message is removed from the outbox once the message response is received from the dst_chain_id.

Channel Response Queue​

All the message responses to the messages in the outbox are validated and added to this queue. The message responses are passed to the application units within the domain. If a response for message arrived earlier that previous message responses, then this response is stored until the previous message responses are delivered.

Channel Nonce​

Channel nonce is used to order messages with in the channel and to avoid replay attacks.

A domain maintains 2 nonces for each domain and channel:

  • Incoming nonce: This nonce is used to order the incoming messages to the domain through the channel from other domain. The nonce starts at 0 and is incremented after each received message.
  • Outgoing nonce: This nonce is used to order the outgoing messages from this domain to the other domains. The nonce starts at 0 and is incremented after every sent message.

Message Proof​

Message proof that can verify the validity of the message from the point-of-view of the consensus chain. Proof combines the storage proofs to validate messages.

The proof consists of the following components:

  • MMR proof for the state root of the parent consensus block
    pub struct MMRProof {
    	// consensus block number below archiving depth at which this MMR proof was generated
    	consensus_block_number,
    	// Leaf data, that contains consensus storage root, is used to verify the `ConfirmedDomainBlocks` storage
    	leaf_data,
    	// merkle proof for this MMR
    	proof,
    }
  • Proof of the source domain state root and XDM inclusion in runtime
    pub struct Proof<BlockNumber, BlockHash, StateRoot> {
      /// MMR proof, which provides the state root of consensus hash MMRProof,
      pub proof: MMRProof,
      /// Storage proof that src chain state_root is confirmed on Consensus chain.
      /// This is optional when the src_chain is Consensus.
      pub domain_confirmed_proof: Option<StorageProof>,
      /// Storage proof that message is processed on src_chain.
      pub message_proof: StorageProof,
    }

Message​

Message encompasses the actual message being sent and metadata about the message itself. MessageID is a unique tuple of (ChannelID , Nonce). There are two types of Message payloads:

  • Protocol payload, used by the protocol to open or close, acknowledge channel connection with other domain.

  • Endpoint payload, used by the protocol to pass messages between endpoint on the src and dst domains.

    pub struct Message<Balance> {
        /// Chain (consensus or domain id) that initiated this message.
        pub src_chain_id: ChainId,
        /// Chain (consensus or domain id) this message is intended for.
        pub dst_chain_id: ChainId,
        /// ChannelId the message was sent through.
        pub channel_id: ChannelId,
        /// Message nonce within the channel.
        pub nonce: Nonce,
        /// Payload of the message
        pub payload: VersionedPayload<Balance>,
        /// Last delivered message response nonce on src_chain.
        pub last_delivered_message_response_nonce: Option<Nonce>,
    }

The response message follows the same structure except the payload contains either Protocol(Response) or Endpoint(Response)

Message Lifecycle​

Conceptually, a message can be in one of the following states during its lifecycle:

From the POV of the sender src_chain_id:

  • Outboxed: A message-request is added to the outbox for the relayers to relay message to dst_chain_id. It stays in the outbox until the message-response is received.
  • Delivered: After message-response is received and executed, the message request is cleared from the outbox.

From the POV of the receiver dst_chain_id:

  • Inboxed: A message-request is added to the inbox for the execution and response. It is not executed until the domain block of src_chain_id containing the originating extrinsic is out of the challenge period or reached archiving depth (if src_chain_id is consensus chain).
  • Cleared: After message-request is executed and message-response is constructed, the message-request is cleared from the inbox and message-response is added to the outbox.

Relayer Component​

A relayer component relays message from src_chain_id to dst_chain_id. Domain operators have builtin relayer to relay messages from the domain to other domains and the consensus chain.

Operators on domain_a relay messages originating in domain_a to the consensus network and listen for messages destined to domain_a from any other domain. Messages are sent through the consensus network where all operators of all domains are present.

The payload for the extrinsic could be a message-request or a message-response.

Fees​

Fees are collected from the sender of the message on src_chain_id to pay for relay and execution of their message on both src_chain_id and dst_chain_id respectively.

Compute fees are computed based on adjusted weights of the exact calls performed on both src_chain_id and dst_chain_id in total. Since we need to account for the block fullness of each src and dst chain, we also include a constant multiplier, currently 5, to the final fee collected from the user.

Collected compute fees for the portion of execution happening on src_chain_id is paid to operators of src_chain_id and the compute fees for the portion of execution happening dst_chain_id. The portion of fees that is to be distributed on dst_chain_id is burned on src_chain_id when message is added to outbox.

The burnt fees are subtracted from src_chain_id bookkeeping balance (if it’s a domain).

The relay fee is split equally among operators on src_chain_id and dst_chain_id who have submitted the ER that includes the message.

On the source chain, this reward is distributed when the message gets the response from the dst_chain_id. On the dst_chain_id, when it receives the next message, it will collect all the messages that are marked delivered on src_chain_id, mints the funds, and, distributes the rewards to the relayer pool on dst_chain_id for each message.

The minted fees are added to dst_chain_id bookkeeping balance (if it’s a domain).

Outbox Message Fees

  1. User sender sends a message from src_chain_id
  2. src_chain_id collects fees to be paid by sender as follows:
    1. Compute fee for message execution on dst_chain_id with XDM multiplier included. This amount is burnt on src_chain_id and minted on dst_chain_id later.
    2. Compute fee for message response execution on src_chain_id with XDM multiplier included.
  3. Message is sent to dst_chain_id.
  4. Once the response is received from dst_chain_id, src_chain_id distributes the rewards from sender to operators.
  5. This message nonce is sent to dst_chain_id as last_delivered_message_response_nonce as an acknowledgement so that it can reward its operators.

Inbox Message Fees

  1. dst_chain_id receives a message from src_chain_id
  2. dst_chain_id mints received fees after message validation
  3. Message is processed and response is sent
  4. After the delivery acknowledgement from src_chain_id, the dst_chain_id distributes the fees from sender equally to operators who have submitted the ER containing the message extrinsic

High-Level Workflow​

The following describes the generic message from one domain to another. This message could be a protocol message to initiate or close channel connection or an endpoint specific message through an established Channel. In either case, the base message passing remains same:

  1. User submits a transaction with the message and the required fees. The funds are locked in users account.
  2. Message with an assigned nonce is added to the outbox of src_chain_id with a runtime event issued.
  3. The operator of src_chain_id extracts the message and prepares it to be relayed it to the transaction pool of domain dst_chain_id. The relayer needs to construct a storage proof to prove that this message was accepted by domain src_chain_id (by proving the message is included in domain src_chain_id runtime state), and a state root of domain src_chain_id (with respect to a consensus chain state root) so that domain dst_chain_id can verify the storage proof.
  4. Operator of src_chain_id waits until the domain block with the ER containing the state root used to construct the storage proof is cleared from the challenge period before gossiping the message.
  5. Operator of src_chain_id gossips the message to the consensus network where all other operators are connected.
  6. Operators of dst_chain_id listen to gossip on consensus and takes the message bound for dst_chain_id into its domain transaction pool while ignoring other messages.
  7. Message proof is validated and if valid added to the inbox of dst_chain_id.
  8. Next message nonce is taken from the inbox of dst_chain_id.
  9. Message is executed and response is stored.
  10. Operator of dst_chain_id waits until the domain block with the message transaction is cleared from the challenge period before gossiping the message response.
  11. Message response on src_chain_id is validated using storage proofs on the consensus chain.
  12. Next Message response nonce is submitted to the endpoint and message is removed from the outbox of src_chain_id.
  13. When the src_chain_id prepares the next message, it will include the latest message nonce that was successful as part of the payload to notify dst_chain_id of message response acknowledgement.

XDM

Networking​

The messaging protocol uses the consensus network to relay messages, since all the operators of all the domains must also be connected to the consensus network.

This model assumes that there is at least one dst_chain_id domain operator to pick the transaction and include in the bundle. If there are no operators to pick the transaction, message could be undelivered until its resubmitted in the network.

Type definitions​

  • ChainId: enum that identifies whether the chain is Consensus or a Domain with DomainId
  • DomainID: uniquely identifies a given Domain, U32.
  • ChannelId: uniquely identifies a channel, U256.
  • AccountId: is the public key of the User account
  • Nonce: is an incrementing value used for ordering message and avoid replay attacks. We also use nonce as the message ID for a given domain for a given channel, U256.
  • MessageId: uniquely identifies a message, a tuple (ChannelId, Nonce)
  • EndpointId : represents a unique id of the endpoint on a domain, U64.

Functions​

Detailed description of each function required to be present in the protocol.

Chain Allow List​

Consensus chain maintains a ChainAllowList to keep track of the authorized domain chains that can establish channels with the Consensus chain. ChainAllowList can be defined at Genesis and can be updated later by the sudo account adding or removing chains.

When a Consensus chain receives an initiate_channel XDM to open the channel, if the dst_chain_id is in the allowlist, then channel is opened else the channel opening is rejected. In the latter case, the channel remains in Initialized state and the submitter can close it and get back the deposit. Practically, this means that a newly initialized domain chain needs to be approved by governance and added to the Consensus chain's ChainAllowList before it can initiate a channel with the Consensus chain.

Similarly, each domain chain maintains its own DomainChainAllowList to keep track of the authorized domains it can establish channels with. This allows the domains to control and restrict which other domains they want to interact with. Updating domain-specific lists is done within a domain by the domain's sudo or governance, without consensus chain approval.

Initiate Channel​

  1. Channel initiate_channel transaction is sent by any user of the src_chain_id domain with the channel opening deposit as a Channel owner.
  2. If the domain is in the allow list, the next available ChannelID is assigned to the new channel.
  3. If no Channel exits, Channel is created and set to Initiated status and cannot accept or receive any messages yet.
  4. When the channel is initiated, reserve deposit is taken from the channel and will be returned when Channel is closed.
  5. If the src_chain_id is not in the allow list of dst_chain_id, destination chain does not open a channel, but rather leaves it in Initiated state and responds with an Err. When the error response is received on source chain, it also does not open then channel and leaves it in the Initiated state.
  6. If both chains are in the allow list of each other, Protocol payload message to open the channel is added to the src_chain_id domain outbox with nonce 0

Open Channel​

Before sending any messages, domain needs to have a channel open with the dst_chain_id:

  1. Channel is initiated by src_chain_id as described in Initiate Channel
  2. Operator on dst_chain_id receives a message with the corresponding Protocol ChannelOpen payload and nonce=0.
  3. Channel status is set to Open and a corresponding event is issued
  4. Operators on dst_domain submits the transaction with message response to src_chain_id
  5. src_domain moves the channel state to Open and starts accepting messages to be sent over channel

Close Channel​

Any domain of either end of an Open or Initiated channel can close the channel:

  1. Channel close transaction is sent by the root user or Channel Owner. We allow closing channel request even if there the chain is not in the allow list so that reserve deposits can be claimed back. In the latter case, the initiator loses 20% of the deposit.
  2. Channel state is set to Closed.
  3. Protocol payload message to close channel is added to the src_chain_id outbox
  4. Operator on src_chain_id gossips the message
  5. Operator on dst_chain_id receives a message with the corresponding Protocol ChannelClose payload.
  6. Channel close response is submitted to src_chain_id
  7. Channel reserve deposit is returned back to the channel owner once closed.

Send message​

When user wants to send message from endpoint on src_chain_id to an endpoint on dst_chain_id with an open channel to dst_chain_id.

  1. User sends a transaction that results in a message to an endpoint on dst_chain_id.
  2. Transaction is included in the runtime state of src_chain_id.
  3. A next incrementing nonce > 0 (0 is always reserved for Channel open message) is assigned to the message bound to dst_chain_id.
  4. Next message nonce storage is updated.
  5. Execution layer stores the message in the Outbox and emits new_message event.
  6. Operator of src_chain_id waits until the domain block with the ER containing the state root used to construct the storage proof is cleared from the challenge period before gossiping the message (or below archiving depth if src_chain_id is consensus chain).
  7. The operator of src_chain_id extracts the message and prepares it to be relayed it to the transaction pool of domain dst_chain_id. The relayer needs to construct a storage proof to prove that this message was accepted by domain src_chain_id (by proving the message is included in domain src_chain_id runtime state), and a state root of domain src_chain_id (with respect to a parent consensus chain state root via MMR) so that domain dst_chain_id can verify the storage proof.
  8. Operator of src_chain_id gossips the message to the consensus network where all other operators are connected.

Receive message​

When a relayer from src_chain_id submits the message to the inbox of the dst_chain_id:

  1. dst_chain_id verifies if src_chain_id is in the allow list. If the chain is not in allow list, then dst_chain_id will return an error to src_chain_id with ChainNotAllowedError and skip next steps. Once the src_chain_id receives the response, it will revert necessary actions taken at the time of sending the request. For transporter, all the SSC burned will be minted back since the dst_chain did not return a mint Ok response.
  2. dst_chain_id verifies the message by verifying the storage proof from the point of view of the consensus chain as follows:
    1. Check if the MMR proof is constructed at a finalized consensus block to ensure the MMR::verify_proof result is deterministic regardless of the consensus chain fork.
    2. Verifies MMR proof using consensus chain MMR::verify_proof function to extract the MMR leaf data and the corresponding state root of consensus chain.
    3. Using consensus_chain_state_root, domain_confirmed_proof is verified and associated domain’s state_root is extracted from DomainBlockInfo
    4. Using domain_state_root, message_proof is verified and actual XDM is extracted from the storage proof.
  3. dst_chain_id adds the message to its inbox.
  4. dst_chain_id listens for next message to process from the inbox in the nonce order.
  5. Message is passed to the endpoint and eventually executed and response is stored for that message.
  6. dst_chain_id takes the latest delivered message nonce on src_chain_id and distributes the rewards to the operators and deletes any stored state pertaining to any message with nonce below the confirmed nonce.

Receive message response​

Relayer from the dst_chain_id will submit the message response to the src_chain_id:

  1. src_chain_id verifies the message response and adds it to the message response queue.
  2. src_chain_id listens for next message response and submits the response to the caller module on the src_chain_id
  3. src_chain_id marks the message nonce as the last confirmed message which is included in the next message bound to dst_chain_id
  4. src_chain_id then deletes the state pertaining to the original message from the runtime.

XDM delays​

Since XDM is inherently a request response protocol it may require a response in some cases.

The use cases such as transfer or sending some payload where response is just an acknowledgement, it is generally ignored. With that in mind, the following table will give the time to send a message from Any chain -> Any chain

Where K = Archiving depth (currently 100 consensus blocks) and D = Domain challenge period (currently 14400 domain blocks)

Domain → Any chainConsensus → Any domain
K + DK + D

For XDM where response is required from, the following table captures the time in blocks

Domain → Consensus → DomainConsensus → Domain → ConsensusDomain A → Domain B → Domain A
2 * (K + D)2 * (K + D)2 * (K + D)

XDM block processing limits​

At the moment, a maximum of 256 XDM messages per channel per chain are allowed to be included in a single Block.

The XDM messages are prioritized over regular transactions.

Example: If there are 2 channels between src and dst chain, Each of the chains will include at the most 256 * 2 XDM messages in a single block.

Runtime calls​

Listed in the order of call index in the runtime.

Initiate Channel​

initiate_channel(dst_chain_id: ChainId)

Anyone can initiate a channel to dst_chain given both dst_chain and src_chain allow other chain to open channel. User would need to deposit fee to initiate channel open and the deposit is reversed once the channel is closed. Currently, for each channel, we set 10_000 maximum outgoing messages. Once the channel reaches 10_000 pending XDM, it will not accept further messages until the pending XDMs are executed.

Once the channel is initiated, dst_chain gets a message to open channel and reverts back to src_chain to confirm the channel open. It would 2 challenge periods to open channel on both the chains. Once, both chains open channels, xdm messages can be sent through the channel.

Note: Ensure both src_chain and dst_chain has other chain in their allowlist. To add chain to the allowlist

Close channel​

close_channel(chain_id: ChainId, channel_id: ChannelId)

Either channel owner, who initiated the channel, or Sudo user can Close the open channel. Once the channel is closed, channel owner will receive the deposit back.

Once the channel is closed, no further xdm messages can be sent through the channel.

Add dst_chain to Consensus Allowlist​

update_consensus_chain_allowlist(update: ChainAllowlistUpdate,)

Only a sudo user can add dst_chain to the Consensus allowlist. Once the chain is added to the allow list, either consensus chain or dst_chain can initiate channel open.

Add dst_chain to Domain allowlist​

initiate_domain_update_chain_allowlist(domain_id: DomainId, update: ChainAllowlistUpdate)

Only a src_domain owner can add dst_chain to the src_domain allowlist. Once the dst_chain is added to the src_domain allowlist, either of the chains can initiate channel open. Ensure that dst_chain also included src_domain in its allow list.