THE GRID:
The Global Resilient Internet Datalink

1.0Introduction

The Grid is a peer-to-peer encrypted storage protocol. Participating nodes, called ZODES, form a network of program-scoped, append-only log stores. Clients encrypt all data locally before upload; ZODES store and serve only opaque ciphertext. The protocol has no consensus layer, no tokens, and no global state.

Existing decentralized storage systems typically require participants to reach consensus on a shared ledger, introduce protocol-native tokens to incentivize storage providers, or maintain globally replicated state that every node must process. These design choices impose coordination overhead, economic complexity, and scalability ceilings that are unnecessary when the primary goal is simply to store and retrieve encrypted blobs. The Grid sidesteps these concerns entirely: a ZODE is a stateless relay that accepts ciphertext, appends it to a local log, and optionally gossips it to peers. There is no ordering protocol beyond append indexing, no fee market, and no finality mechanism. The result is a minimal, high-throughput substrate on which higher-level applications (identity registries, messaging channels, document stores) can be composed as programs without inheriting the weight of a full blockchain stack. A Services layer built atop this substrate provides a standard framework for stateless servers that use programs for persistent state and expose HTTP endpoints, supporting both native and sandboxed WASM runtimes.

Cryptographic privacy is a first-class invariant rather than an optional feature. Every entry is encrypted client-side with either XChaCha20-Poly1305 or a ZK-friendly Poseidon sponge, and signatures are embedded inside the ciphertext so that ZODES can never observe authorship, content structure, or access patterns beyond coarse timing and payload size. Post-quantum resistance is likewise non-negotiable: all signing and key-agreement operations use hybrid constructions that pair classical algorithms (Ed25519, X25519) with their post-quantum counterparts (ML-DSA-65, ML-KEM-768), ensuring that an attacker must break both families simultaneously. This document defines the Grid protocol in implementation-neutral terms. Any conforming implementation, regardless of programming language, that follows the wire formats, cryptographic constructions, and behavioral rules described here can interoperate with any other conforming implementation.

2.0Terminology

3.0Serialization

All protocol messages, program descriptors, and gossip payloads use CBOR (RFC 8949) with deterministic encoding (RFC 8949 §4.2.1).

Implementations MUST produce identical byte sequences for identical logical values. Field ordering within CBOR maps MUST be consistent. The canonical CBOR bytes are the input to all hash operations (ProgramId derivation, CID computation, etc.). Binary fields (payloads, keys, ciphertexts) are encoded as CBOR byte strings.

4.0Cryptographic Identity

4.1Key Hierarchy

All keys derive from a NeuralKey, a 256-bit secret generated by CSPRNG.

NeuralKey (256-bit CSPRNG)
|
+-- IdentitySigningKey (Ed25519 + ML-DSA-65)
|   Derived with: identity_id (16 bytes)
|
+-- MachineKeyPair (per device, per epoch)
    Derived with: identity_id, machine_id, epoch
    Contains:
    +-- Ed25519      (classical signing)
    +-- X25519       (classical key agreement)
    +-- ML-DSA-65    (post-quantum signing)
    +-- ML-KEM-768   (post-quantum encapsulation)

All four key types are always present in every MachineKeyPair. There is no classical-only mode.

4.2Key Derivation

All derivation uses HKDF-SHA256 with salt = None unless otherwise specified.

Identity Signing Key. Two separate derivations from the NeuralKey:

Machine Key Pair. Two-level derivation. Step 1, machine seed:

machine_seed = HKDF-SHA256(
    ikm  = NeuralKey,
    info = "cypher:shared:machine:v1"
           || identity_id || machine_id
           || epoch_be_bytes
)

Step 2, individual key seeds from machine seed:

ML-KEM-768 keys are generated deterministically: the PQ encrypt seed is expanded via HKDF into d ("mlkem768:d") and z ("mlkem768:z") parameters, which are passed to ML-KEM-768 deterministic key generation.

4.3Machine Key Capabilities

Machine keys carry a bitflag:

Capabilities are metadata; enforcement is at the application/policy layer.

4.4Hybrid Signatures

A HybridSignature always contains both components:

Binary format: ed25519_bytes (64) || ml_dsa_bytes (3309)

Both components sign the same message. Both MUST verify for the signature to be considered valid.

4.5Hybrid Key Encapsulation

Key agreement combines X25519 and ML-KEM-768:

x25519_ss = X25519(sender_secret,
                   recipient_public)
(mlkem_ct, mlkem_ss) =
    ML-KEM-768.Encapsulate(recipient_pk)

shared_secret = HKDF-SHA256(
    ikm  = x25519_ss || mlkem_ss,
    salt = None,
    info = "zid:encap:v1"
) -> 32 bytes

The output is a SharedSecret (32 bytes) and an EncapBundle:

An attacker must break both X25519 and ML-KEM-768 to recover the shared secret.

4.6DID Encoding

Ed25519 public keys are encoded as did:key identifiers: did:key:z + base58btc(0xed01 || ed25519_public_key_bytes). The multicodec prefix 0xed01 identifies Ed25519 public keys.

4.7Shamir Secret Sharing

The NeuralKey MAY be split into Shamir shares for backup and recovery. Split: split(secret[32], total, threshold, rng). Combine: combine(shares[≥ threshold]). Each share is serialized as hex: hex(index_byte || share_data).

Identity generation, signing, and machine key derivation can all operate by ephemerally reconstructing the NeuralKey from shares, performing the operation, and immediately zeroizing the secret.

5.0Programs

A Program is the fundamental organizational unit of the Grid. It defines a named, versioned data scope: all storage, subscriptions, topic routing, and encryption policy are scoped by program. Programs are passive descriptors: they specify what data looks like, not how to process it.

5.1ProgramDescriptor

A CBOR map with at minimum:

Program-specific descriptors MAY add additional fields. Two implementations that serialize the same descriptor fields to CBOR MUST produce the same ProgramId.

5.2ProgramId

A 32-byte value:

ProgramId = SHA-256(CBOR(ProgramDescriptor))

Displayed as 64 lowercase hex characters.

5.3ZID (Zero Identity)

ZidMessage: owner_did (text), display_name (optional text), timestamp_ms (uint64), signature (PQ-hybrid, §7.7).

5.4Interlink

v2 Descriptor (current default):

ZMessage: sender_did, channel_id, content (UTF-8), timestamp_ms, signature. Maximum encoded message size: 64 KB.

Channel-to-sector mapping:

SectorId = SHA-256("interlink/channel/"
                   || channel_id_bytes)

ZID (v1) and Interlink (v2) are default programs: ZODES subscribe to them automatically.

6.0Sectors & Storage

Within a program, data is organized into sectors, append-only logs of encrypted entries. A sector is identified by a (ProgramId, SectorId) pair.

6.1SectorId

An opaque 32-byte identifier. In the sector protocol, sector IDs MUST be exactly 32 bytes. ZODES MUST reject requests with non-32-byte sector IDs. Displayed as lowercase hex.

6.2Cid (Content Identifier)

Cid = SHA-256(ciphertext)

A 32-byte content-addressed identifier derived from the stored ciphertext. Displayed as 64 lowercase hex characters.

6.3Append-Only Logs

Each sector is an ordered sequence of entries indexed starting at 0.

6.4Key Layout

key = program_id (32 B) || sector_id (32 B)
   || index (8 B, big-endian)

Total key size: 72 bytes.

6.5Operations

6.6Policy Enforcement

7.0Encryption & Signing

Data in sectors is protected by two complementary mechanisms: encryption ensures confidentiality; signing provides authenticity and integrity. Both happen client-side; ZODES never see plaintext.

7.1SectorKey

A SectorKey is a random 256-bit symmetric key generated via CSPRNG. Key wrapping (§7.5) is unchanged regardless of encryption algorithm.

7.2XChaCha20-Poly1305

sealed = nonce (24 bytes) || ciphertext || tag (16 bytes)

Nonce: 192-bit, randomly generated per encryption. AAD: program_id (32) || sector_id (32), binding ciphertext to its program and sector.

7.3Poseidon Sponge

Parameters: BN254 scalar field, rate=2, capacity=1.

sealed = nonce (32 bytes)
      || ciphertext_elements (32 bytes each)
      || tag (32 bytes)

• Nonce: 256-bit, randomly generated per encryption.
• AAD: program_id || sector_id absorbed into the sponge before the plaintext.
• Field element packing: Plaintext is split into chunks of 30 data bytes. Each chunk is packed into a 32-byte field element as [1-byte length][up to 30 bytes data][zero-pad].
• Duplex mode: The sponge absorbs key, nonce, and AAD elements. For each rate-sized chunk it squeezes keystream, adds to plaintext to produce ciphertext, then absorbs plaintext.
• Tag: Final 32-byte squeeze authenticates the entire encryption.

The choice of algorithm is per-program via proof_system in the ProgramDescriptor (§5.1): None uses XChaCha20-Poly1305; Groth16 uses Poseidon sponge (required for shape proofs, §8).

7.4Padding

Before encryption, content MUST be padded to fixed-size buckets to resist payload-size analysis. Buckets grow in 2× progression:

Padding format:

padded = content_length (4 bytes, LE)
      || content
      || 0x00 * (bucket_size - 4 - content_length)

7.5Key Wrapping

Step 1, Hybrid key agreement: between sender’s MachineKeyPair and the recipient’s MachinePublicKey using §4.5, producing a SharedSecret and EncapBundle.

Step 2, Context-bound wrapping:

wrap_key = HKDF-SHA256(
    ikm  = SharedSecret,
    salt = None,
    info = "grid:sector-key-wrap:v1"
           || program_id || sector_id
)

wrapped_sector_key = XChaCha20-Poly1305(
    key       = wrap_key,
    nonce     = random 192-bit,
    plaintext = sector_key_bytes
)

The result is a KeyEnvelopeEntry:

7.6Sector ID Derivation

For metadata-private storage, sector IDs are derived client-side via two-step HKDF:

derivation_key = HKDF-SHA256(
    ikm=shared_secret,
    salt="grid:sector:v1",
    info="grid:sector:derive-key:v1")

sector_id = HKDF-SHA256(
    ikm=derivation_key,
    salt="grid:sector:v1",
    info=<application-defined string>)

The intermediate derivation key MUST be zeroized after use. Info string convention: "grid:{program}:{purpose}:{...fields}".

Properties: Deterministic (same secret + info → same sector ID), Unlinkable (different info strings → unrelated IDs), Collision-resistant (32-byte HKDF output).

7.7Message Signing

Signatures provide authenticity and integrity for entries within the encrypted blob. They are PQ-hybrid (Ed25519 + ML-DSA-65) and are produced before encryption, verified after decryption. Signatures live inside the encrypted payload; the ZODE never sees them.

7.8Signable Bytes

signable_bytes = canonical_cbor(
    all_fields_except_signature)

Fields MUST be serialized in deterministic CBOR order.

7.9HybridSignature Format

Ed25519 (64) || ML-DSA-65 (3309) = 3,373 bytes total. Both components sign the same signable_bytes. Both MUST verify.

7.10Signing and Verification Flow

1. Signing: Client computes signable_bytes, signs with both algorithms, appends signature, then encrypts.
2. Verification: Recipient decrypts, recomputes signable_bytes, verifies both Ed25519 and ML-DSA-65.

7.11Ed25519-Only Fallback (v1)

For v1 compatibility, programs MAY support Ed25519-only signatures (64 bytes).

8.0Shape Proofs

Shape proofs allow clients to prove that an encrypted blob conforms to a declared schema without revealing plaintext. Programs with proof_system = Groth16 use Poseidon sponge encryption (§7.3) and MAY attach a shape proof to each entry.

8.1ProofSystem Enum

8.2FieldSchema

Message structure is described by a FieldSchema: a sequence of FieldDef (named field + CborType).

schema_hash = SHA-256(canonical_cbor(schema))

8.3ShapeProof Wire Format

8.4Universal Circuit

The shape-proof circuit proves three predicates:

1. shape(B): Plaintext B conforms to the FieldSchema.
2. Poseidon_encrypt(B, K, N, AAD) = C: Ciphertext C is the correct encryption.
3. Poseidon(C) = ciphertext_hash: Public hash matches.

8.5Strong Binding

The ZODE computes Poseidon(received_ciphertext) and MUST verify it equals the attested ciphertext_hash.

8.6Message-Size Buckets

Version 1: Only 1 KB and 4 KB buckets are supported. One (pk, vk) pair per bucket; setup is program-agnostic.

8.7Verification Rules

1. Groth16 proof verifies against vk for the declared size_bucket.
2. Poseidon(received_ciphertext) == ciphertext_hash.
3. schema_hash matches the program’s declared schema.
4. Entry size (post-padding) fits within size_bucket.

9.0Networking

9.1ZODE ID

A libp2p PeerId. On the wire and in storage, the raw PeerId bytes are used. For human display, the canonical format is Zx<PeerId_string>. The Zx prefix MUST NOT appear in wire formats.

9.2Topic Naming

Programs use topics of the form prog/<program_id_hex> where program_id_hex is the 64-char lowercase hex encoding of the 32-byte ProgramId. ZODES subscribe to one or more topics and only accept requests for subscribed programs. Services use topics of the form svc/<service_id_hex> for peer discovery.

9.3Connection

The Grid uses libp2p with the following transports:

• QUIC (/quic-v1): Primary transport.
• TCP + Noise + Yamux: Fallback transport.

Default listen address: /ip4/0.0.0.0/udp/3690/quic-v1.

9.4Protocols

9.5GossipSub Configuration

Message authentication: signed (libp2p keypair). Heartbeat interval: 10 s. Validation mode: permissive. Message ID: hash of message data (content-based dedup).

9.6Peer Discovery

ZODES accept bootstrap peer multiaddrs in configuration. On startup, the ZODE dials each bootstrap peer. When Kademlia is enabled, the ZODE seeds the routing table, triggers initial bootstrap(), and periodically performs random walk queries (default: 30 s interval).

10.0Wire Protocol

The sector protocol is the primary client-to-ZODE interface, operating over /grid/sector/1.0.0.

10.1Request Envelope

SectorRequest = Append(SectorAppendRequest)
  | ReadLog(SectorReadLogRequest)
  | LogLength(SectorLogLengthRequest)
  | BatchAppend(SectorBatchAppendRequest)
  | BatchLogLength(SectorBatchLogLengthRequest)

10.2Response Envelope

Response variants correspond 1:1 to request variants.

10.3Append

Request:

Response: ok (boolean), index (optional uint64), error_code (optional).

10.4ReadLog

Request: program_id, sector_id, from_index (uint64), max_entries (uint32, capped at 64). Response: entries (array of byte strings), error_code (optional). Empty array if sector does not exist.

10.5LogLength

Request: program_id, sector_id. Response: length (uint64), error_code (optional).

10.6BatchAppend

Request: program_id, entries (array of BatchAppendEntry, up to 64). Each entry: sector_id, entry, optional shape_proof. Response: results (array of AppendResult: ok, index, error_code).

10.7BatchLogLength

Request: program_id, sector_ids (array of bytes(32), up to 64). Response: results (array of LogLengthResult: length, error_code).

10.8Batch Limits

Maximum 64 entries and 4 MB total payload per batch. ZODES MUST reject with BatchTooLarge.

10.9Error Codes

Error codes are serialized as CBOR text strings.

11.0Gossip Replication

When a ZODE accepts an Append request, it publishes a GossipSectorAppend message to the program’s GossipSub topic. Other subscribed ZODES store the entry automatically.

11.1GossipSectorAppend

11.2Receiving ZODE Behavior

1. Program check: Discard if not serving program_id.
2. Sector filter: Check allowlist if configured.
3. Entry size check: Discard if oversized.
4. Shape proof: Verify if required; discard on failure.
5. Idempotent insert: Store at given index if unoccupied; silently ignore byte-identical duplicates; on conflict, append at next available index.
6. No re-gossip: GossipSub mesh handles fan-out.

12.0Services Layer

The Services layer provides a framework for stateless servers that run on ZODES, use Programs for persistent state, and expose HTTP endpoints. Programs are passive descriptors that define storage schemas; Services are active processes with runtime lifecycle, request handling, and background tasks. A Service never holds authoritative local state; it reads and writes exclusively through Programs, so any ZODE running the same service can serve the same requests without migration or synchronization.

12.1ServiceDescriptor and ServiceId

A CBOR map with the following fields:

The ServiceId is derived as SHA-256(CBOR(ServiceDescriptor)), mirroring the ProgramId pattern (§5.2). Services can both depend on existing Programs and bring their own storage schemas via owned_programs.

12.2Service Trait

Every service, native or WASM, implements four operations:

12.3ServiceContext and ProgramStore

The ServiceContext is a service’s interface to the ZODE. It provides:

• ProgramStore: a key-value abstraction over sector append-logs. Keys map to SectorId via SHA-256(key). put appends a new entry (latest entry = current value). get reads the last entry. list reads the full log history.
• Ephemeral tokens: HMAC-SHA256 signed, time-limited tokens for stateless short-lived flows (auth challenges, OAuth nonces). The client holds the token and presents it back; no server-side storage is required.
• Event broadcast: services emit Started, Stopped, and RequestHandled events.
• Shutdown token: cooperative cancellation for background tasks.

ProgramStore Operations.

12.4ServiceRegistry

The ServiceRegistry manages the lifecycle of all active services on a ZODE:

• Register: compute ServiceId, reject duplicates, store the service.
• Start all: create a ServiceContext per service, call on_start, emit Started event.
• Stop all: cancel background tasks, call on_stop, emit Stopped event.
• Merged router: build a combined HTTP router with each service nested at /services/{service_id_hex}/.
• Required programs: collect the union of all registered services’ required and owned program IDs for topic subscription.

12.5HTTP Integration

Service HTTP routes are merged into the same server that hosts the JSON-RPC sector endpoint. The resulting route layout is:

POST /rpc                           -- JSON-RPC sector operations (§10)
GET|POST /services/{service_id}/... -- per-service endpoints

Each service defines its own sub-routes (e.g. /resolve, /health, /messages). The ZODE handles TLS, CORS, and connection management; services only see deserialized HTTP requests.

12.6WASM Runtime

The Services layer supports a sandboxed WASM runtime for third-party services using the WebAssembly Component Model. A WIT (WebAssembly Interface Types) contract defines the host–guest boundary:

Host-provided imports:

Guest-exported interface:

WASM services are subject to resource limits: CPU fuel metering, memory caps, and per-request wall-clock timeouts. A misbehaving module is terminated without affecting the ZODE or other services.

12.7Standard Services

Two services are registered by default:

The Identity service wraps the ZID Program with a DID resolution HTTP API. The Interlink service exposes channel message retrieval over HTTP. Both are native (compiled) services and are always registered on ZODE startup.

12.8Programs vs. Services

13.0ZODE Behavior

13.1Startup Sequence

1. Open persistent storage.
2. Start libp2p swarm (GossipSub, request-response, Kademlia).
3. Build ServiceRegistry; register default services (Identity, Interlink).
4. Compute effective topic set default programs ∪ configured programs ∪ service-required programs.
5. Subscribe to each GossipSub topic.
6. Start all registered services (on_start); build merged HTTP router.
7. Start RPC server with sector endpoint and service routes merged.
8. Dial bootstrap peers.
9. If Kademlia enabled, seed routing table and bootstrap.
10. Enter event loop.

13.2Event Loop

The ZODE continuously processes: incoming sector requests, gossip messages, peer events, Kademlia discovery, publish queue, service events, and shutdown signals.

13.3Append + Gossip Flow

On Append: (1) validate program, sector filter, entry size, shape proof; (2) append to local storage; (3) send success response. Gossip propagation is client-triggered: after successful append, the client publishes GossipSectorAppend to the program topic.

13.4Shutdown Sequence

On shutdown: (1) cancel service shutdown tokens; (2) call on_stop on each registered service; (3) stop the RPC server; (4) close the libp2p swarm; (5) flush and close persistent storage.

14.0Visibility and Privacy Properties

What a ZODE can see

What a ZODE cannot see

15.0Security Considerations

• No transport anonymity. IP addresses are visible. Onion routing is out of scope.
• Timing correlation. A ZODE can correlate writes/reads from the same connection.
• No wire-level write authorization. Any client with a ProgramId can append. Access control is application-layer.
• Gossip propagation delay. Entries initially exist on one ZODE. Clients MAY multi-send.
• Key compromise. NeuralKey compromise implies all derived keys are compromised. Shamir splitting mitigates single-point-of-failure.
• Post-quantum readiness. Hybrid constructions require breaking both classical and PQ components.
• WASM service isolation. Sandboxed WASM services are subject to fuel metering, memory limits, and wall-clock timeouts. A misbehaving module cannot affect the ZODE or other services, but resource exhaustion within limits is possible.
• Service statefulness. Services are stateless by construction; all persistent state flows through Programs. A compromised service cannot corrupt state beyond what its declared Programs allow.

16.0Interoperability Requirements

A conforming Grid implementation MUST:

1. Serialize all protocol messages as deterministic CBOR (RFC 8949 §4.2.1).
2. Derive ProgramIds as SHA-256(CBOR(ProgramDescriptor)).
3. Use HKDF-SHA256 with exact domain separation strings (§4.2).
4. Produce hybrid signatures Ed25519 (64 B) + ML-DSA-65 (3,309 B); both must verify.
5. Perform hybrid key encapsulation X25519 + ML-KEM-768 via HKDF (§4.5).
6. Use XChaCha20-Poly1305 or Poseidon sponge per ProgramDescriptor.
7. Implement the padding bucket scheme (§7.4).
8. Reject non-32-byte sector IDs.
9. Use /grid/sector/1.0.0 for sector request-response.
10. Use /grid/kad/1.0.0 for Kademlia DHT.
11. Format GossipSub topics as prog/<64_hex_chars>.
12. Serialize GossipSectorAppend as CBOR.
13. Enforce batch limits 64 entries, 4 MB total.
14. Accept gossip with conflict resolution per §11.2.
15. Derive ServiceIds as SHA-256(CBOR(ServiceDescriptor)).
16. Mount service HTTP routes at /services/{service_id_hex}/ alongside the JSON-RPC endpoint.
17. Format service GossipSub topics as svc/<64_hex_chars>.
18. Map ProgramStore keys to SectorIds via SHA-256(key); put appends, get reads last entry.

Appendix AVersion History