Multi-DC

This document describes the multi-datacenter design for GoAkt and how developers can use it. Each datacenter runs an independent GoAkt cluster. There is no global cluster state shared between datacenters. Cross-DC communication is DC-transparent to callers and relies on a DC control plane (registry) for routing.

Goals and Non-Goals

Goals

• Isolation: Each datacenter remains a standalone GoAkt cluster.

• DC-transparent messaging: Callers address actors/grains by logical identity; the runtime discovers which DC contains the target.

• Control plane as source of truth: The DC control plane registry drives routing and placement.

• DC-aware placement: SpawnOn with WithDataCenter spawns actors in a specific datacenter.

• Liveness: Periodic heartbeats keep routing tables accurate.

Non-Goals

• Global cluster membership or consensus across datacenters.

• Transparent global actor identity with shared state.

• Strong consistency across datacenters.

Quick Start

1. Choose a Control Plane

Use etcd or NATS JetStream as the DC registry. One backend can serve both cluster discovery and the DC control plane (with different namespaces/keys per use).

2. Configure the Actor System

import (
    "github.com/tochemey/goakt/v3/actor"
    "github.com/tochemey/goakt/v3/datacenter"
    "github.com/tochemey/goakt/v3/datacenter/controlplane/etcd"
    "github.com/tochemey/goakt/v3/discovery/nats"
)

// DC control plane (Etcd)
etcdCP, err := etcd.NewControlPlane(&etcd.Config{
    Endpoints:   []string{"127.0.0.1:2379"},
    TTL:         30 * time.Second,
    DialTimeout: 5 * time.Second,
    Timeout:     5 * time.Second,
})
if err != nil {
    return err
}

// Datacenter config
dcConfig := datacenter.NewConfig()
dcConfig.ControlPlane = etcdCP
dcConfig.DataCenter = datacenter.DataCenter{
    Name:   "dc-east",
    Region: "us-east-1",
    Zone:   "a",
}
dcConfig.Endpoints = []string{"127.0.0.1:8080"} // remoting host:port

// Cluster config with multi-DC
clusterConfig := actor.NewClusterConfig().
    WithDiscovery(nats.NewDiscovery(...)).
    WithDataCenter(dcConfig).
    WithKinds(&MyActor{}).
    WithGrains(&MyGrain{})

system, err := actor.NewActorSystem("my-system",
    actor.WithLogger(logger),
    actor.WithRemote(remote.NewConfig("127.0.0.1", 8080)),
    actor.WithCluster(clusterConfig),
)

3. Start and Check Readiness

if err := system.Start(ctx); err != nil {
    return err
}

// Optional: wait for DC controller to be ready before cross-DC operations
if !system.DataCenterReady() {
    return errors.New("datacenter controller not ready")
}

// Optional: inspect last cache refresh
lastRefresh := system.DataCenterLastRefresh()

4. Cross-DC Messaging

Send messages to actors/grains without specifying a DC; the runtime discovers the target:

// SendSync/SendAsync work across DCs transparently
pid.SendAsync(ctx, targetPID, msg)
reply, err := pid.SendSync(ctx, targetPID, msg, timeout)

5. Spawn in a Specific Datacenter

targetDC := &datacenter.DataCenter{Name: "dc-west", Region: "us-west-1", Zone: "a"}
err := system.SpawnOn(ctx, "actor-1", NewMyActor(), actor.WithDataCenter(targetDC))

Architecture

+------------------+        register/heartbeat        +----------------------+
| DC A cluster     | <------------------------------> | DC control plane     |
| nodes + leader   |                                  | (Etcd or NATS)       |
+------------------+                                  +----------------------+
        |                                                      |
        | DC-transparent message                               |
        | resolve via registry                                 |
        v                                                      v
+------------------+                                  +------------------+
| DC B cluster     | <------------------------------> | DC C cluster     |
| nodes + leader   |          registry lookup         | nodes + leader   |
+------------------+                                  +------------------+

• DC leader is the sole writer for its DataCenterRecord; followers do not run the controller.

• Control plane stores records, endpoints, and liveness TTLs/leases.

• Routing uses the control plane cache; local DC is tried first, then remote DCs in parallel.

Developer Guide

Endpoints Configuration

datacenter.Config.Endpoints lists remoting addresses (host:port) advertised to other DCs. Other DCs use these to discover and talk to actors/grains in this DC.

• Single endpoint (e.g. leader only): All cross-DC traffic goes to one node.

• Multiple endpoints: Cross-DC spawns and lookups are spread across nodes.

Example:

// Leader-only: all cross-DC traffic to one node
dcConfig.Endpoints = []string{"192.168.1.1:8080"}

// Multiple nodes: spread across cluster
dcConfig.Endpoints = []string{
    "192.168.1.1:8080",
    "192.168.1.2:8080",
    "192.168.1.3:8080",
}

Endpoints must be reachable from other DCs.

Readiness and Cache Refresh

• DataCenterReady(): Returns true when the DC controller is operational and the cache has been refreshed at least once. Use for readiness probes before cross-DC operations.

• DataCenterLastRefresh(): Returns the time of the last successful cache refresh. Zero when multi-DC is disabled or the controller has never refreshed.

FailOnStaleCache

• true (default): Cross-DC operations (e.g. SpawnOn with WithDataCenter) fail with ErrDataCenterStaleRecords if the cache is stale. Favours consistency.

• false: Operations proceed with best-effort routing using a potentially stale cache. Favours availability.

dcConfig := datacenter.NewConfig()
dcConfig.FailOnStaleCache = false // allow stale cache for higher availability

DC-Aware Spawn (SpawnOn + WithDataCenter)

To spawn an actor in a different datacenter:

targetDC := &datacenter.DataCenter{
    Name:   "dc-west",
    Region: "us-west-1",
    Zone:   "a",
}
err := system.SpawnOn(ctx, "actor-1", NewMyActor(), actor.WithDataCenter(targetDC))

The runtime looks up the target DC by DataCenter.ID() (Zone+Region+Name) in the active records, then sends RemoteSpawn to one of that DC's Endpoints chosen at random.

Errors: ErrDataCenterNotReady, ErrDataCenterStaleRecords, ErrDataCenterRecordNotFound.

Control Plane Providers

Provider	Package	Notes
Etcd	datacenter/controlplane/etcd	Supports TLS, auth, watch
NATS JetStream	datacenter/controlplane/nats	Requires NATS with -js (JetStream)

Control Plane

Interface

type ControlPlane interface {
    Register(ctx context.Context, record DataCenterRecord) (id string, version uint64, err error)
    Heartbeat(ctx context.Context, id string, version uint64) (newVersion uint64, leaseExpiry time.Time, err error)
    SetState(ctx context.Context, id string, state DataCenterState, version uint64) (newVersion uint64, err error)
    ListActive(ctx context.Context) ([]DataCenterRecord, error)
    Watch(ctx context.Context) (<-chan ControlPlaneEvent, error)
    Deregister(ctx context.Context, id string) error
}

• Register: Create a DC record; returns ID and version.

• Heartbeat: Renew lease; extends liveness.

• SetState: Transition record state (e.g. ACTIVE → DRAINING on shutdown).

• ListActive: Return ACTIVE records with valid leases for routing.

• Watch: Stream updates (optional; return ErrWatchNotSupported if unsupported).

• Deregister: Remove record on shutdown for faster cleanup than lease expiry.

Data Model

DataCenterRecord:

Field	Description
ID	Stable identifier (Zone+Region+Name or Name)
DataCenter	Name, Region, Zone, Labels
Endpoints	Advertised remoting addresses
State	REGISTERED, ACTIVE, DRAINING, INACTIVE
LeaseExpiry	Liveness lease expiry
Version	Monotonic revision for conflict-free updates

State transitions:

• REGISTERED → ACTIVE: First heartbeat after start.

• ACTIVE → DRAINING: On actor system shutdown.

• ACTIVE/DRAINING → INACTIVE: TTL expiry or explicit disable.

Built-in Implementations

etcd

import "github.com/tochemey/goakt/v3/datacenter/controlplane/etcd"

cp, err := etcd.NewControlPlane(&etcd.Config{
    Endpoints:   []string{"127.0.0.1:2379"},
    Namespace:   "/goakt/datacenters", // optional, default
    TTL:         30 * time.Second,
    DialTimeout: 5 * time.Second,
    Timeout:     5 * time.Second,
    TLS:         nil, // optional
    Username:    "",  // optional
    Password:    "",  // optional
})

NATS JetStream

import "github.com/tochemey/goakt/v3/datacenter/controlplane/nats"

cp, err := nats.NewControlPlane(&nats.Config{
    URL:             "nats://127.0.0.1:4222",
    Bucket:          "goakt_datacenters", // optional, default
    TTL:             30 * time.Second,
    Timeout:         5 * time.Second,
    ConnectTimeout:  5 * time.Second,
})

NATS must be started with JetStream: nats-server -js.

Contract Guarantees

• Register, SetState, and Heartbeat are linearisable per record (CAS by Version).

• Heartbeat extends the lease only when Version is current.

• ListActive returns only ACTIVE records with unexpired leases.

• Watch emits ordered updates (clients de-dupe by Version).

• Deregister removes the record immediately for clean shutdown.

Configuration Reference

Field	Default	Description
ControlPlane	required	etcd or NATS control plane implementation
DataCenter	required	Name, Region, Zone, Labels
Endpoints	required	Remoting addresses (host:port)
HeartbeatInterval	10s	Heartbeat cadence
CacheRefreshInterval	10s	Cache refresh interval
MaxCacheStaleness	30s	Max age before cache is considered stale
LeaderCheckInterval	5s	Leadership recheck interval
RequestTimeout	5s	Control plane operation timeout
WatchEnabled	true	Use Watch when supported
FailOnStaleCache	true	Fail cross-DC ops when cache is stale
JitterRatio	0.1	Jitter for intervals
MaxBackoff	30s	Max backoff on errors

Cluster Integration

Multi-DC is enabled by adding datacenter config to the cluster config:

clusterConfig := actor.NewClusterConfig().
    WithDiscovery(discoveryProvider).
    WithDataCenter(dcConfig).
    WithKinds(...).
    WithGrains(...)

Failure Handling

• Leader failover: The new leader takes over heartbeats.

• Registry outages: Local DC operation continues; cross-DC routing may use cached data.

• Stale cache: Configurable via FailOnStaleCache (strict vs best-effort).

• Control plane unavailable: Does not block local DC; only affects cross-DC operations.

Routing Flow (DC-Transparent Messaging)

1. Runtime checks local datacenter via ActorOf.

2. If not found locally, fetches active DCs from the control plane cache.

3. Queries all active DC endpoints in parallel via RemoteLookup.

4. Uses the first DC that returns a valid actor address.

5. Cancels remaining lookups.

6. Sends the message via remote messaging to the chosen endpoint.

Sequence: DC Registration

1. DC leader starts and registers DataCenterRecord (metadata + endpoints).

2. Leader sends heartbeats at HeartbeatInterval (less than TTL).

3. Registry expires the record if heartbeats stop.

4. On shutdown, leader transitions ACTIVE → DRAINING → INACTIVE and calls Deregister.