Clustering - Bifrost

Overview

Bifrost Clustering delivers production-ready high availability through a peer-to-peer network architecture with automatic service discovery. The clustering system uses gossip protocols to maintain consistent state across nodes while providing seamless scaling, automatic failover, and zero-downtime deployments.

Why Clustering Matters

Modern AI gateway deployments require robust infrastructure to handle production workloads:

Challenge	Impact	Clustering Solution
Single Point of Failure	Complete service outage if gateway fails	Distributed architecture with automatic failover
Traffic Spikes	Performance degradation under high load	Dynamic load distribution across multiple nodes
Provider Rate Limits	Request throttling and service interruption	Distributed rate limit tracking across cluster
Regional Latency	Poor user experience in distant regions	Geographic distribution with local processing
Maintenance Windows	Service downtime during updates	Rolling updates with zero-downtime deployment
Capacity Planning	Over/under-provisioning resources	Elastic scaling based on real-time demand

Core Features

Feature	Description
Automatic Service Discovery	6 discovery methods for any infrastructure (K8s, Consul, etcd, DNS, UDP, mDNS)
Peer-to-Peer Architecture	No single point of failure with equal node participation
Gossip-Based State Sync	Real-time synchronization of traffic patterns and limits
Automatic Failover	Seamless traffic redistribution when nodes fail
Zero-Downtime Updates	Rolling deployments without service interruption

Architecture

Peer-to-Peer Network Design

Bifrost clustering uses a peer-to-peer (P2P) network where all nodes are equal participants. Each node:

Discovers peers automatically using the configured discovery method
Receives application state and counter updates over gRPC
Tracks cluster membership and node liveness over a memberlist gossip layer
Handles failover automatically

Cluster Communication

Bifrost uses two transports for different responsibilities. Membership and node-liveness signals run over a memberlist gossip layer; everything else (configuration changes, governance counters, routing rules, all replicated entity types) travels over a dedicated gRPC channel.

Transport	Default port	Carries
Memberlist gossip	`10101` (TCP + UDP)	Cluster membership, node join/leave, liveness probes, region metadata
gRPC counter sync	`10102` (TCP)	Application messages: governance usage counters, config sync, routing rules, virtual keys, providers, RBAC, MCP tools, pricing, auth config, and 25+ other replicated entity types

This split lets membership churn (joins, leaves, failure detection) stay isolated from the higher-volume application message stream, and lets each transport be tuned, scaled, and observed independently. The gRPC layer was introduced in v1.4.0; before then, all traffic ran over gossip.

Application messages and entity types

Each replicated message carries an EntityType identifying the kind of state being broadcast. Bifrost replicates 30+ entity types across the cluster, including: model catalog, virtual keys, providers, governance counters, routing rules, RBAC, MCP tools and tool groups, pricing and pricing overrides, access profiles, prompt deployments, auth configuration, and cluster diagnostics. See the Replicated Entity Types reference for the complete list.

Message dedup and invalidation

Each broadcast carries a unique message ID and a SentAt timestamp. Receivers run a deduper (default 5-minute TTL) keyed by message ID, so a node that has already processed a given message ignores re-broadcasts of the same ID. When a newer message with the same ID arrives, the existing entry is invalidated and replaced. Convergence: All nodes converge to the same state within seconds with eventual consistency guarantees.

Node Identity and Region

Each node in the cluster has two pieces of identity metadata:

node_id - configured via cluster_config.node_id and surfaced in cluster status output, the React Flow topology view, and diagnostics. The actual memberlist node name is derived from this value combined with the gossip port. If you omit node_id, set one explicitly per pod or instance to make cluster status readable; UUID-style IDs are fine.
region - free-form region label (e.g. "us-east-1", "eu-west") read from cluster_config.region and propagated in node metadata. Defaults to "unknown" when omitted. Region is used for regional leader election and for region-aware operations; it does not gate gossip scope or membership.

Leader Election

Bifrost runs two leader elections in parallel: one cluster-wide and one per region.

Election	Scope	What it does
Cluster leader	All `StateAlive` nodes in the cluster	Coordinates cluster-wide singleton tasks (e.g. pricing URL fetch and broadcast) so only one node hits the upstream and other nodes receive the result via gRPC
Regional leader	Nodes within the same `region` value	Coordinates region-scoped operations

Election is deterministic: the lexicographically-first healthy member wins. The election loop re-evaluates membership every 30 seconds, so leadership transfers automatically when nodes join, leave, or fail. There is nothing to configure - leader election runs whenever clustering is enabled.

Minimum Node Requirements

Recommended: 3+ nodes minimum for optimal fault tolerance.

Cluster Size	Fault Tolerance	Use Case
3 nodes	1 node failure	Small production deployments
5 nodes	2 node failures	Medium production deployments
7+ nodes	3+ node failures	Large enterprise deployments

Configuration Basics

Core Configuration Structure

The new clustering configuration uses a cluster_config object with integrated service discovery:

{
  "cluster_config": {
    "enabled": true,
    "node_id": "bifrost-1",
    "region": "us-east-1",
    "discovery": {
      "enabled": true,
      "type": "kubernetes",
      "service_name": "bifrost-cluster"
    },
    "gossip": {
      "port": 10101,
      "config": {
        "timeout_seconds": 10,
        "success_threshold": 3,
        "failure_threshold": 3
      }
    },
    "grpc": {
      "port": 10102,
      "dial_timeout_seconds": 5
    }
  }
}

Required ports for v1.4.x and later:

10101/TCP and 10101/UDP for memberlist gossip (membership and liveness)
10102/TCP for the gRPC counter sync transport (application messages)

Both ports must be reachable peer-to-peer between cluster nodes. NetworkPolicies, security groups, and firewall rules need to allow traffic on both.

Discovery-specific fields (e.g. k8s_label_selector, consul_address, etcd_endpoints) slot into the discovery object alongside type - see each method’s section below.

At startup, cluster mode requires either a non-empty peers list or discovery.enabled: true.

Common Discovery Configuration Fields

All discovery methods support these common fields:

Field	Type	Required	Description
`enabled`	boolean	No	Enable/disable discovery (must be `true` to use discovery at runtime)
`type`	string	Yes	Discovery type: `kubernetes`, `consul`, `etcd`, `dns`, `udp`, `mdns`
`service_name`	string	Conditional	Required for `consul`, `etcd`, `udp`, and typically `mdns`; optional for `kubernetes` and `dns`
`bind_port`	integer	No	Port for cluster communication (default: 10101)
`dial_timeout`	duration	No	Discovery timeout (default: 10s)
`allowed_address_space`	array	No	CIDR ranges to filter discovered nodes (e.g., `["10.0.0.0/8"]`)

Gossip Configuration

Field	Description	Default
`port`	Memberlist gossip port (used for both TCP and UDP)	10101
`timeout_seconds`	Health check timeout	10
`success_threshold`	Successful checks to mark healthy	3
`failure_threshold`	Failed checks to mark unhealthy	3

gRPC Configuration

The gRPC transport carries all application messages and counter sync between nodes. It is enabled automatically whenever cluster_config.enabled is true; configuration is optional.

Field	Description	Default
`port`	TCP port for the cluster gRPC server	10102
`dial_timeout_seconds`	Timeout when dialing a peer for gRPC	5

If you omit the grpc block entirely, both defaults apply. Override only when the defaults conflict with your environment (e.g. another service already binding 10102).

Top-level Fields

Field	Type	Required	Description
`enabled`	boolean	Yes	Master switch for cluster mode
`node_id`	string	Recommended	Logical identifier for this node, used in cluster status and topology views
`region`	string	No	Region label (e.g. `us-east-1`); defaults to `unknown` and gates regional leader election
`peers`	array	Conditional	Static peer list (`host:port` per entry). Required if `discovery.enabled` is `false`
`gossip`	object	No	Memberlist gossip settings (see above)
`grpc`	object	No	gRPC counter sync settings (see above)
`discovery`	object	Conditional	Auto-discovery settings. Required if `peers` is empty

Broker Mode

The default mesh clustering described above is peer-to-peer: every node must accept inbound gossip and gRPC connections from every other node. Some environments do not allow that. Google Cloud Run, for example, gives each instance only a single inbound serving port, ephemeral instances with no stable addresses, and no instance-to-instance networking - so memberlist gossip and the gRPC mesh cannot form. Broker mode solves this. Instead of connecting to each other, every node makes a single outbound connection to a central broker process. The broker is a pure relay: a message received from one node is fanned out to all other connected nodes. The broker also pushes a roster (the list of connected node IDs) to every node. A message from Node A travels to the broker, which forwards it to Node B and Node C (never back to A). Because nodes only need outbound connectivity, broker mode runs on any platform that can make an outbound gRPC connection.

How it differs from mesh mode

Aspect	Mesh mode	Broker mode
Connectivity	Every node connects to every node	Each node makes one outbound connection to the broker
Membership	memberlist gossip	Roster pushed by the broker
Discovery	6 discovery methods	Not used - the broker is the rendezvous point
Ports on a node	`10101/TCP+UDP`, `10102/TCP` inbound	None - outbound only
Leader election	Deterministic over gossip members	Deterministic over the broker roster (same algorithm)
Entity replication	Over the gRPC mesh	Over the broker relay - identical entity types

Broker-mode synchronization performance

Broker mode separates cluster traffic into dedicated lanes for general messages, heartbeats, governance usage, KV-store updates, circuit-breaker events, load-balancer logs, and diagnostics. This prevents heavy traffic in one lane from blocking a different kind of cluster traffic. However, all messages of the same type addressed to a node share that node’s corresponding broker stream:

Node B ─┐
Node C ─┼──> Node A's governance stream
Node D ─┘

Mesh mode instead has a separate stream between each pair of nodes. As a result, slow processing or heavy traffic from one peer can delay same-lane messages from other peers in broker mode. Broker mode may therefore synchronize more slowly than mesh mode under contention. Use mesh mode when the environment permits direct node-to-node connectivity and maximum synchronization performance is required. Leadership in broker mode uses the same deterministic rule as mesh mode: the lexicographically-smallest node ID in the roster is the leader. Every node computes this independently from the roster the broker pushes, so there is nothing to configure and no broker-side election.

Configuration

Nodes run in broker mode by setting cluster_config.type to broker and pointing at the broker address:

{
  "cluster_config": {
    "enabled": true,
    "type": "broker",
    "region": "us-east-1",
    "broker": {
      "address": "broker.example.run.app:443",
      "tls": true,
      "auth_token": "your-shared-secret"
    }
  }
}

See the config.json cluster reference for the full field list.

Running the broker

The broker is not a separate binary - the same Bifrost Enterprise image runs as the broker when started with the -mode=broker flag (or the BIFROST_MODE=broker environment variable):

bifrost-enterprise -mode=broker -app-dir /app/data

In broker mode the process branches before the normal server bootstrap: it starts only the relay gRPC server and runs no database, providers, plugins, or HTTP gateway. It reads cluster_config.broker from the same config.json and serves on broker.listen_port (default 50051). A standard gRPC health service is registered for readiness probes.

Deploying on Cloud Run

All nodes must connect to the same broker process. Fan-out cannot span multiple broker instances, so the broker must run as a single instance.

Broker service:

Deploy as a Cloud Run service with min-instances=1 and max-instances=1.
Enable HTTP/2 (end-to-end) so gRPC works.
Expose on :443; nodes use the service URL as broker.address with tls: true.
Set the Cloud Run container port to 50051 so it matches cluster_config.broker.listen_port, or override listen_port to 8080 to match Cloud Run’s default $PORT.
Set auth_token so only your nodes can connect.

Node services:

Deploy normally - they only need outbound access to the broker URL.
Set cluster_config.type to broker and broker.address to the broker URL.

Cloud Run caps a single request - including a streaming gRPC connection - at 60 minutes. When the broker stream is closed by the platform, each node automatically reconnects with exponential backoff, so this is transparent. gRPC keepalive pings are enabled on both sides to keep otherwise-idle streams alive within that window.

Roster and reconnection

The broker pushes the roster on three triggers: when a node connects or disconnects, the full roster to a node as its first frame on join, and a periodic rebroadcast (every ~20s) as a safety net for any node that missed an event-driven update. A node that stops receiving roster heartbeats treats the broker as down and enters its reconnect loop.

There is a brief window after a node disconnects where nodes can disagree on the leader until the updated roster lands everywhere - the same eventual-consistency window that gossip has in mesh mode.

Service Discovery Methods

Bifrost supports 6 service discovery methods to fit any infrastructure. Choose based on your deployment environment:

Kubernetes

Native K8s pod discovery via label selectors

Consul

HashiCorp Consul service mesh integration

etcd

etcd-based distributed discovery

DNS

Traditional DNS SRV record discovery

UDP Broadcast

Local network broadcast discovery

mDNS

Multicast DNS for local development

Kubernetes Discovery

Best for: Kubernetes deployments with StatefulSets or Deployments Kubernetes discovery uses the K8s API to automatically discover pods based on label selectors. This is the most common method for cloud-native deployments.

How It Works

Each Bifrost pod queries the Kubernetes API for pods matching the label selector
Discovers pod IPs automatically as pods scale up/down
Works seamlessly with StatefulSets, Deployments, and DaemonSets
No external dependencies required

Configuration

{
  "cluster_config": {
    "enabled": true,
    "discovery": {
      "enabled": true,
      "type": "kubernetes",
      "service_name": "bifrost-cluster",
      "k8s_namespace": "default",
      "k8s_label_selector": "app=bifrost"
    },
    "gossip": {
      "port": 10101,
      "config": {
        "timeout_seconds": 10,
        "success_threshold": 3,
        "failure_threshold": 3
      }
    }
  }
}

Configuration Parameters

Parameter	Required	Description	Example
`k8s_namespace`	No	Kubernetes namespace to search (if empty we pick the “default” as the namespace). If you are using a custom namespace, make sure to provide it here.	`"production"`
`k8s_label_selector`	Yes	Label selector for pod discovery	`"app=bifrost"`, `"app=bifrost,env=prod"`

Kubernetes Deployment Example

StatefulSet
Deployment

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: bifrost
  namespace: default
spec:
  serviceName: bifrost-cluster
  replicas: 3
  selector:
    matchLabels:
      app: bifrost
  template:
    metadata:
      labels:
        app: bifrost
    spec:
      serviceAccountName: bifrost
      containers:
      - name: bifrost
        image: <enterprise_repo_base_url>/bifrost:latest
        ports:
        - containerPort: 8080
          name: http
        - containerPort: 10101
          name: gossip
        - containerPort: 10102
          name: grpc
        volumeMounts:
        - name: config
          mountPath: /etc/bifrost
      volumes:
      - name: config
        configMap:
          name: bifrost-config
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bifrost
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: bifrost-pod-reader
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: bifrost-pod-reader
  namespace: default
subjects:
- kind: ServiceAccount
  name: bifrost
  namespace: default
roleRef:
  kind: Role
  name: bifrost-pod-reader
  apiGroup: rbac.authorization.k8s.io

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bifrost
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: bifrost
  template:
    metadata:
      labels:
        app: bifrost
    spec:
      serviceAccountName: bifrost
      containers:
      - name: bifrost
        image: <enterprise_repo_base_url>/bifrost:latest
        ports:
        - containerPort: 8080
          name: http
        - containerPort: 10101
          name: gossip
        - containerPort: 10102
          name: grpc
        volumeMounts:
        - name: config
          mountPath: /etc/bifrost
      volumes:
      - name: config
        configMap:
          name: bifrost-config
---
apiVersion: v1
kind: Service
metadata:
  name: bifrost-cluster
  namespace: default
spec:
  clusterIP: None
  selector:
    app: bifrost
  ports:
  - port: 10101
    name: gossip
  - port: 10102
    name: grpc

Troubleshooting

Pods not discovering each other

Symptoms: Cluster shows only 1 member, pods running in isolationSolutions:

Verify ServiceAccount has RBAC permissions to list pods
Check label selector matches pod labels exactly
Ensure namespace is correct (defaults to “default”)
Verify gossip port (10101) and gRPC port (10102) are not blocked by NetworkPolicies
Check logs for “error listing pods” messages

Permission denied errors

Symptoms: “error getting kubernetes config” or “forbidden” errorsSolutions:

Create ServiceAccount for Bifrost pods
Create Role with get, list, watch permissions on pods
Create RoleBinding linking ServiceAccount to Role
Verify RBAC is enabled in cluster

Cluster forms but nodes show as unhealthy

Symptoms: Nodes discovered but marked as “suspect” or “dead”Solutions:

Verify gossip port (10101) and gRPC port (10102) are accessible between pods
Check for NetworkPolicies blocking pod-to-pod communication
Increase timeout_seconds in gossip config if network is slow
Verify pods are in Running state with kubectl get pods

Consul Discovery

Best for: Consul service mesh environments, and environments where the Consul agent lives in a non-default datacenter, namespace, or partition (set via standard CONSUL_* environment variables - see Consul Authentication and TLS) Consul discovery integrates with HashiCorp Consul for service registration and discovery. Ideal for environments already using Consul for service mesh or service discovery.

Consul is external infrastructure - it is not bundled with the Bifrost image and Bifrost does not run or manage a Consul agent for you. You need a reachable Consul deployment (your own, or one your organization already operates) before enabling this discovery type. Bifrost acts only as a Consul client: each node registers itself with, and queries, the Consul agent at consul_address.

How It Works

On startup, each Bifrost node registers itself with Consul as a service instance (ID <service_name>-<hostname>-<gossip_port>, tagged bifrost, cluster), advertising its own address and gossip port.
Consul attaches a TCP health check to that registration, dialing the node’s gossip port every 10s with a 5s timeout.
About 1 second after registering, the node queries Consul’s catalog for all healthy instances of the same service_name and joins them via gossip.
Discovery keeps running in the background on an adaptive interval - starting at 1 minute, doubling (up to a 30-minute ceiling) each time a poll finds no new peers, and resetting straight back to 1 minute if cluster health degrades or the member count drops. A node with no peers yet always retries at the 1-minute floor rather than backing off, so it converges quickly once others appear.
If a node’s check goes critical (e.g. it crashes or is network-partitioned), Consul removes it from the catalog automatically 30 seconds later. A node that shuts down gracefully deregisters itself immediately instead of waiting out that window.

Configuration

{
  "cluster_config": {
    "enabled": true,
    "discovery": {
      "enabled": true,
      "type": "consul",
      "service_name": "bifrost-cluster",
      "consul_address": "consul.service.consul:8500"
    },
    "gossip": {
      "port": 10101,
      "config": {
        "timeout_seconds": 10,
        "success_threshold": 3,
        "failure_threshold": 3
      }
    }
  }
}

Configuration Parameters

Parameter	Required	Description	Example
`consul_address`	No	Consul agent address	`"localhost:8500"`, `"consul.service.consul:8500"` (default: `localhost:8500`)

Consul discovery automatically registers each node with a health check. The check is a TCP dial against the node’s gossip port only (interval 10s, timeout 5s) - it does not check the gRPC port. A node can show healthy in Consul while still failing to form the gRPC application-state mesh; if the cluster looks unhealthy in the topology view despite Consul showing all nodes passing, check gRPC (10102) connectivity separately.

Consul Authentication and TLS

consul_address is the only Consul-specific field in cluster_config.discovery - there is no config.json field for an ACL token, TLS client certificates, namespace, or datacenter. Bifrost builds its Consul client from the standard Consul Go client’s default configuration, which reads these values directly from the process environment - so set the corresponding standard Consul environment variable on the Bifrost process to reach a Consul agent that requires ACLs, TLS, or a non-default namespace/datacenter. These are the same variables the Consul CLI and other official Consul clients read:

Environment variable	Purpose
`CONSUL_HTTP_TOKEN`	ACL token sent with every request
`CONSUL_HTTP_TOKEN_FILE`	Path to a file containing the ACL token
`CONSUL_HTTP_SSL`	Set to `true` to connect over HTTPS
`CONSUL_HTTP_SSL_VERIFY`	Set to `false` to skip TLS certificate verification (not recommended in production)
`CONSUL_CACERT`	Path to the CA certificate file used to verify the Consul agent’s TLS certificate
`CONSUL_CAPATH`	Path to a directory of CA certificates
`CONSUL_CLIENT_CERT` / `CONSUL_CLIENT_KEY`	Client certificate/key for mutual TLS
`CONSUL_TLS_SERVER_NAME`	Server name used for TLS verification (SNI) when it differs from the address
`CONSUL_HTTP_AUTH`	HTTP basic auth credentials, in `username:password` form
`CONSUL_NAMESPACE`	Consul Enterprise namespace to register and query against
`CONSUL_PARTITION`	Consul Enterprise admin partition
`CONSUL_DATACENTER`	Datacenter to target

Set these on every node that uses discovery.type: "consul" - each node registers itself and queries Consul independently, so a missing token or CA file on one node causes that node’s registration or discovery to fail even if the others are configured correctly.

consul_address in config.json takes precedence over CONSUL_HTTP_ADDR - if consul_address is set (non-empty), Bifrost uses it and ignores CONSUL_HTTP_ADDR. CONSUL_HTTP_ADDR only takes effect when consul_address is omitted from config.json. All other settings (token, TLS, namespace, partition, datacenter) are read exclusively from their environment variables - there’s no config.json equivalent to override them.

Docker Compose with Consul

This example runs Consul with agent -dev, which disables ACLs and TLS and keeps all state in memory. It’s fine for trying discovery out locally, but is not a production Consul deployment. A production setup needs a real multi-server Consul cluster with ACLs and TLS enabled, and the corresponding CONSUL_* environment variables set on each Bifrost node - see Consul Authentication and TLS.

version: '3.8'

services:
  consul:
    image: hashicorp/consul:latest
    command: agent -dev -client=0.0.0.0
    ports:
      - "8500:8500"
    networks:
      - bifrost-net

  bifrost-1:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config-node1.json:/etc/bifrost/config.json
    ports:
      - "8080:8080"
    depends_on:
      - consul
    networks:
      - bifrost-net

  bifrost-2:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config-node2.json:/etc/bifrost/config.json
    ports:
      - "8081:8080"
    depends_on:
      - consul
    networks:
      - bifrost-net

  bifrost-3:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config-node3.json:/etc/bifrost/config.json
    ports:
      - "8082:8080"
    depends_on:
      - consul
    networks:
      - bifrost-net

networks:
  bifrost-net:
    driver: bridge

Troubleshooting

Failed to register with Consul

Symptoms: “failed to register service with Consul” errorsSolutions:

Verify Consul agent is accessible at configured address
Check Consul agent logs for registration errors
Ensure CONSUL_HTTP_TOKEN (or CONSUL_HTTP_TOKEN_FILE) is set on the Bifrost process to a token with write permissions if ACLs are enabled - see Consul Authentication and TLS
Verify network connectivity between Bifrost and Consul
Check firewall rules allow connections to port 8500

Services registered but not discovered

Symptoms: Consul UI shows services but nodes don’t join clusterSolutions:

Verify service_name matches across all nodes - a mismatch means nodes register into different catalog entries and never see each other
Check Consul service health checks are passing (the query only returns healthy instances)
Ensure the gossip port (10101) is reachable between nodes - this is what both Consul’s health check and the actual gossip join use
Verify CONSUL_DATACENTER (and CONSUL_NAMESPACE/CONSUL_PARTITION on Consul Enterprise) is either unset or set to the same value on every node - Bifrost’s Consul client scopes both registration and catalog queries to whatever these env vars resolve to, so a node with a different value registers into, and queries, an entirely different partition of the catalog
Check for DNS resolution issues if using service DNS names
If nodes only just started, allow up to a minute - discovery polls on an adaptive interval starting at 1 minute, not continuously

Health checks failing

Symptoms: Services show as critical in Consul UISolutions:

The check only dials the node’s gossip port (10101) over TCP - gRPC (10102) reachability does not affect this check, so don’t spend time on gRPC connectivity for this specific symptom
Check Consul agent can reach node’s gossip port (10101) - firewalls/NetworkPolicies/security groups between the Consul agent and Bifrost nodes are the usual cause
The check uses a 10s interval and 5s timeout by default; a node that’s healthy but briefly unresponsive under load can flap critical - review Bifrost logs for startup or resource-pressure errors around the same time
Ensure nodes have correct IP addresses registered (check service.Address/service.ServiceAddress in Consul’s catalog against the node’s actual reachable IP)
Remember a critical check auto-deregisters the node after 30 seconds - if a node keeps reappearing and disappearing, look for the underlying cause (network flakiness, resource exhaustion) rather than the deregistration itself, which is expected behavior

etcd Discovery

Best for: etcd-based distributed systems, existing etcd infrastructure etcd discovery uses etcd’s distributed key-value store for service registration and discovery. Perfect for environments already using etcd or requiring strong consistency.

How It Works

Each Bifrost node registers itself in etcd with a lease
Nodes maintain lease through keepalive messages
Nodes query etcd prefix to discover other instances
Failed nodes’ leases expire and are automatically removed
Provides strongly consistent service registry

Configuration

{
  "cluster_config": {
    "enabled": true,
    "discovery": {
      "enabled": true,
      "type": "etcd",
      "service_name": "bifrost-cluster",
      "etcd_endpoints": [
        "http://etcd-1:2379",
        "http://etcd-2:2379",
        "http://etcd-3:2379"
      ],
      "dial_timeout": "10s"
    },
    "gossip": {
      "port": 10101,
      "config": {
        "timeout_seconds": 10,
        "success_threshold": 3,
        "failure_threshold": 3
      }
    }
  }
}

Configuration Parameters

Parameter	Required	Description	Example
`etcd_endpoints`	Yes	Array of etcd endpoint URLs	`["http://localhost:2379"]`, `["https://etcd1:2379", "https://etcd2:2379"]`
`dial_timeout`	No	Connection timeout	`"10s"` (default), `"30s"`

Each node registers under /services/{service_name}/{node_id} with a 30-second TTL lease.

Docker Compose with etcd

version: '3.8'

services:
  etcd:
    image: quay.io/coreos/etcd:latest
    command:
      - etcd
      - --advertise-client-urls=http://etcd:2379
      - --listen-client-urls=http://0.0.0.0:2379
      - --listen-peer-urls=http://0.0.0.0:2380
      - --initial-cluster=etcd=http://etcd:2380
      - --initial-advertise-peer-urls=http://etcd:2380
    ports:
      - "2379:2379"
      - "2380:2380"
    networks:
      - bifrost-net

  bifrost-1:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8080:8080"
    depends_on:
      - etcd
    networks:
      - bifrost-net

  bifrost-2:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8081:8080"
    depends_on:
      - etcd
    networks:
      - bifrost-net

  bifrost-3:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8082:8080"
    depends_on:
      - etcd
    networks:
      - bifrost-net

networks:
  bifrost-net:
    driver: bridge

Troubleshooting

Failed to create etcd client

Symptoms: “etcd client error” on startupSolutions:

Verify etcd endpoints are accessible
Check URL format (http:// or https://)
Ensure etcd cluster is healthy and running
Verify network connectivity to etcd endpoints
Check firewall rules allow connections to port 2379
Increase dial_timeout if network is slow

Failed to register with etcd

Symptoms: “failed to register with etcd” errorsSolutions:

Verify etcd cluster is accepting writes
Check etcd cluster has available space
Ensure authentication credentials if etcd has auth enabled
Review etcd logs for permission or quota errors
Verify node can resolve etcd hostnames

Lease keepalive failures

Symptoms: Nodes repeatedly registering/deregisteringSolutions:

Check network stability between nodes and etcd
Verify etcd cluster is not overloaded
Monitor etcd metrics for high latency
Increase lease TTL if network has high latency
Check for etcd leader election issues

DNS Discovery

Best for: Traditional infrastructure, static node addresses, cloud DNS services DNS discovery uses standard DNS resolution to discover cluster nodes. Works with any DNS server and is ideal for static deployments or cloud environments with DNS integration.

How It Works

Configure DNS A records or SRV records for cluster nodes
Bifrost queries DNS to resolve configured names
All returned IP addresses are treated as potential cluster members
Supports multiple DNS names for different node groups
Works with internal DNS, cloud DNS, or public DNS

Configuration

{
  "cluster_config": {
    "enabled": true,
    "discovery": {
      "enabled": true,
      "type": "dns",
      "service_name": "bifrost-cluster",
      "dns_names": [
        "bifrost-cluster.local",
        "bifrost-nodes.internal.company.com"
      ],
      "bind_port": 10101
    },
    "gossip": {
      "port": 10101,
      "config": {
        "timeout_seconds": 10,
        "success_threshold": 3,
        "failure_threshold": 3
      }
    }
  }
}

Configuration Parameters

Parameter	Required	Description	Example
`dns_names`	Yes	Array of DNS names to resolve	`["bifrost.local"]`, `["node1.local", "node2.local", "node3.local"]`
`bind_port`	No	Port appended to discovered IPs	`10101` (default)

DNS discovery is passive - it doesn’t register nodes. You must manage DNS records externally (via DNS server, cloud DNS, or Kubernetes DNS).

Setup Examples

Cloud DNS (AWS Route53)
Kubernetes Headless Service
Local DNS (dnsmasq)

# Create A records for each node
aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890ABC \
  --change-batch '{
    "Changes": [{
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "bifrost-cluster.internal.company.com",
        "Type": "A",
        "TTL": 60,
        "ResourceRecords": [
          {"Value": "10.0.1.10"},
          {"Value": "10.0.1.11"},
          {"Value": "10.0.1.12"}
        ]
      }
    }]
  }'

apiVersion: v1
kind: Service
metadata:
  name: bifrost-cluster
  namespace: default
spec:
  clusterIP: None  # Headless service
  selector:
    app: bifrost
  ports:
  - port: 10101
    name: gossip
---
# DNS will resolve bifrost-cluster.default.svc.cluster.local
# to all pod IPs matching the selector

# /etc/dnsmasq.conf
address=/bifrost-cluster.local/192.168.1.10
address=/bifrost-cluster.local/192.168.1.11
address=/bifrost-cluster.local/192.168.1.12

# Or use /etc/hosts on each node
echo "192.168.1.10 node1.bifrost.local" >> /etc/hosts
echo "192.168.1.11 node2.bifrost.local" >> /etc/hosts
echo "192.168.1.12 node3.bifrost.local" >> /etc/hosts

Troubleshooting

DNS lookup errors

Symptoms: “dns lookup error” in logs, no nodes discoveredSolutions:

Verify DNS names are resolvable: nslookup bifrost-cluster.local
Check DNS server is accessible from Bifrost nodes
Verify /etc/resolv.conf has correct nameserver
Test DNS resolution from inside container if using Docker
Check for DNS caching issues (try flushing DNS cache)

No nodes discovered via DNS

Symptoms: DNS resolves but cluster has 0 membersSolutions:

Verify DNS returns multiple A records (not CNAME)
Check that returned IPs are correct and reachable
Ensure bind_port matches actual gossip port on nodes
Verify nodes are listening on returned IP addresses
Use dig or nslookup to verify DNS response format

Nodes discovered but can't connect

Symptoms: IPs discovered but gossip connection failsSolutions:

Verify gossip port (10101) and gRPC port (10102) are open on all nodes
Check firewall rules between nodes
Ensure nodes are listening on correct network interface
Verify IP addresses match node’s actual network addresses
Test connectivity: telnet <ip> 10101

UDP Broadcast Discovery

Best for: Local network deployments, on-premise infrastructure, development clusters UDP broadcast discovery automatically finds nodes on the same local network using broadcast packets. No external dependencies required.

How It Works

Nodes broadcast UDP discovery beacons on configured port
Other nodes on the same network respond with acknowledgments
Nodes discover each other’s IP addresses automatically
Limited to nodes on the same broadcast domain (subnet)
Requires allowed_address_space for security

Configuration

{
  "cluster_config": {
    "enabled": true,
    "discovery": {
      "enabled": true,
      "type": "udp",
      "service_name": "bifrost-cluster",
      "udp_broadcast_port": 9999,
      "allowed_address_space": [
        "192.168.1.0/24",
        "10.0.0.0/8"
      ],
      "dial_timeout": "10s"
    },
    "gossip": {
      "port": 10101,
      "config": {
        "timeout_seconds": 10,
        "success_threshold": 3,
        "failure_threshold": 3
      }
    }
  }
}

Configuration Parameters

Parameter	Required	Description	Example
`udp_broadcast_port`	Yes	Port for broadcast discovery	`9999`, `8888`
`allowed_address_space`	Yes	CIDR ranges to limit discovery scope	`["192.168.1.0/24"]`, `["10.0.0.0/8", "172.16.0.0/12"]`
`dial_timeout`	No	Time to wait for responses	`"10s"` (default)

UDP broadcast discovery requires allowed_address_space to be configured. This prevents scanning arbitrary networks and limits discovery to trusted subnets.

Docker Compose Example

version: '3.8'

services:
  bifrost-1:
    image: <enterprise_repo_base_url>/bifrost:latest
    network_mode: bridge
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8080:8080"
      - "9999:9999/udp"
      - "10101:10101"
      - "10102:10102"

  bifrost-2:
    image: <enterprise_repo_base_url>/bifrost:latest
    network_mode: bridge
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8081:8080"
      - "9999:9999/udp"
      - "10101:10101"
      - "10102:10102"

  bifrost-3:
    image: <enterprise_repo_base_url>/bifrost:latest
    network_mode: bridge
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8082:8080"
      - "9999:9999/udp"
      - "10101:10101"
      - "10102:10102"

Use network_mode: bridge (default) or host for UDP broadcast. Custom networks may not support broadcast.

Troubleshooting

No nodes discovered via UDP broadcast

Symptoms: Discovery runs but finds 0 nodesSolutions:

Verify allowed_address_space includes node IP addresses
Check UDP broadcast port is open (firewall/security groups)
Ensure nodes are on same subnet/broadcast domain
Verify broadcast is enabled on network interface
Test with tcpdump -i any -n udp port 9999
Check Docker network mode supports broadcast (use bridge or host)

Address space filtering issues

Symptoms: “not in allowed address space” warningsSolutions:

Verify CIDR notation is correct (e.g., 192.168.1.0/24)
Ensure allowed_address_space covers all node IPs
Check node IP addresses: ip addr or ifconfig
Remember to use network address, not host address
Test CIDR match online or with ipcalc

Permission denied on UDP port

Symptoms: “permission denied” or “address already in use”Solutions:

Check if another process is using the UDP broadcast port
Verify port number is > 1024 (non-privileged) or run as root
Use netstat -tulpn | grep 9999 to check port usage
Change udp_broadcast_port to different value
Ensure firewall isn’t blocking UDP on that port

mDNS Discovery

Best for: Local development, testing, zero-configuration setups mDNS (Multicast DNS) provides zero-configuration service discovery on local networks. Perfect for development and testing without requiring any infrastructure setup.

How It Works

Nodes advertise themselves via mDNS (Bonjour/Avahi)
Other nodes browse for mDNS services
Automatic discovery within the same local network
No DNS server or configuration required
Limited to local network segment

Configuration

{
  "cluster_config": {
    "enabled": true,
    "discovery": {
      "enabled": true,
      "type": "mdns",
      "service_name": "bifrost",
      "mdns_service": "_bifrost._tcp",
      "dial_timeout": "10s"
    },
    "gossip": {
      "port": 10101,
      "config": {
        "timeout_seconds": 10,
        "success_threshold": 3,
        "failure_threshold": 3
      }
    }
  }
}

Configuration Parameters

Parameter	Required	Description	Example
`mdns_service`	No	mDNS service type	`"_bifrost._tcp"` (default), `"_myapp._tcp"`
`dial_timeout`	No	Time to wait for mDNS responses	`"10s"` (default)

mDNS is designed for development and testing. For production, use Kubernetes, Consul, or etcd discovery.

Local Development Example

# Start first node
docker run -p 8080:8080 -p 10101:10101 -p 10201:10102 \
  -v $(pwd)/config-mdns.json:/etc/bifrost/config.json \
  <enterprise_repo_base_url>/bifrost:latest

# Start second node (discovers first automatically)
docker run -p 8081:8080 -p 10111:10101 -p 10211:10102 \
  -v $(pwd)/config-mdns.json:/etc/bifrost/config.json \
  <enterprise_repo_base_url>/bifrost:latest

# Start third node (discovers both automatically)
docker run -p 8082:8080 -p 10121:10101 -p 10221:10102 \
  -v $(pwd)/config-mdns.json:/etc/bifrost/config.json \
  <enterprise_repo_base_url>/bifrost:latest

Troubleshooting

mDNS services not discovered

Symptoms: Nodes don’t discover each other via mDNSSolutions:

Verify mDNS is enabled on network (check firewall)
Ensure multicast is enabled on network interface
Check nodes are on same local network segment
Verify mDNS port 5353 is not blocked
Test mDNS resolution: avahi-browse -a (Linux) or dns-sd -B (macOS)
Increase dial_timeout if discovery is slow

Network address validation errors

Symptoms: “skipping invalid host address” warningsSolutions:

This is normal - mDNS returns network/broadcast addresses
mDNS automatically filters invalid addresses (127.x.x.x, *.0, *.255)
Check that nodes have valid non-loopback IP addresses
Ensure nodes are not using 127.0.0.1 for binding
Verify network interface has proper IP configuration

Discovery works but cluster unstable

Symptoms: Nodes discover then disconnect repeatedlySolutions:

mDNS has eventual consistency, allow time for propagation
Check gossip port accessibility between nodes
Verify network doesn’t drop multicast packets
Consider using a more robust discovery method for production
Check for network congestion or packet loss

Deployment Patterns

Docker Compose Deployment

Complete example using Kubernetes-style discovery with a shared config store:

version: '3.8'

services:
  postgres:
    image: postgres:14
    environment:
      POSTGRES_DB: bifrost
      POSTGRES_USER: bifrost
      POSTGRES_PASSWORD: bifrost_password
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - bifrost-net

  consul:
    image: hashicorp/consul:latest
    command: agent -dev -client=0.0.0.0
    ports:
      - "8500:8500"
    networks:
      - bifrost-net

  bifrost-1:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8080:8080"
    depends_on:
      - postgres
      - consul
    networks:
      - bifrost-net

  bifrost-2:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8081:8080"
    depends_on:
      - postgres
      - consul
    networks:
      - bifrost-net

  bifrost-3:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8082:8080"
    depends_on:
      - postgres
      - consul
    networks:
      - bifrost-net

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - bifrost-1
      - bifrost-2
      - bifrost-3
    networks:
      - bifrost-net

volumes:
  postgres_data:

networks:
  bifrost-net:
    driver: bridge

nginx.conf for load balancing:

events {
    worker_connections 1024;
}

http {
    upstream bifrost_cluster {
        least_conn;
        server bifrost-1:8080 max_fails=3 fail_timeout=30s;
        server bifrost-2:8080 max_fails=3 fail_timeout=30s;
        server bifrost-3:8080 max_fails=3 fail_timeout=30s;
    }

    server {
        listen 80;
        
        location / {
            proxy_pass http://bifrost_cluster;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            
            # Timeouts
            proxy_connect_timeout 60s;
            proxy_send_timeout 60s;
            proxy_read_timeout 60s;
        }
        
        location /health {
            access_log off;
            return 200 "healthy\n";
            add_header Content-Type text/plain;
        }
    }
}

Kubernetes Production Deployment

Production-ready Kubernetes deployment with StatefulSet:

If you use PostgreSQL for config_store, ensure the target database is UTF8 encoded. See PostgreSQL UTF8 Requirement.

apiVersion: v1
kind: ConfigMap
metadata:
  name: bifrost-config
  namespace: bifrost
data:
  config.json: |
    {
      "cluster_config": {
        "enabled": true,
        "discovery": {
          "enabled": true,
          "type": "kubernetes",
          "service_name": "bifrost-cluster",
          "k8s_namespace": "bifrost",
          "k8s_label_selector": "app=bifrost,component=gateway"
        },
        "gossip": {
          "port": 10101,
          "config": {
            "timeout_seconds": 10,
            "success_threshold": 3,
            "failure_threshold": 3
          }
        }
      },
      "config_store": {
        "enabled": true,
        "type": "postgres",
        "config": {
          "host": "postgres.bifrost.svc.cluster.local",
          "port": "5432",
          "user": "bifrost",
          "password": "changeme",
          "db_name": "bifrost",
          "ssl_mode": "require"
        }
      }
    }
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bifrost
  namespace: bifrost
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: bifrost-pod-reader
  namespace: bifrost
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: bifrost-pod-reader
  namespace: bifrost
subjects:
- kind: ServiceAccount
  name: bifrost
  namespace: bifrost
roleRef:
  kind: Role
  name: bifrost-pod-reader
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: bifrost
  namespace: bifrost
spec:
  serviceName: bifrost-cluster
  replicas: 3
  selector:
    matchLabels:
      app: bifrost
      component: gateway
  template:
    metadata:
      labels:
        app: bifrost
        component: gateway
    spec:
      serviceAccountName: bifrost
      containers:
      - name: bifrost
        image: <enterprise_repo_base_url>/bifrost:latest
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        - containerPort: 10101
          name: gossip
          protocol: TCP
        - containerPort: 10102
          name: grpc
          protocol: TCP
        env:
        - name: BIFROST_CONFIG
          value: /etc/bifrost/config.json
        volumeMounts:
        - name: config
          mountPath: /etc/bifrost
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            cpu: "2000m"
            memory: "2Gi"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
      volumes:
      - name: config
        configMap:
          name: bifrost-config
---
apiVersion: v1
kind: Service
metadata:
  name: bifrost-cluster
  namespace: bifrost
spec:
  clusterIP: None
  selector:
    app: bifrost
    component: gateway
  ports:
  - port: 10101
    name: gossip
    protocol: TCP
  - port: 10102
    name: grpc
    protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: bifrost
  namespace: bifrost
spec:
  type: LoadBalancer
  selector:
    app: bifrost
    component: gateway
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: bifrost-pdb
  namespace: bifrost
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: bifrost
      component: gateway

Bare Metal / VM Deployment

For bare metal or VM deployments using systemd: Step 1: Install Bifrost on each node

# Download Bifrost Enterprise binary
curl -O https://releases.getmaxim.ai/bifrost-enterprise/latest/bifrost-enterprise-linux-amd64
chmod +x bifrost-enterprise-linux-amd64
sudo mv bifrost-enterprise-linux-amd64 /usr/local/bin/bifrost-enterprise

Step 2: Create configuration file

sudo mkdir -p /etc/bifrost
sudo cat > /etc/bifrost/config.json <<EOF
{
  "cluster_config": {
    "enabled": true,
    "discovery": {
      "enabled": true,
      "type": "dns",
      "service_name": "bifrost-cluster",
      "dns_names": ["bifrost-cluster.internal.company.com"]
    },
    "gossip": {
      "port": 10101,
      "config": {
        "timeout_seconds": 10,
        "success_threshold": 3,
        "failure_threshold": 3
      }
    }
  },
  "config_store": {
    "enabled": true,
    "type": "postgres",
    "config": {
      "host": "postgres.internal.company.com",
      "port": "5432",
      "user": "bifrost",
      "password": "secure_password",
      "db_name": "bifrost",
      "ssl_mode": "require"
    }
  }
}
EOF

Step 3: Create systemd service

sudo cat > /etc/systemd/system/bifrost.service <<EOF
[Unit]
Description=Bifrost Enterprise API Gateway
After=network.target

[Service]
Type=simple
User=bifrost
Group=bifrost
Environment="BIFROST_CONFIG=/etc/bifrost/config.json"
ExecStart=/usr/local/bin/bifrost-enterprise
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/bifrost

[Install]
WantedBy=multi-user.target
EOF

Step 4: Setup DNS records

# Add A records for bifrost-cluster.internal.company.com
# pointing to all node IPs:
# 10.0.1.10  (node1)
# 10.0.1.11  (node2)
# 10.0.1.12  (node3)

Step 5: Start and enable service

sudo useradd -r -s /bin/false bifrost
sudo mkdir -p /var/lib/bifrost
sudo chown bifrost:bifrost /var/lib/bifrost
sudo systemctl daemon-reload
sudo systemctl enable bifrost
sudo systemctl start bifrost
sudo systemctl status bifrost

Step 6: Verify cluster formation

# Check logs on each node
sudo journalctl -u bifrost -f

# Look for messages like:
# "successfully joined X peers on startup"
# "cluster health: HEALTHY"

Cluster Operations

Cluster Topology View

The admin UI ships an interactive cluster graph (React Flow) that renders the live cluster as a circle of nodes with edges colored by reachability. Each node card shows its node_id, region, current state (alive / suspect / dead / left), and a leader badge if it currently holds either the cluster-wide or regional leadership. The view updates in the background and triggers an automatic diagnostic on leader transitions. Single-node clusters render the simplified single-card view.

Cluster Diagnostics

A diagnostic flow lets administrators verify in-cluster reachability end-to-end. When triggered, the local node broadcasts a no-op diagnostic ping to every peer that advertises the ack:v1 capability and streams ACK status back to the caller as each peer responds. Use it to:

Confirm that a newly added node is receiving and acknowledging cluster messages
Surface peers that are partitioned or silently dropping traffic
Validate that NetworkPolicies / firewall rules permit gRPC after a config change

The diagnostic does not mutate state on any node - the message is purely a round-trip probe. It is admin-only and is invoked from the cluster topology view or the equivalent admin endpoint.

Mixed-Version Rollouts

Bifrost negotiates per-peer capabilities so newer and older cluster members can run side-by-side during rolling upgrades. Peers advertise the ack:v1 capability in their gossip metadata; a node only tracks ACKs from peers that advertise it. Older peers still receive broadcasts but are excluded from the pending-ACK set so they don’t trigger false “unacked” metrics or unnecessary retries. In practice this means: you can roll out a new Bifrost version one pod at a time, and the cluster will degrade ACK tracking gracefully for the older pods until they’re replaced. Rolling-update strategies (Kubernetes RollingUpdate, Nomad canaries, etc.) are supported without quorum loss.

Leader-Coordinated Tasks

A small number of cluster-wide tasks run only on the elected leader, with results broadcast over gRPC so followers stay in sync without each one independently doing the work. The clearest example is pricing sync: only the leader fetches the upstream pricing URL on the configured interval and then broadcasts a database reload message; followers reload from the local store rather than each making the upstream HTTP call. Region-scoped variants of this pattern run on the regional leader. There is no configuration to enable this - leader-coordinated tasks engage automatically when clustering is enabled. If the leader fails, the next election (within ~30 seconds) hands the responsibility to a new node.

Replicated Entity Types

Every replicated message carries an EntityType identifying the kind of state being broadcast. The cluster replicates the following entity types over gRPC:

Category	Entity Types
Catalog & Providers	`model_catalog`, `provider`, `model_config`, `pricing`, `pricing_override`
Governance	`governance`, `virtual_key`, `team`, `customer`, `business_unit_governance`, `user_governance`, `access_profile`, `api_key`
Security & Auth	`auth_config`, `rbac`, `proxy_config`, `client_config`
Routing & Load Balancing	`routing_rule`, `load_balancing`, `load_balancing_log`
MCP	`mcp_tool`, `mcp_tool_group`
Guardrails	`guardrail`, `guardrail_config`, `guardrail_sampling`
Plugins & Connectors	`plugin`, `observability_connector`
Prompts & Storage	`prompt_deployment`, `kv_store`
Cluster	`cluster_diagnostic` (no-op probe used by the diagnostic flow)

Each entity type follows the same broadcast and dedup rules: a unique message ID, a SentAt timestamp, and a deduper TTL on the receiver side. Newer messages with the same ID invalidate older ones, so a late-arriving stale message will not overwrite fresh state.

Troubleshooting

General Clustering Issues

Cluster forms but only has 1 member

Symptoms: Each node thinks it’s the only memberCommon Causes & Solutions:

Discovery not configured: Verify discovery.enabled: true and discovery.type is set
Service name mismatch: Ensure all nodes have identical service_name
Gossip port blocked: Check firewall allows TCP port 10101 between nodes
Discovery method issues: See method-specific troubleshooting above
Network isolation: Verify nodes can reach each other on gossip port

Split brain - nodes form separate clusters

Symptoms: Nodes divided into separate clustersCommon Causes & Solutions:

Network partition: Check network connectivity between all nodes
Different discovery configs: Ensure all nodes use same discovery settings
Firewall blocking gossip: Verify bidirectional connectivity on port 10101
Discovery scoped incorrectly: Check label selectors, DNS names, or address spaces
Restart all nodes: Sometimes requires simultaneous restart to reform cluster

High memory usage in cluster

Symptoms: Memory grows over time, especially in large clustersCommon Causes & Solutions:

Large gossip messages: Check size of gossiped data
Too many nodes: Optimize for clusters with 3-7 nodes typically
Message deduplication cache: This is normal, cache TTL is 2 minutes
Increase node resources: Ensure adequate memory allocation

Cluster unstable - nodes flapping

Symptoms: Nodes repeatedly join and leave clusterCommon Causes & Solutions:

Network instability: Check for packet loss or high latency
Resource constraints: Ensure nodes have adequate CPU/memory
Timeout too aggressive: Increase timeout_seconds in gossip config
Health check failures: Review liveness probe configuration
Discovery intervals: Check discovery isn’t running too frequently

Cannot broadcast messages to cluster

Symptoms: Broadcast queue errors, messages not propagatingCommon Causes & Solutions:

Queue not initialized: Check logs for initialization errors
No active members: Verify cluster has multiple healthy members
Gossip port unreachable: Test connectivity between all nodes
Message too large: Check size of broadcast messages

Key log messages to look for:

✅ Successful cluster formation:
- "successfully joined X peers on startup"
- "cluster health: HEALTHY"
- "discovered X nodes"

⚠️ Warning signs:
- "no new nodes discovered"
- "failed to join cluster"
- "cluster health: NOT HEALTHY"
- "node marked as suspect"

❌ Errors:
- "discovery failed"
- "failed to broadcast"
- "timeout waiting for response"

Health Check Endpoints

Monitor cluster health via HTTP endpoints:

# Check if node is healthy
curl http://localhost:8080/health

# Get cluster status (if exposed)
curl http://localhost:8080/cluster/status

# Expected response shows all cluster members
{
  "local_node": "bifrost-remote-10101-...",
  "members": 3,
  "healthy_members": 3,
  "cluster_health": "HEALTHY"
}

This clustering implementation ensures Bifrost can handle enterprise-scale deployments with high availability, automatic service discovery, and intelligent traffic distribution across any infrastructure.

​Overview

​Why Clustering Matters

​Core Features

​Architecture

​Peer-to-Peer Network Design

​Cluster Communication

​Application messages and entity types

​Message dedup and invalidation

​Node Identity and Region

​Leader Election

​Minimum Node Requirements

​Configuration Basics

​Core Configuration Structure

​Common Discovery Configuration Fields

​Gossip Configuration

​gRPC Configuration

​Top-level Fields

​Broker Mode

​How it differs from mesh mode

​Broker-mode synchronization performance

​Configuration

​Running the broker

​Deploying on Cloud Run

​Roster and reconnection

​Service Discovery Methods

Kubernetes

Consul

etcd

DNS

UDP Broadcast

mDNS

​Kubernetes Discovery

​How It Works

​Configuration

​Configuration Parameters

​Kubernetes Deployment Example

​Troubleshooting

​Consul Discovery

​How It Works

​Configuration

​Configuration Parameters

​Consul Authentication and TLS

​Docker Compose with Consul

​Troubleshooting

​etcd Discovery

​How It Works

​Configuration

​Configuration Parameters

​Docker Compose with etcd

​Troubleshooting

​DNS Discovery

​How It Works

​Configuration

​Configuration Parameters

​Setup Examples

​Troubleshooting

​UDP Broadcast Discovery

​How It Works

​Configuration

​Configuration Parameters

​Docker Compose Example

​Troubleshooting

​mDNS Discovery

​How It Works

​Configuration

​Configuration Parameters

​Local Development Example

​Troubleshooting

​Deployment Patterns

​Docker Compose Deployment

​Kubernetes Production Deployment

​Bare Metal / VM Deployment

​Cluster Operations

​Cluster Topology View

​Cluster Diagnostics

​Mixed-Version Rollouts

​Leader-Coordinated Tasks

​Replicated Entity Types

​Troubleshooting

​General Clustering Issues

Overview

Why Clustering Matters

Core Features

Architecture

Peer-to-Peer Network Design

Cluster Communication

Application messages and entity types

Message dedup and invalidation

Node Identity and Region

Leader Election

Minimum Node Requirements

Configuration Basics

Core Configuration Structure

Common Discovery Configuration Fields

Gossip Configuration

gRPC Configuration

Top-level Fields

Broker Mode

How it differs from mesh mode

Broker-mode synchronization performance

Configuration

Running the broker

Deploying on Cloud Run

Roster and reconnection

Service Discovery Methods

Kubernetes Discovery

How It Works

Configuration

Configuration Parameters

Kubernetes Deployment Example

Troubleshooting

Consul Discovery

How It Works

Configuration

Configuration Parameters

Consul Authentication and TLS

Docker Compose with Consul

Troubleshooting

etcd Discovery

How It Works

Configuration

Configuration Parameters

Docker Compose with etcd

Troubleshooting

DNS Discovery

How It Works

Configuration

Configuration Parameters

Setup Examples

Troubleshooting

UDP Broadcast Discovery

How It Works

Configuration

Configuration Parameters

Docker Compose Example

Troubleshooting

mDNS Discovery

How It Works

Configuration

Configuration Parameters

Local Development Example

Troubleshooting

Deployment Patterns

Docker Compose Deployment

Kubernetes Production Deployment

Bare Metal / VM Deployment

Cluster Operations

Cluster Topology View

Cluster Diagnostics

Mixed-Version Rollouts

Leader-Coordinated Tasks

Replicated Entity Types

Troubleshooting

General Clustering Issues

Health Check Endpoints