Skip to content

Enable MOJO HA Cluster

MOJO HA Cluster provides automatic failover between two MOJO nodes, ensuring your infrastructure management platform stays available even if one node goes down. This guide walks you through enabling HA on a new or existing MOJO installation.

How It Works

MOJO HA Cluster pairs two MOJO hosts on the same network in an Active-Standby configuration. A floating Virtual IP (VIP) always points to whichever node is currently active. If the active node becomes unhealthy, the standby automatically takes over — typically within 15 to 30 seconds.

Your managed servers, operators, and any upstream systems (like MGC) continue to use the same hostname and VIP. The failover is transparent.

What Is Protected

Service How It Stays Available
MOJO API & GUI Stateless — starts immediately on the new active node
Database Real-time streaming replication keeps the standby copy up to date. Automatic promotion on failover
Redis Replicated between nodes. Automatic promotion on failover
DHCP Native failover between the two DHCP peers — lease state is shared
TFTP / PXE Stateless — available as soon as the VIP moves
OS Images & Firmware Periodically synced from the active node to the standby

Prerequisites

Before enabling HA, make sure you have:

  • [x] Two MOJO hosts on the same Layer 2 network segment (same subnet/VLAN)
  • [x] MOJO installed on both hosts (see Installing MOJO)
  • [x] A free IP address on the network to use as the floating VIP
  • [x] SSH key-based access between the two hosts (for file synchronization)
  • [x] HA traffic allowed — if you have a firewall, ensure IP protocol 112 is permitted between the two nodes

Existing Installations

You do not need to reinstall MOJO to enable HA. Just re-run mojo-setup on each node to add the HA configuration. Your existing data and settings are preserved.

Enabling HA Cluster

Step 1: Plan Your Configuration

Before starting, decide on the following values. Both nodes must use the same values for items marked "shared":

Setting Description Example
Virtual IP (VIP) Floating IP that MOJO will be accessed through (shared) 192.168.1.100
VIP CIDR Network prefix length (shared) 24
Network Interface The interface connected to your management network (per-node) eth0, ens192
HA Cluster ID A unique number (1-255) identifying this HA pair (shared) 51
HA Cluster Password Shared secret for the HA pair (max 8 characters) (shared) mojo1234
Replication User Database replication username (shared) replicator
Replication Password Database replication password (shared) (auto-generated)

Tip

Point your DNS record for MOJO_HOSTNAME to the Virtual IP. This ensures all clients always reach whichever node is currently active.

Step 2: Configure the Primary Node

On the host you want to be the primary (active) node, run:

cd /opt/mojo
sudo ./mojo-setup

The setup wizard will walk you through the standard MOJO configuration. When you reach the High Availability section:

  1. Enable High Availability: y
  2. HA Role: primary
  3. Peer IP: Enter the IP address of the secondary node
  4. Virtual IP (VIP): Enter the floating IP you chose above
  5. VIP CIDR: Enter the network prefix (e.g., 24)
  6. Network Interface: Enter the interface name on this node
  7. HA Cluster ID: Enter your chosen ID (default: 51)
  8. HA Cluster Password: Enter the shared password
  9. Replication User: Accept the default (replicator) or enter your own
  10. Replication Password: Accept the auto-generated password or enter your own

Write Down the Passwords

The HA Cluster Password and Replication Password must be entered identically on the secondary node. Note them down before proceeding.

Step 3: Start MOJO on the Primary

sudo ./mojo-launcher start mojo --update

Wait for all services to come up. You should see Mojo Platform started successfully! along with HA mode enabled (role: primary).

Step 4: Initialize Database Replication

While MOJO is running on the primary, set up the database for replication:

sudo bash ha/scripts/pg_setup_replication.sh

This creates the replication user and configures the database to accept replication connections from the secondary node.

Step 5: Configure the Secondary Node

On the secondary (standby) host, run:

cd /opt/mojo
sudo ./mojo-setup

Enter the same HA configuration as the primary, except:

  • HA Role: secondary
  • Peer IP: Enter the IP address of the primary node

All other HA settings (VIP, passwords, cluster ID) must match the primary.

Step 6: Initialize the Secondary Database

On the secondary node, clone the database from the primary:

sudo bash ha/scripts/pg_basebackup.sh <primary_ip>

Replace <primary_ip> with the IP address of the primary node. This creates a full copy of the database and configures it as a streaming replica.

Note

This step will stop MOJO on the secondary if it is running, erase the local database, and replace it with a copy from the primary. This is expected during initial setup.

Step 7: Start MOJO on the Secondary

sudo ./mojo-launcher start mojo

The secondary will start in standby mode, continuously receiving database updates from the primary.

Step 8: Run Initial File Synchronization

On the primary node, run a one-time full sync to copy OS images, firmware bundles, and other files to the secondary:

sudo bash ha/scripts/ha_sync_volumes.sh --initial

Ongoing Sync Is Automatic

A background sync job is automatically installed during mojo-setup on the primary node. It runs every 15 minutes to keep files in sync — no manual cron setup needed.

Step 9: Update DNS

Point your MOJO_HOSTNAME DNS record to the Virtual IP (VIP). This ensures that regardless of which node is active, all clients connect through the VIP.

Verifying Your HA Cluster

Once both nodes are running, verify the cluster status:

sudo ./mojo-launcher ha-status

You should see:

  • Primary node: VIP is active, database is in read-write mode, standbys connected: 1
  • Secondary node: VIP is not active, database is in standby (read-only) mode, peer is reachable

HA Cluster is Ready

If both nodes show healthy status and the database shows a connected standby, your HA cluster is operational. The VIP will automatically move to the standby if the primary becomes unavailable.

Day-to-Day Operations

Checking Cluster Status

At any time, check the health of your HA cluster from either node:

sudo ./mojo-launcher ha-status

This shows:

  • Which node currently holds the VIP (active node)
  • Database replication status and lag
  • Redis replication status
  • Peer connectivity
  • Last file synchronization time

You can also view the HA status in the MOJO web console under System Administration.

Planned Maintenance (Manual Failover)

To take the active node offline for maintenance:

  1. Stop MOJO on the active node:

    sudo ./mojo-launcher stop mojo

  2. The standby node automatically detects the outage and takes over:

    • The VIP moves to the standby
    • The standby database promotes to read-write
    • MOJO becomes fully operational on the new active node

  3. Perform your maintenance on the now-offline node.

  4. When ready to bring it back, rebuild it as a standby:

    sudo bash ha/scripts/pg_basebackup.sh <current_active_ip>
sudo ./mojo-launcher start mojo

The returned node joins as a standby and begins receiving replication from the current active node.

No Automatic Failback

MOJO HA Cluster does not automatically switch roles back after a failover. This is intentional to prevent instability. To swap roles, repeat the manual failover process in the opposite direction.

Updating MOJO with HA Enabled

To update MOJO on an HA cluster with minimal downtime:

  1. Update the standby first:

    # On the standby node
sudo ./mojo-launcher update mojo
sudo ./mojo-launcher restart mojo

  2. Failover to the updated standby by stopping the active node:

    # On the current active node
sudo ./mojo-launcher stop mojo

  3. Update the former active node (now standby):

    sudo bash ha/scripts/pg_basebackup.sh <new_active_ip>
sudo ./mojo-launcher update mojo
sudo ./mojo-launcher start mojo

Both nodes are now updated with only a brief failover window.

Troubleshooting

VIP Is Not Moving on Failover

  • Ensure HA traffic (IP protocol 112) is not blocked by a firewall between the two nodes
  • Verify the HA Cluster Password and HA Cluster ID match on both nodes
  • Confirm the Network Interface name is correct on each node (ip addr show to verify)
  • Check HA service logs: sudo docker logs mojo-ha (or sudo podman logs mojo-ha)

Database Replication Lag

Check replication status on the active node:

sudo docker exec mojo-db psql -U mojo -c "SELECT client_addr, state FROM pg_stat_replication;"

If no replicas are connected, verify the standby node is running and the primary's IP is reachable from the standby on port 5432.

DHCP Failover Not Working

  • Ensure port 647 is open between both nodes (DHCP failover peer port)
  • Check DHCP logs: sudo docker logs mojo-dhcpd

File Sync Not Running

Check the sync log:

tail -50 /var/log/mojo-ha-sync.log

Ensure SSH key-based authentication works between the nodes:

ssh <peer_ip> hostname

HA Configuration Reference

All HA settings are stored in configs/mojo.env and can be reconfigured by running sudo ./mojo-setup:

Setting Description Default
HA_ENABLED Enable HA Cluster (y / n) n
HA_ROLE This node's role (primary / secondary) —
HA_PEER_IP IP address of the other node —
HA_VIRTUAL_IP Floating Virtual IP (VIP) —
HA_VIRTUAL_IP_CIDR VIP network prefix length 24
MOJO_HA_INTERFACE Network interface for HA traffic —
MOJO_HA_ROUTER_ID HA Cluster ID (1-255, unique per pair) 51
MOJO_HA_PASSWORD HA Cluster shared secret (max 8 chars) —
HA_REPLICATION_USER Database replication username replicator
HA_REPLICATION_PASSWORD Database replication password —

Support

If you have any questions about MOJO HA Cluster, contact us at support@metify.io.