Overview

Architecture

Router syslog → Ingest (:15514) → Raw (/data/raw/)
    → Indexer (/data/indexed/) → Compressor → Archive (/data/archive/YYYY/MM/DD/HH/)
        → Replicator (rsync) → Storage Node (/srv/cgnat/archive/YYYY/MM/DD/HH/)

Each completed hour folder contains compressed archive files plus manifest.json. The replicator copies the entire hour directory and records success in archive_metadata (SQLite at /data/metadata/cgnat.db).

Prerequisites

1. Storage server with Ubuntu 22.04+ (or similar Linux), root SSH access.
2. Enough disk for long-term archive retention (recommend dedicated volume for /srv/cgnat/archive).
3. Network: app server 10.30.90.249 must reach storage node 10.30.90.250 on TCP 22.
4. On app server: cgnat-compressor and cgnat-replicator services running.
5. Dashboard login with admin or noc role to register the node.

Step 1 — Prepare Storage Node (run on 10.30.90.250)

On the storage server as root, create archive directory and authorize the app server's replicator SSH key.

Option A — Automated script

First run Step 2 on the app server to generate the public key, copy the key file to storage node, then:

# On STORAGE NODE (10.30.90.250) as root
export ARCHIVE_ROOT=/srv/cgnat/archive
export APP_SERVER_IP=10.30.90.249

mkdir -p "$ARCHIVE_ROOT"
chmod 755 "$ARCHIVE_ROOT"
mkdir -p /root/.ssh && chmod 700 /root/.ssh

# Paste app-server public key (from setup_app_storage_replication.sh output):
echo 'PASTE_PUBLIC_KEY_ONE_LINE' >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys

# Optional: restrict SSH to app server
ufw allow from 10.30.90.249 to any port 22 proto tcp comment 'CGNAT replicator' || true

Option B — Use bundled setup script

# Copy pubkey to /tmp/cgnat_replica.pub on storage node first, then:
bash /opt/cgnat-log-system/scripts/setup_storage_node.sh /tmp/cgnat_replica.pub

Step 2 — Configure App Server Replication (run on 10.30.90.249)

On the application server as root, generate SSH key, update .env, register node in database, and configure replicator.

# On APPLICATION SERVER (10.30.90.249) as root
cd /opt/cgnat-log-system

# Automated setup (generates key, updates .env + admin DB):
STORAGE_IP=10.30.90.250 bash scripts/setup_app_storage_replication.sh

# Copy printed public key to storage node authorized_keys (Step 1)
# Then test SSH:
ssh -i /root/.ssh/cgnat_storage_replica root@10.30.90.250 'mkdir -p /srv/cgnat/archive && echo OK'

# Test rsync:
rsync -av -e "ssh -i /root/.ssh/cgnat_storage_replica -o BatchMode=yes" \
  /etc/hostname root@10.30.90.250:/srv/cgnat/archive/.connectivity-test/

# Restart replicator:
systemctl restart cgnat-replicator
systemctl status cgnat-replicator

.env values (reference)

Step 3 — Register in Dashboard

Open Storage Nodes tab and add or verify the node:

Click Refresh List. Expand Details on each node to see live OS, disk capacity, archive usage, and replication status.

Step 4 — Verify (run on app server only)

# Must run on APPLICATION SERVER (10.30.90.249), NOT on storage node
cd /opt/cgnat-log-system
STORAGE_IP=10.30.90.250 bash scripts/verify_storage_node.sh

Expected output: SSH_OK, rsync success, cgnat-replicator active.

When Does Archive Data Appear?

Real data starts after compressor finishes an hour under /data/archive/YYYY/MM/DD/HH/ and replicator syncs it. Check Details for Archive used, hour folders, and Replicated (DB) counts.

# From app server — storage node archive size + hour count
ssh -i /root/.ssh/cgnat_storage_replica root@10.30.90.250 \
  'du -sh /srv/cgnat/archive; find /srv/cgnat/archive -mindepth 4 -maxdepth 4 -type d | wc -l'

# Replicated files in metadata DB
sqlite3 /data/metadata/cgnat.db \
  "SELECT COUNT(*), SUM(size) FROM archive_metadata WHERE storage_node LIKE '%10.30.90.250%'"

Troubleshooting