reworked services
This commit is contained in:
parent
92f19b36d2
commit
ee99a2da9d
7 changed files with 116 additions and 33 deletions
93
CLAUDE.md
93
CLAUDE.md
|
|
@ -4,31 +4,18 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
Proxmox VE homelab cluster running Docker Swarm services. Three Proxmox hosts (pve, adder, game) connected via VXLAN overlay on vmbr1 (10.10.10.0/24). VMs run Docker CE and form a swarm with pve-postgres as manager.
|
Proxmox VE homelab cluster running Docker Swarm services. Three Proxmox hosts connected via VXLAN overlay on vmbr1 (10.10.10.0/24). VMs run Docker CE and form a swarm with pve-postgres as manager.
|
||||||
|
|
||||||
## Deployment
|
## Hardware
|
||||||
|
|
||||||
All services deploy as Docker Swarm stacks from the manager node (pve-postgres). The active service definitions are in `proxmox/services/`. The `services/` directory contains older pre-migration copies.
|
| Node | Machine | RAM | Storage | Role |
|
||||||
|
|------|---------|-----|---------|------|
|
||||||
|
| pve (Meerkat) | Intel TNTGL357 mini PC | 64GB | 4TB NVMe | Proxmox host, data tier |
|
||||||
|
| adder | System76 Adder WS | 32GB | 2TB NVMe + RTX 2070 | Proxmox host, GPU/crypto tier |
|
||||||
|
| game (CyberPower) | Old gaming PC | 16GB | 500GB NVMe + 2TB HDD | Proxmox host |
|
||||||
|
| Synology NAS 425+ | — | — | 48TB usable | NFS backend (VM backups, ISOs, Nextcloud data — NOT live DBs) |
|
||||||
|
|
||||||
```bash
|
Primary workstation (i9-13900KF, 96GB, RTX 4090) is **standalone — not in the Proxmox cluster**. Joins as NATS leaf node for LLM inference.
|
||||||
# Copy yml to manager and deploy (resolve secrets from pass inline)
|
|
||||||
scp proxmox/services/<service>.yml pve-postgres:~/
|
|
||||||
ssh pve-postgres "VAR=$(pass homelab/VAR) docker stack deploy -c <service>.yml <stack_name>"
|
|
||||||
|
|
||||||
# Remove a stack
|
|
||||||
ssh pve-postgres "docker stack rm <stack_name>"
|
|
||||||
```
|
|
||||||
|
|
||||||
Secrets come from `pass` under the `homelab/` prefix. Resolve them inline when deploying. Never hardcode secrets in files.
|
|
||||||
|
|
||||||
## Service YML Conventions
|
|
||||||
|
|
||||||
- All use `version: '3.8'` and Docker Swarm deploy mode
|
|
||||||
- Each service is pinned to a specific node via `node.hostname` constraint
|
|
||||||
- All connect to the external overlay network `homelab-net` via `${OVERLAY_NETWORK:-homelab-net}`
|
|
||||||
- Named volumes for persistence, `on-failure` restart policy with 3 max attempts
|
|
||||||
- Every service that has a web UI or accepts external connections must publish its ports
|
|
||||||
- Only include environment variables required to start the service — no optional/nice-to-have vars
|
|
||||||
|
|
||||||
## Cluster Topology
|
## Cluster Topology
|
||||||
|
|
||||||
|
|
@ -41,13 +28,75 @@ Secrets come from `pass` under the `homelab/` prefix. Resolve them inline when d
|
||||||
| pve-tools | — | 10.10.10.3 | VM on pve, swarm worker |
|
| pve-tools | — | 10.10.10.3 | VM on pve, swarm worker |
|
||||||
| adder-ghost | — | 10.10.10.20 | VM on adder, swarm worker |
|
| adder-ghost | — | 10.10.10.20 | VM on adder, swarm worker |
|
||||||
|
|
||||||
|
All VMs on vmbr1 (internal, not directly LAN-reachable). A gateway LXC on pve is dual-homed vmbr0+vmbr1 and handles WireGuard + iptables DNAT.
|
||||||
|
|
||||||
|
## VM Allocation (Node 1 / pve — current plan)
|
||||||
|
|
||||||
|
```
|
||||||
|
LXC: gateway 256MB — WireGuard client, iptables DNAT, SSH jump host
|
||||||
|
VM: infra 8GB — NATS/JetStream, n8n
|
||||||
|
VM: data 20GB — PostgreSQL (shared: Ghost, Forgejo, all apps)
|
||||||
|
VM: apps 8GB — Ghost instances, Forgejo
|
||||||
|
VM: redis 14GB — Redis (own VM required — needs vm.overcommit_memory=1 kernel tuning)
|
||||||
|
VM: nextcloud 8GB — Nextcloud AIO (manages its own isolated internal Postgres)
|
||||||
|
```
|
||||||
|
|
||||||
|
No MariaDB/MySQL anywhere — PostgreSQL only. Redis must be a VM (not LXC) for kernel tuning.
|
||||||
|
Nextcloud's internal DB must stay isolated — opinionated AIO lifecycle, upgrade migrations must not affect other app DBs.
|
||||||
|
|
||||||
|
## NFS / Database Rules
|
||||||
|
|
||||||
|
- **NFS is NOT for live database files.** Use local NVMe for PostgreSQL, Redis, Neo4j, monerod/zanod data.
|
||||||
|
- LMDB-based services (monerod, zanod) are especially NFS-hostile — mmap is the architecture.
|
||||||
|
- PostgreSQL tolerates NFS better (WAL + buffer pool) but still prefer local.
|
||||||
|
- NAS use: VM backups (PBS encrypted), ISOs, Nextcloud user data, blockchain snapshots only.
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
Active service definitions: `proxmox/services/`. The `services/` directory contains older pre-migration copies.
|
||||||
|
|
||||||
|
**Preferred: use `deploy-stack.sh`** (in `~/private/Knowledge/Homelab/deploy-stack.sh`) — scans the compose file for `${VAR}` references, pulls secrets from `pass` or KeePassXC, writes an ephemeral chmod-600 `.env`, deploys, then nukes `.env` on exit via trap.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Deploy using pass backend (default)
|
||||||
|
./deploy-stack.sh proxmox/services/<service>.yml <stack_name>
|
||||||
|
|
||||||
|
# Deploy to remote swarm manager
|
||||||
|
./deploy-stack.sh proxmox/services/<service>.yml <stack_name> --host pve-postgres
|
||||||
|
|
||||||
|
# Remove a stack
|
||||||
|
ssh pve-postgres "docker stack rm <stack_name>"
|
||||||
|
```
|
||||||
|
|
||||||
|
Secrets live in `pass` under `homelab/<VARNAME>`. Never hardcode secrets in files.
|
||||||
|
|
||||||
|
## Service YML Conventions
|
||||||
|
|
||||||
|
- All use `version: '3.8'` and Docker Swarm deploy mode
|
||||||
|
- Each service pinned to a specific node via `node.hostname` constraint
|
||||||
|
- All connect to external overlay network `homelab-net` via `${OVERLAY_NETWORK:-homelab-net}`
|
||||||
|
- Named volumes for persistence, `on-failure` restart policy with 3 max attempts
|
||||||
|
- Every service with a web UI or external connections must publish its ports
|
||||||
|
- Only include environment variables required to start the service — no optional/nice-to-have vars
|
||||||
|
|
||||||
## SSH Access
|
## SSH Access
|
||||||
|
|
||||||
Use SSH config aliases (`pve-postgres`, `pve-tools`, `adder-ghost`, `pve`, `adder`, `game`). ProxyJump is configured in `~/.ssh/config`. Never manually hop through intermediate nodes.
|
Use SSH config aliases (`pve-postgres`, `pve-tools`, `adder-ghost`, `pve`, `adder`, `game`). ProxyJump is configured in `~/.ssh/config`. Never manually hop through intermediate nodes.
|
||||||
|
|
||||||
|
## External Infrastructure
|
||||||
|
|
||||||
|
- **BuyVM (Luxembourg):** VPS + 1TB block storage, $7/month, Monero payment. Runs: Peertube, Ghost+BTCPay, Nextcloud, Jitsi, Caddy, WireGuard server.
|
||||||
|
- **Hetzner AX102 (pending):** Bare metal Proxmox. One VM runs Caddy + WireGuard server — terminates TLS for public subdomains, tunnels traffic to home cluster via WireGuard.
|
||||||
|
- **Njalla domains:** Accepts Monero, Swedish jurisdiction.
|
||||||
|
|
||||||
|
## Planned Services
|
||||||
|
|
||||||
|
NATS JetStream (3-replica cluster service), PostgreSQL, Redis, Neo4j, FusionAuth, Authentik, n8n, Nextcloud, Garage (object storage), Ghost, monerod, monero-wallet-rpc, zanod, Peertube, BTCPay, Jitsi, Caddy, WireGuard.
|
||||||
|
|
||||||
## Key Files
|
## Key Files
|
||||||
|
|
||||||
- `proxmox/services/*.yml` — Active swarm stack definitions
|
- `proxmox/services/*.yml` — Active swarm stack definitions
|
||||||
- `proxmox/services/nats.conf` — NATS server config (JetStream, websocket, monitoring)
|
- `proxmox/services/nats.conf` — NATS server config (JetStream, websocket, monitoring)
|
||||||
- `proxmox/services/01-init.sql` — Postgres init script (creates users/databases)
|
- `proxmox/services/01-init.sql` — Postgres init script (creates users/databases)
|
||||||
- `proxmox/post_init_node.org` — Fresh Proxmox node setup steps
|
- `proxmox/post_init_node.org` — Fresh Proxmox node setup steps
|
||||||
|
- `~/private/Knowledge/Homelab/deploy-stack.sh` — Preferred deployment script
|
||||||
|
|
|
||||||
|
|
@ -20,8 +20,14 @@ services:
|
||||||
AUTHENTIK_POSTGRESQL__USER: authentik_user
|
AUTHENTIK_POSTGRESQL__USER: authentik_user
|
||||||
AUTHENTIK_POSTGRESQL__PASSWORD: ${AUTHENTIK_DB_PASSWORD}
|
AUTHENTIK_POSTGRESQL__PASSWORD: ${AUTHENTIK_DB_PASSWORD}
|
||||||
ports:
|
ports:
|
||||||
- "9000:9000"
|
- target: 9000
|
||||||
- "9443:9443"
|
published: 9000
|
||||||
|
protocol: tcp
|
||||||
|
mode: host
|
||||||
|
- target: 9443
|
||||||
|
published: 9443
|
||||||
|
protocol: tcp
|
||||||
|
mode: host
|
||||||
volumes:
|
volumes:
|
||||||
- authentik_media:/media
|
- authentik_media:/media
|
||||||
- authentik_templates:/templates
|
- authentik_templates:/templates
|
||||||
|
|
|
||||||
|
|
@ -19,7 +19,10 @@ services:
|
||||||
database__connection__database: ghost1_db
|
database__connection__database: ghost1_db
|
||||||
url: ${GHOST1_URL:-http://localhost:2368}
|
url: ${GHOST1_URL:-http://localhost:2368}
|
||||||
ports:
|
ports:
|
||||||
- "2368:2368"
|
- target: 2368
|
||||||
|
published: 2368
|
||||||
|
protocol: tcp
|
||||||
|
mode: host
|
||||||
volumes:
|
volumes:
|
||||||
- ghost1_data:/var/lib/ghost/content
|
- ghost1_data:/var/lib/ghost/content
|
||||||
networks:
|
networks:
|
||||||
|
|
@ -46,7 +49,10 @@ services:
|
||||||
url: ${GHOST2_URL:-http://localhost:2369}
|
url: ${GHOST2_URL:-http://localhost:2369}
|
||||||
server__port: 2369
|
server__port: 2369
|
||||||
ports:
|
ports:
|
||||||
- "2369:2369"
|
- target: 2369
|
||||||
|
published: 2369
|
||||||
|
protocol: tcp
|
||||||
|
mode: host
|
||||||
volumes:
|
volumes:
|
||||||
- ghost2_data:/var/lib/ghost/content
|
- ghost2_data:/var/lib/ghost/content
|
||||||
networks:
|
networks:
|
||||||
|
|
@ -73,7 +79,10 @@ services:
|
||||||
url: ${GHOST3_URL:-http://localhost:2370}
|
url: ${GHOST3_URL:-http://localhost:2370}
|
||||||
server__port: 2370
|
server__port: 2370
|
||||||
ports:
|
ports:
|
||||||
- "2370:2370"
|
- target: 2370
|
||||||
|
published: 2370
|
||||||
|
protocol: tcp
|
||||||
|
mode: host
|
||||||
volumes:
|
volumes:
|
||||||
- ghost3_data:/var/lib/ghost/content
|
- ghost3_data:/var/lib/ghost/content
|
||||||
networks:
|
networks:
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,10 @@ services:
|
||||||
environment:
|
environment:
|
||||||
MYSQL_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD}
|
MYSQL_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD}
|
||||||
ports:
|
ports:
|
||||||
- "3306:3306"
|
- target: 3306
|
||||||
|
published: 3306
|
||||||
|
protocol: tcp
|
||||||
|
mode: host
|
||||||
volumes:
|
volumes:
|
||||||
- mariadb_data:/var/lib/mysql
|
- mariadb_data:/var/lib/mysql
|
||||||
networks:
|
networks:
|
||||||
|
|
|
||||||
|
|
@ -16,7 +16,10 @@ services:
|
||||||
DB_POSTGRESDB_USER: n8n_user
|
DB_POSTGRESDB_USER: n8n_user
|
||||||
DB_POSTGRESDB_PASSWORD: ${N8N_DB_PASSWORD}
|
DB_POSTGRESDB_PASSWORD: ${N8N_DB_PASSWORD}
|
||||||
ports:
|
ports:
|
||||||
- "5678:5678"
|
- target: 5678
|
||||||
|
published: 5678
|
||||||
|
protocol: tcp
|
||||||
|
mode: host
|
||||||
volumes:
|
volumes:
|
||||||
- n8n_data:/home/node/.n8n
|
- n8n_data:/home/node/.n8n
|
||||||
networks:
|
networks:
|
||||||
|
|
|
||||||
|
|
@ -13,9 +13,18 @@ services:
|
||||||
command:
|
command:
|
||||||
- -c=/etc/nats/nats.conf
|
- -c=/etc/nats/nats.conf
|
||||||
ports:
|
ports:
|
||||||
- "4222:4222"
|
- target: 4222
|
||||||
- "8080:8080"
|
published: 4222
|
||||||
- "8223:8222"
|
protocol: tcp
|
||||||
|
mode: host
|
||||||
|
- target: 8080
|
||||||
|
published: 8080
|
||||||
|
protocol: tcp
|
||||||
|
mode: host
|
||||||
|
- target: 8222
|
||||||
|
published: 8223
|
||||||
|
protocol: tcp
|
||||||
|
mode: host
|
||||||
volumes:
|
volumes:
|
||||||
- nats_data:/data
|
- nats_data:/data
|
||||||
configs:
|
configs:
|
||||||
|
|
|
||||||
|
|
@ -16,8 +16,12 @@ services:
|
||||||
INVITATIONS_ALLOWED: "true"
|
INVITATIONS_ALLOWED: "true"
|
||||||
SHOW_PASSWORD_HINT: "false"
|
SHOW_PASSWORD_HINT: "false"
|
||||||
ROCKET_PORT: 8222
|
ROCKET_PORT: 8222
|
||||||
|
ADMIN_TOKEN: ${VAULT_ADMIN_TOKEN}
|
||||||
ports:
|
ports:
|
||||||
- "8222:8222"
|
- target: 8222
|
||||||
|
published: 8222
|
||||||
|
protocol: tcp
|
||||||
|
mode: host
|
||||||
volumes:
|
volumes:
|
||||||
- vaultwarden_data:/data
|
- vaultwarden_data:/data
|
||||||
networks:
|
networks:
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue