This guide assume basic knowledge in the installation of Proxmox server to your own platform.

Refer to Promox Get Started at https://www.proxmox.com/en/products/proxmox-virtual-environment/get-started.

Setup Promox Cluster

Update and Upgrade

Login to Proxmox GUI
Click on Proxmox server name, click Updates > Repositories
Select and Disable enterprise and pve-enterprise component
Click Add, select No-subscription repository, ensure it is Enabled.
Go to Shell or login with ssh root@

apt update && apt upgrade -y
apt dist-upgrade
apt install pve-kernel-7.0 (already come with v9.2.2)

Create a new Proxmox cluster

Click Datacenter, select Cluster, click Create Cluster, give it a name.
Click Join Information, then Copy Information > go to the next Proxmox server

Join a Proxmox cluster

Login to web GUI on a new Proxmox node
Click Datacenter, select Cluster, click Join Cluster, paste the copied ‘Join Information’ above to Information:
Enter peer’s root password, click Join ‘<cluster>’

Setup Ceph Cluster

Join a Ceph cluster

Select the new node, click Ceph, click Install Ceph
Select No-Subscription repository, click Start squid installation
Proceed to Do you want to continue? Y
Close the Setup screen once Ceph Squid successfully

Create Monitor and Manager

Click Ceph > Monitor
Under Monitor, click Create, select the new Node > Create
Under Manager, click Create, select the new Node > Create

Create OSD disk

If you just want assign your second disk, prefer an nvm or ssd to Ceph cluster.

Click Ceph > OSD > click Create: OSD
Select your disk to join Ceph, click Create

If you want to assign a designated ID to the OSD. Create the OSD above with an automated ID, then select it, click Out, click Stop, click More > Destroy and use Shell command below. This excercise allow Ceph to populate all the keyrings and config before the command works.

# In Shell/SSH
ceph-volume lvm create --osd-id 4 --data /dev/nvme0n1

Configure your desk iops

# Bench test your disk iops
ceph tell osd.4 bench 12288000 4096 4194304 100

# Enter your true hardware baselines
ceph config set osd.4 osd_mclock_max_capacity_iops_ssd 37072 (your iops)

# Set the optimal mClock Profile
ceph config set global osd_mclock_profile high_client_ops

Setup Rados Gateway (s3)

Install tools

sudo apt update && sudo apt install s3cmd haproxy keepalived -y

https://github.com/neni84/pve-ceph-radosgw

Create empty keyring to store keys:

ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring

Generate keys and add them to the keyring

ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.pve1 --gen-key
ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.pve2 --gen-key

Grant access:

ceph-authtool -n client.radosgw.pve1 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
ceph-authtool -n client.radosgw.pve2 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring

Add keys to the cluster:

ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.pve1 -i /etc/ceph/ceph.client.radosgw.keyring
ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.pve2 -i /etc/ceph/ceph.client.radosgw.keyring

On each pve node, copy cluster keyring to local storage:

cp /etc/ceph/ceph.client.radosgw.keyring /etc/pve/priv/

Update /etc/ceph/ceph.conf

nano /etc/ceph/ceph.conf

# add at the end of the file
[client.radosgw.pve1]
    host = pve1
    keyring = /etc/ceph/ceph.client.radosgw.keyring
    log file = /var/log/ceph/client.radosgw.$host.log

[client.radosgw.pve2]
    host = pve2
    keyring = /etc/ceph/ceph.client.radosgw.keyring
    log file = /var/log/ceph/client.radosgw.$host.log

Create and enable ceph-radosgw service on each pve node:

mkdir -p /etc/systemd/system/ceph-radosgw.target.wants
ln -s /lib/systemd/system/ceph-radosgw@.service 
  /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@radosgw.$(hostname).service
systemctl daemon-reload
systemctl enable ceph-radosgw@radosgw.$(hostname)
systemctl start ceph-radosgw@radosgw.$(hostname)

Setup HAProxy for load balancing

Update /etc/haproxy/haproxy.cfg

sudo nano /etc/haproxy/haproxy.cfg

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES>
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log                     global
        mode                    http
        option                  httplog
        option                  dontlognull
        option                  http-server-close
        option                  forwardfor except 127.0.0.0/8
        option                  redispatch
        retries                 3
        timeout queue           20s
        timeout connect         5s
        timeout http-request    1s
        timeout http-keep-alive 5s
        timeout client          10m
        timeout server          10m
        timeout check           5s
        maxconn                 8000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend rgw_https
    bind 192.168.0.200:443 ssl crt /etc/ssl/certs/your-tls.pem  # .pem is a combined .key and .pem
    default_backend rgw_backend

backend rgw_backend
    option forwardfor
    balance roundrobin
    option httpchk HEAD / HTTP/1.0
    server rgw-pve1 192.168.0.1:7480 check inter 2s
    server rgw-pve2 192.168.0.2:7480 check inter 2s

listen stats
    bind *:8080
    mode http
    stats enable
    stats uri /
    stats refresh 5s

Setup HAProxy with the same config on another machine

# Set secured permission for certificate file
chmod 600 /etc/ssl/certs/your-tls.pem

# Check config
haproxy -c -f /etc/haproxy/haproxy.cfg

# Start the service
systemctl enable haproxy
systemctl  start haproxy

Setup keepalived for redundancy

Update /etc/keepalived/keepalived.conf

sudo nano /etc/keepalived/keepalived.conf

global_defs {
    max_auto_priority 0
}

# --- HEALTH CHECKS ---
vrrp_script check_haproxy {
    script "killall -0 haproxy"    # Checks if HAProxy is running
    interval 2
    weight 2
}

# --- SINGLE MULTI-SERVICE BLOCK ---
vrrp_instance VI_SHARED_SERVICES {
    state MASTER                  # Change to BACKUP on second machine
    interface eth0
    virtual_router_id 55          # One single unique ID for this block
    priority 101                  # Change to 100 on second machine
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass 8-characters-password
    }

    virtual_ipaddress {
        192.168.0.200/24         # This one single IP handles redundancy
    }

    # Keepalived will monitor BOTH services.
    # If haproxy service fails, the weight drops and triggers a failover.
    track_script {
        check_haproxy
    }
}

Setup a BACKUP virtual router on another machine

# Create service
systemctl enable keepalived
systemctl start keepalived

Create DNS record for s3 service and test

Create A record for s3.example.com to 192.168.0.200

Browse to https://s3.example.com

Part1: Setup Proxmox and Ceph Cluster