Partial Backup on Real Volumes/Containers

Code to create backups for the containers that we currently use:

• Litellm stack
• OWUI stack
• Wekan
• Nexterm # not anymore, now in monitor
• Whisper (STT)
• Qwenn3 (TTS)
• (Add more when applies)

The code is in /usr/local/sbin/container-backups.sh. Remember to previously create the corresponding backup folder structure.

NOTE: To find more details about how each backup works refer to the following sections: Docker Compose Backup & DB Container Backup

#!/usr/bin/env bash
set -euo pipefail

[[ -e /mnt/bottle/MOUNTED ]] || { echo "Backup drive not mounted"; exit 1; }

# Restart all containers on exit (whether success or failure). Prevention if some script fails after the container was stopped but before it was restarted
trap 'echo "Restarting containers..."; \
    docker start litellm-stack-prometheus-1 \
    litellm-stack-litellm-1 \
    beechat-open-webui-1 \
    wekan-app \
    whisper_gpu \
    qwen3-tts-api' EXIT

# .ENV BACKUP ////////////////////////////////////////
echo "Starting env backup/////////////////////"
tar czf /home/sysadmin/backups/container_env/containers_env_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/ .env
echo -e "Finished env backup///////////////////// \n"

# LITELLM BACKUP ////////////////////////////////////
echo "Starting LITELLM backup/////////////////"

echo "Backing litellm docker compose file"
tar czf /home/sysadmin/backups/litellm-stack/litellm_docker_compose_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/litellm-stack/ docker-compose.yml

echo "Backing litellm config file"
tar czf /home/sysadmin/backups/litellm-stack/litellm_config_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/litellm-stack/ config.yaml

echo "Backing prometheus config file"
tar czf \
/home/sysadmin/backups/litellm-stack/litellm_prometheus_config_$(date +%F).tar.gz \
-C /home/sysadmin/repositories/Apps/litellm-stack/ prometheus.yml

echo "Backing prometheus volume"
echo "Stopping container..."
docker stop litellm-stack-prometheus-1
docker run --rm -v litellm_prometheus_data:/data -v /home/sysadmin/backups/litellm-stack:/backup alpine tar czf /backup/litellm_prometheus_volume_$(date +%F).tar.gz -C /data .
echo "Starting Container..."
docker start litellm-stack-prometheus-1

echo "Creating database dump"
echo "Stopping container..."
docker stop litellm-stack-litellm-1

cd /home/sysadmin/repositories/Apps/litellm-stack/
docker compose exec -T db pg_dump -U llmproxy litellm > /home/sysadmin/backups/litellm-stack/litellm_pg_dump_$(date +%F).sql

echo "Backing database volume"
docker run --rm \
-v litellm_postgres_data:/data \
-v /home/sysadmin/backups/litellm-stack:/backup \
alpine \
tar czf /backup/litellm_pg_volume_$(date +%F).tar.gz -C /data .
echo "Starting Container..."
docker start litellm-stack-litellm-1
echo -e "Finished LITELLM backup///////////////// \n"

# OWUI BACKUP ////////////////////////////////////
echo "Starting OWUI backup..."

echo "Creating backup of OWUI docker compose file"
tar czf /home/sysadmin/backups/beechat-owui-stack/owui_docker_compose_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/beechat-owui-stack/ docker-compose.yml

echo "Creating backup of OWUI config file (openwebui folder)"
tar czf /home/sysadmin/backups/beechat-owui-stack/owui_openwebui_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/beechat-owui-stack openwebui

echo "Creating backup of OWUI volume"
echo "Stopping container..."
docker stop beechat-open-webui-1

docker run --rm \
-v beechat_open-webui:/data \
-v /home/sysadmin/backups/beechat-owui-stack/:/backup \
alpine \
tar czf /backup/owui_volume_$(date +%F).tar.gz -C /data .

echo "Starting container..."
docker start beechat-open-webui-1

echo "Creating backup of OWUI langfuse & pipelines folders"
tar czf /home/sysadmin/backups/beechat-owui-stack/owui_langfuse_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/beechat-owui-stack langfuse
tar czf /home/sysadmin/backups/beechat-owui-stack/owui_pipelines_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/beechat-owui-stack pipelines

echo -e "Finished OWUI backup///////////////// \n"

# WEKAN BACKUP ////////////////////////////////////
echo "\nStarting WEKAN backup..."

echo -e "\nCreating backup docker compose file"
tar czf /home/sysadmin/backups/wekan-stack/wekan_docker_compose_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/wekan-stack/ docker-compose.yml

echo -e "\nCreating Wekan db dump..."
echo "Stopping App Container..."
docker stop wekan-app

docker exec wekan-db mongodump --archive | gzip > /home/sysadmin/backups/wekan-stack/wekan_mongo_$(date +%F).archive.gz

echo "Starting App container..."
docker start wekan-app

echo -e "Finished WEKAN backup/////////////// \n"

# NEXTERM BACKUP ///////////////////////////////
#echo "Starting NEXTERM backup"

#echo "Creating backup docker compose file"
#tar czf /home/sysadmin/backups/nexterm/nexterm_docker_compose_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/nexterm/ docker-compose.yml

#echo "Creating backup volume"
#echo "Stopping container"
#docker stop nexterm

#docker run --rm \
#-v nexterm:/data \
#-v /home/sysadmin/backups/nexterm/:/backup \
#alpine \
#tar czf /backup/nexterm_volume_$(date +%F).tar.gz -C /data .

#echo "Starting container"
#docker start nexterm

#echo -e "Finished NEXTERM backup////////////// \n"

# Whisper BACKUP ////////////////////////////////////
echo "\nStarting WHISPER backup..."

echo "Backing WHISPER docker compose file"
tar czf /home/sysadmin/backups/whisper/whisper_docker_compose_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/whisper/ docker-compose.yml
echo -e "Finished WHISPER backup/////////////// \n"

# Qwenn BACKUP ////////////////////////////////////
echo "\nStarting QWENN backup..."

echo "Backing QWENN docker compose file"
tar czf /home/sysadmin/backups/Qwen3-TTS-Openai-Fastapi/qwenn_docker_compose_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/Qwen3-TTS-Openai-Fastapi/ docker-compose.yml
echo -e "Finished QWENN backup/////////////// \n"

# RESTIC ////////////////////////////
echo "Taking RESTIC snapshots///////"
cd /home/sysadmin/backups

export RESTIC_PASSWORD='------'

# IF YOU ADD A NEW ENTRY DONT FORGET TO ADD THE TAG HERE AND...
TAGS=(container_env_bk full_litellm_bk full_owui_bk full_wekan_bk whisper_bk qwenn_bk)
# THE FOLDER NAMES HERE AS WELL
FOLDER_NAMES=(container_env litellm-stack beechat-owui-stack wekan-stack Qwen3-TTS-Openai-Fastapi whisper)
for i in "${!TAGS[@]}"; do
        restic -r /mnt/bottle/backup/restic-docker backup "./${FOLDER_NAMES[$i]}" --tag "${TAGS[$i]}"
done
echo "Finished taking snapshots"
# DELETE COMPRESSED FILES STORED IN HOST //////
echo -e "\nDeleting generated backup files (we only preserve the ones in restic)"
for folder_name in "${FOLDER_NAMES[@]}"; do
        rm -f "/home/sysadmin/backups/${folder_name}/"*
done
echo "Finished deleting generated compressed backup files"
echo "ALL DONE

For now we skip the restoring part.

Cleanup Script

The cleanup script preserves the last three snapshots from each tag (each app):

#!/usr/bin/env bash
set -euo pipefail
LOCKFILE=/var/lock/restic-docker-cleanup.lock
REPO="/mnt/bottle/backup/restic-docker"
RESTIC_BIN="/usr/bin/restic"
LOGFILE="/var/log/restic-docker-cleanup.log"
KEEP_LAST=3

export RESTIC_PASSWORD=''

# Prevent concurrent runs
exec 9>"$LOCKFILE"
flock -n 9 || { echo "Another instance is already running, exiting."; exit 1; }
TAGS=(container_env_bk full_litellm_bk full_owui_bk full_wekan_bk whisper_bk qwenn_bk)

show_counts() {
        for tag in "${TAGS[@]}"; do
                echo -e "\nSNAPSHOTS WITH TAG: ${tag}:"
                "$RESTIC_BIN" snapshots -r "$REPO" --tag "$tag" --json | jq 'length' || true
        done
}

{
  DRY_ARGS=()
  DRYRUN=${1:-false}
  if [[ "$DRYRUN" == "dry-run" ]]; then
        DRY_ARGS=(--dry-run)
        echo "Running in dry-run mode, nothing will actuallybe deleted"
  fi

  echo "==== $(date -Iseconds) starting restic docker backup cleanup (keep last ${KEEP_LAST}) ===="
  # show current counts
  show_counts

 # Forget old snapshots per tag (no --prune yet)
  for tag in "${TAGS[@]}"; do 
        echo -e "\n---- running restic forget for tag ${tag} --keep-last ${KEEP_LAST} ${DRY_ARGS[*]} ----"
        "$RESTIC_BIN" -r "$REPO" forget --keep-last "$KEEP_LAST" --tag "$tag" "${DRY_ARGS[@]}"
  done

  # Single prune pass
  echo -e "\n---- running restic prune ${DRY_ARGS[*]} ----"
    "$RESTIC_BIN" -r "$REPO" prune "${DRY_ARGS[@]}"

  echo -e "\n---- done; new snapshot counts ----"
    show_counts

  echo -e "\n==== $(date -Iseconds) finished ===="
} 2>&1 | tee -a "$LOGFILE"

Cronjobs

The backup script will be executed daily at 2:00 a.m. and the cleanup script will be executed the first of each month at 5:00 a.m.

0 2 * * * bash /usr/local/sbin/container-backups.sh >> /var/log/container-backups.log 2>&1
0 5 1 * * bash /usr/local/sbin/restic-docker-cleanup.sh