Partial Backup on Real Volumes/Containers

Code to create backups for the containers that we currently use:

• Litellm stack
• OWUI stack
• Wekan
• Nexterm
• (Add more when applies)

The code is in /usr/local/sbin/container-backups.sh. Remember to previously create the corresponding backup folder structure.

NOTE: To find more details about how each backup works refer to the following sections: Docker Compose Backup & DB Container Backup

#!/usr/bin/env bash
set -euo pipefail

[[ -e /mnt/bottle/MOUNTED ]] || { echo "Backup drive not mounted"; exit 1; }

# Restart all containers on exit (whether success or failure). Prevention if some script fails after the container was stopped but before it was restarted
trap 'echo "Restarting containers..."; \
    docker start litellm-stack-prometheus-1 \
    litellm-stack-litellm-1 \
    beechat-open-webui-1 \
    wekan-app \
    nexterm' EXIT

# .ENV BACKUP ////////////////////////////////////////
echo "Starting env backup/////////////////////"
tar czf /home/sysadmin/backups/container_env/containers_env_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/ .env
echo -e "Finished env backup///////////////////// \n"

# LITELLM BACKUP ////////////////////////////////////
echo "Starting LITELLM backup/////////////////"

echo "Backing litellm docker compose file"
tar czf /home/sysadmin/backups/litellm-stack/litellm_docker_compose_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/litellm-stack/ docker-compose.yml

echo "Backing litellm config file"
tar czf /home/sysadmin/backups/litellm-stack/litellm_config_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/litellm-stack/ config.yaml

echo "Backing prometheus config file"
tar czf \
/home/sysadmin/backups/litellm-stack/litellm_prometheus_config_$(date +%F).tar.gz \
-C /home/sysadmin/repositories/Apps/litellm-stack/ prometheus.yml

echo "Backing prometheus volume"
echo "Stopping container..."
docker stop litellm-stack-prometheus-1
docker run --rm -v litellm_prometheus_data:/data -v /home/sysadmin/backups/litellm-stack:/backup alpine tar czf /backup/litellm_prometheus_volume_$(date +%F).tar.gz -C /data .
echo "Starting Container..."
docker start litellm-stack-prometheus-1

echo "Creating database dump"
echo "Stopping container..."
docker stop litellm-stack-litellm-1

cd /home/sysadmin/repositories/Apps/litellm-stack/
docker compose exec -T db pg_dump -U llmproxy litellm > /home/sysadmin/backups/litellm-stack/litellm_pg_dump_$(date +%F).sql

echo "Backing database volume"
docker run --rm \
-v litellm_postgres_data:/data \
-v /home/sysadmin/backups/litellm-stack:/backup \
alpine \
tar czf /backup/litellm_pg_volume_$(date +%F).tar.gz -C /data .
echo "Starting Container..."
docker start litellm-stack-litellm-1
echo -e "Finished LITELLM backup///////////////// \n"

# OWUI BACKUP ////////////////////////////////////
echo "Starting OWUI backup..."

echo "Creating backup of OWUI docker compose file"
tar czf /home/sysadmin/backups/beechat-owui-stack/owui_docker_compose_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/beechat-owui-stack/ docker-compose.yml

echo "Creating backup of OWUI config file (openwebui folder)"
tar czf /home/sysadmin/backups/beechat-owui-stack/owui_openwebui_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/beechat-owui-stack openwebui

echo "Creating backup of OWUI volume"
echo "Stopping container..."
docker stop beechat-open-webui-1

docker run --rm \
-v beechat_open-webui:/data \
-v /home/sysadmin/backups/beechat-owui-stack/:/backup \
alpine \
tar czf /backup/owui_volume_$(date +%F).tar.gz -C /data .

echo "Starting container..."
docker start beechat-open-webui-1

echo "Creating backup of OWUI langfuse & pipelines folders"
tar czf /home/sysadmin/backups/beechat-owui-stack/owui_langfuse_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/beechat-owui-stack langfuse
tar czf /home/sysadmin/backups/beechat-owui-stack/owui_pipelines_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/beechat-owui-stack pipelines

echo -e "Finished OWUI backup///////////////// \n"

# WEKAN BACKUP ////////////////////////////////////
echo "\nStarting WEKAN backup..."

echo -e "\nCreating backup docker compose file"
tar czf /home/sysadmin/backups/wekan-stack/wekan_docker_compose_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/wekan-stack/ docker-compose.yml

echo -e "\nCreating Wekan db dump..."
echo "Stopping App Container..."
docker stop wekan-app

docker exec wekan-db mongodump --archive | gzip > /home/sysadmin/backups/wekan-stack/wekan_mongo_$(date +%F).archive.gz

echo "Starting App container..."
docker start wekan-app

echo -e "Finished WEKAN backup/////////////// \n"

# NEXTERM BACKUP ///////////////////////////////
echo "Starting NEXTERM backup"

echo "Creating backup docker compose file"
tar czf /home/sysadmin/backups/nexterm/nexterm_docker_compose_$(date +%F).tar.gz -C /home/sysadmin/repositories/Apps/nexterm/ docker-compose.yml

echo "Creating backup volume"
echo "Stopping container"
docker stop nexterm

docker run --rm \
-v nexterm:/data \
-v /home/sysadmin/backups/nexterm/:/backup \
alpine \
tar czf /backup/nexterm_volume_$(date +%F).tar.gz -C /data .

echo "Starting container"
docker start nexterm

echo -e "Finished NEXTERM backup////////////// \n"

# RESTIC ////////////////////////////
echo "Taking RESTIC snapshots///////"
cd /home/sysadmin/backups

export RESTIC_PASSWORD='------'

# IF YOU ADD A NEW ENTRY DONT FORGET TO ADD THE TAG HERE AND...
TAGS=(container_env_bk full_litellm_bk full_owui_bk full_wekan_bk full_nexterm_bk)
# THE FOLDER NAMES HERE AS WELL
FOLDER_NAMES=(container_env litellm-stack beechat-owui-stack wekan-stack nexterm)
for i in "${!TAGS[@]}"; do
        restic -r /mnt/bottle/backup/restic-docker backup "./${FOLDER_NAMES[$i]}" --tag "${TAGS[$i]}"
done
echo "Finished taking snapshots"
# DELETE COMPRESSED FILES STORED IN HOST //////
echo -e "\nDeleting generated backup files (we only preserve the ones in restic)"
for folder_name in "${FOLDER_NAMES[@]}"; do
        rm -f "/home/sysadmin/backups/${folder_name}/"*
done
echo "Finished deleting generated compressed backup files"
echo "ALL DONE

For now we skip the restoring part.

Next steps: Automatic Scheduling (CRON) & Clean Up Script

...