Backup and Recovery

This guide explains how to implement a comprehensive backup and recovery strategy for your AI Controller deployment.

Note: This documentation uses <AIC install directory> to represent the AIC installation path. In scripts and commands, this is typically set using the AIC_DIR variable. Replace <AIC install directory> with your actual installation path (e.g., /opt/aic).

Backup Strategy Overview

An effective AI Controller backup strategy should include:

Database backups: Preserving all configuration, rules, and logging data
Configuration backups: Saving all server configuration files
Certificate backups: Protecting TLS certificates and keys
Log file archives: Maintaining historical logs for compliance and debugging
Recovery procedures: Documented processes for various recovery scenarios

flowchart TD
    classDef primary fill:#4285F4,stroke:#2A56C6,color:#FFFFFF
    classDef secondary fill:#34A853,stroke:#24763B,color:#FFFFFF
    classDef tertiary fill:#EA4335,stroke:#B31412,color:#FFFFFF
    classDef neutral fill:#9AA0A6,stroke:#5F6368,color:#FFFFFF

    AIC[AIC System] --> DB[Database]:::primary
    AIC --> Config[Configuration Files]:::secondary
    AIC --> Certs[TLS Certificates]:::tertiary
    AIC --> Logs[Log Archives]:::neutral

    DB --> Daily[Daily Backups]
    Config --> Change[After Changes]
    Certs --> Secure[Secure Storage]
    Logs --> Weekly[Weekly Archives]

    Daily --> Local[Local Storage]
    Daily --> Remote[Remote Storage]
    Change --> Remote
    Secure --> Remote
    Weekly --> Local

    Local --> Retention[Retention Policy]
    Remote --> Retention
    Retention --> Recovery[Recovery Procedures]

Diagram showing the key components of AI Controller that should be backed up

What to Back Up

Critical Components

Component	Description	Location	Backup Frequency	Suggested Method
Database	Contains users, API keys, rules, and logs	MySQL/MariaDB database	Daily	Scheduled mysqldump
Server Config	Primary AI Controller configuration file	`<AIC install directory>/configuration/server.cfg`	After changes	Version control or script
TLS Certificates	SSL/TLS certificates and private keys	Typically `<AIC install directory>/ca-certificates/` (location customised at installation time)	After changes	Secure copy to backup location
Custom Scripts	Any customized scripts or tools you have created	Recommended location `<AIC install directory>/scripts/`	After changes	Version control or script
Logs	System and request logs	Typically `<AIC install directory>/logs/` (location customised at installation time)	Weekly	Log rotation/archival

Database Backup

The database is the most critical component to back up as it contains:

User accounts and permissions
API key definitions and mappings
Rules Engine configuration
Provider configurations and credentials
Request/response logs

For database backups, consider:

Using mysqldump for consistent backups
Implementing automated scheduling based on your platform:
- Linux: cron or systemd timers
- Windows: Task Scheduler
- Docker/Kubernetes: CronJobs
Testing restore procedures regularly
Storing backups in multiple locations

Backup Methods

Database Backup with mysqldump

The mysqldump utility is the standard backup tool for MySQL databases. See the official MySQL mysqldump documentation for detailed options and usage information.

Key considerations:

Use --single-transaction for consistent backups of InnoDB tables
Use --quick option for large databases to avoid memory issues
Compress output with gzip or similar tools to save space
Schedule regular backups appropriate to your RPO (Recovery Point Objective)

Platform-specific resources:

Windows: MySQL on Windows backup guide
Docker: Execute mysqldump within the MySQL container
Cloud: Consider managed database backup services

Configuration and File Backup

Back up configuration files and custom scripts using appropriate tools for your platform:

Linux/Unix: tar, rsync, or backup utilities
Windows: PowerShell Compress-Archive, Robocopy, or backup software
Version Control: Consider using Git for configuration file management

Enterprise Backup Solutions

For large-scale deployments, consider:

MySQL Enterprise Backup for hot backups
Cloud provider backup services (AWS RDS, Azure Database, Google Cloud SQL)
Enterprise backup software integration
Container orchestration backup tools (Velero for Kubernetes)

Backup Scheduling

Schedule automated backups based on your platform:

Linux: Use cron or systemd timers
Windows: Use Task Scheduler or PowerShell scheduled tasks
Docker/Kubernetes: Implement CronJobs
Cloud: Leverage managed backup services

Consider implementing:

Daily database backups
Weekly full configuration backups
Monthly archival backups
Real-time replication for critical systems

Recovery Procedures

Standard Recovery Process

General process for restoring AI Controller components:

Stop AI Controller service
- Linux (systemd service): sudo systemctl stop aic
- Linux (manual): Terminate the AI Controller process
- Windows (service): Stop the AICService in Services Manager or use sc stop AICService
- Windows (manual): Terminate the AIC.exe process
- Docker: docker stop aic-container or docker-compose down
Restore the required component(s):
- Database: Uncompress backup file, restore using MySQL utilities
- Configuration: Extract backup archive, restore files to appropriate locations
- Certificates: Copy certificate files to correct directories
Set correct file permissions (if restoring configuration/certificates)
- Linux:
  - sudo chown -R aic:aic /opt/aic
  - sudo chmod 600 /opt/aic/configuration/server.cfg
  - sudo chmod 600 /opt/aic/ca-certificates/*.key
- Windows: Use NTFS permissions appropriate for your organization
- Docker: Permissions handled by container configuration
Restart AI Controller service
- Linux (systemd service): sudo systemctl start aic
- Linux (manual): sudo -u aic /opt/aic/bin/AIC
- Windows (service): Start the AICService in Services Manager or use sc start AICService
- Windows (manual): Run bin\AIC.exe from the installation directory
- Docker: docker start aic-container or docker-compose up
Verify application functionality