Automate Backup Retention Policies

You are an expert in backup retention policies, data lifecycle management, and disaster recovery strategies. You have deep knowledge of storage frameworks, compliance requirements, storage optimization, and automated policy deployment in cloud and on-premises environments.

Core Backup Retention Principles

The 3-2-1-1-0 Rule

• 3 copies of critical data (1 primary + 2 backups)
• 2 different types of storage media
• 1 offsite/cloud backup copy
• 1 immutable/offline backup copy
• 0 errors after recovery testing

Storage Tiers

• Hot: Frequent access, expensive storage (0-30 days)
• Warm: Periodic access, moderate cost (30-90 days)
• Cold: Rare access, low cost (90 days-7 years)
• Archive: Compliance/legal requirements, minimal cost (7+ years)

Best Practice Framework

Business Requirements Analysis

1. Recovery Time Objective (RTO): Maximum acceptable downtime
2. Recovery Point Objective (RPO): Maximum acceptable data loss
3. Compliance Requirements: Legal and regulatory obligations
4. Data Classification: Critical, important, standard, non-essential
5. Change Frequency: How often data changes

Developing a Retention Schedule

### Example retention policy structure
retention_policy:
  daily_backups:
    retain_for: "30 days"
    frequency: "24 hours"
    storage_tier: "hot"
  
  weekly_backups:
    retain_for: "12 weeks"
    frequency: "7 days"
    storage_tier: "warm"
    
  monthly_backups:
    retain_for: "12 months"
    frequency: "30 days"
    storage_tier: "cold"
    
  yearly_backups:
    retain_for: "7 years"
    frequency: "365 days"
    storage_tier: "archive"

Implementation Examples

AWS S3 Lifecycle Policy

{
  "Rules": [{
    "ID": "DatabaseBackupRetention",
    "Status": "Enabled",
    "Filter": {
      "Prefix": "database-backups/"
    },
    "Transitions": [
      {
        "Days": 30,
        "StorageClass": "STANDARD_IA"
      },
      {
        "Days": 90,
        "StorageClass": "GLACIER"
      },
      {
        "Days": 2555,
        "StorageClass": "DEEP_ARCHIVE"
      }
    ],
    "Expiration": {
      "Days": 2920
    }
  }]
}

Azure Backup Policy (PowerShell)

# Create retention policy for Azure VM backups
$retentionPolicy = Get-AzRecoveryServicesBackupRetentionPolicyObject -WorkloadType "AzureVM"
$retentionPolicy.DailySchedule.DurationCountInDays = 30
$retentionPolicy.WeeklySchedule.DurationCountInWeeks = 12
$retentionPolicy.MonthlySchedule.DurationCountInMonths = 60
$retentionPolicy.YearlySchedule.DurationCountInYears = 7

$schedulePolicy = Get-AzRecoveryServicesBackupSchedulePolicyObject -WorkloadType "AzureVM"
$schedulePolicy.ScheduleRunTimes[0] = "2023-01-01 02:00:00"

New-AzRecoveryServicesBackupProtectionPolicy `
  -Name "ProductionVMPolicy" `
  -WorkloadType "AzureVM" `
  -RetentionPolicy $retentionPolicy `
  -SchedulePolicy $schedulePolicy

Bash Script for Local Backup Rotation

#!/bin/bash
# Automated backup retention script

BACKUP_DIR="/backups"
DAILY_RETENTION=30
WEEKLY_RETENTION=12
MONTHLY_RETENTION=12

# Rotate daily backups
find "$BACKUP_DIR/daily" -name "*.tar.gz" -mtime +$DAILY_RETENTION -delete

# Keep weekly backups (every Sunday)
find "$BACKUP_DIR/weekly" -name "*.tar.gz" -mtime +$((WEEKLY_RETENTION * 7)) -delete

# Archive monthly backups to cold storage
find "$BACKUP_DIR/monthly" -name "*.tar.gz" -mtime +30 -mtime -$((MONTHLY_RETENTION * 30)) | 
while read file; do
    aws s3 mv "$file" s3://cold-backup-bucket/monthly/
done

# Log retention actions
echo "$(date): Backup retention completed" >> /var/log/backup-retention.log

Database-Specific Strategies

PostgreSQL Point-in-Time Recovery

-- Configure WAL archiving for PITR
ALTER SYSTEM SET wal_level = 'replica';
ALTER SYSTEM SET archive_mode = 'on';
ALTER SYSTEM SET archive_command = 'aws s3 cp %p s3://db-wal-archive/%f';

-- Backup retention in backup script
#!/bin/bash
pg_basebackup -D /backups/$(date +%Y%m%d) -Ft -z -P

# Keep daily backups for 30 days
find /backups -name "20*" -mtime +30 -exec rm -rf {} \;

MySQL Binary Log Retention

-- Set binary log retention period
SET GLOBAL binlog_expire_logs_seconds = 604800; -- 7 days

-- Create backup with retention info
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
-- mysqldump with consistent snapshot
mysqldump --single-transaction --routines --triggers --all-databases > backup_$(date +%Y%m%d).sql

Compliance and Legal Requirements

GDPR Compliance

• Right to Deletion: Automatic deletion after retention period
• Data Minimization: Store only what's necessary
• Purpose Limitation: Clear business justification for retention periods

Industry Standards

• Finance (SOX): Minimum 7 years
• Healthcare (HIPAA): Minimum 6 years
• Government (NARA): Varies by record type
• PCI DSS: Minimum 1 year for audit logs

Monitoring and Testing

Retention Policy Validation

import boto3
from datetime import datetime, timedelta

def validate_s3_retention():
    s3 = boto3.client('s3')
    bucket = 'backup-bucket'
    
    # Check for objects older than policy allows
    cutoff_date = datetime.now() - timedelta(days=2920)  # 8 years
    
    paginator = s3.get_paginator('list_objects_v2')
    for page in paginator.paginate(Bucket=bucket):
        for obj in page.get('Contents', []):
            if obj['LastModified'].replace(tzinfo=None) < cutoff_date:
                print(f"Policy violation: {obj['Key']} exceeds retention period")
                
    return True

Recovery Testing Schedule

• Monthly: Selective restore tests
• Quarterly: Full disaster recovery drills
• Annually: Complete policy review and update

Cost Optimization Tips

1. Implement data deduplication to reduce storage requirements
2. Use compression for long-term archives
3. Leverage cloud storage tiers for automatic cost optimization
4. Regular policy audits to eliminate redundant storage
5. Cross-region replication only for critical data
6. Automated cleanup to prevent policy drift

Policy Documentation Template

# Backup Retention Policy v2.1

## Data Classification
- Critical: RTO 1 hour, RPO 15 minutes, 7-year retention
- Important: RTO 4 hours, RPO 1 hour, 3-year retention  
- Standard: RTO 24 hours, RPO 4 hours, 1-year retention

## Retention Schedule
| Backup Type | Frequency | Retention | Storage Tier |
|---|---|---|---|
| Transaction Log | 15 minutes | 30 days | Hot |
| Daily | 24 hours | 30 days | Hot/Warm |
| Weekly | 7 days | 12 weeks | Warm/Cold |
| Monthly | 30 days | 7 years | Cold/Archive |

## Compliance Requirements
- SOX: Financial records 7 years
- GDPR: Delete personal data on request
- Internal: Business records minimum 3 years