# Deployment & Operations Comprehensive Guide

## Overview

This guide consolidates all deployment, operations, and maintenance documentation for the Sojorn platform, covering infrastructure setup, deployment procedures, monitoring, and operational best practices.

All code updates need to be made locally, synced to git, pulled to the server and deployed. Do not directly edit files on the server.

---

## Infrastructure Setup

### Server Requirements

#### Minimum Specifications
- **CPU**: 2 vCPU
- **Memory**: 4GB RAM
- **Storage**: 50GB SSD
- **OS**: Ubuntu 22.04 LTS
- **Network**: Public IP with domain pointing

#### Recommended Specifications
- **CPU**: 4 vCPU
- **Memory**: 8GB RAM
- **Storage**: 100GB SSD
- **OS**: Ubuntu 22.04 LTS
- **Network**: Public IP with SSL certificate

### System Dependencies

#### Core Packages
```bash
sudo apt update && sudo apt install -y \
  postgresql \
  postgresql-contrib \
  postgis \
  nginx \
  certbot \
  python3-certbot-nginx \
  git \
  curl \
  wget \
  htop
```

#### Go Environment
```bash
# Install Go 1.21+
wget https://go.dev/dl/go1.21.0.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.21.0.linux-amd64.tar.gz
echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc
source ~/.bashrc
```

### Database Setup

#### PostgreSQL Configuration
```bash
# Create database
sudo -u postgres createdb sojorn

# Create user
sudo -u postgres createuser --interactive sojorn_user

# Grant privileges
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE sojorn TO sojorn_user;"

# Enable extensions
sudo -u postgres psql sojorn -c "
CREATE EXTENSION IF NOT EXISTS uuid-ossp;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS postgis;
"
```

#### PostgreSQL Performance Tuning
```bash
# Edit postgresql.conf
sudo nano /etc/postgresql/15/main/postgresql.conf
```

Key settings:
```ini
# Memory settings
shared_buffers = 256MB
effective_cache_size = 1GB
work_mem = 4MB
maintenance_work_mem = 64MB

# Connection settings
max_connections = 100
listen_addresses = 'localhost'

# Logging settings
log_statement = 'all'
log_duration = on
log_min_duration_statement = 1000
```

---

## Application Deployment

### Deployment Structure

```
/opt/sojorn/
├── .env                    # Environment variables
├── firebase-service-account.json  # FCM credentials
├── uploads/                # User uploaded files
├── logs/                   # Application logs
└── go-backend/
    ├── bin/
    │   └── api            # Compiled binary
    ├── migrations/        # Database migrations
    └── scripts/          # Deployment scripts
```

### Environment Configuration

#### .env File Template
```bash
# Database
DATABASE_URL=postgres://sojorn_user:password@localhost:5432/sojorn?sslmode=disable

# Server
PORT=8080
HOST=localhost
ENVIRONMENT=production

# JWT
JWT_SECRET=your-super-secret-jwt-key-here
JWT_EXPIRY=24h
REFRESH_TOKEN_EXPIRY=168h

# Firebase/FCM
FIREBASE_CREDENTIALS_FILE=/opt/sojorn/firebase-service-account.json
FIREBASE_WEB_VAPID_KEY=BNxS7_your_vapid_key_here

# Storage
SENDER_API_TOKEN=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJhdWQiOiIxIiwianRpIjoiNT>
R2_ACCOUNT_ID=7041ca6e0f40307190dc2e65e2fb5e0f
R2_PUBLIC_BASE_URL=http://api.sojorn.net:8080/uploads
R2_IMG_DOMAIN=img.sojorn.net
R2_VID_DOMAIN=quips.sojorn.net
R2_API_TOKEN=oR7Vk0Realtx0D6SAGMuYA8pXopSoCYKv8t3JEuk

# Email (Optional)
# Email / SendPulse
SMTP_HOST=smtp-pulse.com
SMTP_PORT=587
SMTP_USER=patrickbritton3@gmail.com
SMTP_PASS=8s4jQBnAFTCXPNM

# Logging
LOG_LEVEL=info
LOG_FORMAT=json
```

### Build Process

#### Compile Go Backend
```bash
cd /opt/sojorn/go-backend

# Build for production
go build -ldflags="-s -w" -o bin/api ./cmd/api/main.go

# Make executable
chmod +x bin/api
```

#### Database Migrations
```bash
cd /opt/sojorn/go-backend

# Run migrations
make migrate-up

# Or manually
go run ./cmd/migrate/main.go up
```

### Systemd Service

#### Service Configuration
Create `/etc/systemd/system/sojorn-api.service`:

```ini
[Unit]
Description=Sojorn API Server
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=patrick
Group=patrick
WorkingDirectory=/opt/sojorn/go-backend
Environment=GO_ENV=production
EnvironmentFile=/opt/sojorn/.env
ExecStart=/opt/sojorn/go-backend/bin/api
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=sojorn-api

# Security
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/sojorn/uploads /opt/sojorn/logs

[Install]
WantedBy=multi-user.target
```

#### Service Management
```bash
# Reload systemd
sudo systemctl daemon-reload

# Enable service
sudo systemctl enable sojorn-api

# Start service
sudo systemctl start sojorn-api

# Check status
sudo systemctl status sojorn-api

# View logs
sudo journalctl -u sojorn-api -f
```

---

## Web Server Configuration

### Nginx Setup

#### Main Configuration
Create `/etc/nginx/sites-available/sojorn-api`:

```nginx
server {
    listen 80;
    server_name api.sojorn.net;
    
    # Redirect to HTTPS
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name api.sojorn.net;
    
    # SSL Configuration
    ssl_certificate /etc/letsencrypt/live/api.sojorn.net/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.sojorn.net/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    
    # Security Headers
    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    
    # Logging
    access_log /var/log/nginx/sojorn-api.access.log;
    error_log /var/log/nginx/sojorn-api.error.log;
    
    # Rate Limiting
    limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
    limit_req zone=api burst=20 nodelay;
    
    # API Proxy
    location / {
        proxy_pass http://localhost:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
        proxy_read_timeout 86400;
    }
    
    # File Uploads
    location /uploads/ {
        alias /opt/sojorn/uploads/;
        expires 1y;
        add_header Cache-Control "public, immutable";
        
        # Security for uploads
        location ~* \.(php|jsp|asp|sh|py)$ {
            deny all;
        }
    }
    
    # Health Check
    location /health {
        access_log off;
        proxy_pass http://localhost:8080/health;
    }
    
    # WebSocket Support (for chat)
    location /ws {
        proxy_pass http://localhost:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 86400;
        proxy_send_timeout 86400;
    }
}
```

#### Enable Site
```bash
# Enable site
sudo ln -s /etc/nginx/sites-available/sojorn-api /etc/nginx/sites-enabled/

# Test configuration
sudo nginx -t

# Reload Nginx
sudo systemctl reload nginx
```

### SSL Certificate Setup

#### Let's Encrypt Certificate
```bash
# Install Certbot
sudo apt install certbot python3-certbot-nginx

# Obtain certificate
sudo certbot --nginx -d api.sojorn.net

# Test renewal
sudo certbot renew --dry-run

# Set up auto-renewal
sudo crontab -e
# Add: 0 12 * * * /usr/bin/certbot renew --quiet
```

---

## Monitoring & Logging

### Application Monitoring

#### Health Check Endpoint
```go
// health_check.go
func (h *HealthHandler) Health(c *gin.Context) {
    status := map[string]interface{}{
        "status":    "healthy",
        "timestamp": time.Now(),
        "version":   os.Getenv("APP_VERSION"),
        "environment": os.Getenv("ENVIRONMENT"),
    }
    
    // Check database
    if err := h.db.Ping(); err != nil {
        status["status"] = "unhealthy"
        status["database"] = "disconnected"
        c.JSON(http.StatusServiceUnavailable, status)
        return
    }
    
    status["database"] = "connected"
    c.JSON(http.StatusOK, status)
}
```

#### Metrics Collection
```go
// metrics.go
var (
    httpRequestsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total number of HTTP requests",
        },
        []string{"method", "endpoint", "status"},
    )
    
    httpRequestDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name: "http_request_duration_seconds",
            Help: "HTTP request duration in seconds",
        },
        []string{"method", "endpoint"},
    )
)

func MetricsMiddleware() gin.HandlerFunc {
    return func(c *gin.Context) {
        start := time.Now()
        
        c.Next()
        
        duration := time.Since(start).Seconds()
        status := fmt.Sprintf("%d", c.Writer.Status())
        
        httpRequestsTotal.WithLabelValues(c.Request.Method, c.FullPath(), status).Inc()
        httpRequestDuration.WithLabelValues(c.Request.Method, c.FullPath()).Observe(duration)
    }
}
```

### System Monitoring

#### Setup Prometheus & Grafana (Optional)
```bash
# Install Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.40.0/prometheus-2.40.0.linux-amd64.tar.gz
tar xvfz prometheus-2.40.0.linux-amd64.tar.gz
sudo mv prometheus-2.40.0.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-2.40.0.linux-amd64/prometheus.yml /etc/prometheus/

# Install Grafana
sudo apt-get install -y adduser libfontconfig1
wget https://dl.grafana.com/oss/release/grafana_9.3.0_amd64.deb
sudo dpkg -i grafana_9.3.0_amd64.deb
```

#### Log Management
```bash
# Configure log rotation
sudo nano /etc/logrotate.d/sojorn-api
```

```
/opt/sojorn/logs/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 644 patrick patrick
    postrotate
        systemctl reload sojorn-api
    endscript
}
```

### Alerting

#### System Alerts Script
```bash
#!/bin/bash
# monitor.sh - System health monitoring

# Check service status
if ! systemctl is-active --quiet sojorn-api; then
    echo "ALERT: sojorn-api service is down" | mail -s "Sojorn Service Alert" admin@sojorn.com
fi

# Check disk space
DISK_USAGE=$(df /opt/sojorn | awk 'NR==2 {print $5}' | sed 's/%//')
if [ $DISK_USAGE -gt 80 ]; then
    echo "ALERT: Disk usage is ${DISK_USAGE}%" | mail -s "Sojorn Disk Alert" admin@sojorn.com
fi

# Check memory usage
MEM_USAGE=$(free | awk 'NR==2{printf "%.0f", $3*100/$2}')
if [ $MEM_USAGE -gt 90 ]; then
    echo "ALERT: Memory usage is ${MEM_USAGE}%" | mail -s "Sojorn Memory Alert" admin@sojorn.com
fi
```

---

## Backup & Recovery

### Database Backups

#### Automated Backup Script
```bash
#!/bin/bash
# backup.sh - Database backup script

BACKUP_DIR="/opt/sojorn/backups"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/sojorn_backup_$DATE.sql"

# Create backup directory
mkdir -p $BACKUP_DIR

# Create backup
pg_dump -h localhost -U sojorn_user sojorn > $BACKUP_FILE

# Compress backup
gzip $BACKUP_FILE

# Remove old backups (keep 7 days)
find $BACKUP_DIR -name "*.sql.gz" -mtime +7 -delete

echo "Backup completed: $BACKUP_FILE.gz"
```

#### Cron Job for Daily Backups
```bash
# Add to crontab
sudo crontab -e

# Daily backup at 2 AM
0 2 * * * /opt/sojorn/scripts/backup.sh
```

### File Backups

#### Upload Directory Backup
```bash
#!/bin/bash
# backup_uploads.sh

BACKUP_DIR="/opt/sojorn/backups"
DATE=$(date +%Y%m%d_%H%M%S)
UPLOAD_DIR="/opt/sojorn/uploads"

# Create compressed backup
tar -czf "$BACKUP_DIR/uploads_backup_$DATE.tar.gz" -C "$UPLOAD_DIR" .

# Remove old backups (keep 7 days)
find $BACKUP_DIR -name "uploads_backup_*.tar.gz" -mtime +7 -delete
```

### Recovery Procedures

#### Database Recovery
```bash
# Stop application
sudo systemctl stop sojorn-api

# Restore database
gunzip -c /opt/sojorn/backups/sojorn_backup_20260130_020000.sql.gz | \
psql -h localhost -U sojorn_user -d sojorn

# Start application
sudo systemctl start sojorn-api

# Verify recovery
curl -f https://api.sojorn.net/health
```

#### File Recovery
```bash
# Restore uploads
tar -xzf /opt/sojorn/backups/uploads_backup_20260130_020000.tar.gz \
-C /opt/sojorn/uploads

# Fix permissions
sudo chown -R patrick:patrick /opt/sojorn/uploads
sudo chmod -R 755 /opt/sojorn/uploads
```

---

## Security Hardening

### System Security

#### Firewall Configuration
```bash
# Install UFW
sudo apt install ufw

# Default policies
sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow SSH
sudo ufw allow ssh

# Allow HTTP/HTTPS
sudo ufw allow 80
sudo ufw allow 443

# Enable firewall
sudo ufw enable
```

#### SSH Security
```bash
# Edit SSH config
sudo nano /etc/ssh/sshd_config
```

Key settings:
```
# Disable root login
PermitRootLogin no

# Use key-based authentication
PasswordAuthentication no
PubkeyAuthentication yes

# Change default port (optional)
Port 2222

# Limit login attempts
MaxAuthTries 3
```

#### Fail2Ban Setup
```bash
# Install Fail2Ban
sudo apt install fail2ban

# Configure for SSH
sudo nano /etc/fail2ban/jail.local
```

```
[DEFAULT]
bantime = 3600
findtime = 600
maxretry = 3

[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
```

### Application Security

#### Environment Variable Protection
```bash
# Secure .env file
sudo chmod 600 /opt/sojorn/.env
sudo chown patrick:patrick /opt/sojorn/.env

# Secure service account file
sudo chmod 600 /opt/sojorn/firebase-service-account.json
sudo chown patrick:patrick /opt/sojorn/firebase-service-account.json
```

#### SSL/TLS Hardening
```nginx
# Add to Nginx config
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;

# HSTS
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

# Other security headers
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header Referrer-Policy "strict-origin-when-cross-origin";
```

---

## Performance Optimization

### Database Optimization

#### Query Optimization
```sql
-- Add indexes for common queries
CREATE INDEX CONCURRENTLY idx_posts_author_created ON posts(author_id, created_at DESC);
CREATE INDEX CONCURRENTLY idx_posts_category_created ON posts(category_id, created_at DESC) WHERE category_id IS NOT NULL;
CREATE INDEX CONCURRENTLY idx_follows_follower ON follows(follower_id);
CREATE INDEX CONCURRENTLY idx_follows_following ON follows(following_id);

-- Analyze table statistics
ANALYZE posts;
ANALYZE follows;
ANALYZE users;
```

#### Connection Pool Tuning
```go
// In database setup
config, err := pgxpool.ParseConfig(databaseURL)
if err != nil {
    return nil, err
}

config.MaxConns = 25
config.MinConns = 5
config.MaxConnLifetime = time.Hour
config.HealthCheckPeriod = time.Minute * 5
config.MaxConnIdleTime = time.Minute * 30
```

### Caching Strategy

#### Redis Integration (Optional)
```bash
# Install Redis
sudo apt install redis-server

# Configure Redis
sudo nano /etc/redis/redis.conf
```

Key settings:
```
# Memory management
maxmemory 256mb
maxmemory-policy allkeys-lru

# Persistence
save 900 1
save 300 10
save 60 10000
```

#### Application Caching
```go
// Cache implementation
type CacheService struct {
    redis *redis.Client
}

func (c *CacheService) Set(key string, value interface{}, expiration time.Duration) error {
    json, err := json.Marshal(value)
    if err != nil {
        return err
    }
    return c.redis.Set(context.Background(), key, json, expiration).Err()
}

func (c *CacheService) Get(key string, dest interface{}) error {
    val, err := c.redis.Get(context.Background(), key).Result()
    if err != nil {
        return err
    }
    return json.Unmarshal([]byte(val), dest)
}
```

---

## Deployment Automation

### Deployment Script

#### Automated Deploy Script
```bash
#!/bin/bash
# deploy.sh - Automated deployment script

set -e  # Exit on any error

# Configuration
REPO_URL="https://github.com/your-org/sojorn.git"
DEPLOY_DIR="/opt/sojorn"
BACKUP_DIR="/opt/sojorn/backups"
SERVICE_NAME="sojorn-api"

echo "Starting deployment..."

# Create backup
echo "Creating backup..."
$DEPLOY_DIR/scripts/backup.sh

# Pull latest code
echo "Pulling latest code..."
cd $DEPLOY_DIR
git pull origin main

# Build application
echo "Building application..."
cd go-backend
go build -ldflags="-s -w" -o bin/api ./cmd/api/main.go

# Run migrations
echo "Running database migrations..."
make migrate-up

# Restart service
echo "Restarting service..."
sudo systemctl restart $SERVICE_NAME

# Wait for service to start
sleep 5

# Health check
echo "Performing health check..."
if curl -f http://localhost:8080/health; then
    echo "✅ Deployment successful!"
else
    echo "❌ Health check failed!"
    echo "Rolling back..."
    sudo systemctl restart $SERVICE_NAME
    exit 1
fi

echo "Deployment completed successfully!"
```

### CI/CD Pipeline (Optional)

#### GitHub Actions Example
```yaml
name: Deploy to Production

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v3
    
    - name: Setup Go
      uses: actions/setup-go@v3
      with:
        go-version: '1.21'
    
    - name: Build application
      run: |
        cd go-backend
        go build -ldflags="-s -w" -o bin/api ./cmd/api/main.go
    
    - name: Deploy to server
      uses: appleboy/ssh-action@v0.1.5
      with:
        host: ${{ secrets.HOST }}
        username: ${{ secrets.USERNAME }}
        key: ${{ secrets.SSH_KEY }}
        script: |
          cd /opt/sojorn
          ./scripts/deploy.sh
```

---

## Troubleshooting Common Issues

### Service Startup Issues

#### Problem: Service fails to start
```bash
# Check service status
sudo systemctl status sojorn-api

# Check logs
sudo journalctl -u sojorn-api -n 50

# Common fixes
# 1. Check .env file permissions
sudo chmod 600 /opt/sojorn/.env

# 2. Check database connection
sudo -u postgres psql -c "SELECT 1;"

# 3. Check port availability
sudo netstat -tlnp | grep :8080
```

#### Problem: Database connection errors
```bash
# Check PostgreSQL status
sudo systemctl status postgresql

# Check connection string
sudo cat /opt/sojorn/.env | grep DATABASE_URL

# Test connection manually
psql "postgres://sojorn_user:password@localhost:5432/sojorn"
```

### Performance Issues

#### Problem: Slow API responses
```bash
# Check system resources
top
htop
iostat -x 1

# Check database queries
sudo -u postgres psql -c "
SELECT query, mean_time, calls 
FROM pg_stat_statements 
ORDER BY mean_time DESC 
LIMIT 10;"

# Check Nginx logs
sudo tail -f /var/log/nginx/sojorn-api.access.log
```

#### Problem: High memory usage
```bash
# Check memory usage
free -h
ps aux --sort=-%mem | head

# Restart service if needed
sudo systemctl restart sojorn-api

# Consider tuning Go memory settings
# In .env: GOMEMLIMIT=256MiB
```

### SSL/TLS Issues

#### Problem: Certificate expired
```bash
# Check certificate expiry
sudo certbot certificates

# Renew certificate
sudo certbot renew

# Reload Nginx
sudo systemctl reload nginx
```

#### Problem: SSL handshake failed
```bash
# Test SSL configuration
openssl s_client -connect api.sojorn.net:443

# Check Nginx configuration
sudo nginx -t

# Reload Nginx
sudo systemctl reload nginx
```

---

## Maintenance Schedule

### Daily Tasks
- [ ] Check service status and logs
- [ ] Monitor system resources
- [ ] Verify backup completion
- [ ] Review error rates

### Weekly Tasks
- [ ] Update security patches
- [ ] Review performance metrics
- [ ] Check disk space usage
- [ ] Test backup restoration

### Monthly Tasks
- [ ] Update application dependencies
- [ ] Review and rotate secrets
- [ ] Audit user access and permissions
- [ ] Performance optimization review

### Quarterly Tasks
- [ ] Security audit and penetration testing
- [ ] Disaster recovery testing
- [ ] Capacity planning review
- [ ] Documentation updates

---

## Emergency Procedures

### Service Outage Response

1. **Immediate Assessment**
   ```bash
   # Check service status
   sudo systemctl status sojorn-api
   
   # Check system resources
   top
   df -h
   
   # Check network connectivity
   ping 8.8.8.8
   ```

2. **Service Recovery**
   ```bash
   # Restart services
   sudo systemctl restart sojorn-api
   sudo systemctl restart nginx
   sudo systemctl restart postgresql
   
   # Verify recovery
   curl -f https://api.sojorn.net/health
   ```

3. **Communication**
   - Notify stakeholders of outage
   - Provide ETA for resolution
   - Document incident and resolution

### Security Incident Response

1. **Containment**
   - Isolate affected systems
   - Preserve evidence
   - Change credentials

2. **Investigation**
   - Review logs and audit trails
   - Identify breach scope
   - Document findings

3. **Recovery**
   - Patch vulnerabilities
   - Restore from clean backups
   - Monitor for suspicious activity

---

**Last Updated**: January 30, 2026
**Version**: 1.0
**Next Review**: February 15, 2026