# Deployment & Operations Comprehensive Guide

## Overview

This guide consolidates all deployment, operations, and maintenance documentation for the Sojorn platform, covering infrastructure setup, deployment procedures, monitoring, and operational best practices.

---

## Infrastructure Setup

### Server Requirements

#### Minimum Specifications
- **CPU**: 2 vCPU
- **Memory**: 4GB RAM
- **Storage**: 50GB SSD
- **OS**: Ubuntu 22.04 LTS
- **Network**: Public IP with domain pointing

#### Recommended Specifications
- **CPU**: 4 vCPU
- **Memory**: 8GB RAM
- **Storage**: 100GB SSD
- **OS**: Ubuntu 22.04 LTS
- **Network**: Public IP with SSL certificate

### System Dependencies

#### Core Packages
```bash
sudo apt update && sudo apt install -y \
  postgresql \
  postgresql-contrib \
  postgis \
  nginx \
  certbot \
  python3-certbot-nginx \
  git \
  curl \
  wget \
  htop
```

#### Go Environment
```bash
# Install Go 1.21+
wget https://go.dev/dl/go1.21.0.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.21.0.linux-amd64.tar.gz
echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc
source ~/.bashrc
```

### Database Setup

#### PostgreSQL Configuration
```bash
# Create database
sudo -u postgres createdb sojorn

# Create user
sudo -u postgres createuser --interactive sojorn_user

# Grant privileges
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE sojorn TO sojorn_user;"

# Enable extensions
sudo -u postgres psql sojorn -c "
CREATE EXTENSION IF NOT EXISTS uuid-ossp;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS postgis;
"
```

#### PostgreSQL Performance Tuning
```bash
# Edit postgresql.conf
sudo nano /etc/postgresql/15/main/postgresql.conf
```

Key settings:
```ini
# Memory settings
shared_buffers = 256MB
effective_cache_size = 1GB
work_mem = 4MB
maintenance_work_mem = 64MB

# Connection settings
max_connections = 100
listen_addresses = 'localhost'

# Logging settings
log_statement = 'all'
log_duration = on
log_min_duration_statement = 1000
```

---

## Application Deployment

### Deployment Structure

```
/opt/sojorn/
├── .env                    # Environment variables
├── firebase-service-account.json  # FCM credentials
├── uploads/                # User uploaded files
├── logs/                   # Application logs
└── go-backend/
    ├── bin/
    │   └── api            # Compiled binary
    ├── migrations/        # Database migrations
    └── scripts/          # Deployment scripts
```

### Environment Configuration

#### .env File Template
```bash
# Database
DATABASE_URL=postgres://sojorn_user:password@localhost:5432/sojorn?sslmode=disable

# Server
PORT=8080
HOST=localhost
ENVIRONMENT=production

# JWT
JWT_SECRET=your-super-secret-jwt-key-here
JWT_EXPIRY=24h
REFRESH_TOKEN_EXPIRY=168h

# Firebase/FCM
FIREBASE_CREDENTIALS_FILE=/opt/sojorn/firebase-service-account.json
FIREBASE_WEB_VAPID_KEY=BNxS7_your_vapid_key_here

# Storage
R2_ACCOUNT_ID=your-r2-account-id
R2_ACCESS_KEY_ID=your-access-key
R2_SECRET_ACCESS_KEY=your-secret-key
R2_BUCKET_NAME=sojorn-uploads
R2_PUBLIC_BASE_URL=https://pub-xxxxx.r2.dev
R2_IMG_DOMAIN=img.sojorn.com
R2_VID_DOMAIN=vid.sojorn.com

# Email (Optional)
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=noreply@sojorn.com
SMTP_PASS=your-app-password

# Logging
LOG_LEVEL=info
LOG_FORMAT=json
```

### Build Process

#### Compile Go Backend
```bash
cd /opt/sojorn/go-backend

# Build for production
go build -ldflags="-s -w" -o bin/api ./cmd/api/main.go

# Make executable
chmod +x bin/api
```

#### Database Migrations
```bash
cd /opt/sojorn/go-backend

# Run migrations
make migrate-up

# Or manually
go run ./cmd/migrate/main.go up
```

### Systemd Service

#### Service Configuration
Create `/etc/systemd/system/sojorn-api.service`:

```ini
[Unit]
Description=Sojorn API Server
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=patrick
Group=patrick
WorkingDirectory=/opt/sojorn/go-backend
Environment=GO_ENV=production
EnvironmentFile=/opt/sojorn/.env
ExecStart=/opt/sojorn/go-backend/bin/api
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=sojorn-api

# Security
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/sojorn/uploads /opt/sojorn/logs

[Install]
WantedBy=multi-user.target
```

#### Service Management
```bash
# Reload systemd
sudo systemctl daemon-reload

# Enable service
sudo systemctl enable sojorn-api

# Start service
sudo systemctl start sojorn-api

# Check status
sudo systemctl status sojorn-api

# View logs
sudo journalctl -u sojorn-api -f
```

---

## Web Server Configuration

### Nginx Setup

#### Main Configuration
Create `/etc/nginx/sites-available/sojorn-api`:

```nginx
server {
    listen 80;
    server_name api.gosojorn.com;
    
    # Redirect to HTTPS
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name api.gosojorn.com;
    
    # SSL Configuration
    ssl_certificate /etc/letsencrypt/live/api.gosojorn.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.gosojorn.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    
    # Security Headers
    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    
    # Logging
    access_log /var/log/nginx/sojorn-api.access.log;
    error_log /var/log/nginx/sojorn-api.error.log;
    
    # Rate Limiting
    limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
    limit_req zone=api burst=20 nodelay;
    
    # API Proxy
    location / {
        proxy_pass http://localhost:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
        proxy_read_timeout 86400;
    }
    
    # File Uploads
    location /uploads/ {
        alias /opt/sojorn/uploads/;
        expires 1y;
        add_header Cache-Control "public, immutable";
        
        # Security for uploads
        location ~* \.(php|jsp|asp|sh|py)$ {
            deny all;
        }
    }
    
    # Health Check
    location /health {
        access_log off;
        proxy_pass http://localhost:8080/health;
    }
    
    # WebSocket Support (for chat)
    location /ws {
        proxy_pass http://localhost:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 86400;
        proxy_send_timeout 86400;
    }
}
```

#### Enable Site
```bash
# Enable site
sudo ln -s /etc/nginx/sites-available/sojorn-api /etc/nginx/sites-enabled/

# Test configuration
sudo nginx -t

# Reload Nginx
sudo systemctl reload nginx
```

### SSL Certificate Setup

#### Let's Encrypt Certificate
```bash
# Install Certbot
sudo apt install certbot python3-certbot-nginx

# Obtain certificate
sudo certbot --nginx -d api.gosojorn.com

# Test renewal
sudo certbot renew --dry-run

# Set up auto-renewal
sudo crontab -e
# Add: 0 12 * * * /usr/bin/certbot renew --quiet
```

---

## Monitoring & Logging

### Application Monitoring

#### Health Check Endpoint
```go
// health_check.go
func (h *HealthHandler) Health(c *gin.Context) {
    status := map[string]interface{}{
        "status":    "healthy",
        "timestamp": time.Now(),
        "version":   os.Getenv("APP_VERSION"),
        "environment": os.Getenv("ENVIRONMENT"),
    }
    
    // Check database
    if err := h.db.Ping(); err != nil {
        status["status"] = "unhealthy"
        status["database"] = "disconnected"
        c.JSON(http.StatusServiceUnavailable, status)
        return
    }
    
    status["database"] = "connected"
    c.JSON(http.StatusOK, status)
}
```

#### Metrics Collection
```go
// metrics.go
var (
    httpRequestsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total number of HTTP requests",
        },
        []string{"method", "endpoint", "status"},
    )
    
    httpRequestDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name: "http_request_duration_seconds",
            Help: "HTTP request duration in seconds",
        },
        []string{"method", "endpoint"},
    )
)

func MetricsMiddleware() gin.HandlerFunc {
    return func(c *gin.Context) {
        start := time.Now()
        
        c.Next()
        
        duration := time.Since(start).Seconds()
        status := fmt.Sprintf("%d", c.Writer.Status())
        
        httpRequestsTotal.WithLabelValues(c.Request.Method, c.FullPath(), status).Inc()
        httpRequestDuration.WithLabelValues(c.Request.Method, c.FullPath()).Observe(duration)
    }
}
```

### System Monitoring

#### Setup Prometheus & Grafana (Optional)
```bash
# Install Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.40.0/prometheus-2.40.0.linux-amd64.tar.gz
tar xvfz prometheus-2.40.0.linux-amd64.tar.gz
sudo mv prometheus-2.40.0.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-2.40.0.linux-amd64/prometheus.yml /etc/prometheus/

# Install Grafana
sudo apt-get install -y adduser libfontconfig1
wget https://dl.grafana.com/oss/release/grafana_9.3.0_amd64.deb
sudo dpkg -i grafana_9.3.0_amd64.deb
```

#### Log Management
```bash
# Configure log rotation
sudo nano /etc/logrotate.d/sojorn-api
```

```
/opt/sojorn/logs/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 644 patrick patrick
    postrotate
        systemctl reload sojorn-api
    endscript
}
```

### Alerting

#### System Alerts Script
```bash
#!/bin/bash
# monitor.sh - System health monitoring

# Check service status
if ! systemctl is-active --quiet sojorn-api; then
    echo "ALERT: sojorn-api service is down" | mail -s "Sojorn Service Alert" admin@sojorn.com
fi

# Check disk space
DISK_USAGE=$(df /opt/sojorn | awk 'NR==2 {print $5}' | sed 's/%//')
if [ $DISK_USAGE -gt 80 ]; then
    echo "ALERT: Disk usage is ${DISK_USAGE}%" | mail -s "Sojorn Disk Alert" admin@sojorn.com
fi

# Check memory usage
MEM_USAGE=$(free | awk 'NR==2{printf "%.0f", $3*100/$2}')
if [ $MEM_USAGE -gt 90 ]; then
    echo "ALERT: Memory usage is ${MEM_USAGE}%" | mail -s "Sojorn Memory Alert" admin@sojorn.com
fi
```

---

## Backup & Recovery

### Database Backups

#### Automated Backup Script
```bash
#!/bin/bash
# backup.sh - Database backup script

BACKUP_DIR="/opt/sojorn/backups"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/sojorn_backup_$DATE.sql"

# Create backup directory
mkdir -p $BACKUP_DIR

# Create backup
pg_dump -h localhost -U sojorn_user sojorn > $BACKUP_FILE

# Compress backup
gzip $BACKUP_FILE

# Remove old backups (keep 7 days)
find $BACKUP_DIR -name "*.sql.gz" -mtime +7 -delete

echo "Backup completed: $BACKUP_FILE.gz"
```

#### Cron Job for Daily Backups
```bash
# Add to crontab
sudo crontab -e

# Daily backup at 2 AM
0 2 * * * /opt/sojorn/scripts/backup.sh
```

### File Backups

#### Upload Directory Backup
```bash
#!/bin/bash
# backup_uploads.sh

BACKUP_DIR="/opt/sojorn/backups"
DATE=$(date +%Y%m%d_%H%M%S)
UPLOAD_DIR="/opt/sojorn/uploads"

# Create compressed backup
tar -czf "$BACKUP_DIR/uploads_backup_$DATE.tar.gz" -C "$UPLOAD_DIR" .

# Remove old backups (keep 7 days)
find $BACKUP_DIR -name "uploads_backup_*.tar.gz" -mtime +7 -delete
```

### Recovery Procedures

#### Database Recovery
```bash
# Stop application
sudo systemctl stop sojorn-api

# Restore database
gunzip -c /opt/sojorn/backups/sojorn_backup_20260130_020000.sql.gz | \
psql -h localhost -U sojorn_user -d sojorn

# Start application
sudo systemctl start sojorn-api

# Verify recovery
curl -f https://api.gosojorn.com/health
```

#### File Recovery
```bash
# Restore uploads
tar -xzf /opt/sojorn/backups/uploads_backup_20260130_020000.tar.gz \
-C /opt/sojorn/uploads

# Fix permissions
sudo chown -R patrick:patrick /opt/sojorn/uploads
sudo chmod -R 755 /opt/sojorn/uploads
```

---

## Security Hardening

### System Security

#### Firewall Configuration
```bash
# Install UFW
sudo apt install ufw

# Default policies
sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow SSH
sudo ufw allow ssh

# Allow HTTP/HTTPS
sudo ufw allow 80
sudo ufw allow 443

# Enable firewall
sudo ufw enable
```

#### SSH Security
```bash
# Edit SSH config
sudo nano /etc/ssh/sshd_config
```

Key settings:
```
# Disable root login
PermitRootLogin no

# Use key-based authentication
PasswordAuthentication no
PubkeyAuthentication yes

# Change default port (optional)
Port 2222

# Limit login attempts
MaxAuthTries 3
```

#### Fail2Ban Setup
```bash
# Install Fail2Ban
sudo apt install fail2ban

# Configure for SSH
sudo nano /etc/fail2ban/jail.local
```

```
[DEFAULT]
bantime = 3600
findtime = 600
maxretry = 3

[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
```

### Application Security

#### Environment Variable Protection
```bash
# Secure .env file
sudo chmod 600 /opt/sojorn/.env
sudo chown patrick:patrick /opt/sojorn/.env

# Secure service account file
sudo chmod 600 /opt/sojorn/firebase-service-account.json
sudo chown patrick:patrick /opt/sojorn/firebase-service-account.json
```

#### SSL/TLS Hardening
```nginx
# Add to Nginx config
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;

# HSTS
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

# Other security headers
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header Referrer-Policy "strict-origin-when-cross-origin";
```

---

## Performance Optimization

### Database Optimization

#### Query Optimization
```sql
-- Add indexes for common queries
CREATE INDEX CONCURRENTLY idx_posts_author_created ON posts(author_id, created_at DESC);
CREATE INDEX CONCURRENTLY idx_posts_category_created ON posts(category_id, created_at DESC) WHERE category_id IS NOT NULL;
CREATE INDEX CONCURRENTLY idx_follows_follower ON follows(follower_id);
CREATE INDEX CONCURRENTLY idx_follows_following ON follows(following_id);

-- Analyze table statistics
ANALYZE posts;
ANALYZE follows;
ANALYZE users;
```

#### Connection Pool Tuning
```go
// In database setup
config, err := pgxpool.ParseConfig(databaseURL)
if err != nil {
    return nil, err
}

config.MaxConns = 25
config.MinConns = 5
config.MaxConnLifetime = time.Hour
config.HealthCheckPeriod = time.Minute * 5
config.MaxConnIdleTime = time.Minute * 30
```

### Caching Strategy

#### Redis Integration (Optional)
```bash
# Install Redis
sudo apt install redis-server

# Configure Redis
sudo nano /etc/redis/redis.conf
```

Key settings:
```
# Memory management
maxmemory 256mb
maxmemory-policy allkeys-lru

# Persistence
save 900 1
save 300 10
save 60 10000
```

#### Application Caching
```go
// Cache implementation
type CacheService struct {
    redis *redis.Client
}

func (c *CacheService) Set(key string, value interface{}, expiration time.Duration) error {
    json, err := json.Marshal(value)
    if err != nil {
        return err
    }
    return c.redis.Set(context.Background(), key, json, expiration).Err()
}

func (c *CacheService) Get(key string, dest interface{}) error {
    val, err := c.redis.Get(context.Background(), key).Result()
    if err != nil {
        return err
    }
    return json.Unmarshal([]byte(val), dest)
}
```

---

## Deployment Automation

### Deployment Script

#### Automated Deploy Script
```bash
#!/bin/bash
# deploy.sh - Automated deployment script

set -e  # Exit on any error

# Configuration
REPO_URL="https://github.com/your-org/sojorn.git"
DEPLOY_DIR="/opt/sojorn"
BACKUP_DIR="/opt/sojorn/backups"
SERVICE_NAME="sojorn-api"

echo "Starting deployment..."

# Create backup
echo "Creating backup..."
$DEPLOY_DIR/scripts/backup.sh

# Pull latest code
echo "Pulling latest code..."
cd $DEPLOY_DIR
git pull origin main

# Build application
echo "Building application..."
cd go-backend
go build -ldflags="-s -w" -o bin/api ./cmd/api/main.go

# Run migrations
echo "Running database migrations..."
make migrate-up

# Restart service
echo "Restarting service..."
sudo systemctl restart $SERVICE_NAME

# Wait for service to start
sleep 5

# Health check
echo "Performing health check..."
if curl -f http://localhost:8080/health; then
    echo "✅ Deployment successful!"
else
    echo "❌ Health check failed!"
    echo "Rolling back..."
    sudo systemctl restart $SERVICE_NAME
    exit 1
fi

echo "Deployment completed successfully!"
```

### CI/CD Pipeline (Optional)

#### GitHub Actions Example
```yaml
name: Deploy to Production

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v3
    
    - name: Setup Go
      uses: actions/setup-go@v3
      with:
        go-version: '1.21'
    
    - name: Build application
      run: |
        cd go-backend
        go build -ldflags="-s -w" -o bin/api ./cmd/api/main.go
    
    - name: Deploy to server
      uses: appleboy/ssh-action@v0.1.5
      with:
        host: ${{ secrets.HOST }}
        username: ${{ secrets.USERNAME }}
        key: ${{ secrets.SSH_KEY }}
        script: |
          cd /opt/sojorn
          ./scripts/deploy.sh
```

---

## Troubleshooting Common Issues

### Service Startup Issues

#### Problem: Service fails to start
```bash
# Check service status
sudo systemctl status sojorn-api

# Check logs
sudo journalctl -u sojorn-api -n 50

# Common fixes
# 1. Check .env file permissions
sudo chmod 600 /opt/sojorn/.env

# 2. Check database connection
sudo -u postgres psql -c "SELECT 1;"

# 3. Check port availability
sudo netstat -tlnp | grep :8080
```

#### Problem: Database connection errors
```bash
# Check PostgreSQL status
sudo systemctl status postgresql

# Check connection string
sudo cat /opt/sojorn/.env | grep DATABASE_URL

# Test connection manually
psql "postgres://sojorn_user:password@localhost:5432/sojorn"
```

### Performance Issues

#### Problem: Slow API responses
```bash
# Check system resources
top
htop
iostat -x 1

# Check database queries
sudo -u postgres psql -c "
SELECT query, mean_time, calls 
FROM pg_stat_statements 
ORDER BY mean_time DESC 
LIMIT 10;"

# Check Nginx logs
sudo tail -f /var/log/nginx/sojorn-api.access.log
```

#### Problem: High memory usage
```bash
# Check memory usage
free -h
ps aux --sort=-%mem | head

# Restart service if needed
sudo systemctl restart sojorn-api

# Consider tuning Go memory settings
# In .env: GOMEMLIMIT=256MiB
```

### SSL/TLS Issues

#### Problem: Certificate expired
```bash
# Check certificate expiry
sudo certbot certificates

# Renew certificate
sudo certbot renew

# Reload Nginx
sudo systemctl reload nginx
```

#### Problem: SSL handshake failed
```bash
# Test SSL configuration
openssl s_client -connect api.gosojorn.com:443

# Check Nginx configuration
sudo nginx -t

# Reload Nginx
sudo systemctl reload nginx
```

---

## Maintenance Schedule

### Daily Tasks
- [ ] Check service status and logs
- [ ] Monitor system resources
- [ ] Verify backup completion
- [ ] Review error rates

### Weekly Tasks
- [ ] Update security patches
- [ ] Review performance metrics
- [ ] Check disk space usage
- [ ] Test backup restoration

### Monthly Tasks
- [ ] Update application dependencies
- [ ] Review and rotate secrets
- [ ] Audit user access and permissions
- [ ] Performance optimization review

### Quarterly Tasks
- [ ] Security audit and penetration testing
- [ ] Disaster recovery testing
- [ ] Capacity planning review
- [ ] Documentation updates

---

## Emergency Procedures

### Service Outage Response

1. **Immediate Assessment**
   ```bash
   # Check service status
   sudo systemctl status sojorn-api
   
   # Check system resources
   top
   df -h
   
   # Check network connectivity
   ping 8.8.8.8
   ```

2. **Service Recovery**
   ```bash
   # Restart services
   sudo systemctl restart sojorn-api
   sudo systemctl restart nginx
   sudo systemctl restart postgresql
   
   # Verify recovery
   curl -f https://api.gosojorn.com/health
   ```

3. **Communication**
   - Notify stakeholders of outage
   - Provide ETA for resolution
   - Document incident and resolution

### Security Incident Response

1. **Containment**
   - Isolate affected systems
   - Preserve evidence
   - Change credentials

2. **Investigation**
   - Review logs and audit trails
   - Identify breach scope
   - Document findings

3. **Recovery**
   - Patch vulnerabilities
   - Restore from clean backups
   - Monitor for suspicious activity

---

**Last Updated**: January 30, 2026
**Version**: 1.0
**Next Review**: February 15, 2026