21 KiB
21 KiB
Deployment & Operations Comprehensive Guide
Overview
This guide consolidates all deployment, operations, and maintenance documentation for the Sojorn platform, covering infrastructure setup, deployment procedures, monitoring, and operational best practices.
All code updates need to be made locally, synced to git, pulled to the server and deployed. Do not directly edit files on the server.
Infrastructure Setup
Server Requirements
Minimum Specifications
- CPU: 2 vCPU
- Memory: 4GB RAM
- Storage: 50GB SSD
- OS: Ubuntu 22.04 LTS
- Network: Public IP with domain pointing
Recommended Specifications
- CPU: 4 vCPU
- Memory: 8GB RAM
- Storage: 100GB SSD
- OS: Ubuntu 22.04 LTS
- Network: Public IP with SSL certificate
System Dependencies
Core Packages
sudo apt update && sudo apt install -y \
postgresql \
postgresql-contrib \
postgis \
nginx \
certbot \
python3-certbot-nginx \
git \
curl \
wget \
htop
Go Environment
# Install Go 1.21+
wget https://go.dev/dl/go1.21.0.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.21.0.linux-amd64.tar.gz
echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc
source ~/.bashrc
Database Setup
PostgreSQL Configuration
# Create database
sudo -u postgres createdb sojorn
# Create user
sudo -u postgres createuser --interactive sojorn_user
# Grant privileges
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE sojorn TO sojorn_user;"
# Enable extensions
sudo -u postgres psql sojorn -c "
CREATE EXTENSION IF NOT EXISTS uuid-ossp;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS postgis;
"
PostgreSQL Performance Tuning
# Edit postgresql.conf
sudo nano /etc/postgresql/15/main/postgresql.conf
Key settings:
# Memory settings
shared_buffers = 256MB
effective_cache_size = 1GB
work_mem = 4MB
maintenance_work_mem = 64MB
# Connection settings
max_connections = 100
listen_addresses = 'localhost'
# Logging settings
log_statement = 'all'
log_duration = on
log_min_duration_statement = 1000
Application Deployment
Deployment Structure
/opt/sojorn/
├── .env # Environment variables
├── firebase-service-account.json # FCM credentials
├── uploads/ # User uploaded files
├── logs/ # Application logs
└── go-backend/
├── bin/
│ └── api # Compiled binary
├── migrations/ # Database migrations
└── scripts/ # Deployment scripts
Environment Configuration
.env File Template
# Database
DATABASE_URL=postgres://sojorn_user:password@localhost:5432/sojorn?sslmode=disable
# Server
PORT=8080
HOST=localhost
ENVIRONMENT=production
# JWT
JWT_SECRET=your-super-secret-jwt-key-here
JWT_EXPIRY=24h
REFRESH_TOKEN_EXPIRY=168h
# Firebase/FCM
FIREBASE_CREDENTIALS_FILE=/opt/sojorn/firebase-service-account.json
FIREBASE_WEB_VAPID_KEY=BNxS7_your_vapid_key_here
# Storage
SENDER_API_TOKEN=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJhdWQiOiIxIiwianRpIjoiNT>
R2_ACCOUNT_ID=7041ca6e0f40307190dc2e65e2fb5e0f
R2_PUBLIC_BASE_URL=http://api.sojorn.net:8080/uploads
R2_IMG_DOMAIN=img.sojorn.net
R2_VID_DOMAIN=quips.sojorn.net
R2_API_TOKEN=oR7Vk0Realtx0D6SAGMuYA8pXopSoCYKv8t3JEuk
# Email (Optional)
# Email / SendPulse
SMTP_HOST=smtp-pulse.com
SMTP_PORT=587
SMTP_USER=patrickbritton3@gmail.com
SMTP_PASS=8s4jQBnAFTCXPNM
# Logging
LOG_LEVEL=info
LOG_FORMAT=json
Build Process
Compile Go Backend
cd /opt/sojorn/go-backend
# Build for production
go build -ldflags="-s -w" -o bin/api ./cmd/api/main.go
# Make executable
chmod +x bin/api
Database Migrations
cd /opt/sojorn/go-backend
# Run migrations
make migrate-up
# Or manually
go run ./cmd/migrate/main.go up
Systemd Service
Service Configuration
Create /etc/systemd/system/sojorn-api.service:
[Unit]
Description=Sojorn API Server
After=network.target postgresql.service
Requires=postgresql.service
[Service]
Type=simple
User=patrick
Group=patrick
WorkingDirectory=/opt/sojorn/go-backend
Environment=GO_ENV=production
EnvironmentFile=/opt/sojorn/.env
ExecStart=/opt/sojorn/bin/api
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=sojorn-api
# Security
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/sojorn/uploads /opt/sojorn/logs
[Install]
WantedBy=multi-user.target
Service Management
# Reload systemd
sudo systemctl daemon-reload
# Enable service
sudo systemctl enable sojorn-api
# Start service
sudo systemctl start sojorn-api
# Check status
sudo systemctl status sojorn-api
# View logs
sudo journalctl -u sojorn-api -f
Web Server Configuration
Nginx Setup
Main Configuration
Create /etc/nginx/sites-available/sojorn-api:
server {
listen 80;
server_name api.sojorn.net;
# Redirect to HTTPS
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name api.sojorn.net;
# SSL Configuration
ssl_certificate /etc/letsencrypt/live/api.sojorn.net/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/api.sojorn.net/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
# Security Headers
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
# Logging
access_log /var/log/nginx/sojorn-api.access.log;
error_log /var/log/nginx/sojorn-api.error.log;
# Rate Limiting
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
limit_req zone=api burst=20 nodelay;
# API Proxy
location / {
proxy_pass http://localhost:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_cache_bypass $http_upgrade;
proxy_read_timeout 86400;
}
# File Uploads
location /uploads/ {
alias /opt/sojorn/uploads/;
expires 1y;
add_header Cache-Control "public, immutable";
# Security for uploads
location ~* \.(php|jsp|asp|sh|py)$ {
deny all;
}
}
# Health Check
location /health {
access_log off;
proxy_pass http://localhost:8080/health;
}
# WebSocket Support (for chat)
location /ws {
proxy_pass http://localhost:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 86400;
proxy_send_timeout 86400;
}
}
Enable Site
# Enable site
sudo ln -s /etc/nginx/sites-available/sojorn-api /etc/nginx/sites-enabled/
# Test configuration
sudo nginx -t
# Reload Nginx
sudo systemctl reload nginx
SSL Certificate Setup
Let's Encrypt Certificate
# Install Certbot
sudo apt install certbot python3-certbot-nginx
# Obtain certificate
sudo certbot --nginx -d api.sojorn.net
# Test renewal
sudo certbot renew --dry-run
# Set up auto-renewal
sudo crontab -e
# Add: 0 12 * * * /usr/bin/certbot renew --quiet
Monitoring & Logging
Application Monitoring
Health Check Endpoint
// health_check.go
func (h *HealthHandler) Health(c *gin.Context) {
status := map[string]interface{}{
"status": "healthy",
"timestamp": time.Now(),
"version": os.Getenv("APP_VERSION"),
"environment": os.Getenv("ENVIRONMENT"),
}
// Check database
if err := h.db.Ping(); err != nil {
status["status"] = "unhealthy"
status["database"] = "disconnected"
c.JSON(http.StatusServiceUnavailable, status)
return
}
status["database"] = "connected"
c.JSON(http.StatusOK, status)
}
Metrics Collection
// metrics.go
var (
httpRequestsTotal = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "Total number of HTTP requests",
},
[]string{"method", "endpoint", "status"},
)
httpRequestDuration = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "http_request_duration_seconds",
Help: "HTTP request duration in seconds",
},
[]string{"method", "endpoint"},
)
)
func MetricsMiddleware() gin.HandlerFunc {
return func(c *gin.Context) {
start := time.Now()
c.Next()
duration := time.Since(start).Seconds()
status := fmt.Sprintf("%d", c.Writer.Status())
httpRequestsTotal.WithLabelValues(c.Request.Method, c.FullPath(), status).Inc()
httpRequestDuration.WithLabelValues(c.Request.Method, c.FullPath()).Observe(duration)
}
}
System Monitoring
Setup Prometheus & Grafana (Optional)
# Install Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.40.0/prometheus-2.40.0.linux-amd64.tar.gz
tar xvfz prometheus-2.40.0.linux-amd64.tar.gz
sudo mv prometheus-2.40.0.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-2.40.0.linux-amd64/prometheus.yml /etc/prometheus/
# Install Grafana
sudo apt-get install -y adduser libfontconfig1
wget https://dl.grafana.com/oss/release/grafana_9.3.0_amd64.deb
sudo dpkg -i grafana_9.3.0_amd64.deb
Log Management
# Configure log rotation
sudo nano /etc/logrotate.d/sojorn-api
/opt/sojorn/logs/*.log {
daily
missingok
rotate 30
compress
delaycompress
notifempty
create 644 patrick patrick
postrotate
systemctl reload sojorn-api
endscript
}
Alerting
System Alerts Script
#!/bin/bash
# monitor.sh - System health monitoring
# Check service status
if ! systemctl is-active --quiet sojorn-api; then
echo "ALERT: sojorn-api service is down" | mail -s "Sojorn Service Alert" admin@sojorn.com
fi
# Check disk space
DISK_USAGE=$(df /opt/sojorn | awk 'NR==2 {print $5}' | sed 's/%//')
if [ $DISK_USAGE -gt 80 ]; then
echo "ALERT: Disk usage is ${DISK_USAGE}%" | mail -s "Sojorn Disk Alert" admin@sojorn.com
fi
# Check memory usage
MEM_USAGE=$(free | awk 'NR==2{printf "%.0f", $3*100/$2}')
if [ $MEM_USAGE -gt 90 ]; then
echo "ALERT: Memory usage is ${MEM_USAGE}%" | mail -s "Sojorn Memory Alert" admin@sojorn.com
fi
Backup & Recovery
Database Backups
Automated Backup Script
#!/bin/bash
# backup.sh - Database backup script
BACKUP_DIR="/opt/sojorn/backups"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/sojorn_backup_$DATE.sql"
# Create backup directory
mkdir -p $BACKUP_DIR
# Create backup
pg_dump -h localhost -U sojorn_user sojorn > $BACKUP_FILE
# Compress backup
gzip $BACKUP_FILE
# Remove old backups (keep 7 days)
find $BACKUP_DIR -name "*.sql.gz" -mtime +7 -delete
echo "Backup completed: $BACKUP_FILE.gz"
Cron Job for Daily Backups
# Add to crontab
sudo crontab -e
# Daily backup at 2 AM
0 2 * * * /opt/sojorn/scripts/backup.sh
File Backups
Upload Directory Backup
#!/bin/bash
# backup_uploads.sh
BACKUP_DIR="/opt/sojorn/backups"
DATE=$(date +%Y%m%d_%H%M%S)
UPLOAD_DIR="/opt/sojorn/uploads"
# Create compressed backup
tar -czf "$BACKUP_DIR/uploads_backup_$DATE.tar.gz" -C "$UPLOAD_DIR" .
# Remove old backups (keep 7 days)
find $BACKUP_DIR -name "uploads_backup_*.tar.gz" -mtime +7 -delete
Recovery Procedures
Database Recovery
# Stop application
sudo systemctl stop sojorn-api
# Restore database
gunzip -c /opt/sojorn/backups/sojorn_backup_20260130_020000.sql.gz | \
psql -h localhost -U sojorn_user -d sojorn
# Start application
sudo systemctl start sojorn-api
# Verify recovery
curl -f https://api.sojorn.net/health
File Recovery
# Restore uploads
tar -xzf /opt/sojorn/backups/uploads_backup_20260130_020000.tar.gz \
-C /opt/sojorn/uploads
# Fix permissions
sudo chown -R patrick:patrick /opt/sojorn/uploads
sudo chmod -R 755 /opt/sojorn/uploads
Security Hardening
System Security
Firewall Configuration
# Install UFW
sudo apt install ufw
# Default policies
sudo ufw default deny incoming
sudo ufw default allow outgoing
# Allow SSH
sudo ufw allow ssh
# Allow HTTP/HTTPS
sudo ufw allow 80
sudo ufw allow 443
# Enable firewall
sudo ufw enable
SSH Security
# Edit SSH config
sudo nano /etc/ssh/sshd_config
Key settings:
# Disable root login
PermitRootLogin no
# Use key-based authentication
PasswordAuthentication no
PubkeyAuthentication yes
# Change default port (optional)
Port 2222
# Limit login attempts
MaxAuthTries 3
Fail2Ban Setup
# Install Fail2Ban
sudo apt install fail2ban
# Configure for SSH
sudo nano /etc/fail2ban/jail.local
[DEFAULT]
bantime = 3600
findtime = 600
maxretry = 3
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
Application Security
Environment Variable Protection
# Secure .env file
sudo chmod 600 /opt/sojorn/.env
sudo chown patrick:patrick /opt/sojorn/.env
# Secure service account file
sudo chmod 600 /opt/sojorn/firebase-service-account.json
sudo chown patrick:patrick /opt/sojorn/firebase-service-account.json
SSL/TLS Hardening
# Add to Nginx config
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
# HSTS
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
# Other security headers
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header Referrer-Policy "strict-origin-when-cross-origin";
Performance Optimization
Database Optimization
Query Optimization
-- Add indexes for common queries
CREATE INDEX CONCURRENTLY idx_posts_author_created ON posts(author_id, created_at DESC);
CREATE INDEX CONCURRENTLY idx_posts_category_created ON posts(category_id, created_at DESC) WHERE category_id IS NOT NULL;
CREATE INDEX CONCURRENTLY idx_follows_follower ON follows(follower_id);
CREATE INDEX CONCURRENTLY idx_follows_following ON follows(following_id);
-- Analyze table statistics
ANALYZE posts;
ANALYZE follows;
ANALYZE users;
Connection Pool Tuning
// In database setup
config, err := pgxpool.ParseConfig(databaseURL)
if err != nil {
return nil, err
}
config.MaxConns = 25
config.MinConns = 5
config.MaxConnLifetime = time.Hour
config.HealthCheckPeriod = time.Minute * 5
config.MaxConnIdleTime = time.Minute * 30
Caching Strategy
Redis Integration (Optional)
# Install Redis
sudo apt install redis-server
# Configure Redis
sudo nano /etc/redis/redis.conf
Key settings:
# Memory management
maxmemory 256mb
maxmemory-policy allkeys-lru
# Persistence
save 900 1
save 300 10
save 60 10000
Application Caching
// Cache implementation
type CacheService struct {
redis *redis.Client
}
func (c *CacheService) Set(key string, value interface{}, expiration time.Duration) error {
json, err := json.Marshal(value)
if err != nil {
return err
}
return c.redis.Set(context.Background(), key, json, expiration).Err()
}
func (c *CacheService) Get(key string, dest interface{}) error {
val, err := c.redis.Get(context.Background(), key).Result()
if err != nil {
return err
}
return json.Unmarshal([]byte(val), dest)
}
Deployment Automation
Deployment Script
Automated Deploy Script
#!/bin/bash
# deploy.sh - Automated deployment script
set -e # Exit on any error
# Configuration
REPO_URL="https://github.com/your-org/sojorn.git"
DEPLOY_DIR="/opt/sojorn"
BACKUP_DIR="/opt/sojorn/backups"
SERVICE_NAME="sojorn-api"
echo "Starting deployment..."
# Create backup
echo "Creating backup..."
$DEPLOY_DIR/scripts/backup.sh
# Pull latest code
echo "Pulling latest code..."
cd $DEPLOY_DIR
git pull origin main
# Build application
echo "Building application..."
cd go-backend
go build -ldflags="-s -w" -o bin/api ./cmd/api/main.go
# Run migrations
echo "Running database migrations..."
make migrate-up
# Restart service
echo "Restarting service..."
sudo systemctl restart $SERVICE_NAME
# Wait for service to start
sleep 5
# Health check
echo "Performing health check..."
if curl -f http://localhost:8080/health; then
echo "✅ Deployment successful!"
else
echo "❌ Health check failed!"
echo "Rolling back..."
sudo systemctl restart $SERVICE_NAME
exit 1
fi
echo "Deployment completed successfully!"
CI/CD Pipeline (Optional)
GitHub Actions Example
name: Deploy to Production
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.21'
- name: Build application
run: |
cd go-backend
go build -ldflags="-s -w" -o bin/api ./cmd/api/main.go
- name: Deploy to server
uses: appleboy/ssh-action@v0.1.5
with:
host: ${{ secrets.HOST }}
username: ${{ secrets.USERNAME }}
key: ${{ secrets.SSH_KEY }}
script: |
cd /opt/sojorn
./scripts/deploy.sh
Troubleshooting Common Issues
Service Startup Issues
Problem: Service fails to start
# Check service status
sudo systemctl status sojorn-api
# Check logs
sudo journalctl -u sojorn-api -n 50
# Common fixes
# 1. Check .env file permissions
sudo chmod 600 /opt/sojorn/.env
# 2. Check database connection
sudo -u postgres psql -c "SELECT 1;"
# 3. Check port availability
sudo netstat -tlnp | grep :8080
Problem: Database connection errors
# Check PostgreSQL status
sudo systemctl status postgresql
# Check connection string
sudo cat /opt/sojorn/.env | grep DATABASE_URL
# Test connection manually
psql "postgres://sojorn_user:password@localhost:5432/sojorn"
Performance Issues
Problem: Slow API responses
# Check system resources
top
htop
iostat -x 1
# Check database queries
sudo -u postgres psql -c "
SELECT query, mean_time, calls
FROM pg_stat_statements
ORDER BY mean_time DESC
LIMIT 10;"
# Check Nginx logs
sudo tail -f /var/log/nginx/sojorn-api.access.log
Problem: High memory usage
# Check memory usage
free -h
ps aux --sort=-%mem | head
# Restart service if needed
sudo systemctl restart sojorn-api
# Consider tuning Go memory settings
# In .env: GOMEMLIMIT=256MiB
SSL/TLS Issues
Problem: Certificate expired
# Check certificate expiry
sudo certbot certificates
# Renew certificate
sudo certbot renew
# Reload Nginx
sudo systemctl reload nginx
Problem: SSL handshake failed
# Test SSL configuration
openssl s_client -connect api.sojorn.net:443
# Check Nginx configuration
sudo nginx -t
# Reload Nginx
sudo systemctl reload nginx
Maintenance Schedule
Daily Tasks
- Check service status and logs
- Monitor system resources
- Verify backup completion
- Review error rates
Weekly Tasks
- Update security patches
- Review performance metrics
- Check disk space usage
- Test backup restoration
Monthly Tasks
- Update application dependencies
- Review and rotate secrets
- Audit user access and permissions
- Performance optimization review
Quarterly Tasks
- Security audit and penetration testing
- Disaster recovery testing
- Capacity planning review
- Documentation updates
Emergency Procedures
Service Outage Response
-
Immediate Assessment
# Check service status sudo systemctl status sojorn-api # Check system resources top df -h # Check network connectivity ping 8.8.8.8 -
Service Recovery
# Restart services sudo systemctl restart sojorn-api sudo systemctl restart nginx sudo systemctl restart postgresql # Verify recovery curl -f https://api.sojorn.net/health -
Communication
- Notify stakeholders of outage
- Provide ETA for resolution
- Document incident and resolution
Security Incident Response
-
Containment
- Isolate affected systems
- Preserve evidence
- Change credentials
-
Investigation
- Review logs and audit trails
- Identify breach scope
- Document findings
-
Recovery
- Patch vulnerabilities
- Restore from clean backups
- Monitor for suspicious activity
Last Updated: January 30, 2026 Version: 1.0 Next Review: February 15, 2026