sojorn/sojorn_docs/E2EE_COMPREHENSIVE_GUIDE.md

# End-to-End Encryption (E2EE) Comprehensive Guide

## Overview

This document consolidates all E2EE implementation knowledge for Sojorn, covering the complete evolution from simple stateless encryption to the current X3DH-based production system.

## Current Architecture (Production System)

### Cryptographic Foundation
- **Flutter Client**: Uses X25519 for key exchange, Ed25519 for signatures, AES-GCM for encryption
- **Go Backend**: Stores key bundles in PostgreSQL, serves encryption keys
- **Protocol**: X3DH (Extended Triple Diffie-Hellman) for key agreement

### Key Components

#### 1. Key Storage
- **FlutterSecureStorage**: Local key persistence with `e2ee_keys_v3` key
- **PostgreSQL Tables**: `profiles`, `signed_prekeys`, `one_time_prekeys`
- **Key Format**: Identity keys stored as `Ed25519:X25519` (base64 concatenated with colon)

#### 2. Key Generation Flow
1. Generate Ed25519 signing key pair (for signatures)
2. Generate X25519 identity key pair (for DH)
3. Generate X25519 signed prekey with Ed25519 signature
4. Generate 20 X25519 one-time prekeys (OTKs)
5. Upload key bundle to backend

#### 3. Message Encryption Flow
1. Fetch recipient's key bundle from backend
2. Verify signed prekey signature with Ed25519
3. Perform X3DH key agreement
4. Derive shared secret using KDF (SHA-256)
5. Encrypt message with AES-GCM
6. Delete used OTK from server

## Historical Evolution

### Phase 1: Simple Stateless E2EE (Legacy)

**Description**: Basic stateless system using X25519 + AES-GCM with single static identity keys.

**Architecture**:
- Each user had a single static identity key pair
- Each message used a fresh ephemeral key pair
- Shared secret derived via X25519 ECDH
- Sender could not decrypt their own message history

**Data Model**:
```
profiles.identity_key (base64 X25519 public key)
encrypted_conversations (conversation metadata)
encrypted_messages (ciphertext + header + metadata)
```

**Message Header Format**:
```json
{
  "epk": "<base64 sender ephemeral public key>",
  "n":   "<base64 nonce>",
  "m":   "<base64 MAC>",
  "v":   1
}
```

**Limitations**:
- No forward secrecy beyond individual messages
- No multi-device support
- Senders couldn't decrypt their own message history
- No key recovery mechanism

### Phase 2: X3DH Implementation (Current)

**Description**: Full X3DH implementation with signed prekeys, one-time prekeys, and proper key management.

**Improvements**:
- ✅ Perfect Forward Secrecy via OTKs
- ✅ Post-Compromise Security via key rotation
- ✅ Authentication via Ed25519 signatures
- ✅ Confidentiality via AES-GCM
- ✅ Cross-platform compatibility (Android↔Web)
- ✅ Automatic key management

## Issues Encountered & Resolutions

### Issue #1: 208-bit Key Bug ❌→✅
**Problem**: Keys were 26 characters (208 bits) instead of 32 bytes (256 bits)
**Root Cause**: Using string-based KDF instead of proper byte-based KDF
**Fix**: Updated `_kdf` method to use SHA-256 on byte arrays
**Files Modified**: `simple_e2ee_service.dart`

### Issue #2: Database Constraint Error ❌→✅
**Problem**: `SQLSTATE 42P10` - ON CONFLICT constraint mismatch
**Root Cause**: Go code used `ON CONFLICT (user_id)` but DB had `PRIMARY KEY (user_id, key_id)`
**Fix**: Updated Go code to use correct constraint `ON CONFLICT (user_id, key_id)`
**Files Modified**: `user_repository.go`

### Issue #3: Fake Zero Signatures ❌→✅
**Problem**: SPK signatures were all zeros (`AAAAAAAA...`)
**Root Cause**: Manual upload used fake signature for testing
**Fix**: Updated manual upload to generate real Ed25519 signatures
**Files Modified**: `simple_e2ee_service.dart`

### Issue #4: Asymmetric Security ❌→✅
**Problem**: One user skipped signature verification (legacy), other enforced it
**Root Cause**: Legacy user detection created security asymmetry
**Fix**: Removed legacy logic, enforced signature verification for all users
**Files Modified**: `simple_e2ee_service.dart`

### Issue #5: Key Upload Not Automatic ❌→✅
**Problem**: Keys loaded locally but never uploaded to backend
**Root Cause**: `_doInitialize` returned early after loading keys
**Fix**: Added backend existence check and automatic upload
**Files Modified**: `simple_e2ee_service.dart`

### Issue #6: NULL Database Values ❌→✅
**Problem**: `registration_id` was NULL causing scan errors
**Root Cause**: Database column allowed NULL values
**Fix**: Updated Go code to handle `sql.NullInt64` with default values
**Files Modified**: `user_repository.go`

### Issue #7: Noisy WebSocket Logs ❌→✅
**Problem**: Ping/pong messages cluttered console
**Root Cause**: WebSocket heartbeat logging
**Fix**: Filtered out ping/pong messages completely
**Files Modified**: `secure_chat_service.dart`

### Issue #8: Modal Header Override ❌→✅
**Problem**: AppBar changes in chat screen were hidden by modal wrapper
**Root Cause**: `SecureChatModal` had custom header overriding `SecureChatScreen` AppBar
**Fix**: Added upload button to modal header instead
**Files Modified**: `secure_chat_modal_sheet.dart`

## Current Status ✅

### Working Components
- ✅ 32-byte key generation
- ✅ Valid Ed25519 signatures
- ✅ Signature verification
- ✅ Key bundle upload/download
- ✅ X3DH key agreement
- ✅ AES-GCM encryption/decryption
- ✅ OTK management (generation, usage, deletion)
- ✅ Backend key storage/retrieval
- ✅ Cross-platform encryption (Android↔Web)

### Key Files Modified
```
Flutter:
- lib/services/simple_e2ee_service.dart (core E2EE logic)
- lib/services/secure_chat_service.dart (WebSocket + key management)
- lib/screens/secure_chat/secure_chat_modal_sheet.dart (UI upload button)

Go Backend:
- internal/handlers/key_handler.go (API endpoints + validation)
- internal/repository/user_repository.go (database operations)
```

### Database Schema
```sql
-- Key storage tables
profiles (identity_key, registration_id)
signed_prekeys (user_id, key_id, public_key, signature)
one_time_prekeys (user_id, key_id, public_key)
```

## Testing Checklist

### Before Testing
1. Ensure both users have valid keys (check `[E2EE] Keys exist on backend - ready`)
2. Verify signatures are non-zero (check backend logs)
3. Confirm OTKs are available (should have 20 OTKs each)

### Test Flow
1. **Key Upload**: Tap "🔑" button → should see `[E2EE] Key bundle uploaded successfully`
2. **Message Send**: Type message → should see `[E2EE] SPK signature verified successfully`
3. **Message Receive**: Should see `[DECRYPT] SUCCESS: Decrypted message: "..."`
4. **OTK Deletion**: Should see `[E2EE] Deleted used OTK #[id] from server`

### Expected Logs
```
Sender:
[ENCRYPT] Fetching key bundle for recipient: [...]
[E2EE] SPK signature verified successfully.
[E2EE] Deleted used OTK #[id] from server

Receiver:
[DECRYPT] Used OTK with key_id: [id]
[DECRYPT] SUCCESS: Decrypted message: "[message_text]"
```

## Next Steps: Message Recovery

### Problem
When users uninstall the app or lose local keys, they cannot decrypt historical messages.

### Solution Requirements
1. **Key Backup Strategy**: Securely backup encryption keys
2. **Message Recovery**: Allow decryption of historical messages after key recovery
3. **Security**: Maintain E2EE guarantees while enabling recovery

### Proposed Solutions

#### Option 1: Cloud Key Backup (Recommended)
- Encrypt identity keys with user password
- Store encrypted backup in cloud storage
- Recover keys with password authentication

**Pros**:
- Most user-friendly
- Maintains security (password-protected)
- Technically straightforward
- Reversible if needed

#### Option 2: Social Recovery
- Allow trusted contacts to help recover keys
- Use Shamir's Secret Sharing for security
- Requires multiple trusted contacts

#### Option 3: Server-Side Recovery (Limited)
- Store encrypted key backups on server
- Server cannot decrypt without user password
- Similar to Signal's approach

#### Option 4: Message Re-encryption
- Store messages encrypted with server keys
- Re-encrypt with new keys after recovery
- Breaks perfect forward secrecy

### Implementation Plan for Key Recovery

#### Phase 1: Key Backup
1. Add password-based key encryption
2. Implement cloud backup storage
3. Add backup/restore UI
4. Test backup/restore flow

#### Phase 2: Message Recovery
1. Store message headers for re-decryption
2. Implement batch message re-decryption
3. Add recovery progress indicators
4. Test with historical messages

#### Phase 3: Security Enhancements
1. Add backup encryption verification
2. Implement backup rotation
3. Add recovery security checks
4. Monitor recovery success rates

## Security Considerations

### Current Security Model
- ✅ Perfect Forward Secrecy (PFS) via OTKs
- ✅ Post-Compromise Security via key rotation
- ✅ Authentication via Ed25519 signatures
- ✅ Confidentiality via AES-GCM

### Recovery Security Impact
- ⚠️ Breaks PFS for recovered messages
- ✅ Maintains confidentiality with password protection
- ✅ Preserves authentication via signature verification
- ⚠️ Requires trust in backup storage

### Mitigation Strategies
1. Use strong password requirements
2. Implement backup encryption verification
3. Add backup expiration policies
4. Monitor backup access patterns

## Migration Notes

### From Simple E2EE to X3DH
- Old messages encrypted with simple protocol are not decryptable with new system
- Full reset required clearing `encrypted_messages` and `profiles.identity_key`
- Multi-device support still not implemented; one account per device

### Database Migration
- Added `signed_prekeys` and `one_time_prekeys` tables
- Updated `profiles` table with new key format
- Migration scripts available in `migrations_archive/`

## Conclusion

The E2EE implementation is now fully functional with all major issues resolved. The system provides:

- Strong cryptographic guarantees
- Cross-platform compatibility
- Automatic key management
- Secure message transmission

The next phase focuses on key recovery to handle user device changes while maintaining security principles.

---

**Last Updated**: January 30, 2026
**Status**: ✅ Production Ready (except key recovery)
**Next Priority**: Implement key recovery system