sojorn/sojorn_docs/E2EE_COMPREHENSIVE_GUIDE.md

End-to-End Encryption (E2EE) Comprehensive Guide

Overview

This document consolidates all E2EE implementation knowledge for Sojorn, covering the complete evolution from simple stateless encryption to the current X3DH-based production system.

Current Architecture (Production System)

Cryptographic Foundation

• Flutter Client: Uses X25519 for key exchange, Ed25519 for signatures, AES-GCM for encryption
• Go Backend: Stores key bundles in PostgreSQL, serves encryption keys
• Protocol: X3DH (Extended Triple Diffie-Hellman) for key agreement

Key Components

1. Key Storage

• FlutterSecureStorage: Local key persistence with e2ee_keys_v3 key
• PostgreSQL Tables: profiles, signed_prekeys, one_time_prekeys
• Key Format: Identity keys stored as Ed25519:X25519 (base64 concatenated with colon)

2. Key Generation Flow

1. Generate Ed25519 signing key pair (for signatures)
2. Generate X25519 identity key pair (for DH)
3. Generate X25519 signed prekey with Ed25519 signature
4. Generate 20 X25519 one-time prekeys (OTKs)
5. Upload key bundle to backend

3. Message Encryption Flow

1. Fetch recipient's key bundle from backend
2. Verify signed prekey signature with Ed25519
3. Perform X3DH key agreement
4. Derive shared secret using KDF (SHA-256)
5. Encrypt message with AES-GCM
6. Delete used OTK from server

Historical Evolution

Phase 1: Simple Stateless E2EE (Legacy)

Description: Basic stateless system using X25519 + AES-GCM with single static identity keys.

Architecture:

• Each user had a single static identity key pair
• Each message used a fresh ephemeral key pair
• Shared secret derived via X25519 ECDH
• Sender could not decrypt their own message history

Data Model:

profiles.identity_key (base64 X25519 public key)
encrypted_conversations (conversation metadata)
encrypted_messages (ciphertext + header + metadata)

Message Header Format:

{
  "epk": "<base64 sender ephemeral public key>",
  "n":   "<base64 nonce>",
  "m":   "<base64 MAC>",
  "v":   1
}

Limitations:

• No forward secrecy beyond individual messages
• No multi-device support
• Senders couldn't decrypt their own message history
• No key recovery mechanism

Phase 2: X3DH Implementation (Current)

Description: Full X3DH implementation with signed prekeys, one-time prekeys, and proper key management.

Improvements:

• ✅ Perfect Forward Secrecy via OTKs
• ✅ Post-Compromise Security via key rotation
• ✅ Authentication via Ed25519 signatures
• ✅ Confidentiality via AES-GCM
• ✅ Cross-platform compatibility (Android↔Web)
• ✅ Automatic key management

Issues Encountered & Resolutions

Issue #1: 208-bit Key Bug ❌→✅

Problem: Keys were 26 characters (208 bits) instead of 32 bytes (256 bits) Root Cause: Using string-based KDF instead of proper byte-based KDF Fix: Updated _kdf method to use SHA-256 on byte arrays Files Modified: simple_e2ee_service.dart

Issue #2: Database Constraint Error ❌→✅

Problem: SQLSTATE 42P10 - ON CONFLICT constraint mismatch Root Cause: Go code used ON CONFLICT (user_id) but DB had PRIMARY KEY (user_id, key_id) Fix: Updated Go code to use correct constraint ON CONFLICT (user_id, key_id) Files Modified: user_repository.go

Issue #3: Fake Zero Signatures ❌→✅

Problem: SPK signatures were all zeros (AAAAAAAA...) Root Cause: Manual upload used fake signature for testing Fix: Updated manual upload to generate real Ed25519 signatures Files Modified: simple_e2ee_service.dart

Issue #4: Asymmetric Security ❌→✅

Problem: One user skipped signature verification (legacy), other enforced it Root Cause: Legacy user detection created security asymmetry Fix: Removed legacy logic, enforced signature verification for all users Files Modified: simple_e2ee_service.dart

Issue #5: Key Upload Not Automatic ❌→✅

Problem: Keys loaded locally but never uploaded to backend Root Cause: _doInitialize returned early after loading keys Fix: Added backend existence check and automatic upload Files Modified: simple_e2ee_service.dart

Issue #6: NULL Database Values ❌→✅

Problem: registration_id was NULL causing scan errors Root Cause: Database column allowed NULL values Fix: Updated Go code to handle sql.NullInt64 with default values Files Modified: user_repository.go

Issue #7: Noisy WebSocket Logs ❌→✅

Problem: Ping/pong messages cluttered console Root Cause: WebSocket heartbeat logging Fix: Filtered out ping/pong messages completely Files Modified: secure_chat_service.dart

Issue #8: Modal Header Override ❌→✅

Problem: AppBar changes in chat screen were hidden by modal wrapper Root Cause: SecureChatModal had custom header overriding SecureChatScreen AppBar Fix: Added upload button to modal header instead Files Modified: secure_chat_modal_sheet.dart

Current Status ✅

Working Components

• ✅ 32-byte key generation
• ✅ Valid Ed25519 signatures
• ✅ Signature verification
• ✅ Key bundle upload/download
• ✅ X3DH key agreement
• ✅ AES-GCM encryption/decryption
• ✅ OTK management (generation, usage, deletion)
• ✅ Backend key storage/retrieval
• ✅ Cross-platform encryption (Android↔Web)
• ✅ Full Backup & Recovery (Keys + Messages)

Key Files Modified

Flutter:
- lib/services/simple_e2ee_service.dart (core E2EE logic)
- lib/services/secure_chat_service.dart (WebSocket + key management)
- lib/screens/secure_chat/secure_chat_modal_sheet.dart (UI upload button)

Go Backend:
- internal/handlers/key_handler.go (API endpoints + validation)
- internal/repository/user_repository.go (database operations)

Database Schema

-- Key storage tables
profiles (identity_key, registration_id)
signed_prekeys (user_id, key_id, public_key, signature)
one_time_prekeys (user_id, key_id, public_key)

Testing Checklist

Before Testing

1. Ensure both users have valid keys (check [E2EE] Keys exist on backend - ready)
2. Verify signatures are non-zero (check backend logs)
3. Confirm OTKs are available (should have 20 OTKs each)

Test Flow

1. Key Upload: Tap "🔑" button → should see [E2EE] Key bundle uploaded successfully
2. Message Send: Type message → should see [E2EE] SPK signature verified successfully
3. Message Receive: Should see [DECRYPT] SUCCESS: Decrypted message: "..."
4. OTK Deletion: Should see [E2EE] Deleted used OTK #[id] from server

Expected Logs

Sender:
[ENCRYPT] Fetching key bundle for recipient: [...]
[E2EE] SPK signature verified successfully.
[E2EE] Deleted used OTK #[id] from server

Receiver:
[DECRYPT] Used OTK with key_id: [id]
[DECRYPT] SUCCESS: Decrypted message: "[message_text]"

Backup & Recovery System ✅

Overview

A robust local backup and recovery system has been implemented to address the risk of data loss on device changes or app uninstalls. This system allows users to export their cryptographic identity and message history into a secure, portable file.

Architecture

1. Security Model

• Encryption: AES-256-GCM
• Key Derivation: Argon2id (from user password)
• Storage: Local file system (portable JSON file)
• Trust: Zero-knowledge (server never sees the backup file or password)

2. Backup Content

The encrypted backup file contains two main components:

1. Key Material:
   • Identity Key Pair (Ed25519 & X25519)
   • Signed PreKey Pair (with signature)
   • One-Time PreKeys (all unused keys)
2. Message History (Optional):
   • Full plaintext message history
   • Metadata (sender, timestamp, etc.)
   • Note: Messages are decrypted from local storage and re-encrypted with the backup password for portability.

3. Backup Flow

1. User Initiation: User selects "Full Backup & Recovery" in settings.
2. Password Entry: User sets a strong backup password.
3. Data Gathering:
   • SimpleE2EEService exports all key pairs.
   • (Optional) LocalMessageStore exports all message records.
4. Encryption:
   • Salt & Nonce generated.
   • Key derived from password via Argon2id.
   • Payload (keys + messages) encrypted via AES-GCM.
5. File Generation: JSON file containing ciphertext, salt, nonce, and metadata is saved to device.

4. Restore Flow

1. File Selection: User selects the .json backup file.
2. Decryption:
   • User enters password.
   • Key derived using stored salt.
   • Payload decrypted.
3. Import:
   • Keys are imported into SimpleE2EEService and persisted to secure storage.
   • Messages are imported into LocalMessageStore and re-encrypted with the device's new local storage key.

Technical Implementation

• Service: LocalKeyBackupService handles the encryption/decryption pipeline.
• Store: LocalMessageStore provides bulk export/import methods (getAllMessageRecords, saveMessageRecord).
• UI: LocalBackupScreen provides the interface for creating and restoring backups.

Security Considerations

Current Security Model

• ✅ Perfect Forward Secrecy (PFS) via OTKs
• ✅ Post-Compromise Security via key rotation
• ✅ Authentication via Ed25519 signatures
• ✅ Confidentiality via AES-GCM

Recovery Security Impact

• ⚠️ Breaks PFS for recovered messages
• ✅ Maintains confidentiality with password protection
• ✅ Preserves authentication via signature verification
• ⚠️ Requires trust in backup storage

Mitigation Strategies

1. Use strong password requirements
2. Implement backup encryption verification
3. Add backup expiration policies
4. Monitor backup access patterns

Migration Notes

From Simple E2EE to X3DH

• Old messages encrypted with simple protocol are not decryptable with new system
• Full reset required clearing encrypted_messages and profiles.identity_key
• Multi-device support still not implemented; one account per device

Database Migration

• Added signed_prekeys and one_time_prekeys tables
• Updated profiles table with new key format
• Migration scripts available in migrations_archive/

Conclusion

The E2EE implementation is now fully functional with all major issues resolved. The system provides:

• Strong cryptographic guarantees
• Cross-platform compatibility
• Automatic key management
• Secure message transmission

The next phase focuses on device management to handle users with multiple active devices simultaneously.

---

Last Updated: February 2, 2026 Status: ✅ Production Ready (including key/message recovery) Next Priority: Device Management (Multi-device support)