diff --git a/docs/DATABASE.md b/docs/DATABASE.md deleted file mode 100644 index 52df42e0b..000000000 --- a/docs/DATABASE.md +++ /dev/null @@ -1,736 +0,0 @@ -# Trilium Database Architecture - -> **Related:** [ARCHITECTURE.md](ARCHITECTURE.md) | [Database Schema](Developer%20Guide/Developer%20Guide/Development%20and%20architecture/Database/) - -## Overview - -Trilium uses **SQLite** as its embedded database engine, providing a reliable, file-based storage system that requires no separate database server. The database stores all notes, their relationships, metadata, and configuration. - -## Database File - -**Location:** -- Desktop: `~/.local/share/trilium-data/document.db` (Linux/Mac) or `%APPDATA%/trilium-data/document.db` (Windows) -- Server: Configured via `TRILIUM_DATA_DIR` environment variable -- Docker: Mounted volume at `/home/node/trilium-data/` - -**Characteristics:** -- Single-file database -- Embedded (no server required) -- ACID compliant -- Cross-platform -- Supports up to 281 TB database size -- Efficient for 100k+ notes - -## Database Driver - -**Library:** `better-sqlite3` - -**Why better-sqlite3:** -- Native performance (C++ bindings) -- Synchronous API (simpler code) -- Prepared statements -- Transaction support -- Type safety - -**Usage:** -```typescript -// apps/server/src/services/sql.ts -import Database from 'better-sqlite3' - -const db = new Database('document.db') -const stmt = db.prepare('SELECT * FROM notes WHERE noteId = ?') -const note = stmt.get(noteId) -``` - -## Schema Overview - -Schema location: `apps/server/src/assets/db/schema.sql` - -**Entity Tables:** -- `notes` - Core note data -- `branches` - Tree relationships -- `attributes` - Metadata (labels/relations) -- `revisions` - Version history -- `attachments` - File attachments -- `blobs` - Binary content storage - -**System Tables:** -- `options` - Application configuration -- `entity_changes` - Change tracking for sync -- `recent_notes` - Recently accessed notes -- `etapi_tokens` - API authentication tokens -- `user_data` - User credentials -- `sessions` - Web session storage - -## Entity Tables - -### Notes Table - -```sql -CREATE TABLE notes ( - noteId TEXT NOT NULL PRIMARY KEY, - title TEXT NOT NULL DEFAULT "note", - isProtected INT NOT NULL DEFAULT 0, - type TEXT NOT NULL DEFAULT 'text', - mime TEXT NOT NULL DEFAULT 'text/html', - blobId TEXT DEFAULT NULL, - isDeleted INT NOT NULL DEFAULT 0, - deleteId TEXT DEFAULT NULL, - dateCreated TEXT NOT NULL, - dateModified TEXT NOT NULL, - utcDateCreated TEXT NOT NULL, - utcDateModified TEXT NOT NULL -); - --- Indexes for performance -CREATE INDEX IDX_notes_title ON notes (title); -CREATE INDEX IDX_notes_type ON notes (type); -CREATE INDEX IDX_notes_dateCreated ON notes (dateCreated); -CREATE INDEX IDX_notes_dateModified ON notes (dateModified); -CREATE INDEX IDX_notes_utcDateModified ON notes (utcDateModified); -CREATE INDEX IDX_notes_blobId ON notes (blobId); -``` - -**Field Descriptions:** - -| Field | Type | Description | -|-------|------|-------------| -| `noteId` | TEXT | Unique identifier (UUID or custom) | -| `title` | TEXT | Note title (displayed in tree) | -| `isProtected` | INT | 1 if encrypted, 0 if not | -| `type` | TEXT | Note type: text, code, file, image, etc. | -| `mime` | TEXT | MIME type: text/html, application/json, etc. | -| `blobId` | TEXT | Reference to content in blobs table | -| `isDeleted` | INT | Soft delete flag | -| `deleteId` | TEXT | Unique delete operation ID | -| `dateCreated` | TEXT | Creation date (local timezone) | -| `dateModified` | TEXT | Last modified (local timezone) | -| `utcDateCreated` | TEXT | Creation date (UTC) | -| `utcDateModified` | TEXT | Last modified (UTC) | - -**Note Types:** -- `text` - Rich text with HTML -- `code` - Source code -- `file` - Binary file -- `image` - Image file -- `search` - Saved search -- `render` - Custom HTML rendering -- `relation-map` - Relationship diagram -- `canvas` - Excalidraw drawing -- `mermaid` - Mermaid diagram -- `book` - Container for documentation -- `web-view` - Embedded web page -- `mindmap` - Mind map -- `geomap` - Geographical map - -### Branches Table - -```sql -CREATE TABLE branches ( - branchId TEXT NOT NULL PRIMARY KEY, - noteId TEXT NOT NULL, - parentNoteId TEXT NOT NULL, - notePosition INTEGER NOT NULL, - prefix TEXT, - isExpanded INTEGER NOT NULL DEFAULT 0, - isDeleted INTEGER NOT NULL DEFAULT 0, - deleteId TEXT DEFAULT NULL, - utcDateModified TEXT NOT NULL -); - --- Indexes -CREATE INDEX IDX_branches_noteId_parentNoteId ON branches (noteId, parentNoteId); -CREATE INDEX IDX_branches_parentNoteId ON branches (parentNoteId); -``` - -**Field Descriptions:** - -| Field | Type | Description | -|-------|------|-------------| -| `branchId` | TEXT | Unique identifier for this branch | -| `noteId` | TEXT | Child note ID | -| `parentNoteId` | TEXT | Parent note ID | -| `notePosition` | INT | Sort order among siblings | -| `prefix` | TEXT | Optional prefix text (e.g., "Chapter 1:") | -| `isExpanded` | INT | Tree expansion state | -| `isDeleted` | INT | Soft delete flag | -| `deleteId` | TEXT | Delete operation ID | -| `utcDateModified` | TEXT | Last modified (UTC) | - -**Key Concepts:** -- **Cloning:** A note can have multiple branches (multiple parents) -- **Position:** Siblings ordered by `notePosition` -- **Prefix:** Display text before note title in tree -- **Soft Delete:** Allows sync before permanent deletion - -### Attributes Table - -```sql -CREATE TABLE attributes ( - attributeId TEXT NOT NULL PRIMARY KEY, - noteId TEXT NOT NULL, - type TEXT NOT NULL, - name TEXT NOT NULL, - value TEXT DEFAULT '' NOT NULL, - position INT DEFAULT 0 NOT NULL, - utcDateModified TEXT NOT NULL, - isDeleted INT NOT NULL, - deleteId TEXT DEFAULT NULL, - isInheritable INT DEFAULT 0 NULL -); - --- Indexes -CREATE INDEX IDX_attributes_name_value ON attributes (name, value); -CREATE INDEX IDX_attributes_noteId ON attributes (noteId); -CREATE INDEX IDX_attributes_value ON attributes (value); -``` - -**Field Descriptions:** - -| Field | Type | Description | -|-------|------|-------------| -| `attributeId` | TEXT | Unique identifier | -| `noteId` | TEXT | Note this attribute belongs to | -| `type` | TEXT | 'label' or 'relation' | -| `name` | TEXT | Attribute name | -| `value` | TEXT | Attribute value (text for labels, noteId for relations) | -| `position` | INT | Display order | -| `utcDateModified` | TEXT | Last modified (UTC) | -| `isDeleted` | INT | Soft delete flag | -| `deleteId` | TEXT | Delete operation ID | -| `isInheritable` | INT | Inherited by child notes | - -**Attribute Types:** - -**Labels** (key-value pairs): -```sql --- Example: #priority=high -INSERT INTO attributes (attributeId, noteId, type, name, value) -VALUES ('attr1', 'note123', 'label', 'priority', 'high') -``` - -**Relations** (links to other notes): -```sql --- Example: ~author=[[noteId]] -INSERT INTO attributes (attributeId, noteId, type, name, value) -VALUES ('attr2', 'note123', 'relation', 'author', 'author-note-id') -``` - -**Special Attributes:** -- `#run=frontendStartup` - Execute script on frontend load -- `#run=backendStartup` - Execute script on backend load -- `#customWidget` - Custom widget implementation -- `#iconClass` - Custom tree icon -- `#cssClass` - CSS class for note -- `#sorted` - Auto-sort children -- `#hideChildrenOverview` - Don't show child list - -### Revisions Table - -```sql -CREATE TABLE revisions ( - revisionId TEXT NOT NULL PRIMARY KEY, - noteId TEXT NOT NULL, - type TEXT DEFAULT '' NOT NULL, - mime TEXT DEFAULT '' NOT NULL, - title TEXT NOT NULL, - isProtected INT NOT NULL DEFAULT 0, - blobId TEXT DEFAULT NULL, - utcDateLastEdited TEXT NOT NULL, - utcDateCreated TEXT NOT NULL, - utcDateModified TEXT NOT NULL, - dateLastEdited TEXT NOT NULL, - dateCreated TEXT NOT NULL -); - --- Indexes -CREATE INDEX IDX_revisions_noteId ON revisions (noteId); -CREATE INDEX IDX_revisions_utcDateCreated ON revisions (utcDateCreated); -CREATE INDEX IDX_revisions_utcDateLastEdited ON revisions (utcDateLastEdited); -CREATE INDEX IDX_revisions_blobId ON revisions (blobId); -``` - -**Revision Strategy:** -- Automatic revision created on note modification -- Configurable interval (default: daily max) -- Stores complete note snapshot -- Allows reverting to previous versions -- Can be disabled with `#disableVersioning` - -### Attachments Table - -```sql -CREATE TABLE attachments ( - attachmentId TEXT NOT NULL PRIMARY KEY, - ownerId TEXT NOT NULL, - role TEXT NOT NULL, - mime TEXT NOT NULL, - title TEXT NOT NULL, - isProtected INT NOT NULL DEFAULT 0, - position INT DEFAULT 0 NOT NULL, - blobId TEXT DEFAULT NULL, - dateModified TEXT NOT NULL, - utcDateModified TEXT NOT NULL, - utcDateScheduledForErasureSince TEXT DEFAULT NULL, - isDeleted INT NOT NULL, - deleteId TEXT DEFAULT NULL -); - --- Indexes -CREATE INDEX IDX_attachments_ownerId_role ON attachments (ownerId, role); -CREATE INDEX IDX_attachments_blobId ON attachments (blobId); -``` - -**Attachment Roles:** -- `file` - Regular file attachment -- `image` - Image file -- `cover-image` - Note cover image -- Custom roles for specific purposes - -### Blobs Table - -```sql -CREATE TABLE blobs ( - blobId TEXT NOT NULL PRIMARY KEY, - content TEXT NULL DEFAULT NULL, - dateModified TEXT NOT NULL, - utcDateModified TEXT NOT NULL -); -``` - -**Blob Usage:** -- Stores actual content (text or binary) -- Referenced by notes, revisions, attachments -- Deduplication via hash-based blobId -- TEXT type stores both text and binary (base64) - -**Content Types:** -- **Text notes:** HTML content -- **Code notes:** Plain text source code -- **Binary notes:** Base64 encoded data -- **Protected notes:** Encrypted content - -## System Tables - -### Options Table - -```sql -CREATE TABLE options ( - name TEXT NOT NULL PRIMARY KEY, - value TEXT NOT NULL, - isSynced INTEGER DEFAULT 0 NOT NULL, - utcDateModified TEXT NOT NULL -); -``` - -**Key Options:** -- `documentId` - Unique installation ID -- `dbVersion` - Schema version -- `syncVersion` - Sync protocol version -- `passwordVerificationHash` - Password verification -- `encryptedDataKey` - Encryption key (encrypted) -- `theme` - UI theme -- Various feature flags and settings - -**Synced Options:** -- `isSynced = 1` - Synced across devices -- `isSynced = 0` - Local to this installation - -### Entity Changes Table - -```sql -CREATE TABLE entity_changes ( - id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, - entityName TEXT NOT NULL, - entityId TEXT NOT NULL, - hash TEXT NOT NULL, - isErased INT NOT NULL, - changeId TEXT NOT NULL, - componentId TEXT NOT NULL, - instanceId TEXT NOT NULL, - isSynced INTEGER NOT NULL, - utcDateChanged TEXT NOT NULL -); - --- Indexes -CREATE UNIQUE INDEX IDX_entityChanges_entityName_entityId - ON entity_changes (entityName, entityId); -CREATE INDEX IDX_entity_changes_changeId ON entity_changes (changeId); -``` - -**Purpose:** Track all entity modifications for synchronization - -**Entity Types:** -- `notes` -- `branches` -- `attributes` -- `revisions` -- `attachments` -- `options` -- `etapi_tokens` - -### Recent Notes Table - -```sql -CREATE TABLE recent_notes ( - noteId TEXT NOT NULL PRIMARY KEY, - notePath TEXT NOT NULL, - utcDateCreated TEXT NOT NULL -); -``` - -**Purpose:** Track recently accessed notes for quick access - -### Sessions Table - -```sql -CREATE TABLE sessions ( - sid TEXT PRIMARY KEY, - sess TEXT NOT NULL, - expired TEXT NOT NULL -); -``` - -**Purpose:** HTTP session storage for web interface - -### User Data Table - -```sql -CREATE TABLE user_data ( - tmpID INT PRIMARY KEY, - username TEXT, - email TEXT, - userIDEncryptedDataKey TEXT, - userIDVerificationHash TEXT, - salt TEXT, - derivedKey TEXT, - isSetup TEXT DEFAULT "false" -); -``` - -**Purpose:** Store user authentication credentials - -### ETAPI Tokens Table - -```sql -CREATE TABLE etapi_tokens ( - etapiTokenId TEXT PRIMARY KEY NOT NULL, - name TEXT NOT NULL, - tokenHash TEXT NOT NULL, - utcDateCreated TEXT NOT NULL, - utcDateModified TEXT NOT NULL, - isDeleted INT NOT NULL DEFAULT 0 -); -``` - -**Purpose:** API token authentication for external access - -## Data Relationships - -```mermaid -graph TB - Notes[Notes] - Branches[Branches] - Attributes[Attributes] - Attachments[Attachments] - Blobs[(Blobs)] - Revisions[Revisions] - - Notes --> Branches - Notes --> Attributes - Notes --> Attachments - Notes --> Blobs - Notes --> Revisions - - Branches --> Blobs - Attachments --> Blobs - Revisions --> Blobs - - style Notes fill:#e1f5ff - style Blobs fill:#ffe1e1 -``` - -**Relationships:** -- Notes ↔ Branches (many-to-many via noteId) -- Notes → Attributes (one-to-many) -- Notes → Blobs (one-to-one) -- Notes → Revisions (one-to-many) -- Notes → Attachments (one-to-many) -- Attachments → Blobs (one-to-one) -- Revisions → Blobs (one-to-one) - -## Database Access Patterns - -### Direct SQL Access - -**Location:** `apps/server/src/services/sql.ts` - -```typescript -// Execute query (returns rows) -const notes = sql.getRows('SELECT * FROM notes WHERE type = ?', ['text']) - -// Execute query (returns single row) -const note = sql.getRow('SELECT * FROM notes WHERE noteId = ?', [noteId]) - -// Execute statement (no return) -sql.execute('UPDATE notes SET title = ? WHERE noteId = ?', [title, noteId]) - -// Insert -sql.insert('notes', { - noteId: 'new-note-id', - title: 'New Note', - type: 'text', - // ... -}) - -// Transactions -sql.transactional(() => { - sql.execute('UPDATE ...') - sql.execute('INSERT ...') -}) -``` - -### Entity-Based Access (Recommended) - -**Via Becca Cache:** - -```typescript -// Get entity from cache -const note = becca.getNote(noteId) - -// Modify and save -note.title = 'Updated Title' -note.save() // Writes to database - -// Create new -const newNote = becca.createNote({ - parentNoteId: 'root', - title: 'New Note', - type: 'text', - content: 'Hello World' -}) - -// Delete -note.markAsDeleted() -``` - -## Database Migrations - -**Location:** `apps/server/src/migrations/` - -**Migration Files:** -- Format: `XXXX_migration_name.sql` or `XXXX_migration_name.js` -- Executed in numerical order -- Version tracked in `options.dbVersion` - -**SQL Migration Example:** -```sql --- 0280_add_new_column.sql -ALTER TABLE notes ADD COLUMN newField TEXT DEFAULT NULL; - -UPDATE options SET value = '280' WHERE name = 'dbVersion'; -``` - -**JavaScript Migration Example:** -```javascript -// 0285_complex_migration.js -module.exports = () => { - const notes = sql.getRows('SELECT * FROM notes WHERE type = ?', ['old-type']) - - for (const note of notes) { - sql.execute('UPDATE notes SET type = ? WHERE noteId = ?', - ['new-type', note.noteId]) - } -} -``` - -**Migration Process:** -1. Server checks `dbVersion` on startup -2. Compares with latest migration number -3. Executes pending migrations in order -4. Updates `dbVersion` after each -5. Restarts if migrations ran - -## Database Maintenance - -### Backup - -**Full Backup:** -```bash -# Copy database file -cp document.db document.db.backup - -# Or use Trilium's backup feature -# Settings → Backup -``` - -**Automatic Backups:** -- Daily backup (configurable) -- Stored in `backup/` directory -- Retention policy (keep last N backups) - -### Vacuum - -**Purpose:** Reclaim unused space, defragment - -```sql -VACUUM; -``` - -**When to vacuum:** -- After deleting many notes -- Database file size larger than expected -- Performance degradation - -### Integrity Check - -```sql -PRAGMA integrity_check; -``` - -**Result:** "ok" or list of errors - -### Consistency Checks - -**Built-in Consistency Checks:** - -Location: `apps/server/src/services/consistency_checks.ts` - -- Orphaned branches -- Missing parent notes -- Circular dependencies -- Invalid entity references -- Blob reference integrity - -**Run Checks:** -```typescript -// Via API -POST /api/consistency-check - -// Or from backend script -api.runConsistencyChecks() -``` - -## Performance Optimization - -### Indexes - -**Existing Indexes:** -- `notes.title` - Fast title searches -- `notes.type` - Filter by type -- `notes.dateCreated/Modified` - Time-based queries -- `branches.noteId_parentNoteId` - Tree navigation -- `attributes.name_value` - Attribute searches - -**Query Optimization:** -```sql --- Use indexed columns in WHERE clause -SELECT * FROM notes WHERE type = 'text' -- Uses index - --- Avoid functions on indexed columns -SELECT * FROM notes WHERE LOWER(title) = 'test' -- No index - --- Better -SELECT * FROM notes WHERE title = 'Test' -- Uses index -``` - -### Connection Settings - -```typescript -// apps/server/src/services/sql.ts -const db = new Database('document.db', { - // Enable WAL mode for better concurrency - verbose: console.log -}) - -db.pragma('journal_mode = WAL') -db.pragma('synchronous = NORMAL') -db.pragma('cache_size = -64000') // 64MB cache -db.pragma('temp_store = MEMORY') -``` - -**WAL Mode Benefits:** -- Better concurrency (readers don't block writers) -- Faster commits -- More robust - -### Query Performance - -**Use EXPLAIN QUERY PLAN:** -```sql -EXPLAIN QUERY PLAN -SELECT * FROM notes -WHERE type = 'text' - AND dateCreated > '2025-01-01' -``` - -**Analyze slow queries:** -- Check index usage -- Avoid SELECT * -- Use prepared statements -- Batch operations in transactions - -## Database Size Management - -**Typical Sizes:** -- 1,000 notes: ~5-10 MB -- 10,000 notes: ~50-100 MB -- 100,000 notes: ~500 MB - 1 GB - -**Size Reduction Strategies:** - -1. **Delete old revisions** -2. **Remove large attachments** -3. **Vacuum database** -4. **Compact blobs** -5. **Archive old notes** - -**Blob Deduplication:** -- Blobs identified by content hash -- Identical content shares one blob -- Automatic deduplication on insert - -## Security Considerations - -### Protected Notes Encryption - -**Encryption Process:** -```typescript -// Encrypt blob content -const encryptedContent = encrypt(content, dataKey) -blob.content = encryptedContent - -// Store encrypted -sql.insert('blobs', { blobId, content: encryptedContent }) -``` - -**Encryption Details:** -- Algorithm: AES-256-CBC -- Key derivation: PBKDF2 (10,000 iterations) -- Per-note encryption -- Master key encrypted with user password - -### SQL Injection Prevention - -**Always use parameterized queries:** -```typescript -// GOOD - Safe from SQL injection -sql.execute('SELECT * FROM notes WHERE title = ?', [userInput]) - -// BAD - Vulnerable to SQL injection -sql.execute(`SELECT * FROM notes WHERE title = '${userInput}'`) -``` - -### Database File Protection - -**File Permissions:** -- Owner read/write only -- No group/other access -- Located in user-specific directory - ---- - -**See Also:** -- [ARCHITECTURE.md](ARCHITECTURE.md) - Overall architecture -- [Database Schema Files](Developer%20Guide/Developer%20Guide/Development%20and%20architecture/Database/) -- [Migration Scripts](../apps/server/src/migrations/) diff --git a/docs/Developer Guide/!!!meta.json b/docs/Developer Guide/!!!meta.json index f19dbc16a..397bdcab4 100644 --- a/docs/Developer Guide/!!!meta.json +++ b/docs/Developer Guide/!!!meta.json @@ -172,64 +172,57 @@ "children": [ { "isClone": false, - "noteId": "2DJZgzpTJ078", + "noteId": "dsMq2EIOMOBU", "notePath": [ "jdjRLhLV3TtI", "MhwWMgxwDTZL", - "2DJZgzpTJ078" + "dsMq2EIOMOBU" ], - "title": "Client-server architecture", + "title": "Frontend", "notePosition": 10, "prefix": null, "isExpanded": false, "type": "text", "mime": "text/html", - "attributes": [], - "format": "markdown", - "attachments": [], - "dirFileName": "Client-server architecture", - "children": [ + "attributes": [ { - "isClone": false, - "noteId": "dsMq2EIOMOBU", - "notePath": [ - "jdjRLhLV3TtI", - "MhwWMgxwDTZL", - "2DJZgzpTJ078", - "dsMq2EIOMOBU" - ], - "title": "Frontend", - "notePosition": 10, - "prefix": null, - "isExpanded": false, - "type": "text", - "mime": "text/html", - "attributes": [], - "format": "markdown", - "dataFileName": "Frontend.md", - "attachments": [] - }, - { - "isClone": false, - "noteId": "tsswRlmHEnYW", - "notePath": [ - "jdjRLhLV3TtI", - "MhwWMgxwDTZL", - "2DJZgzpTJ078", - "tsswRlmHEnYW" - ], - "title": "Backend", - "notePosition": 20, - "prefix": null, - "isExpanded": false, - "type": "text", - "mime": "text/html", - "attributes": [], - "format": "markdown", - "dataFileName": "Backend.md", - "attachments": [] + "type": "label", + "name": "shareAlias", + "value": "frontend", + "isInheritable": false, + "position": 20 } - ] + ], + "format": "markdown", + "dataFileName": "Frontend.md", + "attachments": [] + }, + { + "isClone": false, + "noteId": "tsswRlmHEnYW", + "notePath": [ + "jdjRLhLV3TtI", + "MhwWMgxwDTZL", + "tsswRlmHEnYW" + ], + "title": "Backend", + "notePosition": 20, + "prefix": null, + "isExpanded": false, + "type": "text", + "mime": "text/html", + "attributes": [ + { + "type": "label", + "name": "shareAlias", + "value": "backend", + "isInheritable": false, + "position": 20 + } + ], + "format": "markdown", + "dataFileName": "Backend.md", + "attachments": [] }, { "isClone": false, @@ -240,7 +233,7 @@ "pRZhrVIGCbMu" ], "title": "Database", - "notePosition": 20, + "notePosition": 40, "prefix": null, "isExpanded": false, "type": "text", @@ -785,15 +778,23 @@ "MhwWMgxwDTZL", "Wxn82Em8B7U5" ], - "title": "API", - "notePosition": 30, + "title": "APIs", + "notePosition": 50, "prefix": null, "isExpanded": false, "type": "text", "mime": "text/html", - "attributes": [], + "attributes": [ + { + "type": "label", + "name": "shareAlias", + "value": "api", + "isInheritable": false, + "position": 20 + } + ], "format": "markdown", - "dataFileName": "API.md", + "dataFileName": "APIs.md", "attachments": [] }, { @@ -805,7 +806,7 @@ "Vk4zD1Iirarg" ], "title": "Arhitecture Decision Records", - "notePosition": 40, + "notePosition": 60, "prefix": null, "isExpanded": false, "type": "text", @@ -817,6 +818,13 @@ "value": "Jg7clqogFOyD", "isInheritable": false, "position": 20 + }, + { + "type": "label", + "name": "shareAlias", + "value": "adr", + "isInheritable": false, + "position": 30 } ], "format": "markdown", @@ -825,14 +833,14 @@ }, { "isClone": false, - "noteId": "QW1MB7RZB5Gf", + "noteId": "RHbKw3xiwk3S", "notePath": [ "jdjRLhLV3TtI", "MhwWMgxwDTZL", - "QW1MB7RZB5Gf" + "RHbKw3xiwk3S" ], - "title": "Security Architecture", - "notePosition": 50, + "title": "Security", + "notePosition": 80, "prefix": null, "isExpanded": false, "type": "text", @@ -841,13 +849,13 @@ { "type": "label", "name": "shareAlias", - "value": "security-architecture", + "value": "security", "isInheritable": false, "position": 20 } ], "format": "markdown", - "dataFileName": "Security Architecture.md", + "dataFileName": "Security.md", "attachments": [] } ] @@ -1153,6 +1161,13 @@ "value": "bx bx-rocket", "isInheritable": false, "position": 30 + }, + { + "type": "label", + "name": "shareAlias", + "value": "releasing", + "isInheritable": false, + "position": 40 } ], "format": "markdown", @@ -1181,6 +1196,13 @@ "value": "bx bxs-component", "isInheritable": false, "position": 20 + }, + { + "type": "label", + "name": "shareAlias", + "value": "dependencies", + "isInheritable": false, + "position": 30 } ], "format": "markdown", @@ -1527,6 +1549,13 @@ "value": "bx bx-microchip", "isInheritable": false, "position": 20 + }, + { + "type": "label", + "name": "shareAlias", + "value": "cache", + "isInheritable": false, + "position": 30 } ], "format": "markdown", @@ -2001,7 +2030,15 @@ "isExpanded": false, "type": "text", "mime": "text/html", - "attributes": [], + "attributes": [ + { + "type": "label", + "name": "shareAlias", + "value": "note-types", + "isInheritable": false, + "position": 20 + } + ], "format": "markdown", "attachments": [], "dirFileName": "Note Types", @@ -2547,6 +2584,7 @@ } ], "format": "markdown", + "dataFileName": "Synchronisation.md", "attachments": [], "dirFileName": "Synchronisation", "children": [ @@ -2794,7 +2832,15 @@ "isExpanded": false, "type": "text", "mime": "text/html", - "attributes": [], + "attributes": [ + { + "type": "label", + "name": "shareAlias", + "value": "unit-tests", + "isInheritable": false, + "position": 20 + } + ], "format": "markdown", "dataFileName": "Unit tests.md", "attachments": [] diff --git a/docs/Developer Guide/Developer Guide/Architecture.md b/docs/Developer Guide/Developer Guide/Architecture.md index 48a91fcde..66cb82eea 100644 --- a/docs/Developer Guide/Developer Guide/Architecture.md +++ b/docs/Developer Guide/Developer Guide/Architecture.md @@ -118,4 +118,85 @@ desktop → client → commons server → client → commons client → ckeditor5, codemirror, highlightjs ckeditor5 → ckeditor5-* plugins -``` \ No newline at end of file +``` + +## Security summary + +### Encryption System + +**Per-Note Encryption:** + +* Notes can be individually protected +* AES-128-CBC encryption for encrypted notes. +* Separate protected session management + +**Protected Session:** + +* Time-limited access to protected notes +* Automatic timeout +* Re-authentication required +* Frontend: `protected_session.ts` +* Backend: `protected_session.ts` + +### Authentication + +**Password Auth:** + +* PBKDF2 key derivation +* Salt per installation +* Hash verification + +**OpenID Connect:** + +* External identity provider support +* OAuth 2.0 flow +* Configurable providers + +**TOTP (2FA):** + +* Time-based one-time passwords +* QR code setup +* Backup codes + +### Authorization + +**Single-User Model:** + +* Desktop: single user (owner) +* Server: single user per installation + +**Share Notes:** + +* Public access without authentication +* Separate Shaca cache +* Read-only access + +### CSRF Protection + +**CSRF Tokens:** + +* Required for state-changing operations +* Token in header or cookie +* Validation middleware + +### Input Sanitization + +**XSS Prevention:** + +* DOMPurify for HTML sanitization +* CKEditor content filtering +* CSP headers + +**SQL Injection:** + +* Parameterized queries only +* Better-sqlite3 prepared statements +* No string concatenation in SQL + +### Dependency Security + +**Vulnerability Scanning:** + +* Renovate bot for updates +* npm audit integration +* Override vulnerable sub-dependencies \ No newline at end of file diff --git a/docs/Developer Guide/Developer Guide/Architecture/API.md b/docs/Developer Guide/Developer Guide/Architecture/APIs.md similarity index 99% rename from docs/Developer Guide/Developer Guide/Architecture/API.md rename to docs/Developer Guide/Developer Guide/Architecture/APIs.md index 92f6fb813..cdd954a61 100644 --- a/docs/Developer Guide/Developer Guide/Architecture/API.md +++ b/docs/Developer Guide/Developer Guide/Architecture/APIs.md @@ -1,4 +1,4 @@ -# API +# APIs ### Internal API **REST Endpoints** (`/api/*`) diff --git a/docs/Developer Guide/Developer Guide/Architecture/Client-server architecture/Backend.md b/docs/Developer Guide/Developer Guide/Architecture/Backend.md similarity index 100% rename from docs/Developer Guide/Developer Guide/Architecture/Client-server architecture/Backend.md rename to docs/Developer Guide/Developer Guide/Architecture/Backend.md diff --git a/docs/Developer Guide/Developer Guide/Architecture/Database.md b/docs/Developer Guide/Developer Guide/Architecture/Database.md index bb80ccdb1..b8836b56d 100644 --- a/docs/Developer Guide/Developer Guide/Architecture/Database.md +++ b/docs/Developer Guide/Developer Guide/Architecture/Database.md @@ -1,5 +1,5 @@ # Database -Trilium uses **SQLite** as its database engine, managed via `better-sqlite3`. +Trilium uses **SQLite** (via `better-sqlite3`) as its embedded database engine, providing a reliable, file-based storage system that requires no separate database server. The database stores all notes, their relationships, metadata, and configuration. Schema location: `apps/server/src/assets/db/schema.sql` diff --git a/docs/Developer Guide/Developer Guide/Architecture/Client-server architecture/Frontend.md b/docs/Developer Guide/Developer Guide/Architecture/Frontend.md similarity index 100% rename from docs/Developer Guide/Developer Guide/Architecture/Client-server architecture/Frontend.md rename to docs/Developer Guide/Developer Guide/Architecture/Frontend.md diff --git a/docs/Developer Guide/Developer Guide/Architecture/Security Architecture.md b/docs/Developer Guide/Developer Guide/Architecture/Security Architecture.md deleted file mode 100644 index ab0c88c06..000000000 --- a/docs/Developer Guide/Developer Guide/Architecture/Security Architecture.md +++ /dev/null @@ -1,79 +0,0 @@ -# Security Architecture -### Encryption System - -**Per-Note Encryption:** - -* Notes can be individually protected -* AES-128-CBC encryption for encrypted notes. -* Separate protected session management - -**Protected Session:** - -* Time-limited access to protected notes -* Automatic timeout -* Re-authentication required -* Frontend: `protected_session.ts` -* Backend: `protected_session.ts` - -### Authentication - -**Password Auth:** - -* PBKDF2 key derivation -* Salt per installation -* Hash verification - -**OpenID Connect:** - -* External identity provider support -* OAuth 2.0 flow -* Configurable providers - -**TOTP (2FA):** - -* Time-based one-time passwords -* QR code setup -* Backup codes - -### Authorization - -**Single-User Model:** - -* Desktop: single user (owner) -* Server: single user per installation - -**Share Notes:** - -* Public access without authentication -* Separate Shaca cache -* Read-only access - -### CSRF Protection - -**CSRF Tokens:** - -* Required for state-changing operations -* Token in header or cookie -* Validation middleware - -### Input Sanitization - -**XSS Prevention:** - -* DOMPurify for HTML sanitization -* CKEditor content filtering -* CSP headers - -**SQL Injection:** - -* Parameterized queries only -* Better-sqlite3 prepared statements -* No string concatenation in SQL - -### Dependency Security - -**Vulnerability Scanning:** - -* Renovate bot for updates -* npm audit integration -* Override vulnerable sub-dependencies \ No newline at end of file diff --git a/docs/Developer Guide/Developer Guide/Architecture/Security.md b/docs/Developer Guide/Developer Guide/Architecture/Security.md new file mode 100644 index 000000000..61fdc7377 --- /dev/null +++ b/docs/Developer Guide/Developer Guide/Architecture/Security.md @@ -0,0 +1,464 @@ +# Security +Trilium implements a **defense-in-depth security model** with multiple layers of protection for user data. The security architecture covers authentication, authorization, encryption, input sanitization, and secure communication. + +## Security Principles + +1. **Data Privacy**: User data is protected at rest and in transit +2. **Encryption**: Per-note encryption for sensitive content +3. **Authentication**: Multiple authentication methods supported +4. **Authorization**: Single-user model with granular protected sessions +5. **Input Validation**: All user input sanitized +6. **Secure Defaults**: Security features enabled by default +7. **Transparency**: Open source allows security audits + +## Threat Model + +### Threats Considered + +1. **Unauthorized Access** + * Physical access to device + * Network eavesdropping + * Stolen credentials + * Session hijacking +2. **Data Exfiltration** + * Malicious scripts + * XSS attacks + * SQL injection + * CSRF attacks +3. **Data Corruption** + * Malicious modifications + * Database tampering + * Sync conflicts +4. **Privacy Leaks** + * Unencrypted backups + * Search indexing + * Temporary files + * Memory dumps + +### Out of Scope + +* Nation-state attackers +* Zero-day vulnerabilities in dependencies +* Hardware vulnerabilities (Spectre, Meltdown) +* Physical access with unlimited time +* Quantum computing attacks + +## Authentication + +### Password Authentication + +**Implementation:** `apps/server/src/services/password.ts` + +### TOTP (Two-Factor Authentication) + +**Implementation:** `apps/server/src/routes/api/login.ts` + +### OpenID Connect + +**Implementation:** `apps/server/src/routes/api/login.ts` + +**Supported Providers:** + +* Any OpenID Connect compatible provider +* Google, GitHub, Auth0, etc. + +**Flow:** + +```typescript +// 1. Redirect to provider +GET /api/login/openid + +// 2. Provider redirects back with code +GET /api/login/openid/callback?code=... + +// 3. Exchange code for tokens +const tokens = await openidClient.callback(redirectUri, req.query) + +// 4. Verify ID token +const claims = tokens.claims() + +// 5. Create session +req.session.loggedIn = true +``` + +### Session Management + +**Session Storage:** SQLite database (sessions table) + +**Session Configuration:** + +```typescript +app.use(session({ + secret: sessionSecret, + resave: false, + saveUninitialized: false, + rolling: true, + cookie: { + maxAge: 7 * 24 * 60 * 60 * 1000, // 7 days + httpOnly: true, + secure: isHttps, + sameSite: 'lax' + }, + store: new SqliteStore({ + db: db, + table: 'sessions' + }) +})) +``` + +**Session Invalidation:** + +* Automatic timeout after inactivity +* Manual logout clears session +* Server restart invalidates all sessions (optional) + +## Authorization + +### Single-User Model + +**Desktop:** + +* Single user (owner of device) +* No multi-user support +* Full access to all notes + +**Server:** + +* Single user per installation +* Authentication required for all operations +* No user roles or permissions + +### Protected Sessions + +**Purpose:** Temporary access to encrypted (protected) notes + +**Implementation:** `apps/server/src/services/protected_session.ts` + +**Workflow:** + +```typescript +// 1. User enters password for protected notes +POST /api/protected-session/enter +Body: { password: "protected-password" } + +// 2. Derive encryption key +const protectedDataKey = deriveKey(password) + +// 3. Verify password (decrypt known encrypted value) +const decrypted = decrypt(testValue, protectedDataKey) +if (decrypted === expectedValue) { + // 4. Store in memory (not in session) + protectedSessionHolder.setProtectedDataKey(protectedDataKey) + + // 5. Set timeout + setTimeout(() => { + protectedSessionHolder.clearProtectedDataKey() + }, timeout) +} +``` + +**Protected Session Timeout:** + +* Default: 10 minutes (configurable) +* Extends on activity +* Cleared on browser close +* Separate from main session + +### API Authorization + +**Internal API:** + +* Requires authenticated session +* CSRF token validation +* Same-origin policy + +**ETAPI (External API):** + +* Token-based authentication +* No session required +* Rate limiting + +## Encryption + +### Note Encryption + +**Encryption Algorithm:** AES-256-CBC + +**Key Hierarchy:** + +``` +User Password + ↓ (scrypt) +Data Key (for protected notes) + ↓ (AES-128) +Protected Note Content +``` + +**Protected Note Metadata:** + +* Content IS encrypted +* Type and MIME are NOT encrypted +* Attributes are NOT encrypted + +### Data Key Management + +**Key Rotation:** + +* Not currently supported +* Requires re-encrypting all protected notes + +### Transport Encryption + +**HTTPS:** + +* Recommended for server installations +* TLS 1.2+ only +* Strong cipher suites preferred +* Certificate validation enabled + +**Desktop:** + +* Local communication (no network) +* No HTTPS required + +### Backup Encryption + +**Database Backups:** + +* Protected notes remain encrypted in backup +* Backup file should be protected separately +* Consider encrypting backup storage location + +## Input Sanitization + +### XSS Prevention + +* **HTML Sanitization** +* **CKEditor Configuration:** + + ``` + // apps/client/src/widgets/type_widgets/text_type_widget.ts + ClassicEditor.create(element, { + // Restrict allowed content + htmlSupport: { + allow: [ + { name: /./, attributes: true, classes: true, styles: true } + ], + disallow: [ + { name: 'script' }, + { name: 'iframe', attributes: /^(?!src$).*/ } + ] + } + }) + ``` +* Content Security Policy + +### SQL Injection Prevention + +**Parameterized Queries:** + +```typescript +const notes = sql.getRows( + 'SELECT * FROM notes WHERE title = ?', + [userInput] +) +``` + +**ORM Usage:** + +```typescript +// Entity-based access prevents SQL injection +const note = becca.getNote(noteId) +note.title = userInput // Sanitized by entity +note.save() // Parameterized query +``` + +### CSRF Prevention + +**CSRF Token Validation:** + +Location: `apps/server/src/routes/csrf_protection.ts` + +Stateless CSRF using [Double Submit Cookie Pattern](https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#double-submit-cookie) via [`csrf-csrf`](https://github.com/Psifi-Solutions/csrf-csrf). + +### File Upload Validation + +**Validation:** + +```typescript +// Validate file size +const maxSize = 100 * 1024 * 1024 // 100 MB +if (file.size > maxSize) { + throw new Error('File too large') +} +``` + +## Network Security + +### HTTPS Configuration + +**Certificate Validation:** + +* Require valid certificates in production +* Self-signed certificates allowed for development +* Certificate pinning not implemented + +### Rate Limiting + +**Login Rate Limiting:** + +```typescript +const loginLimiter = rateLimit({ + windowMs: 15 * 60 * 1000, + max: 10, // 10 failed attempts + skipSuccessfulRequests: true +}) + +app.post('/api/login/password', loginLimiter, loginHandler) +``` + +## Data Security + +### Secure Data Deletion + +**Soft Delete:** + +```typescript +// Mark as deleted (sync first) +note.isDeleted = 1 +note.deleteId = generateUUID() +note.save() + +// Entity change tracked for sync +addEntityChange('notes', noteId, note) +``` + +**Hard Delete (Erase):** + +```typescript +// After sync completed +sql.execute('DELETE FROM notes WHERE noteId = ?', [noteId]) +sql.execute('DELETE FROM branches WHERE noteId = ?', [noteId]) +sql.execute('DELETE FROM attributes WHERE noteId = ?', [noteId]) + +// Mark entity change as erased +sql.execute('UPDATE entity_changes SET isErased = 1 WHERE entityId = ?', [noteId]) +``` + +**Blob Cleanup:** + +```typescript +// Find orphaned blobs (not referenced by any note/revision/attachment) +const orphanedBlobs = sql.getRows(` + SELECT blobId FROM blobs + WHERE blobId NOT IN (SELECT blobId FROM notes WHERE blobId IS NOT NULL) + AND blobId NOT IN (SELECT blobId FROM revisions WHERE blobId IS NOT NULL) + AND blobId NOT IN (SELECT blobId FROM attachments WHERE blobId IS NOT NULL) +`) + +// Delete orphaned blobs +for (const blob of orphanedBlobs) { + sql.execute('DELETE FROM blobs WHERE blobId = ?', [blob.blobId]) +} +``` + +### Memory Security + +**Protected Data in Memory:** + +* Protected data keys stored in memory only +* Cleared on timeout +* Not written to disk +* Not in session storage + +## Dependency Security + +### Vulnerability Scanning + +**Tools:** + +* Renovate bot - Automatic dependency updates +* `pnpm audit` - Check for known vulnerabilities +* GitHub Dependabot alerts + +**Process:** + +```sh +# Check for vulnerabilities +npm audit + +# Fix automatically +npm audit fix + +# Manual review for breaking changes +npm audit fix --force +``` + +### Dependency Pinning + +**package.json:** + +``` +{ + "dependencies": { + "express": "4.18.2", // Exact version + "better-sqlite3": "^9.2.2" // Compatible versions + } +} +``` + +**pnpm Overrides:** + +``` +{ + "pnpm": { + "overrides": { + "lodash@<4.17.21": ">=4.17.21", // Force minimum version + "axios@<0.21.2": ">=0.21.2" + } + } +} +``` + +### Patch Management + +**pnpm Patches:** + +```sh +# Create patch +pnpm patch @ckeditor/ckeditor5 + +# Edit files in temporary directory +# ... + +# Generate patch file +pnpm patch-commit /tmp/ckeditor5-patch + +# Patch applied automatically on install +``` + +## Security Auditing + +### Logs + +**Security Events Logged:** + +* Login attempts (success/failure) +* Protected session access +* Password changes +* ETAPI token usage +* Failed CSRF validations + +**Log Location:** + +* Desktop: Console output +* Server: Log files or stdout + +### Monitoring + +**Metrics to Monitor:** + +* Failed login attempts +* API error rates +* Unusual database changes +* Large exports/imports \ No newline at end of file diff --git a/docs/SYNCHRONIZATION.md b/docs/Developer Guide/Developer Guide/Concepts/Synchronisation.md similarity index 52% rename from docs/SYNCHRONIZATION.md rename to docs/Developer Guide/Developer Guide/Concepts/Synchronisation.md index 0945e533a..f97bd3c4e 100644 --- a/docs/SYNCHRONIZATION.md +++ b/docs/Developer Guide/Developer Guide/Concepts/Synchronisation.md @@ -1,16 +1,11 @@ -# Trilium Synchronization Architecture +# Synchronisation +Trilium implements a **bidirectional synchronization system** that allows users to sync their note databases across multiple devices (desktop clients and server instances). The sync protocol is designed to handle: -> **Related:** [ARCHITECTURE.md](ARCHITECTURE.md) | [User Guide: Synchronization](https://triliumnext.github.io/Docs/Wiki/synchronization) - -## Overview - -Trilium implements a sophisticated **bidirectional synchronization system** that allows users to sync their note databases across multiple devices (desktop clients and server instances). The sync protocol is designed to handle: - -- Concurrent modifications across devices -- Conflict resolution -- Partial sync (only changed entities) -- Protected note synchronization -- Efficient bandwidth usage +* Concurrent modifications across devices +* Simple conflict resolution (without “merge conflict” indication). +* Partial sync (only changed entities) +* Protected note synchronization +* Efficient bandwidth usage ## Sync Architecture @@ -35,7 +30,7 @@ graph TB Every modification to any entity (note, branch, attribute, etc.) creates an **entity change** record: -```sql +``` entity_changes ( id, -- Auto-increment ID entityName, -- 'notes', 'branches', 'attributes', etc. @@ -43,7 +38,7 @@ entity_changes ( hash, -- Content hash for integrity isErased, -- If entity was erased (deleted permanently) changeId, -- Unique change identifier - componentId, -- Installation identifier + componentId, -- Unique component/widget identifier instanceId, -- Process instance identifier isSynced, -- Whether synced to server utcDateChanged -- When change occurred @@ -51,17 +46,19 @@ entity_changes ( ``` **Key Properties:** -- **changeId**: Globally unique identifier (UUID) for the change -- **componentId**: Unique per Trilium installation (persists across restarts) -- **instanceId**: Unique per process (changes on restart) -- **hash**: SHA-256 hash of entity data for integrity verification + +* **changeId**: Globally unique identifier (UUID) for the change +* **componentId**: Unique identifier of the component/widget that generated to change (can be used to avoid refreshing the widget being edited). +* **instanceId**: Unique per process (changes on restart) +* **hash**: SHA-256 hash of entity data for integrity verification ### Sync Versions Each Trilium installation tracks: -- **Local sync version**: Highest change ID seen locally -- **Server sync version**: Highest change ID on server -- **Entity versions**: Last sync version for each entity type + +* **Local sync version**: Highest change ID seen locally +* **Server sync version**: Highest change ID on server +* **Entity versions**: Last sync version for each entity type ### Change Tracking @@ -87,11 +84,12 @@ function addEntityChange(entityName, entityId, entity) { ``` **Entity modification triggers:** -- Note content update -- Note metadata change -- Branch creation/deletion/reorder -- Attribute addition/removal -- Options modification + +* Note content update +* Note metadata change +* Branch creation/deletion/reorder +* Attribute addition/removal +* Options modification ## Sync Protocol @@ -122,9 +120,9 @@ Response: **Step 3: Decision** -- If `entityChanges > 0`: Pull changes from server -- If `outstandingPushCount > 0`: Push changes to server -- Both can happen in sequence +* If `entityChanges > 0`: Pull changes from server +* If `outstandingPushCount > 0`: Push changes to server +* Both can happen in sequence ### Pull Sync (Server → Client) @@ -159,10 +157,10 @@ Response: **Client Processing:** -1. Apply entity changes to local database -2. Update Froca cache -3. Update local sync version -4. Trigger UI refresh +1. Apply entity changes to local database +2. Update Froca cache +3. Update local sync version +4. Trigger UI refresh ### Push Sync (Client → Server) @@ -191,12 +189,12 @@ POST /api/sync/push **Server Processing:** -1. Validate changes -2. Check for conflicts -3. Apply changes to database -4. Update Becca cache -5. Mark as synced -6. Broadcast to other connected clients via WebSocket +1. Validate changes +2. Check for conflicts +3. Apply changes to database +4. Update Becca cache +5. Mark as synced +6. Broadcast to other connected clients via WebSocket **Conflict Detection:** @@ -215,21 +213,25 @@ if (serverLastModified > clientSyncVersion) { ### Conflict Types -**1. Content Conflict** -- Both client and server modified same note content -- **Resolution**: Last-write-wins based on `utcDateModified` +**1\. Content Conflict** -**2. Structure Conflict** -- Branch moved/deleted on one side, modified on other -- **Resolution**: Tombstone records, reconciliation +* Both client and server modified same note content +* **Resolution**: Last-write-wins based on `utcDateModified` -**3. Attribute Conflict** -- Same attribute modified differently -- **Resolution**: Last-write-wins +**2\. Structure Conflict** + +* Branch moved/deleted on one side, modified on other +* **Resolution**: Tombstone records, reconciliation + +**3\. Attribute Conflict** + +* Same attribute modified differently +* **Resolution**: Last-write-wins ### Conflict Resolution Strategy **Last-Write-Wins:** + ```typescript if (clientEntity.utcDateModified > serverEntity.utcDateModified) { // Client wins, apply client changes @@ -241,9 +243,10 @@ if (clientEntity.utcDateModified > serverEntity.utcDateModified) { ``` **Tombstone Records:** -- Deleted entities leave tombstone in `entity_changes` -- Prevents re-sync of deleted items -- `isErased = 1` for permanent deletions + +* Deleted entities leave tombstone in `entity_changes` +* Prevents re-sync of deleted items +* `isErased = 1` for permanent deletions ### Protected Notes Sync @@ -251,43 +254,26 @@ if (clientEntity.utcDateModified > serverEntity.utcDateModified) { **Solution:** -1. **Protected session required**: User must unlock protected notes -2. **Encrypted sync**: Content synced in encrypted form -3. **Hash verification**: Integrity checked without decryption -4. **Lazy decryption**: Only decrypt when accessed - -**Sync Flow:** - -```typescript -// Client side -if (note.isProtected && !protectedSessionHolder.isProtectedSessionAvailable()) { - // Skip protected notes if session not active - continue -} - -// Server side -if (note.isProtected) { - // Sync encrypted blob - // Don't decrypt for sync - syncEncryptedBlob(note.blobId) -} -``` +1. **Encrypted sync**: Content synced in encrypted form +2. **Hash verification**: Integrity checked without decryption +3. **Lazy decryption**: Only decrypt when accessed ## Sync States ### Connection States -- **Connected**: WebSocket connection active -- **Disconnected**: No connection to sync server -- **Syncing**: Actively transferring data -- **Conflict**: Sync paused due to conflict +* **Connected**: WebSocket connection active +* **Disconnected**: No connection to sync server +* **Syncing**: Actively transferring data +* **Conflict**: Sync paused due to conflict ### Entity Sync States Each entity can be in: -- **Synced**: In sync with server -- **Pending**: Local changes not yet pushed -- **Conflict**: Conflicting changes detected + +* **Synced**: In sync with server +* **Pending**: Local changes not yet pushed +* **Conflict**: Conflicting changes detected ### UI Indicators @@ -299,8 +285,6 @@ class SyncStatusWidget { showIcon('synced') } else if (isSyncing) { showIcon('syncing-spinner') - } else if (hasConflicts) { - showIcon('conflict-warning') } else { showIcon('not-synced') } @@ -314,7 +298,7 @@ class SyncStatusWidget { Only entities changed since last sync are transferred: -```sql +``` SELECT * FROM entity_changes WHERE id > :lastSyncedChangeId ORDER BY id ASC @@ -357,26 +341,12 @@ res.send(gzip(syncData)) ### Network Errors -**Retry Strategy:** -```typescript -const RETRY_DELAYS = [1000, 2000, 5000, 10000, 30000] - -async function syncWithRetry(attempt = 0) { - try { - await performSync() - } catch (error) { - if (attempt < RETRY_DELAYS.length) { - setTimeout(() => { - syncWithRetry(attempt + 1) - }, RETRY_DELAYS[attempt]) - } - } -} -``` +Reported to the user and the sync will be retried after the interval passes. ### Sync Integrity Checks **Hash Verification:** + ```typescript // Verify entity hash matches const calculatedHash = calculateHash(entity) @@ -388,16 +358,18 @@ if (calculatedHash !== receivedHash) { ``` **Consistency Checks:** -- Orphaned branches detection -- Missing parent notes -- Invalid entity references -- Circular dependencies + +* Orphaned branches detection +* Missing parent notes +* Invalid entity references +* Circular dependencies ## Sync Server Configuration ### Server Setup **Required Options:** + ```javascript { "syncServerHost": "https://sync.example.com", @@ -407,59 +379,26 @@ if (calculatedHash !== receivedHash) { ``` **Authentication:** -- Username/password or -- Sync token (generated on server) -### Client Setup - -**Desktop Client:** -```javascript -// Settings → Sync -{ - "syncServerHost": "https://sync.example.com", - "username": "user@example.com", - "password": "********" -} -``` - -**Test Connection:** -```typescript -POST /api/sync/test -Response: { "success": true } -``` +* Username/password or +* Sync token (generated on server) ## Sync API Endpoints Located at: `apps/server/src/routes/api/sync.ts` -**Endpoints:** - -- `POST /api/sync/check` - Check sync status -- `POST /api/sync/pull` - Pull changes from server -- `POST /api/sync/push` - Push changes to server -- `POST /api/sync/finished` - Mark sync complete -- `POST /api/sync/test` - Test connection -- `GET /api/sync/stats` - Sync statistics - ## WebSocket Sync Updates Real-time sync via WebSocket: ```typescript // Server broadcasts change to all connected clients -ws.broadcast('entity-change', { - entityName: 'notes', - entityId: 'abc123', - changeId: 'change-uuid', - sourceId: 'originating-component-id' +ws.broadcast('frontend-update', { + lastSyncedPush, + entityChanges }) -// Client receives and applies -ws.on('entity-change', (data) => { - if (data.sourceId !== myComponentId) { - froca.processEntityChange(data) - } -}) +// Client receives and processed the information. ``` ## Sync Scheduling @@ -467,107 +406,79 @@ ws.on('entity-change', (data) => { ### Automatic Sync **Desktop:** -- Sync on startup -- Periodic sync (configurable interval, default: 60s) -- Sync before shutdown + +* Sync on startup +* Periodic sync (configurable interval, default: 60s) **Server:** -- Sync on entity modification -- WebSocket push to connected clients + +* Sync on entity modification +* WebSocket push to connected clients ### Manual Sync User can trigger: -- Full sync -- Sync now -- Sync specific subtree + +* Full sync +* Sync now +* Sync specific subtree ## Troubleshooting ### Common Issues **Sync stuck:** -```sql + +``` -- Reset sync state UPDATE entity_changes SET isSynced = 0; DELETE FROM options WHERE name LIKE 'sync%'; ``` **Hash mismatch:** -- Data corruption detected -- Re-sync from backup -- Check database integrity + +* Data corruption detected +* Re-sync from backup +* Check database integrity **Conflict loop:** -- Manual intervention required -- Export conflicting notes -- Choose winning version -- Re-sync -### Sync Diagnostics - -**Check sync status:** -```typescript -GET /api/sync/stats -Response: { - "unsyncedChanges": 0, - "lastSyncDate": "2025-11-02T12:00:00Z", - "syncVersion": 12890 -} -``` - -**Entity change log:** -```sql -SELECT * FROM entity_changes -WHERE isSynced = 0 -ORDER BY id DESC; -``` +* Manual intervention required +* Export conflicting notes +* Choose winning version +* Re-sync ## Security Considerations ### Encrypted Sync -- Protected notes synced encrypted -- No plain text over network -- Server cannot read protected content +* Protected notes synced encrypted +* No plain text over network +* Server cannot read protected content ### Authentication -- Username/password over HTTPS only -- Sync tokens for token-based auth -- Session cookies with CSRF protection +* Username/password over HTTPS only +* Sync tokens for token-based auth +* Session cookies with CSRF protection ### Authorization -- Users can only sync their own data -- No cross-user sync support -- Sync server validates ownership +* Users can only sync their own data +* No cross-user sync support +* Sync server validates ownership ## Performance Metrics **Typical Sync Performance:** -- 1000 changes: ~2-5 seconds -- 10000 changes: ~20-50 seconds -- Initial full sync (100k notes): ~5-10 minutes + +* 1000 changes: ~2-5 seconds +* 10000 changes: ~20-50 seconds +* Initial full sync (100k notes): ~5-10 minutes **Factors:** -- Network latency -- Database size -- Number of protected notes -- Attachment sizes -## Future Improvements - -**Planned Enhancements:** -- Differential sync (binary diff) -- Peer-to-peer sync (no central server) -- Multi-server sync -- Partial sync (subtree only) -- Sync over Tor/I2P - ---- - -**See Also:** -- [ARCHITECTURE.md](ARCHITECTURE.md) - Overall architecture -- [Sync User Guide](https://triliumnext.github.io/Docs/Wiki/synchronization) -- [Sync API Source](../apps/server/src/routes/api/sync.ts) +* Network latency +* Database size +* Number of protected notes +* Attachment sizes \ No newline at end of file diff --git a/docs/QUICK_REFERENCE.md b/docs/QUICK_REFERENCE.md deleted file mode 100644 index 396cc0dd6..000000000 --- a/docs/QUICK_REFERENCE.md +++ /dev/null @@ -1,155 +0,0 @@ -# Trilium Technical Documentation - Quick Reference - -> **Start here:** [TECHNICAL_DOCUMENTATION.md](TECHNICAL_DOCUMENTATION.md) - Complete index of all documentation - -## 📖 Documentation Files - -| Document | Description | Size | Lines | -|----------|-------------|------|-------| -| [TECHNICAL_DOCUMENTATION.md](TECHNICAL_DOCUMENTATION.md) | Main index and navigation hub | 13KB | 423 | -| [ARCHITECTURE.md](ARCHITECTURE.md) | Complete system architecture | 30KB | 1,016 | -| [DATABASE.md](DATABASE.md) | Database schema and operations | 19KB | 736 | -| [SYNCHRONIZATION.md](SYNCHRONIZATION.md) | Sync protocol and implementation | 14KB | 583 | -| [SCRIPTING.md](SCRIPTING.md) | User scripting system guide | 17KB | 734 | -| [SECURITY_ARCHITECTURE.md](SECURITY_ARCHITECTURE.md) | Security implementation details | 19KB | 834 | - -**Total:** 112KB of comprehensive documentation across 4,326 lines! - -## 🎯 Quick Access by Role - -### 👤 End Users -- **Getting Started:** [User Guide](User%20Guide/User%20Guide/) -- **Scripting:** [SCRIPTING.md](SCRIPTING.md) -- **Sync Setup:** [SYNCHRONIZATION.md](SYNCHRONIZATION.md) - -### 💻 Developers -- **Architecture:** [ARCHITECTURE.md](ARCHITECTURE.md) -- **Development Setup:** [Developer Guide](Developer%20Guide/Developer%20Guide/Environment%20Setup.md) -- **Database:** [DATABASE.md](DATABASE.md) - -### 🔒 Security Auditors -- **Security:** [SECURITY_ARCHITECTURE.md](SECURITY_ARCHITECTURE.md) -- **Encryption:** [SECURITY_ARCHITECTURE.md#encryption](SECURITY_ARCHITECTURE.md#encryption) -- **Auth:** [SECURITY_ARCHITECTURE.md#authentication](SECURITY_ARCHITECTURE.md#authentication) - -### 🏗️ System Architects -- **Overall Design:** [ARCHITECTURE.md](ARCHITECTURE.md) -- **Cache System:** [ARCHITECTURE.md#three-layer-cache-system](ARCHITECTURE.md#three-layer-cache-system) -- **Entity Model:** [ARCHITECTURE.md#entity-system](ARCHITECTURE.md#entity-system) - -### 🔧 DevOps Engineers -- **Server Installation:** [User Guide - Server Installation](User%20Guide/User%20Guide/Installation%20&%20Setup/Server%20Installation.md) -- **Docker:** [Developer Guide - Docker](Developer%20Guide/Developer%20Guide/Development%20and%20architecture/Docker.md) -- **Sync Server:** [SYNCHRONIZATION.md#sync-server-configuration](SYNCHRONIZATION.md#sync-server-configuration) - -### 📊 Database Administrators -- **Schema:** [DATABASE.md#database-schema](DATABASE.md#database-schema) -- **Maintenance:** [DATABASE.md#database-maintenance](DATABASE.md#database-maintenance) -- **Performance:** [DATABASE.md#performance-optimization](DATABASE.md#performance-optimization) - -## 🔍 Quick Topic Finder - -### Core Concepts -- **Becca Cache:** [ARCHITECTURE.md#1-becca-backend-cache](ARCHITECTURE.md#1-becca-backend-cache) -- **Froca Cache:** [ARCHITECTURE.md#2-froca-frontend-cache](ARCHITECTURE.md#2-froca-frontend-cache) -- **Entity System:** [ARCHITECTURE.md#entity-system](ARCHITECTURE.md#entity-system) -- **Widget System:** [ARCHITECTURE.md#widget-based-ui](ARCHITECTURE.md#widget-based-ui) - -### Database -- **Schema Overview:** [DATABASE.md#schema-overview](DATABASE.md#schema-overview) -- **Notes Table:** [DATABASE.md#notes-table](DATABASE.md#notes-table) -- **Branches Table:** [DATABASE.md#branches-table](DATABASE.md#branches-table) -- **Migrations:** [DATABASE.md#database-migrations](DATABASE.md#database-migrations) - -### Synchronization -- **Sync Protocol:** [SYNCHRONIZATION.md#sync-protocol](SYNCHRONIZATION.md#sync-protocol) -- **Conflict Resolution:** [SYNCHRONIZATION.md#conflict-resolution](SYNCHRONIZATION.md#conflict-resolution) -- **Entity Changes:** [SYNCHRONIZATION.md#entity-changes](SYNCHRONIZATION.md#entity-changes) - -### Scripting -- **Frontend Scripts:** [SCRIPTING.md#frontend-scripts](SCRIPTING.md#frontend-scripts) -- **Backend Scripts:** [SCRIPTING.md#backend-scripts](SCRIPTING.md#backend-scripts) -- **Script Examples:** [SCRIPTING.md#script-examples](SCRIPTING.md#script-examples) -- **API Reference:** [SCRIPTING.md#script-api](SCRIPTING.md#script-api) - -### Security -- **Authentication:** [SECURITY_ARCHITECTURE.md#authentication](SECURITY_ARCHITECTURE.md#authentication) -- **Encryption:** [SECURITY_ARCHITECTURE.md#encryption](SECURITY_ARCHITECTURE.md#encryption) -- **Input Sanitization:** [SECURITY_ARCHITECTURE.md#input-sanitization](SECURITY_ARCHITECTURE.md#input-sanitization) -- **Best Practices:** [SECURITY_ARCHITECTURE.md#security-best-practices](SECURITY_ARCHITECTURE.md#security-best-practices) - -## 📚 Learning Paths - -### New to Trilium Development -1. Read [ARCHITECTURE.md](ARCHITECTURE.md) - System overview -2. Setup environment: [Environment Setup](Developer%20Guide/Developer%20Guide/Environment%20Setup.md) -3. Explore [DATABASE.md](DATABASE.md) - Understand data model -4. Check [Developer Guide](Developer%20Guide/Developer%20Guide/) - -### Want to Create Scripts -1. Read [SCRIPTING.md](SCRIPTING.md) - Complete guide -2. Check [Script API](Script%20API/) - API reference -3. Review examples: [SCRIPTING.md#script-examples](SCRIPTING.md#script-examples) -4. Explore [Advanced Showcases](https://triliumnext.github.io/Docs/Wiki/advanced-showcases) - -### Setting Up Sync -1. Understand protocol: [SYNCHRONIZATION.md](SYNCHRONIZATION.md) -2. Configure server: [SYNCHRONIZATION.md#sync-server-configuration](SYNCHRONIZATION.md#sync-server-configuration) -3. Setup clients: [SYNCHRONIZATION.md#client-setup](SYNCHRONIZATION.md#client-setup) -4. Troubleshoot: [SYNCHRONIZATION.md#troubleshooting](SYNCHRONIZATION.md#troubleshooting) - -### Security Review -1. Read threat model: [SECURITY_ARCHITECTURE.md#threat-model](SECURITY_ARCHITECTURE.md#threat-model) -2. Review authentication: [SECURITY_ARCHITECTURE.md#authentication](SECURITY_ARCHITECTURE.md#authentication) -3. Check encryption: [SECURITY_ARCHITECTURE.md#encryption](SECURITY_ARCHITECTURE.md#encryption) -4. Verify best practices: [SECURITY_ARCHITECTURE.md#security-best-practices](SECURITY_ARCHITECTURE.md#security-best-practices) - -## 🗺️ Documentation Map - -``` -docs/ -├── TECHNICAL_DOCUMENTATION.md ← START HERE (Index) -│ -├── Core Technical Docs -│ ├── ARCHITECTURE.md (System design) -│ ├── DATABASE.md (Data layer) -│ ├── SYNCHRONIZATION.md (Sync system) -│ ├── SCRIPTING.md (User scripting) -│ └── SECURITY_ARCHITECTURE.md (Security) -│ -├── Developer Guide/ -│ └── Developer Guide/ (Development setup) -│ -├── User Guide/ -│ └── User Guide/ (End-user docs) -│ -└── Script API/ (API reference) -``` - -## 💡 Tips for Reading Documentation - -1. **Start with the index:** [TECHNICAL_DOCUMENTATION.md](TECHNICAL_DOCUMENTATION.md) provides an overview -2. **Use search:** Press Ctrl+F / Cmd+F to find specific topics -3. **Follow links:** Documents are cross-referenced for easy navigation -4. **Code examples:** Most docs include practical code examples -5. **See Also sections:** Check bottom of each doc for related resources - -## 🔗 External Resources - -- **Website:** https://triliumnotes.org -- **Online Docs:** https://docs.triliumnotes.org -- **GitHub:** https://github.com/TriliumNext/Trilium -- **Discussions:** https://github.com/TriliumNext/Trilium/discussions -- **Matrix Chat:** https://matrix.to/#/#triliumnext:matrix.org - -## 🤝 Contributing to Documentation - -Found an error or want to improve the docs? See: -- [Contributing Guide](../README.md#-contribute) -- [Documentation Standards](TECHNICAL_DOCUMENTATION.md#documentation-conventions) - ---- - -**Version:** 0.99.3 -**Last Updated:** November 2025 -**Maintained by:** TriliumNext Team diff --git a/docs/SCRIPTING.md b/docs/SCRIPTING.md deleted file mode 100644 index 49e548c50..000000000 --- a/docs/SCRIPTING.md +++ /dev/null @@ -1,734 +0,0 @@ -# Trilium Scripting System - -> **Related:** [ARCHITECTURE.md](ARCHITECTURE.md) | [Script API Documentation](Script%20API/) - -## Overview - -Trilium features a **powerful scripting system** that allows users to extend and customize the application without modifying source code. Scripts are written in JavaScript and can execute both in the **frontend (browser)** and **backend (Node.js)** contexts. - -## Script Types - -### Frontend Scripts - -**Location:** Attached to notes with `#run=frontendStartup` attribute - -**Execution Context:** Browser environment - -**Access:** -- Trilium Frontend API -- Browser APIs (DOM, localStorage, etc.) -- Froca (frontend cache) -- UI widgets -- No direct file system access - -**Lifecycle:** -- `frontendStartup` - Run once when Trilium loads -- `frontendReload` - Run on every note context change - -**Example:** -```javascript -// Attach to note with #run=frontendStartup -const api = window.api - -// Add custom button to toolbar -api.addButtonToToolbar({ - title: 'My Button', - icon: 'star', - action: () => { - api.showMessage('Hello from frontend!') - } -}) -``` - -### Backend Scripts - -**Location:** Attached to notes with `#run=backendStartup` attribute - -**Execution Context:** Node.js server environment - -**Access:** -- Trilium Backend API -- Node.js APIs (fs, http, etc.) -- Becca (backend cache) -- Database (SQL) -- External libraries (via require) - -**Lifecycle:** -- `backendStartup` - Run once when server starts -- Event handlers (custom events) - -**Example:** -```javascript -// Attach to note with #run=backendStartup -const api = require('@triliumnext/api') - -// Listen for note creation -api.dayjs // Example: access dayjs library - -api.onNoteCreated((note) => { - if (note.title.includes('TODO')) { - note.setLabel('priority', 'high') - } -}) -``` - -### Render Scripts - -**Location:** Attached to notes with `#customWidget` or similar attributes - -**Purpose:** Custom note rendering/widgets - -**Example:** -```javascript -// Custom widget for a note -class MyWidget extends api.NoteContextAwareWidget { - doRender() { - this.$widget = $('