NIP-78 encrypted transport, instant UI, re-sync recovery

Nostr transport:
- Switch from kind 1 to NIP-78 kind 30078 (application-specific data)
- Relays store events persistently — no inventory protocol needed
- AES-256-GCM encryption via Web Crypto API (room key = shared secret)
- PBKDF2 key derivation (100k iterations) from room key
- Chunking for large events (images >48KB) with per-chunk encryption
- Auto re-publish missing local events on room join
- Manual "Re-sync" button for recovery of failed publishes

Performance (all variants):
- Emit 'change' immediately after local store write (IDB/NeDB/SQLite)
- Folder and Nostr writes run fire-and-forget in background
- Fast event hashing: fingerprint metadata only, skip full payload SHA-256
- Saving spinner in paste UI while write completes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jason Tudisco 2026-03-18 01:11:29 -06:00
parent fb3147f0c9
commit 7077191c0c
13 changed files with 558 additions and 242 deletions

View File

@ -116,15 +116,7 @@ export class SyncEngine {
await this.applyEvent(event); await this.applyEvent(event);
await this.idb.markEventApplied(filename); await this.idb.markEventApplied(filename);
// Then persist to folder (if available) // Emit change immediately after IDB write
if (this.folder.hasHandle && (await this.folder.hasPermission())) {
try {
await this.folder.writeEvent(filename, event);
} catch (err) {
this.emitter.emit('folder:lost-permission', err);
}
}
this.emitter.emit('change', { this.emitter.emit('change', {
type: event.type, type: event.type,
store: event.store, store: event.store,
@ -132,6 +124,20 @@ export class SyncEngine {
id: event.id, id: event.id,
data: event.data, data: event.data,
}); });
// Persist to folder in the background (fire and forget)
this.persistToFolder(filename, event);
}
/** Write event to folder store without blocking the caller. */
private persistToFolder(filename: string, event: SyncEvent): void {
if (!this.folder.hasHandle) return;
this.folder.hasPermission().then((ok) => {
if (!ok) return;
this.folder.writeEvent(filename, event).catch((err) => {
this.emitter.emit('folder:lost-permission', err);
});
});
} }
// ── Sync: import folder events into IDB ──────────────────── // ── Sync: import folder events into IDB ────────────────────

View File

@ -103,14 +103,8 @@ export class SyncEngine {
await this.applyEvent(event); await this.applyEvent(event);
await this.store.markEventApplied(filename); await this.store.markEventApplied(filename);
if (this.folder.hasHandle && (await this.folder.hasPermission())) { // Emit change immediately so the UI updates without waiting for the
try { // (potentially slow) folder write.
await this.folder.writeEvent(filename, event);
} catch (err) {
this.emitter.emit('folder:lost-permission', err);
}
}
this.emitter.emit('change', { this.emitter.emit('change', {
type: event.type, type: event.type,
store: event.store, store: event.store,
@ -118,6 +112,13 @@ export class SyncEngine {
id: event.id, id: event.id,
data: event.data, data: event.data,
}); });
// Fire-and-forget: persist to the sync folder in the background.
if (this.folder.hasHandle && (await this.folder.hasPermission())) {
this.folder.writeEvent(filename, event).catch((err) => {
this.emitter.emit('folder:lost-permission', err);
});
}
} }
// ── Sync ─────────────────────────────────────────────────── // ── Sync ───────────────────────────────────────────────────

View File

@ -29,19 +29,64 @@ Use either or both:
## How it works ## How it works
**On write:** ### NIP-78 application data (kind 30078)
1. Update IndexedDB (fast)
2. Write event file to folder (if folder connected)
3. Publish event to Nostr relay (if room joined)
**On sync:** Events are stored on relays as **NIP-78 application-specific data** (kind 30078). This is the Nostr standard for apps that use relays as a personal database. Unlike regular kind 1 notes, relays are designed to store and index kind 30078 events long-term.
1. Scan folder for new events from other local browsers
2. Check Nostr relay cache for events from remote devices
3. Merge both, deduplicate by filename
4. Apply unseen events to IndexedDB
5. Bridge: folder events get published to Nostr, Nostr events get written to folder
**Real-time push:** When a Nostr subscription receives a new event, `sync()` is triggered immediately — no waiting for the polling interval. Each sync event becomes one NIP-78 event:
- **`d` tag** = `foldersync:{roomKey}:{filename}` — makes each event addressable
- **`t` tag** = `foldersync-{roomKey}` — indexed by relays for room-wide queries
- **content** = AES-256-GCM encrypted JSON (see Encryption below)
### Encryption
All data is **end-to-end encrypted** using **AES-256-GCM** via the browser's native Web Crypto API.
```
Room key string
PBKDF2 (100k iterations, SHA-256)
AES-256-GCM key
encrypt(JSON payload) → base64(iv + ciphertext)
stored on Nostr relay (relay sees only opaque bytes)
```
- The room key IS the shared secret — only peers who know it can decrypt
- Each message gets a random 96-bit IV (no IV reuse)
- PBKDF2 key derivation makes brute-force expensive
- Relay operators cannot read your data
- Zero extra dependencies — uses only `crypto.subtle`
### Write flow
1. Update IndexedDB (fast, immediate)
2. Emit `change` event (UI updates instantly)
3. Write event file to folder (background, fire-and-forget)
4. Encrypt + publish to Nostr relay (background, fire-and-forget)
### Sync / join flow
1. Query relay: `{ kinds: [30078], '#t': ['foldersync-{room}'] }` → full history
2. Decrypt each event with the room key
3. Subscribe for real-time updates
4. Scan folder for new local events
5. Merge both, deduplicate by filename
6. Apply unseen events to IndexedDB
7. Bridge: folder events get published to Nostr, Nostr events get written to folder
### Large data (images)
Events larger than 48KB are automatically **chunked** into multiple NIP-78 events:
- Each chunk gets its own `d` tag: `foldersync:{room}:{filename}:chunk:{n}`
- Chunks are individually encrypted
- Reassembled on the receiving side before decryption
### Real-time push
When a Nostr subscription receives a new event, `sync()` is triggered immediately — no waiting for the polling interval.
## Quick start ## Quick start
@ -50,18 +95,13 @@ import { FolderSyncDB } from './nostr/src/index.ts';
const db = await FolderSyncDB.open({ const db = await FolderSyncDB.open({
autoSyncIntervalMs: 5000, autoSyncIntervalMs: 5000,
relays: [ // optional, defaults to popular public relays
'wss://relay.damus.io',
'wss://nos.lol',
'wss://relay.nostr.band',
],
}); });
// Local folder sync (same as other variants) // Local folder sync (optional)
await db.selectFolder(); await db.selectFolder();
// Cross-device sync via Nostr // Cross-device sync via Nostr (encrypted)
await db.joinRoom('my-shared-room-key'); await db.joinRoom('my-secret-room-key');
// Use normally // Use normally
await db.kv.set('theme', 'dark'); await db.kv.set('theme', 'dark');
@ -78,7 +118,7 @@ On top of the standard FolderSyncDB API, this variant adds:
| Method | Description | | Method | Description |
|--------|-------------| |--------|-------------|
| `joinRoom(roomKey)` | Join a Nostr sync room. All clients with the same key sync together. | | `joinRoom(roomKey)` | Join an encrypted Nostr sync room. All clients with the same key sync together. |
| `leaveRoom()` | Disconnect from the current room. | | `leaveRoom()` | Disconnect from the current room. |
| `isConnected()` | Whether a room is currently joined and relay is connected. | | `isConnected()` | Whether a room is currently joined and relay is connected. |
| `currentRoom` | The current room key, or `null`. | | `currentRoom` | The current room key, or `null`. |
@ -97,13 +137,14 @@ On top of the standard FolderSyncDB API, this variant adds:
| `nostr:connected` | `{ roomKey }` | Joined a Nostr room | | `nostr:connected` | `{ roomKey }` | Joined a Nostr room |
| `nostr:disconnected` | -- | Left a Nostr room | | `nostr:disconnected` | -- | Left a Nostr room |
## Room key ## Security model
The room key is simply a shared string that identifies your sync group. It's used as a Nostr tag — all clients subscribed to the same tag receive each other's events. - **Room key = encryption key** — treat it like a password
- **AES-256-GCM** — authenticated encryption, tamper-proof
- Anyone who knows the room key can join and sync - **PBKDF2 key derivation** — 100k iterations slows brute-force
- Events are not encrypted (v1) — use random room keys for privacy through obscurity - **Per-message random IV** — no ciphertext patterns
- Each client generates its own Nostr keypair (stored in IndexedDB) - **Relay sees nothing** — tags contain the room name (which you can obscure), content is encrypted
- Each client generates its own Nostr keypair (stored in IndexedDB) for signing
## Works from `file://` ## Works from `file://`

87
nostr/src/crypto.ts Normal file
View File

@ -0,0 +1,87 @@
/**
* AES-256-GCM encryption using the Web Crypto API.
* Derives a symmetric key from the room key (shared secret).
* Zero dependencies uses only browser-native crypto.
*/
const ALGO = 'AES-GCM';
const KEY_LENGTH = 256;
const IV_BYTES = 12; // 96-bit IV for GCM
const SALT = 'foldersync-v1'; // fixed salt for key derivation
/** Derive an AES-256-GCM key from a room key string. */
export async function deriveKey(roomKey: string): Promise<CryptoKey> {
const enc = new TextEncoder();
const keyMaterial = await crypto.subtle.importKey(
'raw',
enc.encode(roomKey),
'PBKDF2',
false,
['deriveKey'],
);
return crypto.subtle.deriveKey(
{
name: 'PBKDF2',
salt: enc.encode(SALT),
iterations: 100_000,
hash: 'SHA-256',
},
keyMaterial,
{ name: ALGO, length: KEY_LENGTH },
false,
['encrypt', 'decrypt'],
);
}
/**
* Encrypt a string base64(iv + ciphertext).
* Fast: uses native AES-256-GCM via SubtleCrypto.
*/
export async function encrypt(key: CryptoKey, plaintext: string): Promise<string> {
const iv = crypto.getRandomValues(new Uint8Array(IV_BYTES));
const data = new TextEncoder().encode(plaintext);
const ciphertext = await crypto.subtle.encrypt(
{ name: ALGO, iv },
key,
data,
);
// Prepend IV to ciphertext, then base64
const combined = new Uint8Array(IV_BYTES + ciphertext.byteLength);
combined.set(iv, 0);
combined.set(new Uint8Array(ciphertext), IV_BYTES);
return uint8ToBase64(combined);
}
/**
* Decrypt base64(iv + ciphertext) string.
*/
export async function decrypt(key: CryptoKey, encoded: string): Promise<string> {
const combined = base64ToUint8(encoded);
const iv = combined.slice(0, IV_BYTES);
const ciphertext = combined.slice(IV_BYTES);
const plainBuffer = await crypto.subtle.decrypt(
{ name: ALGO, iv },
key,
ciphertext,
);
return new TextDecoder().decode(plainBuffer);
}
// ── Base64 helpers (browser-native, no btoa unicode issues) ──
function uint8ToBase64(bytes: Uint8Array): string {
let binary = '';
for (let i = 0; i < bytes.length; i++) {
binary += String.fromCharCode(bytes[i]);
}
return btoa(binary);
}
function base64ToUint8(b64: string): Uint8Array {
const binary = atob(b64);
const bytes = new Uint8Array(binary.length);
for (let i = 0; i < binary.length; i++) {
bytes[i] = binary.charCodeAt(i);
}
return bytes;
}

View File

@ -1,6 +1,6 @@
import type { import type {
OpenOptions, StoreOptions, CollectionApi, KVApi, OpenOptions, StoreOptions, CollectionApi, KVApi,
SyncDBEventName, SyncDBEventHandler, SyncEvent, SyncDBEventName, SyncDBEventHandler,
} from './types.js'; } from './types.js';
import { generateClientId } from './utils.js'; import { generateClientId } from './utils.js';
import { Emitter } from './emitter.js'; import { Emitter } from './emitter.js';
@ -31,11 +31,8 @@ export class FolderSyncDB {
private autoSyncTimer: ReturnType<typeof setInterval> | null = null; private autoSyncTimer: ReturnType<typeof setInterval> | null = null;
private collections = new Map<string, Collection<any>>(); private collections = new Map<string, Collection<any>>();
/** Public KV API — available immediately after open(). */
kv!: KVApi; kv!: KVApi;
// ── Construction (use static open()) ───────────────────────
private constructor() {} private constructor() {}
static async open(options?: OpenOptions): Promise<FolderSyncDB> { static async open(options?: OpenOptions): Promise<FolderSyncDB> {
@ -50,21 +47,20 @@ export class FolderSyncDB {
this.idb = await IDBStore.open(dbName); this.idb = await IDBStore.open(dbName);
this.folderStore = new FolderStore(); this.folderStore = new FolderStore();
// ── Nostr transport setup ──────────────────────────────── // ── Nostr transport ───────────────────────────────────────
const relays = opts.relays ?? DEFAULT_RELAYS; const relays = opts.relays ?? DEFAULT_RELAYS;
this.nostrTransport = new NostrTransport(relays); this.nostrTransport = new NostrTransport(relays);
// Load or generate Nostr keypair // Load or generate keypair (persisted in IDB)
let skHex = await this.idb.getMeta<string>(META_NOSTR_SK); let skHex = await this.idb.getMeta<string>(META_NOSTR_SK);
if (skHex) { if (skHex) {
const sk = hexToBytes(skHex); this.nostrTransport.setKeypair(hexToBytes(skHex));
this.nostrTransport.setKeypair(sk);
} else { } else {
const sk = this.nostrTransport.generateKeypair(); const sk = this.nostrTransport.generateKeypair();
await this.idb.setMeta(META_NOSTR_SK, bytesToHex(sk)); await this.idb.setMeta(META_NOSTR_SK, bytesToHex(sk));
} }
// ── Client ID ──────────────────────────────────────────── // ── Client ID ────────────────────────────────────────────
let clientId = opts.clientId; let clientId = opts.clientId;
if (!clientId) { if (!clientId) {
clientId = await this.idb.getMeta<string>(META_CLIENT_ID); clientId = await this.idb.getMeta<string>(META_CLIENT_ID);
@ -76,10 +72,7 @@ export class FolderSyncDB {
await this.idb.setMeta(META_CLIENT_ID, clientId); await this.idb.setMeta(META_CLIENT_ID, clientId);
} }
// ── Nostr transport identity ───────────────────────────── // ── Sync engine ───────────────────────────────────────────
this.nostrTransport.setClientId(clientId);
// ── Sync engine (dual transport) ─────────────────────────
this.syncEngine = new SyncEngine( this.syncEngine = new SyncEngine(
this.idb, this.folderStore, this.nostrTransport, this.idb, this.folderStore, this.nostrTransport,
this.emitter, clientId, opts.conflictResolver, this.emitter, clientId, opts.conflictResolver,
@ -87,58 +80,21 @@ export class FolderSyncDB {
this.kv = new KVStore(this.idb, this.syncEngine); this.kv = new KVStore(this.idb, this.syncEngine);
// ── Restore folder handle ──────────────────────────────── // ── Restore folder handle ────────────────────────────────
await this.tryRestoreHandle(); await this.tryRestoreHandle();
// ── Restore Nostr room ─────────────────────────────────── // ── Restore Nostr room ───────────────────────────────────
const savedRoom = opts.roomKey ?? await this.idb.getMeta<string>(META_NOSTR_ROOM); const savedRoom = opts.roomKey ?? await this.idb.getMeta<string>(META_NOSTR_ROOM);
if (savedRoom) { if (savedRoom) {
await this.joinRoom(savedRoom); await this.joinRoom(savedRoom);
} }
// ── Real-time push from Nostr triggers immediate sync ──── // ── Real-time push: new Nostr event → immediate sync ──────
this.nostrTransport.onNewEvent(() => { this.nostrTransport.onNewEvent(() => {
this.sync().catch(() => {}); this.sync().catch(() => {});
}); });
// ── Inventory-based set reconciliation ──────────────────── // ── Auto-sync ─────────────────────────────────────────────
this.nostrTransport.onInventory(
async (theirKnown: Set<string>, round: number, _fromClientId: string) => {
try {
const ourApplied = await this.idb.getAppliedSet();
// Find events WE have that THEY don't
const theyNeed: Array<{ filename: string; event: SyncEvent }> = [];
for (const filename of ourApplied) {
if (theirKnown.has(filename)) continue;
// Try Nostr cache first, then folder
let event = await this.nostrTransport.readEventFile(filename);
if (!event && this.folderStore.hasHandle) {
event = await this.folderStore.readEventFile(filename);
}
if (event) theyNeed.push({ filename, event });
}
// Send them what they're missing
if (theyNeed.length > 0) {
theyNeed.sort((a, b) => (a.event.ts ?? 0) - (b.event.ts ?? 0));
await this.nostrTransport.republishEvents(theyNeed);
console.log(`[nostr] sent ${theyNeed.length} events peer was missing`);
}
// If this was their initial inventory (round 0), send ours back
// so they can send us what WE'RE missing. Max 1 reply round.
if (round < 1) {
const ourFilenames = Array.from(ourApplied);
await this.nostrTransport.publishInventory(ourFilenames, round + 1);
}
} catch (err) {
console.warn('[nostr] inventory reconciliation failed:', err);
}
},
);
// ── Auto-sync ────────────────────────────────────────────
if (opts.autoSyncIntervalMs && opts.autoSyncIntervalMs > 0) { if (opts.autoSyncIntervalMs && opts.autoSyncIntervalMs > 0) {
this.autoSyncTimer = setInterval(() => { this.autoSyncTimer = setInterval(() => {
this.sync().catch(() => {}); this.sync().catch(() => {});
@ -146,7 +102,7 @@ export class FolderSyncDB {
} }
} }
// ── Folder management (same as indexeddb variant) ────────── // ── Folder management ──────────────────────────────────────
async selectFolder(): Promise<void> { async selectFolder(): Promise<void> {
const handle = await this.folderStore.selectFolder(); const handle = await this.folderStore.selectFolder();
@ -164,17 +120,22 @@ export class FolderSyncDB {
return granted; return granted;
} }
// ── Nostr room management (NEW) ──────────────────────────── // ── Nostr room management ──────────────────────────────────
/**
* Join a Nostr room. The relay stores all events as NIP-78 kind 30078,
* so on join we get the full history no inventory exchange needed.
*/
async joinRoom(roomKey: string): Promise<void> { async joinRoom(roomKey: string): Promise<void> {
await this.nostrTransport.joinRoom(roomKey); await this.nostrTransport.joinRoom(roomKey);
await this.idb.setMeta(META_NOSTR_ROOM, roomKey); await this.idb.setMeta(META_NOSTR_ROOM, roomKey);
this.emitter.emit('nostr:connected', { roomKey }); this.emitter.emit('nostr:connected', { roomKey });
await this.sync(); await this.sync();
// Announce our inventory so peers can send us what we're missing // Re-publish any local events the relay doesn't have yet
const appliedSet = await this.idb.getAppliedSet(); this.republishMissing().catch((err) =>
await this.nostrTransport.publishInventory(Array.from(appliedSet), 0); console.warn('[nostr] republish failed:', err),
);
} }
leaveRoom(): void { leaveRoom(): void {
@ -190,12 +151,58 @@ export class FolderSyncDB {
return this.nostrTransport.currentRoom; return this.nostrTransport.currentRoom;
} }
// ── Sync (folder + Nostr) ────────────────────────────────── // ── Sync ───────────────────────────────────────────────────
async sync(): Promise<void> { async sync(): Promise<void> {
return this.syncEngine.sync(); return this.syncEngine.sync();
} }
/**
* Re-publish all local events that the relay doesn't have.
* Reads raw event files from the folder and publishes to Nostr.
* Call this manually or it runs automatically on joinRoom.
*/
async republishMissing(): Promise<void> {
if (!this.nostrTransport.isConnected) return;
if (!this.folderStore.hasHandle) {
console.log('[nostr] no folder connected — cannot re-publish missing events');
return;
}
const hasAccess = await this.folderStore.hasPermission();
if (!hasAccess) return;
const ourApplied = await this.idb.getAppliedSet();
const relayKnown = new Set(await this.nostrTransport.scanEventFilenames());
// Find events we have locally but relay doesn't
const missing: string[] = [];
for (const filename of ourApplied) {
if (!relayKnown.has(filename)) missing.push(filename);
}
if (missing.length === 0) {
console.log('[nostr] relay is up to date with local events');
return;
}
console.log(`[nostr] re-publishing ${missing.length} events relay is missing...`);
let published = 0;
for (const filename of missing) {
try {
const event = await this.folderStore.readEventFile(filename);
if (event) {
await this.nostrTransport.writeEvent(filename, event);
published++;
}
} catch {
// skip unreadable events
}
}
console.log(`[nostr] re-published ${published}/${missing.length} events`);
}
// ── Collections ──────────────────────────────────────────── // ── Collections ────────────────────────────────────────────
collection<T extends { id: string }>(options: StoreOptions<T>): CollectionApi<T> { collection<T extends { id: string }>(options: StoreOptions<T>): CollectionApi<T> {
@ -230,12 +237,9 @@ export class FolderSyncDB {
private async tryRestoreHandle(): Promise<void> { private async tryRestoreHandle(): Promise<void> {
try { try {
const handle = await this.idb.getMeta<FileSystemDirectoryHandle>(META_DIR_HANDLE); const handle = await this.idb.getMeta<FileSystemDirectoryHandle>(META_DIR_HANDLE);
if (!handle) return; if (!handle || typeof handle.queryPermission !== 'function') return;
if (typeof handle.queryPermission !== 'function') return;
this.folderStore.setHandle(handle); this.folderStore.setHandle(handle);
} catch { } catch {}
// Handle was not restorable
}
} }
} }

View File

@ -1,26 +1,29 @@
import type { SyncEvent } from './types.js'; import type { SyncEvent } from './types.js';
/** /**
* Nostr relay transport layer with inventory-based set reconciliation. * Nostr relay transport using NIP-78 (kind 30078, application-specific data).
* *
* When a peer joins a room it publishes an "inventory" listing every * Each sync event is stored as an addressable/replaceable Nostr event:
* event filename it already knows. Other peers diff against their own * kind: 30078
* set and re-publish anything the newcomer is missing, then send their * d-tag: foldersync:{roomKey}:{filename} (unique per event)
* own inventory so the newcomer can do the same. Two rounds max. * t-tag: foldersync-{roomKey} (indexable room filter)
* *
* Works from file:// origins via WebSocket. * Relays index and persist kind 30078 events by design.
* On join we query the full room history no inventory protocol needed.
*
* Large events (images) are chunked into multiple 30078 events
* and reassembled on the receiving side.
*/ */
const NOSTR_EVENT_KIND = 1; const NIP78_KIND = 30078;
const TOPIC_PREFIX = 'foldersync-'; const TOPIC_PREFIX = 'foldersync-';
const D_TAG_PREFIX = 'foldersync:';
const MAX_CONTENT_BYTES = 48_000; // safe limit per Nostr event
import { generateSecretKey, getPublicKey, finalizeEvent } from 'nostr-tools/pure'; import { generateSecretKey, getPublicKey, finalizeEvent } from 'nostr-tools/pure';
import { SimplePool, type SubCloser } from 'nostr-tools/pool'; import { SimplePool, type SubCloser } from 'nostr-tools/pool';
import type { Filter } from 'nostr-tools/filter'; import type { Filter } from 'nostr-tools/filter';
import { deriveKey, encrypt, decrypt } from './crypto.js';
/** Internal message envelope. */
type NostrMessage =
| { _type: 'event'; filename: string; payload: SyncEvent }
| { _type: 'inventory'; clientId: string; known: string[]; round: number };
export class NostrTransport { export class NostrTransport {
private relays: string[]; private relays: string[];
@ -28,27 +31,21 @@ export class NostrTransport {
private roomKey: string | null = null; private roomKey: string | null = null;
private topicTag: string | null = null; private topicTag: string | null = null;
private secretKey: Uint8Array | null = null; private secretKey: Uint8Array | null = null;
private clientId: string = ''; private pubkey: string = '';
private cryptoKey: CryptoKey | null = null;
/** Cached events keyed by filename. */ /** Cached events keyed by filename. */
private eventCache = new Map<string, SyncEvent>(); private eventCache = new Map<string, SyncEvent>();
/** Buffered chunks waiting for assembly: filename → (index → data). */
private chunkBuffer = new Map<string, Map<number, { data: string; total: number }>>();
/** Active relay subscription. */ /** Active relay subscription. */
private sub: SubCloser | null = null; private sub: SubCloser | null = null;
/** Callback: new sync event arrived. */ /** Callback: new sync event arrived. */
private _onNewEvent?: () => void; private _onNewEvent?: () => void;
/**
* Callback: peer inventory received.
* Handler should call respondToInventory() with events the peer is missing.
*/
private _onInventory?: (
theirKnown: Set<string>,
round: number,
fromClientId: string,
) => void;
constructor(relays: string[]) { constructor(relays: string[]) {
this.relays = relays; this.relays = relays;
this.pool = new SimplePool(); this.pool = new SimplePool();
@ -58,7 +55,7 @@ export class NostrTransport {
setKeypair(sk: Uint8Array): void { setKeypair(sk: Uint8Array): void {
this.secretKey = sk; this.secretKey = sk;
getPublicKey(sk); this.pubkey = getPublicKey(sk);
} }
generateKeypair(): Uint8Array { generateKeypair(): Uint8Array {
@ -67,10 +64,6 @@ export class NostrTransport {
return sk; return sk;
} }
setClientId(id: string): void {
this.clientId = id;
}
// ── Room management ───────────────────────────────────────── // ── Room management ─────────────────────────────────────────
async joinRoom(roomKey: string): Promise<void> { async joinRoom(roomKey: string): Promise<void> {
@ -78,17 +71,18 @@ export class NostrTransport {
this.leaveRoom(); this.leaveRoom();
this.roomKey = roomKey; this.roomKey = roomKey;
this.topicTag = TOPIC_PREFIX + roomKey; this.topicTag = TOPIC_PREFIX + roomKey;
this.cryptoKey = await deriveKey(roomKey);
// Fetch any events the relay still has // Fetch ALL history from relays (NIP-78 events are persistent)
const fetchFilter: Filter = { const fetchFilter: Filter = {
kinds: [NOSTR_EVENT_KIND], kinds: [NIP78_KIND],
'#t': [this.topicTag], '#t': [this.topicTag],
limit: 5000, limit: 10000,
}; };
try { try {
const events = await this.pool.querySync(this.relays, fetchFilter); const events = await this.pool.querySync(this.relays, fetchFilter);
console.log(`[nostr] fetched ${events.length} existing events from relay`); console.log(`[nostr] fetched ${events.length} NIP-78 events from relays`);
for (const ev of events) { for (const ev of events) {
this.handleIncoming(ev); this.handleIncoming(ev);
} }
@ -96,9 +90,9 @@ export class NostrTransport {
console.warn('[nostr] relay fetch failed:', err); console.warn('[nostr] relay fetch failed:', err);
} }
// Subscribe for real-time events // Subscribe for real-time updates
const subFilter: Filter = { const subFilter: Filter = {
kinds: [NOSTR_EVENT_KIND], kinds: [NIP78_KIND],
'#t': [this.topicTag], '#t': [this.topicTag],
since: Math.floor(Date.now() / 1000) - 1, since: Math.floor(Date.now() / 1000) - 1,
}; };
@ -110,23 +104,6 @@ export class NostrTransport {
console.log(`[nostr] joined room "${roomKey}" on ${this.relays.length} relays`); console.log(`[nostr] joined room "${roomKey}" on ${this.relays.length} relays`);
} }
/**
* Publish our inventory to the room.
* Called by FolderSyncDB after joining (it has the full applied set from IDB).
*/
async publishInventory(allKnownFilenames: string[], round = 0): Promise<void> {
const msg: NostrMessage = {
_type: 'inventory',
clientId: this.clientId,
known: allKnownFilenames,
round,
};
await this.publishToRelay(JSON.stringify(msg));
console.log(
`[nostr] sent inventory (round ${round}, ${allKnownFilenames.length} events known)`,
);
}
leaveRoom(): void { leaveRoom(): void {
if (this.sub) { if (this.sub) {
this.sub.close(); this.sub.close();
@ -134,7 +111,9 @@ export class NostrTransport {
} }
this.roomKey = null; this.roomKey = null;
this.topicTag = null; this.topicTag = null;
this.cryptoKey = null;
this.eventCache.clear(); this.eventCache.clear();
this.chunkBuffer.clear();
} }
get isConnected(): boolean { get isConnected(): boolean {
@ -151,38 +130,26 @@ export class NostrTransport {
this._onNewEvent = cb; this._onNewEvent = cb;
} }
onInventory(
cb: (theirKnown: Set<string>, round: number, fromClientId: string) => void,
): void {
this._onInventory = cb;
}
// ── Event I/O ─────────────────────────────────────────────── // ── Event I/O ───────────────────────────────────────────────
async writeEvent(filename: string, event: SyncEvent): Promise<void> { async writeEvent(filename: string, event: SyncEvent): Promise<void> {
if (!this.topicTag || !this.secretKey) return; if (!this.topicTag || !this.secretKey || !this.roomKey) return;
this.eventCache.set(filename, event); this.eventCache.set(filename, event);
const msg: NostrMessage = { const plaintext = JSON.stringify(event);
_type: 'event', const plaintextBytes = new TextEncoder().encode(plaintext).length;
filename,
payload: event,
};
await this.publishToRelay(JSON.stringify(msg)); if (plaintextBytes <= MAX_CONTENT_BYTES) {
console.log(`[nostr] published ${filename}`); // Small enough for one event — encrypt whole thing
} const content = this.cryptoKey
? await encrypt(this.cryptoKey, plaintext)
async republishEvents( : plaintext;
events: Array<{ filename: string; event: SyncEvent }>, await this.publishNip78(filename, content);
): Promise<void> { console.log(`[nostr] published ${filename} (${plaintextBytes} bytes, encrypted)`);
if (!this.topicTag || !this.secretKey || events.length === 0) return; } else {
// Too large — chunk the PLAINTEXT, each chunk encrypted individually
console.log(`[nostr] re-publishing ${events.length} events for peer catch-up`); await this.publishChunked(filename, plaintext);
for (const { filename, event } of events) {
const msg: NostrMessage = { _type: 'event', filename, payload: event };
await this.publishToRelay(JSON.stringify(msg));
} }
} }
@ -201,16 +168,23 @@ export class NostrTransport {
this.pool.close(this.relays); this.pool.close(this.relays);
} }
// ── Internals ─────────────────────────────────────────────── // ── NIP-78 publishing ──────────────────────────────────────
private async publishToRelay(content: string): Promise<void> { private async publishNip78(filename: string, content: string, extraTags?: string[][]): Promise<void> {
if (!this.topicTag || !this.secretKey) return; if (!this.topicTag || !this.secretKey || !this.roomKey) return;
const dTag = `${D_TAG_PREFIX}${this.roomKey}:${filename}`;
const tags: string[][] = [
['d', dTag],
['t', this.topicTag],
];
if (extraTags) tags.push(...extraTags);
const nostrEvent = finalizeEvent( const nostrEvent = finalizeEvent(
{ {
kind: NOSTR_EVENT_KIND, kind: NIP78_KIND,
created_at: Math.floor(Date.now() / 1000), created_at: Math.floor(Date.now() / 1000),
tags: [['t', this.topicTag]], tags,
content, content,
}, },
this.secretKey, this.secretKey,
@ -219,40 +193,136 @@ export class NostrTransport {
try { try {
await Promise.any(this.pool.publish(this.relays, nostrEvent as any)); await Promise.any(this.pool.publish(this.relays, nostrEvent as any));
} catch { } catch {
/* relay publish failed */ console.warn(`[nostr] publish failed for ${filename}`);
} }
} }
// ── Chunking for large events ──────────────────────────────
private async publishChunked(filename: string, content: string): Promise<void> {
const chunks: string[] = [];
for (let i = 0; i < content.length; i += MAX_CONTENT_BYTES) {
chunks.push(content.slice(i, i + MAX_CONTENT_BYTES));
}
console.log(`[nostr] chunking ${filename} into ${chunks.length} parts (${content.length} bytes)`);
for (let i = 0; i < chunks.length; i++) {
const chunkPlain = JSON.stringify({
_chunk: true,
filename,
index: i,
total: chunks.length,
data: chunks[i],
});
const chunkPayload = this.cryptoKey
? await encrypt(this.cryptoKey, chunkPlain)
: chunkPlain;
// Each chunk gets its own unique d-tag so relay stores all of them
const chunkFilename = `${filename}:chunk:${i}`;
await this.publishNip78(chunkFilename, chunkPayload, [
['chunk_of', filename],
['chunk_index', String(i)],
['chunk_total', String(chunks.length)],
]);
}
console.log(`[nostr] published ${chunks.length} chunks for ${filename}`);
}
// ── Incoming event handling ────────────────────────────────
private handleIncoming(nostrEvent: any): void { private handleIncoming(nostrEvent: any): void {
// Skip our own events
if (nostrEvent.pubkey === this.pubkey) return;
// Decrypt + process async
this.decryptAndProcess(nostrEvent).catch(() => {});
}
private async decryptAndProcess(nostrEvent: any): Promise<void> {
try { try {
const hasTag = nostrEvent.tags?.some( let content = nostrEvent.content;
(t: string[]) => t[0] === 't' && t[1]?.startsWith(TOPIC_PREFIX), if (!content) return;
);
if (!hasTag) return;
const msg: NostrMessage = JSON.parse(nostrEvent.content); // Decrypt if we have a key and content looks like base64 (not JSON)
if (this.cryptoKey && !content.startsWith('{') && !content.startsWith('[')) {
// ── Inventory message ─────────────────────────────────── try {
if (msg._type === 'inventory') { content = await decrypt(this.cryptoKey, content);
if (msg.clientId === this.clientId) return; // ignore our own
console.log(
`[nostr] peer ${msg.clientId.slice(0, 8)} sent inventory ` +
`(round ${msg.round}, ${msg.known.length} events)`,
);
this._onInventory?.(new Set(msg.known), msg.round, msg.clientId);
return;
}
// ── Sync event ──────────────────────────────────────────
if (msg._type === 'event' && msg.filename && msg.payload) {
if (this.eventCache.has(msg.filename)) return;
this.eventCache.set(msg.filename, msg.payload);
console.log(`[nostr] cached event ${msg.filename}`);
this._onNewEvent?.();
return;
}
} catch { } catch {
/* not a valid message */ // Decryption failed — wrong room key or unencrypted legacy event
return;
}
}
let parsed: any;
try {
parsed = JSON.parse(content);
} catch {
return;
}
if (parsed?._chunk) {
this.handleChunk(parsed);
return;
}
// Regular sync event — extract filename from d-tag
const dTag = nostrEvent.tags?.find((t: string[]) => t[0] === 'd')?.[1];
if (!dTag || !dTag.startsWith(D_TAG_PREFIX)) return;
const afterPrefix = dTag.slice(D_TAG_PREFIX.length);
const colonIdx = afterPrefix.indexOf(':');
if (colonIdx < 0) return;
const filename = afterPrefix.slice(colonIdx + 1);
if (filename.includes(':chunk:')) return;
if (this.eventCache.has(filename)) return;
const event = parsed as SyncEvent;
if (!event.type || !event.store || !event.key) return;
this.eventCache.set(filename, event);
console.log(`[nostr] cached event ${filename}`);
this._onNewEvent?.();
} catch {
/* malformed event */
}
}
private handleChunk(chunk: { filename: string; index: number; total: number; data: string }): void {
const { filename, index, total, data } = chunk;
// Already have the assembled event
if (this.eventCache.has(filename)) return;
if (!this.chunkBuffer.has(filename)) {
this.chunkBuffer.set(filename, new Map());
}
const chunks = this.chunkBuffer.get(filename)!;
chunks.set(index, { data, total });
console.log(`[nostr] received chunk ${index + 1}/${total} for ${filename}`);
// Check if all chunks arrived
if (chunks.size === total) {
// Assemble in order
const parts: string[] = [];
for (let i = 0; i < total; i++) {
const c = chunks.get(i);
if (!c) return; // missing chunk
parts.push(c.data);
}
try {
const event = JSON.parse(parts.join('')) as SyncEvent;
this.eventCache.set(filename, event);
this.chunkBuffer.delete(filename);
console.log(`[nostr] assembled ${total} chunks → ${filename}`);
this._onNewEvent?.();
} catch {
console.warn(`[nostr] chunk assembly failed for ${filename}`);
}
} }
} }
} }

View File

@ -3,7 +3,7 @@ import type { IDBStore } from './idb-store.js';
import type { FolderStore } from './folder-store.js'; import type { FolderStore } from './folder-store.js';
import type { NostrTransport } from './nostr-transport.js'; import type { NostrTransport } from './nostr-transport.js';
import type { Emitter } from './emitter.js'; import type { Emitter } from './emitter.js';
import { canonicalJson, sha256Hex, eventFilename } from './utils.js'; import { eventHash, eventFilename } from './utils.js';
/** /**
* Dual-transport sync engine. * Dual-transport sync engine.
@ -76,36 +76,44 @@ export class SyncEngine {
// ── Core persist: IDB + folder + Nostr ───────────────────── // ── Core persist: IDB + folder + Nostr ─────────────────────
private async persistEvent(event: SyncEvent): Promise<void> { private async persistEvent(event: SyncEvent): Promise<void> {
const canonical = canonicalJson(event); const hash = await eventHash(event);
const hash = await sha256Hex(canonical);
const filename = eventFilename(event.ts, hash); const filename = eventFilename(event.ts, hash);
// 1. Write to IDB (fast path) // 1. Write to IDB (fast, synchronous-feeling)
await this.applyEvent(event); await this.applyEvent(event);
await this.idb.markEventApplied(filename); await this.idb.markEventApplied(filename);
// 2. Write to folder (if connected) // 2. Emit change IMMEDIATELY so UI updates without waiting
if (this.folder.hasHandle && (await this.folder.hasPermission())) {
try {
await this.folder.writeEvent(filename, event);
} catch (err) {
this.emitter.emit('folder:lost-permission', err);
}
}
// 3. Publish to Nostr (if joined)
if (this.nostr.isConnected) {
try {
await this.nostr.writeEvent(filename, event);
} catch {
// Nostr publish failed — event is still in IDB + folder
}
}
this.emitter.emit('change', { this.emitter.emit('change', {
type: event.type, store: event.store, type: event.type, store: event.store,
key: event.key, id: event.id, data: event.data, key: event.key, id: event.id, data: event.data,
}); });
// 3. Folder + Nostr writes happen in background (best-effort)
this.persistToTransports(filename, event);
}
/**
* Write event to folder and Nostr without blocking the caller.
* IDB is the source of truth; these are durable copies.
*/
private persistToTransports(filename: string, event: SyncEvent): void {
// Folder (fire and forget)
if (this.folder.hasHandle) {
this.folder.hasPermission().then((ok) => {
if (!ok) return;
return this.folder.writeEvent(filename, event);
}).catch((err) => {
this.emitter.emit('folder:lost-permission', err);
});
}
// Nostr (fire and forget)
if (this.nostr.isConnected) {
this.nostr.writeEvent(filename, event).catch(() => {
// Nostr publish failed — event is still safe in IDB
});
}
} }
// ── Sync: import from BOTH folder and Nostr ──────────────── // ── Sync: import from BOTH folder and Nostr ────────────────

View File

@ -26,6 +26,29 @@ export async function sha256Hex(data: string): Promise<string> {
.join(''); .join('');
} }
/**
* Fast event fingerprint for filename hashing.
* Hashes only the metadata + data shape (not the full payload).
* This avoids SHA-256 over megabytes of base64 image data.
*/
export async function eventHash(event: {
type: string; store: string; key: string;
id?: string; ts: number; clientId: string; rev?: number;
data?: unknown;
}): Promise<string> {
const dataStr = event.data !== undefined ? JSON.stringify(event.data) : '';
const fingerprint = [
event.type, event.store, event.key, event.id ?? '',
String(event.ts), event.clientId, String(event.rev ?? 0),
String(dataStr.length),
// Include first+last 128 chars of data for uniqueness without full hash
dataStr.slice(0, 128),
dataStr.length > 128 ? dataStr.slice(-128) : '',
].join('\0');
return sha256Hex(fingerprint);
}
// ── Event filenames ────────────────────────────────────────── // ── Event filenames ──────────────────────────────────────────
const HASH_PREFIX_LEN = 12; const HASH_PREFIX_LEN = 12;

View File

@ -52,6 +52,29 @@ document.addEventListener('DOMContentLoaded', async () => {
if (e.key === 'Enter') joinBtn.click(); if (e.key === 'Enter') joinBtn.click();
}); });
// ── Re-sync button ───────────────────────────────────────
const resyncBtn = document.querySelector('#resync-btn') as HTMLButtonElement;
resyncBtn.addEventListener('click', async () => {
if (!db.isConnected()) {
nostrStatus.textContent = 'Join a room first';
nostrStatus.className = 'status err';
return;
}
resyncBtn.disabled = true;
resyncBtn.textContent = 'Syncing...';
try {
await db.republishMissing();
await db.sync();
resyncBtn.textContent = 'Done!';
setTimeout(() => { resyncBtn.textContent = 'Re-sync'; resyncBtn.disabled = false; }, 2000);
} catch (e: unknown) {
resyncBtn.textContent = 'Re-sync';
resyncBtn.disabled = false;
nostrStatus.textContent = 'Re-sync error: ' + (e as Error).message;
nostrStatus.className = 'status err';
}
});
// Restore saved room // Restore saved room
if (db.isConnected()) { if (db.isConnected()) {
updateNostrUI(true, db.currentRoom ?? undefined); updateNostrUI(true, db.currentRoom ?? undefined);

View File

@ -18,6 +18,7 @@
<input type="text" id="room-input" placeholder="room key" autocomplete="off" <input type="text" id="room-input" placeholder="room key" autocomplete="off"
style="padding:6px 10px;border:1px solid #555;border-radius:6px;background:#1e1e2e;color:#cdd6f4;font-size:0.85rem;width:180px"> style="padding:6px 10px;border:1px solid #555;border-radius:6px;background:#1e1e2e;color:#cdd6f4;font-size:0.85rem;width:180px">
<button id="join-room" class="folder-btn">Join room</button> <button id="join-room" class="folder-btn">Join room</button>
<button id="resync-btn" class="folder-btn" style="margin-left:4px;font-size:0.7rem;padding:4px 8px" title="Re-publish all local events to relay">Re-sync</button>
<span id="nostr-status" class="status"></span> <span id="nostr-status" class="status"></span>
</div> </div>
</header> </header>

View File

@ -122,6 +122,17 @@ export async function initPasteApp(db: PasteDB, variantLabel: string) {
updateFolderUI(false); updateFolderUI(false);
} }
// ── Saving spinner ─────────────────────────────────────────
const spinner = document.createElement('div');
spinner.className = 'saving-spinner hidden';
spinner.innerHTML = '<span class="spinner-dot"></span> Syncing...';
document.body.appendChild(spinner);
function showSaving(active: boolean) {
spinner.classList.toggle('hidden', !active);
}
// ── Status ─────────────────────────────────────────────── // ── Status ───────────────────────────────────────────────
function setStatus(msg: string, type?: string) { function setStatus(msg: string, type?: string) {
@ -163,7 +174,7 @@ export async function initPasteApp(db: PasteDB, variantLabel: string) {
description: string, description: string,
fileName: string, fileName: string,
) { ) {
setStatus('Saving...'); setStatus('Reading file...');
try { try {
const base64 = await fileToBase64(file); const base64 = await fileToBase64(file);
const paste: Paste = { const paste: Paste = {
@ -175,10 +186,16 @@ export async function initPasteApp(db: PasteDB, variantLabel: string) {
fileName, fileName,
fileData: base64, fileData: base64,
}; };
// Show spinner — IDB write is fast, but hashing + transport are slow
showSaving(true);
await pastes.put(paste); await pastes.put(paste);
// UI refreshes immediately via 'change' event (IDB write triggers it)
// Transports (folder, Nostr) continue in background
showSaving(false);
setStatus('Saved', 'ok'); setStatus('Saved', 'ok');
await refreshItems();
} catch (e: unknown) { } catch (e: unknown) {
showSaving(false);
setStatus('Error: ' + (e as Error).message, 'err'); setStatus('Error: ' + (e as Error).message, 'err');
} }
} }

View File

@ -188,3 +188,40 @@ header .sub {
padding: 3rem 1rem; padding: 3rem 1rem;
font-size: 0.9rem; font-size: 0.9rem;
} }
/* ── Saving spinner ────────────────────────────────────── */
.saving-spinner {
position: fixed;
top: 12px;
right: 16px;
display: flex;
align-items: center;
gap: 6px;
background: var(--bg-card);
border: 1px solid var(--accent-dim);
color: var(--accent);
font-size: 0.75rem;
font-family: var(--mono);
padding: 5px 12px;
border-radius: var(--radius);
z-index: 1000;
animation: fade-in 0.15s ease;
}
.saving-spinner.hidden {
display: none;
}
.spinner-dot {
width: 8px;
height: 8px;
border-radius: 50%;
border: 2px solid var(--accent-dim);
border-top-color: var(--accent);
animation: spin 0.6s linear infinite;
}
@keyframes spin {
to { transform: rotate(360deg); }
}
@keyframes fade-in {
from { opacity: 0; transform: translateY(-4px); }
to { opacity: 1; transform: translateY(0); }
}

View File

@ -53,17 +53,15 @@ export class SyncEngine {
await this.applyEvent(event); await this.applyEvent(event);
await this.store.markEventApplied(filename); await this.store.markEventApplied(filename);
if (this.folder.hasHandle && (await this.folder.hasPermission())) {
try {
await this.folder.writeEvent(filename, event);
} catch (err) {
this.emitter.emit('folder:lost-permission', err);
}
}
this.emitter.emit('change', { this.emitter.emit('change', {
type: event.type, store: event.store, key: event.key, id: event.id, data: event.data, type: event.type, store: event.store, key: event.key, id: event.id, data: event.data,
}); });
if (this.folder.hasHandle && (await this.folder.hasPermission())) {
this.folder.writeEvent(filename, event).catch((err) => {
this.emitter.emit('folder:lost-permission', err);
});
}
} }
async sync(): Promise<void> { async sync(): Promise<void> {