Initial commit: local-first browser sync library experiment

Four variants of the same sync library (IndexedDB, NeDB, SQLite WASM, sql.js)
plus a paste-bin demo app for testing multi-browser sync via shared folders.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jason Tudisco 2026-03-17 22:04:08 -06:00
commit 6ebe02ad56
78 changed files with 7704 additions and 0 deletions

29
.gitignore vendored Normal file
View File

@ -0,0 +1,29 @@
# Dependencies
node_modules/
# Build output
dist/
# Test data
Data/
# OS files
.DS_Store
Thumbs.db
Desktop.ini
# Editor / IDE
.vscode/
.idea/
*.swp
*.swo
*~
# Bun
bun.lockb
# npm
package-lock.json
# TypeScript incremental
*.tsbuildinfo

189
README.md Normal file
View File

@ -0,0 +1,189 @@
# IndexSyncFile
A local-first key-value and document store for the browser that syncs across multiple browser windows via a user-selected folder on disk.
**No server. No backend. No cloud.** Just browsers and a shared folder.
## The experiment
Can multiple browser tabs (or separate browsers entirely) stay in sync using nothing but a shared folder on the local file system?
This repo explores that question by building the same sync library four times, each with a different in-browser storage engine. The sync layer and public API are identical. Only the local cache differs.
The goal is to compare trade-offs: bundle size, startup speed, query power, persistence behavior, and `file://` compatibility.
## How it works
```
Browser A Shared Folder Browser B
--------- ------------- ---------
kv.set("x", 1)
|
+---> writes to local
| cache (fast)
|
+---> writes event file ---> /events/1234_abc.json
<--- sync() reads new events
|
+---> applies to
local cache
```
1. Each browser keeps a fast local cache for reads/writes (IndexedDB, SQLite WASM, NeDB, or sql.js)
2. Every mutation is written as an immutable JSON file in `/events/` inside a user-selected folder
3. When any browser calls `sync()`, it scans `/events/`, finds files it hasn't seen, and merges them into its local cache
4. Conflicts are resolved by last-write-wins (configurable)
### Why events?
The event log is what makes multi-browser sync possible. Each mutation is a separate file, so concurrent writes from different browsers never overwrite each other. Both sides' changes are captured and merged on the next `sync()`.
Without events, if two browsers both wrote a single database file, the last save would silently destroy the other browser's changes.
## Four variants
Each variant uses a different local cache engine. The API and sync behavior are identical.
| Variant | Local cache | Persistence | `file://` works? | Dependencies |
|---------|------------|-------------|-------------------|--------------|
| [`indexeddb/`](./indexeddb/) | IndexedDB | Browser-managed | Yes | Zero |
| [`nedb/`](./nedb/) | NeDB in-memory | NDJSON snapshots to `/data/` | Yes | `@seald-io/nedb` |
| [`sqlite/`](./sqlite/) | SQLite WASM | OPFS or in-memory | No (needs HTTP + headers) | `@sqlite.org/sqlite-wasm` |
| [`sql-js/`](./sql-js/) | sql.js (asm.js) | SQLite binary to `/data/store.sqlite` | Yes | `sql.js` |
### Which one should I use?
- **Just want it to work** — use `indexeddb/`. Zero deps, works from `file://`, instant startup.
- **Need MongoDB-style queries** (`$gt`, `$in`, `$regex`) — use `nedb/`.
- **Need real SQL + proper numeric range queries** — use `sql-js/`. Works from `file://`, no special headers.
- **Need OPFS-backed persistence + SQL** — use `sqlite/`. Requires HTTP server and may need COOP/COEP headers.
## Quick start (same for all variants)
```ts
import { FolderSyncDB } from 'index-sync-file'; // pick your variant
const db = await FolderSyncDB.open({
autoSyncIntervalMs: 5000,
});
// User picks a folder (one-time, persisted across sessions)
await db.selectFolder();
// Key-value
await db.kv.set('theme', 'dark');
const theme = await db.kv.get('theme'); // 'dark'
// Document collections with indexes
const todos = db.collection({
name: 'todos',
indexes: [{ name: 'byStatus', fields: ['status'] }],
});
await todos.put({ id: '1', title: 'Ship it', status: 'active' });
await todos.put({ id: '2', title: 'Test it', status: 'done' });
const active = await todos.findByIndex('byStatus', 'active');
// [{ id: '1', title: 'Ship it', status: 'active' }]
// Listen for changes (including from other browsers)
db.on('change', (e) => console.log('changed:', e));
db.on('conflict', (e) => console.log('conflict resolved:', e));
```
## Folder layout on disk
```
your-selected-folder/
events/
1710000000000_a1b2c3d4e5f6.json <-- immutable event files
1710000001000_f6e5d4c3b2a1.json
...
data/ <-- nedb and sql-js variants
kv.db <-- nedb: NDJSON snapshot
docs.db
store.sqlite <-- sql-js: full SQLite database file
```
## Demo app: Paste
The [`paste/`](./paste/) folder contains a working paste-bin demo app built with all four variants. Each compiles to a single self-contained HTML file that you can open directly from disk (except the SQLite WASM variant, which needs HTTP).
See the [paste README](./paste/README.md) for details.
## Browser requirements
Requires the **File System Access API** (`showDirectoryPicker`). Supported in:
- Chrome 86+
- Edge 86+
- Brave
- Opera 72+
Firefox and Safari do not support `showDirectoryPicker`. The library throws a clear error if the API is unavailable.
## Building
Each variant is a self-contained npm package:
```bash
cd indexeddb && npm install && npm run build
cd nedb && npm install && npm run build
cd sqlite && npm install && npm run build
cd sql-js && npm install && npm run build
```
The paste demo uses Bun:
```bash
cd paste && bun run build.ts
```
## API reference
See the README in each variant's folder for variant-specific details.
### `FolderSyncDB`
| Method | Description |
|--------|-------------|
| `FolderSyncDB.open(options?)` | Create and initialize the database |
| `selectFolder()` | Prompt user to pick a sync folder |
| `hasFolderAccess()` | Check if folder permission is granted |
| `requestFolderAccess()` | Re-request permission (needs user gesture) |
| `sync()` | Manually trigger a sync with the folder |
| `close()` | Stop auto-sync, flush writes, release resources |
| `kv` | Key-value store (see below) |
| `collection(options)` | Create/get a document collection (see below) |
| `on(event, handler)` | Listen for events, returns unsubscribe function |
### `kv` (Key-Value Store)
| Method | Description |
|--------|-------------|
| `get(key)` | Get value by key |
| `set(key, value)` | Set a key-value pair |
| `delete(key)` | Delete a key |
| `has(key)` | Check if key exists |
| `keys()` | Get all keys |
| `entries()` | Get all [key, value] pairs |
### `collection(options)` (Document Collections)
| Method | Description |
|--------|-------------|
| `get(id)` | Get document by ID |
| `put(doc)` | Insert or update a document (must have `id` field) |
| `delete(id)` | Delete a document (tombstone) |
| `all()` | Get all documents |
| `findByIndex(indexName, value)` | Exact-match query on an index |
| `queryByIndex(indexName, range)` | Range query (`{ gte, lte, gt, lt }`) |
### Events
| Event | Payload | When |
|-------|---------|------|
| `sync:start` | -- | Sync begins |
| `sync:end` | `{ imported }` or `{ error }` | Sync completes |
| `change` | `{ type, store, key, id, data }` | Any data mutation (local or synced) |
| `conflict` | `{ filename, event }` | A conflict was detected and resolved |
| `folder:lost-permission` | error? | Folder access was lost |

79
indexeddb/README.md Normal file
View File

@ -0,0 +1,79 @@
# IndexSyncFile — IndexedDB Variant
Local-first key-value and document store using **IndexedDB** as the local cache, syncing with a user-selected folder via the File System Access API.
## Why IndexedDB
- **Zero dependencies** — uses the browser's built-in IndexedDB API directly
- **Persistent by default** — IndexedDB data survives page reloads and browser restarts without any extra work
- **No WASM loading** — instant startup, no async module initialization
- **Battle-tested** — IndexedDB is the most widely supported browser storage API
## Architecture
```
Write: app ---> IndexedDB (immediate) ---> /events/timestamp_hash.json
Sync: /events/*.json ---> sort by timestamp ---> skip applied ---> apply to IndexedDB
```
IndexedDB serves as both the fast query layer and the local persistence layer. The folder's event log is the sync mechanism that keeps multiple browsers converged.
### IDB schema (5 object stores)
| Store | Key | Purpose |
|-------|-----|---------|
| `kv` | `key` | Key-value records |
| `docs` | `[store, id]` | Document collection records |
| `idx` | `[store, indexName, id]` | Secondary index entries |
| `applied` | `filename` | Tracks which event files have been processed |
| `meta` | `key` | Client ID, directory handle persistence |
The `FileSystemDirectoryHandle` is stored directly in the `meta` IDB store via structured clone, so the user doesn't need to re-select the folder on every page load.
## Install and build
```bash
npm install
npm run build
```
## Usage
```ts
import { FolderSyncDB } from './dist/index.js';
const db = await FolderSyncDB.open({
dbName: 'MyApp',
autoSyncIntervalMs: 5000,
});
await db.selectFolder();
// KV
await db.kv.set('user', { name: 'Alice' });
// Collections
const notes = db.collection({
name: 'notes',
indexes: [{ name: 'byTag', fields: ['tag'] }],
});
await notes.put({ id: 'n1', title: 'Hello', tag: 'work' });
// Exact match
const workNotes = await notes.findByIndex('byTag', 'work');
// Range query (IDB compound key ranges)
const recent = await notes.queryByIndex('byDate', { gte: '2024-01-01' });
```
## Index implementation
Secondary indexes are maintained in a dedicated `idx` IDB object store using compound keys `[store, indexName, id]`. An IDB index on `[store, indexName, value]` enables efficient exact-match and range queries via `IDBKeyRange`.
Index entries are updated atomically with document writes inside a single IDB transaction. On first use of a collection, indexes are rebuilt from existing documents.
## Variant-specific notes
- Directory handle is persisted in IDB (no separate storage needed)
- IDB transactions are used for atomic document + index updates
- No external dependencies

26
indexeddb/package.json Normal file
View File

@ -0,0 +1,26 @@
{
"name": "index-sync-file",
"version": "1.0.0",
"description": "Local-first key-value and document store that syncs with a user-selected folder via File System Access API",
"type": "module",
"main": "dist/index.js",
"module": "dist/index.js",
"types": "dist/index.d.ts",
"exports": {
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.js"
}
},
"files": ["dist"],
"scripts": {
"build": "tsc",
"dev": "tsc --watch",
"clean": "rm -rf dist"
},
"keywords": ["local-first", "indexeddb", "file-system-access-api", "key-value", "document-store", "sync"],
"license": "MIT",
"devDependencies": {
"typescript": "^5.7.0"
}
}

137
indexeddb/src/collection.ts Normal file
View File

@ -0,0 +1,137 @@
import type { CollectionApi, IndexDefinition, StoreOptions } from './types.js';
import type { IDBStore } from './idb-store.js';
import type { SyncEngine } from './sync-engine.js';
export class Collection<T extends { id: string }> implements CollectionApi<T> {
private readonly storeName: string;
private readonly indexes: IndexDefinition<T>[];
private indexesBuilt = false;
constructor(
options: StoreOptions<T>,
private idb: IDBStore,
private sync: SyncEngine,
) {
this.storeName = options.name;
this.indexes = options.indexes ?? [];
this.sync.registerIndexes(this.storeName, this.indexes as IndexDefinition[]);
}
/** Lazily rebuild indexes on first access. */
private async ensureIndexes(): Promise<void> {
if (this.indexesBuilt || this.indexes.length === 0) return;
await this.idb.rebuildIndexes(
this.storeName,
this.indexes as IndexDefinition[],
);
this.indexesBuilt = true;
}
async get(id: string): Promise<T | undefined> {
const rec = await this.idb.getDoc(this.storeName, id);
if (!rec) return undefined;
return rec.data as T;
}
async put(doc: T): Promise<void> {
await this.ensureIndexes();
const existing = await this.idb.getRawDoc(this.storeName, doc.id);
const rev = (existing?.rev ?? 0) + 1;
const ts = Date.now();
// SyncEngine.persistEvent handles the IDB write + folder write
await this.sync.writeDoc(this.storeName, doc.id, doc, ts, rev);
}
async delete(id: string): Promise<void> {
await this.ensureIndexes();
const existing = await this.idb.getRawDoc(this.storeName, id);
if (!existing || existing.deleted) return;
const rev = existing.rev + 1;
const ts = Date.now();
await this.sync.deleteDocEvent(this.storeName, id, ts, rev);
}
async all(): Promise<T[]> {
const docs = await this.idb.getAllDocs(this.storeName);
return docs.map((d) => d.data as T);
}
async findByIndex(indexName: string, value: IDBValidKey): Promise<T[]> {
await this.ensureIndexes();
this.assertIndex(indexName);
const ids = await this.idb.findByIndex(this.storeName, indexName, value);
return this.fetchByIds(ids);
}
async queryByIndex(
indexName: string,
range: {
gt?: IDBValidKey;
gte?: IDBValidKey;
lt?: IDBValidKey;
lte?: IDBValidKey;
},
): Promise<T[]> {
await this.ensureIndexes();
this.assertIndex(indexName);
const lower = range.gte ?? range.gt;
const upper = range.lte ?? range.lt;
const lowerOpen = range.gte === undefined && range.gt !== undefined;
const upperOpen = range.lte === undefined && range.lt !== undefined;
let idbRange: IDBKeyRange;
if (lower !== undefined && upper !== undefined) {
idbRange = IDBKeyRange.bound(
[this.storeName, indexName, lower],
[this.storeName, indexName, upper],
lowerOpen,
upperOpen,
);
} else if (lower !== undefined) {
idbRange = IDBKeyRange.lowerBound(
[this.storeName, indexName, lower],
lowerOpen,
);
} else if (upper !== undefined) {
idbRange = IDBKeyRange.upperBound(
[this.storeName, indexName, upper],
upperOpen,
);
} else {
// No bounds: return all entries for this index
idbRange = IDBKeyRange.bound(
[this.storeName, indexName, -Infinity],
[this.storeName, indexName, []],
);
}
const ids = await this.idb.queryByIndex(
this.storeName,
indexName,
idbRange,
);
return this.fetchByIds(ids);
}
// ── Helpers ────────────────────────────────────────────────
private assertIndex(name: string): void {
if (!this.indexes.some((i) => i.name === name)) {
throw new Error(
`Index "${name}" is not defined on collection "${this.storeName}". ` +
`Defined indexes: ${this.indexes.map((i) => i.name).join(', ') || '(none)'}`,
);
}
}
private async fetchByIds(ids: string[]): Promise<T[]> {
const results: T[] = [];
for (const id of ids) {
const doc = await this.idb.getDoc(this.storeName, id);
if (doc) results.push(doc.data as T);
}
return results;
}
}

33
indexeddb/src/emitter.ts Normal file
View File

@ -0,0 +1,33 @@
import type { SyncDBEventName, SyncDBEventHandler } from './types.js';
export class Emitter {
private listeners = new Map<SyncDBEventName, Set<SyncDBEventHandler>>();
on(event: SyncDBEventName, handler: SyncDBEventHandler): () => void {
let set = this.listeners.get(event);
if (!set) {
set = new Set();
this.listeners.set(event, set);
}
set.add(handler);
return () => {
set!.delete(handler);
};
}
emit(event: SyncDBEventName, ...args: unknown[]): void {
const set = this.listeners.get(event);
if (!set) return;
for (const fn of set) {
try {
fn(...args);
} catch {
// listener errors must not break the library
}
}
}
removeAll(): void {
this.listeners.clear();
}
}

View File

@ -0,0 +1,122 @@
import type { SyncEvent } from './types.js';
import { parseEventFilename } from './utils.js';
const EVENTS_DIR = 'events';
export class FolderStore {
private dirHandle: FileSystemDirectoryHandle | null = null;
// ── Handle management ──────────────────────────────────────
get hasHandle(): boolean {
return this.dirHandle !== null;
}
getHandle(): FileSystemDirectoryHandle | null {
return this.dirHandle;
}
setHandle(handle: FileSystemDirectoryHandle): void {
this.dirHandle = handle;
}
async selectFolder(): Promise<FileSystemDirectoryHandle> {
if (typeof window.showDirectoryPicker !== 'function') {
throw new Error(
'File System Access API is not supported in this browser. ' +
'Use a Chromium-based browser (Chrome, Edge, Brave, etc.).',
);
}
this.dirHandle = await window.showDirectoryPicker({
mode: 'readwrite',
});
// Ensure events directory exists
await this.dirHandle.getDirectoryHandle(EVENTS_DIR, { create: true });
return this.dirHandle;
}
// ── Permission checks ─────────────────────────────────────
async hasPermission(): Promise<boolean> {
if (!this.dirHandle) return false;
try {
const perm = await this.dirHandle.queryPermission({ mode: 'readwrite' });
return perm === 'granted';
} catch {
return false;
}
}
async requestPermission(): Promise<boolean> {
if (!this.dirHandle) return false;
try {
const perm = await this.dirHandle.requestPermission({
mode: 'readwrite',
});
return perm === 'granted';
} catch {
return false;
}
}
// ── Write event file ──────────────────────────────────────
async writeEvent(filename: string, event: SyncEvent): Promise<void> {
const eventsDir = await this.getEventsDir();
const fileHandle = await eventsDir.getFileHandle(filename, {
create: true,
});
const writable = await fileHandle.createWritable();
try {
await writable.write(JSON.stringify(event, null, 2));
} finally {
await writable.close();
}
}
// ── Scan events ────────────────────────────────────────────
async scanEventFilenames(): Promise<string[]> {
const eventsDir = await this.getEventsDir();
const names: string[] = [];
for await (const [name, handle] of eventsDir.entries()) {
if (handle.kind === 'file' && name.endsWith('.json')) {
if (parseEventFilename(name)) {
names.push(name);
}
}
}
// Sort by (timestamp, full filename) for deterministic ordering
names.sort((a, b) => {
const pa = parseEventFilename(a)!;
const pb = parseEventFilename(b)!;
if (pa.ts !== pb.ts) return pa.ts - pb.ts;
return a.localeCompare(b);
});
return names;
}
async readEventFile(filename: string): Promise<SyncEvent | null> {
const eventsDir = await this.getEventsDir();
try {
const fh = await eventsDir.getFileHandle(filename);
const file = await fh.getFile();
const text = await file.text();
return JSON.parse(text) as SyncEvent;
} catch {
return null;
}
}
// ── Internals ──────────────────────────────────────────────
private async getEventsDir(): Promise<FileSystemDirectoryHandle> {
if (!this.dirHandle) {
throw new Error('No folder selected. Call selectFolder() first.');
}
return this.dirHandle.getDirectoryHandle(EVENTS_DIR, { create: true });
}
}

View File

@ -0,0 +1,162 @@
import type {
OpenOptions,
StoreOptions,
CollectionApi,
KVApi,
SyncDBEventName,
SyncDBEventHandler,
} from './types.js';
import { generateClientId } from './utils.js';
import { Emitter } from './emitter.js';
import { IDBStore } from './idb-store.js';
import { FolderStore } from './folder-store.js';
import { SyncEngine } from './sync-engine.js';
import { KVStore } from './kv-store.js';
import { Collection } from './collection.js';
const META_CLIENT_ID = 'clientId';
const META_DIR_HANDLE = 'dirHandle';
export class FolderSyncDB {
private idb!: IDBStore;
private folderStore!: FolderStore;
private syncEngine!: SyncEngine;
private emitter!: Emitter;
private autoSyncTimer: ReturnType<typeof setInterval> | null = null;
private collections = new Map<string, Collection<any>>();
/** Public KV API — available immediately after open(). */
kv!: KVApi;
// ── Construction (use static open()) ───────────────────────
private constructor() {}
static async open(options?: OpenOptions): Promise<FolderSyncDB> {
const db = new FolderSyncDB();
await db.init(options ?? {});
return db;
}
private async init(opts: OpenOptions): Promise<void> {
const dbName = opts.dbName ?? 'FolderSyncDB';
this.emitter = new Emitter();
this.idb = await IDBStore.open(dbName);
this.folderStore = new FolderStore();
// Client ID: use provided, or load persisted, or generate new
let clientId = opts.clientId;
if (!clientId) {
clientId = await this.idb.getMeta<string>(META_CLIENT_ID);
if (!clientId) {
clientId = generateClientId();
await this.idb.setMeta(META_CLIENT_ID, clientId);
}
} else {
await this.idb.setMeta(META_CLIENT_ID, clientId);
}
this.syncEngine = new SyncEngine(
this.idb,
this.folderStore,
this.emitter,
clientId,
opts.conflictResolver,
);
this.kv = new KVStore(this.idb, this.syncEngine);
// Restore persisted directory handle
await this.tryRestoreHandle();
// Auto-sync
if (opts.autoSyncIntervalMs && opts.autoSyncIntervalMs > 0) {
this.autoSyncTimer = setInterval(() => {
this.sync().catch(() => {
/* swallow — errors are emitted as events */
});
}, opts.autoSyncIntervalMs);
}
}
// ── Folder management ──────────────────────────────────────
async selectFolder(): Promise<void> {
const handle = await this.folderStore.selectFolder();
// Persist handle for future sessions
await this.idb.setMeta(META_DIR_HANDLE, handle);
// Run initial sync to import existing events
await this.sync();
}
async hasFolderAccess(): Promise<boolean> {
return this.folderStore.hasPermission();
}
async requestFolderAccess(): Promise<boolean> {
const granted = await this.folderStore.requestPermission();
if (granted) {
await this.sync();
}
return granted;
}
// ── Sync ───────────────────────────────────────────────────
async sync(): Promise<void> {
return this.syncEngine.sync();
}
// ── Collections ────────────────────────────────────────────
collection<T extends { id: string }>(
options: StoreOptions<T>,
): CollectionApi<T> {
const cached = this.collections.get(options.name);
if (cached) return cached as Collection<T>;
const col = new Collection<T>(options, this.idb, this.syncEngine);
this.collections.set(options.name, col);
return col;
}
// ── Events ─────────────────────────────────────────────────
on(event: SyncDBEventName, handler: SyncDBEventHandler): () => void {
return this.emitter.on(event, handler);
}
// ── Lifecycle ──────────────────────────────────────────────
async close(): Promise<void> {
if (this.autoSyncTimer !== null) {
clearInterval(this.autoSyncTimer);
this.autoSyncTimer = null;
}
this.emitter.removeAll();
this.collections.clear();
this.idb.close();
}
// ── Internals ──────────────────────────────────────────────
private async tryRestoreHandle(): Promise<void> {
try {
const handle = await this.idb.getMeta<FileSystemDirectoryHandle>(
META_DIR_HANDLE,
);
if (!handle) return;
// Verify it's a real handle (IDB structured-clone survives reload)
if (typeof handle.queryPermission !== 'function') return;
this.folderStore.setHandle(handle);
// If permission is already granted (e.g. persistent permissions),
// the handle is ready. Otherwise the user must call
// requestFolderAccess() which requires a user gesture.
} catch {
// Handle was not restorable — user must re-select
}
}
}

88
indexeddb/src/fs-access.d.ts vendored Normal file
View File

@ -0,0 +1,88 @@
/**
* Type declarations for the File System Access API.
* These APIs are available in Chromium-based browsers only.
* @see https://developer.mozilla.org/en-US/docs/Web/API/File_System_Access_API
*/
interface FileSystemHandlePermissionDescriptor {
mode?: 'read' | 'readwrite';
}
interface FileSystemDirectoryHandle {
readonly kind: 'directory';
readonly name: string;
getDirectoryHandle(
name: string,
options?: { create?: boolean },
): Promise<FileSystemDirectoryHandle>;
getFileHandle(
name: string,
options?: { create?: boolean },
): Promise<FileSystemFileHandle>;
entries(): AsyncIterableIterator<
[string, FileSystemDirectoryHandle | FileSystemFileHandle]
>;
values(): AsyncIterableIterator<
FileSystemDirectoryHandle | FileSystemFileHandle
>;
keys(): AsyncIterableIterator<string>;
queryPermission(
descriptor?: FileSystemHandlePermissionDescriptor,
): Promise<PermissionState>;
requestPermission(
descriptor?: FileSystemHandlePermissionDescriptor,
): Promise<PermissionState>;
}
interface FileSystemFileHandle {
readonly kind: 'file';
readonly name: string;
getFile(): Promise<File>;
createWritable(
options?: FileSystemCreateWritableOptions,
): Promise<FileSystemWritableFileStream>;
}
interface FileSystemCreateWritableOptions {
keepExistingData?: boolean;
}
interface FileSystemWritableFileStream extends WritableStream {
write(data: BufferSource | Blob | string | WriteParams): Promise<void>;
seek(position: number): Promise<void>;
truncate(size: number): Promise<void>;
close(): Promise<void>;
}
interface WriteParams {
type: 'write' | 'seek' | 'truncate';
data?: BufferSource | Blob | string;
position?: number;
size?: number;
}
interface ShowDirectoryPickerOptions {
id?: string;
mode?: 'read' | 'readwrite';
startIn?:
| FileSystemDirectoryHandle
| 'desktop'
| 'documents'
| 'downloads'
| 'music'
| 'pictures'
| 'videos';
}
interface Window {
showDirectoryPicker(
options?: ShowDirectoryPickerOptions,
): Promise<FileSystemDirectoryHandle>;
}

308
indexeddb/src/idb-store.ts Normal file
View File

@ -0,0 +1,308 @@
import type {
KVRecord,
DocRecord,
IdxEntry,
AppliedRecord,
MetaRecord,
IndexDefinition,
} from './types.js';
import { extractIndexValue } from './utils.js';
// ── Store names ──────────────────────────────────────────────
const S_KV = 'kv';
const S_DOCS = 'docs';
const S_IDX = 'idx';
const S_APPLIED = 'applied';
const S_META = 'meta';
const DB_VERSION = 1;
// ── Promise wrappers for raw IDB ─────────────────────────────
function reqP<T>(req: IDBRequest<T>): Promise<T> {
return new Promise((resolve, reject) => {
req.onsuccess = () => resolve(req.result);
req.onerror = () => reject(req.error);
});
}
function txDone(tx: IDBTransaction): Promise<void> {
return new Promise((resolve, reject) => {
tx.oncomplete = () => resolve();
tx.onerror = () => reject(tx.error);
tx.onabort = () => reject(tx.error ?? new Error('Transaction aborted'));
});
}
// ── IDBStore ─────────────────────────────────────────────────
export class IDBStore {
private constructor(private db: IDBDatabase) {}
// ── Open ───────────────────────────────────────────────────
static open(name: string): Promise<IDBStore> {
return new Promise((resolve, reject) => {
const req = indexedDB.open(name, DB_VERSION);
req.onupgradeneeded = () => {
const db = req.result;
// KV store
if (!db.objectStoreNames.contains(S_KV)) {
db.createObjectStore(S_KV, { keyPath: 'key' });
}
// Document store (compound primary key)
if (!db.objectStoreNames.contains(S_DOCS)) {
const docs = db.createObjectStore(S_DOCS, {
keyPath: ['store', 'id'],
});
docs.createIndex('byStore', 'store', { unique: false });
}
// Secondary index entries
if (!db.objectStoreNames.contains(S_IDX)) {
const idx = db.createObjectStore(S_IDX, {
keyPath: ['store', 'indexName', 'id'],
});
idx.createIndex('lookup', ['store', 'indexName', 'value'], {
unique: false,
});
idx.createIndex('byDoc', ['store', 'id'], { unique: false });
}
// Applied event dedup
if (!db.objectStoreNames.contains(S_APPLIED)) {
db.createObjectStore(S_APPLIED, { keyPath: 'filename' });
}
// Metadata
if (!db.objectStoreNames.contains(S_META)) {
db.createObjectStore(S_META, { keyPath: 'key' });
}
};
req.onsuccess = () => resolve(new IDBStore(req.result));
req.onerror = () => reject(req.error);
});
}
// ── KV operations ──────────────────────────────────────────
async getKV(key: string): Promise<KVRecord | undefined> {
const tx = this.db.transaction(S_KV, 'readonly');
return reqP(tx.objectStore(S_KV).get(key));
}
async putKV(record: KVRecord): Promise<void> {
const tx = this.db.transaction(S_KV, 'readwrite');
tx.objectStore(S_KV).put(record);
return txDone(tx);
}
async deleteKV(key: string): Promise<void> {
const tx = this.db.transaction(S_KV, 'readwrite');
tx.objectStore(S_KV).delete(key);
return txDone(tx);
}
async getAllKVKeys(): Promise<string[]> {
const tx = this.db.transaction(S_KV, 'readonly');
const records: KVRecord[] = await reqP(tx.objectStore(S_KV).getAll());
return records.map((r) => r.key);
}
async getAllKVEntries(): Promise<KVRecord[]> {
const tx = this.db.transaction(S_KV, 'readonly');
return reqP(tx.objectStore(S_KV).getAll());
}
// ── Doc operations ─────────────────────────────────────────
async getDoc(store: string, id: string): Promise<DocRecord | undefined> {
const tx = this.db.transaction(S_DOCS, 'readonly');
const rec: DocRecord | undefined = await reqP(
tx.objectStore(S_DOCS).get([store, id]),
);
if (rec && rec.deleted) return undefined;
return rec;
}
async getRawDoc(store: string, id: string): Promise<DocRecord | undefined> {
const tx = this.db.transaction(S_DOCS, 'readonly');
return reqP(tx.objectStore(S_DOCS).get([store, id]));
}
async putDoc(
record: DocRecord,
indexes: IndexDefinition[],
): Promise<void> {
const tx = this.db.transaction([S_DOCS, S_IDX], 'readwrite');
const docsOS = tx.objectStore(S_DOCS);
const idxOS = tx.objectStore(S_IDX);
// Remove old index entries for this doc (chain off getAll request)
const byDocIdx = idxOS.index('byDoc');
const getOld = byDocIdx.getAll([record.store, record.id]);
getOld.onsuccess = () => {
const oldEntries: IdxEntry[] = getOld.result;
for (const entry of oldEntries) {
idxOS.delete([entry.store, entry.indexName, entry.id]);
}
// Write document
docsOS.put(record);
// Create new index entries (only for non-deleted docs)
if (!record.deleted) {
for (const def of indexes) {
const val = extractIndexValue(record.data, def.fields as string[]);
if (val !== undefined) {
idxOS.put({
store: record.store,
indexName: def.name,
id: record.id,
value: val,
} satisfies IdxEntry);
}
}
}
};
return txDone(tx);
}
async deleteDoc(
store: string,
id: string,
ts: number,
rev: number,
indexes: IndexDefinition[],
): Promise<void> {
return this.putDoc(
{ store, id, data: null, ts, rev, deleted: true },
indexes,
);
}
async getAllDocs(store: string): Promise<DocRecord[]> {
const tx = this.db.transaction(S_DOCS, 'readonly');
const idx = tx.objectStore(S_DOCS).index('byStore');
const all: DocRecord[] = await reqP(idx.getAll(store));
return all.filter((r) => !r.deleted);
}
async findByIndex(
store: string,
indexName: string,
value: IDBValidKey,
): Promise<string[]> {
const tx = this.db.transaction(S_IDX, 'readonly');
const idx = tx.objectStore(S_IDX).index('lookup');
const entries: IdxEntry[] = await reqP(
idx.getAll([store, indexName, value]),
);
return entries.map((e) => e.id);
}
async queryByIndex(
store: string,
indexName: string,
range: IDBKeyRange,
): Promise<string[]> {
const tx = this.db.transaction(S_IDX, 'readonly');
const idx = tx.objectStore(S_IDX).index('lookup');
const entries: IdxEntry[] = await reqP(idx.getAll(range));
// Filter to only this store+index (the range may be broader)
return entries
.filter((e) => e.store === store && e.indexName === indexName)
.map((e) => e.id);
}
async rebuildIndexes(
store: string,
indexes: IndexDefinition[],
): Promise<void> {
const docs = await this.getAllDocs(store);
const tx = this.db.transaction(S_IDX, 'readwrite');
const idxOS = tx.objectStore(S_IDX);
// Delete all existing index entries for this store
const byDocIdx = idxOS.index('byDoc');
for (const doc of docs) {
const req = byDocIdx.getAll([store, doc.id]);
req.onsuccess = () => {
for (const entry of req.result as IdxEntry[]) {
idxOS.delete([entry.store, entry.indexName, entry.id]);
}
};
}
// Re-create all entries
for (const doc of docs) {
for (const def of indexes) {
const val = extractIndexValue(doc.data, def.fields as string[]);
if (val !== undefined) {
idxOS.put({
store,
indexName: def.name,
id: doc.id,
value: val,
} satisfies IdxEntry);
}
}
}
return txDone(tx);
}
// ── Applied-event tracking ─────────────────────────────────
async isEventApplied(filename: string): Promise<boolean> {
const tx = this.db.transaction(S_APPLIED, 'readonly');
const rec = await reqP(tx.objectStore(S_APPLIED).get(filename));
return rec !== undefined;
}
async markEventApplied(filename: string): Promise<void> {
const tx = this.db.transaction(S_APPLIED, 'readwrite');
tx.objectStore(S_APPLIED).put({
filename,
appliedAt: Date.now(),
} satisfies AppliedRecord);
return txDone(tx);
}
async getAppliedSet(): Promise<Set<string>> {
const tx = this.db.transaction(S_APPLIED, 'readonly');
const all: AppliedRecord[] = await reqP(
tx.objectStore(S_APPLIED).getAll(),
);
return new Set(all.map((r) => r.filename));
}
// ── Meta ───────────────────────────────────────────────────
async getMeta<T = unknown>(key: string): Promise<T | undefined> {
const tx = this.db.transaction(S_META, 'readonly');
const rec: MetaRecord | undefined = await reqP(
tx.objectStore(S_META).get(key),
);
return rec?.value as T | undefined;
}
async setMeta(key: string, value: unknown): Promise<void> {
const tx = this.db.transaction(S_META, 'readwrite');
tx.objectStore(S_META).put({ key, value } satisfies MetaRecord);
return txDone(tx);
}
// ── Lifecycle ──────────────────────────────────────────────
close(): void {
this.db.close();
}
}

12
indexeddb/src/index.ts Normal file
View File

@ -0,0 +1,12 @@
export { FolderSyncDB } from './folder-sync-db.js';
export type {
OpenOptions,
StoreOptions,
IndexDefinition,
SyncEvent,
KVApi,
CollectionApi,
SyncDBEventName,
SyncDBEventHandler,
} from './types.js';

45
indexeddb/src/kv-store.ts Normal file
View File

@ -0,0 +1,45 @@
import type { KVApi } from './types.js';
import type { IDBStore } from './idb-store.js';
import type { SyncEngine } from './sync-engine.js';
export class KVStore implements KVApi {
constructor(
private idb: IDBStore,
private sync: SyncEngine,
) {}
async get<T = unknown>(key: string): Promise<T | undefined> {
const rec = await this.idb.getKV(key);
return rec?.value as T | undefined;
}
async set<T = unknown>(key: string, value: T): Promise<void> {
const existing = await this.idb.getKV(key);
const rev = (existing?.rev ?? 0) + 1;
const ts = Date.now();
// SyncEngine.persistEvent handles the IDB write + folder write
await this.sync.writeKV(key, value, ts, rev);
}
async delete(key: string): Promise<void> {
const existing = await this.idb.getKV(key);
if (!existing) return;
const rev = existing.rev + 1;
const ts = Date.now();
await this.sync.deleteKV(key, ts, rev);
}
async has(key: string): Promise<boolean> {
const rec = await this.idb.getKV(key);
return rec !== undefined;
}
async keys(): Promise<string[]> {
return this.idb.getAllKVKeys();
}
async entries<T = unknown>(): Promise<Array<[string, T]>> {
const records = await this.idb.getAllKVEntries();
return records.map((r) => [r.key, r.value as T]);
}
}

View File

@ -0,0 +1,282 @@
import type { SyncEvent, IndexDefinition, OpenOptions } from './types.js';
import type { IDBStore } from './idb-store.js';
import type { FolderStore } from './folder-store.js';
import type { Emitter } from './emitter.js';
import { canonicalJson, sha256Hex, eventFilename } from './utils.js';
/**
* Knows about every registered collection's index definitions so
* it can maintain indexes when replaying events during sync.
*/
export class SyncEngine {
private clientId: string;
private conflictResolver?: OpenOptions['conflictResolver'];
private collectionIndexes = new Map<string, IndexDefinition[]>();
private syncing = false;
constructor(
private idb: IDBStore,
private folder: FolderStore,
private emitter: Emitter,
clientId: string,
conflictResolver?: OpenOptions['conflictResolver'],
) {
this.clientId = clientId;
this.conflictResolver = conflictResolver;
}
// ── Collection index registration ──────────────────────────
registerIndexes(store: string, indexes: IndexDefinition[]): void {
this.collectionIndexes.set(store, indexes);
}
getIndexes(store: string): IndexDefinition[] {
return this.collectionIndexes.get(store) ?? [];
}
// ── Local writes (write to IDB + folder) ───────────────────
async writeKV(
key: string,
value: unknown,
ts: number,
rev: number,
): Promise<void> {
const event: SyncEvent = {
type: 'put',
store: 'kv',
key,
ts,
clientId: this.clientId,
data: value,
rev,
};
await this.persistEvent(event);
}
async deleteKV(key: string, ts: number, rev: number): Promise<void> {
const event: SyncEvent = {
type: 'delete',
store: 'kv',
key,
ts,
clientId: this.clientId,
rev,
};
await this.persistEvent(event);
}
async writeDoc(
store: string,
id: string,
data: unknown,
ts: number,
rev: number,
): Promise<void> {
const event: SyncEvent = {
type: 'put',
store,
key: id,
id,
ts,
clientId: this.clientId,
data,
rev,
};
await this.persistEvent(event);
}
async deleteDocEvent(
store: string,
id: string,
ts: number,
rev: number,
): Promise<void> {
const event: SyncEvent = {
type: 'delete',
store,
key: id,
id,
ts,
clientId: this.clientId,
rev,
};
await this.persistEvent(event);
}
// ── Core persist: IDB + folder ─────────────────────────────
private async persistEvent(event: SyncEvent): Promise<void> {
const canonical = canonicalJson(event);
const hash = await sha256Hex(canonical);
const filename = eventFilename(event.ts, hash);
// Write to IDB first (fast path)
await this.applyEvent(event);
await this.idb.markEventApplied(filename);
// Then persist to folder (if available)
if (this.folder.hasHandle && (await this.folder.hasPermission())) {
try {
await this.folder.writeEvent(filename, event);
} catch (err) {
this.emitter.emit('folder:lost-permission', err);
}
}
this.emitter.emit('change', {
type: event.type,
store: event.store,
key: event.key,
id: event.id,
data: event.data,
});
}
// ── Sync: import folder events into IDB ────────────────────
async sync(): Promise<void> {
if (this.syncing) return;
if (!this.folder.hasHandle) return;
const hasAccess = await this.folder.hasPermission();
if (!hasAccess) {
this.emitter.emit('folder:lost-permission');
return;
}
this.syncing = true;
this.emitter.emit('sync:start');
try {
const appliedSet = await this.idb.getAppliedSet();
const filenames = await this.folder.scanEventFilenames();
let importCount = 0;
for (const name of filenames) {
if (appliedSet.has(name)) continue;
const event = await this.folder.readEventFile(name);
if (!event) continue;
const hadConflict = await this.applyEventWithConflictCheck(event);
await this.idb.markEventApplied(name);
importCount++;
if (hadConflict) {
this.emitter.emit('conflict', {
filename: name,
event,
});
}
this.emitter.emit('change', {
type: event.type,
store: event.store,
key: event.key,
id: event.id,
data: event.data,
source: 'sync',
});
}
this.emitter.emit('sync:end', { imported: importCount });
} catch (err) {
this.emitter.emit('sync:end', { error: err });
} finally {
this.syncing = false;
}
}
// ── Apply a single event to IDB ────────────────────────────
private async applyEvent(event: SyncEvent): Promise<void> {
if (event.store === 'kv') {
if (event.type === 'put') {
await this.idb.putKV({
key: event.key,
value: event.data,
ts: event.ts,
rev: event.rev ?? 0,
});
} else {
await this.idb.deleteKV(event.key);
}
} else {
const indexes = this.getIndexes(event.store);
if (event.type === 'put') {
await this.idb.putDoc(
{
store: event.store,
id: event.id ?? event.key,
data: event.data,
ts: event.ts,
rev: event.rev ?? 0,
},
indexes,
);
} else {
await this.idb.deleteDoc(
event.store,
event.id ?? event.key,
event.ts,
event.rev ?? 0,
indexes,
);
}
}
}
/**
* Apply an incoming event, checking for conflicts using LWW.
* Returns true if a conflict was detected (even if resolved).
*/
private async applyEventWithConflictCheck(
event: SyncEvent,
): Promise<boolean> {
let hadConflict = false;
if (event.store === 'kv') {
const existing = await this.idb.getKV(event.key);
if (existing) {
if (event.ts > existing.ts) {
// incoming wins
hadConflict = true;
} else if (event.ts === existing.ts) {
// tie-break: use custom resolver or skip
hadConflict = true;
if (this.conflictResolver) {
const resolved = this.conflictResolver(existing.value, event.data);
event = { ...event, data: resolved };
}
// With equal ts, still apply (deterministic: both sides converge)
} else {
// existing is newer — skip
return true; // was a conflict but existing wins
}
}
} else {
const existing = await this.idb.getRawDoc(
event.store,
event.id ?? event.key,
);
if (existing && !existing.deleted) {
if (event.ts > existing.ts) {
hadConflict = true;
} else if (event.ts === existing.ts) {
hadConflict = true;
if (this.conflictResolver) {
const resolved = this.conflictResolver(existing.data, event.data);
event = { ...event, data: resolved };
}
} else {
return true;
}
}
}
await this.applyEvent(event);
return hadConflict;
}
}

104
indexeddb/src/types.ts Normal file
View File

@ -0,0 +1,104 @@
// ── Event types ──────────────────────────────────────────────
export type EventType = 'put' | 'delete';
export interface SyncEvent {
type: EventType;
store: string; // "kv" | collection name
key: string;
id?: string; // present for collection documents
ts: number; // millisecond Unix epoch
clientId: string;
data?: unknown; // absent on deletes
rev?: number;
}
// ── Index / collection options ───────────────────────────────
export interface IndexDefinition<T = unknown> {
name: string;
fields: (keyof T & string)[] | string[];
}
export interface StoreOptions<T = unknown> {
name: string;
indexes?: IndexDefinition<T>[];
}
export interface OpenOptions {
dbName?: string;
autoSyncIntervalMs?: number;
clientId?: string;
conflictResolver?: (current: unknown, incoming: unknown) => unknown;
}
// ── IDB record shapes ────────────────────────────────────────
export interface KVRecord {
key: string;
value: unknown;
ts: number;
rev: number;
}
export interface DocRecord {
store: string;
id: string;
data: unknown;
ts: number;
rev: number;
deleted?: boolean;
}
export interface IdxEntry {
store: string;
indexName: string;
id: string;
value: IDBValidKey;
}
export interface AppliedRecord {
filename: string;
appliedAt: number;
}
export interface MetaRecord {
key: string;
value: unknown;
}
// ── Event emitter ────────────────────────────────────────────
export type SyncDBEventName =
| 'sync:start'
| 'sync:end'
| 'change'
| 'conflict'
| 'folder:lost-permission';
export type SyncDBEventHandler = (...args: unknown[]) => void;
// ── KV API surface ───────────────────────────────────────────
export interface KVApi {
get<T = unknown>(key: string): Promise<T | undefined>;
set<T = unknown>(key: string, value: T): Promise<void>;
delete(key: string): Promise<void>;
has(key: string): Promise<boolean>;
keys(): Promise<string[]>;
entries<T = unknown>(): Promise<Array<[string, T]>>;
}
// ── Collection API surface ───────────────────────────────────
export interface CollectionApi<T extends { id: string }> {
get(id: string): Promise<T | undefined>;
put(doc: T): Promise<void>;
delete(id: string): Promise<void>;
all(): Promise<T[]>;
findByIndex(indexName: string, value: IDBValidKey): Promise<T[]>;
queryByIndex(
indexName: string,
range: { gt?: IDBValidKey; gte?: IDBValidKey; lt?: IDBValidKey; lte?: IDBValidKey },
): Promise<T[]>;
}

94
indexeddb/src/utils.ts Normal file
View File

@ -0,0 +1,94 @@
// ── Canonical JSON ───────────────────────────────────────────
function sortKeys(obj: unknown): unknown {
if (obj === null || obj === undefined || typeof obj !== 'object') return obj;
if (Array.isArray(obj)) return obj.map(sortKeys);
const sorted: Record<string, unknown> = {};
for (const key of Object.keys(obj as Record<string, unknown>).sort()) {
sorted[key] = sortKeys((obj as Record<string, unknown>)[key]);
}
return sorted;
}
export function canonicalJson(obj: unknown): string {
return JSON.stringify(sortKeys(obj));
}
// ── Hashing ──────────────────────────────────────────────────
export async function sha256Hex(data: string): Promise<string> {
const buf = await crypto.subtle.digest(
'SHA-256',
new TextEncoder().encode(data),
);
return Array.from(new Uint8Array(buf))
.map((b) => b.toString(16).padStart(2, '0'))
.join('');
}
// ── Event filenames ──────────────────────────────────────────
const HASH_PREFIX_LEN = 12;
export function eventFilename(ts: number, fullHash: string): string {
return `${ts}_${fullHash.slice(0, HASH_PREFIX_LEN)}.json`;
}
const EVENT_FILE_RE = /^(\d+)_([0-9a-f]+)\.json$/;
export function parseEventFilename(
name: string,
): { ts: number; hash: string } | null {
const m = EVENT_FILE_RE.exec(name);
if (!m) return null;
return { ts: Number(m[1]), hash: m[2] };
}
// ── Client ID ────────────────────────────────────────────────
export function generateClientId(): string {
return crypto.randomUUID();
}
// ── Index value extraction ───────────────────────────────────
export function extractIndexValue(
data: unknown,
fields: string[],
): IDBValidKey | undefined {
if (!data || typeof data !== 'object') return undefined;
const rec = data as Record<string, unknown>;
if (fields.length === 1) {
const v = getNestedValue(rec, fields[0]);
if (v === undefined || v === null) return undefined;
return toIDBKey(v);
}
const parts: IDBValidKey[] = [];
for (const f of fields) {
const v = getNestedValue(rec, f);
if (v === undefined || v === null) return undefined;
parts.push(toIDBKey(v));
}
return parts;
}
function getNestedValue(obj: Record<string, unknown>, path: string): unknown {
let cur: unknown = obj;
for (const seg of path.split('.')) {
if (cur === null || cur === undefined || typeof cur !== 'object') {
return undefined;
}
cur = (cur as Record<string, unknown>)[seg];
}
return cur;
}
function toIDBKey(v: unknown): IDBValidKey {
if (typeof v === 'string' || typeof v === 'number') return v;
if (v instanceof Date) return v;
if (Array.isArray(v)) return v.map(toIDBKey);
// Fallback: stringify
return String(v);
}

21
indexeddb/tsconfig.json Normal file
View File

@ -0,0 +1,21 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "ES2022",
"moduleResolution": "bundler",
"lib": ["ES2022", "DOM", "DOM.Iterable"],
"outDir": "dist",
"rootDir": "src",
"declaration": true,
"declarationMap": true,
"sourceMap": true,
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noFallthroughCasesInSwitch": true,
"forceConsistentCasingInFileNames": true,
"skipLibCheck": true
},
"include": ["src"],
"exclude": ["node_modules", "dist"]
}

115
nedb/README.md Normal file
View File

@ -0,0 +1,115 @@
# IndexSyncFile — NeDB Variant
Local-first key-value and document store using **NeDB** (`@seald-io/nedb`, the maintained fork) as the local cache, syncing with a user-selected folder via the File System Access API.
## Why NeDB
- **MongoDB-style queries**`$gt`, `$lt`, `$in`, `$nin`, `$regex`, `$exists`, `$elemMatch`, and more
- **Native indexing**`ensureIndex` handles index creation and maintenance automatically
- **Dot-notation queries** — query nested fields like `data.user.name` naturally
- **Promise-based API**`@seald-io/nedb` provides native `*Async` methods, no callback wrappers needed
## Architecture
```
Write: app ---> NeDB in-memory (immediate) ---> /events/timestamp_hash.json
---> /data/*.db (debounced NDJSON snapshot)
Sync: /events/*.json ---> sort by timestamp ---> skip applied ---> apply to NeDB
---> persist to /data/*.db
Startup: load /data/*.db (instant) ---> sync only NEW events from /events/
```
### Two persistence layers
| Layer | Purpose | Format |
|-------|---------|--------|
| `/events/` | **Sync mechanism** — immutable event files enable multi-browser sync without data loss | One JSON file per mutation |
| `/data/` | **Fast reload** — NDJSON snapshots so startup doesn't replay the entire event history | One NDJSON file per datastore |
The event log is essential for sync. The data files are a startup optimization — without them, every page load would replay all events from scratch.
### Data files (NDJSON)
```
your-folder/data/
kv.db # all key-value records
docs.db # all collection documents
applied.db # which event files have been processed
meta.db # client ID and other metadata
```
Each file is newline-delimited JSON (one record per line). Human-readable, diffable, and written with a 250ms debounce to avoid excessive disk writes during batch operations.
### NeDB datastores (4 in-memory)
| Datastore | `_id` | Purpose |
|-----------|-------|---------|
| `kvDB` | KV key | Key-value records |
| `docsDB` | `"store\0id"` | All collection documents (with `_store`, `_docId`, `_deleted`, `data` fields) |
| `appliedDB` | event filename | Tracks processed events |
| `metaDB` | meta key | Client ID, etc. |
Document data is stored in a `data` field to avoid field name conflicts. NeDB indexes are created on `data.<fieldName>` using dot notation.
## Install and build
```bash
npm install
npm run build
```
**Dependency:** `@seald-io/nedb` ^4.1.0 — actively maintained fork of NeDB with native async/await support. Zero npm vulnerabilities.
## Usage
```ts
import { FolderSyncDB } from './dist/index.js';
const db = await FolderSyncDB.open({
dbName: 'MyApp',
autoSyncIntervalMs: 5000,
});
await db.selectFolder();
// KV
await db.kv.set('user', { name: 'Alice', role: 'admin' });
// Collections with indexes
const logs = db.collection({
name: 'logs',
indexes: [
{ name: 'byLevel', fields: ['level'] },
{ name: 'byTimestamp', fields: ['ts'] },
],
});
await logs.put({ id: 'log1', level: 'error', ts: Date.now(), msg: 'disk full' });
await logs.put({ id: 'log2', level: 'info', ts: Date.now(), msg: 'started' });
// Exact match (uses NeDB index)
const errors = await logs.findByIndex('byLevel', 'error');
// Range query (uses NeDB $gte/$lte operators)
const recent = await logs.queryByIndex('byTimestamp', {
gte: Date.now() - 3600000,
});
// Multi-field indexes work too
const items = db.collection({
name: 'items',
indexes: [{ name: 'byTypeAndStatus', fields: ['type', 'status'] }],
});
const match = await items.findByIndex('byTypeAndStatus', ['widget', 'active']);
```
## Variant-specific notes
- NeDB runs in-memory only in the browser (no Node.js `fs` access)
- Persistence comes from two sources: the event log (for sync) and the NDJSON snapshots (for fast reload)
- Directory handle is stored in a tiny IDB sidecar (NeDB can't store DOM objects)
- Range queries are only supported on single-field indexes (NeDB limitation)
- Consumer's bundler must handle NeDB's CommonJS-to-ESM conversion and the `browser` field in NeDB's `package.json`
- `getAllData()` is used for synchronous zero-copy dumps (no async overhead for persistence writes)

29
nedb/package.json Normal file
View File

@ -0,0 +1,29 @@
{
"name": "index-sync-file-nedb",
"version": "1.0.0",
"description": "Local-first key-value and document store backed by NeDB, syncing with a user-selected folder via File System Access API",
"type": "module",
"main": "dist/index.js",
"module": "dist/index.js",
"types": "dist/index.d.ts",
"exports": {
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.js"
}
},
"files": ["dist"],
"scripts": {
"build": "tsc",
"dev": "tsc --watch",
"clean": "rm -rf dist"
},
"keywords": ["local-first", "nedb", "file-system-access-api", "key-value", "document-store", "sync"],
"license": "MIT",
"dependencies": {
"@seald-io/nedb": "^4.1.0"
},
"devDependencies": {
"typescript": "^5.7.0"
}
}

93
nedb/src/collection.ts Normal file
View File

@ -0,0 +1,93 @@
import type {
CollectionApi,
IndexDefinition,
IndexRange,
StoreOptions,
} from './types.js';
import type { NeDBStore } from './nedb-store.js';
import type { SyncEngine } from './sync-engine.js';
export class Collection<T extends { id: string }> implements CollectionApi<T> {
private readonly storeName: string;
private readonly indexes: IndexDefinition<T>[];
private indexesBuilt = false;
constructor(
options: StoreOptions<T>,
private store: NeDBStore,
private sync: SyncEngine,
) {
this.storeName = options.name;
this.indexes = options.indexes ?? [];
this.sync.registerIndexes(
this.storeName,
this.indexes as IndexDefinition[],
);
}
private async ensureIndexes(): Promise<void> {
if (this.indexesBuilt || this.indexes.length === 0) return;
await this.store.rebuildIndexes(
this.storeName,
this.indexes as IndexDefinition[],
);
this.indexesBuilt = true;
}
async get(id: string): Promise<T | undefined> {
const rec = await this.store.getDoc(this.storeName, id);
if (!rec) return undefined;
return rec.data as T;
}
async put(doc: T): Promise<void> {
await this.ensureIndexes();
const existing = await this.store.getRawDoc(this.storeName, doc.id);
const rev = (existing?.rev ?? 0) + 1;
const ts = Date.now();
await this.sync.writeDoc(this.storeName, doc.id, doc, ts, rev);
}
async delete(id: string): Promise<void> {
await this.ensureIndexes();
const existing = await this.store.getRawDoc(this.storeName, id);
if (!existing || existing.deleted) return;
const rev = existing.rev + 1;
const ts = Date.now();
await this.sync.deleteDocEvent(this.storeName, id, ts, rev);
}
async all(): Promise<T[]> {
const docs = await this.store.getAllDocs(this.storeName);
return docs.map((d) => d.data as T);
}
async findByIndex(indexName: string, value: unknown): Promise<T[]> {
await this.ensureIndexes();
const ids = await this.store.findByIndex(
this.storeName,
indexName,
value,
);
return this.fetchByIds(ids);
}
async queryByIndex(indexName: string, range: IndexRange): Promise<T[]> {
await this.ensureIndexes();
const ids = await this.store.queryByIndex(
this.storeName,
indexName,
range,
);
return this.fetchByIds(ids);
}
private async fetchByIds(ids: string[]): Promise<T[]> {
const results: T[] = [];
for (const id of ids) {
const doc = await this.store.getDoc(this.storeName, id);
if (doc) results.push(doc.data as T);
}
return results;
}
}

33
nedb/src/emitter.ts Normal file
View File

@ -0,0 +1,33 @@
import type { SyncDBEventName, SyncDBEventHandler } from './types.js';
export class Emitter {
private listeners = new Map<SyncDBEventName, Set<SyncDBEventHandler>>();
on(event: SyncDBEventName, handler: SyncDBEventHandler): () => void {
let set = this.listeners.get(event);
if (!set) {
set = new Set();
this.listeners.set(event, set);
}
set.add(handler);
return () => {
set!.delete(handler);
};
}
emit(event: SyncDBEventName, ...args: unknown[]): void {
const set = this.listeners.get(event);
if (!set) return;
for (const fn of set) {
try {
fn(...args);
} catch {
// listener errors must not break the library
}
}
}
removeAll(): void {
this.listeners.clear();
}
}

153
nedb/src/folder-store.ts Normal file
View File

@ -0,0 +1,153 @@
import type { SyncEvent } from './types.js';
import { parseEventFilename } from './utils.js';
const EVENTS_DIR = 'events';
export class FolderStore {
private dirHandle: FileSystemDirectoryHandle | null = null;
// ── Handle management ──────────────────────────────────────
get hasHandle(): boolean {
return this.dirHandle !== null;
}
getHandle(): FileSystemDirectoryHandle | null {
return this.dirHandle;
}
setHandle(handle: FileSystemDirectoryHandle): void {
this.dirHandle = handle;
}
async selectFolder(): Promise<FileSystemDirectoryHandle> {
if (typeof window.showDirectoryPicker !== 'function') {
throw new Error(
'File System Access API is not supported in this browser. ' +
'Use a Chromium-based browser (Chrome, Edge, Brave, etc.).',
);
}
this.dirHandle = await window.showDirectoryPicker({
mode: 'readwrite',
});
// Ensure events directory exists
await this.dirHandle.getDirectoryHandle(EVENTS_DIR, { create: true });
return this.dirHandle;
}
// ── Permission checks ─────────────────────────────────────
async hasPermission(): Promise<boolean> {
if (!this.dirHandle) return false;
try {
const perm = await this.dirHandle.queryPermission({ mode: 'readwrite' });
return perm === 'granted';
} catch {
return false;
}
}
async requestPermission(): Promise<boolean> {
if (!this.dirHandle) return false;
try {
const perm = await this.dirHandle.requestPermission({
mode: 'readwrite',
});
return perm === 'granted';
} catch {
return false;
}
}
// ── Write event file ──────────────────────────────────────
async writeEvent(filename: string, event: SyncEvent): Promise<void> {
const eventsDir = await this.getEventsDir();
const fileHandle = await eventsDir.getFileHandle(filename, {
create: true,
});
const writable = await fileHandle.createWritable();
try {
await writable.write(JSON.stringify(event, null, 2));
} finally {
await writable.close();
}
}
// ── Scan events ────────────────────────────────────────────
async scanEventFilenames(): Promise<string[]> {
const eventsDir = await this.getEventsDir();
const names: string[] = [];
for await (const [name, handle] of eventsDir.entries()) {
if (handle.kind === 'file' && name.endsWith('.json')) {
if (parseEventFilename(name)) {
names.push(name);
}
}
}
// Sort by (timestamp, full filename) for deterministic ordering
names.sort((a, b) => {
const pa = parseEventFilename(a)!;
const pb = parseEventFilename(b)!;
if (pa.ts !== pb.ts) return pa.ts - pb.ts;
return a.localeCompare(b);
});
return names;
}
async readEventFile(filename: string): Promise<SyncEvent | null> {
const eventsDir = await this.getEventsDir();
try {
const fh = await eventsDir.getFileHandle(filename);
const file = await fh.getFile();
const text = await file.text();
return JSON.parse(text) as SyncEvent;
} catch {
return null;
}
}
// ── Data files (NeDB NDJSON persistence) ────────────────────
async writeDataFile(filename: string, content: string): Promise<void> {
const dataDir = await this.getDataDir();
const fh = await dataDir.getFileHandle(filename, { create: true });
const writable = await fh.createWritable();
try {
await writable.write(content);
} finally {
await writable.close();
}
}
async readDataFile(filename: string): Promise<string | null> {
try {
const dataDir = await this.getDataDir();
const fh = await dataDir.getFileHandle(filename);
const file = await fh.getFile();
return await file.text();
} catch {
return null;
}
}
// ── Internals ──────────────────────────────────────────────
private async getEventsDir(): Promise<FileSystemDirectoryHandle> {
if (!this.dirHandle) {
throw new Error('No folder selected. Call selectFolder() first.');
}
return this.dirHandle.getDirectoryHandle(EVENTS_DIR, { create: true });
}
private async getDataDir(): Promise<FileSystemDirectoryHandle> {
if (!this.dirHandle) {
throw new Error('No folder selected. Call selectFolder() first.');
}
return this.dirHandle.getDirectoryHandle('data', { create: true });
}
}

236
nedb/src/folder-sync-db.ts Normal file
View File

@ -0,0 +1,236 @@
import type {
OpenOptions,
StoreOptions,
CollectionApi,
KVApi,
SyncDBEventName,
SyncDBEventHandler,
} from './types.js';
import { generateClientId } from './utils.js';
import { Emitter } from './emitter.js';
import { NeDBStore } from './nedb-store.js';
import { FolderStore } from './folder-store.js';
import { SyncEngine } from './sync-engine.js';
import { KVStore } from './kv-store.js';
import { Collection } from './collection.js';
import { storeHandle, loadHandle } from './handle-store.js';
const META_CLIENT_ID = 'clientId';
const HANDLE_KEY = 'dirHandle';
/** NDJSON data files written to /data/ in the user-selected folder. */
const DATA_FILES = ['kv.db', 'docs.db', 'applied.db', 'meta.db'] as const;
/** Debounce interval for disk persistence (ms). */
const PERSIST_DEBOUNCE_MS = 250;
export class FolderSyncDB {
private store!: NeDBStore;
private folderStore!: FolderStore;
private syncEngine!: SyncEngine;
private emitter!: Emitter;
private autoSyncTimer: ReturnType<typeof setInterval> | null = null;
private collections = new Map<string, Collection<any>>();
// ── Disk persistence state ─────────────────────────────────
private flushTimer: ReturnType<typeof setTimeout> | null = null;
private dirty = false;
kv!: KVApi;
private constructor() {}
static async open(options?: OpenOptions): Promise<FolderSyncDB> {
const db = new FolderSyncDB();
await db.init(options ?? {});
return db;
}
private async init(opts: OpenOptions): Promise<void> {
const dbName = opts.dbName ?? 'FolderSyncDB';
this.emitter = new Emitter();
this.store = await NeDBStore.open(dbName);
this.folderStore = new FolderStore();
// Restore directory handle and load disk state BEFORE client ID setup
// so getMeta finds the persisted clientId from a previous session.
await this.tryRestoreHandle();
if (this.folderStore.hasHandle && (await this.folderStore.hasPermission())) {
await this.loadFromDisk();
}
let clientId = opts.clientId;
if (!clientId) {
clientId = await this.store.getMeta<string>(META_CLIENT_ID);
if (!clientId) {
clientId = generateClientId();
await this.store.setMeta(META_CLIENT_ID, clientId);
}
} else {
await this.store.setMeta(META_CLIENT_ID, clientId);
}
this.syncEngine = new SyncEngine(
this.store,
this.folderStore,
this.emitter,
clientId,
opts.conflictResolver,
);
this.kv = new KVStore(this.store, this.syncEngine);
// Sync any new events not yet in the loaded data
if (this.folderStore.hasHandle && (await this.folderStore.hasPermission())) {
await this.sync();
}
// Schedule disk writes whenever data changes
this.emitter.on('change', () => this.schedulePersist());
// Auto-sync
if (opts.autoSyncIntervalMs && opts.autoSyncIntervalMs > 0) {
this.autoSyncTimer = setInterval(() => {
this.sync().catch(() => {});
}, opts.autoSyncIntervalMs);
}
}
// ── Folder management ──────────────────────────────────────
async selectFolder(): Promise<void> {
const handle = await this.folderStore.selectFolder();
await storeHandle(HANDLE_KEY, handle);
await this.loadFromDisk();
await this.sync();
await this.persistNow();
}
async hasFolderAccess(): Promise<boolean> {
return this.folderStore.hasPermission();
}
async requestFolderAccess(): Promise<boolean> {
const granted = await this.folderStore.requestPermission();
if (granted) {
await this.loadFromDisk();
await this.sync();
}
return granted;
}
// ── Sync ───────────────────────────────────────────────────
async sync(): Promise<void> {
await this.syncEngine.sync();
// Persist immediately after sync (may have imported many events)
await this.persistNow();
}
// ── Collections ────────────────────────────────────────────
collection<T extends { id: string }>(
options: StoreOptions<T>,
): CollectionApi<T> {
const cached = this.collections.get(options.name);
if (cached) return cached as Collection<T>;
const col = new Collection<T>(options, this.store, this.syncEngine);
this.collections.set(options.name, col);
return col;
}
// ── Events ─────────────────────────────────────────────────
on(event: SyncDBEventName, handler: SyncDBEventHandler): () => void {
return this.emitter.on(event, handler);
}
// ── Lifecycle ──────────────────────────────────────────────
async close(): Promise<void> {
if (this.autoSyncTimer !== null) {
clearInterval(this.autoSyncTimer);
this.autoSyncTimer = null;
}
if (this.flushTimer !== null) {
clearTimeout(this.flushTimer);
this.flushTimer = null;
}
// Flush any pending changes before closing
await this.persistNow();
this.emitter.removeAll();
this.collections.clear();
this.store.close();
}
// ── Disk persistence (debounced) ───────────────────────────
private schedulePersist(): void {
this.dirty = true;
if (this.flushTimer) clearTimeout(this.flushTimer);
this.flushTimer = setTimeout(() => {
this.persistNow().catch(() => {});
}, PERSIST_DEBOUNCE_MS);
}
/**
* Write all NeDB datastores to NDJSON files in /data/.
* No-ops if nothing changed or folder is unavailable.
*/
private async persistNow(): Promise<void> {
if (!this.dirty) return;
if (!this.folderStore.hasHandle) return;
const hasAccess = await this.folderStore.hasPermission();
if (!hasAccess) {
this.emitter.emit('folder:lost-permission');
return;
}
try {
const dumps = this.store.dumpAll();
for (const [filename, content] of Object.entries(dumps)) {
await this.folderStore.writeDataFile(filename, content);
}
this.dirty = false;
} catch (err) {
// Keep dirty so we retry on next trigger
this.emitter.emit('folder:lost-permission', err);
}
}
/**
* Load NeDB state from NDJSON files in /data/.
* Called once on startup before syncing events.
*/
private async loadFromDisk(): Promise<void> {
if (!this.folderStore.hasHandle) return;
const hasAccess = await this.folderStore.hasPermission();
if (!hasAccess) return;
const files: Record<string, string> = {};
for (const name of DATA_FILES) {
const content = await this.folderStore.readDataFile(name);
if (content !== null) files[name] = content;
}
if (Object.keys(files).length > 0) {
await this.store.loadAll(files);
}
}
// ── Internals ──────────────────────────────────────────────
private async tryRestoreHandle(): Promise<void> {
try {
const handle = await loadHandle(HANDLE_KEY);
if (!handle) return;
if (typeof handle.queryPermission !== 'function') return;
this.folderStore.setHandle(handle);
} catch {
// Handle was not restorable
}
}
}

88
nedb/src/fs-access.d.ts vendored Normal file
View File

@ -0,0 +1,88 @@
/**
* Type declarations for the File System Access API.
* These APIs are available in Chromium-based browsers only.
* @see https://developer.mozilla.org/en-US/docs/Web/API/File_System_Access_API
*/
interface FileSystemHandlePermissionDescriptor {
mode?: 'read' | 'readwrite';
}
interface FileSystemDirectoryHandle {
readonly kind: 'directory';
readonly name: string;
getDirectoryHandle(
name: string,
options?: { create?: boolean },
): Promise<FileSystemDirectoryHandle>;
getFileHandle(
name: string,
options?: { create?: boolean },
): Promise<FileSystemFileHandle>;
entries(): AsyncIterableIterator<
[string, FileSystemDirectoryHandle | FileSystemFileHandle]
>;
values(): AsyncIterableIterator<
FileSystemDirectoryHandle | FileSystemFileHandle
>;
keys(): AsyncIterableIterator<string>;
queryPermission(
descriptor?: FileSystemHandlePermissionDescriptor,
): Promise<PermissionState>;
requestPermission(
descriptor?: FileSystemHandlePermissionDescriptor,
): Promise<PermissionState>;
}
interface FileSystemFileHandle {
readonly kind: 'file';
readonly name: string;
getFile(): Promise<File>;
createWritable(
options?: FileSystemCreateWritableOptions,
): Promise<FileSystemWritableFileStream>;
}
interface FileSystemCreateWritableOptions {
keepExistingData?: boolean;
}
interface FileSystemWritableFileStream extends WritableStream {
write(data: BufferSource | Blob | string | WriteParams): Promise<void>;
seek(position: number): Promise<void>;
truncate(size: number): Promise<void>;
close(): Promise<void>;
}
interface WriteParams {
type: 'write' | 'seek' | 'truncate';
data?: BufferSource | Blob | string;
position?: number;
size?: number;
}
interface ShowDirectoryPickerOptions {
id?: string;
mode?: 'read' | 'readwrite';
startIn?:
| FileSystemDirectoryHandle
| 'desktop'
| 'documents'
| 'downloads'
| 'music'
| 'pictures'
| 'videos';
}
interface Window {
showDirectoryPicker(
options?: ShowDirectoryPickerOptions,
): Promise<FileSystemDirectoryHandle>;
}

56
nedb/src/handle-store.ts Normal file
View File

@ -0,0 +1,56 @@
/**
* Tiny IndexedDB sidecar for persisting FileSystemDirectoryHandle.
*
* SQLite can't store DOM objects (they require structured clone),
* so we use a minimal IDB store exclusively for the folder handle.
*/
const DB_NAME = 'FolderSyncDB_handles';
const STORE_NAME = 'handles';
const DB_VERSION = 1;
function openHandleDB(): Promise<IDBDatabase> {
return new Promise((resolve, reject) => {
const req = indexedDB.open(DB_NAME, DB_VERSION);
req.onupgradeneeded = () => {
req.result.createObjectStore(STORE_NAME, { keyPath: 'key' });
};
req.onsuccess = () => resolve(req.result);
req.onerror = () => reject(req.error);
});
}
export async function storeHandle(
key: string,
handle: FileSystemDirectoryHandle,
): Promise<void> {
const db = await openHandleDB();
try {
const tx = db.transaction(STORE_NAME, 'readwrite');
tx.objectStore(STORE_NAME).put({ key, handle });
await new Promise<void>((resolve, reject) => {
tx.oncomplete = () => resolve();
tx.onerror = () => reject(tx.error);
});
} finally {
db.close();
}
}
export async function loadHandle(
key: string,
): Promise<FileSystemDirectoryHandle | null> {
const db = await openHandleDB();
try {
const tx = db.transaction(STORE_NAME, 'readonly');
const req = tx.objectStore(STORE_NAME).get(key);
return await new Promise<FileSystemDirectoryHandle | null>(
(resolve, reject) => {
req.onsuccess = () => resolve(req.result?.handle ?? null);
req.onerror = () => reject(req.error);
},
);
} finally {
db.close();
}
}

13
nedb/src/index.ts Normal file
View File

@ -0,0 +1,13 @@
export { FolderSyncDB } from './folder-sync-db.js';
export type {
OpenOptions,
StoreOptions,
IndexDefinition,
IndexRange,
SyncEvent,
KVApi,
CollectionApi,
SyncDBEventName,
SyncDBEventHandler,
} from './types.js';

44
nedb/src/kv-store.ts Normal file
View File

@ -0,0 +1,44 @@
import type { KVApi } from './types.js';
import type { NeDBStore } from './nedb-store.js';
import type { SyncEngine } from './sync-engine.js';
export class KVStore implements KVApi {
constructor(
private store: NeDBStore,
private sync: SyncEngine,
) {}
async get<T = unknown>(key: string): Promise<T | undefined> {
const rec = await this.store.getKV(key);
return rec?.value as T | undefined;
}
async set<T = unknown>(key: string, value: T): Promise<void> {
const existing = await this.store.getKV(key);
const rev = (existing?.rev ?? 0) + 1;
const ts = Date.now();
await this.sync.writeKV(key, value, ts, rev);
}
async delete(key: string): Promise<void> {
const existing = await this.store.getKV(key);
if (!existing) return;
const rev = existing.rev + 1;
const ts = Date.now();
await this.sync.deleteKV(key, ts, rev);
}
async has(key: string): Promise<boolean> {
const rec = await this.store.getKV(key);
return rec !== undefined;
}
async keys(): Promise<string[]> {
return this.store.getAllKVKeys();
}
async entries<T = unknown>(): Promise<Array<[string, T]>> {
const records = await this.store.getAllKVEntries();
return records.map((r) => [r.key, r.value as T]);
}
}

356
nedb/src/nedb-store.ts Normal file
View File

@ -0,0 +1,356 @@
import Datastore from '@seald-io/nedb';
import type { KVRecord, DocRecord, IndexDefinition } from './types.js';
// ── Internal record shapes ───────────────────────────────────
interface KVDoc {
_id: string;
value: unknown;
ts: number;
rev: number;
}
interface DocDoc {
_id: string; // pk = "store\0id"
_store: string;
_docId: string;
_ts: number;
_rev: number;
_deleted: boolean;
data: unknown;
}
interface AppliedDoc {
_id: string; // filename
appliedAt: number;
}
interface MetaDoc {
_id: string;
value: unknown;
}
// ── NeDBStore ────────────────────────────────────────────────
export class NeDBStore {
private kvDB = new Datastore<KVDoc>();
private docsDB = new Datastore<DocDoc>();
private appliedDB = new Datastore<AppliedDoc>();
private metaDB = new Datastore<MetaDoc>();
private indexDefs = new Map<string, Map<string, IndexDefinition>>();
private constructor() {}
static async open(_name: string): Promise<NeDBStore> {
const store = new NeDBStore();
await store.docsDB.ensureIndexAsync({ fieldName: '_store' });
return store;
}
// ── Index registration ─────────────────────────────────────
async registerIndexes(
store: string,
indexes: IndexDefinition[],
): Promise<void> {
this.indexDefs.set(
store,
new Map(indexes.map((i) => [i.name, i])),
);
for (const idx of indexes) {
for (const field of idx.fields as string[]) {
await this.docsDB.ensureIndexAsync({ fieldName: `data.${field}` });
}
}
}
private getIndexDef(store: string, indexName: string): IndexDefinition {
const defs = this.indexDefs.get(store);
if (!defs) {
throw new Error(`No indexes registered for collection "${store}"`);
}
const def = defs.get(indexName);
if (!def) {
throw new Error(
`Index "${indexName}" is not defined on collection "${store}". ` +
`Defined indexes: ${[...defs.keys()].join(', ') || '(none)'}`,
);
}
return def;
}
// ── KV operations ──────────────────────────────────────────
async getKV(key: string): Promise<KVRecord | undefined> {
const doc = await this.kvDB.findOneAsync({ _id: key });
if (!doc) return undefined;
return { key: doc._id, value: doc.value, ts: doc.ts, rev: doc.rev };
}
async putKV(record: KVRecord): Promise<void> {
await this.kvDB.updateAsync(
{ _id: record.key },
{ _id: record.key, value: record.value, ts: record.ts, rev: record.rev },
{ upsert: true },
);
}
async deleteKV(key: string): Promise<void> {
await this.kvDB.removeAsync({ _id: key }, {});
}
async getAllKVKeys(): Promise<string[]> {
const docs = await this.kvDB.findAsync({});
return docs.map((d) => d._id);
}
async getAllKVEntries(): Promise<KVRecord[]> {
const docs = await this.kvDB.findAsync({});
return docs.map((d) => ({
key: d._id,
value: d.value,
ts: d.ts,
rev: d.rev,
}));
}
// ── Doc operations ─────────────────────────────────────────
private docPk(store: string, id: string): string {
return `${store}\0${id}`;
}
async getDoc(store: string, id: string): Promise<DocRecord | undefined> {
const doc = await this.docsDB.findOneAsync({
_id: this.docPk(store, id),
_deleted: false,
});
if (!doc) return undefined;
return this.toDocRecord(doc);
}
async getRawDoc(store: string, id: string): Promise<DocRecord | undefined> {
const doc = await this.docsDB.findOneAsync({
_id: this.docPk(store, id),
});
if (!doc) return undefined;
return this.toDocRecord(doc);
}
async putDoc(
record: DocRecord,
_indexes: IndexDefinition[],
): Promise<void> {
const pk = this.docPk(record.store, record.id);
await this.docsDB.updateAsync(
{ _id: pk },
{
_id: pk,
_store: record.store,
_docId: record.id,
_ts: record.ts,
_rev: record.rev,
_deleted: !!record.deleted,
data: record.data,
},
{ upsert: true },
);
}
async deleteDoc(
store: string,
id: string,
ts: number,
rev: number,
indexes: IndexDefinition[],
): Promise<void> {
return this.putDoc(
{ store, id, data: null, ts, rev, deleted: true },
indexes,
);
}
async getAllDocs(store: string): Promise<DocRecord[]> {
const docs = await this.docsDB.findAsync({
_store: store,
_deleted: false,
});
return docs.map((d) => this.toDocRecord(d));
}
// ── Index queries ──────────────────────────────────────────
async findByIndex(
store: string,
indexName: string,
value: unknown,
): Promise<string[]> {
const def = this.getIndexDef(store, indexName);
const query = this.buildExactQuery(store, def, value);
const docs = await this.docsDB.findAsync(query);
return docs.map((d) => d._docId);
}
async queryByIndex(
store: string,
indexName: string,
range: { gt?: unknown; gte?: unknown; lt?: unknown; lte?: unknown },
): Promise<string[]> {
const def = this.getIndexDef(store, indexName);
const fields = def.fields as string[];
if (fields.length !== 1) {
throw new Error(
'Range queries are only supported on single-field indexes',
);
}
const field = `data.${fields[0]}`;
const query: Record<string, unknown> = {
_store: store,
_deleted: false,
};
const rangeOp: Record<string, unknown> = {};
if (range.gte !== undefined) rangeOp['$gte'] = range.gte;
if (range.gt !== undefined) rangeOp['$gt'] = range.gt;
if (range.lte !== undefined) rangeOp['$lte'] = range.lte;
if (range.lt !== undefined) rangeOp['$lt'] = range.lt;
query[field] = rangeOp;
const docs = await this.docsDB.findAsync(query);
return docs.map((d) => d._docId);
}
async rebuildIndexes(
store: string,
indexes: IndexDefinition[],
): Promise<void> {
await this.registerIndexes(store, indexes);
}
// ── Applied-event tracking ─────────────────────────────────
async isEventApplied(filename: string): Promise<boolean> {
const doc = await this.appliedDB.findOneAsync({ _id: filename });
return doc !== null;
}
async markEventApplied(filename: string): Promise<void> {
await this.appliedDB.updateAsync(
{ _id: filename },
{ _id: filename, appliedAt: Date.now() },
{ upsert: true },
);
}
async getAppliedSet(): Promise<Set<string>> {
const docs = await this.appliedDB.findAsync({});
return new Set(docs.map((d) => d._id));
}
// ── Meta ───────────────────────────────────────────────────
async getMeta<T = unknown>(key: string): Promise<T | undefined> {
const doc = await this.metaDB.findOneAsync({ _id: key });
if (!doc) return undefined;
return doc.value as T;
}
async setMeta(key: string, value: unknown): Promise<void> {
await this.metaDB.updateAsync(
{ _id: key },
{ _id: key, value },
{ upsert: true },
);
}
// ── Disk persistence (NDJSON dump / load) ──────────────────
/**
* Dump all four datastores as NDJSON strings.
* Uses getAllData() for synchronous, zero-copy access.
*/
dumpAll(): Record<string, string> {
return {
'kv.db': toNDJSON(this.kvDB.getAllData()),
'docs.db': toNDJSON(this.docsDB.getAllData()),
'applied.db': toNDJSON(this.appliedDB.getAllData()),
'meta.db': toNDJSON(this.metaDB.getAllData()),
};
}
/**
* Load NDJSON data files into the (empty) datastores.
* Must be called before any other operations.
*/
async loadAll(files: Record<string, string>): Promise<void> {
await this.loadInto(this.kvDB, files['kv.db']);
await this.loadInto(this.docsDB, files['docs.db']);
await this.loadInto(this.appliedDB, files['applied.db']);
await this.loadInto(this.metaDB, files['meta.db']);
}
// eslint-disable-next-line @typescript-eslint/no-explicit-any
private async loadInto(
db: Datastore<any>,
content: string | undefined,
): Promise<void> {
if (!content?.trim()) return;
const docs = content
.split('\n')
.filter((l) => l.trim())
.map((l) => JSON.parse(l));
if (docs.length === 0) return;
// Clear existing data first — disk state replaces in-memory state
await db.removeAsync({}, { multi: true });
await db.insertAsync(docs);
}
// ── Lifecycle ──────────────────────────────────────────────
close(): void {
// NeDB in-memory — no cleanup needed
}
// ── Helpers ────────────────────────────────────────────────
private toDocRecord(doc: DocDoc): DocRecord {
return {
store: doc._store,
id: doc._docId,
data: doc.data,
ts: doc._ts,
rev: doc._rev,
deleted: doc._deleted,
};
}
private buildExactQuery(
store: string,
def: IndexDefinition,
value: unknown,
): Record<string, unknown> {
const query: Record<string, unknown> = {
_store: store,
_deleted: false,
};
const fields = def.fields as string[];
if (fields.length === 1) {
query[`data.${fields[0]}`] = value;
} else {
const values = value as unknown[];
for (let i = 0; i < fields.length; i++) {
query[`data.${fields[i]}`] = values[i];
}
}
return query;
}
}
// ── Utilities ────────────────────────────────────────────────
function toNDJSON(docs: unknown[]): string {
return docs.map((d) => JSON.stringify(d)).join('\n');
}

87
nedb/src/nedb-types.d.ts vendored Normal file
View File

@ -0,0 +1,87 @@
/**
* Minimal type declarations for @seald-io/nedb (v4.x).
* Covers the async API surface used by this library.
*/
declare module '@seald-io/nedb' {
interface DatastoreOptions {
filename?: string;
inMemoryOnly?: boolean;
autoload?: boolean;
timestampData?: boolean;
onload?: (err: Error | null) => void;
afterSerialization?: (doc: string) => string;
beforeDeserialization?: (doc: string) => string;
corruptAlertThreshold?: number;
compareStrings?: (a: string, b: string) => number;
}
interface Cursor<T> {
sort(query: Record<string, 1 | -1>): Cursor<T>;
skip(n: number): Cursor<T>;
limit(n: number): Cursor<T>;
projection(query: Record<string, 0 | 1>): Cursor<T>;
execAsync(): Promise<T>;
then: Promise<T>['then'];
}
interface EnsureIndexOptions {
fieldName: string;
unique?: boolean;
sparse?: boolean;
expireAfterSeconds?: number;
}
interface UpdateOptions {
multi?: boolean;
upsert?: boolean;
returnUpdatedDocs?: boolean;
}
interface UpdateResult<T> {
numAffected: number;
affectedDocuments: T | T[] | null;
upsert: boolean;
}
interface RemoveOptions {
multi?: boolean;
}
class Datastore<T = Record<string, unknown>> {
constructor(options?: DatastoreOptions | string);
// ── Async API (native promises) ──────────────────────────
loadDatabaseAsync(): Promise<void>;
insertAsync(doc: T): Promise<T>;
insertAsync(docs: T[]): Promise<T[]>;
findAsync(
query: Record<string, unknown>,
projection?: Record<string, 0 | 1>,
): Cursor<T[]>;
findOneAsync(
query: Record<string, unknown>,
projection?: Record<string, 0 | 1>,
): Cursor<T | null>;
updateAsync(
query: Record<string, unknown>,
update: Record<string, unknown> | T,
options?: UpdateOptions,
): Promise<UpdateResult<T>>;
removeAsync(
query: Record<string, unknown>,
options?: RemoveOptions,
): Promise<number>;
countAsync(
query: Record<string, unknown>,
): Cursor<number>;
ensureIndexAsync(options: EnsureIndexOptions): Promise<void>;
removeIndexAsync(fieldName: string): Promise<void>;
compactDatafileAsync(): Promise<void>;
dropDatabaseAsync(): Promise<void>;
// ── Synchronous helpers ──────────────────────────────────
getAllData(): T[];
}
export default Datastore;
}

257
nedb/src/sync-engine.ts Normal file
View File

@ -0,0 +1,257 @@
import type { SyncEvent, IndexDefinition, OpenOptions } from './types.js';
import type { NeDBStore } from './nedb-store.js';
import type { FolderStore } from './folder-store.js';
import type { Emitter } from './emitter.js';
import { canonicalJson, sha256Hex, eventFilename } from './utils.js';
export class SyncEngine {
private clientId: string;
private conflictResolver?: OpenOptions['conflictResolver'];
private collectionIndexes = new Map<string, IndexDefinition[]>();
private syncing = false;
constructor(
private store: NeDBStore,
private folder: FolderStore,
private emitter: Emitter,
clientId: string,
conflictResolver?: OpenOptions['conflictResolver'],
) {
this.clientId = clientId;
this.conflictResolver = conflictResolver;
}
registerIndexes(store: string, indexes: IndexDefinition[]): void {
this.collectionIndexes.set(store, indexes);
}
getIndexes(store: string): IndexDefinition[] {
return this.collectionIndexes.get(store) ?? [];
}
async writeKV(
key: string,
value: unknown,
ts: number,
rev: number,
): Promise<void> {
await this.persistEvent({
type: 'put',
store: 'kv',
key,
ts,
clientId: this.clientId,
data: value,
rev,
});
}
async deleteKV(key: string, ts: number, rev: number): Promise<void> {
await this.persistEvent({
type: 'delete',
store: 'kv',
key,
ts,
clientId: this.clientId,
rev,
});
}
async writeDoc(
store: string,
id: string,
data: unknown,
ts: number,
rev: number,
): Promise<void> {
await this.persistEvent({
type: 'put',
store,
key: id,
id,
ts,
clientId: this.clientId,
data,
rev,
});
}
async deleteDocEvent(
store: string,
id: string,
ts: number,
rev: number,
): Promise<void> {
await this.persistEvent({
type: 'delete',
store,
key: id,
id,
ts,
clientId: this.clientId,
rev,
});
}
// ── Core persist ───────────────────────────────────────────
private async persistEvent(event: SyncEvent): Promise<void> {
const canonical = canonicalJson(event);
const hash = await sha256Hex(canonical);
const filename = eventFilename(event.ts, hash);
await this.applyEvent(event);
await this.store.markEventApplied(filename);
if (this.folder.hasHandle && (await this.folder.hasPermission())) {
try {
await this.folder.writeEvent(filename, event);
} catch (err) {
this.emitter.emit('folder:lost-permission', err);
}
}
this.emitter.emit('change', {
type: event.type,
store: event.store,
key: event.key,
id: event.id,
data: event.data,
});
}
// ── Sync ───────────────────────────────────────────────────
async sync(): Promise<void> {
if (this.syncing) return;
if (!this.folder.hasHandle) return;
const hasAccess = await this.folder.hasPermission();
if (!hasAccess) {
this.emitter.emit('folder:lost-permission');
return;
}
this.syncing = true;
this.emitter.emit('sync:start');
try {
const appliedSet = await this.store.getAppliedSet();
const filenames = await this.folder.scanEventFilenames();
let importCount = 0;
for (const name of filenames) {
if (appliedSet.has(name)) continue;
const event = await this.folder.readEventFile(name);
if (!event) continue;
const hadConflict = await this.applyEventWithConflictCheck(event);
await this.store.markEventApplied(name);
importCount++;
if (hadConflict) {
this.emitter.emit('conflict', { filename: name, event });
}
this.emitter.emit('change', {
type: event.type,
store: event.store,
key: event.key,
id: event.id,
data: event.data,
source: 'sync',
});
}
this.emitter.emit('sync:end', { imported: importCount });
} catch (err) {
this.emitter.emit('sync:end', { error: err });
} finally {
this.syncing = false;
}
}
// ── Apply ──────────────────────────────────────────────────
private async applyEvent(event: SyncEvent): Promise<void> {
if (event.store === 'kv') {
if (event.type === 'put') {
await this.store.putKV({
key: event.key,
value: event.data,
ts: event.ts,
rev: event.rev ?? 0,
});
} else {
await this.store.deleteKV(event.key);
}
} else {
const indexes = this.getIndexes(event.store);
if (event.type === 'put') {
await this.store.putDoc(
{
store: event.store,
id: event.id ?? event.key,
data: event.data,
ts: event.ts,
rev: event.rev ?? 0,
},
indexes,
);
} else {
await this.store.deleteDoc(
event.store,
event.id ?? event.key,
event.ts,
event.rev ?? 0,
indexes,
);
}
}
}
private async applyEventWithConflictCheck(
event: SyncEvent,
): Promise<boolean> {
let hadConflict = false;
if (event.store === 'kv') {
const existing = await this.store.getKV(event.key);
if (existing) {
if (event.ts > existing.ts) {
hadConflict = true;
} else if (event.ts === existing.ts) {
hadConflict = true;
if (this.conflictResolver) {
const resolved = this.conflictResolver(existing.value, event.data);
event = { ...event, data: resolved };
}
} else {
return true;
}
}
} else {
const existing = await this.store.getRawDoc(
event.store,
event.id ?? event.key,
);
if (existing && !existing.deleted) {
if (event.ts > existing.ts) {
hadConflict = true;
} else if (event.ts === existing.ts) {
hadConflict = true;
if (this.conflictResolver) {
const resolved = this.conflictResolver(existing.data, event.data);
event = { ...event, data: resolved };
}
} else {
return true;
}
}
}
await this.applyEvent(event);
return hadConflict;
}
}

93
nedb/src/types.ts Normal file
View File

@ -0,0 +1,93 @@
// ── Event types ──────────────────────────────────────────────
export type EventType = 'put' | 'delete';
export interface SyncEvent {
type: EventType;
store: string; // "kv" | collection name
key: string;
id?: string; // present for collection documents
ts: number; // millisecond Unix epoch
clientId: string;
data?: unknown; // absent on deletes
rev?: number;
}
// ── Index / collection options ───────────────────────────────
export interface IndexDefinition<T = unknown> {
name: string;
fields: (keyof T & string)[] | string[];
}
export interface StoreOptions<T = unknown> {
name: string;
indexes?: IndexDefinition<T>[];
}
export interface OpenOptions {
dbName?: string;
autoSyncIntervalMs?: number;
clientId?: string;
conflictResolver?: (current: unknown, incoming: unknown) => unknown;
}
// ── Store record shapes ──────────────────────────────────────
export interface KVRecord {
key: string;
value: unknown;
ts: number;
rev: number;
}
export interface DocRecord {
store: string;
id: string;
data: unknown;
ts: number;
rev: number;
deleted?: boolean;
}
// ── Index range (replaces IDBKeyRange for SQLite) ────────────
export interface IndexRange {
gt?: unknown;
gte?: unknown;
lt?: unknown;
lte?: unknown;
}
// ── Event emitter ────────────────────────────────────────────
export type SyncDBEventName =
| 'sync:start'
| 'sync:end'
| 'change'
| 'conflict'
| 'folder:lost-permission';
export type SyncDBEventHandler = (...args: unknown[]) => void;
// ── KV API surface ───────────────────────────────────────────
export interface KVApi {
get<T = unknown>(key: string): Promise<T | undefined>;
set<T = unknown>(key: string, value: T): Promise<void>;
delete(key: string): Promise<void>;
has(key: string): Promise<boolean>;
keys(): Promise<string[]>;
entries<T = unknown>(): Promise<Array<[string, T]>>;
}
// ── Collection API surface ───────────────────────────────────
export interface CollectionApi<T extends { id: string }> {
get(id: string): Promise<T | undefined>;
put(doc: T): Promise<void>;
delete(id: string): Promise<void>;
all(): Promise<T[]>;
findByIndex(indexName: string, value: unknown): Promise<T[]>;
queryByIndex(indexName: string, range: IndexRange): Promise<T[]>;
}

94
nedb/src/utils.ts Normal file
View File

@ -0,0 +1,94 @@
// ── Canonical JSON ───────────────────────────────────────────
function sortKeys(obj: unknown): unknown {
if (obj === null || obj === undefined || typeof obj !== 'object') return obj;
if (Array.isArray(obj)) return obj.map(sortKeys);
const sorted: Record<string, unknown> = {};
for (const key of Object.keys(obj as Record<string, unknown>).sort()) {
sorted[key] = sortKeys((obj as Record<string, unknown>)[key]);
}
return sorted;
}
export function canonicalJson(obj: unknown): string {
return JSON.stringify(sortKeys(obj));
}
// ── Hashing ──────────────────────────────────────────────────
export async function sha256Hex(data: string): Promise<string> {
const buf = await crypto.subtle.digest(
'SHA-256',
new TextEncoder().encode(data),
);
return Array.from(new Uint8Array(buf))
.map((b) => b.toString(16).padStart(2, '0'))
.join('');
}
// ── Event filenames ──────────────────────────────────────────
const HASH_PREFIX_LEN = 12;
export function eventFilename(ts: number, fullHash: string): string {
return `${ts}_${fullHash.slice(0, HASH_PREFIX_LEN)}.json`;
}
const EVENT_FILE_RE = /^(\d+)_([0-9a-f]+)\.json$/;
export function parseEventFilename(
name: string,
): { ts: number; hash: string } | null {
const m = EVENT_FILE_RE.exec(name);
if (!m) return null;
return { ts: Number(m[1]), hash: m[2] };
}
// ── Client ID ────────────────────────────────────────────────
export function generateClientId(): string {
return crypto.randomUUID();
}
// ── Index value extraction ───────────────────────────────────
export function extractIndexValue(
data: unknown,
fields: string[],
): IDBValidKey | undefined {
if (!data || typeof data !== 'object') return undefined;
const rec = data as Record<string, unknown>;
if (fields.length === 1) {
const v = getNestedValue(rec, fields[0]);
if (v === undefined || v === null) return undefined;
return toIDBKey(v);
}
const parts: IDBValidKey[] = [];
for (const f of fields) {
const v = getNestedValue(rec, f);
if (v === undefined || v === null) return undefined;
parts.push(toIDBKey(v));
}
return parts;
}
function getNestedValue(obj: Record<string, unknown>, path: string): unknown {
let cur: unknown = obj;
for (const seg of path.split('.')) {
if (cur === null || cur === undefined || typeof cur !== 'object') {
return undefined;
}
cur = (cur as Record<string, unknown>)[seg];
}
return cur;
}
function toIDBKey(v: unknown): IDBValidKey {
if (typeof v === 'string' || typeof v === 'number') return v;
if (v instanceof Date) return v;
if (Array.isArray(v)) return v.map(toIDBKey);
// Fallback: stringify
return String(v);
}

23
nedb/tsconfig.json Normal file
View File

@ -0,0 +1,23 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "ES2022",
"moduleResolution": "bundler",
"lib": ["ES2022", "DOM", "DOM.Iterable"],
"outDir": "dist",
"rootDir": "src",
"declaration": true,
"declarationMap": true,
"sourceMap": true,
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noFallthroughCasesInSwitch": true,
"forceConsistentCasingInFileNames": true,
"esModuleInterop": true,
"allowSyntheticDefaultImports": true,
"skipLibCheck": true
},
"include": ["src"],
"exclude": ["node_modules", "dist"]
}

123
paste/README.md Normal file
View File

@ -0,0 +1,123 @@
# Paste -- Demo App
A minimal paste-bin app used to test and compare all four IndexSyncFile storage variants. Each version compiles to a single self-contained HTML file.
## What it does
1. You type or paste text into a text area
2. It saves automatically to a local database
3. You pick a sync folder on disk
4. Open the same HTML file in another browser window, point it at the same folder
5. Both windows stay in sync (auto-sync every 3 seconds)
That's it. No server, no accounts, no network. Just two browser windows and a folder.
## Four versions
| File | Storage engine | Opens from `file://`? | Size |
|------|---------------|----------------------|------|
| `paste-indexeddb.html` | IndexedDB (browser built-in) | Yes | ~23 KB |
| `paste-nedb.html` | NeDB (in-memory, MongoDB-style) | Yes | ~115 KB |
| `paste-sql-js.html` | sql.js (SQLite via asm.js) | Yes | ~1.9 MB |
| `paste-sqlite.html` | SQLite WASM (official build) | No (needs HTTP) | ~1.4 MB |
### paste-indexeddb.html
Uses the browser's built-in IndexedDB. Smallest file, fastest startup, zero dependencies. The best choice for most use cases.
### paste-nedb.html
Uses NeDB (`@seald-io/nedb`) running in-memory. Persists NDJSON snapshots to `/data/` in the selected folder for fast reload. Useful if you need MongoDB-style query operators (`$gt`, `$in`, `$regex`).
### paste-sql-js.html
Uses sql.js, which is SQLite compiled to pure JavaScript (asm.js, not WebAssembly). The full SQLite database is exported as a binary file to `/data/store.sqlite` in the selected folder. You can open that file with any SQLite tool. Larger bundle but works from `file://` with no special setup.
### paste-sqlite.html
Uses the official `@sqlite.org/sqlite-wasm` package. Requires an HTTP server because the browser must `fetch()` the `.wasm` binary at runtime. May also need COOP/COEP headers for OPFS persistence. Run `bun run serve.ts` to test this variant.
## How to test
### Quick (three variants)
Double-click or drag any of these into Chrome:
```
dist/paste-indexeddb.html
dist/paste-nedb.html
dist/paste-sql-js.html
```
They work directly from `file://`.
### SQLite WASM variant (needs HTTP)
```bash
bun run serve.ts
```
Then open `http://localhost:4040/paste-sqlite.html`.
### Multi-browser sync test
1. Open `paste-indexeddb.html` in Chrome
2. Click "Select sync folder" and pick an empty folder
3. Type something
4. Open the same `paste-indexeddb.html` in a second Chrome window
5. Click "Select sync folder" and pick the **same folder**
6. Wait a few seconds -- the text should appear in the second window
7. Edit in either window -- changes propagate both ways
## Building
Requires [Bun](https://bun.sh/) 1.3.10+.
```bash
bun run build.ts
```
This produces four single-file HTMLs in `dist/`. The build script:
1. Runs `Bun.build()` for each variant (bundles TypeScript, minifies)
2. Inlines all JS and CSS chunks into the HTML
3. Copies `sqlite3.wasm` to `dist/` for the WASM variant
## Project structure
```
paste/
indexeddb/ source for IndexedDB paste app
app.ts
index.html
nedb/ source for NeDB paste app
app.ts
index.html
sql-js/ source for sql.js paste app
app.ts
index.html
sqlite/ source for SQLite WASM paste app
app.ts
index.html
shared.ts shared UI logic (all variants import this)
styles.css shared styles
build.ts bun build script
serve.ts dev server for SQLite WASM variant
dist/ built output (single HTML files)
```
## Folder layout on disk
After selecting a sync folder, you'll see:
```
your-folder/
events/
1710000000000_a1b2c3d4e5f6.json
1710000001000_f6e5d4c3b2a1.json
data/ (nedb and sql-js only)
kv.db (nedb: NDJSON)
docs.db (nedb: NDJSON)
store.sqlite (sql-js: SQLite binary)
```
The `events/` folder is shared by all variants. If you point two different variant HTMLs at the same folder, they will sync with each other (the event format is identical across all four).

99
paste/build.ts Normal file
View File

@ -0,0 +1,99 @@
/**
* Build script for paste app variants.
*
* Usage:
* bun run build.ts
*
* Produces self-contained HTML files in dist/:
* dist/paste-indexeddb.html
* dist/paste-sqlite.html
* dist/paste-nedb.html
*
* Each file is a single self-contained HTML with all JS and CSS inlined.
*
* If bun >= 1.3.10, you can also use:
* bun build --compile --target=browser ./indexeddb/index.html
*
* Docs: https://bun.com/docs/bundler/html-static
*/
import { readFileSync } from 'fs';
import { join, basename } from 'path';
const variants = ['indexeddb', 'sqlite', 'nedb', 'sql-js'] as const;
for (const variant of variants) {
console.log(`Building paste-${variant}...`);
const outdir = `./dist/${variant}`;
const result = await Bun.build({
entrypoints: [`./${variant}/index.html`],
outdir,
target: 'browser',
minify: true,
});
if (!result.success) {
console.error(` FAILED: paste-${variant}`);
for (const log of result.logs) {
console.error(' ', log);
}
process.exit(1);
}
// Read the generated HTML and inline all referenced chunks
const htmlPath = join(outdir, 'index.html');
let html = readFileSync(htmlPath, 'utf-8');
// Inline JS chunks: <script type="module" src="./chunk-xxx.js">
html = html.replace(
/<script[^>]+src="\.\/([^"]+\.js)"[^>]*><\/script>/g,
(_match, jsFile) => {
const jsPath = join(outdir, jsFile);
try {
const js = readFileSync(jsPath, 'utf-8');
return `<script type="module">${js}</script>`;
} catch {
console.warn(` Warning: could not inline ${jsFile}`);
return _match;
}
},
);
// Inline CSS chunks: <link rel="stylesheet" href="./chunk-xxx.css">
html = html.replace(
/<link[^>]+href="\.\/([^"]+\.css)"[^>]*\/?>/g,
(_match, cssFile) => {
const cssPath = join(outdir, cssFile);
try {
const css = readFileSync(cssPath, 'utf-8');
return `<style>${css}</style>`;
} catch {
console.warn(` Warning: could not inline ${cssFile}`);
return _match;
}
},
);
// Write the single self-contained HTML
const outFile = `./dist/paste-${variant}.html`;
await Bun.write(outFile, html);
console.log(` OK: ${outFile}`);
}
// Copy sqlite3.wasm to dist/ (needed for SQLite variant over HTTP)
const wasmSrc = '../sqlite/node_modules/@sqlite.org/sqlite-wasm/dist/sqlite3.wasm';
const wasmDst = './dist/sqlite3.wasm';
try {
const wasmFile = Bun.file(wasmSrc);
if (await wasmFile.exists()) {
await Bun.write(wasmDst, wasmFile);
console.log(`\nCopied sqlite3.wasm (${Math.round(wasmFile.size! / 1024)}KB) to dist/`);
}
} catch {
console.warn('\nWarning: could not copy sqlite3.wasm — run `npm install` in sqlite/ first');
}
console.log('\nAll builds complete. Single-file HTMLs in paste/dist/');
console.log('Note: paste-sqlite.html requires HTTP — run `bun run serve.ts`');

7
paste/indexeddb/app.ts Normal file
View File

@ -0,0 +1,7 @@
import { FolderSyncDB } from '../../indexeddb/src/index.ts';
import { initPasteApp } from '../shared.ts';
document.addEventListener('DOMContentLoaded', async () => {
const db = await FolderSyncDB.open({ autoSyncIntervalMs: 3000 });
await initPasteApp(db as any, 'indexeddb');
});

View File

@ -0,0 +1,37 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>paste — IndexedDB</title>
<link rel="stylesheet" href="../styles.css">
</head>
<body>
<header>
<h1>paste <span id="variant-label" class="variant">indexeddb</span></h1>
<p class="sub">type text + enter, or paste an image &mdash; use #hashtags to add tags</p>
<div class="folder-bar">
<button id="select-folder" class="folder-btn">Select sync folder</button>
<span id="folder-status" class="status"></span>
</div>
</header>
<main>
<div class="input-area">
<div class="input-row">
<input type="text" id="paste-input"
placeholder="type something and press Enter, or Ctrl+V an image"
autocomplete="off" spellcheck="false">
<button class="clip-btn" id="clip-btn" title="Attach a file">
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path d="M21.44 11.05l-9.19 9.19a6 6 0 01-8.49-8.49l9.19-9.19a4 4 0 015.66 5.66l-9.2 9.19a2 2 0 01-2.83-2.83l8.49-8.48"/>
</svg>
</button>
<input type="file" id="file-input" hidden>
</div>
<div id="status" class="status"></div>
</div>
<div id="items" class="items"></div>
</main>
<script type="module" src="./app.ts"></script>
</body>
</html>

7
paste/nedb/app.ts Normal file
View File

@ -0,0 +1,7 @@
import { FolderSyncDB } from '../../nedb/src/index.ts';
import { initPasteApp } from '../shared.ts';
document.addEventListener('DOMContentLoaded', async () => {
const db = await FolderSyncDB.open({ autoSyncIntervalMs: 3000 });
await initPasteApp(db as any, 'nedb');
});

37
paste/nedb/index.html Normal file
View File

@ -0,0 +1,37 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>paste — NeDB</title>
<link rel="stylesheet" href="../styles.css">
</head>
<body>
<header>
<h1>paste <span id="variant-label" class="variant">nedb</span></h1>
<p class="sub">type text + enter, or paste an image &mdash; use #hashtags to add tags</p>
<div class="folder-bar">
<button id="select-folder" class="folder-btn">Select sync folder</button>
<span id="folder-status" class="status"></span>
</div>
</header>
<main>
<div class="input-area">
<div class="input-row">
<input type="text" id="paste-input"
placeholder="type something and press Enter, or Ctrl+V an image"
autocomplete="off" spellcheck="false">
<button class="clip-btn" id="clip-btn" title="Attach a file">
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path d="M21.44 11.05l-9.19 9.19a6 6 0 01-8.49-8.49l9.19-9.19a4 4 0 015.66 5.66l-9.2 9.19a2 2 0 01-2.83-2.83l8.49-8.48"/>
</svg>
</button>
<input type="file" id="file-input" hidden>
</div>
<div id="status" class="status"></div>
</div>
<div id="items" class="items"></div>
</main>
<script type="module" src="./app.ts"></script>
</body>
</html>

52
paste/serve.ts Normal file
View File

@ -0,0 +1,52 @@
/**
* Simple static file server for testing paste variants.
*
* Usage:
* bun run serve.ts
*
* Then open:
* http://localhost:3456/paste-indexeddb.html
* http://localhost:3456/paste-sqlite.html
* http://localhost:3456/paste-nedb.html
*
* The SQLite variant REQUIRES HTTP (can't load .wasm from file://).
* The IndexedDB and NeDB variants also work from file:// directly.
*/
const PORT = 3456;
const mimeTypes: Record<string, string> = {
'.html': 'text/html',
'.js': 'text/javascript',
'.css': 'text/css',
'.wasm': 'application/wasm',
'.json': 'application/json',
};
Bun.serve({
port: PORT,
async fetch(req) {
const url = new URL(req.url);
let path = url.pathname;
if (path === '/') path = '/index.html';
const filePath = './dist' + path;
const file = Bun.file(filePath);
if (!(await file.exists())) {
return new Response('Not found', { status: 404 });
}
const ext = path.substring(path.lastIndexOf('.'));
const contentType = mimeTypes[ext] || 'application/octet-stream';
return new Response(file, {
headers: { 'Content-Type': contentType },
});
},
});
console.log(`Serving paste/dist/ at http://localhost:${PORT}/`);
console.log(` http://localhost:${PORT}/paste-indexeddb.html`);
console.log(` http://localhost:${PORT}/paste-sqlite.html`);
console.log(` http://localhost:${PORT}/paste-nedb.html`);

318
paste/shared.ts Normal file
View File

@ -0,0 +1,318 @@
// ── Types ────────────────────────────────────────────────────
interface Paste {
id: string;
mimeType: string;
description: string;
tags: string[];
timestamp: number;
fileName: string;
fileData: string; // base64 (empty for text-only)
}
// Duck-typed DB interface — all three variants satisfy this
interface PasteDB {
selectFolder(): Promise<void>;
hasFolderAccess(): Promise<boolean>;
requestFolderAccess(): Promise<boolean>;
sync(): Promise<void>;
collection(opts: {
name: string;
indexes: { name: string; fields: string[] }[];
}): {
put(doc: Paste): Promise<void>;
all(): Promise<Paste[]>;
delete(id: string): Promise<void>;
};
on(event: string, handler: (...args: unknown[]) => void): () => void;
}
// ── Helpers ──────────────────────────────────────────────────
function genId(): string {
return Date.now().toString(36) + Math.random().toString(36).slice(2, 8);
}
function extractTags(text: string): string[] {
return text
.split(/\s+/)
.filter((w) => w.startsWith('#') && w.length > 1)
.map((w) => w.slice(1).replace(/[^a-zA-Z0-9_-]+$/, ''))
.filter((t) => t.length > 0);
}
function fileToBase64(file: Blob): Promise<string> {
return new Promise((resolve, reject) => {
const reader = new FileReader();
reader.onload = () => {
const result = reader.result as string;
resolve(result.split(',')[1] || '');
};
reader.onerror = reject;
reader.readAsDataURL(file);
});
}
function escapeHtml(s: string): string {
const d = document.createElement('div');
d.textContent = s;
return d.innerHTML;
}
function fmtTime(ts: number): string {
const d = new Date(ts);
const p = (n: number) => String(n).padStart(2, '0');
return `${d.getFullYear()}-${p(d.getMonth() + 1)}-${p(d.getDate())} ${p(d.getHours())}:${p(d.getMinutes())}:${p(d.getSeconds())}`;
}
const pdfSvg =
'<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"><path d="M14 2H6a2 2 0 00-2 2v16a2 2 0 002 2h12a2 2 0 002-2V8z"/><polyline points="14 2 14 8 20 8"/><text x="12" y="17" text-anchor="middle" fill="currentColor" stroke="none" font-size="6" font-weight="bold" font-family="sans-serif">PDF</text></svg>';
const fileSvg =
'<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"><path d="M14 2H6a2 2 0 00-2 2v16a2 2 0 002 2h12a2 2 0 002-2V8z"/><polyline points="14 2 14 8 20 8"/><line x1="16" y1="13" x2="8" y2="13"/><line x1="16" y1="17" x2="8" y2="17"/></svg>';
// ── Main init ────────────────────────────────────────────────
export async function initPasteApp(db: PasteDB, variantLabel: string) {
const pastes = db.collection({
name: 'pastes',
indexes: [{ name: 'byTimestamp', fields: ['timestamp'] }],
});
const $ = (s: string) => document.querySelector(s)!;
const input = $('#paste-input') as HTMLInputElement;
const statusEl = $('#status')!;
const itemsEl = $('#items')!;
const selectFolderBtn = $('#select-folder') as HTMLButtonElement;
const folderStatusEl = $('#folder-status')!;
const variantEl = $('#variant-label');
if (variantEl) variantEl.textContent = variantLabel;
// ── Folder management ────────────────────────────────────
function updateFolderUI(connected: boolean) {
if (connected) {
folderStatusEl.textContent = 'Folder connected';
folderStatusEl.className = 'status ok';
selectFolderBtn.textContent = 'Change folder';
} else {
folderStatusEl.textContent = 'No folder selected';
folderStatusEl.className = 'status';
selectFolderBtn.textContent = 'Select sync folder';
}
}
selectFolderBtn.addEventListener('click', async () => {
try {
await db.selectFolder();
updateFolderUI(true);
await refreshItems();
} catch (e: unknown) {
setStatus('Folder: ' + (e as Error).message, 'err');
}
});
// Check if we already have access from a previous session
const hasAccess = await db.hasFolderAccess();
if (hasAccess) {
const granted = await db.requestFolderAccess();
updateFolderUI(granted);
} else {
updateFolderUI(false);
}
// ── Status ───────────────────────────────────────────────
function setStatus(msg: string, type?: string) {
statusEl.textContent = msg;
statusEl.className = 'status' + (type ? ' ' + type : '');
if (type === 'ok')
setTimeout(() => {
statusEl.textContent = '';
statusEl.className = 'status';
}, 2500);
}
// ── Save text ────────────────────────────────────────────
async function pasteText(text: string) {
setStatus('Saving...');
try {
const paste: Paste = {
id: genId(),
mimeType: 'text/plain',
description: text,
tags: extractTags(text),
timestamp: Date.now(),
fileName: '',
fileData: '',
};
await pastes.put(paste);
setStatus('Saved', 'ok');
await refreshItems();
} catch (e: unknown) {
setStatus('Error: ' + (e as Error).message, 'err');
}
}
// ── Save file ────────────────────────────────────────────
async function pasteFile(
file: Blob,
description: string,
fileName: string,
) {
setStatus('Saving...');
try {
const base64 = await fileToBase64(file);
const paste: Paste = {
id: genId(),
mimeType: file.type || 'application/octet-stream',
description,
tags: extractTags(description),
timestamp: Date.now(),
fileName,
fileData: base64,
};
await pastes.put(paste);
setStatus('Saved', 'ok');
await refreshItems();
} catch (e: unknown) {
setStatus('Error: ' + (e as Error).message, 'err');
}
}
// ── Refresh & render ─────────────────────────────────────
async function refreshItems() {
try {
const all: Paste[] = await pastes.all();
all.sort((a, b) => b.timestamp - a.timestamp);
renderItems(all);
} catch (e: unknown) {
setStatus('Load error: ' + (e as Error).message, 'err');
}
}
function renderItems(items: Paste[]) {
if (!items || items.length === 0) {
itemsEl.innerHTML =
'<div class="empty">Nothing here yet. Type something or paste an image.</div>';
return;
}
itemsEl.innerHTML = items
.map((it) => {
const time = fmtTime(it.timestamp);
const shortId = it.id.substring(0, 12);
const isImage = it.mimeType.startsWith('image/');
const isPdf = it.mimeType === 'application/pdf';
const isText = it.mimeType.startsWith('text/');
let thumb = '';
if (isImage && it.fileData) {
const src = `data:${it.mimeType};base64,${it.fileData}`;
thumb = `<img class="item-thumb" src="${src}" alt="">`;
} else if (isPdf) {
thumb = `<div class="item-icon pdf">${pdfSvg}</div>`;
} else if (!isText && it.fileData) {
thumb = `<div class="item-icon file">${fileSvg}</div>`;
}
let content = '';
if (isImage && it.fileData) {
const src = `data:${it.mimeType};base64,${it.fileData}`;
content = `<div class="item-image"><img src="${src}" alt="pasted image" loading="lazy"></div>`;
}
if (it.description) {
content += `<div class="item-content">${escapeHtml(it.description)}</div>`;
}
let fileLink = '';
if (it.fileName && !isText) {
fileLink = `<div class="item-filename">${escapeHtml(it.fileName)}</div>`;
}
let tagsHtml = '';
if (it.tags && it.tags.length > 0) {
tagsHtml =
'<div class="item-tags">' +
it.tags
.map((t) => `<span class="tag">#${escapeHtml(t)}</span>`)
.join('') +
'</div>';
}
return (
`<div class="item">` +
thumb +
`<div class="item-body">` +
`<div class="item-meta">` +
`<span class="time">${time}</span>` +
`<span class="hash">${shortId}</span>` +
`<span>${escapeHtml(it.mimeType)}</span>` +
`</div>` +
fileLink +
content +
tagsHtml +
`</div></div>`
);
})
.join('');
}
// ── Event listeners ──────────────────────────────────────
input.addEventListener('keydown', (e: KeyboardEvent) => {
if (e.key === 'Enter' && input.value.trim()) {
pasteText(input.value.trim());
input.value = '';
}
});
document.addEventListener('paste', (e: ClipboardEvent) => {
const items = e.clipboardData?.items;
if (!items) return;
for (const item of items) {
if (item.type.startsWith('image/')) {
e.preventDefault();
const blob = item.getAsFile();
if (blob) {
const ext =
blob.type === 'image/png'
? '.png'
: blob.type === 'image/jpeg'
? '.jpg'
: blob.type === 'image/gif'
? '.gif'
: blob.type === 'image/webp'
? '.webp'
: '.bin';
pasteFile(blob, input.value.trim(), 'clipboard' + ext);
input.value = '';
}
return;
}
}
});
const fileInput = $('#file-input') as HTMLInputElement;
($('#clip-btn') as HTMLElement).addEventListener('click', () =>
fileInput.click(),
);
fileInput.addEventListener('change', () => {
const file = fileInput.files?.[0];
if (file) {
pasteFile(file, input.value.trim(), file.name);
input.value = '';
fileInput.value = '';
}
});
// Live updates from sync
db.on('change', () => refreshItems());
// Initial load
await refreshItems();
}

7
paste/sql-js/app.ts Normal file
View File

@ -0,0 +1,7 @@
import { FolderSyncDB } from '../../sql-js/src/index.ts';
import { initPasteApp } from '../shared.ts';
document.addEventListener('DOMContentLoaded', async () => {
const db = await FolderSyncDB.open({ autoSyncIntervalMs: 3000 });
await initPasteApp(db as any, 'sql.js');
});

37
paste/sql-js/index.html Normal file
View File

@ -0,0 +1,37 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>paste — sql.js</title>
<link rel="stylesheet" href="../styles.css">
</head>
<body>
<header>
<h1>paste <span id="variant-label" class="variant">sql.js</span></h1>
<p class="sub">type text + enter, or paste an image &mdash; use #hashtags to add tags</p>
<div class="folder-bar">
<button id="select-folder" class="folder-btn">Select sync folder</button>
<span id="folder-status" class="status"></span>
</div>
</header>
<main>
<div class="input-area">
<div class="input-row">
<input type="text" id="paste-input"
placeholder="type something and press Enter, or Ctrl+V an image"
autocomplete="off" spellcheck="false">
<button class="clip-btn" id="clip-btn" title="Attach a file">
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path d="M21.44 11.05l-9.19 9.19a6 6 0 01-8.49-8.49l9.19-9.19a4 4 0 015.66 5.66l-9.2 9.19a2 2 0 01-2.83-2.83l8.49-8.48"/>
</svg>
</button>
<input type="file" id="file-input" hidden>
</div>
<div id="status" class="status"></div>
</div>
<div id="items" class="items"></div>
</main>
<script type="module" src="./app.ts"></script>
</body>
</html>

33
paste/sqlite/app.ts Normal file
View File

@ -0,0 +1,33 @@
import { SQLITE3_WASM_BASE64 } from './sqlite3-wasm.ts';
import { FolderSyncDB } from '../../sqlite/src/index.ts';
import { initPasteApp } from '../shared.ts';
// Decode embedded WASM → blob URL so sqlite3InitModule loads
// without fetch(), enabling file:// usage (no server needed).
function base64ToBlob(b64: string, mime: string): Blob {
const bin = atob(b64);
const bytes = new Uint8Array(bin.length);
for (let i = 0; i < bin.length; i++) bytes[i] = bin.charCodeAt(i);
return new Blob([bytes], { type: mime });
}
const wasmBlobUrl = URL.createObjectURL(
base64ToBlob(SQLITE3_WASM_BASE64, 'application/wasm'),
);
document.addEventListener('DOMContentLoaded', async () => {
try {
const db = await FolderSyncDB.open({
autoSyncIntervalMs: 3000,
sqlite3Config: {
locateFile: (path: string) =>
path.endsWith('.wasm') ? wasmBlobUrl : path,
},
});
await initPasteApp(db as any, 'sqlite');
} catch (e) {
console.error('SQLite init failed:', e);
document.body.innerHTML = `<div style="color:#e05555;padding:2rem;font-family:monospace">
SQLite init failed: ${(e as Error).message}</div>`;
}
});

37
paste/sqlite/index.html Normal file
View File

@ -0,0 +1,37 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>paste — SQLite</title>
<link rel="stylesheet" href="../styles.css">
</head>
<body>
<header>
<h1>paste <span id="variant-label" class="variant">sqlite</span></h1>
<p class="sub">type text + enter, or paste an image &mdash; use #hashtags to add tags</p>
<div class="folder-bar">
<button id="select-folder" class="folder-btn">Select sync folder</button>
<span id="folder-status" class="status"></span>
</div>
</header>
<main>
<div class="input-area">
<div class="input-row">
<input type="text" id="paste-input"
placeholder="type something and press Enter, or Ctrl+V an image"
autocomplete="off" spellcheck="false">
<button class="clip-btn" id="clip-btn" title="Attach a file">
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path d="M21.44 11.05l-9.19 9.19a6 6 0 01-8.49-8.49l9.19-9.19a4 4 0 015.66 5.66l-9.2 9.19a2 2 0 01-2.83-2.83l8.49-8.48"/>
</svg>
</button>
<input type="file" id="file-input" hidden>
</div>
<div id="status" class="status"></div>
</div>
<div id="items" class="items"></div>
</main>
<script type="module" src="./app.ts"></script>
</body>
</html>

File diff suppressed because one or more lines are too long

190
paste/styles.css Normal file
View File

@ -0,0 +1,190 @@
:root {
--bg: #1a1a1e;
--bg-card: #24242a;
--bg-input: #2c2c34;
--border: #38383f;
--text: #e4e4e8;
--text-muted: #8888a0;
--accent: #6c8cff;
--accent-dim: #4a6ad0;
--success: #4caf80;
--error: #e05555;
--radius: 6px;
--mono: "SF Mono", "Cascadia Code", "Consolas", monospace;
--sans: -apple-system, "Segoe UI", system-ui, sans-serif;
}
*, *::before, *::after { margin: 0; padding: 0; box-sizing: border-box; }
body {
background: var(--bg);
color: var(--text);
font-family: var(--sans);
line-height: 1.5;
max-width: 720px;
margin: 0 auto;
padding: 2rem 1rem;
}
header { margin-bottom: 1.5rem; }
header h1 {
font-size: 1.4rem;
font-weight: 600;
color: var(--accent);
font-family: var(--mono);
letter-spacing: -0.02em;
}
.variant {
font-size: 0.75rem;
font-weight: 400;
color: var(--text-muted);
background: var(--bg-input);
padding: 0.15rem 0.5rem;
border-radius: 3px;
vertical-align: middle;
}
header .sub {
color: var(--text-muted);
font-size: 0.82rem;
margin-top: 0.2rem;
}
.folder-bar {
display: flex;
align-items: center;
gap: 0.75rem;
margin-top: 0.6rem;
}
.folder-btn {
padding: 0.45rem 0.9rem;
background: var(--accent);
color: #fff;
border: none;
border-radius: var(--radius);
font-size: 0.82rem;
font-family: var(--sans);
cursor: pointer;
transition: background 0.15s;
}
.folder-btn:hover { background: var(--accent-dim); }
.input-area { margin-bottom: 1.5rem; }
.input-row {
display: flex;
gap: 0.5rem;
align-items: center;
}
#paste-input {
flex: 1;
padding: 0.7rem 1rem;
background: var(--bg-input);
border: 1px solid var(--border);
border-radius: var(--radius);
color: var(--text);
font-size: 0.95rem;
font-family: var(--sans);
outline: none;
transition: border-color 0.15s;
}
#paste-input:focus { border-color: var(--accent); }
#paste-input::placeholder { color: var(--text-muted); }
.clip-btn {
flex-shrink: 0;
width: 38px;
height: 38px;
background: var(--bg-input);
border: 1px solid var(--border);
border-radius: var(--radius);
color: var(--text-muted);
cursor: pointer;
display: flex;
align-items: center;
justify-content: center;
transition: border-color 0.15s, color 0.15s;
}
.clip-btn:hover { border-color: var(--accent); color: var(--accent); }
.clip-btn svg { width: 18px; height: 18px; }
.status {
font-size: 0.78rem;
min-height: 1.2em;
color: var(--text-muted);
}
.status.ok { color: var(--success); }
.status.err { color: var(--error); }
.items { display: flex; flex-direction: column; gap: 0.5rem; }
.item {
background: var(--bg-card);
border: 1px solid var(--border);
border-radius: var(--radius);
padding: 0.65rem 0.85rem;
display: flex;
gap: 0.7rem;
align-items: flex-start;
}
.item-thumb {
flex-shrink: 0;
width: 48px;
height: 48px;
border-radius: 4px;
object-fit: cover;
background: var(--bg);
}
.item-icon {
flex-shrink: 0;
width: 48px;
height: 48px;
border-radius: 4px;
background: var(--bg);
display: flex;
align-items: center;
justify-content: center;
}
.item-icon svg { width: 26px; height: 26px; }
.item-icon.pdf { color: #e05555; }
.item-icon.file { color: var(--text-muted); }
.item-filename {
font-size: 0.78rem;
margin-top: 0.2rem;
color: var(--accent);
}
.item-body { flex: 1; min-width: 0; }
.item-meta {
display: flex;
flex-wrap: wrap;
gap: 0.6rem;
font-size: 0.73rem;
color: var(--text-muted);
margin-bottom: 0.25rem;
}
.item-meta .hash {
font-family: var(--mono);
color: var(--accent-dim);
}
.item-content {
font-size: 0.88rem;
white-space: pre-wrap;
word-break: break-word;
max-height: 6em;
overflow: hidden;
}
.item-image { margin-top: 0.4rem; }
.item-image img {
max-width: 100%;
max-height: 200px;
border-radius: 4px;
}
.item-tags {
display: flex;
flex-wrap: wrap;
gap: 0.3rem;
margin-top: 0.3rem;
}
.tag {
font-size: 0.7rem;
font-family: var(--mono);
background: #2a2f45;
color: var(--accent);
padding: 0.1rem 0.45rem;
border-radius: 3px;
}
.empty {
text-align: center;
color: var(--text-muted);
padding: 3rem 1rem;
font-size: 0.9rem;
}

115
sql-js/README.md Normal file
View File

@ -0,0 +1,115 @@
# IndexSyncFile -- sql.js Variant
Local-first key-value and document store using **sql.js** (SQLite compiled to JavaScript via asm.js) as the local cache, syncing with a user-selected folder via the File System Access API.
## Why sql.js
- **Works from `file://`** -- pure JavaScript, no `.wasm` file to fetch, no COOP/COEP headers needed
- **Real SQL** -- full SQLite query engine with proper indexes, joins, and transactions
- **Single-file friendly** -- bun/webpack/rollup can inline everything into one HTML file
- **Correct numeric ordering** -- range queries on numbers compare as numbers, not strings
- **Portable database file** -- `db.export()` produces a standard SQLite binary that any SQLite tool can open
## Architecture
```
Write: app ---> sql.js in-memory (immediate) ---> /events/timestamp_hash.json
---> /data/store.sqlite (debounced)
Sync: /events/*.json ---> sort by timestamp ---> skip applied ---> apply to sql.js
---> persist store.sqlite
Startup: load /data/store.sqlite (instant) ---> sync only NEW events from /events/
```
### Two persistence layers
| Layer | Purpose | Format |
|-------|---------|--------|
| `/events/` | **Sync mechanism** -- immutable event files enable multi-browser sync without data loss | One JSON file per mutation |
| `/data/store.sqlite` | **Fast reload** -- full SQLite database binary so startup doesn't replay the entire event history | Standard SQLite 3 file |
The event log is essential for sync. The SQLite file is a startup optimization. Without it, every page load would replay all events from scratch.
### The SQLite file is a real database
The `store.sqlite` file written to `/data/` is a standard SQLite database. You can open it with any SQLite tool (DB Browser for SQLite, the `sqlite3` CLI, Python's `sqlite3` module) to inspect, query, or debug your data.
### SQL schema (5 tables)
```sql
kv (key TEXT PK, value TEXT, ts INTEGER, rev INTEGER)
docs (store TEXT, id TEXT, data TEXT, ts INTEGER, rev INTEGER, deleted INTEGER, PK(store,id))
idx_entries (store TEXT, index_name TEXT, id TEXT, value ANY, PK(store,index_name,id))
applied (filename TEXT PK, applied_at INTEGER)
meta (key TEXT PK, value TEXT)
```
## How it differs from the `sqlite/` variant
| | `sql-js/` (this) | `sqlite/` |
|--|-------------------|-----------|
| Engine | sql.js (asm.js, pure JS) | `@sqlite.org/sqlite-wasm` (WebAssembly) |
| `.wasm` file | None | Required, fetched at runtime |
| `file://` support | Yes | No |
| HTTP headers | None needed | May need COOP/COEP for OPFS |
| Speed | ~2-3x slower than WASM | Fastest SQLite in browser |
| Browser persistence | None (in-memory only) | OPFS (survives reload natively) |
| Disk persistence | Exports full `.sqlite` to folder | Event log only |
| Bundle size | ~2MB (asm.js is large) | ~800KB + separate .wasm |
For this library's use case (small-to-medium datasets, infrequent queries, folder-based persistence), the speed difference is negligible.
## Install and build
```bash
npm install
npm run build
```
**Dependency:** `sql.js` -- SQLite compiled to JavaScript. The asm.js build is used (not the WASM build) so everything works from `file://` with zero fetches.
## Usage
```ts
import { FolderSyncDB } from './dist/index.js';
const db = await FolderSyncDB.open({
dbName: 'MyApp',
autoSyncIntervalMs: 5000,
});
await db.selectFolder();
// KV
await db.kv.set('config', { theme: 'dark', lang: 'en' });
// Collections
const tasks = db.collection({
name: 'tasks',
indexes: [{ name: 'byPriority', fields: ['priority'] }],
});
await tasks.put({ id: 't1', title: 'Deploy', priority: 1 });
// Range queries with proper numeric ordering
const urgent = await tasks.queryByIndex('byPriority', { gte: 1, lte: 3 });
```
## Folder layout
```
your-folder/
events/
1710000000000_a1b2c3d4e5f6.json
1710000001000_f6e5d4c3b2a1.json
data/
store.sqlite <-- standard SQLite 3 database file
```
## Variant-specific notes
- sql.js runs entirely in-memory; persistence comes from the event log and the exported `.sqlite` file
- The `.sqlite` file is written with a 250ms debounce to avoid excessive disk writes during batch operations
- Directory handle is stored in a tiny IDB sidecar (sql.js cannot store DOM objects)
- On `close()`, any pending writes are flushed before releasing resources
- Index values are stored with SQLite type affinity, so numbers compare as numbers in range queries

29
sql-js/package.json Normal file
View File

@ -0,0 +1,29 @@
{
"name": "index-sync-file-sql-js",
"version": "1.0.0",
"description": "Local-first key-value and document store backed by sql.js (pure JS SQLite), syncing with a user-selected folder via File System Access API",
"type": "module",
"main": "dist/index.js",
"module": "dist/index.js",
"types": "dist/index.d.ts",
"exports": {
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.js"
}
},
"files": ["dist"],
"scripts": {
"build": "tsc",
"dev": "tsc --watch",
"clean": "rm -rf dist"
},
"keywords": ["local-first", "sql.js", "sqlite", "file-system-access-api", "key-value", "document-store", "sync"],
"license": "MIT",
"dependencies": {
"sql.js": "^1.14.0"
},
"devDependencies": {
"typescript": "^5.7.0"
}
}

64
sql-js/src/collection.ts Normal file
View File

@ -0,0 +1,64 @@
import type { CollectionApi, IndexDefinition, IndexRange, StoreOptions } from './types.js';
import type { SqlJsStore } from './sqljs-store.js';
import type { SyncEngine } from './sync-engine.js';
export class Collection<T extends { id: string }> implements CollectionApi<T> {
private readonly storeName: string;
private readonly indexes: IndexDefinition<T>[];
private indexesBuilt = false;
constructor(options: StoreOptions<T>, private store: SqlJsStore, private sync: SyncEngine) {
this.storeName = options.name;
this.indexes = options.indexes ?? [];
this.sync.registerIndexes(this.storeName, this.indexes as IndexDefinition[]);
}
private async ensureIndexes(): Promise<void> {
if (this.indexesBuilt || this.indexes.length === 0) return;
await this.store.rebuildIndexes(this.storeName, this.indexes as IndexDefinition[]);
this.indexesBuilt = true;
}
async get(id: string): Promise<T | undefined> {
const rec = await this.store.getDoc(this.storeName, id);
return rec ? (rec.data as T) : undefined;
}
async put(doc: T): Promise<void> {
await this.ensureIndexes();
const existing = await this.store.getRawDoc(this.storeName, doc.id);
await this.sync.writeDoc(this.storeName, doc.id, doc, Date.now(), (existing?.rev ?? 0) + 1);
}
async delete(id: string): Promise<void> {
await this.ensureIndexes();
const existing = await this.store.getRawDoc(this.storeName, id);
if (!existing || existing.deleted) return;
await this.sync.deleteDocEvent(this.storeName, id, Date.now(), existing.rev + 1);
}
async all(): Promise<T[]> {
return (await this.store.getAllDocs(this.storeName)).map((d) => d.data as T);
}
async findByIndex(indexName: string, value: unknown): Promise<T[]> {
await this.ensureIndexes();
const ids = await this.store.findByIndex(this.storeName, indexName, value);
return this.fetchByIds(ids);
}
async queryByIndex(indexName: string, range: IndexRange): Promise<T[]> {
await this.ensureIndexes();
const ids = await this.store.queryByIndex(this.storeName, indexName, range);
return this.fetchByIds(ids);
}
private async fetchByIds(ids: string[]): Promise<T[]> {
const results: T[] = [];
for (const id of ids) {
const doc = await this.store.getDoc(this.storeName, id);
if (doc) results.push(doc.data as T);
}
return results;
}
}

33
sql-js/src/emitter.ts Normal file
View File

@ -0,0 +1,33 @@
import type { SyncDBEventName, SyncDBEventHandler } from './types.js';
export class Emitter {
private listeners = new Map<SyncDBEventName, Set<SyncDBEventHandler>>();
on(event: SyncDBEventName, handler: SyncDBEventHandler): () => void {
let set = this.listeners.get(event);
if (!set) {
set = new Set();
this.listeners.set(event, set);
}
set.add(handler);
return () => {
set!.delete(handler);
};
}
emit(event: SyncDBEventName, ...args: unknown[]): void {
const set = this.listeners.get(event);
if (!set) return;
for (const fn of set) {
try {
fn(...args);
} catch {
// listener errors must not break the library
}
}
}
removeAll(): void {
this.listeners.clear();
}
}

154
sql-js/src/folder-store.ts Normal file
View File

@ -0,0 +1,154 @@
import type { SyncEvent } from './types.js';
import { parseEventFilename } from './utils.js';
const EVENTS_DIR = 'events';
export class FolderStore {
private dirHandle: FileSystemDirectoryHandle | null = null;
// ── Handle management ──────────────────────────────────────
get hasHandle(): boolean {
return this.dirHandle !== null;
}
getHandle(): FileSystemDirectoryHandle | null {
return this.dirHandle;
}
setHandle(handle: FileSystemDirectoryHandle): void {
this.dirHandle = handle;
}
async selectFolder(): Promise<FileSystemDirectoryHandle> {
if (typeof window.showDirectoryPicker !== 'function') {
throw new Error(
'File System Access API is not supported in this browser. ' +
'Use a Chromium-based browser (Chrome, Edge, Brave, etc.).',
);
}
this.dirHandle = await window.showDirectoryPicker({
mode: 'readwrite',
});
// Ensure events directory exists
await this.dirHandle.getDirectoryHandle(EVENTS_DIR, { create: true });
return this.dirHandle;
}
// ── Permission checks ─────────────────────────────────────
async hasPermission(): Promise<boolean> {
if (!this.dirHandle) return false;
try {
const perm = await this.dirHandle.queryPermission({ mode: 'readwrite' });
return perm === 'granted';
} catch {
return false;
}
}
async requestPermission(): Promise<boolean> {
if (!this.dirHandle) return false;
try {
const perm = await this.dirHandle.requestPermission({
mode: 'readwrite',
});
return perm === 'granted';
} catch {
return false;
}
}
// ── Write event file ──────────────────────────────────────
async writeEvent(filename: string, event: SyncEvent): Promise<void> {
const eventsDir = await this.getEventsDir();
const fileHandle = await eventsDir.getFileHandle(filename, {
create: true,
});
const writable = await fileHandle.createWritable();
try {
await writable.write(JSON.stringify(event, null, 2));
} finally {
await writable.close();
}
}
// ── Scan events ────────────────────────────────────────────
async scanEventFilenames(): Promise<string[]> {
const eventsDir = await this.getEventsDir();
const names: string[] = [];
for await (const [name, handle] of eventsDir.entries()) {
if (handle.kind === 'file' && name.endsWith('.json')) {
if (parseEventFilename(name)) {
names.push(name);
}
}
}
// Sort by (timestamp, full filename) for deterministic ordering
names.sort((a, b) => {
const pa = parseEventFilename(a)!;
const pb = parseEventFilename(b)!;
if (pa.ts !== pb.ts) return pa.ts - pb.ts;
return a.localeCompare(b);
});
return names;
}
async readEventFile(filename: string): Promise<SyncEvent | null> {
const eventsDir = await this.getEventsDir();
try {
const fh = await eventsDir.getFileHandle(filename);
const file = await fh.getFile();
const text = await file.text();
return JSON.parse(text) as SyncEvent;
} catch {
return null;
}
}
// ── Data files (SQLite database persistence) ────────────────
async writeDataFile(filename: string, data: Uint8Array): Promise<void> {
const dataDir = await this.getDataDir();
const fh = await dataDir.getFileHandle(filename, { create: true });
const writable = await fh.createWritable();
try {
await writable.write(data.buffer as ArrayBuffer);
} finally {
await writable.close();
}
}
async readDataFile(filename: string): Promise<Uint8Array | null> {
try {
const dataDir = await this.getDataDir();
const fh = await dataDir.getFileHandle(filename);
const file = await fh.getFile();
const buf = await file.arrayBuffer();
return new Uint8Array(buf);
} catch {
return null;
}
}
// ── Internals ──────────────────────────────────────────────
private async getEventsDir(): Promise<FileSystemDirectoryHandle> {
if (!this.dirHandle) {
throw new Error('No folder selected. Call selectFolder() first.');
}
return this.dirHandle.getDirectoryHandle(EVENTS_DIR, { create: true });
}
private async getDataDir(): Promise<FileSystemDirectoryHandle> {
if (!this.dirHandle) {
throw new Error('No folder selected. Call selectFolder() first.');
}
return this.dirHandle.getDirectoryHandle('data', { create: true });
}
}

View File

@ -0,0 +1,185 @@
import type {
OpenOptions, StoreOptions, CollectionApi, KVApi,
SyncDBEventName, SyncDBEventHandler,
} from './types.js';
import { generateClientId } from './utils.js';
import { Emitter } from './emitter.js';
import { SqlJsStore } from './sqljs-store.js';
import { FolderStore } from './folder-store.js';
import { SyncEngine } from './sync-engine.js';
import { KVStore } from './kv-store.js';
import { Collection } from './collection.js';
import { storeHandle, loadHandle } from './handle-store.js';
const META_CLIENT_ID = 'clientId';
const HANDLE_KEY = 'dirHandle';
const DB_FILE = 'store.sqlite';
const PERSIST_DEBOUNCE_MS = 250;
export class FolderSyncDB {
private store!: SqlJsStore;
private folderStore!: FolderStore;
private syncEngine!: SyncEngine;
private emitter!: Emitter;
private autoSyncTimer: ReturnType<typeof setInterval> | null = null;
private collections = new Map<string, Collection<any>>();
// Disk persistence state
private flushTimer: ReturnType<typeof setTimeout> | null = null;
private dirty = false;
kv!: KVApi;
private constructor() {}
static async open(options?: OpenOptions): Promise<FolderSyncDB> {
const db = new FolderSyncDB();
await db.init(options ?? {});
return db;
}
private async init(opts: OpenOptions): Promise<void> {
const dbName = opts.dbName ?? 'FolderSyncDB';
this.emitter = new Emitter();
this.folderStore = new FolderStore();
// Restore directory handle and load SQLite from disk BEFORE client ID setup
await this.tryRestoreHandle();
let existingData: Uint8Array | undefined;
if (this.folderStore.hasHandle && (await this.folderStore.hasPermission())) {
const data = await this.folderStore.readDataFile(DB_FILE);
if (data) existingData = data;
}
this.store = await SqlJsStore.open(dbName, existingData);
let clientId = opts.clientId;
if (!clientId) {
clientId = await this.store.getMeta<string>(META_CLIENT_ID);
if (!clientId) {
clientId = generateClientId();
await this.store.setMeta(META_CLIENT_ID, clientId);
}
} else {
await this.store.setMeta(META_CLIENT_ID, clientId);
}
this.syncEngine = new SyncEngine(
this.store, this.folderStore, this.emitter, clientId, opts.conflictResolver,
);
this.kv = new KVStore(this.store, this.syncEngine);
// Sync any new events from folder
if (this.folderStore.hasHandle && (await this.folderStore.hasPermission())) {
await this.syncEngine.sync();
await this.persistNow();
}
// Schedule disk writes whenever data changes
this.emitter.on('change', () => this.schedulePersist());
if (opts.autoSyncIntervalMs && opts.autoSyncIntervalMs > 0) {
this.autoSyncTimer = setInterval(() => { this.sync().catch(() => {}); }, opts.autoSyncIntervalMs);
}
}
// ── Folder management ──────────────────────────────────────
async selectFolder(): Promise<void> {
const handle = await this.folderStore.selectFolder();
await storeHandle(HANDLE_KEY, handle);
await this.sync();
await this.persistNow();
}
async hasFolderAccess(): Promise<boolean> {
return this.folderStore.hasPermission();
}
async requestFolderAccess(): Promise<boolean> {
const granted = await this.folderStore.requestPermission();
if (granted) {
await this.sync();
await this.persistNow();
}
return granted;
}
// ── Sync ───────────────────────────────────────────────────
async sync(): Promise<void> {
await this.syncEngine.sync();
await this.persistNow();
}
// ── Collections ────────────────────────────────────────────
collection<T extends { id: string }>(options: StoreOptions<T>): CollectionApi<T> {
const cached = this.collections.get(options.name);
if (cached) return cached as Collection<T>;
const col = new Collection<T>(options, this.store, this.syncEngine);
this.collections.set(options.name, col);
return col;
}
// ── Events ─────────────────────────────────────────────────
on(event: SyncDBEventName, handler: SyncDBEventHandler): () => void {
return this.emitter.on(event, handler);
}
// ── Lifecycle ──────────────────────────────────────────────
async close(): Promise<void> {
if (this.autoSyncTimer !== null) { clearInterval(this.autoSyncTimer); this.autoSyncTimer = null; }
if (this.flushTimer !== null) { clearTimeout(this.flushTimer); this.flushTimer = null; }
await this.persistNow();
this.emitter.removeAll();
this.collections.clear();
this.store.close();
}
// ── Disk persistence (debounced) ───────────────────────────
private schedulePersist(): void {
this.dirty = true;
if (this.flushTimer) clearTimeout(this.flushTimer);
this.flushTimer = setTimeout(() => {
this.persistNow().catch(() => {});
}, PERSIST_DEBOUNCE_MS);
}
/**
* Write the full SQLite database to /data/store.sqlite.
*/
private async persistNow(): Promise<void> {
if (!this.dirty) return;
if (!this.folderStore.hasHandle) return;
const hasAccess = await this.folderStore.hasPermission();
if (!hasAccess) {
this.emitter.emit('folder:lost-permission');
return;
}
try {
const data = this.store.exportDB();
await this.folderStore.writeDataFile(DB_FILE, data);
this.dirty = false;
} catch (err) {
this.emitter.emit('folder:lost-permission', err);
}
}
// ── Internals ──────────────────────────────────────────────
private async tryRestoreHandle(): Promise<void> {
try {
const handle = await loadHandle(HANDLE_KEY);
if (!handle) return;
if (typeof handle.queryPermission !== 'function') return;
this.folderStore.setHandle(handle);
} catch {}
}
}

88
sql-js/src/fs-access.d.ts vendored Normal file
View File

@ -0,0 +1,88 @@
/**
* Type declarations for the File System Access API.
* These APIs are available in Chromium-based browsers only.
* @see https://developer.mozilla.org/en-US/docs/Web/API/File_System_Access_API
*/
interface FileSystemHandlePermissionDescriptor {
mode?: 'read' | 'readwrite';
}
interface FileSystemDirectoryHandle {
readonly kind: 'directory';
readonly name: string;
getDirectoryHandle(
name: string,
options?: { create?: boolean },
): Promise<FileSystemDirectoryHandle>;
getFileHandle(
name: string,
options?: { create?: boolean },
): Promise<FileSystemFileHandle>;
entries(): AsyncIterableIterator<
[string, FileSystemDirectoryHandle | FileSystemFileHandle]
>;
values(): AsyncIterableIterator<
FileSystemDirectoryHandle | FileSystemFileHandle
>;
keys(): AsyncIterableIterator<string>;
queryPermission(
descriptor?: FileSystemHandlePermissionDescriptor,
): Promise<PermissionState>;
requestPermission(
descriptor?: FileSystemHandlePermissionDescriptor,
): Promise<PermissionState>;
}
interface FileSystemFileHandle {
readonly kind: 'file';
readonly name: string;
getFile(): Promise<File>;
createWritable(
options?: FileSystemCreateWritableOptions,
): Promise<FileSystemWritableFileStream>;
}
interface FileSystemCreateWritableOptions {
keepExistingData?: boolean;
}
interface FileSystemWritableFileStream extends WritableStream {
write(data: BufferSource | Blob | string | WriteParams): Promise<void>;
seek(position: number): Promise<void>;
truncate(size: number): Promise<void>;
close(): Promise<void>;
}
interface WriteParams {
type: 'write' | 'seek' | 'truncate';
data?: BufferSource | Blob | string;
position?: number;
size?: number;
}
interface ShowDirectoryPickerOptions {
id?: string;
mode?: 'read' | 'readwrite';
startIn?:
| FileSystemDirectoryHandle
| 'desktop'
| 'documents'
| 'downloads'
| 'music'
| 'pictures'
| 'videos';
}
interface Window {
showDirectoryPicker(
options?: ShowDirectoryPickerOptions,
): Promise<FileSystemDirectoryHandle>;
}

View File

@ -0,0 +1,56 @@
/**
* Tiny IndexedDB sidecar for persisting FileSystemDirectoryHandle.
*
* SQLite can't store DOM objects (they require structured clone),
* so we use a minimal IDB store exclusively for the folder handle.
*/
const DB_NAME = 'FolderSyncDB_handles';
const STORE_NAME = 'handles';
const DB_VERSION = 1;
function openHandleDB(): Promise<IDBDatabase> {
return new Promise((resolve, reject) => {
const req = indexedDB.open(DB_NAME, DB_VERSION);
req.onupgradeneeded = () => {
req.result.createObjectStore(STORE_NAME, { keyPath: 'key' });
};
req.onsuccess = () => resolve(req.result);
req.onerror = () => reject(req.error);
});
}
export async function storeHandle(
key: string,
handle: FileSystemDirectoryHandle,
): Promise<void> {
const db = await openHandleDB();
try {
const tx = db.transaction(STORE_NAME, 'readwrite');
tx.objectStore(STORE_NAME).put({ key, handle });
await new Promise<void>((resolve, reject) => {
tx.oncomplete = () => resolve();
tx.onerror = () => reject(tx.error);
});
} finally {
db.close();
}
}
export async function loadHandle(
key: string,
): Promise<FileSystemDirectoryHandle | null> {
const db = await openHandleDB();
try {
const tx = db.transaction(STORE_NAME, 'readonly');
const req = tx.objectStore(STORE_NAME).get(key);
return await new Promise<FileSystemDirectoryHandle | null>(
(resolve, reject) => {
req.onsuccess = () => resolve(req.result?.handle ?? null);
req.onerror = () => reject(req.error);
},
);
} finally {
db.close();
}
}

13
sql-js/src/index.ts Normal file
View File

@ -0,0 +1,13 @@
export { FolderSyncDB } from './folder-sync-db.js';
export type {
OpenOptions,
StoreOptions,
IndexDefinition,
IndexRange,
SyncEvent,
KVApi,
CollectionApi,
SyncDBEventName,
SyncDBEventHandler,
} from './types.js';

40
sql-js/src/kv-store.ts Normal file
View File

@ -0,0 +1,40 @@
import type { KVApi } from './types.js';
import type { SqlJsStore } from './sqljs-store.js';
import type { SyncEngine } from './sync-engine.js';
export class KVStore implements KVApi {
constructor(
private store: SqlJsStore,
private sync: SyncEngine,
) {}
async get<T = unknown>(key: string): Promise<T | undefined> {
const rec = await this.store.getKV(key);
return rec?.value as T | undefined;
}
async set<T = unknown>(key: string, value: T): Promise<void> {
const existing = await this.store.getKV(key);
const rev = (existing?.rev ?? 0) + 1;
await this.sync.writeKV(key, value, Date.now(), rev);
}
async delete(key: string): Promise<void> {
const existing = await this.store.getKV(key);
if (!existing) return;
await this.sync.deleteKV(key, Date.now(), existing.rev + 1);
}
async has(key: string): Promise<boolean> {
return (await this.store.getKV(key)) !== undefined;
}
async keys(): Promise<string[]> {
return this.store.getAllKVKeys();
}
async entries<T = unknown>(): Promise<Array<[string, T]>> {
const records = await this.store.getAllKVEntries();
return records.map((r) => [r.key, r.value as T]);
}
}

341
sql-js/src/sqljs-store.ts Normal file
View File

@ -0,0 +1,341 @@
// Use the asm.js build — pure JS, no .wasm file, works from file://
import initSqlJs from 'sql.js/dist/sql-asm.js';
import type { Database } from 'sql.js';
import type { KVRecord, DocRecord, IndexDefinition } from './types.js';
import { extractIndexValue } from './utils.js';
// ── Value serialization ──────────────────────────────────────
function serializeIndexValue(v: unknown): unknown {
if (typeof v === 'number' || typeof v === 'string') return v;
if (v instanceof Date) return v.getTime();
if (Array.isArray(v)) return JSON.stringify(v);
return String(v);
}
// ── SqlJsStore ───────────────────────────────────────────────
export class SqlJsStore {
private constructor(private db: Database) {}
// ── Open ───────────────────────────────────────────────────
static async open(
_name: string,
existingData?: Uint8Array,
): Promise<SqlJsStore> {
// sql.js asm.js build: no .wasm file, no fetch, no headers
const SQL = await initSqlJs();
const db = existingData
? new SQL.Database(existingData)
: new SQL.Database();
const store = new SqlJsStore(db);
store.createSchema(); // no-op if tables already exist (IF NOT EXISTS)
return store;
}
// ── Disk persistence ────────────────────────────────────────
/** Export the entire SQLite database as a binary Uint8Array. */
exportDB(): Uint8Array {
return this.db.export();
}
private createSchema(): void {
this.db.run(`
CREATE TABLE IF NOT EXISTS kv (
key TEXT PRIMARY KEY,
value TEXT NOT NULL,
ts INTEGER NOT NULL,
rev INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS docs (
store TEXT NOT NULL,
id TEXT NOT NULL,
data TEXT NOT NULL,
ts INTEGER NOT NULL,
rev INTEGER NOT NULL,
deleted INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (store, id)
);
CREATE INDEX IF NOT EXISTS docs_by_store
ON docs (store) WHERE deleted = 0;
CREATE TABLE IF NOT EXISTS idx_entries (
store TEXT NOT NULL,
index_name TEXT NOT NULL,
id TEXT NOT NULL,
value ANY,
PRIMARY KEY (store, index_name, id)
);
CREATE INDEX IF NOT EXISTS idx_lookup
ON idx_entries (store, index_name, value);
CREATE INDEX IF NOT EXISTS idx_by_doc
ON idx_entries (store, id);
CREATE TABLE IF NOT EXISTS applied (
filename TEXT PRIMARY KEY,
applied_at INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS meta (
key TEXT PRIMARY KEY,
value TEXT
);
`);
}
// ── KV operations ──────────────────────────────────────────
async getKV(key: string): Promise<KVRecord | undefined> {
const rows = this.query(
'SELECT key, value, ts, rev FROM kv WHERE key = ?',
[key],
);
if (rows.length === 0) return undefined;
const r = rows[0];
return {
key: r.key as string,
value: JSON.parse(r.value as string),
ts: r.ts as number,
rev: r.rev as number,
};
}
async putKV(record: KVRecord): Promise<void> {
this.db.run(
'INSERT OR REPLACE INTO kv (key, value, ts, rev) VALUES (?,?,?,?)',
[record.key, JSON.stringify(record.value), record.ts, record.rev],
);
}
async deleteKV(key: string): Promise<void> {
this.db.run('DELETE FROM kv WHERE key = ?', [key]);
}
async getAllKVKeys(): Promise<string[]> {
return this.query('SELECT key FROM kv').map((r) => r.key as string);
}
async getAllKVEntries(): Promise<KVRecord[]> {
return this.query('SELECT key, value, ts, rev FROM kv').map((r) => ({
key: r.key as string,
value: JSON.parse(r.value as string),
ts: r.ts as number,
rev: r.rev as number,
}));
}
// ── Doc operations ─────────────────────────────────────────
async getDoc(store: string, id: string): Promise<DocRecord | undefined> {
const rows = this.query(
'SELECT store, id, data, ts, rev, deleted FROM docs WHERE store = ? AND id = ? AND deleted = 0',
[store, id],
);
if (rows.length === 0) return undefined;
return this.rowToDoc(rows[0]);
}
async getRawDoc(store: string, id: string): Promise<DocRecord | undefined> {
const rows = this.query(
'SELECT store, id, data, ts, rev, deleted FROM docs WHERE store = ? AND id = ?',
[store, id],
);
if (rows.length === 0) return undefined;
return this.rowToDoc(rows[0]);
}
async putDoc(record: DocRecord, indexes: IndexDefinition[]): Promise<void> {
this.db.run(
`INSERT OR REPLACE INTO docs (store, id, data, ts, rev, deleted)
VALUES (?,?,?,?,?,?)`,
[
record.store,
record.id,
JSON.stringify(record.data),
record.ts,
record.rev,
record.deleted ? 1 : 0,
],
);
// Update index entries
this.db.run(
'DELETE FROM idx_entries WHERE store = ? AND id = ?',
[record.store, record.id],
);
if (!record.deleted) {
for (const def of indexes) {
const val = extractIndexValue(record.data, def.fields as string[]);
if (val !== undefined) {
this.db.run(
`INSERT OR REPLACE INTO idx_entries (store, index_name, id, value)
VALUES (?,?,?,?)`,
[record.store, def.name, record.id, serializeIndexValue(val)],
);
}
}
}
}
async deleteDoc(
store: string,
id: string,
ts: number,
rev: number,
indexes: IndexDefinition[],
): Promise<void> {
return this.putDoc(
{ store, id, data: null, ts, rev, deleted: true },
indexes,
);
}
async getAllDocs(store: string): Promise<DocRecord[]> {
return this.query(
'SELECT store, id, data, ts, rev, deleted FROM docs WHERE store = ? AND deleted = 0',
[store],
).map((r) => this.rowToDoc(r));
}
// ── Index queries ──────────────────────────────────────────
async findByIndex(
store: string,
indexName: string,
value: unknown,
): Promise<string[]> {
return this.query(
'SELECT id FROM idx_entries WHERE store = ? AND index_name = ? AND value = ?',
[store, indexName, serializeIndexValue(value)],
).map((r) => r.id as string);
}
async queryByIndex(
store: string,
indexName: string,
range: { gt?: unknown; gte?: unknown; lt?: unknown; lte?: unknown },
): Promise<string[]> {
const conditions = ['store = ?', 'index_name = ?'];
const params: unknown[] = [store, indexName];
if (range.gte !== undefined) {
conditions.push('value >= ?');
params.push(serializeIndexValue(range.gte));
} else if (range.gt !== undefined) {
conditions.push('value > ?');
params.push(serializeIndexValue(range.gt));
}
if (range.lte !== undefined) {
conditions.push('value <= ?');
params.push(serializeIndexValue(range.lte));
} else if (range.lt !== undefined) {
conditions.push('value < ?');
params.push(serializeIndexValue(range.lt));
}
return this.query(
`SELECT id FROM idx_entries WHERE ${conditions.join(' AND ')}`,
params,
).map((r) => r.id as string);
}
async rebuildIndexes(
store: string,
indexes: IndexDefinition[],
): Promise<void> {
this.db.run('DELETE FROM idx_entries WHERE store = ?', [store]);
const docs = this.query(
'SELECT id, data FROM docs WHERE store = ? AND deleted = 0',
[store],
);
for (const row of docs) {
const data = JSON.parse(row.data as string);
for (const def of indexes) {
const val = extractIndexValue(data, def.fields as string[]);
if (val !== undefined) {
this.db.run(
`INSERT OR REPLACE INTO idx_entries (store, index_name, id, value)
VALUES (?,?,?,?)`,
[store, def.name, row.id, serializeIndexValue(val)],
);
}
}
}
}
// ── Applied-event tracking ─────────────────────────────────
async isEventApplied(filename: string): Promise<boolean> {
return (
this.query('SELECT 1 FROM applied WHERE filename = ?', [filename])
.length > 0
);
}
async markEventApplied(filename: string): Promise<void> {
this.db.run(
'INSERT OR IGNORE INTO applied (filename, applied_at) VALUES (?,?)',
[filename, Date.now()],
);
}
async getAppliedSet(): Promise<Set<string>> {
const rows = this.query('SELECT filename FROM applied');
return new Set(rows.map((r) => r.filename as string));
}
// ── Meta ───────────────────────────────────────────────────
async getMeta<T = unknown>(key: string): Promise<T | undefined> {
const rows = this.query('SELECT value FROM meta WHERE key = ?', [key]);
if (rows.length === 0) return undefined;
const raw = rows[0].value as string | null;
if (raw === null) return undefined;
return JSON.parse(raw) as T;
}
async setMeta(key: string, value: unknown): Promise<void> {
this.db.run('INSERT OR REPLACE INTO meta (key, value) VALUES (?,?)', [
key,
JSON.stringify(value),
]);
}
// ── Lifecycle ──────────────────────────────────────────────
close(): void {
this.db.close();
}
// ── Helpers ────────────────────────────────────────────────
private query(
sql: string,
params?: unknown[],
): Record<string, unknown>[] {
const results = this.db.exec(sql, params);
if (results.length === 0) return [];
const { columns, values } = results[0];
return values.map((row) => {
const obj: Record<string, unknown> = {};
for (let i = 0; i < columns.length; i++) {
obj[columns[i]] = row[i];
}
return obj;
});
}
private rowToDoc(row: Record<string, unknown>): DocRecord {
return {
store: row.store as string,
id: row.id as string,
data: JSON.parse(row.data as string),
ts: row.ts as number,
rev: row.rev as number,
deleted: (row.deleted as number) === 1,
};
}
}

53
sql-js/src/sqljs-types.d.ts vendored Normal file
View File

@ -0,0 +1,53 @@
/**
* Minimal type declarations for sql.js (v1.x).
* Covers the API surface used by this library.
*/
declare module 'sql.js' {
interface SqlJsStatic {
Database: new (data?: ArrayLike<number> | Buffer | null) => Database;
}
interface Database {
run(sql: string, params?: BindParams): Database;
exec(sql: string, params?: BindParams): QueryExecResult[];
prepare(sql: string): Statement;
getRowsModified(): number;
close(): void;
export(): Uint8Array;
}
interface Statement {
bind(params?: BindParams): boolean;
step(): boolean;
getAsObject(params?: BindParams): Record<string, unknown>;
get(params?: BindParams): unknown[];
free(): boolean;
reset(): void;
}
interface QueryExecResult {
columns: string[];
values: unknown[][];
}
type BindParams =
| Record<string, unknown>
| unknown[]
| null;
interface InitSqlJsOptions {
locateFile?: (filename: string) => string;
}
export default function initSqlJs(
options?: InitSqlJsOptions,
): Promise<SqlJsStatic>;
}
// asm.js subpath — same API, no .wasm file needed
declare module 'sql.js/dist/sql-asm.js' {
import type { SqlJsStatic, InitSqlJsOptions } from 'sql.js';
export default function initSqlJs(
options?: InitSqlJsOptions,
): Promise<SqlJsStatic>;
}

151
sql-js/src/sync-engine.ts Normal file
View File

@ -0,0 +1,151 @@
import type { SyncEvent, IndexDefinition, OpenOptions } from './types.js';
import type { SqlJsStore } from './sqljs-store.js';
import type { FolderStore } from './folder-store.js';
import type { Emitter } from './emitter.js';
import { canonicalJson, sha256Hex, eventFilename } from './utils.js';
export class SyncEngine {
private clientId: string;
private conflictResolver?: OpenOptions['conflictResolver'];
private collectionIndexes = new Map<string, IndexDefinition[]>();
private syncing = false;
constructor(
private store: SqlJsStore,
private folder: FolderStore,
private emitter: Emitter,
clientId: string,
conflictResolver?: OpenOptions['conflictResolver'],
) {
this.clientId = clientId;
this.conflictResolver = conflictResolver;
}
registerIndexes(store: string, indexes: IndexDefinition[]): void {
this.collectionIndexes.set(store, indexes);
}
getIndexes(store: string): IndexDefinition[] {
return this.collectionIndexes.get(store) ?? [];
}
async writeKV(key: string, value: unknown, ts: number, rev: number): Promise<void> {
await this.persistEvent({ type: 'put', store: 'kv', key, ts, clientId: this.clientId, data: value, rev });
}
async deleteKV(key: string, ts: number, rev: number): Promise<void> {
await this.persistEvent({ type: 'delete', store: 'kv', key, ts, clientId: this.clientId, rev });
}
async writeDoc(store: string, id: string, data: unknown, ts: number, rev: number): Promise<void> {
await this.persistEvent({ type: 'put', store, key: id, id, ts, clientId: this.clientId, data, rev });
}
async deleteDocEvent(store: string, id: string, ts: number, rev: number): Promise<void> {
await this.persistEvent({ type: 'delete', store, key: id, id, ts, clientId: this.clientId, rev });
}
private async persistEvent(event: SyncEvent): Promise<void> {
const canonical = canonicalJson(event);
const hash = await sha256Hex(canonical);
const filename = eventFilename(event.ts, hash);
await this.applyEvent(event);
await this.store.markEventApplied(filename);
if (this.folder.hasHandle && (await this.folder.hasPermission())) {
try {
await this.folder.writeEvent(filename, event);
} catch (err) {
this.emitter.emit('folder:lost-permission', err);
}
}
this.emitter.emit('change', {
type: event.type, store: event.store, key: event.key, id: event.id, data: event.data,
});
}
async sync(): Promise<void> {
if (this.syncing) return;
if (!this.folder.hasHandle) return;
if (!(await this.folder.hasPermission())) {
this.emitter.emit('folder:lost-permission');
return;
}
this.syncing = true;
this.emitter.emit('sync:start');
try {
const appliedSet = await this.store.getAppliedSet();
const filenames = await this.folder.scanEventFilenames();
let importCount = 0;
for (const name of filenames) {
if (appliedSet.has(name)) continue;
const event = await this.folder.readEventFile(name);
if (!event) continue;
const hadConflict = await this.applyEventWithConflictCheck(event);
await this.store.markEventApplied(name);
importCount++;
if (hadConflict) this.emitter.emit('conflict', { filename: name, event });
this.emitter.emit('change', {
type: event.type, store: event.store, key: event.key, id: event.id, data: event.data, source: 'sync',
});
}
this.emitter.emit('sync:end', { imported: importCount });
} catch (err) {
this.emitter.emit('sync:end', { error: err });
} finally {
this.syncing = false;
}
}
private async applyEvent(event: SyncEvent): Promise<void> {
if (event.store === 'kv') {
if (event.type === 'put') {
await this.store.putKV({ key: event.key, value: event.data, ts: event.ts, rev: event.rev ?? 0 });
} else {
await this.store.deleteKV(event.key);
}
} else {
const indexes = this.getIndexes(event.store);
if (event.type === 'put') {
await this.store.putDoc({ store: event.store, id: event.id ?? event.key, data: event.data, ts: event.ts, rev: event.rev ?? 0 }, indexes);
} else {
await this.store.deleteDoc(event.store, event.id ?? event.key, event.ts, event.rev ?? 0, indexes);
}
}
}
private async applyEventWithConflictCheck(event: SyncEvent): Promise<boolean> {
let hadConflict = false;
if (event.store === 'kv') {
const existing = await this.store.getKV(event.key);
if (existing) {
if (event.ts > existing.ts) { hadConflict = true; }
else if (event.ts === existing.ts) {
hadConflict = true;
if (this.conflictResolver) event = { ...event, data: this.conflictResolver(existing.value, event.data) };
} else { return true; }
}
} else {
const existing = await this.store.getRawDoc(event.store, event.id ?? event.key);
if (existing && !existing.deleted) {
if (event.ts > existing.ts) { hadConflict = true; }
else if (event.ts === existing.ts) {
hadConflict = true;
if (this.conflictResolver) event = { ...event, data: this.conflictResolver(existing.data, event.data) };
} else { return true; }
}
}
await this.applyEvent(event);
return hadConflict;
}
}

95
sql-js/src/types.ts Normal file
View File

@ -0,0 +1,95 @@
// ── Event types ──────────────────────────────────────────────
export type EventType = 'put' | 'delete';
export interface SyncEvent {
type: EventType;
store: string; // "kv" | collection name
key: string;
id?: string; // present for collection documents
ts: number; // millisecond Unix epoch
clientId: string;
data?: unknown; // absent on deletes
rev?: number;
}
// ── Index / collection options ───────────────────────────────
export interface IndexDefinition<T = unknown> {
name: string;
fields: (keyof T & string)[] | string[];
}
export interface StoreOptions<T = unknown> {
name: string;
indexes?: IndexDefinition<T>[];
}
export interface OpenOptions {
dbName?: string;
autoSyncIntervalMs?: number;
clientId?: string;
conflictResolver?: (current: unknown, incoming: unknown) => unknown;
/** Passed to sqlite3InitModule(). Use locateFile to override .wasm loading. */
sqlite3Config?: Record<string, unknown>;
}
// ── Store record shapes ──────────────────────────────────────
export interface KVRecord {
key: string;
value: unknown;
ts: number;
rev: number;
}
export interface DocRecord {
store: string;
id: string;
data: unknown;
ts: number;
rev: number;
deleted?: boolean;
}
// ── Index range (replaces IDBKeyRange for SQLite) ────────────
export interface IndexRange {
gt?: unknown;
gte?: unknown;
lt?: unknown;
lte?: unknown;
}
// ── Event emitter ────────────────────────────────────────────
export type SyncDBEventName =
| 'sync:start'
| 'sync:end'
| 'change'
| 'conflict'
| 'folder:lost-permission';
export type SyncDBEventHandler = (...args: unknown[]) => void;
// ── KV API surface ───────────────────────────────────────────
export interface KVApi {
get<T = unknown>(key: string): Promise<T | undefined>;
set<T = unknown>(key: string, value: T): Promise<void>;
delete(key: string): Promise<void>;
has(key: string): Promise<boolean>;
keys(): Promise<string[]>;
entries<T = unknown>(): Promise<Array<[string, T]>>;
}
// ── Collection API surface ───────────────────────────────────
export interface CollectionApi<T extends { id: string }> {
get(id: string): Promise<T | undefined>;
put(doc: T): Promise<void>;
delete(id: string): Promise<void>;
all(): Promise<T[]>;
findByIndex(indexName: string, value: unknown): Promise<T[]>;
queryByIndex(indexName: string, range: IndexRange): Promise<T[]>;
}

94
sql-js/src/utils.ts Normal file
View File

@ -0,0 +1,94 @@
// ── Canonical JSON ───────────────────────────────────────────
function sortKeys(obj: unknown): unknown {
if (obj === null || obj === undefined || typeof obj !== 'object') return obj;
if (Array.isArray(obj)) return obj.map(sortKeys);
const sorted: Record<string, unknown> = {};
for (const key of Object.keys(obj as Record<string, unknown>).sort()) {
sorted[key] = sortKeys((obj as Record<string, unknown>)[key]);
}
return sorted;
}
export function canonicalJson(obj: unknown): string {
return JSON.stringify(sortKeys(obj));
}
// ── Hashing ──────────────────────────────────────────────────
export async function sha256Hex(data: string): Promise<string> {
const buf = await crypto.subtle.digest(
'SHA-256',
new TextEncoder().encode(data),
);
return Array.from(new Uint8Array(buf))
.map((b) => b.toString(16).padStart(2, '0'))
.join('');
}
// ── Event filenames ──────────────────────────────────────────
const HASH_PREFIX_LEN = 12;
export function eventFilename(ts: number, fullHash: string): string {
return `${ts}_${fullHash.slice(0, HASH_PREFIX_LEN)}.json`;
}
const EVENT_FILE_RE = /^(\d+)_([0-9a-f]+)\.json$/;
export function parseEventFilename(
name: string,
): { ts: number; hash: string } | null {
const m = EVENT_FILE_RE.exec(name);
if (!m) return null;
return { ts: Number(m[1]), hash: m[2] };
}
// ── Client ID ────────────────────────────────────────────────
export function generateClientId(): string {
return crypto.randomUUID();
}
// ── Index value extraction ───────────────────────────────────
export function extractIndexValue(
data: unknown,
fields: string[],
): IDBValidKey | undefined {
if (!data || typeof data !== 'object') return undefined;
const rec = data as Record<string, unknown>;
if (fields.length === 1) {
const v = getNestedValue(rec, fields[0]);
if (v === undefined || v === null) return undefined;
return toIDBKey(v);
}
const parts: IDBValidKey[] = [];
for (const f of fields) {
const v = getNestedValue(rec, f);
if (v === undefined || v === null) return undefined;
parts.push(toIDBKey(v));
}
return parts;
}
function getNestedValue(obj: Record<string, unknown>, path: string): unknown {
let cur: unknown = obj;
for (const seg of path.split('.')) {
if (cur === null || cur === undefined || typeof cur !== 'object') {
return undefined;
}
cur = (cur as Record<string, unknown>)[seg];
}
return cur;
}
function toIDBKey(v: unknown): IDBValidKey {
if (typeof v === 'string' || typeof v === 'number') return v;
if (v instanceof Date) return v;
if (Array.isArray(v)) return v.map(toIDBKey);
// Fallback: stringify
return String(v);
}

23
sql-js/tsconfig.json Normal file
View File

@ -0,0 +1,23 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "ES2022",
"moduleResolution": "bundler",
"lib": ["ES2022", "DOM", "DOM.Iterable"],
"outDir": "dist",
"rootDir": "src",
"declaration": true,
"declarationMap": true,
"sourceMap": true,
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noFallthroughCasesInSwitch": true,
"forceConsistentCasingInFileNames": true,
"esModuleInterop": true,
"allowSyntheticDefaultImports": true,
"skipLibCheck": true
},
"include": ["src"],
"exclude": ["node_modules", "dist"]
}

100
sqlite/README.md Normal file
View File

@ -0,0 +1,100 @@
# IndexSyncFile -- SQLite WASM Variant
Local-first key-value and document store using the **official SQLite WASM** build (`@sqlite.org/sqlite-wasm`) as the local cache, syncing with a user-selected folder via the File System Access API.
> **Note:** This variant requires an HTTP server. It cannot run from `file://` because the browser must `fetch()` the `.wasm` binary at runtime. If you need SQLite without a server, use the [`sql-js/`](../sql-js/) variant instead.
## Why SQLite WASM
- **Fastest SQLite in browser** -- native WebAssembly, ~2-3x faster than the asm.js alternative
- **Real SQL** -- full query power with proper indexes, joins, and ACID transactions
- **OPFS persistence** -- via `opfs-sahpool` VFS, the database survives page reloads without extra code
- **Atomic transactions** -- document + index updates in a single SQLite `transaction()`, no partial writes
- **Correct numeric ordering** -- range queries on numbers work properly (unlike JSON-serialized keys)
## Architecture
```
Write: app ---> SQLite in-memory/OPFS (immediate) ---> /events/timestamp_hash.json
Sync: /events/*.json ---> sort by timestamp ---> skip applied ---> apply to SQLite
```
SQLite is the fast query layer. The folder's event log is the sync mechanism.
### Persistence cascade
The library tries three SQLite VFS backends in order:
1. **opfs-sahpool** (best) -- no COOP/COEP headers needed, main-thread compatible
2. **OpfsDb** -- requires a Web Worker and COOP/COEP HTTP headers
3. **In-memory** -- fallback if OPFS is unavailable; folder sync rebuilds state on each page load
Check `db.isPersistent` after opening to see which mode was selected.
### HTTP headers (may be required)
Some OPFS modes require these response headers on the HTML page:
```
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp
```
The `opfs-sahpool` VFS works without these headers in most cases. The library tries it first.
### SQL schema (5 tables)
```sql
kv (key TEXT PK, value TEXT, ts INTEGER, rev INTEGER)
docs (store TEXT, id TEXT, data TEXT, ts INTEGER, rev INTEGER, deleted INTEGER, PK(store,id))
idx_entries (store TEXT, index_name TEXT, id TEXT, value ANY, PK(store,index_name,id))
applied (filename TEXT PK, applied_at INTEGER)
meta (key TEXT PK, value TEXT)
```
## Install and build
```bash
npm install
npm run build
```
**Dependency:** `@sqlite.org/sqlite-wasm` -- the official SQLite WASM build from the SQLite project.
## Usage
```ts
import { FolderSyncDB } from './dist/index.js';
const db = await FolderSyncDB.open({
dbName: 'MyApp',
autoSyncIntervalMs: 5000,
});
console.log('OPFS-backed:', db.isPersistent);
await db.selectFolder();
// KV
await db.kv.set('config', { theme: 'dark', lang: 'en' });
// Collections
const tasks = db.collection({
name: 'tasks',
indexes: [{ name: 'byPriority', fields: ['priority'] }],
});
await tasks.put({ id: 't1', title: 'Deploy', priority: 1 });
// Range queries with proper numeric ordering
const urgent = await tasks.queryByIndex('byPriority', { gte: 1, lte: 3 });
```
## Variant-specific notes
- **Cannot run from `file://`** -- the browser must fetch `sqlite3.wasm` over HTTP
- Directory handle is stored in a tiny IDB sidecar (SQLite can't store DOM objects via structured clone)
- All document data is JSON-serialized in TEXT columns
- Index values are stored with SQLite type affinity -- numbers compare as numbers
- The `@sqlite.org/sqlite-wasm` package uses pre-release version tags (e.g., `3.51.2-build8`)
- Consumer's bundler must serve the `.wasm` files from the package
- For a SQLite option that works from `file://`, see the [`sql-js/`](../sql-js/) variant

29
sqlite/package.json Normal file
View File

@ -0,0 +1,29 @@
{
"name": "index-sync-file-sqlite",
"version": "1.0.0",
"description": "Local-first key-value and document store backed by SQLite WASM, syncing with a user-selected folder via File System Access API",
"type": "module",
"main": "dist/index.js",
"module": "dist/index.js",
"types": "dist/index.d.ts",
"exports": {
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.js"
}
},
"files": ["dist"],
"scripts": {
"build": "tsc",
"dev": "tsc --watch",
"clean": "rm -rf dist"
},
"keywords": ["local-first", "sqlite", "wasm", "opfs", "file-system-access-api", "key-value", "document-store", "sync"],
"license": "MIT",
"dependencies": {
"@sqlite.org/sqlite-wasm": "3.51.2-build8"
},
"devDependencies": {
"typescript": "^5.7.0"
}
}

106
sqlite/src/collection.ts Normal file
View File

@ -0,0 +1,106 @@
import type {
CollectionApi,
IndexDefinition,
IndexRange,
StoreOptions,
} from './types.js';
import type { SQLiteStore } from './sqlite-store.js';
import type { SyncEngine } from './sync-engine.js';
export class Collection<T extends { id: string }> implements CollectionApi<T> {
private readonly storeName: string;
private readonly indexes: IndexDefinition<T>[];
private indexesBuilt = false;
constructor(
options: StoreOptions<T>,
private store: SQLiteStore,
private sync: SyncEngine,
) {
this.storeName = options.name;
this.indexes = options.indexes ?? [];
this.sync.registerIndexes(
this.storeName,
this.indexes as IndexDefinition[],
);
}
private async ensureIndexes(): Promise<void> {
if (this.indexesBuilt || this.indexes.length === 0) return;
await this.store.rebuildIndexes(
this.storeName,
this.indexes as IndexDefinition[],
);
this.indexesBuilt = true;
}
async get(id: string): Promise<T | undefined> {
const rec = await this.store.getDoc(this.storeName, id);
if (!rec) return undefined;
return rec.data as T;
}
async put(doc: T): Promise<void> {
await this.ensureIndexes();
const existing = await this.store.getRawDoc(this.storeName, doc.id);
const rev = (existing?.rev ?? 0) + 1;
const ts = Date.now();
await this.sync.writeDoc(this.storeName, doc.id, doc, ts, rev);
}
async delete(id: string): Promise<void> {
await this.ensureIndexes();
const existing = await this.store.getRawDoc(this.storeName, id);
if (!existing || existing.deleted) return;
const rev = existing.rev + 1;
const ts = Date.now();
await this.sync.deleteDocEvent(this.storeName, id, ts, rev);
}
async all(): Promise<T[]> {
const docs = await this.store.getAllDocs(this.storeName);
return docs.map((d) => d.data as T);
}
async findByIndex(indexName: string, value: unknown): Promise<T[]> {
await this.ensureIndexes();
this.assertIndex(indexName);
const ids = await this.store.findByIndex(
this.storeName,
indexName,
value,
);
return this.fetchByIds(ids);
}
async queryByIndex(indexName: string, range: IndexRange): Promise<T[]> {
await this.ensureIndexes();
this.assertIndex(indexName);
const ids = await this.store.queryByIndex(
this.storeName,
indexName,
range,
);
return this.fetchByIds(ids);
}
// ── Helpers ────────────────────────────────────────────────
private assertIndex(name: string): void {
if (!this.indexes.some((i) => i.name === name)) {
throw new Error(
`Index "${name}" is not defined on collection "${this.storeName}". ` +
`Defined indexes: ${this.indexes.map((i) => i.name).join(', ') || '(none)'}`,
);
}
}
private async fetchByIds(ids: string[]): Promise<T[]> {
const results: T[] = [];
for (const id of ids) {
const doc = await this.store.getDoc(this.storeName, id);
if (doc) results.push(doc.data as T);
}
return results;
}
}

33
sqlite/src/emitter.ts Normal file
View File

@ -0,0 +1,33 @@
import type { SyncDBEventName, SyncDBEventHandler } from './types.js';
export class Emitter {
private listeners = new Map<SyncDBEventName, Set<SyncDBEventHandler>>();
on(event: SyncDBEventName, handler: SyncDBEventHandler): () => void {
let set = this.listeners.get(event);
if (!set) {
set = new Set();
this.listeners.set(event, set);
}
set.add(handler);
return () => {
set!.delete(handler);
};
}
emit(event: SyncDBEventName, ...args: unknown[]): void {
const set = this.listeners.get(event);
if (!set) return;
for (const fn of set) {
try {
fn(...args);
} catch {
// listener errors must not break the library
}
}
}
removeAll(): void {
this.listeners.clear();
}
}

122
sqlite/src/folder-store.ts Normal file
View File

@ -0,0 +1,122 @@
import type { SyncEvent } from './types.js';
import { parseEventFilename } from './utils.js';
const EVENTS_DIR = 'events';
export class FolderStore {
private dirHandle: FileSystemDirectoryHandle | null = null;
// ── Handle management ──────────────────────────────────────
get hasHandle(): boolean {
return this.dirHandle !== null;
}
getHandle(): FileSystemDirectoryHandle | null {
return this.dirHandle;
}
setHandle(handle: FileSystemDirectoryHandle): void {
this.dirHandle = handle;
}
async selectFolder(): Promise<FileSystemDirectoryHandle> {
if (typeof window.showDirectoryPicker !== 'function') {
throw new Error(
'File System Access API is not supported in this browser. ' +
'Use a Chromium-based browser (Chrome, Edge, Brave, etc.).',
);
}
this.dirHandle = await window.showDirectoryPicker({
mode: 'readwrite',
});
// Ensure events directory exists
await this.dirHandle.getDirectoryHandle(EVENTS_DIR, { create: true });
return this.dirHandle;
}
// ── Permission checks ─────────────────────────────────────
async hasPermission(): Promise<boolean> {
if (!this.dirHandle) return false;
try {
const perm = await this.dirHandle.queryPermission({ mode: 'readwrite' });
return perm === 'granted';
} catch {
return false;
}
}
async requestPermission(): Promise<boolean> {
if (!this.dirHandle) return false;
try {
const perm = await this.dirHandle.requestPermission({
mode: 'readwrite',
});
return perm === 'granted';
} catch {
return false;
}
}
// ── Write event file ──────────────────────────────────────
async writeEvent(filename: string, event: SyncEvent): Promise<void> {
const eventsDir = await this.getEventsDir();
const fileHandle = await eventsDir.getFileHandle(filename, {
create: true,
});
const writable = await fileHandle.createWritable();
try {
await writable.write(JSON.stringify(event, null, 2));
} finally {
await writable.close();
}
}
// ── Scan events ────────────────────────────────────────────
async scanEventFilenames(): Promise<string[]> {
const eventsDir = await this.getEventsDir();
const names: string[] = [];
for await (const [name, handle] of eventsDir.entries()) {
if (handle.kind === 'file' && name.endsWith('.json')) {
if (parseEventFilename(name)) {
names.push(name);
}
}
}
// Sort by (timestamp, full filename) for deterministic ordering
names.sort((a, b) => {
const pa = parseEventFilename(a)!;
const pb = parseEventFilename(b)!;
if (pa.ts !== pb.ts) return pa.ts - pb.ts;
return a.localeCompare(b);
});
return names;
}
async readEventFile(filename: string): Promise<SyncEvent | null> {
const eventsDir = await this.getEventsDir();
try {
const fh = await eventsDir.getFileHandle(filename);
const file = await fh.getFile();
const text = await file.text();
return JSON.parse(text) as SyncEvent;
} catch {
return null;
}
}
// ── Internals ──────────────────────────────────────────────
private async getEventsDir(): Promise<FileSystemDirectoryHandle> {
if (!this.dirHandle) {
throw new Error('No folder selected. Call selectFolder() first.');
}
return this.dirHandle.getDirectoryHandle(EVENTS_DIR, { create: true });
}
}

View File

@ -0,0 +1,161 @@
import type {
OpenOptions,
StoreOptions,
CollectionApi,
KVApi,
SyncDBEventName,
SyncDBEventHandler,
} from './types.js';
import { generateClientId } from './utils.js';
import { Emitter } from './emitter.js';
import { SQLiteStore } from './sqlite-store.js';
import { FolderStore } from './folder-store.js';
import { SyncEngine } from './sync-engine.js';
import { KVStore } from './kv-store.js';
import { Collection } from './collection.js';
import { storeHandle, loadHandle } from './handle-store.js';
const META_CLIENT_ID = 'clientId';
const HANDLE_KEY = 'dirHandle';
export class FolderSyncDB {
private store!: SQLiteStore;
private folderStore!: FolderStore;
private syncEngine!: SyncEngine;
private emitter!: Emitter;
private autoSyncTimer: ReturnType<typeof setInterval> | null = null;
private collections = new Map<string, Collection<any>>();
/** Public KV API — available immediately after open(). */
kv!: KVApi;
/**
* Whether the underlying SQLite database is backed by OPFS.
* If false, data lives only in memory and the folder is the
* sole durable store (sync rebuilds state on each page load).
*/
get isPersistent(): boolean {
return this.store.isPersistent;
}
// ── Construction (use static open()) ───────────────────────
private constructor() {}
static async open(options?: OpenOptions): Promise<FolderSyncDB> {
const db = new FolderSyncDB();
await db.init(options ?? {});
return db;
}
private async init(opts: OpenOptions): Promise<void> {
const dbName = opts.dbName ?? 'FolderSyncDB';
this.emitter = new Emitter();
this.store = await SQLiteStore.open(dbName, opts.sqlite3Config);
this.folderStore = new FolderStore();
// Client ID: use provided, or load persisted, or generate new
let clientId = opts.clientId;
if (!clientId) {
clientId = await this.store.getMeta<string>(META_CLIENT_ID);
if (!clientId) {
clientId = generateClientId();
await this.store.setMeta(META_CLIENT_ID, clientId);
}
} else {
await this.store.setMeta(META_CLIENT_ID, clientId);
}
this.syncEngine = new SyncEngine(
this.store,
this.folderStore,
this.emitter,
clientId,
opts.conflictResolver,
);
this.kv = new KVStore(this.store, this.syncEngine);
// Restore persisted directory handle (stored in IDB sidecar)
await this.tryRestoreHandle();
// Auto-sync
if (opts.autoSyncIntervalMs && opts.autoSyncIntervalMs > 0) {
this.autoSyncTimer = setInterval(() => {
this.sync().catch(() => {});
}, opts.autoSyncIntervalMs);
}
}
// ── Folder management ──────────────────────────────────────
async selectFolder(): Promise<void> {
const handle = await this.folderStore.selectFolder();
// Persist handle in IDB sidecar (SQLite can't store DOM objects)
await storeHandle(HANDLE_KEY, handle);
// Run initial sync to import existing events
await this.sync();
}
async hasFolderAccess(): Promise<boolean> {
return this.folderStore.hasPermission();
}
async requestFolderAccess(): Promise<boolean> {
const granted = await this.folderStore.requestPermission();
if (granted) {
await this.sync();
}
return granted;
}
// ── Sync ───────────────────────────────────────────────────
async sync(): Promise<void> {
return this.syncEngine.sync();
}
// ── Collections ────────────────────────────────────────────
collection<T extends { id: string }>(
options: StoreOptions<T>,
): CollectionApi<T> {
const cached = this.collections.get(options.name);
if (cached) return cached as Collection<T>;
const col = new Collection<T>(options, this.store, this.syncEngine);
this.collections.set(options.name, col);
return col;
}
// ── Events ─────────────────────────────────────────────────
on(event: SyncDBEventName, handler: SyncDBEventHandler): () => void {
return this.emitter.on(event, handler);
}
// ── Lifecycle ──────────────────────────────────────────────
async close(): Promise<void> {
if (this.autoSyncTimer !== null) {
clearInterval(this.autoSyncTimer);
this.autoSyncTimer = null;
}
this.emitter.removeAll();
this.collections.clear();
this.store.close();
}
// ── Internals ──────────────────────────────────────────────
private async tryRestoreHandle(): Promise<void> {
try {
const handle = await loadHandle(HANDLE_KEY);
if (!handle) return;
if (typeof handle.queryPermission !== 'function') return;
this.folderStore.setHandle(handle);
} catch {
// Handle was not restorable — user must re-select
}
}
}

88
sqlite/src/fs-access.d.ts vendored Normal file
View File

@ -0,0 +1,88 @@
/**
* Type declarations for the File System Access API.
* These APIs are available in Chromium-based browsers only.
* @see https://developer.mozilla.org/en-US/docs/Web/API/File_System_Access_API
*/
interface FileSystemHandlePermissionDescriptor {
mode?: 'read' | 'readwrite';
}
interface FileSystemDirectoryHandle {
readonly kind: 'directory';
readonly name: string;
getDirectoryHandle(
name: string,
options?: { create?: boolean },
): Promise<FileSystemDirectoryHandle>;
getFileHandle(
name: string,
options?: { create?: boolean },
): Promise<FileSystemFileHandle>;
entries(): AsyncIterableIterator<
[string, FileSystemDirectoryHandle | FileSystemFileHandle]
>;
values(): AsyncIterableIterator<
FileSystemDirectoryHandle | FileSystemFileHandle
>;
keys(): AsyncIterableIterator<string>;
queryPermission(
descriptor?: FileSystemHandlePermissionDescriptor,
): Promise<PermissionState>;
requestPermission(
descriptor?: FileSystemHandlePermissionDescriptor,
): Promise<PermissionState>;
}
interface FileSystemFileHandle {
readonly kind: 'file';
readonly name: string;
getFile(): Promise<File>;
createWritable(
options?: FileSystemCreateWritableOptions,
): Promise<FileSystemWritableFileStream>;
}
interface FileSystemCreateWritableOptions {
keepExistingData?: boolean;
}
interface FileSystemWritableFileStream extends WritableStream {
write(data: BufferSource | Blob | string | WriteParams): Promise<void>;
seek(position: number): Promise<void>;
truncate(size: number): Promise<void>;
close(): Promise<void>;
}
interface WriteParams {
type: 'write' | 'seek' | 'truncate';
data?: BufferSource | Blob | string;
position?: number;
size?: number;
}
interface ShowDirectoryPickerOptions {
id?: string;
mode?: 'read' | 'readwrite';
startIn?:
| FileSystemDirectoryHandle
| 'desktop'
| 'documents'
| 'downloads'
| 'music'
| 'pictures'
| 'videos';
}
interface Window {
showDirectoryPicker(
options?: ShowDirectoryPickerOptions,
): Promise<FileSystemDirectoryHandle>;
}

View File

@ -0,0 +1,56 @@
/**
* Tiny IndexedDB sidecar for persisting FileSystemDirectoryHandle.
*
* SQLite can't store DOM objects (they require structured clone),
* so we use a minimal IDB store exclusively for the folder handle.
*/
const DB_NAME = 'FolderSyncDB_handles';
const STORE_NAME = 'handles';
const DB_VERSION = 1;
function openHandleDB(): Promise<IDBDatabase> {
return new Promise((resolve, reject) => {
const req = indexedDB.open(DB_NAME, DB_VERSION);
req.onupgradeneeded = () => {
req.result.createObjectStore(STORE_NAME, { keyPath: 'key' });
};
req.onsuccess = () => resolve(req.result);
req.onerror = () => reject(req.error);
});
}
export async function storeHandle(
key: string,
handle: FileSystemDirectoryHandle,
): Promise<void> {
const db = await openHandleDB();
try {
const tx = db.transaction(STORE_NAME, 'readwrite');
tx.objectStore(STORE_NAME).put({ key, handle });
await new Promise<void>((resolve, reject) => {
tx.oncomplete = () => resolve();
tx.onerror = () => reject(tx.error);
});
} finally {
db.close();
}
}
export async function loadHandle(
key: string,
): Promise<FileSystemDirectoryHandle | null> {
const db = await openHandleDB();
try {
const tx = db.transaction(STORE_NAME, 'readonly');
const req = tx.objectStore(STORE_NAME).get(key);
return await new Promise<FileSystemDirectoryHandle | null>(
(resolve, reject) => {
req.onsuccess = () => resolve(req.result?.handle ?? null);
req.onerror = () => reject(req.error);
},
);
} finally {
db.close();
}
}

13
sqlite/src/index.ts Normal file
View File

@ -0,0 +1,13 @@
export { FolderSyncDB } from './folder-sync-db.js';
export type {
OpenOptions,
StoreOptions,
IndexDefinition,
IndexRange,
SyncEvent,
KVApi,
CollectionApi,
SyncDBEventName,
SyncDBEventHandler,
} from './types.js';

45
sqlite/src/kv-store.ts Normal file
View File

@ -0,0 +1,45 @@
import type { KVApi } from './types.js';
import type { SQLiteStore } from './sqlite-store.js';
import type { SyncEngine } from './sync-engine.js';
export class KVStore implements KVApi {
constructor(
private store: SQLiteStore,
private sync: SyncEngine,
) {}
async get<T = unknown>(key: string): Promise<T | undefined> {
const rec = await this.store.getKV(key);
return rec?.value as T | undefined;
}
async set<T = unknown>(key: string, value: T): Promise<void> {
const existing = await this.store.getKV(key);
const rev = (existing?.rev ?? 0) + 1;
const ts = Date.now();
// SyncEngine.persistEvent handles the SQLite write + folder write
await this.sync.writeKV(key, value, ts, rev);
}
async delete(key: string): Promise<void> {
const existing = await this.store.getKV(key);
if (!existing) return;
const rev = existing.rev + 1;
const ts = Date.now();
await this.sync.deleteKV(key, ts, rev);
}
async has(key: string): Promise<boolean> {
const rec = await this.store.getKV(key);
return rec !== undefined;
}
async keys(): Promise<string[]> {
return this.store.getAllKVKeys();
}
async entries<T = unknown>(): Promise<Array<[string, T]>> {
const records = await this.store.getAllKVEntries();
return records.map((r) => [r.key, r.value as T]);
}
}

389
sqlite/src/sqlite-store.ts Normal file
View File

@ -0,0 +1,389 @@
import sqlite3InitModule from '@sqlite.org/sqlite-wasm';
import type { OO1Database } from '@sqlite.org/sqlite-wasm';
import type { KVRecord, DocRecord, IndexDefinition } from './types.js';
import { extractIndexValue } from './utils.js';
// ── Value serialization for index column ─────────────────────
function serializeIndexValue(v: unknown): unknown {
if (typeof v === 'number' || typeof v === 'string') return v;
if (v instanceof Date) return v.getTime();
if (Array.isArray(v)) return JSON.stringify(v);
return String(v);
}
// ── SQLiteStore ──────────────────────────────────────────────
export class SQLiteStore {
private constructor(
private db: OO1Database,
private persistent: boolean,
) {}
/** Whether the database is backed by OPFS (survives page reload). */
get isPersistent(): boolean {
return this.persistent;
}
// ── Open ───────────────────────────────────────────────────
static async open(
name: string,
sqlite3Config?: Record<string, unknown>,
): Promise<SQLiteStore> {
const sqlite3 = await sqlite3InitModule(sqlite3Config);
let db: OO1Database | null = null;
let persistent = false;
// 1. Try opfs-sahpool (best: no COOP/COEP headers, main-thread OK)
if (sqlite3.installOpfsSAHPoolVfs) {
try {
const pool = await sqlite3.installOpfsSAHPoolVfs({
initialCapacity: 4,
});
db = new pool.OpfsSAHPoolDb(`/${name}.sqlite3`);
persistent = true;
} catch {
// OPFS not available in this context
}
}
// 2. Try OpfsDb (requires worker + COOP/COEP headers)
if (!db && sqlite3.oo1.OpfsDb) {
try {
db = new sqlite3.oo1.OpfsDb(`/${name}.sqlite3`, 'c');
persistent = true;
} catch {
// Not in a Worker or headers missing
}
}
// 3. Fall back to in-memory (folder sync rebuilds state on reload)
if (!db) {
db = new sqlite3.oo1.DB(':memory:', 'c');
}
const store = new SQLiteStore(db, persistent);
store.createSchema();
return store;
}
private createSchema(): void {
this.db.exec(`
CREATE TABLE IF NOT EXISTS kv (
key TEXT PRIMARY KEY,
value TEXT NOT NULL,
ts INTEGER NOT NULL,
rev INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS docs (
store TEXT NOT NULL,
id TEXT NOT NULL,
data TEXT NOT NULL,
ts INTEGER NOT NULL,
rev INTEGER NOT NULL,
deleted INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (store, id)
);
CREATE INDEX IF NOT EXISTS docs_by_store
ON docs (store) WHERE deleted = 0;
CREATE TABLE IF NOT EXISTS idx_entries (
store TEXT NOT NULL,
index_name TEXT NOT NULL,
id TEXT NOT NULL,
value ANY,
PRIMARY KEY (store, index_name, id)
);
CREATE INDEX IF NOT EXISTS idx_lookup
ON idx_entries (store, index_name, value);
CREATE INDEX IF NOT EXISTS idx_by_doc
ON idx_entries (store, id);
CREATE TABLE IF NOT EXISTS applied (
filename TEXT PRIMARY KEY,
applied_at INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS meta (
key TEXT PRIMARY KEY,
value TEXT
);
`);
}
// ── KV operations ──────────────────────────────────────────
async getKV(key: string): Promise<KVRecord | undefined> {
const rows = this.db.selectObjects(
'SELECT key, value, ts, rev FROM kv WHERE key = ?',
[key],
);
if (rows.length === 0) return undefined;
const r = rows[0];
return {
key: r.key as string,
value: JSON.parse(r.value as string),
ts: r.ts as number,
rev: r.rev as number,
};
}
async putKV(record: KVRecord): Promise<void> {
this.db.exec({
sql: 'INSERT OR REPLACE INTO kv (key, value, ts, rev) VALUES (?,?,?,?)',
bind: [record.key, JSON.stringify(record.value), record.ts, record.rev],
});
}
async deleteKV(key: string): Promise<void> {
this.db.exec({ sql: 'DELETE FROM kv WHERE key = ?', bind: [key] });
}
async getAllKVKeys(): Promise<string[]> {
return this.db.selectValues('SELECT key FROM kv') as string[];
}
async getAllKVEntries(): Promise<KVRecord[]> {
return this.db
.selectObjects('SELECT key, value, ts, rev FROM kv')
.map((r) => ({
key: r.key as string,
value: JSON.parse(r.value as string),
ts: r.ts as number,
rev: r.rev as number,
}));
}
// ── Doc operations ─────────────────────────────────────────
async getDoc(store: string, id: string): Promise<DocRecord | undefined> {
const rows = this.db.selectObjects(
'SELECT store, id, data, ts, rev, deleted FROM docs WHERE store = ? AND id = ? AND deleted = 0',
[store, id],
);
if (rows.length === 0) return undefined;
return this.rowToDoc(rows[0]);
}
async getRawDoc(store: string, id: string): Promise<DocRecord | undefined> {
const rows = this.db.selectObjects(
'SELECT store, id, data, ts, rev, deleted FROM docs WHERE store = ? AND id = ?',
[store, id],
);
if (rows.length === 0) return undefined;
return this.rowToDoc(rows[0]);
}
async putDoc(record: DocRecord, indexes: IndexDefinition[]): Promise<void> {
this.db.transaction(() => {
// Upsert document
this.db.exec({
sql: `INSERT OR REPLACE INTO docs (store, id, data, ts, rev, deleted)
VALUES (?,?,?,?,?,?)`,
bind: [
record.store,
record.id,
JSON.stringify(record.data),
record.ts,
record.rev,
record.deleted ? 1 : 0,
],
});
// Remove old index entries for this doc
this.db.exec({
sql: 'DELETE FROM idx_entries WHERE store = ? AND id = ?',
bind: [record.store, record.id],
});
// Insert new index entries (only for non-deleted docs)
if (!record.deleted) {
for (const def of indexes) {
const val = extractIndexValue(record.data, def.fields as string[]);
if (val !== undefined) {
this.db.exec({
sql: `INSERT OR REPLACE INTO idx_entries (store, index_name, id, value)
VALUES (?,?,?,?)`,
bind: [
record.store,
def.name,
record.id,
serializeIndexValue(val),
],
});
}
}
}
});
}
async deleteDoc(
store: string,
id: string,
ts: number,
rev: number,
indexes: IndexDefinition[],
): Promise<void> {
return this.putDoc(
{ store, id, data: null, ts, rev, deleted: true },
indexes,
);
}
async getAllDocs(store: string): Promise<DocRecord[]> {
return this.db
.selectObjects(
'SELECT store, id, data, ts, rev, deleted FROM docs WHERE store = ? AND deleted = 0',
[store],
)
.map(this.rowToDoc);
}
// ── Index queries ──────────────────────────────────────────
async findByIndex(
store: string,
indexName: string,
value: unknown,
): Promise<string[]> {
return this.db.selectValues(
'SELECT id FROM idx_entries WHERE store = ? AND index_name = ? AND value = ?',
[store, indexName, serializeIndexValue(value)],
) as string[];
}
async queryByIndex(
store: string,
indexName: string,
range: {
gt?: unknown;
gte?: unknown;
lt?: unknown;
lte?: unknown;
},
): Promise<string[]> {
const conditions = ['store = ?', 'index_name = ?'];
const params: unknown[] = [store, indexName];
if (range.gte !== undefined) {
conditions.push('value >= ?');
params.push(serializeIndexValue(range.gte));
} else if (range.gt !== undefined) {
conditions.push('value > ?');
params.push(serializeIndexValue(range.gt));
}
if (range.lte !== undefined) {
conditions.push('value <= ?');
params.push(serializeIndexValue(range.lte));
} else if (range.lt !== undefined) {
conditions.push('value < ?');
params.push(serializeIndexValue(range.lt));
}
const sql = `SELECT id FROM idx_entries WHERE ${conditions.join(' AND ')}`;
return this.db.selectValues(sql, params) as string[];
}
async rebuildIndexes(
store: string,
indexes: IndexDefinition[],
): Promise<void> {
this.db.transaction(() => {
// Clear existing index entries for this collection
this.db.exec({
sql: 'DELETE FROM idx_entries WHERE store = ?',
bind: [store],
});
// Read all non-deleted docs
const docs = this.db.selectObjects(
'SELECT id, data FROM docs WHERE store = ? AND deleted = 0',
[store],
);
// Rebuild
for (const row of docs) {
const data = JSON.parse(row.data as string);
for (const def of indexes) {
const val = extractIndexValue(data, def.fields as string[]);
if (val !== undefined) {
this.db.exec({
sql: `INSERT OR REPLACE INTO idx_entries (store, index_name, id, value)
VALUES (?,?,?,?)`,
bind: [store, def.name, row.id, serializeIndexValue(val)],
});
}
}
}
});
}
// ── Applied-event tracking ─────────────────────────────────
async isEventApplied(filename: string): Promise<boolean> {
const rows = this.db.selectValues(
'SELECT 1 FROM applied WHERE filename = ?',
[filename],
);
return rows.length > 0;
}
async markEventApplied(filename: string): Promise<void> {
this.db.exec({
sql: 'INSERT OR IGNORE INTO applied (filename, applied_at) VALUES (?,?)',
bind: [filename, Date.now()],
});
}
async getAppliedSet(): Promise<Set<string>> {
const names = this.db.selectValues(
'SELECT filename FROM applied',
) as string[];
return new Set(names);
}
// ── Meta ───────────────────────────────────────────────────
async getMeta<T = unknown>(key: string): Promise<T | undefined> {
const rows = this.db.selectObjects(
'SELECT value FROM meta WHERE key = ?',
[key],
);
if (rows.length === 0) return undefined;
const raw = rows[0].value as string | null;
if (raw === null) return undefined;
return JSON.parse(raw) as T;
}
async setMeta(key: string, value: unknown): Promise<void> {
this.db.exec({
sql: 'INSERT OR REPLACE INTO meta (key, value) VALUES (?,?)',
bind: [key, JSON.stringify(value)],
});
}
// ── Lifecycle ──────────────────────────────────────────────
close(): void {
if (this.db.isOpen()) {
this.db.close();
}
}
// ── Helpers ────────────────────────────────────────────────
private rowToDoc(row: Record<string, unknown>): DocRecord {
return {
store: row.store as string,
id: row.id as string,
data: JSON.parse(row.data as string),
ts: row.ts as number,
rev: row.rev as number,
deleted: (row.deleted as number) === 1,
};
}
}

65
sqlite/src/sqlite3-types.d.ts vendored Normal file
View File

@ -0,0 +1,65 @@
/**
* Minimal type declarations for @sqlite.org/sqlite-wasm.
* Covers the OO1 API surface used by this library.
*/
declare module '@sqlite.org/sqlite-wasm' {
export interface SQLite3Static {
oo1: {
DB: new (
filename?: string,
flags?: string,
vfs?: string,
) => OO1Database;
OpfsDb?: new (filename: string, flags?: string) => OO1Database;
};
installOpfsSAHPoolVfs?: (options?: {
clearOnInit?: boolean;
initialCapacity?: number;
directory?: string;
name?: string;
}) => Promise<SAHPoolUtil>;
capi: Record<string, unknown>;
wasm: Record<string, unknown>;
}
export interface OO1Database {
exec(options: {
sql: string;
bind?: unknown[];
rowMode?: 'array' | 'object' | 'stmt';
resultRows?: unknown[];
columnNames?: string[];
returnValue?: 'this' | 'resultRows' | 'saveSql';
}): OO1Database;
exec(sql: string): OO1Database;
selectObjects(sql: string, bind?: unknown[]): Record<string, unknown>[];
selectValues(sql: string, bind?: unknown[]): unknown[];
prepare(sql: string): OO1Statement;
transaction(fn: (db: OO1Database) => void): OO1Database;
close(): void;
isOpen(): boolean;
changes(total?: boolean): number;
}
export interface OO1Statement {
bind(values: unknown[]): OO1Statement;
bind(index: number, value: unknown): OO1Statement;
step(): boolean;
stepFinalize(): boolean;
get(target?: unknown[] | Record<string, unknown>): unknown;
getColumnName(index: number): string;
reset(clearBindings?: boolean): OO1Statement;
finalize(): void;
}
export interface SAHPoolUtil {
OpfsSAHPoolDb: new (filename: string, flags?: string) => OO1Database;
getCapacity(): number;
getFileCount(): number;
removeVfs(): Promise<void>;
}
export default function sqlite3InitModule(
config?: Record<string, unknown>,
): Promise<SQLite3Static>;
}

267
sqlite/src/sync-engine.ts Normal file
View File

@ -0,0 +1,267 @@
import type { SyncEvent, IndexDefinition, OpenOptions } from './types.js';
import type { SQLiteStore } from './sqlite-store.js';
import type { FolderStore } from './folder-store.js';
import type { Emitter } from './emitter.js';
import { canonicalJson, sha256Hex, eventFilename } from './utils.js';
export class SyncEngine {
private clientId: string;
private conflictResolver?: OpenOptions['conflictResolver'];
private collectionIndexes = new Map<string, IndexDefinition[]>();
private syncing = false;
constructor(
private store: SQLiteStore,
private folder: FolderStore,
private emitter: Emitter,
clientId: string,
conflictResolver?: OpenOptions['conflictResolver'],
) {
this.clientId = clientId;
this.conflictResolver = conflictResolver;
}
// ── Collection index registration ──────────────────────────
registerIndexes(store: string, indexes: IndexDefinition[]): void {
this.collectionIndexes.set(store, indexes);
}
getIndexes(store: string): IndexDefinition[] {
return this.collectionIndexes.get(store) ?? [];
}
// ── Local writes (write to SQLite + folder) ────────────────
async writeKV(
key: string,
value: unknown,
ts: number,
rev: number,
): Promise<void> {
const event: SyncEvent = {
type: 'put',
store: 'kv',
key,
ts,
clientId: this.clientId,
data: value,
rev,
};
await this.persistEvent(event);
}
async deleteKV(key: string, ts: number, rev: number): Promise<void> {
const event: SyncEvent = {
type: 'delete',
store: 'kv',
key,
ts,
clientId: this.clientId,
rev,
};
await this.persistEvent(event);
}
async writeDoc(
store: string,
id: string,
data: unknown,
ts: number,
rev: number,
): Promise<void> {
const event: SyncEvent = {
type: 'put',
store,
key: id,
id,
ts,
clientId: this.clientId,
data,
rev,
};
await this.persistEvent(event);
}
async deleteDocEvent(
store: string,
id: string,
ts: number,
rev: number,
): Promise<void> {
const event: SyncEvent = {
type: 'delete',
store,
key: id,
id,
ts,
clientId: this.clientId,
rev,
};
await this.persistEvent(event);
}
// ── Core persist: SQLite + folder ──────────────────────────
private async persistEvent(event: SyncEvent): Promise<void> {
const canonical = canonicalJson(event);
const hash = await sha256Hex(canonical);
const filename = eventFilename(event.ts, hash);
// Write to SQLite first (fast path)
await this.applyEvent(event);
await this.store.markEventApplied(filename);
// Then persist to folder (if available)
if (this.folder.hasHandle && (await this.folder.hasPermission())) {
try {
await this.folder.writeEvent(filename, event);
} catch (err) {
this.emitter.emit('folder:lost-permission', err);
}
}
this.emitter.emit('change', {
type: event.type,
store: event.store,
key: event.key,
id: event.id,
data: event.data,
});
}
// ── Sync: import folder events into SQLite ─────────────────
async sync(): Promise<void> {
if (this.syncing) return;
if (!this.folder.hasHandle) return;
const hasAccess = await this.folder.hasPermission();
if (!hasAccess) {
this.emitter.emit('folder:lost-permission');
return;
}
this.syncing = true;
this.emitter.emit('sync:start');
try {
const appliedSet = await this.store.getAppliedSet();
const filenames = await this.folder.scanEventFilenames();
let importCount = 0;
for (const name of filenames) {
if (appliedSet.has(name)) continue;
const event = await this.folder.readEventFile(name);
if (!event) continue;
const hadConflict = await this.applyEventWithConflictCheck(event);
await this.store.markEventApplied(name);
importCount++;
if (hadConflict) {
this.emitter.emit('conflict', { filename: name, event });
}
this.emitter.emit('change', {
type: event.type,
store: event.store,
key: event.key,
id: event.id,
data: event.data,
source: 'sync',
});
}
this.emitter.emit('sync:end', { imported: importCount });
} catch (err) {
this.emitter.emit('sync:end', { error: err });
} finally {
this.syncing = false;
}
}
// ── Apply a single event to SQLite ─────────────────────────
private async applyEvent(event: SyncEvent): Promise<void> {
if (event.store === 'kv') {
if (event.type === 'put') {
await this.store.putKV({
key: event.key,
value: event.data,
ts: event.ts,
rev: event.rev ?? 0,
});
} else {
await this.store.deleteKV(event.key);
}
} else {
const indexes = this.getIndexes(event.store);
if (event.type === 'put') {
await this.store.putDoc(
{
store: event.store,
id: event.id ?? event.key,
data: event.data,
ts: event.ts,
rev: event.rev ?? 0,
},
indexes,
);
} else {
await this.store.deleteDoc(
event.store,
event.id ?? event.key,
event.ts,
event.rev ?? 0,
indexes,
);
}
}
}
private async applyEventWithConflictCheck(
event: SyncEvent,
): Promise<boolean> {
let hadConflict = false;
if (event.store === 'kv') {
const existing = await this.store.getKV(event.key);
if (existing) {
if (event.ts > existing.ts) {
hadConflict = true;
} else if (event.ts === existing.ts) {
hadConflict = true;
if (this.conflictResolver) {
const resolved = this.conflictResolver(existing.value, event.data);
event = { ...event, data: resolved };
}
} else {
return true;
}
}
} else {
const existing = await this.store.getRawDoc(
event.store,
event.id ?? event.key,
);
if (existing && !existing.deleted) {
if (event.ts > existing.ts) {
hadConflict = true;
} else if (event.ts === existing.ts) {
hadConflict = true;
if (this.conflictResolver) {
const resolved = this.conflictResolver(existing.data, event.data);
event = { ...event, data: resolved };
}
} else {
return true;
}
}
}
await this.applyEvent(event);
return hadConflict;
}
}

95
sqlite/src/types.ts Normal file
View File

@ -0,0 +1,95 @@
// ── Event types ──────────────────────────────────────────────
export type EventType = 'put' | 'delete';
export interface SyncEvent {
type: EventType;
store: string; // "kv" | collection name
key: string;
id?: string; // present for collection documents
ts: number; // millisecond Unix epoch
clientId: string;
data?: unknown; // absent on deletes
rev?: number;
}
// ── Index / collection options ───────────────────────────────
export interface IndexDefinition<T = unknown> {
name: string;
fields: (keyof T & string)[] | string[];
}
export interface StoreOptions<T = unknown> {
name: string;
indexes?: IndexDefinition<T>[];
}
export interface OpenOptions {
dbName?: string;
autoSyncIntervalMs?: number;
clientId?: string;
conflictResolver?: (current: unknown, incoming: unknown) => unknown;
/** Passed to sqlite3InitModule(). Use locateFile to override .wasm loading. */
sqlite3Config?: Record<string, unknown>;
}
// ── Store record shapes ──────────────────────────────────────
export interface KVRecord {
key: string;
value: unknown;
ts: number;
rev: number;
}
export interface DocRecord {
store: string;
id: string;
data: unknown;
ts: number;
rev: number;
deleted?: boolean;
}
// ── Index range (replaces IDBKeyRange for SQLite) ────────────
export interface IndexRange {
gt?: unknown;
gte?: unknown;
lt?: unknown;
lte?: unknown;
}
// ── Event emitter ────────────────────────────────────────────
export type SyncDBEventName =
| 'sync:start'
| 'sync:end'
| 'change'
| 'conflict'
| 'folder:lost-permission';
export type SyncDBEventHandler = (...args: unknown[]) => void;
// ── KV API surface ───────────────────────────────────────────
export interface KVApi {
get<T = unknown>(key: string): Promise<T | undefined>;
set<T = unknown>(key: string, value: T): Promise<void>;
delete(key: string): Promise<void>;
has(key: string): Promise<boolean>;
keys(): Promise<string[]>;
entries<T = unknown>(): Promise<Array<[string, T]>>;
}
// ── Collection API surface ───────────────────────────────────
export interface CollectionApi<T extends { id: string }> {
get(id: string): Promise<T | undefined>;
put(doc: T): Promise<void>;
delete(id: string): Promise<void>;
all(): Promise<T[]>;
findByIndex(indexName: string, value: unknown): Promise<T[]>;
queryByIndex(indexName: string, range: IndexRange): Promise<T[]>;
}

94
sqlite/src/utils.ts Normal file
View File

@ -0,0 +1,94 @@
// ── Canonical JSON ───────────────────────────────────────────
function sortKeys(obj: unknown): unknown {
if (obj === null || obj === undefined || typeof obj !== 'object') return obj;
if (Array.isArray(obj)) return obj.map(sortKeys);
const sorted: Record<string, unknown> = {};
for (const key of Object.keys(obj as Record<string, unknown>).sort()) {
sorted[key] = sortKeys((obj as Record<string, unknown>)[key]);
}
return sorted;
}
export function canonicalJson(obj: unknown): string {
return JSON.stringify(sortKeys(obj));
}
// ── Hashing ──────────────────────────────────────────────────
export async function sha256Hex(data: string): Promise<string> {
const buf = await crypto.subtle.digest(
'SHA-256',
new TextEncoder().encode(data),
);
return Array.from(new Uint8Array(buf))
.map((b) => b.toString(16).padStart(2, '0'))
.join('');
}
// ── Event filenames ──────────────────────────────────────────
const HASH_PREFIX_LEN = 12;
export function eventFilename(ts: number, fullHash: string): string {
return `${ts}_${fullHash.slice(0, HASH_PREFIX_LEN)}.json`;
}
const EVENT_FILE_RE = /^(\d+)_([0-9a-f]+)\.json$/;
export function parseEventFilename(
name: string,
): { ts: number; hash: string } | null {
const m = EVENT_FILE_RE.exec(name);
if (!m) return null;
return { ts: Number(m[1]), hash: m[2] };
}
// ── Client ID ────────────────────────────────────────────────
export function generateClientId(): string {
return crypto.randomUUID();
}
// ── Index value extraction ───────────────────────────────────
export function extractIndexValue(
data: unknown,
fields: string[],
): IDBValidKey | undefined {
if (!data || typeof data !== 'object') return undefined;
const rec = data as Record<string, unknown>;
if (fields.length === 1) {
const v = getNestedValue(rec, fields[0]);
if (v === undefined || v === null) return undefined;
return toIDBKey(v);
}
const parts: IDBValidKey[] = [];
for (const f of fields) {
const v = getNestedValue(rec, f);
if (v === undefined || v === null) return undefined;
parts.push(toIDBKey(v));
}
return parts;
}
function getNestedValue(obj: Record<string, unknown>, path: string): unknown {
let cur: unknown = obj;
for (const seg of path.split('.')) {
if (cur === null || cur === undefined || typeof cur !== 'object') {
return undefined;
}
cur = (cur as Record<string, unknown>)[seg];
}
return cur;
}
function toIDBKey(v: unknown): IDBValidKey {
if (typeof v === 'string' || typeof v === 'number') return v;
if (v instanceof Date) return v;
if (Array.isArray(v)) return v.map(toIDBKey);
// Fallback: stringify
return String(v);
}

21
sqlite/tsconfig.json Normal file
View File

@ -0,0 +1,21 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "ES2022",
"moduleResolution": "bundler",
"lib": ["ES2022", "DOM", "DOM.Iterable"],
"outDir": "dist",
"rootDir": "src",
"declaration": true,
"declarationMap": true,
"sourceMap": true,
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noFallthroughCasesInSwitch": true,
"forceConsistentCasingInFileNames": true,
"skipLibCheck": true
},
"include": ["src"],
"exclude": ["node_modules", "dist"]
}