Main README covers quick start, API overview, and links to example READMEs.
Each example (paste, filemanager, can-sync, canfs) gets its own README
with setup instructions, architecture, and configuration details.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comments help non-Rust users understand what each function, struct, and
module does. Covers the core service (18 source files) and all four
example projects (can-sync, canfs, filemanager, paste).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Files now stored in type subdirectories: images/, pdf/, documents/,
video/, audio/, others/
- Filename format: YYYY-MM-DD_HH-MM_{hash8}.{ext} (human-readable)
- Added mime_to_type_dir() for MIME-to-directory classification
- write_asset() auto-creates type subdirectories
- Backward compatible: old flat filenames still work via DB lookup
- Filemanager: added TYPE/ virtual tree root for browsing by content type
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- CAN service: public SSE endpoint at /api/v1/can/0/events broadcasts
new_asset events on ingest and sync push (no auth required)
- Paste backend: SSE proxy at /paste/events streams from CAN service,
with auto-reconnect on connection loss
- Paste frontend: EventSource subscribes to /paste/events and calls
loadItems() on new_asset events for instant UI refresh
- When assets arrive via P2P sync, paste updates automatically
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Peers sharing the same sync_passphrase can now find each other
automatically over the internet without manual ticket exchange or
port forwarding. Uses n0's public pkarr relay servers as a
rendezvous point.
How it works:
- Derive 8 deterministic Ed25519 keypair "slots" from the passphrase
- Each peer claims a slot by publishing its EndpointId as a TXT record
- All peers scan all 8 slots every 15s to discover new peers
- Re-publish every 60s with 5min TTL to stay visible
- Discovered EndpointIds feed into the same peer channel as gossip
This runs alongside the existing gossip discovery (which still needs
bootstrap peers) and direct ticket-file connections (used by tests).
All 6 stress tests pass (102 assets, 63+ MB/s bidirectional).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Same as go_example_1.ps1 but for bash — starts CAN service, Paste UI,
and sync agent. Uses trap for clean shutdown on Ctrl+C.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The go script now builds and starts the can-sync agent alongside CAN
service and Paste UI. Any machine that clones the repo and runs the
script will auto-discover other instances via iroh's relay network
using the shared passphrase "duke-canman-sync" — no port forwarding
or manual peer configuration needed.
Changes:
- Add sync_api_key to root config.yaml (enables sync API)
- Update can-sync config.yaml with matching key and shared passphrase
- Update go_example_1.ps1 to build and launch can-sync agent
- Script now manages 3 processes: CAN service, Paste UI, sync agent
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace polling-based sync detection with SSE (Server-Sent Events) from
CAN service for instant push notifications on new asset ingests. Add
incremental hash queries via ?since=timestamp parameter to avoid
transferring full hash lists on every sync cycle.
CAN service changes:
- Add broadcast channel (SyncEventSender) in AppState for SSE events
- Add GET /sync/events SSE endpoint with auth via header or query param
- Fire broadcast events on both ingest and sync push
- Add db::get_assets_since() for incremental queries
- Support ?since= parameter on POST /sync/hashes
can-sync agent changes:
- Add SSE subscription with auto-reconnect in can_client
- Add get_hashes_since() for incremental catch-up
- Rewrite live push loop: SSE-driven with 30s fallback poll
- Remove poll_interval parameter from live sync functions
All 6 stress tests pass (102 assets, 63 MB/s bidirectional).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rewrite integration test with 6 scenarios:
- Single file each direction (basic sanity)
- Rapid burst 25 files A→B with mixed sizes (12MB+ large files)
- Rapid burst 25 files B→A with mixed sizes
- Simultaneous burst: 25 files on EACH side at the same time
- Final full-mirror verification (102 assets, perfect match)
Throughput measured at ~20 MB/s per direction, ~31 MB/s
bidirectional. All tests pass including simultaneous ingestion
of 162 MB across 50 files with zero conflicts or missing assets.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add mpsc channel between live receive and push loops so hashes
received from a peer aren't pushed right back (echo prevention).
Change initial reconciliation to use tokio::join! for concurrent
send/receive, avoiding QUIC flow-control deadlock when both sides
have large transfers. Update known_hashes to union-insert so
peer-received hashes persist across poll cycles.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix bidirectional stream handling: responder uses accept_bi() instead
of open_bi() so both sides communicate on the same stream
- Add live_receive_loop to accept incoming bi-streams during ongoing
sync (peer's push loop opens new streams per batch)
- Split live_sync_loop into live_push_loop + live_receive_loop running
concurrently via tokio::select in new run_live_sync()
- Update handle_incoming to run live sync after initial reconciliation
- Add direct peer connection via ticket files (EndpointAddr JSON
exchange) for local testing without gossip bootstrap
- Add CAN_PORT env var override for running multiple CAN instances
- Add integration test binary (sync_test.rs): starts 2 CAN services +
2 sync agents, ingests files on each side, verifies bidirectional
sync with 4 test cases (A→B, B→A, batch, count match)
- Add PowerShell script (run-integration-test.ps1) for one-command test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the over-engineered iroh-docs/libraries/filters architecture
with a simple peer-to-peer sync using:
- iroh 0.96 Endpoint for QUIC transport + NAT traversal
- iroh-gossip for peer discovery via shared passphrase
- Protobuf messages over QUIC streams for asset transfer
- CAN service's private /sync/* API for local data access
Deleted: announcer, fetcher, library, manifest, node, routes (2860 lines)
Added: discovery, peer, protocol (simplified ~600 lines)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New /sync/* endpoints for peer-to-peer replication:
- POST /sync/hashes - list all asset digests for reconciliation
- POST /sync/pull - pull full assets (metadata + content) by hash
- POST /sync/push - push asset with explicit timestamp for deterministic hashing
- POST /sync/meta - update metadata (tags, description, trash state)
All endpoints use protobuf encoding (prost) and require X-Sync-Key header
matching config.sync_api_key. Sync API is disabled when no key is configured.
Also adds db::get_all_assets() for sync reconciliation (includes trashed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CAN Service: content-addressable storage with HTTP API, SQLite metadata,
file-based blob storage, thumbnail generation, and integrity verification.
can-sync v1: P2P sync sidecar using iroh-docs for encrypted peer-to-peer
replication with library/filter-based selective sync. Fully builds but
being superseded by v2 (simplified full-mirror approach).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
