Replace polling-based sync detection with SSE (Server-Sent Events) from
CAN service for instant push notifications on new asset ingests. Add
incremental hash queries via ?since=timestamp parameter to avoid
transferring full hash lists on every sync cycle.
CAN service changes:
- Add broadcast channel (SyncEventSender) in AppState for SSE events
- Add GET /sync/events SSE endpoint with auth via header or query param
- Fire broadcast events on both ingest and sync push
- Add db::get_assets_since() for incremental queries
- Support ?since= parameter on POST /sync/hashes
can-sync agent changes:
- Add SSE subscription with auto-reconnect in can_client
- Add get_hashes_since() for incremental catch-up
- Rewrite live push loop: SSE-driven with 30s fallback poll
- Remove poll_interval parameter from live sync functions
All 6 stress tests pass (102 assets, 63 MB/s bidirectional).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rewrite integration test with 6 scenarios:
- Single file each direction (basic sanity)
- Rapid burst 25 files A→B with mixed sizes (12MB+ large files)
- Rapid burst 25 files B→A with mixed sizes
- Simultaneous burst: 25 files on EACH side at the same time
- Final full-mirror verification (102 assets, perfect match)
Throughput measured at ~20 MB/s per direction, ~31 MB/s
bidirectional. All tests pass including simultaneous ingestion
of 162 MB across 50 files with zero conflicts or missing assets.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add mpsc channel between live receive and push loops so hashes
received from a peer aren't pushed right back (echo prevention).
Change initial reconciliation to use tokio::join! for concurrent
send/receive, avoiding QUIC flow-control deadlock when both sides
have large transfers. Update known_hashes to union-insert so
peer-received hashes persist across poll cycles.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix bidirectional stream handling: responder uses accept_bi() instead
of open_bi() so both sides communicate on the same stream
- Add live_receive_loop to accept incoming bi-streams during ongoing
sync (peer's push loop opens new streams per batch)
- Split live_sync_loop into live_push_loop + live_receive_loop running
concurrently via tokio::select in new run_live_sync()
- Update handle_incoming to run live sync after initial reconciliation
- Add direct peer connection via ticket files (EndpointAddr JSON
exchange) for local testing without gossip bootstrap
- Add CAN_PORT env var override for running multiple CAN instances
- Add integration test binary (sync_test.rs): starts 2 CAN services +
2 sync agents, ingests files on each side, verifies bidirectional
sync with 4 test cases (A→B, B→A, batch, count match)
- Add PowerShell script (run-integration-test.ps1) for one-command test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the over-engineered iroh-docs/libraries/filters architecture
with a simple peer-to-peer sync using:
- iroh 0.96 Endpoint for QUIC transport + NAT traversal
- iroh-gossip for peer discovery via shared passphrase
- Protobuf messages over QUIC streams for asset transfer
- CAN service's private /sync/* API for local data access
Deleted: announcer, fetcher, library, manifest, node, routes (2860 lines)
Added: discovery, peer, protocol (simplified ~600 lines)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New /sync/* endpoints for peer-to-peer replication:
- POST /sync/hashes - list all asset digests for reconciliation
- POST /sync/pull - pull full assets (metadata + content) by hash
- POST /sync/push - push asset with explicit timestamp for deterministic hashing
- POST /sync/meta - update metadata (tags, description, trash state)
All endpoints use protobuf encoding (prost) and require X-Sync-Key header
matching config.sync_api_key. Sync API is disabled when no key is configured.
Also adds db::get_all_assets() for sync reconciliation (includes trashed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CAN Service: content-addressable storage with HTTP API, SQLite metadata,
file-based blob storage, thumbnail generation, and integrity verification.
can-sync v1: P2P sync sidecar using iroh-docs for encrypted peer-to-peer
replication with library/filter-based selective sync. Fully builds but
being superseded by v2 (simplified full-mirror approach).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
