CanMan/README.md

CAN Service

Containerized Asset Network -- A self-healing local storage daemon with HTTP REST and Protobuf APIs for ingesting, managing, and retrieving files.

CAN stores any file you throw at it, tags it with metadata, verifies integrity in the background, and syncs between machines over encrypted P2P connections. Think of it as a personal S3 that runs on your laptop and replicates to your other devices automatically.

---

Quick Start

# Build and run (listens on port 3210)
cargo run

The service reads config.yaml from the current directory:

storage_root: "./can_data"
admin_token: "super_secret_rebuild"
enable_thumbnail_cache: true
verify_interval_hours: 12
sync_api_key: "can-sync-default-key"   # enables P2P sync endpoints

Override the port with CAN_PORT=8080 cargo run.

Upload a file

curl -X POST http://localhost:3210/api/v1/can/0/ingest \
  -F "file=@photo.jpg" \
  -F "tags=vacation,summer" \
  -F "application=my-app"

Upload JSON data (agent-friendly)

curl -X POST http://localhost:3210/api/v1/can/0/ingest/data \
  -H "Content-Type: application/json" \
  -d '{"data": {"key": "value"}, "tags": "config,backup"}'

Download a file

curl http://localhost:3210/api/v1/can/0/asset/{hash} -o file.jpg

---

How It Works

                 +-----------+
    Upload  ---->|           |----> SQLite index (millisecond queries)
                 | CAN       |
    Download <---|  Service  |----> Flat file storage (one file per asset)
                 |           |
    Search  ---->| port 3210 |----> OS file attributes (disaster recovery)
                 |           |
    SSE     <----|           |----> Background verifier (integrity checks)
                 +-----------+
                      |
              P2P Sync (protobuf over QUIC)
                      |
                 +-----------+
                 | CAN       |
                 |  Service  |  (another machine)
                 | port 3210 |
                 +-----------+

Each asset is saved as {timestamp}_{sha256hash}_{tags}.{ext} in a flat directory. Metadata lives in SQLite for fast queries and is redundantly written to OS-level file attributes (xattr on macOS/Linux, NTFS ADS on Windows) so you can recover even if the database is lost.

A background verifier re-hashes every file periodically and flags corruption. It also watches for filesystem changes in real time.

---

API

All endpoints live under /api/v1/can/0/. See API.md for the full specification.

Method	Path	Description
POST	/ingest	Upload a file (multipart form)
POST	/ingest/data	Upload JSON data (no multipart needed)
GET	/asset/{hash}	Download an asset by its SHA-256 hash
GET	/asset/{hash}/meta	Get metadata as JSON
PATCH	/asset/{hash}	Update tags and/or description
GET	/asset/{hash}/thumb/{w}/{h}	Get a resized thumbnail (images only)
GET	/list	Paginated listing with filters
GET	/search	Search by hash prefix, time range, MIME, tags, etc.
GET	/events	SSE stream of new asset notifications

Private sync endpoints (/sync/*) use protobuf and require the X-Sync-Key header.

---

Examples

Four example apps show what you can build on top of CAN:

Example	Port	Description	README
Paste	3211	Pastebin -- type text or paste images, auto-tags with #hashtags	README
File Manager	3212	Web file browser with grid/list views, search, and filters	README
CAN Sync	--	P2P replication agent -- encrypted sync via shared passphrase	README
CanFS	--	Mount assets as a read-only Windows drive (WinFSP)	README

Run everything at once

# Windows
.\go_example_1.ps1

# macOS / Linux
./go_example_1.sh

Starts CAN Service + Sync Agent + Paste, builds everything, cleans up on Ctrl+C.

---

Configuration

Field	Default	Description
storage_root	(required)	Directory where assets and the database are stored
admin_token	"changeme"	Bearer token for admin endpoints
enable_thumbnail_cache	true	Cache resized thumbnails in .thumbs/
rebuild_error_threshold	50	Max errors before triggering a full rebuild
verify_interval_hours	12	Hours between full integrity scans
sync_api_key	(none)	API key for sync endpoints; omit to disable sync

---

Project Structure

src/
  main.rs          Entry point: config, DB, verifier, HTTP server
  config.rs        YAML config loading
  db.rs            SQLite CRUD (assets, tags, search)
  hash.rs          SHA-256 content hashing
  storage.rs       File I/O (write, read, trash, filename parsing)
  verifier.rs      Background integrity checker + file watcher
  xattr.rs         OS-level file attributes (xattr / NTFS ADS)
  routes/          HTTP API handlers (ingest, asset, list, search, thumb, sync, events)
examples/
  paste/           Pastebin web app
  filemanager/     File browser web app
  can-sync/        P2P sync agent (iroh + gossip + pkarr)
  canfs/           Windows virtual filesystem (WinFSP)

Requirements

• Rust 1.75+
• SQLite bundled (no system install needed)
• WinFSP only for the canfs example (Windows only)