DukeInc/CanMan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: CAN Service + examples (can-sync v1, canfs, filemanager, paste)

Browse Source 
CAN Service: content-addressable storage with HTTP API, SQLite metadata,
file-based blob storage, thumbnail generation, and integrity verification.

can-sync v1: P2P sync sidecar using iroh-docs for encrypted peer-to-peer
replication with library/filter-based selective sync. Fully builds but
being superseded by v2 (simplified full-mirror approach).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Jason Tudisco 2026-03-12 10:32:04 -06:00
commit 360ecbdad0
55 changed files with 23583 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
.gitignore vendored Normal file
View File

@ -0,0 +1,24 @@
# Build artifacts
/target/
examples/*/target/
# Data files (runtime-generated)
can_data/
*.db
*.db-shm
*.db-wal
can_sync_data/
examples/can-sync/can_sync_data/
# IDE / Editor
.vscode/
.idea/
*.swp
*.swo
# OS files
.DS_Store
Thumbs.db
# Claude local settings
.claude/

483
API.md Normal file
View File

@ -0,0 +1,483 @@
# CAN Service API Reference
**Base URL:** `http://localhost:3210`
**API Prefix:** `/api/v1/can/0`
**Content-Type:** `application/json` (all responses are JSON)
> The `0` in the path is the `can_id`. MVP supports only container `0`.
---
## Quick Start
```bash
# Start the server
cargo run
# Store a file
curl -X POST http://localhost:3210/api/v1/can/0/ingest \
-F "file=@photo.jpg" \
-F "tags=vacation,beach" \
-F "description=Summer trip"
# Store data (no file needed)
curl -X POST http://localhost:3210/api/v1/can/0/ingest/data \
-H "Content-Type: application/json" \
-d '{"data": {"sensor": "temp", "value": 22.5}, "tags": "iot,sensor"}'
# List everything
curl http://localhost:3210/api/v1/can/0/list
# Search by tag
curl "http://localhost:3210/api/v1/can/0/search?tags=vacation"
```
---
## Response Envelope
All responses use a standard wrapper:
```json
// Success
{
"status": "success",
"data": { ... }
}
// Error
{
"status": "error",
"error": "Human-readable error message"
}
```
**HTTP Status Codes:**
| Code | Meaning |
|------|---------|
| 200 | Success |
| 400 | Bad request (missing/invalid parameters) |
| 404 | Asset not found |
| 500 | Internal error / corrupted asset |
---
## Endpoints
### 1. Ingest File (Multipart)
Upload a binary file with optional metadata.
```
POST /api/v1/can/0/ingest
Content-Type: multipart/form-data
```
**Form Fields:**
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file` | Binary | **Yes** | The file to store |
| `mime_type` | String | No | Override MIME type (auto-detected from filename if omitted) |
| `human_file_name` | String | No | Logical filename (e.g. `report.pdf`) |
| `human_readable_path` | String | No | Logical folder path (e.g. `/docs/reports/`) |
| `application` | String | No | Application that produced this file |
| `user` | String | No | User or agent identity |
| `tags` | String | No | Comma-separated tags (e.g. `finance,Q4,report`) |
| `description` | String | No | Human-readable description |
**Response:**
```json
{
"status": "success",
"data": {
"timestamp": 1773014400123,
"hash": "a3b2c4d5e6f7...",
"filename": "1773014400123_a3b2c4d5e6f7_finance_Q4.pdf"
}
}
```
**Example:**
```bash
curl -X POST http://localhost:3210/api/v1/can/0/ingest \
-F "file=@quarterly_report.pdf" \
-F "application=WebUI" \
-F "user=jason" \
-F "tags=finance,Q4,report" \
-F "description=Q4 2025 financial report" \
-F "human_file_name=quarterly_report.pdf" \
-F "human_readable_path=/finance/reports/"
```
---
### 2. Ingest Data (JSON)
Store any JSON value directly -- no multipart needed. Designed for agents and programmatic use.
```
POST /api/v1/can/0/ingest/data
Content-Type: application/json
```
**JSON Body:**
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `data` | Any JSON | **Yes** | -- | The payload to store. Object, array, string, number, boolean, or null. |
| `mime_type` | String | No | `application/json` | Override MIME type (e.g. `text/plain` to store as `.txt`) |
| `human_file_name` | String | No | | Logical filename |
| `human_readable_path` | String | No | | Logical folder path |
| `application` | String | No | | Application/agent that produced this data |
| `user` | String | No | | User or agent identity |
| `tags` | String | No | | Comma-separated tags |
| `description` | String | No | | Human-readable description |
The `data` field is serialized to pretty-printed JSON and stored as a `.json` file (or `.txt` etc. if you override `mime_type`).
**Response:** Same as file ingest.
**Examples:**
```bash
# Minimal -- just dump an object
curl -X POST http://localhost:3210/api/v1/can/0/ingest/data \
-H "Content-Type: application/json" \
-d '{"data": {"key": "value"}}'
# Agent saving structured output
curl -X POST http://localhost:3210/api/v1/can/0/ingest/data \
-H "Content-Type: application/json" \
-d '{
"data": {
"agent_id": "planner-v2",
"session": "abc-123",
"steps": ["research", "outline", "draft"]
},
"application": "AgentOrchestrator",
"user": "planner_agent",
"tags": "agent,plan,session",
"description": "Planning output for session abc-123",
"human_file_name": "plan_output.json",
"human_readable_path": "/agents/planner/"
}'
# Store a plain string
curl -X POST http://localhost:3210/api/v1/can/0/ingest/data \
-H "Content-Type: application/json" \
-d '{"data": "Log: task completed at 14:30", "tags": "log"}'
# Store an array
curl -X POST http://localhost:3210/api/v1/can/0/ingest/data \
-H "Content-Type: application/json" \
-d '{"data": [1, 2, 3, "four"], "tags": "test"}'
# Store as plain text instead of JSON
curl -X POST http://localhost:3210/api/v1/can/0/ingest/data \
-H "Content-Type: application/json" \
-d '{"data": "plain text content", "mime_type": "text/plain"}'
```
---
### 3. Retrieve Asset
Download the physical file by its hash.
```
GET /api/v1/can/0/asset/{hash}
```
**Path Parameters:**
| Param | Type | Description |
|-------|------|-------------|
| `hash` | String | The SHA-256 hash returned from ingest |
**Response:** Raw file bytes with headers:
- `Content-Type` set to the asset's MIME type
- `Content-Disposition: attachment; filename="<human_filename>"`
Returns `500` if the asset is flagged as corrupted.
**Example:**
```bash
curl -o output.pdf http://localhost:3210/api/v1/can/0/asset/a3b2c4d5e6f7...
```
---
### 4. Get Asset Metadata
Retrieve all metadata for an asset without downloading the file.
```
GET /api/v1/can/0/asset/{hash}/meta
```
**Response:**
```json
{
"status": "success",
"data": {
"hash": "a3b2c4d5e6f7...",
"mime_type": "application/pdf",
"application": "WebUI",
"user": "jason",
"tags": ["finance", "Q4", "report"],
"description": "Q4 2025 financial report",
"human_filename": "quarterly_report.pdf",
"human_path": "/finance/reports/",
"timestamp": 1773014400123,
"is_trashed": false,
"is_corrupted": false
}
}
```
---
### 5. Update Metadata
Modify tags and/or description for an existing asset. The physical file is unchanged.
```
PATCH /api/v1/can/0/asset/{hash}
Content-Type: application/json
```
**JSON Body:**
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `tags` | String[] | No | New tag list (replaces all existing tags) |
| `description` | String | No | New description |
Both fields are optional. Only provided fields are updated.
**Example:**
```bash
curl -X PATCH http://localhost:3210/api/v1/can/0/asset/a3b2c4d5e6f7... \
-H "Content-Type: application/json" \
-d '{
"tags": ["finance", "Q4", "report", "reviewed"],
"description": "Q4 2025 report - reviewed by CFO"
}'
```
**Response:**
```json
{
"status": "success",
"data": "updated"
}
```
---
### 6. List Assets
Paginated listing of all assets with optional filtering.
```
GET /api/v1/can/0/list
```
**Query Parameters:**
| Param | Type | Default | Description |
|-------|------|---------|-------------|
| `limit` | Integer | `50` | Page size |
| `offset` | Integer | `0` | Starting position |
| `offset_time` | Integer | -- | Epoch ms cursor. Lists items strictly before/after this timestamp (based on `order`). Faster than offset for large datasets. |
| `order` | String | `desc` | Sort direction: `asc` or `desc` (by timestamp) |
| `application` | String | -- | Filter to assets from this application |
| `include_trashed` | Boolean | `false` | Include soft-deleted assets |
| `include_corrupted` | Boolean | `false` | Include corrupted assets |
**Response:**
```json
{
"status": "success",
"data": {
"items": [
{
"hash": "a3b2...",
"mime_type": "application/pdf",
"application": "WebUI",
"user": "jason",
"tags": ["finance"],
"description": "...",
"human_filename": "report.pdf",
"human_path": "/docs/",
"timestamp": 1773014400123,
"is_trashed": false,
"is_corrupted": false
}
],
"pagination": {
"limit": 50,
"offset": 0,
"total": 142
}
}
}
```
**Examples:**
```bash
# First page, 10 items
curl "http://localhost:3210/api/v1/can/0/list?limit=10"
# Second page
curl "http://localhost:3210/api/v1/can/0/list?limit=10&offset=10"
# Only assets from a specific app, oldest first
curl "http://localhost:3210/api/v1/can/0/list?application=IoTAgent&order=asc"
# Include trashed assets
curl "http://localhost:3210/api/v1/can/0/list?include_trashed=true"
```
---
### 7. Search Assets
Search with multiple filters. All filters are AND-combined.
```
GET /api/v1/can/0/search
```
**Query Parameters:**
| Param | Type | Default | Description |
|-------|------|---------|-------------|
| `hash` | String | -- | Exact hash or prefix match (e.g. `a3b2` matches `a3b2c4d5...`) |
| `start_time` | Integer | -- | Epoch ms lower bound (inclusive) |
| `end_time` | Integer | -- | Epoch ms upper bound (inclusive) |
| `tags` | String | -- | Comma-separated. AND logic: asset must have **all** specified tags. |
| `mime_type` | String | -- | Exact MIME type match (e.g. `image/jpeg`) |
| `user` | String | -- | Exact user/agent identity match |
| `application` | String | -- | Exact application match |
| `limit` | Integer | `50` | Page size |
| `offset` | Integer | `0` | Starting position |
| `order` | String | `desc` | Sort direction: `asc` or `desc` |
| `include_trashed` | Boolean | `false` | Include soft-deleted assets |
| `include_corrupted` | Boolean | `false` | Include corrupted assets |
**Response:** Same structure as List (items + pagination).
**Examples:**
```bash
# Find by hash prefix
curl "http://localhost:3210/api/v1/can/0/search?hash=a3b2"
# All JPEG images from last 24 hours
curl "http://localhost:3210/api/v1/can/0/search?mime_type=image/jpeg&start_time=1773014400000"
# Assets tagged with BOTH "sensor" AND "temperature"
curl "http://localhost:3210/api/v1/can/0/search?tags=sensor,temperature"
# Everything from a specific agent
curl "http://localhost:3210/api/v1/can/0/search?application=PlannerAgent&user=agent_v2"
# Combine filters
curl "http://localhost:3210/api/v1/can/0/search?tags=report&application=WebUI&order=asc&limit=5"
```
---
### 8. Get Thumbnail
Generate a resized thumbnail for image assets. Non-image assets return a placeholder SVG.
```
GET /api/v1/can/0/asset/{hash}/thumb/{max_width}/{max_height}
```
**Path Parameters:**
| Param | Type | Description |
|-------|------|-------------|
| `hash` | String | Asset hash |
| `max_width` | Integer | Maximum width in pixels |
| `max_height` | Integer | Maximum height in pixels |
Aspect ratio is always preserved. The image fits within the `max_width x max_height` box.
**Response:**
- **Image assets:** JPEG bytes (`Content-Type: image/jpeg`). Cached in `.thumbs/` if enabled.
- **Non-image assets:** SVG placeholder icon (`Content-Type: image/svg+xml`).
**Example:**
```bash
# 200x200 thumbnail
curl -o thumb.jpg http://localhost:3210/api/v1/can/0/asset/a3b2.../thumb/200/200
```
---
## Configuration
`config.yaml` at the project root (or pass a custom path as the first CLI argument):
```yaml
storage_root: "/var/lib/can_data" # Where files are stored
admin_token: "super_secret_rebuild" # Bearer token for admin operations
enable_thumbnail_cache: true # Cache thumbnails in .thumbs/
rebuild_error_threshold: 50 # Error tolerance before hard rebuild
verify_interval_hours: 12 # Hours between integrity scrubs
```
---
## Concepts
### Hash
Every asset gets a unique SHA-256 hash computed as `SHA256(timestamp_be_bytes + file_content)`. This hash is the primary identifier used in all API calls. Because the timestamp is mixed in, even identical file content produces different hashes at different times.
### Physical Filename
Files are stored as: `{timestamp}_{hash}_{tags}.{extension}`
For example: `1773014400123_a3b2c4d5e6f7_finance_Q4.pdf`
This naming allows offline integrity verification -- you can recompute the hash from the timestamp and file contents and compare it to the filename.
### Tags
Tags are comma-separated strings. On ingest and in search, pass them as a single string: `"finance,Q4,report"`. In metadata responses, they come back as an array: `["finance", "Q4", "report"]`.
When searching, tag matching uses AND logic: `?tags=finance,Q4` finds only assets that have **both** tags.
When patching, tags are **replaced** entirely (not merged).
### Integrity Verification
A background verifier runs automatically:
1. **On startup:** Scrubs all assets against their hashes.
2. **On file change:** Watches the storage directory for modifications.
3. **Periodically:** Every `verify_interval_hours`.
Corrupted assets are flagged (`is_corrupted: true`) and excluded from standard list/search results unless `include_corrupted=true` is passed. Retrieving a corrupted asset via GET returns a 500 error.
### OS File Attributes
Critical metadata is written to the host OS as file attributes for disaster recovery:
- **Linux/macOS:** Extended attributes (`xattr`) under the `user.can.*` namespace
- **Windows:** NTFS Alternate Data Streams (`:can.*`)
This means the SQLite database can be rebuilt from scratch by scanning the storage directory.

2630
Cargo.lock generated Normal file
View File

File diff suppressed because it is too large Load Diff

52
Cargo.toml Normal file
View File

@ -0,0 +1,52 @@
[package]
name = "can-service"
version = "0.1.0"
edition = "2021"
description = "Containerized Asset Network - a self-healing local storage daemon"
[dependencies]
# Web framework
axum = { version = "0.8", features = ["multipart"] }
tokio = { version = "1", features = ["full"] }
tower-http = { version = "0.6", features = ["cors", "trace"] }
tokio-util = { version = "0.7", features = ["io"] }
# Database
rusqlite = { version = "0.32", features = ["bundled"] }
# Serialization
serde = { version = "1", features = ["derive"] }
serde_json = "1"
serde_yaml = "0.9"
# Hashing
sha2 = "0.10"
hex = "0.4"
# Image processing
image = { version = "0.25", default-features = false, features = ["jpeg", "png", "gif", "webp"] }
# File system watching
notify = "7"
# MIME type detection
mime_guess = "2"
mime = "0.3"
# Logging
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
# Utilities
chrono = { version = "0.4", features = ["serde"] }
anyhow = "1"
thiserror = "2"
# OS attributes (unix only, windows uses custom ADS)
[target.'cfg(unix)'.dependencies]
xattr = "1"
[dev-dependencies]
tempfile = "3"
reqwest = { version = "0.12", features = ["multipart", "json"] }
tokio-test = "0.4"

5
config.yaml Normal file
View File

@ -0,0 +1,5 @@
storage_root: "./can_data"
admin_token: "super_secret_rebuild"
enable_thumbnail_cache: true
rebuild_error_threshold: 50
verify_interval_hours: 12

5575
examples/can-sync/Cargo.lock generated Normal file
View File

File diff suppressed because it is too large Load Diff

44
examples/can-sync/Cargo.toml Normal file
View File

@ -0,0 +1,44 @@
[package]
name = "can-sync"
version = "0.1.0"
edition = "2021"
description = "P2P sync service for CAN content-addressable storage"
[[bin]]
name = "can-sync"
path = "src/main.rs"
[dependencies]
# P2P networking
iroh = "0.96"
iroh-blobs = "0.98"
iroh-docs = "0.96"
iroh-gossip = "0.96"
# HTTP server + client
axum = "0.8"
tokio = { version = "1", features = ["full"] }
reqwest = { version = "0.12", features = ["json", "multipart"] }
tower-http = { version = "0.6", features = ["cors"] }
# Serialization
serde = { version = "1", features = ["derive"] }
serde_json = "1"
serde_yaml = "0.9"
postcard = { version = "1", features = ["alloc"] }
# Storage
rusqlite = { version = "0.32", features = ["bundled"] }
# Utilities
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
anyhow = "1"
open = "5"
sha2 = "0.10"
hex = "0.4"
uuid = { version = "1", features = ["v4"] }
chrono = { version = "0.4", features = ["serde"] }
bytes = "1"
futures-lite = "2"
tokio-util = { version = "0.7", features = ["io"] }

263
examples/can-sync/README.md Normal file
View File

@ -0,0 +1,263 @@
# CAN Sync
P2P file synchronization service that runs on top of [CAN Service](../../). Uses [iroh](https://iroh.computer/) for encrypted peer-to-peer networking with NAT traversal.
```
┌─────────────┐ HTTP API ┌─────────────┐ iroh (QUIC) ┌─────────────┐
│ CAN Service │◄───────────►│ CAN Sync │◄─────────────►│ CAN Sync │
│ (port 3210)│ │ (port 3213)│ │ (remote) │
│ storage + │ │ P2P node + │ │ │
│ SQLite │ │ libraries │ │ │
└─────────────┘ └─────────────┘ └─────────────┘
```
CAN Sync communicates with CAN Service **only** via its public HTTP API — zero changes to CAN Service required.
## Quick Start
1. **Start CAN Service** (default port 3210):
```bash
cd ../..
cargo run
```
2. **Edit config** (optional — defaults work out of the box):
```bash
cp config.yaml my-config.yaml
# edit my-config.yaml if needed
```
3. **Start CAN Sync**:
```bash
cargo run
# or with a custom config:
cargo run -- my-config.yaml
```
CAN Sync starts on `http://127.0.0.1:3213` and connects to CAN Service at `http://127.0.0.1:3210/api/v1/can/0`.
## Configuration
`config.yaml`:
```yaml
# URL of the local CAN Service API
can_service_url: "http://127.0.0.1:3210/api/v1/can/0"
# Address for the CAN Sync HTTP API
listen_addr: "127.0.0.1:3213"
# Directory for persistent data (peer key, sync state DB)
data_dir: "./can_sync_data"
# Custom relay server URL (null = iroh's public relay)
relay_url: null
# Seconds between fast polls for new assets
poll_interval_secs: 5
# Seconds between full scans of all assets
full_scan_interval_secs: 300
```
## Concepts
### Libraries
A **library** is a shared collection of CAN assets that syncs between peers. Each library has a **filter** that determines which assets belong to it.
Filter options (combined with AND logic):
- `application` — match assets with this application tag (e.g. `"paste"`)
- `tags` — match assets with any of these tags (e.g. `["photos", "backup"]`)
- `user` — match assets from this user identity
- `mime_prefix` — match assets whose MIME type starts with this (e.g. `"image/"`)
- `hashes` — manual list of specific asset hashes to include
### Sync Flow
**Outbound** (local → remote):
1. Announcer polls CAN Service for new/changed assets
2. Assets matching a library's filter get announced to the library's iroh document
3. iroh replicates the entry to all subscribed peers
4. Remote peer's fetcher downloads the blob and ingests it into their local CAN Service
**Inbound** (remote → local):
1. iroh document receives new entry from remote peer
2. Fetcher downloads the blob via iroh's encrypted QUIC transport
3. Fetcher verifies the CAN hash (SHA-256) independently
4. Fetcher ingests the file into local CAN Service with all metadata preserved
## API
All endpoints return JSON with `{ "status": "success", "data": ... }` or `{ "status": "error", "error": "..." }`.
### Status & Peers
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/status` | Node status, CAN service health, library count |
| GET | `/peers` | Connected peers list |
### Libraries
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/libraries` | Create a library |
| GET | `/libraries` | List all libraries |
| GET | `/libraries/{id}` | Get library details |
| DELETE | `/libraries/{id}` | Remove a library |
### Sharing
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/libraries/{id}/invite` | Generate a share ticket |
| POST | `/join` | Join a library from a ticket |
### Examples
**Create a library** that syncs all assets with `application=paste`:
```bash
curl -X POST http://127.0.0.1:3213/libraries \
-H "Content-Type: application/json" \
-d '{"name": "my-pastes", "filter": {"application": "paste"}}'
```
**Create a library** that syncs all images:
```bash
curl -X POST http://127.0.0.1:3213/libraries \
-H "Content-Type: application/json" \
-d '{"name": "images", "filter": {"mime_prefix": "image/"}}'
```
**Generate an invite ticket** to share with another machine:
```bash
curl -X POST http://127.0.0.1:3213/libraries/{id}/invite
```
**Join a library** on another machine using the ticket:
```bash
curl -X POST http://127.0.0.1:3213/join \
-H "Content-Type: application/json" \
-d '{"ticket": "eyJsaWJyYXJ5X25hbWUiOi..."}'
```
**List all libraries**:
```bash
curl http://127.0.0.1:3213/libraries
```
**Check status**:
```bash
curl http://127.0.0.1:3213/status
```
## Two-Machine Setup
### Machine A (the host)
**1. Start CAN Service** (default port 3210):
```bash
cd /path/to/CanService
cargo run
```
**2. Start CAN Sync** with default config (port 3213):
```bash
cd examples/can-sync
cargo run
```
**3. Create a library** (e.g. sync all images):
```bash
curl -X POST http://127.0.0.1:3213/libraries \
-H "Content-Type: application/json" \
-d '{"name": "shared-images", "filter": {"mime_prefix": "image/"}}'
```
Save the `id` from the response (e.g. `"id": "a1b2c3d4-..."`).
**4. Generate an invite ticket:**
```bash
curl -X POST http://127.0.0.1:3213/libraries/a1b2c3d4-.../invite
```
Copy the `ticket` string from the response — this is what Machine B needs.
### Machine B (the joiner)
**1. Start CAN Service** on a different port:
```bash
cd /path/to/CanService
CAN_PORT=3220 cargo run
```
**2. Create a config file** for CAN Sync pointing at Machine B's CAN Service:
```yaml
# machine-b-config.yaml
can_service_url: "http://127.0.0.1:3220/api/v1/can/0"
listen_addr: "127.0.0.1:3223"
data_dir: "./can_sync_data_b"
```
**3. Start CAN Sync** with that config:
```bash
cd examples/can-sync
cargo run -- machine-b-config.yaml
```
**4. Join the library** using Machine A's ticket:
```bash
curl -X POST http://127.0.0.1:3223/join \
-H "Content-Type: application/json" \
-d '{"ticket": "eyJsaWJyYXJ5X25hbWUiOi..."}'
```
### Verify it works
**Ingest a file on Machine A:**
```bash
curl -X POST http://127.0.0.1:3210/api/v1/can/0/ingest \
-F "file=@photo.jpg" \
-F "mime_type=image/jpeg"
```
**Check Machine B** — the file should appear within a few seconds:
```bash
curl http://127.0.0.1:3220/api/v1/can/0/list?limit=5
```
The same image (with matching hash and metadata) will be in Machine B's CAN Service, synced over iroh's encrypted P2P connection.
## Architecture
```
src/
├── main.rs — entry point: config, iroh node, announcer, fetcher, HTTP server
├── config.rs — YAML config loading
├── can_client.rs — HTTP client for CAN Service API (list, search, ingest, meta, etc.)
├── node.rs — iroh endpoint + blobs + docs + gossip + router
├── library.rs — library/filter definitions + SQLite state tracking
├── manifest.rs — AssetSyncEntry serialized into iroh document entries
├── announcer.rs — polls CAN Service, announces matching assets to libraries
├── fetcher.rs — receives remote entries, downloads blobs, ingests into CAN Service
└── routes.rs — Axum HTTP API handlers
```
## Security
- **Transport**: All peer-to-peer traffic is encrypted with QUIC + TLS 1.3 (mandatory in iroh)
- **Identity**: Each node has an Ed25519 keypair generated on first run
- **Access control**: Library access via cryptographic capability tickets — only peers with a valid ticket can read/write
- **NAT traversal**: iroh's built-in relay servers and hole-punching
- **Hash verification**: Downloaded files are independently verified against CAN's SHA-256 hash before ingestion
## Current Status
The service compiles and runs with the following fully implemented:
- iroh P2P node startup with all protocol handlers (blobs, docs, gossip)
- CAN Service HTTP client with full API coverage
- Library management with SQLite persistence
- Announcer polling loop (fast + full scan) with real iroh-docs writes
- Fetcher with iroh document event subscription for real-time sync
- Fetcher blob download via iroh and CAN hash verification before ingestion
- Real DocTicket-based invite/join with cryptographic capability tokens
- HTTP API for library CRUD, invite, and join

7
examples/can-sync/config.yaml Normal file
View File

@ -0,0 +1,7 @@
# CAN Sync configuration
can_service_url: "http://127.0.0.1:3210/api/v1/can/0"
listen_addr: "127.0.0.1:3213"
data_dir: "./can_sync_data"
relay_url: null
poll_interval_secs: 5
full_scan_interval_secs: 300

234
examples/can-sync/src/announcer.rs Normal file
View File

@ -0,0 +1,234 @@
use std::sync::Arc;
use std::time::Duration;
use anyhow::Result;
use tracing::{debug, error, info, warn};
use crate::can_client::CanClient;
use crate::library::SyncState;
use crate::manifest::AssetSyncEntry;
use crate::node::SyncNode;
/// The announcer periodically polls CAN service for new or changed assets
/// and writes matching entries into iroh library documents.
pub struct Announcer {
can: CanClient,
state: Arc<SyncState>,
node: Arc<SyncNode>,
poll_interval: Duration,
full_scan_interval: Duration,
}
impl Announcer {
pub fn new(
can: CanClient,
state: Arc<SyncState>,
node: Arc<SyncNode>,
poll_interval_secs: u64,
full_scan_interval_secs: u64,
) -> Self {
Self {
can,
state,
node,
poll_interval: Duration::from_secs(poll_interval_secs),
full_scan_interval: Duration::from_secs(full_scan_interval_secs),
}
}
/// Run the announcer loop — fast polls + periodic full scans
pub async fn run(self) {
let mut fast_tick = tokio::time::interval(self.poll_interval);
let mut full_tick = tokio::time::interval(self.full_scan_interval);
// Skip the first immediate tick for full scan (let fast poll get first data)
full_tick.tick().await;
info!(
"Announcer started (fast poll: {}s, full scan: {}s)",
self.poll_interval.as_secs(),
self.full_scan_interval.as_secs(),
);
loop {
tokio::select! {
_ = fast_tick.tick() => {
if let Err(e) = self.fast_poll().await {
warn!("Fast poll error: {:#}", e);
}
}
_ = full_tick.tick() => {
if let Err(e) = self.full_scan().await {
warn!("Full scan error: {:#}", e);
}
}
}
}
}
/// Fast poll: check for recently ingested assets
async fn fast_poll(&self) -> Result<()> {
let last_ts = self
.state
.get_state("last_seen_timestamp")?
.and_then(|s| s.parse::<i64>().ok())
.unwrap_or(0);
// Get recent assets ordered newest first
let resp = self.can.list(50, 0, "desc", Some(last_ts)).await?;
if resp.items.is_empty() {
return Ok(());
}
debug!("Fast poll found {} new assets since ts={}", resp.items.len(), last_ts);
// Track the newest timestamp we see
let mut max_ts = last_ts;
for asset in &resp.items {
if asset.timestamp > max_ts {
max_ts = asset.timestamp;
}
}
// Process assets against libraries
let libraries = self.state.list_libraries()?;
for asset in &resp.items {
for lib in &libraries {
if lib.filter.matches(asset) {
self.announce_asset(lib, asset).await?;
}
}
}
// Update last seen timestamp
self.state.set_state("last_seen_timestamp", &max_ts.to_string())?;
Ok(())
}
/// Full scan: paginate through all assets, checking for metadata changes
async fn full_scan(&self) -> Result<()> {
info!("Starting full scan...");
let libraries = self.state.list_libraries()?;
if libraries.is_empty() {
debug!("No libraries configured, skipping full scan");
return Ok(());
}
let page_size = 100;
let mut offset = 0;
let mut total_scanned = 0;
let mut total_announced = 0;
loop {
let resp = self.can.list_all(page_size, offset, true).await?;
let count = resp.items.len();
total_scanned += count;
for asset in &resp.items {
for lib in &libraries {
if lib.filter.matches(asset) {
let was_new = self.announce_asset(lib, asset).await?;
if was_new {
total_announced += 1;
}
}
}
}
if (count as i64) < page_size {
break;
}
offset += page_size;
}
info!(
"Full scan complete: scanned {}, announced {} new/updated",
total_scanned, total_announced
);
Ok(())
}
/// Announce a single asset to a library's iroh document.
/// Returns true if the asset was newly announced or updated.
async fn announce_asset(
&self,
lib: &crate::library::Library,
asset: &crate::can_client::AssetMeta,
) -> Result<bool> {
let doc_id = match &lib.doc_id {
Some(id) => id.clone(),
None => {
debug!("Library '{}' has no doc_id yet, skipping", lib.name);
return Ok(false);
}
};
// Check if already announced at current version
if self.state.is_announced(&lib.id, &asset.hash)? {
// Already announced — skip unless metadata changed
// (full scan handles re-announcement on metadata change)
return Ok(false);
}
// Download file content from CAN service and add as iroh blob
let iroh_blob_hash = match self.can.get_asset(&asset.hash).await {
Ok(content) => {
// Add to iroh blob store so remote peers can download it
match self.node.blobs.add_bytes(content).await {
Ok(tag_info) => Some(tag_info.hash.to_string()),
Err(e) => {
warn!(
"Failed to add blob for asset {}: {:#}",
&asset.hash[..12],
e
);
None
}
}
}
Err(e) => {
warn!(
"Failed to download asset {} from CAN service: {:#}",
&asset.hash[..12],
e
);
None
}
};
// Create sync entry with the iroh blob hash
let mut entry = AssetSyncEntry::from_asset_meta(asset, &self.node.peer_id());
entry.iroh_blob_hash = iroh_blob_hash;
let entry_bytes = entry.to_bytes();
// Write to iroh document (CRDT — concurrent writes merge automatically)
if let Err(e) = self
.node
.write_to_doc(&doc_id, asset.hash.as_bytes(), &entry_bytes)
.await
{
error!(
"Failed to write asset {} to doc {}: {:#}",
&asset.hash[..12],
&doc_id[..12],
e
);
return Ok(false);
}
// Mark as announced in local state
self.state.mark_announced(&lib.id, &asset.hash, entry.version)?;
debug!(
"Announced asset {} to library '{}' (doc {})",
&asset.hash[..12],
lib.name,
&doc_id[..12]
);
Ok(true)
}
}

291
examples/can-sync/src/can_client.rs Normal file
View File

@ -0,0 +1,291 @@
use anyhow::{Context, Result};
use bytes::Bytes;
use reqwest::multipart;
use serde::{Deserialize, Serialize};
/// HTTP client for CAN service API
#[derive(Debug, Clone)]
pub struct CanClient {
client: reqwest::Client,
base_url: String,
}
// ── API response types (mirror CAN service) ──
#[derive(Debug, Deserialize)]
pub struct ApiResponse<T> {
pub status: String,
pub data: T,
}
#[derive(Debug, Deserialize)]
pub struct ErrorResponse {
pub status: String,
pub error: String,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AssetMeta {
pub hash: String,
pub mime_type: String,
pub application: Option<String>,
pub user: Option<String>,
pub tags: Vec<String>,
pub description: Option<String>,
pub human_filename: Option<String>,
pub human_path: Option<String>,
pub timestamp: i64,
pub is_trashed: bool,
#[serde(default)]
pub is_corrupted: bool,
pub size: i64,
}
#[derive(Debug, Deserialize)]
pub struct ListResponse {
pub items: Vec<AssetMeta>,
pub pagination: Pagination,
}
#[derive(Debug, Deserialize)]
pub struct Pagination {
pub limit: i64,
pub offset: i64,
pub total: i64,
}
#[derive(Debug, Deserialize)]
pub struct IngestResult {
pub timestamp: i64,
pub hash: String,
pub filename: String,
}
// ── Search parameters ──
#[derive(Debug, Default, Serialize)]
pub struct SearchParams {
#[serde(skip_serializing_if = "Option::is_none")]
pub hash: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub start_time: Option<i64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub end_time: Option<i64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tags: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub mime_type: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub user: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub application: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub limit: Option<i64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub offset: Option<i64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub order: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub include_trashed: Option<bool>,
}
// ── Ingest metadata ──
#[derive(Debug, Default)]
pub struct IngestMeta {
pub mime_type: Option<String>,
pub human_file_name: Option<String>,
pub human_readable_path: Option<String>,
pub application: Option<String>,
pub user: Option<String>,
pub tags: Option<String>,
pub description: Option<String>,
}
// ── Client implementation ──
impl CanClient {
pub fn new(base_url: &str) -> Self {
Self {
client: reqwest::Client::new(),
base_url: base_url.trim_end_matches('/').to_string(),
}
}
/// List assets with pagination and ordering
pub async fn list(
&self,
limit: i64,
offset: i64,
order: &str,
offset_time: Option<i64>,
) -> Result<ListResponse> {
let mut url = format!("{}/list?limit={}&offset={}&order={}", self.base_url, limit, offset, order);
if let Some(ts) = offset_time {
url.push_str(&format!("&offset_time={}", ts));
}
let resp = self.client.get(&url).send().await.context("list request failed")?;
let status = resp.status();
if !status.is_success() {
let text = resp.text().await.unwrap_or_default();
anyhow::bail!("CAN list failed ({}): {}", status, text);
}
let api: ApiResponse<ListResponse> = resp.json().await.context("parse list response")?;
Ok(api.data)
}
/// List all assets (paginated, including trashed for full sync)
pub async fn list_all(
&self,
limit: i64,
offset: i64,
include_trashed: bool,
) -> Result<ListResponse> {
let mut url = format!("{}/list?limit={}&offset={}&order=asc", self.base_url, limit, offset);
if include_trashed {
url.push_str("&include_trashed=true");
}
let resp = self.client.get(&url).send().await.context("list_all request failed")?;
let status = resp.status();
if !status.is_success() {
let text = resp.text().await.unwrap_or_default();
anyhow::bail!("CAN list_all failed ({}): {}", status, text);
}
let api: ApiResponse<ListResponse> = resp.json().await.context("parse list_all response")?;
Ok(api.data)
}
/// Search assets by filters
pub async fn search(&self, params: &SearchParams) -> Result<ListResponse> {
let resp = self
.client
.get(&format!("{}/search", self.base_url))
.query(params)
.send()
.await
.context("search request failed")?;
let status = resp.status();
if !status.is_success() {
let text = resp.text().await.unwrap_or_default();
anyhow::bail!("CAN search failed ({}): {}", status, text);
}
let api: ApiResponse<ListResponse> = resp.json().await.context("parse search response")?;
Ok(api.data)
}
/// Download asset content by hash
pub async fn get_asset(&self, hash: &str) -> Result<Bytes> {
let resp = self
.client
.get(&format!("{}/asset/{}", self.base_url, hash))
.send()
.await
.context("get_asset request failed")?;
let status = resp.status();
if !status.is_success() {
let text = resp.text().await.unwrap_or_default();
anyhow::bail!("CAN get_asset failed ({}): {}", status, text);
}
resp.bytes().await.context("read asset bytes")
}
/// Get asset metadata by hash
pub async fn get_meta(&self, hash: &str) -> Result<AssetMeta> {
let resp = self
.client
.get(&format!("{}/asset/{}/meta", self.base_url, hash))
.send()
.await
.context("get_meta request failed")?;
let status = resp.status();
if !status.is_success() {
let text = resp.text().await.unwrap_or_default();
anyhow::bail!("CAN get_meta failed ({}): {}", status, text);
}
let api: ApiResponse<AssetMeta> = resp.json().await.context("parse meta response")?;
Ok(api.data)
}
/// Ingest a file into CAN service via multipart upload
pub async fn ingest(&self, content: Bytes, meta: IngestMeta) -> Result<IngestResult> {
let file_part = multipart::Part::bytes(content.to_vec())
.file_name(meta.human_file_name.clone().unwrap_or_else(|| "file".to_string()))
.mime_str(meta.mime_type.as_deref().unwrap_or("application/octet-stream"))?;
let mut form = multipart::Form::new().part("file", file_part);
if let Some(ref v) = meta.mime_type {
form = form.text("mime_type", v.clone());
}
if let Some(ref v) = meta.human_file_name {
form = form.text("human_file_name", v.clone());
}
if let Some(ref v) = meta.human_readable_path {
form = form.text("human_readable_path", v.clone());
}
if let Some(ref v) = meta.application {
form = form.text("application", v.clone());
}
if let Some(ref v) = meta.user {
form = form.text("user", v.clone());
}
if let Some(ref v) = meta.tags {
form = form.text("tags", v.clone());
}
if let Some(ref v) = meta.description {
form = form.text("description", v.clone());
}
let resp = self
.client
.post(&format!("{}/ingest", self.base_url))
.multipart(form)
.send()
.await
.context("ingest request failed")?;
let status = resp.status();
if !status.is_success() {
let text = resp.text().await.unwrap_or_default();
anyhow::bail!("CAN ingest failed ({}): {}", status, text);
}
let api: ApiResponse<IngestResult> = resp.json().await.context("parse ingest response")?;
Ok(api.data)
}
/// Update asset metadata (tags, description)
pub async fn update_meta(
&self,
hash: &str,
tags: Option<Vec<String>>,
description: Option<String>,
) -> Result<()> {
#[derive(Serialize)]
struct MetadataUpdate {
#[serde(skip_serializing_if = "Option::is_none")]
tags: Option<Vec<String>>,
#[serde(skip_serializing_if = "Option::is_none")]
description: Option<String>,
}
let resp = self
.client
.patch(&format!("{}/asset/{}", self.base_url, hash))
.json(&MetadataUpdate { tags, description })
.send()
.await
.context("update_meta request failed")?;
let status = resp.status();
if !status.is_success() {
let text = resp.text().await.unwrap_or_default();
anyhow::bail!("CAN update_meta failed ({}): {}", status, text);
}
Ok(())
}
/// Check if CAN service is reachable
pub async fn health_check(&self) -> Result<bool> {
match self.list(1, 0, "desc", None).await {
Ok(_) => Ok(true),
Err(_) => Ok(false),
}
}
}

78
examples/can-sync/src/config.rs Normal file
View File

@ -0,0 +1,78 @@
use anyhow::{Context, Result};
use serde::{Deserialize, Serialize};
use std::path::{Path, PathBuf};
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SyncConfig {
/// Base URL for the CAN service API (e.g. "http://127.0.0.1:3210/api/v1/can/0")
pub can_service_url: String,
/// Address for the CAN Sync HTTP API (e.g. "127.0.0.1:3213")
pub listen_addr: String,
/// Directory for persistent data (peer key, sync state DB)
pub data_dir: String,
/// Optional custom relay URL; null uses iroh's public relay
pub relay_url: Option<String>,
/// Seconds between fast polls for new assets
#[serde(default = "default_poll_interval")]
pub poll_interval_secs: u64,
/// Seconds between full scans of all assets
#[serde(default = "default_full_scan_interval")]
pub full_scan_interval_secs: u64,
}
fn default_poll_interval() -> u64 {
5
}
fn default_full_scan_interval() -> u64 {
300
}
impl SyncConfig {
/// Load config from a YAML file, falling back to defaults if not found
pub fn load(path: &Path) -> Result<Self> {
if path.exists() {
let contents =
std::fs::read_to_string(path).context("Failed to read config file")?;
let config: SyncConfig =
serde_yaml::from_str(&contents).context("Failed to parse config YAML")?;
Ok(config)
} else {
tracing::warn!("Config file not found at {}, using defaults", path.display());
Ok(Self::default())
}
}
/// Resolved data directory path
pub fn data_path(&self) -> PathBuf {
PathBuf::from(&self.data_dir)
}
/// Path to the peer keypair file
pub fn peer_key_path(&self) -> PathBuf {
self.data_path().join("peer_key")
}
/// Path to the sync state SQLite database
pub fn db_path(&self) -> PathBuf {
self.data_path().join("can_sync.db")
}
}
impl Default for SyncConfig {
fn default() -> Self {
Self {
can_service_url: "http://127.0.0.1:3210/api/v1/can/0".to_string(),
listen_addr: "127.0.0.1:3213".to_string(),
data_dir: "./can_sync_data".to_string(),
relay_url: None,
poll_interval_secs: default_poll_interval(),
full_scan_interval_secs: default_full_scan_interval(),
}
}
}

352
examples/can-sync/src/fetcher.rs Normal file
View File

@ -0,0 +1,352 @@
use std::sync::Arc;
use anyhow::Result;
use futures_lite::StreamExt;
use sha2::{Digest, Sha256};
use tokio::io::AsyncReadExt;
use tracing::{debug, error, info, warn};
use crate::can_client::{CanClient, IngestMeta};
use crate::library::SyncState;
use crate::manifest::AssetSyncEntry;
use crate::node::SyncNode;
/// The fetcher receives remote asset entries from iroh documents
/// and ingests them into the local CAN service.
pub struct Fetcher {
can: CanClient,
state: Arc<SyncState>,
node: Arc<SyncNode>,
}
impl Fetcher {
pub fn new(can: CanClient, state: Arc<SyncState>, node: Arc<SyncNode>) -> Self {
Self { can, state, node }
}
/// Run the fetcher — subscribes to library document events for real-time sync,
/// falls back to periodic polling for documents without active subscriptions
pub async fn run(self) {
info!("Fetcher started — watching for remote asset entries");
// Run two loops concurrently:
// 1. Subscription watcher — subscribes to active library docs
// 2. Periodic checker — catches anything missed
let poll_interval = tokio::time::interval(std::time::Duration::from_secs(10));
let sub_interval = tokio::time::interval(std::time::Duration::from_secs(5));
tokio::pin!(poll_interval);
tokio::pin!(sub_interval);
loop {
tokio::select! {
_ = poll_interval.tick() => {
if let Err(e) = self.check_for_new_entries().await {
warn!("Fetcher poll error: {:#}", e);
}
}
_ = sub_interval.tick() => {
// Try to subscribe to any library docs that we haven't subscribed to yet
if let Err(e) = self.subscribe_to_libraries().await {
debug!("Fetcher subscription check: {:#}", e);
}
}
}
}
}
/// Subscribe to document events for all libraries that have doc_ids
async fn subscribe_to_libraries(&self) -> Result<()> {
let libraries = self.state.list_libraries()?;
for lib in &libraries {
if let Some(ref doc_id_hex) = lib.doc_id {
// Open the doc and subscribe to events
let doc = match self.node.open_doc(doc_id_hex).await {
Ok(d) => d,
Err(_) => continue,
};
let mut events = match doc.subscribe().await {
Ok(e) => e,
Err(_) => continue,
};
// Spawn a task to process events from this doc
let can = self.can.clone();
let node_peer_id = self.node.peer_id();
let node = self.node.clone();
let lib_name = lib.name.clone();
tokio::spawn(async move {
while let Some(event) = events.next().await {
match event {
Ok(iroh_docs::engine::LiveEvent::InsertRemote {
entry,
content_status,
..
}) => {
let key = entry.key().to_vec();
let can_hash = String::from_utf8_lossy(&key).to_string();
if content_status == iroh_docs::ContentStatus::Complete {
// The entry value (our AssetSyncEntry) is available
// Read the entry content from the blob store
let content_hash = entry.content_hash();
let mut reader = node.blobs.reader(content_hash);
let mut buf = Vec::new();
if reader.read_to_end(&mut buf).await.is_ok() {
if let Ok(sync_entry) = AssetSyncEntry::from_bytes(&buf) {
if sync_entry.last_modified_by == node_peer_id {
continue; // Skip our own entries
}
info!(
"Received remote entry for {} in library '{}'",
&can_hash[..can_hash.len().min(12)],
lib_name
);
if let Err(e) = process_remote_entry(
&can,
&node,
&node_peer_id,
&can_hash,
sync_entry,
)
.await
{
error!(
"Error processing remote entry {}: {:#}",
&can_hash[..can_hash.len().min(12)],
e
);
}
}
}
}
}
Ok(iroh_docs::engine::LiveEvent::NeighborUp(peer)) => {
info!("Peer connected: {}", peer.fmt_short());
}
Ok(iroh_docs::engine::LiveEvent::NeighborDown(peer)) => {
info!("Peer disconnected: {}", peer.fmt_short());
}
Ok(_) => {} // Ignore other events
Err(e) => {
warn!("Document event error: {:#}", e);
break;
}
}
}
});
// Only subscribe to one doc per tick to avoid overwhelming
return Ok(());
}
}
Ok(())
}
/// Check all library documents for entries we don't have locally (polling fallback)
async fn check_for_new_entries(&self) -> Result<()> {
let libraries = self.state.list_libraries()?;
for lib in &libraries {
if let Some(ref doc_id_hex) = lib.doc_id {
let doc = match self.node.open_doc(doc_id_hex).await {
Ok(d) => d,
Err(_) => continue,
};
// Query all entries (latest per key)
let query = iroh_docs::store::Query::single_latest_per_key().build();
let entries = match doc.get_many(query).await {
Ok(e) => e,
Err(_) => continue,
};
tokio::pin!(entries);
while let Some(Ok(entry)) = entries.next().await {
let key = entry.key().to_vec();
let can_hash = String::from_utf8_lossy(&key).to_string();
// Read the entry value (AssetSyncEntry)
let content_hash = entry.content_hash();
let mut reader = self.node.blobs.reader(content_hash);
let mut buf = Vec::new();
if reader.read_to_end(&mut buf).await.is_err() {
continue;
}
let sync_entry = match AssetSyncEntry::from_bytes(&buf) {
Ok(e) => e,
Err(_) => continue,
};
// Skip our own entries
if sync_entry.last_modified_by == self.node.peer_id() {
continue;
}
// Check if already processed
if self.state.is_announced(&lib.id, &can_hash).unwrap_or(false) {
continue;
}
info!(
"Polling found remote entry for {} in library '{}'",
&can_hash[..can_hash.len().min(12)],
lib.name
);
if let Err(e) = process_remote_entry(
&self.can,
&self.node,
&self.node.peer_id(),
&can_hash,
sync_entry,
)
.await
{
error!(
"Error processing remote entry {}: {:#}",
&can_hash[..can_hash.len().min(12)],
e
);
}
// Mark as processed
let _ = self.state.mark_announced(&lib.id, &can_hash, 1);
}
}
}
Ok(())
}
}
/// Process a remote asset entry — download blob and ingest into CAN service
async fn process_remote_entry(
can: &CanClient,
node: &SyncNode,
local_peer_id: &str,
can_hash: &str,
entry: AssetSyncEntry,
) -> Result<()> {
// Skip if this is our own entry
if entry.last_modified_by == local_peer_id {
return Ok(());
}
// Check if already in local CAN service
match can.get_meta(can_hash).await {
Ok(existing) => {
// Asset exists — check if metadata needs updating
if entry.tags != existing.tags
|| entry.description != existing.description
|| entry.is_trashed != existing.is_trashed
{
info!("Updating metadata for {} from remote peer", &can_hash[..12]);
can.update_meta(
can_hash,
Some(entry.tags.clone()),
entry.description.clone(),
)
.await?;
}
return Ok(());
}
Err(_) => {
// Asset not found locally — need to fetch and ingest
}
}
info!(
"Fetching remote asset {} ({}B) from peer {}",
&can_hash[..12],
entry.size,
&entry.last_modified_by[..entry.last_modified_by.len().min(12)]
);
// Download blob via iroh
let content = download_blob(node, &entry).await?;
if content.is_empty() {
warn!("Downloaded empty blob for {} — skipping", &can_hash[..12]);
return Ok(());
}
// Verify CAN hash: SHA256(timestamp_bytes + content)
if !verify_can_hash(can_hash, entry.timestamp, &content) {
error!(
"CAN hash verification failed for {} — rejecting",
&can_hash[..12]
);
return Ok(());
}
// Ingest into local CAN service
let meta = IngestMeta {
mime_type: Some(entry.mime_type.clone()),
human_file_name: entry.human_filename.clone(),
human_readable_path: entry.human_path.clone(),
application: entry.application.clone(),
user: entry.user.clone(),
tags: if entry.tags.is_empty() {
None
} else {
Some(entry.tags.join(","))
},
description: entry.description.clone(),
};
match can.ingest(content.into(), meta).await {
Ok(result) => {
info!(
"Ingested remote asset: hash={}, filename={}",
&result.hash[..12],
result.filename
);
}
Err(e) => {
error!("Failed to ingest remote asset {}: {:#}", &can_hash[..12], e);
}
}
Ok(())
}
/// Download a blob via iroh using the blob hash from the sync entry
async fn download_blob(node: &SyncNode, entry: &AssetSyncEntry) -> Result<Vec<u8>> {
let blob_hash_str = match &entry.iroh_blob_hash {
Some(h) => h,
None => {
warn!("No iroh blob hash in sync entry — cannot download");
return Ok(Vec::new());
}
};
// Parse the BLAKE3 hash
let blob_hash: iroh_blobs::Hash = blob_hash_str
.parse()
.map_err(|_| anyhow::anyhow!("Invalid iroh blob hash: {}", &blob_hash_str[..12]))?;
// Read from the local blob store (iroh-docs should have synced it)
let mut reader = node.blobs.reader(blob_hash);
let mut buf = Vec::with_capacity(entry.size as usize);
reader.read_to_end(&mut buf).await?;
debug!(
"Downloaded blob {} ({} bytes)",
&blob_hash_str[..12],
buf.len()
);
Ok(buf)
}
/// Verify CAN hash: SHA256(timestamp_string + content) matches expected hash
fn verify_can_hash(expected_hash: &str, timestamp: i64, content: &[u8]) -> bool {
let mut hasher = Sha256::new();
hasher.update(timestamp.to_string().as_bytes());
hasher.update(content);
let computed = hex::encode(hasher.finalize());
computed == expected_hash
}

288
examples/can-sync/src/library.rs Normal file
View File

@ -0,0 +1,288 @@
use anyhow::{Context, Result};
use rusqlite::Connection;
use serde::{Deserialize, Serialize};
use crate::can_client::AssetMeta;
/// Filter criteria that determines which CAN assets belong to a library
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct LibraryFilter {
/// Match assets with this application tag
#[serde(skip_serializing_if = "Option::is_none")]
pub application: Option<String>,
/// Match assets with any of these tags
#[serde(skip_serializing_if = "Option::is_none")]
pub tags: Option<Vec<String>>,
/// Match assets from this user
#[serde(skip_serializing_if = "Option::is_none")]
pub user: Option<String>,
/// Match assets with MIME type prefix (e.g. "image/")
#[serde(skip_serializing_if = "Option::is_none")]
pub mime_prefix: Option<String>,
/// Manual list of specific hashes to include
#[serde(skip_serializing_if = "Option::is_none")]
pub hashes: Option<Vec<String>>,
}
impl LibraryFilter {
/// Check if an asset matches this filter
pub fn matches(&self, asset: &AssetMeta) -> bool {
// If hashes list is set, only match those exact hashes
if let Some(ref hashes) = self.hashes {
return hashes.contains(&asset.hash);
}
// All set criteria must match (AND logic)
if let Some(ref app) = self.application {
if asset.application.as_deref() != Some(app.as_str()) {
return false;
}
}
if let Some(ref required_tags) = self.tags {
// Asset must have at least one of the required tags
if !required_tags.iter().any(|t| asset.tags.contains(t)) {
return false;
}
}
if let Some(ref user) = self.user {
if asset.user.as_deref() != Some(user.as_str()) {
return false;
}
}
if let Some(ref prefix) = self.mime_prefix {
if !asset.mime_type.starts_with(prefix.as_str()) {
return false;
}
}
true
}
}
/// A library definition stored locally
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Library {
/// Unique library ID (UUID)
pub id: String,
/// Human-readable name
pub name: String,
/// Filter criteria
pub filter: LibraryFilter,
/// iroh document ID (namespace) — set after creation
pub doc_id: Option<String>,
/// Whether this library was created locally or joined from remote
pub is_local: bool,
/// Creation timestamp
pub created_at: i64,
}
/// Tracks which assets have been announced to which libraries.
/// Uses std::sync::Mutex because rusqlite::Connection is !Send,
/// so tokio::sync::RwLock won't work across .await points.
pub struct SyncState {
db: std::sync::Mutex<Connection>,
}
impl SyncState {
/// Open or create the sync state database
pub fn open(path: &std::path::Path) -> Result<Self> {
let db = Connection::open(path).context("open sync state DB")?;
db.execute_batch(
"
CREATE TABLE IF NOT EXISTS libraries (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
filter_json TEXT NOT NULL,
doc_id TEXT,
is_local INTEGER NOT NULL DEFAULT 1,
created_at INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS announced_assets (
library_id TEXT NOT NULL,
hash TEXT NOT NULL,
version INTEGER NOT NULL DEFAULT 1,
announced_at INTEGER NOT NULL,
PRIMARY KEY (library_id, hash),
FOREIGN KEY (library_id) REFERENCES libraries(id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS sync_state (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
",
)
.context("init sync state tables")?;
Ok(Self {
db: std::sync::Mutex::new(db),
})
}
fn lock_db(&self) -> std::sync::MutexGuard<'_, Connection> {
self.db.lock().expect("sync state DB lock poisoned")
}
// ── Library CRUD ──
pub fn save_library(&self, lib: &Library) -> Result<()> {
let db = self.lock_db();
let filter_json = serde_json::to_string(&lib.filter)?;
db.execute(
"INSERT OR REPLACE INTO libraries (id, name, filter_json, doc_id, is_local, created_at)
VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
rusqlite::params![
lib.id,
lib.name,
filter_json,
lib.doc_id,
lib.is_local as i32,
lib.created_at,
],
)?;
Ok(())
}
pub fn list_libraries(&self) -> Result<Vec<Library>> {
let db = self.lock_db();
let mut stmt =
db.prepare("SELECT id, name, filter_json, doc_id, is_local, created_at FROM libraries")?;
let libs = stmt
.query_map([], |row| {
let filter_json: String = row.get(2)?;
Ok(Library {
id: row.get(0)?,
name: row.get(1)?,
filter: serde_json::from_str(&filter_json).unwrap_or(LibraryFilter {
application: None,
tags: None,
user: None,
mime_prefix: None,
hashes: None,
}),
doc_id: row.get(3)?,
is_local: row.get::<_, i32>(4)? != 0,
created_at: row.get(5)?,
})
})?
.collect::<Result<Vec<_>, _>>()?;
Ok(libs)
}
pub fn get_library(&self, id: &str) -> Result<Option<Library>> {
let db = self.lock_db();
let mut stmt = db.prepare(
"SELECT id, name, filter_json, doc_id, is_local, created_at FROM libraries WHERE id = ?1",
)?;
let mut rows = stmt.query_map([id], |row| {
let filter_json: String = row.get(2)?;
Ok(Library {
id: row.get(0)?,
name: row.get(1)?,
filter: serde_json::from_str(&filter_json).unwrap_or(LibraryFilter {
application: None,
tags: None,
user: None,
mime_prefix: None,
hashes: None,
}),
doc_id: row.get(3)?,
is_local: row.get::<_, i32>(4)? != 0,
created_at: row.get(5)?,
})
})?;
match rows.next() {
Some(Ok(lib)) => Ok(Some(lib)),
Some(Err(e)) => Err(e.into()),
None => Ok(None),
}
}
pub fn delete_library(&self, id: &str) -> Result<()> {
let db = self.lock_db();
db.execute("DELETE FROM announced_assets WHERE library_id = ?1", [id])?;
db.execute("DELETE FROM libraries WHERE id = ?1", [id])?;
Ok(())
}
pub fn update_library_doc_id(&self, id: &str, doc_id: &str) -> Result<()> {
let db = self.lock_db();
db.execute(
"UPDATE libraries SET doc_id = ?1 WHERE id = ?2",
[doc_id, id],
)?;
Ok(())
}
// ── Asset announcement tracking ──
pub fn is_announced(&self, library_id: &str, hash: &str) -> Result<bool> {
let db = self.lock_db();
let count: i64 = db.query_row(
"SELECT COUNT(*) FROM announced_assets WHERE library_id = ?1 AND hash = ?2",
[library_id, hash],
|row| row.get(0),
)?;
Ok(count > 0)
}
pub fn get_announced_version(&self, library_id: &str, hash: &str) -> Result<Option<u64>> {
let db = self.lock_db();
let mut stmt = db.prepare(
"SELECT version FROM announced_assets WHERE library_id = ?1 AND hash = ?2",
)?;
let mut rows = stmt.query_map(rusqlite::params![library_id, hash], |row| {
row.get::<_, i64>(0)
})?;
match rows.next() {
Some(Ok(v)) => Ok(Some(v as u64)),
Some(Err(e)) => Err(e.into()),
None => Ok(None),
}
}
pub fn mark_announced(&self, library_id: &str, hash: &str, version: u64) -> Result<()> {
let db = self.lock_db();
let now = chrono::Utc::now().timestamp_millis();
db.execute(
"INSERT OR REPLACE INTO announced_assets (library_id, hash, version, announced_at)
VALUES (?1, ?2, ?3, ?4)",
rusqlite::params![library_id, hash, version as i64, now],
)?;
Ok(())
}
pub fn remove_announced(&self, library_id: &str, hash: &str) -> Result<()> {
let db = self.lock_db();
db.execute(
"DELETE FROM announced_assets WHERE library_id = ?1 AND hash = ?2",
[library_id, hash],
)?;
Ok(())
}
// ── General state ──
pub fn get_state(&self, key: &str) -> Result<Option<String>> {
let db = self.lock_db();
let mut stmt = db.prepare("SELECT value FROM sync_state WHERE key = ?1")?;
let mut rows = stmt.query_map([key], |row| row.get::<_, String>(0))?;
match rows.next() {
Some(Ok(v)) => Ok(Some(v)),
Some(Err(e)) => Err(e.into()),
None => Ok(None),
}
}
pub fn set_state(&self, key: &str, value: &str) -> Result<()> {
let db = self.lock_db();
db.execute(
"INSERT OR REPLACE INTO sync_state (key, value) VALUES (?1, ?2)",
[key, value],
)?;
Ok(())
}
}

121
examples/can-sync/src/main.rs Normal file
View File

@ -0,0 +1,121 @@
#![allow(dead_code)]
mod announcer;
mod can_client;
mod config;
mod fetcher;
mod library;
mod manifest;
mod node;
mod routes;
use std::path::PathBuf;
use std::sync::Arc;
use anyhow::{Context, Result};
use tracing::info;
use crate::announcer::Announcer;
use crate::can_client::CanClient;
use crate::config::SyncConfig;
use crate::fetcher::Fetcher;
use crate::library::SyncState;
use crate::node::SyncNode;
use crate::routes::AppState;
#[tokio::main]
async fn main() -> Result<()> {
// Initialize tracing
tracing_subscriber::fmt()
.with_env_filter(
tracing_subscriber::EnvFilter::try_from_default_env()
.unwrap_or_else(|_| "can_sync=info,iroh=warn".parse().unwrap()),
)
.init();
// Load config
let config_path = std::env::args()
.nth(1)
.map(PathBuf::from)
.unwrap_or_else(|| PathBuf::from("config.yaml"));
let config = SyncConfig::load(&config_path)?;
info!("CAN Sync starting...");
info!(" CAN service: {}", config.can_service_url);
info!(" Listen addr: {}", config.listen_addr);
info!(" Data dir: {}", config.data_dir);
// Ensure data directory exists
std::fs::create_dir_all(config.data_path())
.context("Failed to create data directory")?;
// Initialize CAN service client
let can = CanClient::new(&config.can_service_url);
// Check CAN service health
match can.health_check().await {
Ok(true) => info!("CAN service is reachable"),
Ok(false) | Err(_) => {
tracing::warn!(
"CAN service at {} is not reachable — will retry on each poll",
config.can_service_url
);
}
}
// Open sync state database
let state = SyncState::open(&config.db_path())?;
let state = Arc::new(state);
info!("Sync state DB opened at {}", config.db_path().display());
// Start iroh P2P node
let node = SyncNode::spawn(&config).await?;
let node = Arc::new(node);
info!("iroh node ID: {}", node.peer_id());
// Build shared app state
let app_state = Arc::new(AppState {
node: node.clone(),
state: state.clone(),
can: can.clone(),
});
// Start the announcer (polls CAN service for new assets)
let announcer = Announcer::new(
can.clone(),
state.clone(),
node.clone(),
config.poll_interval_secs,
config.full_scan_interval_secs,
);
tokio::spawn(async move {
announcer.run().await;
});
// Start the fetcher (receives remote assets and ingests them)
let fetcher = Fetcher::new(can.clone(), state.clone(), node.clone());
tokio::spawn(async move {
fetcher.run().await;
});
// Build HTTP router
let router = routes::build_router(app_state);
// Start HTTP server
let listener = tokio::net::TcpListener::bind(&config.listen_addr)
.await
.context("Failed to bind HTTP listener")?;
info!("CAN Sync API listening on http://{}", config.listen_addr);
// Open browser to status page
let status_url = format!("http://{}/status", config.listen_addr);
if open::that(&status_url).is_err() {
info!("Open {} in your browser to check status", status_url);
}
axum::serve(listener, router)
.await
.context("HTTP server error")?;
Ok(())
}

75
examples/can-sync/src/manifest.rs Normal file
View File

@ -0,0 +1,75 @@
use serde::{Deserialize, Serialize};
use crate::can_client::AssetMeta;
/// Entry stored in iroh documents for each synced asset.
/// Key = CAN hash, Value = serialized AssetSyncEntry
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AssetSyncEntry {
/// CAN timestamp (milliseconds since epoch)
pub timestamp: i64,
/// MIME type
pub mime_type: String,
/// Application tag
pub application: Option<String>,
/// User identity
pub user: Option<String>,
/// Tags list
pub tags: Vec<String>,
/// Description
pub description: Option<String>,
/// Original human-readable filename
pub human_filename: Option<String>,
/// Original human-readable path
pub human_path: Option<String>,
/// File size in bytes
pub size: i64,
/// Whether the asset is trashed
pub is_trashed: bool,
/// iroh blob hash (BLAKE3) for downloading via iroh
pub iroh_blob_hash: Option<String>,
/// Version counter for conflict resolution (higher wins)
pub version: u64,
/// Peer ID that last modified this entry
pub last_modified_by: String,
}
impl AssetSyncEntry {
/// Create from CAN service asset metadata
pub fn from_asset_meta(meta: &AssetMeta, peer_id: &str) -> Self {
Self {
timestamp: meta.timestamp,
mime_type: meta.mime_type.clone(),
application: meta.application.clone(),
user: meta.user.clone(),
tags: meta.tags.clone(),
description: meta.description.clone(),
human_filename: meta.human_filename.clone(),
human_path: meta.human_path.clone(),
size: meta.size,
is_trashed: meta.is_trashed,
iroh_blob_hash: None,
version: 1,
last_modified_by: peer_id.to_string(),
}
}
/// Serialize to bytes for storage in iroh document
pub fn to_bytes(&self) -> Vec<u8> {
postcard::to_allocvec(self).expect("serialize AssetSyncEntry")
}
/// Deserialize from bytes
pub fn from_bytes(bytes: &[u8]) -> anyhow::Result<Self> {
Ok(postcard::from_bytes(bytes)?)
}
/// Check if metadata differs from a CAN asset (indicates update needed)
pub fn metadata_differs(&self, meta: &AssetMeta) -> bool {
self.tags != meta.tags
|| self.description != meta.description
|| self.is_trashed != meta.is_trashed
|| self.human_filename != meta.human_filename
|| self.human_path != meta.human_path
}
}

150
examples/can-sync/src/node.rs Normal file
View File

@ -0,0 +1,150 @@
use anyhow::{Context, Result};
use iroh::protocol::Router;
use iroh::Endpoint;
use iroh_blobs::store::mem::MemStore;
use iroh_blobs::{BlobsProtocol, ALPN as BLOBS_ALPN};
use iroh_docs::api::protocol::{AddrInfoOptions, ShareMode};
use iroh_docs::protocol::Docs;
use iroh_docs::{AuthorId, DocTicket, NamespaceId, ALPN as DOCS_ALPN};
use iroh_gossip::net::Gossip;
use iroh_gossip::ALPN as GOSSIP_ALPN;
use tokio::sync::OnceCell;
use crate::config::SyncConfig;
/// Holds all iroh subsystems for the P2P node
pub struct SyncNode {
pub endpoint: Endpoint,
pub blobs: BlobsProtocol,
pub docs: Docs,
pub gossip: Gossip,
pub router: Router,
/// Cached default author ID (created once on startup)
author_id: OnceCell<AuthorId>,
}
impl SyncNode {
/// Start the iroh node with all protocol handlers
pub async fn spawn(_config: &SyncConfig) -> Result<Self> {
// Build endpoint (Ed25519 keypair auto-generated and cached)
let endpoint = Endpoint::bind()
.await
.map_err(|e| anyhow::anyhow!("Failed to bind iroh endpoint: {}", e))?;
tracing::info!(
"iroh node started — EndpointID: {}",
endpoint.id()
);
// Gossip for peer communication
let gossip = Gossip::builder().spawn(endpoint.clone());
// Blob store (in-memory — blobs are transient, CAN service is authoritative)
let mem_store = MemStore::default();
let blobs_store: &iroh_blobs::api::Store = &mem_store;
let blobs = BlobsProtocol::new(blobs_store, None);
// Document sync (CRDT-replicated key-value store)
let docs = Docs::memory()
.spawn(endpoint.clone(), blobs_store.clone(), gossip.clone())
.await
.context("Failed to spawn iroh-docs")?;
// Router accepts incoming connections and dispatches to handlers
let router = Router::builder(endpoint.clone())
.accept(BLOBS_ALPN, blobs.clone())
.accept(GOSSIP_ALPN, gossip.clone())
.accept(DOCS_ALPN, docs.clone())
.spawn();
Ok(Self {
endpoint,
blobs,
docs,
gossip,
router,
author_id: OnceCell::new(),
})
}
/// Get this node's peer ID as a string
pub fn peer_id(&self) -> String {
self.endpoint.id().to_string()
}
/// Get the node's endpoint address info for sharing
pub fn endpoint_addr(&self) -> iroh::EndpointAddr {
self.endpoint.addr()
}
/// Get or create the default author for writing to documents
pub async fn author(&self) -> Result<AuthorId> {
self.author_id
.get_or_try_init(|| async {
self.docs.author_default().await
})
.await
.copied()
}
/// Create a new iroh document and return its NamespaceId as a hex string
pub async fn create_doc(&self) -> Result<String> {
let doc = self.docs.create().await?;
let ns_id = doc.id();
Ok(hex::encode(ns_id.to_bytes()))
}
/// Open an existing document by its hex-encoded namespace ID
pub async fn open_doc(&self, doc_id_hex: &str) -> Result<iroh_docs::api::Doc> {
let ns_id = parse_namespace_id(doc_id_hex)?;
self.docs
.open(ns_id)
.await?
.ok_or_else(|| anyhow::anyhow!("Document {} not found", &doc_id_hex[..12]))
}
/// Write a key-value entry to a document
pub async fn write_to_doc(
&self,
doc_id_hex: &str,
key: &[u8],
value: &[u8],
) -> Result<()> {
let doc = self.open_doc(doc_id_hex).await?;
let author = self.author().await?;
doc.set_bytes(author, key.to_vec(), value.to_vec()).await?;
Ok(())
}
/// Generate a share ticket (DocTicket) for a document
pub async fn share_doc(&self, doc_id_hex: &str) -> Result<DocTicket> {
let doc = self.open_doc(doc_id_hex).await?;
let ticket = doc
.share(ShareMode::Write, AddrInfoOptions::RelayAndAddresses)
.await?;
Ok(ticket)
}
/// Import a document from a DocTicket, returns the namespace ID as hex
pub async fn import_doc(&self, ticket: DocTicket) -> Result<String> {
let doc = self.docs.import(ticket).await?;
let ns_id = doc.id();
Ok(hex::encode(ns_id.to_bytes()))
}
/// Graceful shutdown
pub async fn shutdown(self) -> Result<()> {
tracing::info!("Shutting down iroh node...");
self.router.shutdown().await?;
Ok(())
}
}
/// Parse a hex-encoded NamespaceId
pub fn parse_namespace_id(hex_str: &str) -> Result<NamespaceId> {
let bytes: [u8; 32] = hex::decode(hex_str)
.context("Invalid hex in doc_id")?
.try_into()
.map_err(|_| anyhow::anyhow!("doc_id must be 32 bytes (64 hex chars)"))?;
Ok(NamespaceId::from(bytes))
}

430
examples/can-sync/src/routes.rs Normal file
View File

@ -0,0 +1,430 @@
use std::sync::Arc;
use axum::{
extract::{Path, State},
http::StatusCode,
response::IntoResponse,
routing::{get, post},
Json, Router,
};
use serde::{Deserialize, Serialize};
use crate::can_client::CanClient;
use crate::library::{Library, LibraryFilter, SyncState};
use crate::node::SyncNode;
/// Shared application state for route handlers
pub struct AppState {
pub node: Arc<SyncNode>,
pub state: Arc<SyncState>,
pub can: CanClient,
}
// ── Request/Response types ──
#[derive(Serialize)]
struct StatusResponse {
peer_id: String,
can_service_healthy: bool,
library_count: usize,
}
#[derive(Serialize)]
struct PeerInfo {
peer_id: String,
}
#[derive(Deserialize)]
pub struct CreateLibraryRequest {
pub name: String,
pub filter: LibraryFilter,
}
#[derive(Serialize)]
struct LibraryResponse {
id: String,
name: String,
filter: LibraryFilter,
doc_id: Option<String>,
is_local: bool,
created_at: i64,
}
#[derive(Serialize)]
struct InviteResponse {
ticket: String,
}
#[derive(Deserialize)]
pub struct JoinRequest {
pub ticket: String,
}
#[derive(Serialize)]
struct JoinResponse {
library_id: String,
message: String,
}
#[derive(Serialize)]
struct ApiResp<T: Serialize> {
status: String,
data: T,
}
#[derive(Serialize)]
struct ApiErr {
status: String,
error: String,
}
fn ok_json<T: Serialize>(data: T) -> Json<ApiResp<T>> {
Json(ApiResp {
status: "success".to_string(),
data,
})
}
fn err_resp(status: StatusCode, msg: &str) -> (StatusCode, Json<ApiErr>) {
(
status,
Json(ApiErr {
status: "error".to_string(),
error: msg.to_string(),
}),
)
}
// ── Routes ──
pub fn build_router(app_state: Arc<AppState>) -> Router {
Router::new()
.route("/status", get(get_status))
.route("/peers", get(get_peers))
.route("/libraries", post(create_library).get(list_libraries))
.route(
"/libraries/{id}",
get(get_library).delete(delete_library),
)
.route("/libraries/{id}/invite", post(create_invite))
.route("/join", post(join_library))
.with_state(app_state)
}
// ── Handlers ──
async fn get_status(State(app): State<Arc<AppState>>) -> impl IntoResponse {
let can_healthy = app.can.health_check().await.unwrap_or(false);
let lib_count = app.state.list_libraries().unwrap_or_default().len();
ok_json(StatusResponse {
peer_id: app.node.peer_id(),
can_service_healthy: can_healthy,
library_count: lib_count,
})
.into_response()
}
async fn get_peers(State(app): State<Arc<AppState>>) -> impl IntoResponse {
let peers: Vec<PeerInfo> = vec![PeerInfo {
peer_id: app.node.peer_id(),
}];
ok_json(peers).into_response()
}
async fn create_library(
State(app): State<Arc<AppState>>,
Json(req): Json<CreateLibraryRequest>,
) -> impl IntoResponse {
// Create an iroh document for this library
let doc_id = match app.node.create_doc().await {
Ok(id) => Some(id),
Err(e) => {
tracing::warn!("Failed to create iroh document for library: {:#}", e);
None
}
};
let lib = Library {
id: uuid::Uuid::new_v4().to_string(),
name: req.name,
filter: req.filter,
doc_id,
is_local: true,
created_at: chrono::Utc::now().timestamp_millis(),
};
if let Err(e) = app.state.save_library(&lib) {
return err_resp(
StatusCode::INTERNAL_SERVER_ERROR,
&format!("save failed: {}", e),
)
.into_response();
}
tracing::info!(
"Created library '{}' (id={}, doc_id={:?})",
lib.name,
&lib.id[..8],
lib.doc_id.as_deref().map(|d| &d[..12.min(d.len())])
);
ok_json(LibraryResponse {
id: lib.id,
name: lib.name,
filter: lib.filter,
doc_id: lib.doc_id,
is_local: lib.is_local,
created_at: lib.created_at,
})
.into_response()
}
async fn list_libraries(State(app): State<Arc<AppState>>) -> impl IntoResponse {
match app.state.list_libraries() {
Ok(libs) => {
let responses: Vec<LibraryResponse> = libs
.into_iter()
.map(|lib| LibraryResponse {
id: lib.id,
name: lib.name,
filter: lib.filter,
doc_id: lib.doc_id,
is_local: lib.is_local,
created_at: lib.created_at,
})
.collect();
ok_json(responses).into_response()
}
Err(e) => {
err_resp(StatusCode::INTERNAL_SERVER_ERROR, &format!("{}", e)).into_response()
}
}
}
async fn get_library(
State(app): State<Arc<AppState>>,
Path(id): Path<String>,
) -> impl IntoResponse {
match app.state.get_library(&id) {
Ok(Some(lib)) => ok_json(LibraryResponse {
id: lib.id,
name: lib.name,
filter: lib.filter,
doc_id: lib.doc_id,
is_local: lib.is_local,
created_at: lib.created_at,
})
.into_response(),
Ok(None) => err_resp(StatusCode::NOT_FOUND, "Library not found").into_response(),
Err(e) => {
err_resp(StatusCode::INTERNAL_SERVER_ERROR, &format!("{}", e)).into_response()
}
}
}
async fn delete_library(
State(app): State<Arc<AppState>>,
Path(id): Path<String>,
) -> impl IntoResponse {
match app.state.delete_library(&id) {
Ok(()) => ok_json("deleted").into_response(),
Err(e) => {
err_resp(StatusCode::INTERNAL_SERVER_ERROR, &format!("{}", e)).into_response()
}
}
}
async fn create_invite(
State(app): State<Arc<AppState>>,
Path(id): Path<String>,
) -> impl IntoResponse {
match app.state.get_library(&id) {
Ok(Some(lib)) => {
let doc_id = match &lib.doc_id {
Some(d) => d,
None => {
return err_resp(
StatusCode::BAD_REQUEST,
"Library has no iroh document — cannot create invite",
)
.into_response()
}
};
// Generate a real DocTicket via iroh
match app.node.share_doc(doc_id).await {
Ok(ticket) => {
// DocTicket implements Display via iroh's Ticket trait (base32 serialization)
let ticket_str = ticket.to_string();
// Wrap with library metadata so the joiner knows the name and filter
let invite_data = serde_json::json!({
"ticket": ticket_str,
"library_name": lib.name,
"filter": lib.filter,
});
let invite_b64 = base64_encode(
&serde_json::to_vec(&invite_data).unwrap(),
);
ok_json(InviteResponse { ticket: invite_b64 }).into_response()
}
Err(e) => err_resp(
StatusCode::INTERNAL_SERVER_ERROR,
&format!("Failed to create invite: {}", e),
)
.into_response(),
}
}
Ok(None) => err_resp(StatusCode::NOT_FOUND, "Library not found").into_response(),
Err(e) => {
err_resp(StatusCode::INTERNAL_SERVER_ERROR, &format!("{}", e)).into_response()
}
}
}
async fn join_library(
State(app): State<Arc<AppState>>,
Json(req): Json<JoinRequest>,
) -> impl IntoResponse {
// Decode our envelope
let ticket_bytes = match base64_decode(&req.ticket) {
Ok(b) => b,
Err(_) => {
return err_resp(StatusCode::BAD_REQUEST, "Invalid ticket encoding").into_response()
}
};
let ticket_data: serde_json::Value = match serde_json::from_slice(&ticket_bytes) {
Ok(v) => v,
Err(_) => {
return err_resp(StatusCode::BAD_REQUEST, "Invalid ticket data").into_response()
}
};
// Extract the real DocTicket string
let ticket_str = match ticket_data["ticket"].as_str() {
Some(s) => s,
None => {
return err_resp(StatusCode::BAD_REQUEST, "Missing 'ticket' field in invite")
.into_response()
}
};
// Parse DocTicket from the serialized string
let doc_ticket: iroh_docs::DocTicket = match ticket_str.parse() {
Ok(t) => t,
Err(e) => {
return err_resp(
StatusCode::BAD_REQUEST,
&format!("Invalid DocTicket: {}", e),
)
.into_response()
}
};
// Import the document via iroh (starts sync with remote peers)
let doc_id_hex = match app.node.import_doc(doc_ticket).await {
Ok(id) => id,
Err(e) => {
return err_resp(
StatusCode::INTERNAL_SERVER_ERROR,
&format!("Failed to join document: {}", e),
)
.into_response()
}
};
let name = ticket_data["library_name"]
.as_str()
.unwrap_or("remote library")
.to_string();
let filter: LibraryFilter = serde_json::from_value(ticket_data["filter"].clone())
.unwrap_or(LibraryFilter {
application: None,
tags: None,
user: None,
mime_prefix: None,
hashes: None,
});
let lib = Library {
id: uuid::Uuid::new_v4().to_string(),
name: name.clone(),
filter,
doc_id: Some(doc_id_hex),
is_local: false,
created_at: chrono::Utc::now().timestamp_millis(),
};
if let Err(e) = app.state.save_library(&lib) {
return err_resp(
StatusCode::INTERNAL_SERVER_ERROR,
&format!("save failed: {}", e),
)
.into_response();
}
tracing::info!(
"Joined library '{}' (id={}, doc_id={:?})",
name,
&lib.id[..8],
lib.doc_id.as_deref().map(|d| &d[..12.min(d.len())])
);
ok_json(JoinResponse {
library_id: lib.id,
message: "Joined library successfully".to_string(),
})
.into_response()
}
// ── Base64 helpers ──
fn base64_encode(data: &[u8]) -> String {
const CHARS: &[u8] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
let mut result = Vec::new();
for chunk in data.chunks(3) {
let b0 = chunk[0] as u32;
let b1 = if chunk.len() > 1 { chunk[1] as u32 } else { 0 };
let b2 = if chunk.len() > 2 { chunk[2] as u32 } else { 0 };
let triple = (b0 << 16) | (b1 << 8) | b2;
result.push(CHARS[((triple >> 18) & 0x3F) as usize]);
result.push(CHARS[((triple >> 12) & 0x3F) as usize]);
if chunk.len() > 1 {
result.push(CHARS[((triple >> 6) & 0x3F) as usize]);
} else {
result.push(b'=');
}
if chunk.len() > 2 {
result.push(CHARS[(triple & 0x3F) as usize]);
} else {
result.push(b'=');
}
}
String::from_utf8(result).unwrap()
}
fn base64_decode(input: &str) -> Result<Vec<u8>, &'static str> {
const CHARS: &[u8] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
let input = input.trim_end_matches('=');
let bytes: Vec<u8> = input
.bytes()
.filter_map(|b| CHARS.iter().position(|&c| c == b).map(|p| p as u8))
.collect();
let mut buf = Vec::new();
for chunk in bytes.chunks(4) {
if chunk.len() >= 2 {
buf.push((chunk[0] << 2) | (chunk[1] >> 4));
}
if chunk.len() >= 3 {
buf.push((chunk[1] << 4) | (chunk[2] >> 2));
}
if chunk.len() >= 4 {
buf.push((chunk[2] << 6) | chunk[3]);
}
}
Ok(buf)
}

2377
examples/canfs/Cargo.lock generated Normal file
View File

File diff suppressed because it is too large Load Diff

27
examples/canfs/Cargo.toml Normal file
View File

@ -0,0 +1,27 @@
[package]
name = "canfs"
version = "0.1.0"
edition = "2021"
publish = false
description = "Mount CAN service assets as a virtual Windows filesystem via WinFSP"
[[bin]]
name = "canfs"
path = "src/main.rs"
[dependencies]
winfsp = "0.12"
widestring = "1"
reqwest = { version = "0.12", features = ["json", "blocking"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
chrono = "0.4"
parking_lot = "0.12"
clap = { version = "4", features = ["derive"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
anyhow = "1"
ctrlc = "3"
[build-dependencies]
winfsp = { version = "0.12", features = ["delayload"] }

3
examples/canfs/build.rs Normal file
View File

@ -0,0 +1,3 @@
fn main() {
winfsp::build::winfsp_link_delayload();
}

4
examples/canfs/run.bat Normal file
View File

@ -0,0 +1,4 @@
@echo off
set PATH=C:\Program Files (x86)\WinFsp\bin;%PATH%
cd /d "%~dp0"
cargo run -- --mount J:

104
examples/canfs/src/api.rs Normal file
View File

@ -0,0 +1,104 @@
use anyhow::{Context, Result};
use serde::Deserialize;
/// Mirrors the server's AssetMeta response type.
#[derive(Debug, Clone, Deserialize)]
#[allow(dead_code)]
pub struct AssetMeta {
pub hash: String,
pub mime_type: String,
pub application: Option<String>,
pub user: Option<String>,
pub tags: Vec<String>,
pub description: Option<String>,
pub human_filename: Option<String>,
pub human_path: Option<String>,
pub timestamp: i64,
pub is_trashed: bool,
pub is_corrupted: bool,
#[serde(default)]
pub size: i64,
}
#[derive(Debug, Deserialize)]
struct ApiResponse<T> {
#[allow(dead_code)]
status: String,
data: T,
}
#[derive(Debug, Deserialize)]
struct ListData {
items: Vec<AssetMeta>,
pagination: Pagination,
}
#[derive(Debug, Deserialize)]
struct Pagination {
#[allow(dead_code)]
limit: i64,
#[allow(dead_code)]
offset: i64,
total: i64,
}
/// Blocking HTTP client for the CAN service API.
pub struct CanClient {
client: reqwest::blocking::Client,
base_url: String,
}
impl CanClient {
pub fn new(base_url: &str) -> Self {
Self {
client: reqwest::blocking::Client::new(),
base_url: base_url.trim_end_matches('/').to_string(),
}
}
/// Fetch all non-trashed assets by paginating through the list endpoint.
pub fn list_all(&self) -> Result<Vec<AssetMeta>> {
let mut all = Vec::new();
let page_size = 500;
let mut offset = 0i64;
loop {
let url = format!(
"{}/list?limit={}&offset={}&order=desc",
self.base_url, page_size, offset
);
let resp: ApiResponse<ListData> = self
.client
.get(&url)
.send()
.context("failed to reach CAN service")?
.json()
.context("failed to parse list response")?;
let count = resp.data.items.len() as i64;
all.extend(resp.data.items);
if all.len() as i64 >= resp.data.pagination.total || count < page_size {
break;
}
offset += count;
}
// Filter out trashed and corrupted
all.retain(|a| !a.is_trashed && !a.is_corrupted);
Ok(all)
}
/// Download the raw bytes of an asset by hash.
pub fn fetch_bytes(&self, hash: &str) -> Result<Vec<u8>> {
let url = format!("{}/asset/{}", self.base_url, hash);
let bytes = self
.client
.get(&url)
.send()
.context("failed to fetch asset")?
.bytes()
.context("failed to read asset body")?;
Ok(bytes.to_vec())
}
}

316
examples/canfs/src/fs.rs Normal file
View File

@ -0,0 +1,316 @@
use std::os::raw::c_void;
use std::sync::Arc;
use parking_lot::{Mutex, RwLock};
use tracing::{debug, warn};
use widestring::U16CStr;
use winfsp::filesystem::{
DirBuffer, DirInfo, DirMarker, FileInfo, FileSecurity, FileSystemContext, OpenFileInfo,
VolumeInfo, WideNameInfo,
};
use winfsp::FspError;
use crate::api::{AssetMeta, CanClient};
use crate::tree::{NodeId, NodeKind, VirtualTree};
use crate::util;
// NTSTATUS constants (raw i32 values to avoid windows crate version conflicts)
const STATUS_OBJECT_NAME_NOT_FOUND: i32 = 0xC0000034_u32 as i32;
const STATUS_NOT_A_DIRECTORY: i32 = 0xC0000103_u32 as i32;
const STATUS_UNEXPECTED_NETWORK_ERROR: i32 = 0xC00000C4_u32 as i32;
const STATUS_INVALID_DEVICE_REQUEST: i32 = 0xC0000010_u32 as i32;
// File attribute constants
const FILE_ATTRIBUTE_DIRECTORY: u32 = 0x10;
const FILE_ATTRIBUTE_READONLY: u32 = 0x01;
const FILE_ATTRIBUTE_ARCHIVE: u32 = 0x20;
fn ntstatus(code: i32) -> FspError {
FspError::NTSTATUS(code)
}
/// Shared cache state: asset list + virtual tree.
pub struct CacheState {
pub assets: Vec<AssetMeta>,
pub tree: VirtualTree,
}
/// The WinFSP filesystem context for CAN service.
pub struct CanFs {
pub cache: Arc<RwLock<CacheState>>,
pub client: Arc<CanClient>,
}
/// Per-open-handle context.
pub struct CanFileContext {
node_id: NodeId,
/// Lazily fetched file bytes.
content: Mutex<Option<Vec<u8>>>,
/// Directory enumeration buffer.
dir_buffer: DirBuffer,
}
impl FileSystemContext for CanFs {
type FileContext = CanFileContext;
fn get_security_by_name(
&self,
file_name: &U16CStr,
_security_descriptor: Option<&mut [c_void]>,
_resolve_reparse_points: impl FnOnce(&U16CStr) -> Option<FileSecurity>,
) -> winfsp::Result<FileSecurity> {
let path = util::normalize_path(file_name);
debug!("get_security_by_name: {}", path);
let cache = self.cache.read();
let node_id = cache
.tree
.lookup(&path)
.ok_or(ntstatus(STATUS_OBJECT_NAME_NOT_FOUND))?;
let node = cache.tree.get(node_id);
let attributes = if node.is_directory() {
FILE_ATTRIBUTE_DIRECTORY
} else {
FILE_ATTRIBUTE_READONLY | FILE_ATTRIBUTE_ARCHIVE
};
Ok(FileSecurity {
reparse: false,
sz_security_descriptor: 0,
attributes,
})
}
fn open(
&self,
file_name: &U16CStr,
_create_options: u32,
_granted_access: u32,
file_info: &mut OpenFileInfo,
) -> winfsp::Result<Self::FileContext> {
let path = util::normalize_path(file_name);
debug!("open: {}", path);
let cache = self.cache.read();
let node_id = cache
.tree
.lookup(&path)
.ok_or(ntstatus(STATUS_OBJECT_NAME_NOT_FOUND))?;
let node = cache.tree.get(node_id);
let fi = file_info.as_mut();
if node.is_directory() {
fi.file_attributes = FILE_ATTRIBUTE_DIRECTORY;
fi.file_size = 0;
fi.allocation_size = 0;
} else {
fi.file_attributes = FILE_ATTRIBUTE_READONLY | FILE_ATTRIBUTE_ARCHIVE;
if let NodeKind::File { asset_index } = &node.kind {
let sz = cache.assets[*asset_index].size as u64;
fi.file_size = sz;
fi.allocation_size = sz;
} else {
fi.file_size = 0;
fi.allocation_size = 0;
}
}
if let NodeKind::File { asset_index } = &node.kind {
let ts = util::epoch_ms_to_filetime(cache.assets[*asset_index].timestamp);
fi.creation_time = ts;
fi.last_access_time = ts;
fi.last_write_time = ts;
fi.change_time = ts;
} else {
fi.creation_time = 0;
fi.last_access_time = 0;
fi.last_write_time = 0;
fi.change_time = 0;
}
fi.index_number = 0;
fi.hard_links = 0;
fi.ea_size = 0;
fi.reparse_tag = 0;
Ok(CanFileContext {
node_id,
content: Mutex::new(None),
dir_buffer: DirBuffer::new(),
})
}
fn close(&self, _context: Self::FileContext) {}
fn get_file_info(
&self,
context: &Self::FileContext,
file_info: &mut FileInfo,
) -> winfsp::Result<()> {
let cache = self.cache.read();
let node = cache.tree.get(context.node_id);
if node.is_directory() {
file_info.file_attributes = FILE_ATTRIBUTE_DIRECTORY;
file_info.file_size = 0;
file_info.allocation_size = 0;
file_info.creation_time = 0;
file_info.last_access_time = 0;
file_info.last_write_time = 0;
file_info.change_time = 0;
} else {
file_info.file_attributes = FILE_ATTRIBUTE_READONLY | FILE_ATTRIBUTE_ARCHIVE;
// Use actual downloaded size if available, otherwise metadata size
let content = context.content.lock();
if let Some(ref bytes) = *content {
let sz = bytes.len() as u64;
file_info.file_size = sz;
file_info.allocation_size = sz;
} else if let NodeKind::File { asset_index } = &node.kind {
let sz = cache.assets[*asset_index].size as u64;
file_info.file_size = sz;
file_info.allocation_size = sz;
} else {
file_info.file_size = 0;
file_info.allocation_size = 0;
}
if let NodeKind::File { asset_index } = &node.kind {
let ts = util::epoch_ms_to_filetime(cache.assets[*asset_index].timestamp);
file_info.creation_time = ts;
file_info.last_access_time = ts;
file_info.last_write_time = ts;
file_info.change_time = ts;
}
}
file_info.index_number = 0;
file_info.hard_links = 0;
file_info.ea_size = 0;
file_info.reparse_tag = 0;
Ok(())
}
fn read(
&self,
context: &Self::FileContext,
buffer: &mut [u8],
offset: u64,
) -> winfsp::Result<u32> {
let mut content = context.content.lock();
if content.is_none() {
let cache = self.cache.read();
let node = cache.tree.get(context.node_id);
if let NodeKind::File { asset_index } = &node.kind {
let hash = &cache.assets[*asset_index].hash;
debug!("fetching bytes for {}", hash);
match self.client.fetch_bytes(hash) {
Ok(bytes) => {
*content = Some(bytes);
}
Err(e) => {
warn!("failed to fetch asset: {}", e);
return Err(ntstatus(STATUS_UNEXPECTED_NETWORK_ERROR));
}
}
} else {
return Err(ntstatus(STATUS_INVALID_DEVICE_REQUEST));
}
}
let bytes = content.as_ref().unwrap();
let offset = offset as usize;
if offset >= bytes.len() {
return Ok(0);
}
let end = (offset + buffer.len()).min(bytes.len());
let count = end - offset;
buffer[..count].copy_from_slice(&bytes[offset..end]);
Ok(count as u32)
}
fn read_directory(
&self,
context: &Self::FileContext,
_pattern: Option<&U16CStr>,
marker: DirMarker,
buffer: &mut [u8],
) -> winfsp::Result<u32> {
let cache = self.cache.read();
let node = cache.tree.get(context.node_id);
if !node.is_directory() {
return Err(ntstatus(STATUS_NOT_A_DIRECTORY));
}
if let Ok(dir_buffer_lock) = context.dir_buffer.acquire(marker.is_none(), None) {
// "." entry
{
let mut di: DirInfo = DirInfo::new();
let _ = di.set_name(std::ffi::OsStr::new("."));
di.file_info_mut().file_attributes = FILE_ATTRIBUTE_DIRECTORY;
let _ = dir_buffer_lock.write(&mut di);
}
// ".." entry
{
let mut di: DirInfo = DirInfo::new();
let _ = di.set_name(std::ffi::OsStr::new(".."));
di.file_info_mut().file_attributes = FILE_ATTRIBUTE_DIRECTORY;
let _ = dir_buffer_lock.write(&mut di);
}
for &child_id in &node.children {
let child = cache.tree.get(child_id);
let mut di: DirInfo = DirInfo::new();
if di.set_name(std::ffi::OsStr::new(&child.name)).is_err() {
continue;
}
let fi = di.file_info_mut();
if child.is_directory() {
fi.file_attributes = FILE_ATTRIBUTE_DIRECTORY;
fi.file_size = 0;
fi.allocation_size = 0;
} else {
fi.file_attributes = FILE_ATTRIBUTE_READONLY | FILE_ATTRIBUTE_ARCHIVE;
if let NodeKind::File { asset_index } = &child.kind {
let sz = cache.assets[*asset_index].size as u64;
fi.file_size = sz;
fi.allocation_size = sz;
} else {
fi.file_size = 0;
fi.allocation_size = 0;
}
}
if let NodeKind::File { asset_index } = &child.kind {
let ts = util::epoch_ms_to_filetime(cache.assets[*asset_index].timestamp);
fi.creation_time = ts;
fi.last_access_time = ts;
fi.last_write_time = ts;
fi.change_time = ts;
}
fi.index_number = 0;
fi.hard_links = 0;
fi.ea_size = 0;
fi.reparse_tag = 0;
let _ = dir_buffer_lock.write(&mut di);
}
}
Ok(context.dir_buffer.read(marker, buffer))
}
fn get_volume_info(&self, out_volume_info: &mut VolumeInfo) -> winfsp::Result<()> {
out_volume_info.total_size = 1024 * 1024 * 1024; // 1 GB
out_volume_info.free_size = 0;
Ok(())
}
}

139
examples/canfs/src/main.rs Normal file
View File

@ -0,0 +1,139 @@
mod api;
mod fs;
mod tree;
mod util;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::Duration;
use clap::Parser;
use parking_lot::RwLock;
use tracing::{error, info};
use winfsp::host::{FileSystemHost, FileSystemParams, VolumeParams};
use winfsp::winfsp_init_or_die;
use crate::api::CanClient;
use crate::fs::{CacheState, CanFs};
use crate::tree::VirtualTree;
#[derive(Parser)]
#[command(name = "canfs", about = "Mount CAN service assets as a virtual drive")]
struct Args {
/// Mount point: a drive letter like "X:" or a directory path.
#[arg(short, long, default_value = "X:")]
mount: String,
/// CAN service base URL.
#[arg(long, default_value = "http://127.0.0.1:3210/api/v1/can/0")]
can_url: String,
/// Cache refresh interval in seconds.
#[arg(long, default_value = "60")]
refresh_secs: u64,
}
fn main() {
tracing_subscriber::fmt()
.with_env_filter(
tracing_subscriber::EnvFilter::try_from_default_env()
.unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")),
)
.init();
let _init = winfsp_init_or_die();
let args = Args::parse();
info!("connecting to CAN service at {}", args.can_url);
let client = Arc::new(CanClient::new(&args.can_url));
let assets = match client.list_all() {
Ok(a) => a,
Err(e) => {
error!("initial fetch failed — is the CAN service running? {}", e);
std::process::exit(1);
}
};
info!("loaded {} assets", assets.len());
let tree = VirtualTree::build(&assets);
let cache = Arc::new(RwLock::new(CacheState { assets, tree }));
// Background refresh thread
{
let cache = Arc::clone(&cache);
let client = Arc::clone(&client);
let interval = Duration::from_secs(args.refresh_secs);
std::thread::spawn(move || loop {
std::thread::sleep(interval);
match client.list_all() {
Ok(assets) => {
let tree = VirtualTree::build(&assets);
let count = assets.len();
*cache.write() = CacheState { assets, tree };
info!("cache refreshed: {} assets", count);
}
Err(e) => {
error!("cache refresh failed: {}", e);
}
}
});
}
let canfs = CanFs {
cache: Arc::clone(&cache),
client,
};
let mut volume_params = VolumeParams::new();
volume_params
.filesystem_name("CanFS")
.sector_size(512)
.sectors_per_allocation_unit(1)
.file_info_timeout(1000)
.case_sensitive_search(false)
.case_preserved_names(true)
.read_only_volume(true)
.unicode_on_disk(true)
.persistent_acls(false);
let params = FileSystemParams::default_params(volume_params);
let mut host = match FileSystemHost::new_with_options(params, canfs) {
Ok(h) => h,
Err(e) => {
error!("failed to create filesystem host: {:?}", e);
std::process::exit(1);
}
};
info!("mounting on {}", args.mount);
if let Err(e) = host.mount(std::ffi::OsStr::new(&args.mount)) {
error!("failed to mount: {:?}", e);
std::process::exit(1);
}
if let Err(e) = host.start() {
error!("failed to start: {:?}", e);
std::process::exit(1);
}
info!("CanFS mounted on {} — press Ctrl+C to unmount", args.mount);
// Wait for Ctrl+C
let running = Arc::new(AtomicBool::new(true));
{
let running = Arc::clone(&running);
ctrlc::set_handler(move || {
info!("shutting down...");
running.store(false, Ordering::SeqCst);
})
.expect("failed to set Ctrl+C handler");
}
while running.load(Ordering::SeqCst) {
std::thread::sleep(Duration::from_millis(100));
}
host.stop();
info!("unmounted");
}

259
examples/canfs/src/tree.rs Normal file
View File

@ -0,0 +1,259 @@
use std::collections::{HashMap, HashSet};
use crate::api::AssetMeta;
use crate::util;
use chrono::DateTime;
/// Unique identifier for a node in the virtual tree.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct NodeId(pub usize);
/// A node is either a directory or a file reference.
#[derive(Debug, Clone)]
pub enum NodeKind {
Directory,
/// Points to an index in the flat asset list.
File { asset_index: usize },
}
/// A node in the virtual directory tree.
#[derive(Debug, Clone)]
pub struct VNode {
pub name: String,
pub kind: NodeKind,
pub children: Vec<NodeId>,
#[allow(dead_code)]
pub parent: Option<NodeId>,
}
impl VNode {
pub fn is_directory(&self) -> bool {
matches!(self.kind, NodeKind::Directory)
}
}
/// The complete virtual directory tree built from a list of assets.
pub struct VirtualTree {
nodes: Vec<VNode>,
/// Normalized path -> NodeId lookup.
path_index: HashMap<String, NodeId>,
}
impl VirtualTree {
/// Build the virtual tree from a flat list of assets.
pub fn build(assets: &[AssetMeta]) -> Self {
let mut tree = TreeBuilder::new();
// Create top-level directories
let root = tree.root();
let can_dir = tree.add_dir("CAN", root);
let app_dir = tree.add_dir("APPLICATION", root);
let dates_dir = tree.add_dir("DATES", root);
let tags_dir = tree.add_dir("TAGS", root);
for (i, asset) in assets.iter().enumerate() {
let ext = util::mime_to_ext(&asset.mime_type);
let hash8 = &asset.hash[..asset.hash.len().min(8)];
// 1) CAN/ — always: {timestamp}_{hash8}.ext
let can_name = format!("{}_{}.{}", asset.timestamp, hash8, ext);
let can_name = util::sanitize_filename(&can_name);
tree.add_file(&can_name, can_dir, i);
// Display name for other folders: human_filename if available, else hash8.ext
let display_name = if let Some(ref hf) = asset.human_filename {
util::sanitize_filename(hf)
} else {
format!("{}.{}", hash8, ext)
};
// 2) APPLICATION/{app}/ — if application is set
if let Some(ref app) = asset.application {
let app_name = util::sanitize_filename(app);
let app_sub = tree.ensure_dir(&app_name, app_dir);
tree.add_file(&display_name, app_sub, i);
}
// 3) DATES/{year}/{month:02}/
if let Some(dt) = DateTime::from_timestamp_millis(asset.timestamp) {
let year_str = dt.format("%Y").to_string();
let month_str = dt.format("%m").to_string();
let year_dir = tree.ensure_dir(&year_str, dates_dir);
let month_dir = tree.ensure_dir(&month_str, year_dir);
tree.add_file(&display_name, month_dir, i);
}
// 4) TAGS/{tag}/
for tag in &asset.tags {
let tag_name = util::sanitize_filename(tag);
if tag_name.is_empty() {
continue;
}
let tag_sub = tree.ensure_dir(&tag_name, tags_dir);
tree.add_file(&display_name, tag_sub, i);
}
}
// Sort children and build path index
tree.finalize()
}
/// Look up a node by its normalized path (e.g., `\can\file.txt`).
pub fn lookup(&self, path: &str) -> Option<NodeId> {
let normalized = path.to_lowercase().replace('/', "\\");
let normalized = if normalized.len() > 1 && normalized.ends_with('\\') {
&normalized[..normalized.len() - 1]
} else {
&normalized
};
self.path_index.get(normalized).copied()
}
/// Get a node by ID.
pub fn get(&self, id: NodeId) -> &VNode {
&self.nodes[id.0]
}
/// Get the root node ID.
#[allow(dead_code)]
pub fn root(&self) -> NodeId {
NodeId(0)
}
}
/// Builder helper for constructing the virtual tree.
struct TreeBuilder {
nodes: Vec<VNode>,
/// Track names used per directory to resolve collisions.
dir_names: HashMap<usize, HashSet<String>>,
/// Cache for ensure_dir: (parent_id, name) -> NodeId
dir_cache: HashMap<(usize, String), NodeId>,
}
impl TreeBuilder {
fn new() -> Self {
let root = VNode {
name: String::new(),
kind: NodeKind::Directory,
children: Vec::new(),
parent: None,
};
let mut dir_names = HashMap::new();
dir_names.insert(0, HashSet::new());
Self {
nodes: vec![root],
dir_names,
dir_cache: HashMap::new(),
}
}
fn root(&self) -> NodeId {
NodeId(0)
}
/// Add a directory as a child of `parent`. Returns its NodeId.
fn add_dir(&mut self, name: &str, parent: NodeId) -> NodeId {
let id = NodeId(self.nodes.len());
self.nodes.push(VNode {
name: name.to_string(),
kind: NodeKind::Directory,
children: Vec::new(),
parent: Some(parent),
});
self.nodes[parent.0].children.push(id);
self.dir_names.insert(id.0, HashSet::new());
self.dir_names
.entry(parent.0)
.or_default()
.insert(name.to_lowercase());
self.dir_cache
.insert((parent.0, name.to_lowercase()), id);
id
}
/// Get or create a subdirectory by name under `parent`.
fn ensure_dir(&mut self, name: &str, parent: NodeId) -> NodeId {
let key = (parent.0, name.to_lowercase());
if let Some(&id) = self.dir_cache.get(&key) {
return id;
}
self.add_dir(name, parent)
}
/// Add a file node as a child of `parent`, deduplicating names.
fn add_file(&mut self, name: &str, parent: NodeId, asset_index: usize) {
let used = self.dir_names.entry(parent.0).or_default();
let lower = name.to_lowercase();
let final_name = if !used.contains(&lower) {
used.insert(lower);
name.to_string()
} else {
// Deduplicate: try _2, _3, etc.
let (stem, ext) = if let Some(dot_pos) = name.rfind('.') {
(&name[..dot_pos], &name[dot_pos..])
} else {
(name, "")
};
let mut n = 2;
loop {
let candidate = format!("{}_{}{}", stem, n, ext);
let cand_lower = candidate.to_lowercase();
if !used.contains(&cand_lower) {
used.insert(cand_lower);
break candidate;
}
n += 1;
}
};
let id = NodeId(self.nodes.len());
self.nodes.push(VNode {
name: final_name,
kind: NodeKind::File { asset_index },
children: Vec::new(),
parent: Some(parent),
});
self.nodes[parent.0].children.push(id);
}
/// Sort children and build the path index.
fn finalize(mut self) -> VirtualTree {
// Sort children by name (case-insensitive)
let names: Vec<String> = self.nodes.iter().map(|n| n.name.to_lowercase()).collect();
for node in &mut self.nodes {
node.children.sort_by(|a, b| names[a.0].cmp(&names[b.0]));
}
// Build path index by walking the tree
let mut path_index = HashMap::new();
path_index.insert("\\".to_string(), NodeId(0));
fn walk(
nodes: &[VNode],
id: NodeId,
prefix: &str,
index: &mut HashMap<String, NodeId>,
) {
for &child_id in &nodes[id.0].children {
let child = &nodes[child_id.0];
let path = if prefix == "\\" {
format!("\\{}", child.name.to_lowercase())
} else {
format!("{}\\{}", prefix, child.name.to_lowercase())
};
index.insert(path.clone(), child_id);
if child.is_directory() {
walk(nodes, child_id, &path, index);
}
}
}
walk(&self.nodes, NodeId(0), "\\", &mut path_index);
VirtualTree {
nodes: self.nodes,
path_index,
}
}
}

76
examples/canfs/src/util.rs Normal file
View File

@ -0,0 +1,76 @@
use widestring::U16CStr;
/// Convert a MIME type string to a file extension (without dot).
pub fn mime_to_ext(mime: &str) -> &'static str {
// Common overrides for types where mime_guess gives odd results
match mime {
"text/plain" => "txt",
"text/html" => "html",
"text/css" => "css",
"text/javascript" | "application/javascript" => "js",
"application/json" => "json",
"application/pdf" => "pdf",
"application/zip" => "zip",
"application/gzip" => "gz",
"image/jpeg" => "jpg",
"image/png" => "png",
"image/gif" => "gif",
"image/webp" => "webp",
"image/svg+xml" => "svg",
"audio/mpeg" => "mp3",
"audio/ogg" => "ogg",
"video/mp4" => "mp4",
"video/webm" => "webm",
"application/octet-stream" => "bin",
other => {
// Try to extract from subtype: "image/tiff" -> "tiff"
if let Some((_main, sub)) = other.split_once('/') {
let sub = sub.split('+').next().unwrap_or(sub);
let sub = sub.split('.').next_back().unwrap_or(sub);
// Leak a static string for the extension - acceptable for a small set
// In practice we'll hit the match arms above for common types
Box::leak(sub.to_string().into_boxed_str())
} else {
"bin"
}
}
}
}
/// Convert Unix epoch milliseconds to Windows FILETIME (100ns intervals since 1601-01-01).
pub fn epoch_ms_to_filetime(ts_ms: i64) -> u64 {
// Windows epoch is 1601-01-01, Unix epoch is 1970-01-01
// Difference: 11644473600 seconds
const EPOCH_DIFF_SECS: i64 = 11_644_473_600;
const TICKS_PER_SEC: i64 = 10_000_000;
const TICKS_PER_MS: i64 = 10_000;
let secs = ts_ms / 1000;
let ms_remainder = ts_ms % 1000;
((secs + EPOCH_DIFF_SECS) * TICKS_PER_SEC + ms_remainder * TICKS_PER_MS) as u64
}
/// Normalize a WinFSP U16CStr path to a lowercase String with backslash separators.
/// Strips trailing backslash (except for root "\").
pub fn normalize_path(path: &U16CStr) -> String {
let s = path.to_string_lossy();
let s = s.to_lowercase();
let s = s.replace('/', "\\");
if s.len() > 1 && s.ends_with('\\') {
s[..s.len() - 1].to_string()
} else {
s
}
}
/// Sanitize a string for use as a filename: replace invalid chars with underscore.
pub fn sanitize_filename(s: &str) -> String {
s.chars()
.map(|c| match c {
'<' | '>' | ':' | '"' | '/' | '\\' | '|' | '?' | '*' => '_',
c if c.is_control() => '_',
_ => c,
})
.collect()
}

1955
examples/filemanager/Cargo.lock generated Normal file
View File

File diff suppressed because it is too large Load Diff

20
examples/filemanager/Cargo.toml Normal file
View File

@ -0,0 +1,20 @@
[package]
name = "filemanager"
version = "0.1.0"
edition = "2021"
publish = false
description = "Web-based file manager for CAN service assets"
[[bin]]
name = "filemanager"
path = "src/main.rs"
[dependencies]
axum = "0.8"
tokio = { version = "1", features = ["full"] }
reqwest = { version = "0.12", features = ["json"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
open = "5"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }

992
examples/filemanager/src/html.rs Normal file
View File

@ -0,0 +1,992 @@
pub const INDEX_HTML: &str = r##"<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>CAN File Manager</title>
<style>
:root {
--bg: #1a1a1e;
--bg2: #222228;
--bg3: #2a2a32;
--bg-hover: #32323c;
--border: #3a3a44;
--text: #e0e0e6;
--text2: #9898a4;
--accent: #6c8cff;
--accent-dim: #4a6ad0;
--folder: #f0c040;
--mono: 'SF Mono', 'Cascadia Code', 'Consolas', monospace;
--sans: -apple-system, 'Segoe UI', system-ui, sans-serif;
--radius: 6px;
--transition: 0.15s ease;
}
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
font-family: var(--sans);
background: var(--bg);
color: var(--text);
height: 100vh;
display: flex;
flex-direction: column;
overflow: hidden;
}
/* ── Toolbar ── */
.toolbar {
display: flex;
align-items: center;
gap: 8px;
padding: 8px 12px;
background: var(--bg2);
border-bottom: 1px solid var(--border);
flex-shrink: 0;
}
.toolbar .logo {
font-weight: 700;
font-size: 14px;
color: var(--accent);
white-space: nowrap;
margin-right: 8px;
}
.search-box {
flex: 1;
max-width: 500px;
position: relative;
}
.search-box input {
width: 100%;
padding: 7px 12px 7px 32px;
background: var(--bg3);
border: 1px solid var(--border);
border-radius: var(--radius);
color: var(--text);
font-size: 13px;
outline: none;
transition: border var(--transition);
}
.search-box input:focus { border-color: var(--accent); }
.search-box .icon {
position: absolute;
left: 10px;
top: 50%;
transform: translateY(-50%);
color: var(--text2);
font-size: 13px;
pointer-events: none;
}
.toolbar-actions {
display: flex;
gap: 4px;
margin-left: auto;
}
.tb-btn {
background: var(--bg3);
border: 1px solid var(--border);
color: var(--text2);
padding: 6px 10px;
border-radius: var(--radius);
cursor: pointer;
font-size: 13px;
transition: all var(--transition);
}
.tb-btn:hover { background: var(--bg-hover); color: var(--text); }
.tb-btn.active { color: var(--accent); border-color: var(--accent-dim); }
/* ── Filter bar ── */
.filter-bar {
display: none;
gap: 8px;
padding: 8px 12px;
background: var(--bg2);
border-bottom: 1px solid var(--border);
flex-wrap: wrap;
align-items: center;
flex-shrink: 0;
}
.filter-bar.show { display: flex; }
.filter-bar label { font-size: 12px; color: var(--text2); }
.filter-bar select, .filter-bar input[type="date"] {
padding: 4px 8px;
background: var(--bg3);
border: 1px solid var(--border);
border-radius: var(--radius);
color: var(--text);
font-size: 12px;
outline: none;
}
.filter-bar select:focus, .filter-bar input[type="date"]:focus {
border-color: var(--accent);
}
.filter-clear {
background: none;
border: none;
color: var(--accent);
cursor: pointer;
font-size: 12px;
padding: 4px 8px;
}
/* ── Main layout ── */
.main {
display: flex;
flex: 1;
overflow: hidden;
}
/* ── Sidebar ── */
.sidebar {
width: 220px;
min-width: 220px;
background: var(--bg2);
border-right: 1px solid var(--border);
overflow-y: auto;
padding: 8px 0;
flex-shrink: 0;
}
.tree-item {
display: flex;
align-items: center;
padding: 4px 8px 4px calc(8px + var(--depth, 0) * 16px);
cursor: pointer;
font-size: 13px;
color: var(--text2);
transition: all var(--transition);
user-select: none;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
.tree-item:hover { background: var(--bg-hover); color: var(--text); }
.tree-item.active { background: var(--bg3); color: var(--accent); }
.tree-item .arrow {
width: 16px;
text-align: center;
font-size: 10px;
flex-shrink: 0;
transition: transform var(--transition);
}
.tree-item .arrow.open { transform: rotate(90deg); }
.tree-item .folder-icon { margin-right: 6px; flex-shrink: 0; }
.tree-children { display: none; }
.tree-children.open { display: block; }
/* ── Content area ── */
.content {
flex: 1;
display: flex;
flex-direction: column;
overflow: hidden;
}
.breadcrumb {
display: flex;
align-items: center;
gap: 4px;
padding: 8px 16px;
font-size: 13px;
color: var(--text2);
border-bottom: 1px solid var(--border);
flex-shrink: 0;
flex-wrap: wrap;
}
.breadcrumb span { cursor: pointer; transition: color var(--transition); }
.breadcrumb span:hover { color: var(--accent); }
.breadcrumb .sep { color: var(--border); cursor: default; }
.breadcrumb .sep:hover { color: var(--border); }
.breadcrumb .current { color: var(--text); cursor: default; }
.breadcrumb .current:hover { color: var(--text); }
.file-area {
flex: 1;
overflow-y: auto;
padding: 12px 16px;
}
/* ── Status ── */
.status-bar {
padding: 4px 16px;
font-size: 12px;
color: var(--text2);
border-top: 1px solid var(--border);
background: var(--bg2);
flex-shrink: 0;
}
/* ── Grid view ── */
.file-grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(160px, 1fr));
gap: 12px;
}
.file-card {
background: var(--bg2);
border: 1px solid var(--border);
border-radius: var(--radius);
padding: 8px;
cursor: pointer;
transition: all var(--transition);
overflow: hidden;
}
.file-card:hover { border-color: var(--accent-dim); transform: translateY(-1px); }
.file-card .thumb {
width: 100%;
aspect-ratio: 1;
background: var(--bg3);
border-radius: 4px;
display: flex;
align-items: center;
justify-content: center;
margin-bottom: 8px;
overflow: hidden;
}
.file-card .thumb img {
width: 100%;
height: 100%;
object-fit: cover;
border-radius: 4px;
}
.file-card .thumb .file-icon {
font-size: 36px;
opacity: 0.4;
}
.file-card .name {
font-size: 12px;
font-weight: 500;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
margin-bottom: 2px;
}
.file-card .meta {
font-size: 11px;
color: var(--text2);
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
/* folder cards */
.folder-card {
background: var(--bg2);
border: 1px solid var(--border);
border-radius: var(--radius);
padding: 12px;
cursor: pointer;
transition: all var(--transition);
display: flex;
align-items: center;
gap: 10px;
}
.folder-card:hover { border-color: var(--folder); }
.folder-card .folder-icon { font-size: 28px; }
.folder-card .folder-name { font-size: 13px; font-weight: 500; }
.folder-card .folder-count { font-size: 11px; color: var(--text2); }
/* ── List view ── */
.file-list { width: 100%; }
.file-list-header, .file-list-row {
display: grid;
grid-template-columns: 32px 1fr 90px 120px 140px;
gap: 8px;
align-items: center;
padding: 6px 8px;
font-size: 12px;
}
.file-list-header {
color: var(--text2);
font-weight: 600;
border-bottom: 1px solid var(--border);
position: sticky;
top: 0;
background: var(--bg);
z-index: 1;
}
.file-list-row {
cursor: pointer;
border-radius: var(--radius);
transition: background var(--transition);
}
.file-list-row:hover { background: var(--bg-hover); }
.file-list-row .list-icon { text-align: center; font-size: 16px; }
.file-list-row .list-name {
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
.file-list-row .list-size { color: var(--text2); text-align: right; }
.file-list-row .list-type {
color: var(--text2);
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
.file-list-row .list-date { color: var(--text2); }
/* ── Detail modal ── */
.modal-overlay {
display: none;
position: fixed;
inset: 0;
background: rgba(0,0,0,0.7);
z-index: 100;
align-items: center;
justify-content: center;
}
.modal-overlay.show { display: flex; }
.modal {
background: var(--bg2);
border: 1px solid var(--border);
border-radius: 8px;
max-width: 600px;
width: 90%;
max-height: 85vh;
overflow-y: auto;
padding: 20px;
}
.modal .close-btn {
float: right;
background: none;
border: none;
color: var(--text2);
font-size: 20px;
cursor: pointer;
}
.modal .close-btn:hover { color: var(--text); }
.modal .preview {
width: 100%;
max-height: 300px;
object-fit: contain;
border-radius: var(--radius);
background: var(--bg3);
margin: 12px 0;
}
.modal h3 { font-size: 16px; margin-bottom: 12px; }
.modal .detail-row {
display: flex;
padding: 6px 0;
font-size: 13px;
border-bottom: 1px solid var(--border);
}
.modal .detail-label {
width: 100px;
color: var(--text2);
flex-shrink: 0;
}
.modal .detail-value {
word-break: break-all;
flex: 1;
}
.modal .tag-pill {
display: inline-block;
background: var(--accent-dim);
color: #fff;
padding: 2px 8px;
border-radius: 10px;
font-size: 11px;
margin: 2px;
}
.modal .actions {
display: flex;
gap: 8px;
margin-top: 16px;
}
.modal .actions a, .modal .actions button {
padding: 8px 16px;
border-radius: var(--radius);
font-size: 13px;
text-decoration: none;
cursor: pointer;
border: 1px solid var(--border);
transition: all var(--transition);
}
.modal .actions .primary {
background: var(--accent);
color: #fff;
border-color: var(--accent);
}
.modal .actions .primary:hover { background: var(--accent-dim); }
.modal .actions .secondary {
background: var(--bg3);
color: var(--text);
}
.modal .actions .secondary:hover { background: var(--bg-hover); }
/* ── Empty state ── */
.empty-state {
text-align: center;
padding: 60px 20px;
color: var(--text2);
}
.empty-state .icon { font-size: 48px; margin-bottom: 12px; opacity: 0.3; }
.empty-state p { font-size: 14px; }
/* ── Scrollbar ── */
::-webkit-scrollbar { width: 8px; }
::-webkit-scrollbar-track { background: transparent; }
::-webkit-scrollbar-thumb { background: var(--border); border-radius: 4px; }
::-webkit-scrollbar-thumb:hover { background: var(--text2); }
</style>
</head>
<body>
<!-- Toolbar -->
<div class="toolbar">
<div class="logo">CAN Files</div>
<div class="search-box">
<span class="icon">&#128269;</span>
<input type="text" id="searchInput" placeholder="Search files...">
</div>
<div class="toolbar-actions">
<button class="tb-btn" id="filterToggle" title="Toggle filters">&#9776; Filters</button>
<button class="tb-btn active" id="gridBtn" title="Grid view">&#9638;</button>
<button class="tb-btn" id="listBtn" title="List view">&#9776;</button>
</div>
</div>
<!-- Filter bar -->
<div class="filter-bar" id="filterBar">
<label>App:</label>
<select id="filterApp"><option value="">All</option></select>
<label>Type:</label>
<select id="filterMime"><option value="">All</option></select>
<label>Tag:</label>
<select id="filterTag"><option value="">All</option></select>
<label>From:</label>
<input type="date" id="filterFrom">
<label>To:</label>
<input type="date" id="filterTo">
<button class="filter-clear" id="clearFilters">Clear</button>
</div>
<!-- Main -->
<div class="main">
<div class="sidebar" id="sidebar"></div>
<div class="content">
<div class="breadcrumb" id="breadcrumb"></div>
<div class="file-area" id="fileArea"></div>
</div>
</div>
<!-- Status -->
<div class="status-bar" id="statusBar">Loading...</div>
<!-- Detail modal -->
<div class="modal-overlay" id="modalOverlay">
<div class="modal" id="modal"></div>
</div>
<script>
// ── State ──
let allAssets = [];
let vtree = null; // virtual tree root {children: {name: node}}
let currentPath = []; // e.g. ['CAN'] or ['DATES','2025','01']
let viewMode = 'grid';
let searchQuery = '';
let activeFilters = {};
// ── Init ──
document.addEventListener('DOMContentLoaded', () => {
loadAssets();
document.getElementById('searchInput').addEventListener('input', onSearch);
document.getElementById('filterToggle').addEventListener('click', toggleFilters);
document.getElementById('gridBtn').addEventListener('click', () => setView('grid'));
document.getElementById('listBtn').addEventListener('click', () => setView('list'));
document.getElementById('clearFilters').addEventListener('click', clearFilters);
document.getElementById('modalOverlay').addEventListener('click', e => {
if (e.target === e.currentTarget) closeModal();
});
['filterApp','filterMime','filterTag','filterFrom','filterTo'].forEach(id => {
document.getElementById(id).addEventListener('change', onFilterChange);
});
document.addEventListener('keydown', e => {
if (e.key === 'Escape') closeModal();
});
});
async function loadAssets() {
try {
const r = await fetch('/fm/list?limit=10000&order=desc');
const json = await r.json();
if (json.status !== 'success') throw new Error(json.error || 'load failed');
allAssets = json.data.items.filter(a => !a.is_trashed && !a.is_corrupted);
vtree = buildVirtualTree(allAssets);
populateFilterOptions();
renderSidebar();
navigateTo([]);
updateStatus();
} catch (e) {
document.getElementById('statusBar').textContent = 'Error: ' + e.message;
}
}
// ── Virtual Tree ──
function mimeToExt(mime) {
const map = {
'application/pdf':'pdf','application/json':'json','text/plain':'txt',
'text/html':'html','text/css':'css','text/csv':'csv',
'image/jpeg':'jpg','image/png':'png','image/gif':'gif','image/webp':'webp',
'image/svg+xml':'svg','audio/mpeg':'mp3','video/mp4':'mp4',
'application/zip':'zip','application/xml':'xml',
};
return map[mime] || mime.split('/').pop() || 'bin';
}
function buildVirtualTree(assets) {
const root = { name: '', type: 'dir', children: {}, items: [] };
function ensureDir(parent, name) {
if (!parent.children[name]) {
parent.children[name] = { name, type: 'dir', children: {}, items: [] };
}
return parent.children[name];
}
function addFile(parent, fileName, asset) {
// deduplicate names
let name = fileName;
let i = 2;
while (parent.children[name]) {
const dot = fileName.lastIndexOf('.');
if (dot > 0) {
name = fileName.slice(0, dot) + '_' + i + fileName.slice(dot);
} else {
name = fileName + '_' + i;
}
i++;
}
parent.children[name] = { name, type: 'file', asset };
}
for (const asset of assets) {
const ext = mimeToExt(asset.mime_type);
const hash8 = asset.hash.slice(0, 8);
const canName = asset.timestamp + '_' + hash8 + '.' + ext;
const friendlyName = asset.human_filename || (hash8 + '.' + ext);
// CAN/
const canDir = ensureDir(root, 'CAN');
addFile(canDir, canName, asset);
// APPLICATION/
if (asset.application) {
const appRoot = ensureDir(root, 'APPLICATION');
const appDir = ensureDir(appRoot, asset.application);
addFile(appDir, friendlyName, asset);
}
// DATES/
const d = new Date(asset.timestamp);
if (!isNaN(d.getTime())) {
const datesRoot = ensureDir(root, 'DATES');
const yearDir = ensureDir(datesRoot, String(d.getFullYear()));
const monthDir = ensureDir(yearDir, String(d.getMonth() + 1).padStart(2, '0'));
addFile(monthDir, friendlyName, asset);
}
// TAGS/
if (asset.tags && asset.tags.length > 0) {
const tagsRoot = ensureDir(root, 'TAGS');
for (const tag of asset.tags) {
const tagDir = ensureDir(tagsRoot, tag);
addFile(tagDir, friendlyName, asset);
}
}
}
return root;
}
// ── Navigation ──
function getNode(path) {
let node = vtree;
for (const seg of path) {
if (!node || !node.children || !node.children[seg]) return null;
node = node.children[seg];
}
return node;
}
function navigateTo(path) {
currentPath = path;
searchQuery = '';
document.getElementById('searchInput').value = '';
renderBreadcrumb();
renderContent();
highlightSidebar();
updateStatus();
}
function renderBreadcrumb() {
const bc = document.getElementById('breadcrumb');
let html = '<span onclick="navigateTo([])">Root</span>';
for (let i = 0; i < currentPath.length; i++) {
html += '<span class="sep">&#9656;</span>';
if (i === currentPath.length - 1) {
html += '<span class="current">' + esc(currentPath[i]) + '</span>';
} else {
const p = currentPath.slice(0, i + 1);
html += '<span onclick="navigateTo(' + esc(JSON.stringify(p)) + ')">' + esc(currentPath[i]) + '</span>';
}
}
bc.innerHTML = html;
}
function renderContent() {
const area = document.getElementById('fileArea');
const node = getNode(currentPath);
if (!node || !node.children) {
area.innerHTML = '<div class="empty-state"><div class="icon">&#128194;</div><p>Empty folder</p></div>';
return;
}
// Separate dirs and files
const entries = Object.values(node.children);
let dirs = entries.filter(e => e.type === 'dir');
let files = entries.filter(e => e.type === 'file');
// Apply search filter
if (searchQuery) {
const q = searchQuery.toLowerCase();
files = files.filter(f => {
const a = f.asset;
return f.name.toLowerCase().includes(q)
|| (a.description && a.description.toLowerCase().includes(q))
|| (a.human_filename && a.human_filename.toLowerCase().includes(q))
|| a.hash.toLowerCase().startsWith(q);
});
dirs = dirs.filter(d => d.name.toLowerCase().includes(q));
}
// Apply active filters — filter files that match
if (hasActiveFilters()) {
files = files.filter(f => matchesFilters(f.asset));
}
dirs.sort((a, b) => a.name.localeCompare(b.name));
files.sort((a, b) => b.asset.timestamp - a.asset.timestamp);
if (dirs.length === 0 && files.length === 0) {
area.innerHTML = '<div class="empty-state"><div class="icon">&#128269;</div><p>No items found</p></div>';
return;
}
if (viewMode === 'grid') {
renderGrid(area, dirs, files);
} else {
renderList(area, dirs, files);
}
}
// ── Grid view ──
function renderGrid(area, dirs, files) {
let html = '<div class="file-grid">';
for (const d of dirs) {
const childCount = Object.keys(d.children).length;
const p = esc(JSON.stringify([...currentPath, d.name]));
html += '<div class="folder-card" onclick="navigateTo(' + p + ')">'
+ '<div class="folder-icon">&#128193;</div>'
+ '<div><div class="folder-name">' + esc(d.name) + '</div>'
+ '<div class="folder-count">' + childCount + ' items</div></div>'
+ '</div>';
}
for (const f of files) {
const a = f.asset;
const isImage = a.mime_type.startsWith('image/');
const thumbHtml = isImage
? '<img src="/fm/thumb/' + a.hash + '" loading="lazy" alt="">'
: '<div class="file-icon">' + fileIcon(a.mime_type) + '</div>';
html += '<div class="file-card" onclick="showDetail(\'' + a.hash + '\')">'
+ '<div class="thumb">' + thumbHtml + '</div>'
+ '<div class="name" title="' + esc(f.name) + '">' + esc(f.name) + '</div>'
+ '<div class="meta">' + formatSize(a.size) + ' &middot; ' + shortDate(a.timestamp) + '</div>'
+ '</div>';
}
html += '</div>';
area.innerHTML = html;
}
// ── List view ──
function renderList(area, dirs, files) {
let html = '<div class="file-list">'
+ '<div class="file-list-header">'
+ '<div></div><div>Name</div><div style="text-align:right">Size</div><div>Type</div><div>Date</div>'
+ '</div>';
for (const d of dirs) {
const p = esc(JSON.stringify([...currentPath, d.name]));
html += '<div class="file-list-row" onclick="navigateTo(' + p + ')">'
+ '<div class="list-icon">&#128193;</div>'
+ '<div class="list-name">' + esc(d.name) + '</div>'
+ '<div class="list-size">&mdash;</div>'
+ '<div class="list-type">Folder</div>'
+ '<div class="list-date">&mdash;</div>'
+ '</div>';
}
for (const f of files) {
const a = f.asset;
html += '<div class="file-list-row" onclick="showDetail(\'' + a.hash + '\')">'
+ '<div class="list-icon">' + fileIcon(a.mime_type) + '</div>'
+ '<div class="list-name" title="' + esc(f.name) + '">' + esc(f.name) + '</div>'
+ '<div class="list-size">' + formatSize(a.size) + '</div>'
+ '<div class="list-type">' + esc(shortMime(a.mime_type)) + '</div>'
+ '<div class="list-date">' + shortDate(a.timestamp) + '</div>'
+ '</div>';
}
html += '</div>';
area.innerHTML = html;
}
// ── Sidebar tree ──
function renderSidebar() {
if (!vtree) return;
const sb = document.getElementById('sidebar');
sb.innerHTML = buildTreeHtml(vtree, [], 0);
}
function buildTreeHtml(node, path, depth) {
if (!node.children) return '';
const dirs = Object.values(node.children).filter(c => c.type === 'dir');
dirs.sort((a, b) => a.name.localeCompare(b.name));
let html = '';
for (const d of dirs) {
const cPath = [...path, d.name];
const key = cPath.join('/');
const hasSubs = Object.values(d.children).some(c => c.type === 'dir');
const arrowCls = hasSubs ? 'arrow' : 'arrow';
const arrowChar = hasSubs ? '&#9656;' : '';
html += '<div class="tree-item" data-path="' + esc(key) + '" '
+ 'style="--depth:' + depth + '" '
+ 'onclick="onTreeClick(event, ' + esc(JSON.stringify(cPath)) + ')">'
+ '<span class="' + arrowCls + '">' + arrowChar + '</span>'
+ '<span class="folder-icon">&#128193;</span>'
+ esc(d.name)
+ '</div>';
if (hasSubs) {
html += '<div class="tree-children" data-path="' + esc(key) + '">';
html += buildTreeHtml(d, cPath, depth + 1);
html += '</div>';
}
}
return html;
}
function onTreeClick(e, path) {
e.stopPropagation();
const key = path.join('/');
// toggle expand
const children = document.querySelector('.tree-children[data-path="' + key + '"]');
const arrow = e.currentTarget.querySelector('.arrow');
if (children) {
children.classList.toggle('open');
if (arrow) arrow.classList.toggle('open');
}
navigateTo(path);
}
function highlightSidebar() {
const key = currentPath.join('/');
document.querySelectorAll('.tree-item').forEach(el => {
el.classList.toggle('active', el.dataset.path === key);
});
// auto-expand parents
for (let i = 1; i <= currentPath.length; i++) {
const parentKey = currentPath.slice(0, i).join('/');
const ch = document.querySelector('.tree-children[data-path="' + parentKey + '"]');
if (ch) ch.classList.add('open');
const arrow = document.querySelector('.tree-item[data-path="' + parentKey + '"] .arrow');
if (arrow) arrow.classList.add('open');
}
}
// ── Search ──
function onSearch(e) {
searchQuery = e.target.value.trim();
renderContent();
updateStatus();
}
// ── Filters ──
function toggleFilters() {
document.getElementById('filterBar').classList.toggle('show');
document.getElementById('filterToggle').classList.toggle('active');
}
function populateFilterOptions() {
const apps = new Set(), mimes = new Set(), tags = new Set();
for (const a of allAssets) {
if (a.application) apps.add(a.application);
mimes.add(a.mime_type);
if (a.tags) a.tags.forEach(t => tags.add(t));
}
fillSelect('filterApp', [...apps].sort());
fillSelect('filterMime', [...mimes].sort());
fillSelect('filterTag', [...tags].sort());
}
function fillSelect(id, values) {
const sel = document.getElementById(id);
const current = sel.value;
sel.innerHTML = '<option value="">All</option>';
for (const v of values) {
sel.innerHTML += '<option value="' + esc(v) + '">' + esc(v) + '</option>';
}
sel.value = current;
}
function onFilterChange() {
activeFilters = {
application: document.getElementById('filterApp').value,
mime_type: document.getElementById('filterMime').value,
tag: document.getElementById('filterTag').value,
from: document.getElementById('filterFrom').value,
to: document.getElementById('filterTo').value,
};
renderContent();
updateStatus();
}
function clearFilters() {
['filterApp','filterMime','filterTag','filterFrom','filterTo'].forEach(id => {
document.getElementById(id).value = '';
});
activeFilters = {};
renderContent();
updateStatus();
}
function hasActiveFilters() {
return Object.values(activeFilters).some(v => v);
}
function matchesFilters(asset) {
const f = activeFilters;
if (f.application && asset.application !== f.application) return false;
if (f.mime_type && asset.mime_type !== f.mime_type) return false;
if (f.tag && (!asset.tags || !asset.tags.includes(f.tag))) return false;
if (f.from) {
const from = new Date(f.from).getTime();
if (asset.timestamp < from) return false;
}
if (f.to) {
const to = new Date(f.to).getTime() + 86400000; // end of day
if (asset.timestamp >= to) return false;
}
return true;
}
// ── Detail modal ──
function showDetail(hash) {
const asset = allAssets.find(a => a.hash === hash);
if (!asset) return;
const isImage = asset.mime_type.startsWith('image/');
const previewHtml = isImage
? '<img class="preview" src="/fm/asset/' + hash + '" alt="preview">'
: '';
let tagsHtml = '';
if (asset.tags && asset.tags.length) {
tagsHtml = asset.tags.map(t => '<span class="tag-pill">' + esc(t) + '</span>').join(' ');
}
const modal = document.getElementById('modal');
modal.innerHTML = '<button class="close-btn" onclick="closeModal()">&times;</button>'
+ '<h3>' + esc(asset.human_filename || asset.hash.slice(0, 12)) + '</h3>'
+ previewHtml
+ '<div class="detail-row"><div class="detail-label">Hash</div><div class="detail-value" style="font-family:var(--mono);font-size:12px">' + esc(asset.hash) + '</div></div>'
+ '<div class="detail-row"><div class="detail-label">Type</div><div class="detail-value">' + esc(asset.mime_type) + '</div></div>'
+ '<div class="detail-row"><div class="detail-label">Size</div><div class="detail-value">' + formatSize(asset.size) + '</div></div>'
+ '<div class="detail-row"><div class="detail-label">Date</div><div class="detail-value">' + fullDate(asset.timestamp) + '</div></div>'
+ (asset.application ? '<div class="detail-row"><div class="detail-label">Application</div><div class="detail-value">' + esc(asset.application) + '</div></div>' : '')
+ (asset.user ? '<div class="detail-row"><div class="detail-label">User</div><div class="detail-value">' + esc(asset.user) + '</div></div>' : '')
+ (asset.description ? '<div class="detail-row"><div class="detail-label">Description</div><div class="detail-value">' + esc(asset.description) + '</div></div>' : '')
+ (tagsHtml ? '<div class="detail-row"><div class="detail-label">Tags</div><div class="detail-value">' + tagsHtml + '</div></div>' : '')
+ '<div class="actions">'
+ '<a class="primary" href="/fm/asset/' + hash + '" target="_blank">Open</a>'
+ '<a class="secondary" href="/fm/asset/' + hash + '" download>Download</a>'
+ '</div>';
document.getElementById('modalOverlay').classList.add('show');
}
function closeModal() {
document.getElementById('modalOverlay').classList.remove('show');
}
// ── View toggle ──
function setView(mode) {
viewMode = mode;
document.getElementById('gridBtn').classList.toggle('active', mode === 'grid');
document.getElementById('listBtn').classList.toggle('active', mode === 'list');
renderContent();
}
// ── Status bar ──
function updateStatus() {
const node = getNode(currentPath);
if (!node) {
document.getElementById('statusBar').textContent = allAssets.length + ' total assets';
return;
}
const entries = Object.values(node.children || {});
const dirs = entries.filter(e => e.type === 'dir').length;
const files = entries.filter(e => e.type === 'file').length;
let text = '';
if (dirs > 0) text += dirs + ' folder' + (dirs !== 1 ? 's' : '');
if (files > 0) text += (text ? ', ' : '') + files + ' file' + (files !== 1 ? 's' : '');
if (!text) text = 'Empty';
text += ' | ' + allAssets.length + ' total assets';
document.getElementById('statusBar').textContent = text;
}
// ── Helpers ──
function esc(s) {
if (!s) return '';
return s.replace(/&/g,'&amp;').replace(/</g,'&lt;').replace(/>/g,'&gt;').replace(/"/g,'&quot;');
}
function formatSize(bytes) {
if (!bytes || bytes === 0) return '0 B';
const units = ['B', 'KB', 'MB', 'GB'];
let i = 0;
let v = bytes;
while (v >= 1024 && i < units.length - 1) { v /= 1024; i++; }
return (i === 0 ? v : v.toFixed(1)) + ' ' + units[i];
}
function shortDate(ts) {
const d = new Date(ts);
return d.toLocaleDateString(undefined, { year:'numeric', month:'short', day:'numeric' });
}
function fullDate(ts) {
return new Date(ts).toLocaleString();
}
function shortMime(mime) {
const parts = mime.split('/');
return parts.length > 1 ? parts[1] : mime;
}
function fileIcon(mime) {
if (mime.startsWith('image/')) return '&#128247;';
if (mime.startsWith('video/')) return '&#127909;';
if (mime.startsWith('audio/')) return '&#127925;';
if (mime === 'application/pdf') return '&#128196;';
if (mime.startsWith('text/')) return '&#128196;';
if (mime === 'application/json') return '&#128196;';
return '&#128196;';
}
</script>
</body>
</html>"##;

161
examples/filemanager/src/main.rs Normal file
View File

@ -0,0 +1,161 @@
mod html;
use axum::extract::{Path, Query, State};
use axum::http::{HeaderMap, HeaderValue, StatusCode};
use axum::response::{Html, IntoResponse, Response};
use axum::routing::get;
use axum::Router;
use std::collections::HashMap;
const CAN_API: &str = "http://127.0.0.1:3210/api/v1/can/0";
#[derive(Clone)]
struct AppState {
client: reqwest::Client,
}
#[tokio::main]
async fn main() {
tracing_subscriber::fmt()
.with_env_filter(
tracing_subscriber::EnvFilter::try_from_default_env()
.unwrap_or_else(|_| "filemanager=info".into()),
)
.init();
let state = AppState {
client: reqwest::Client::new(),
};
let app = Router::new()
.route("/", get(serve_index))
.route("/fm/list", get(proxy_list))
.route("/fm/search", get(proxy_search))
.route("/fm/asset/{hash}", get(proxy_asset))
.route("/fm/asset/{hash}/meta", get(proxy_meta))
.route("/fm/thumb/{hash}", get(proxy_thumb))
.with_state(state);
let addr = "127.0.0.1:3212";
let url = format!("http://{}", addr);
tracing::info!("File Manager listening on {}", url);
// Best-effort browser open
let _ = open::that(&url);
let listener = tokio::net::TcpListener::bind(addr).await.unwrap();
axum::serve(listener, app).await.unwrap();
}
async fn serve_index() -> Html<&'static str> {
Html(html::INDEX_HTML)
}
/// Forward a reqwest response to the axum caller, preserving status + content-type.
async fn forward(resp: Result<reqwest::Response, reqwest::Error>) -> Response {
match resp {
Ok(r) => {
let status = StatusCode::from_u16(r.status().as_u16()).unwrap_or(StatusCode::OK);
let ct = r
.headers()
.get("content-type")
.and_then(|v| v.to_str().ok())
.unwrap_or("application/octet-stream")
.to_string();
let body = r.bytes().await.unwrap_or_default();
let mut headers = HeaderMap::new();
headers.insert("content-type", HeaderValue::from_str(&ct).unwrap());
(status, headers, body).into_response()
}
Err(e) => {
tracing::warn!("CAN service error: {}", e);
(StatusCode::BAD_GATEWAY, "CAN service unavailable").into_response()
}
}
}
/// Build a query string from a HashMap of params.
fn build_qs(params: &HashMap<String, String>) -> String {
if params.is_empty() {
return String::new();
}
let qs: Vec<String> = params
.iter()
.map(|(k, v)| format!("{}={}", urlencoding(k), urlencoding(v)))
.collect();
format!("?{}", qs.join("&"))
}
fn urlencoding(s: &str) -> String {
s.chars()
.map(|c| match c {
'A'..='Z' | 'a'..='z' | '0'..='9' | '-' | '_' | '.' | '~' => c.to_string(),
_ => format!("%{:02X}", c as u8),
})
.collect()
}
/// Proxy list: pass all query params through to CAN /list.
async fn proxy_list(
State(state): State<AppState>,
Query(params): Query<HashMap<String, String>>,
) -> Response {
let url = format!("{}/list{}", CAN_API, build_qs(&params));
forward(state.client.get(&url).send().await).await
}
/// Proxy search: pass all query params through to CAN /search.
async fn proxy_search(
State(state): State<AppState>,
Query(params): Query<HashMap<String, String>>,
) -> Response {
let url = format!("{}/search{}", CAN_API, build_qs(&params));
forward(state.client.get(&url).send().await).await
}
/// Proxy asset download.
async fn proxy_asset(
State(state): State<AppState>,
Path(hash): Path<String>,
) -> Response {
let url = format!("{}/asset/{}", CAN_API, hash);
let resp = state.client.get(&url).send().await;
// Drop Content-Disposition so images render inline
match resp {
Ok(r) => {
let status = StatusCode::from_u16(r.status().as_u16()).unwrap_or(StatusCode::OK);
let ct = r
.headers()
.get("content-type")
.and_then(|v| v.to_str().ok())
.unwrap_or("application/octet-stream")
.to_string();
let body = r.bytes().await.unwrap_or_default();
let mut headers = HeaderMap::new();
headers.insert("content-type", HeaderValue::from_str(&ct).unwrap());
(status, headers, body).into_response()
}
Err(e) => {
tracing::warn!("CAN service error: {}", e);
(StatusCode::BAD_GATEWAY, "CAN service unavailable").into_response()
}
}
}
/// Proxy asset metadata.
async fn proxy_meta(
State(state): State<AppState>,
Path(hash): Path<String>,
) -> Response {
let url = format!("{}/asset/{}/meta", CAN_API, hash);
forward(state.client.get(&url).send().await).await
}
/// Proxy thumbnail (200x200).
async fn proxy_thumb(
State(state): State<AppState>,
Path(hash): Path<String>,
) -> Response {
let url = format!("{}/asset/{}/thumb/200/200", CAN_API, hash);
forward(state.client.get(&url).send().await).await
}

2003
examples/paste/Cargo.lock generated Normal file
View File

File diff suppressed because it is too large Load Diff

20
examples/paste/Cargo.toml Normal file
View File

@ -0,0 +1,20 @@
[package]
name = "paste"
version = "0.1.0"
edition = "2021"
publish = false
description = "Clipboard log UI — example app for CanService"
[[bin]]
name = "paste"
path = "src/main.rs"
[dependencies]
axum = { version = "0.8", features = ["multipart"] }
tokio = { version = "1", features = ["full"] }
reqwest = { version = "0.12", features = ["multipart", "json"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
open = "5"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }

382
examples/paste/src/html.rs Normal file
View File

@ -0,0 +1,382 @@
pub const INDEX_HTML: &str = r##"<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>paste</title>
<style>
:root {
--bg: #1a1a1e;
--bg-card: #24242a;
--bg-input: #2c2c34;
--border: #38383f;
--text: #e4e4e8;
--text-muted: #8888a0;
--accent: #6c8cff;
--accent-dim: #4a6ad0;
--success: #4caf80;
--error: #e05555;
--radius: 6px;
--mono: "SF Mono", "Cascadia Code", "Consolas", monospace;
--sans: -apple-system, "Segoe UI", system-ui, sans-serif;
}
*, *::before, *::after { margin: 0; padding: 0; box-sizing: border-box; }
body {
background: var(--bg);
color: var(--text);
font-family: var(--sans);
line-height: 1.5;
max-width: 720px;
margin: 0 auto;
padding: 2rem 1rem;
}
header { margin-bottom: 1.5rem; }
header h1 {
font-size: 1.4rem;
font-weight: 600;
color: var(--accent);
font-family: var(--mono);
letter-spacing: -0.02em;
}
header .sub {
color: var(--text-muted);
font-size: 0.82rem;
margin-top: 0.2rem;
}
.input-area { margin-bottom: 1.5rem; }
.input-row {
display: flex;
gap: 0.5rem;
align-items: center;
}
#paste-input {
flex: 1;
padding: 0.7rem 1rem;
background: var(--bg-input);
border: 1px solid var(--border);
border-radius: var(--radius);
color: var(--text);
font-size: 0.95rem;
font-family: var(--sans);
outline: none;
transition: border-color 0.15s;
}
#paste-input:focus { border-color: var(--accent); }
#paste-input::placeholder { color: var(--text-muted); }
.clip-btn {
flex-shrink: 0;
width: 38px;
height: 38px;
background: var(--bg-input);
border: 1px solid var(--border);
border-radius: var(--radius);
color: var(--text-muted);
cursor: pointer;
display: flex;
align-items: center;
justify-content: center;
transition: border-color 0.15s, color 0.15s;
}
.clip-btn:hover { border-color: var(--accent); color: var(--accent); }
.clip-btn svg { width: 18px; height: 18px; }
.status {
font-size: 0.78rem;
margin-top: 0.35rem;
min-height: 1.2em;
color: var(--text-muted);
}
.status.ok { color: var(--success); }
.status.err { color: var(--error); }
.items { display: flex; flex-direction: column; gap: 0.5rem; }
.item {
background: var(--bg-card);
border: 1px solid var(--border);
border-radius: var(--radius);
padding: 0.65rem 0.85rem;
display: flex;
gap: 0.7rem;
align-items: flex-start;
}
.item-thumb {
flex-shrink: 0;
width: 48px;
height: 48px;
border-radius: 4px;
object-fit: cover;
background: var(--bg);
cursor: pointer;
}
.item-icon {
flex-shrink: 0;
width: 48px;
height: 48px;
border-radius: 4px;
background: var(--bg);
display: flex;
align-items: center;
justify-content: center;
cursor: pointer;
}
.item-icon svg { width: 26px; height: 26px; }
.item-icon.pdf { color: #e05555; }
.item-icon.file { color: var(--text-muted); }
.item-filename {
font-size: 0.78rem;
margin-top: 0.2rem;
}
.item-filename a {
color: var(--accent);
text-decoration: none;
}
.item-filename a:hover { text-decoration: underline; }
.item-body { flex: 1; min-width: 0; }
.item-meta {
display: flex;
flex-wrap: wrap;
gap: 0.6rem;
font-size: 0.73rem;
color: var(--text-muted);
margin-bottom: 0.25rem;
}
.item-meta .hash {
font-family: var(--mono);
color: var(--accent-dim);
}
.item-content {
font-size: 0.88rem;
white-space: pre-wrap;
word-break: break-word;
max-height: 6em;
overflow: hidden;
}
.item-image { margin-top: 0.4rem; }
.item-image img {
max-width: 100%;
max-height: 200px;
border-radius: 4px;
}
.item-tags {
display: flex;
flex-wrap: wrap;
gap: 0.3rem;
margin-top: 0.3rem;
}
.tag {
font-size: 0.7rem;
font-family: var(--mono);
background: #2a2f45;
color: var(--accent);
padding: 0.1rem 0.45rem;
border-radius: 3px;
}
.empty {
text-align: center;
color: var(--text-muted);
padding: 3rem 1rem;
font-size: 0.9rem;
}
</style>
</head>
<body>
<header>
<h1>paste</h1>
<p class="sub">type text + enter, or paste an image &mdash; use #hashtags to add tags</p>
</header>
<main>
<div class="input-area">
<div class="input-row">
<input type="text" id="paste-input"
placeholder="type something and press Enter, or Ctrl+V an image"
autocomplete="off" spellcheck="false">
<button class="clip-btn" id="clip-btn" title="Attach a file">
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path d="M21.44 11.05l-9.19 9.19a6 6 0 01-8.49-8.49l9.19-9.19a4 4 0 015.66 5.66l-9.2 9.19a2 2 0 01-2.83-2.83l8.49-8.48"/>
</svg>
</button>
<input type="file" id="file-input" hidden>
</div>
<div id="status" class="status"></div>
</div>
<div id="items" class="items"></div>
</main>
<script>
const $ = s => document.querySelector(s);
const input = $('#paste-input');
const statusEl = $('#status');
const itemsEl = $('#items');
function setStatus(msg, type) {
statusEl.textContent = msg;
statusEl.className = 'status' + (type ? ' ' + type : '');
if (type === 'ok') setTimeout(() => { statusEl.textContent = ''; statusEl.className = 'status'; }, 2500);
}
function escapeHtml(s) {
const d = document.createElement('div');
d.textContent = s;
return d.innerHTML;
}
function fmtTime(ts) {
const d = new Date(ts);
const pad = n => String(n).padStart(2, '0');
return d.getFullYear() + '-' + pad(d.getMonth()+1) + '-' + pad(d.getDate())
+ ' ' + pad(d.getHours()) + ':' + pad(d.getMinutes()) + ':' + pad(d.getSeconds());
}
async function pasteText(text) {
setStatus('Sending...');
try {
const r = await fetch('/paste/text', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ text })
});
if (!r.ok) throw new Error(await r.text());
setStatus('Saved', 'ok');
loadItems();
} catch (e) {
setStatus('Error: ' + e.message, 'err');
}
}
async function pasteFile(file, description, fileName) {
setStatus('Uploading...');
try {
const form = new FormData();
form.append('file', file, fileName || file.name || 'clipboard.bin');
if (description) form.append('description', description);
const r = await fetch('/paste/file', { method: 'POST', body: form });
if (!r.ok) throw new Error(await r.text());
setStatus('Saved', 'ok');
loadItems();
} catch (e) {
setStatus('Error: ' + e.message, 'err');
}
}
async function loadItems() {
try {
const r = await fetch('/paste/list');
const json = await r.json();
if (json.status !== 'success') throw new Error(json.error || 'load failed');
renderItems(json.data.items);
} catch (e) {
setStatus('Load error: ' + e.message, 'err');
}
}
function renderItems(items) {
if (!items || items.length === 0) {
itemsEl.innerHTML = '<div class="empty">Nothing here yet. Type something or paste an image.</div>';
return;
}
const pdfSvg = '<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"><path d="M14 2H6a2 2 0 00-2 2v16a2 2 0 002 2h12a2 2 0 002-2V8z"/><polyline points="14 2 14 8 20 8"/><text x="12" y="17" text-anchor="middle" fill="currentColor" stroke="none" font-size="6" font-weight="bold" font-family="sans-serif">PDF</text></svg>';
const fileSvg = '<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"><path d="M14 2H6a2 2 0 00-2 2v16a2 2 0 002 2h12a2 2 0 002-2V8z"/><polyline points="14 2 14 8 20 8"/><line x1="16" y1="13" x2="8" y2="13"/><line x1="16" y1="17" x2="8" y2="17"/></svg>';
itemsEl.innerHTML = items.map(it => {
const time = fmtTime(it.timestamp);
const shortHash = it.hash.substring(0, 12);
const isImage = it.mime_type && it.mime_type.startsWith('image/');
const isPdf = it.mime_type === 'application/pdf';
const isText = it.mime_type && it.mime_type.startsWith('text/');
const assetUrl = '/paste/asset/' + it.hash;
const thumbUrl = '/paste/thumb/' + it.hash;
let thumb = '';
if (isImage) {
thumb = '<a href="' + assetUrl + '" target="_blank"><img class="item-thumb" src="' + thumbUrl + '" alt=""></a>';
} else if (isPdf) {
thumb = '<a href="' + assetUrl + '" target="_blank" class="item-icon pdf">' + pdfSvg + '</a>';
} else if (!isText) {
thumb = '<a href="' + assetUrl + '" target="_blank" class="item-icon file">' + fileSvg + '</a>';
}
let content = '';
if (isImage) {
content = '<div class="item-image"><a href="' + assetUrl + '" target="_blank"><img src="' + assetUrl + '" alt="pasted image" loading="lazy"></a></div>';
}
if (it.description) {
content += '<div class="item-content">' + escapeHtml(it.description) + '</div>';
}
let fileLink = '';
const fname = it.human_filename || it.human_file_name;
if (fname && !isText) {
fileLink = '<div class="item-filename"><a href="' + assetUrl + '" target="_blank">' + escapeHtml(fname) + '</a></div>';
}
let tagsHtml = '';
if (it.tags && it.tags.length > 0) {
tagsHtml = '<div class="item-tags">'
+ it.tags.map(t => '<span class="tag">#' + escapeHtml(t) + '</span>').join('')
+ '</div>';
}
return '<div class="item">'
+ thumb
+ '<div class="item-body">'
+ '<div class="item-meta">'
+ '<span class="time">' + time + '</span>'
+ '<span class="hash">' + shortHash + '</span>'
+ '<span>' + escapeHtml(it.mime_type || '') + '</span>'
+ '</div>'
+ fileLink
+ content
+ tagsHtml
+ '</div>'
+ '</div>';
}).join('');
}
// Enter key sends text
input.addEventListener('keydown', e => {
if (e.key === 'Enter' && input.value.trim()) {
pasteText(input.value.trim());
input.value = '';
}
});
// Clipboard paste: intercept images
document.addEventListener('paste', e => {
const items = e.clipboardData && e.clipboardData.items;
if (!items) return;
for (const item of items) {
if (item.type.startsWith('image/')) {
e.preventDefault();
const blob = item.getAsFile();
if (blob) {
const ext = blob.type === 'image/png' ? '.png'
: blob.type === 'image/jpeg' ? '.jpg'
: blob.type === 'image/gif' ? '.gif'
: blob.type === 'image/webp' ? '.webp' : '.bin';
pasteFile(blob, input.value.trim(), 'clipboard' + ext);
input.value = '';
}
return;
}
}
// text paste: let browser fill the input naturally
});
// Paperclip file picker
const fileInput = $('#file-input');
$('#clip-btn').addEventListener('click', () => fileInput.click());
fileInput.addEventListener('change', () => {
const file = fileInput.files[0];
if (file) {
pasteFile(file, input.value.trim());
input.value = '';
fileInput.value = '';
}
});
// Initial load
loadItems();
</script>
</body>
</html>"##;

263
examples/paste/src/main.rs Normal file
View File

@ -0,0 +1,263 @@
mod html;
use axum::extract::{DefaultBodyLimit, Multipart, Path, State};
use axum::http::{header, StatusCode};
use axum::response::{Html, IntoResponse, Response};
use axum::routing::{get, post};
use axum::{Json, Router};
use serde::Deserialize;
use std::net::SocketAddr;
const CAN_API: &str = "http://127.0.0.1:3210/api/v1/can/0";
#[derive(Clone)]
struct AppState {
client: reqwest::Client,
}
#[derive(Deserialize)]
struct PasteTextRequest {
text: String,
}
// ── Helpers ──────────────────────────────────────────────────────────────
/// Extract #hashtags from text, returning the comma-separated tag string.
/// e.g. "some #chicken and #food" -> "chicken,food"
fn extract_tags(text: &str) -> String {
text.split_whitespace()
.filter(|w| w.starts_with('#') && w.len() > 1)
.map(|w| w[1..].trim_end_matches(|c: char| !c.is_alphanumeric() && c != '_' && c != '-'))
.filter(|t| !t.is_empty())
.collect::<Vec<_>>()
.join(",")
}
/// Convert a reqwest response into an axum response, copying status +
/// content-type + body. Intentionally drops Content-Disposition so that
/// images render inline rather than triggering a download.
async fn forward(resp: Result<reqwest::Response, reqwest::Error>) -> Response {
match resp {
Ok(r) => {
let status = StatusCode::from_u16(r.status().as_u16())
.unwrap_or(StatusCode::BAD_GATEWAY);
let ct = r
.headers()
.get("content-type")
.and_then(|v| v.to_str().ok())
.unwrap_or("application/octet-stream")
.to_string();
let bytes = r.bytes().await.unwrap_or_default();
(status, [(header::CONTENT_TYPE, ct)], bytes).into_response()
}
Err(e) => (
StatusCode::BAD_GATEWAY,
format!("CanService unreachable: {e}"),
)
.into_response(),
}
}
// ── Handlers ─────────────────────────────────────────────────────────────
async fn serve_index() -> Html<&'static str> {
Html(html::INDEX_HTML)
}
/// Accept `{ "text": "..." }` from the frontend, forward to CanService as
/// a multipart text/plain file so the stored content is raw text (not
/// JSON-wrapped).
async fn paste_text(
State(state): State<AppState>,
Json(body): Json<PasteTextRequest>,
) -> Response {
let desc = if body.text.len() > 200 {
format!("{}...", &body.text[..body.text.char_indices().nth(200).map(|(i, _)| i).unwrap_or(body.text.len())])
} else {
body.text.clone()
};
let tags = extract_tags(&desc);
let part = match reqwest::multipart::Part::bytes(body.text.into_bytes())
.file_name("paste.txt")
.mime_str("text/plain")
{
Ok(p) => p,
Err(e) => return (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()).into_response(),
};
let mut form = reqwest::multipart::Form::new()
.part("file", part)
.text("application", "paste")
.text("description", desc);
if !tags.is_empty() {
form = form.text("tags", tags);
}
let resp = state
.client
.post(format!("{CAN_API}/ingest"))
.multipart(form)
.send()
.await;
forward(resp).await
}
/// Accept a multipart upload from the frontend (clipboard image).
/// Re-packages it into a new multipart request for CanService.
async fn paste_file(
State(state): State<AppState>,
mut multipart: Multipart,
) -> Response {
let mut file_bytes: Option<Vec<u8>> = None;
let mut file_name = "clipboard.png".to_string();
let mut content_type = "image/png".to_string();
let mut description = String::new();
loop {
match multipart.next_field().await {
Ok(Some(field)) => {
let name = field.name().unwrap_or("").to_string();
match name.as_str() {
"file" => {
if let Some(fname) = field.file_name() {
file_name = fname.to_string();
}
if let Some(ct) = field.content_type() {
content_type = ct.to_string();
}
match field.bytes().await {
Ok(b) => file_bytes = Some(b.to_vec()),
Err(e) => {
return (StatusCode::BAD_REQUEST, format!("Failed to read file: {e}")).into_response();
}
}
}
"description" => {
description = field.text().await.unwrap_or_default();
}
_ => {}
}
}
Ok(None) => break,
Err(e) => {
return (StatusCode::BAD_REQUEST, format!("Multipart error: {e}")).into_response();
}
}
}
let Some(bytes) = file_bytes else {
return (StatusCode::BAD_REQUEST, "Missing file field").into_response();
};
let part = match reqwest::multipart::Part::bytes(bytes)
.file_name(file_name)
.mime_str(&content_type)
{
Ok(p) => p,
Err(e) => return (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()).into_response(),
};
let tags = extract_tags(&description);
let mut form = reqwest::multipart::Form::new()
.part("file", part)
.text("application", "paste");
if !description.is_empty() {
form = form.text("description", description);
}
if !tags.is_empty() {
form = form.text("tags", tags);
}
let resp = state
.client
.post(format!("{CAN_API}/ingest"))
.multipart(form)
.send()
.await;
forward(resp).await
}
/// List items with application=paste, newest first.
async fn paste_list(State(state): State<AppState>) -> Response {
let resp = state
.client
.get(format!(
"{CAN_API}/list?application=paste&order=desc&limit=100"
))
.send()
.await;
forward(resp).await
}
/// Proxy asset download by hash.
async fn proxy_asset(
State(state): State<AppState>,
Path(hash): Path<String>,
) -> Response {
let resp = state
.client
.get(format!("{CAN_API}/asset/{hash}"))
.send()
.await;
forward(resp).await
}
/// Proxy thumbnail (200x200) by hash.
async fn proxy_thumb(
State(state): State<AppState>,
Path(hash): Path<String>,
) -> Response {
let resp = state
.client
.get(format!("{CAN_API}/asset/{hash}/thumb/200/200"))
.send()
.await;
forward(resp).await
}
// ── Main ─────────────────────────────────────────────────────────────────
#[tokio::main]
async fn main() {
tracing_subscriber::fmt()
.with_env_filter(
tracing_subscriber::EnvFilter::try_from_default_env()
.unwrap_or_else(|_| "paste=info".into()),
)
.init();
let state = AppState {
client: reqwest::Client::new(),
};
let app = Router::new()
.route("/", get(serve_index))
.route("/paste/text", post(paste_text))
.route("/paste/file", post(paste_file))
.route("/paste/list", get(paste_list))
.route("/paste/asset/{hash}", get(proxy_asset))
.route("/paste/thumb/{hash}", get(proxy_thumb))
.layer(DefaultBodyLimit::max(100 * 1024 * 1024)) // 100 MB
.with_state(state);
let addr = SocketAddr::from(([127, 0, 0, 1], 3211));
tracing::info!("paste running at http://{addr}");
tracing::info!("requires CanService at http://127.0.0.1:3210");
// Open browser (best-effort, won't crash on headless)
let url = format!("http://{addr}");
let _ = open::that(&url);
let listener = tokio::net::TcpListener::bind(addr).await.unwrap();
axum::serve(listener, app).await.unwrap();
}

69
go_example_1.ps1 Normal file
View File

@ -0,0 +1,69 @@
# go_example_1.ps1 — Start CanService + Paste example, open browser
$ErrorActionPreference = "Stop"
$root = $PSScriptRoot
# Kill any leftover processes on our ports
Write-Host "Cleaning up stale processes..." -ForegroundColor Yellow
Get-NetTCPConnection -LocalPort 3210 -ErrorAction SilentlyContinue |
ForEach-Object { Stop-Process -Id $_.OwningProcess -Force -ErrorAction SilentlyContinue }
Get-NetTCPConnection -LocalPort 3211 -ErrorAction SilentlyContinue |
ForEach-Object { Stop-Process -Id $_.OwningProcess -Force -ErrorAction SilentlyContinue }
Start-Sleep -Milliseconds 500
Write-Host "Building CanService..." -ForegroundColor Cyan
cargo build --manifest-path "$root\Cargo.toml"
Write-Host "Building Paste example..." -ForegroundColor Cyan
cargo build --manifest-path "$root\examples\paste\Cargo.toml"
# Start CanService in background
Write-Host "Starting CanService on port 3210..." -ForegroundColor Green
$canService = Start-Process -FilePath "cargo" `
-ArgumentList "run --manifest-path `"$root\Cargo.toml`"" `
-WorkingDirectory $root `
-PassThru -NoNewWindow
# Wait for CanService to be ready
Write-Host "Waiting for CanService..." -ForegroundColor Yellow
$ready = $false
for ($i = 0; $i -lt 30; $i++) {
try {
$null = Invoke-WebRequest -Uri "http://127.0.0.1:3210/api/v1/can/0/list" -TimeoutSec 1 -ErrorAction Stop
$ready = $true
break
} catch {
Start-Sleep -Milliseconds 500
}
}
if (-not $ready) {
Write-Host "CanService failed to start within 15s" -ForegroundColor Red
Stop-Process -Id $canService.Id -Force -ErrorAction SilentlyContinue
exit 1
}
Write-Host "CanService ready." -ForegroundColor Green
# Start Paste example (it opens the browser itself)
Write-Host "Starting Paste on port 3211..." -ForegroundColor Green
$paste = Start-Process -FilePath "cargo" `
-ArgumentList "run --manifest-path `"$root\examples\paste\Cargo.toml`"" `
-WorkingDirectory $root `
-PassThru -NoNewWindow
Write-Host ""
Write-Host "Running:" -ForegroundColor Cyan
Write-Host " CanService -> http://127.0.0.1:3210"
Write-Host " Paste UI -> http://127.0.0.1:3211"
Write-Host ""
Write-Host "Press Ctrl+C to stop both." -ForegroundColor Yellow
# Wait for either process to exit, then clean up both
try {
while (-not $canService.HasExited -and -not $paste.HasExited) {
Start-Sleep -Seconds 1
}
} finally {
Write-Host "Shutting down..." -ForegroundColor Yellow
Stop-Process -Id $canService.Id -Force -ErrorAction SilentlyContinue
Stop-Process -Id $paste.Id -Force -ErrorAction SilentlyContinue
}

185
spec.md Normal file
View File

@ -0,0 +1,185 @@
# CAN (Containerized Asset Network) Service Specification
**Version:** 1.0 (Final MVP)
**Target Language:** Rust
## 1. System Overview
The CAN service is a robust, self-healing local network daemon designed to simulate a high-speed, append-oriented file system. It provides an HTTP REST and Protobuf API to ingest, manage, and retrieve assets (files and data).
To bypass the slow nature of traditional OS file searches, it uses an embedded SQLite database for millisecond querying. To ensure 100% disaster-recovery readiness, critical metadata is redundantly written to the host's native OS file attributes.
**MVP Scope:** This version supports a single, default container. To future-proof the API, all routes require a `{can_id}` parameter, which **must always be `0`**. Physically, all data is mapped flatly to the configured storage root.
---
## 2. Directory Structure & Configuration
The system uses a flat directory structure within the configured root folder.
**Physical Structure:**
```text
/var/lib/can_data/ # Defined by storage_root
├── .can.db # Master SQLite Index (Hidden)
├── .trash/ # Soft-deleted physical assets (Hidden)
├── .thumbs/ # Cached thumbnail images (Hidden, if enabled)
├── 1773014400123_a3b2... # Physical Asset
└── 1773014405999_f8c9... # Physical Asset
```
**Configuration (`config.yaml`):**
```yaml
storage_root: "/var/lib/can_data" # Absolute path to the storage folder
admin_token: "super_secret_rebuild" # Bearer token for admin operations
enable_thumbnail_cache: true # Toggle caching in .thumbs/
rebuild_error_threshold: 50 # Tolerance before triggering a hard rebuild
verify_interval_hours: 12 # Frequency of full background hash verification
```
---
## 3. Storage Mechanics & Disaster Recovery
### 3.1 Cryptographic Naming Convention
Files are written with a strict physical naming format to allow offline, mathematical verification of integrity.
**Format:** `{timestamp}_{sha256}_{truncated_tags}.{extension}`
* `timestamp`: Epoch Unix timestamp in milliseconds (e.g., `1773014400123`).
* `sha256`: A SHA-256 hash calculated exactly as: `SHA256([timestamp_bytes] + [raw_file_content_bytes])`.
* `truncated_tags`: Tags joined by underscores (`_`). Non-alphanumeric characters stripped. Safely truncated to ensure the total filename stays safely under OS path limits (~255 chars). Omitted if no tags provided.
* `extension`: Derived from the `mime_type` or magic bytes (e.g., `.pdf`, `.json`).
### 3.2 Native OS File Attributes
To guarantee the SQLite database can be rebuilt from scratch, critical metadata is bound directly to the file using OS-level attributes (Extended Attributes / `xattr` on Linux/macOS; NTFS Alternate Data Streams on Windows).
**Required Attributes:**
* `can.application`: Software that ingested the file.
* `can.user`: User identity.
* `can.tags`: **The complete, unbounded, comma-separated list of tags.**
* `can.description`: Human-readable description.
* `can.human_filename`: The logical filename provided during ingestion.
* `can.human_path`: The logical folder path provided during ingestion.
---
## 4. Metadata Indexing (`.can.db`)
A fully normalized SQLite database located at `{storage_root}/.can.db`.
**Schema:**
```sql
CREATE TABLE assets (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp INTEGER NOT NULL,
hash TEXT NOT NULL UNIQUE,
mime_type TEXT NOT NULL,
application TEXT,
user_identity TEXT,
description TEXT,
actual_filename TEXT NOT NULL,
human_filename TEXT,
human_path TEXT,
is_trashed BOOLEAN NOT NULL DEFAULT 0,
is_corrupted BOOLEAN NOT NULL DEFAULT 0
);
CREATE TABLE tags (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL UNIQUE
);
CREATE TABLE asset_tags (
asset_id INTEGER NOT NULL,
tag_id INTEGER NOT NULL,
PRIMARY KEY (asset_id, tag_id),
FOREIGN KEY (asset_id) REFERENCES assets(id) ON DELETE CASCADE,
FOREIGN KEY (tag_id) REFERENCES tags(id) ON DELETE CASCADE
);
-- Optimization Indexes
CREATE INDEX idx_hash ON assets(hash);
CREATE INDEX idx_timestamp ON assets(timestamp);
CREATE INDEX idx_application ON assets(application);
CREATE INDEX idx_user ON assets(user_identity);
CREATE INDEX idx_trashed ON assets(is_trashed);
CREATE INDEX idx_tag_name ON tags(name);
```
---
## 5. Background Verifier Subsystem
A low-priority background thread dedicated to data integrity.
1. **Initial Scrub:** Runs on startup. Verifies `SHA256(timestamp + content)` for all files against their filenames.
2. **Continuous Monitoring:** Hooks into OS file system events (e.g., `inotify`). If a file is touched or altered by an external program, the verifier immediately rescans it.
3. **Periodic Scrub:** Runs every `verify_interval_hours` to catch silent bit rot.
4. **Corruption Handling:** If a hash mismatch is found, it flags `is_corrupted = 1` in `.can.db`. Corrupted files are explicitly marked in API responses and excluded from standard operations.
---
## 6. API Endpoints
**Protocol Negotation:**
All endpoints communicate in JSON by default. Clients can request/send **Protocol Buffers** by providing the HTTP headers:
* `Accept: application/x-protobuf`
* `Content-Type: application/x-protobuf`
*(Note: Endpoint paths below use `{can_id}` which must be passed as `0`)*
### 6.1 Ingest Data
* **Method:** `POST`
* **Path:** `/api/v1/can/0/ingest`
* **Content-Type:** `multipart/form-data`
* **Form Payload:**
* `file` (Binary File) - **Required**
* `mime_type` (String) - *Optional*
* `human_file_name` (String) - *Optional*
* `human_readable_path` (String) - *Optional*
* `application` (String) - *Optional*
* `user` (String) - *Optional*
* `tags` (String) - *Optional* (comma-separated)
* `description` (String) - *Optional*
* **Action:** Hashes file, writes to `{storage_root}`, attaches OS attributes, logs to DB.
* **Response (JSON):**
`{ "status": "success", "data": { "timestamp": 1773014400123, "hash": "abc...", "filename": "1773014400123_abc_tag.pdf" } }`
### 6.2 Retrieve Physical Asset
* **Method:** `GET`
* **Path:** `/api/v1/can/0/asset/{hash}`
* **Action:** Streams the physical file. Sets `Content-Type` via DB mapping and `Content-Disposition` using `human_filename`. Returns 500/Warning if `is_corrupted = 1`.
### 6.3 Retrieve Asset Metadata
* **Method:** `GET`
* **Path:** `/api/v1/can/0/asset/{hash}/meta`
* **Action:** Returns DB record.
* **Response (JSON):**
`{ "status": "success", "data": { "hash": "abc...", "mime_type": "image/jpeg", "application": "WebUI", "user": "Jason", "tags": ["tag1", "tag2"], "description": "...", "human_filename": "photo.jpg", "human_path": "/img/", "timestamp": 1773014400123, "is_trashed": false, "is_corrupted": false } }`
### 6.4 Retrieve Thumbnail
* **Method:** `GET`
* **Path:** `/api/v1/can/0/asset/{hash}/thumb/{max_width}/{max_height}`
* **Action:** Resizes image strictly preserving aspect ratio. Falls back to static icon (SVG/PNG) for non-images. If `enable_thumbnail_cache=true`, reads/writes to `{storage_root}/.thumbs/{hash}_{max_width}x{max_height}.jpg`. Streams byte payload.
### 6.5 Modify Metadata
* **Method:** `PATCH`
* **Path:** `/api/v1/can/0/asset/{hash}`
* **Body (JSON/Protobuf):**
`{ "tags": ["new_tag1", "new_tag2"], "description": "New description" }`
* **Action:** Updates `can.tags` and `can.description` OS Attributes. Updates SQLite `assets`, `tags`, and `asset_tags` tables inside a transaction. Physical filename remains unchanged.
### 6.6 List Assets
* **Method:** `GET`
* **Path:** `/api/v1/can/0/list`
* **Query Parameters:**
* `limit` (Integer) - Default `50`
* `offset` (Integer) - Default `0`
* `offset_time` (Integer) - *Optional*. Epoch ms. High-speed cursor. Lists items strictly after/before this timestamp based on `order`.
* `order` (String) - `asc` or `desc`. Default `desc`.
* `application` (String) - *Optional*. Scopes list exclusively to files ingested by this Application ID.
* `include_trashed` (Boolean) - Default `false`.
* `include_corrupted` (Boolean) - Default `false`.
* **Response:** Paginated array of metadata objects (matching 6.3 output) + `pagination` block.
### 6.7 Search Assets
* **Method:** `GET`
* **Path:** `/api/v1/can/0/search`
* **Query Parameters:**
* `hash` (String) - Exact or partial prefix.
* `start_time` (Integer) - Epoch ms.
* `end_time` (Integer) -

112
src/config.rs Normal file
View File

@ -0,0 +1,112 @@
use serde::Deserialize;
use std::path::{Path, PathBuf};
#[derive(Debug, Clone, Deserialize)]
pub struct Config {
pub storage_root: PathBuf,
#[serde(default = "default_admin_token")]
pub admin_token: String,
#[serde(default = "default_true")]
pub enable_thumbnail_cache: bool,
#[serde(default = "default_rebuild_threshold")]
pub rebuild_error_threshold: u32,
#[serde(default = "default_verify_interval")]
pub verify_interval_hours: u64,
}
fn default_admin_token() -> String {
"changeme".to_string()
}
fn default_true() -> bool {
true
}
fn default_rebuild_threshold() -> u32 {
50
}
fn default_verify_interval() -> u64 {
12
}
impl Config {
pub fn load(path: &Path) -> anyhow::Result<Self> {
let contents = std::fs::read_to_string(path)?;
let config: Config = serde_yaml::from_str(&contents)?;
Ok(config)
}
pub fn db_path(&self) -> PathBuf {
self.storage_root.join(".can.db")
}
pub fn trash_dir(&self) -> PathBuf {
self.storage_root.join(".trash")
}
pub fn thumbs_dir(&self) -> PathBuf {
self.storage_root.join(".thumbs")
}
pub fn ensure_dirs(&self) -> anyhow::Result<()> {
std::fs::create_dir_all(&self.storage_root)?;
std::fs::create_dir_all(self.trash_dir())?;
if self.enable_thumbnail_cache {
std::fs::create_dir_all(self.thumbs_dir())?;
}
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
use tempfile::TempDir;
#[test]
fn test_load_config() {
let dir = TempDir::new().unwrap();
let config_path = dir.path().join("config.yaml");
std::fs::write(
&config_path,
r#"
storage_root: "/tmp/can_test"
admin_token: "test_token"
enable_thumbnail_cache: false
rebuild_error_threshold: 10
verify_interval_hours: 6
"#,
)
.unwrap();
let config = Config::load(&config_path).unwrap();
assert_eq!(config.storage_root, PathBuf::from("/tmp/can_test"));
assert_eq!(config.admin_token, "test_token");
assert!(!config.enable_thumbnail_cache);
assert_eq!(config.rebuild_error_threshold, 10);
assert_eq!(config.verify_interval_hours, 6);
}
#[test]
fn test_config_defaults() {
let dir = TempDir::new().unwrap();
let config_path = dir.path().join("config.yaml");
std::fs::write(&config_path, "storage_root: /tmp/test\n").unwrap();
let config = Config::load(&config_path).unwrap();
assert_eq!(config.admin_token, "changeme");
assert!(config.enable_thumbnail_cache);
assert_eq!(config.rebuild_error_threshold, 50);
assert_eq!(config.verify_interval_hours, 12);
}
#[test]
fn test_derived_paths() {
let dir = TempDir::new().unwrap();
let config_path = dir.path().join("config.yaml");
std::fs::write(&config_path, "storage_root: /data/can\n").unwrap();
let config = Config::load(&config_path).unwrap();
assert_eq!(config.db_path(), PathBuf::from("/data/can/.can.db"));
assert_eq!(config.trash_dir(), PathBuf::from("/data/can/.trash"));
assert_eq!(config.thumbs_dir(), PathBuf::from("/data/can/.thumbs"));
}
}

663
src/db.rs Normal file
View File

@ -0,0 +1,663 @@
use rusqlite::{params, Connection, OptionalExtension};
use std::path::Path;
use std::sync::{Arc, Mutex};
use crate::models::{Asset, AssetMeta, ListParams, SearchParams};
pub type Db = Arc<Mutex<Connection>>;
pub fn open(path: &Path) -> anyhow::Result<Db> {
let conn = Connection::open(path)?;
conn.execute_batch("PRAGMA journal_mode=WAL; PRAGMA foreign_keys=ON;")?;
init_schema(&conn)?;
Ok(Arc::new(Mutex::new(conn)))
}
pub fn open_in_memory() -> anyhow::Result<Db> {
let conn = Connection::open_in_memory()?;
conn.execute_batch("PRAGMA foreign_keys=ON;")?;
init_schema(&conn)?;
Ok(Arc::new(Mutex::new(conn)))
}
fn init_schema(conn: &Connection) -> rusqlite::Result<()> {
conn.execute_batch(
"
CREATE TABLE IF NOT EXISTS assets (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp INTEGER NOT NULL,
hash TEXT NOT NULL UNIQUE,
mime_type TEXT NOT NULL,
application TEXT,
user_identity TEXT,
description TEXT,
actual_filename TEXT NOT NULL,
human_filename TEXT,
human_path TEXT,
is_trashed BOOLEAN NOT NULL DEFAULT 0,
is_corrupted BOOLEAN NOT NULL DEFAULT 0
);
CREATE TABLE IF NOT EXISTS tags (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL UNIQUE
);
CREATE TABLE IF NOT EXISTS asset_tags (
asset_id INTEGER NOT NULL,
tag_id INTEGER NOT NULL,
PRIMARY KEY (asset_id, tag_id),
FOREIGN KEY (asset_id) REFERENCES assets(id) ON DELETE CASCADE,
FOREIGN KEY (tag_id) REFERENCES tags(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_hash ON assets(hash);
CREATE INDEX IF NOT EXISTS idx_timestamp ON assets(timestamp);
CREATE INDEX IF NOT EXISTS idx_application ON assets(application);
CREATE INDEX IF NOT EXISTS idx_user ON assets(user_identity);
CREATE INDEX IF NOT EXISTS idx_trashed ON assets(is_trashed);
CREATE INDEX IF NOT EXISTS idx_tag_name ON tags(name);
",
)?;
// Migration: add size column (ignore error if column already exists)
let _ = conn.execute("ALTER TABLE assets ADD COLUMN size INTEGER NOT NULL DEFAULT 0", []);
Ok(())
}
/// Insert a new asset. Returns the row id.
pub fn insert_asset(conn: &Connection, asset: &Asset) -> rusqlite::Result<i64> {
conn.execute(
"INSERT INTO assets (timestamp, hash, mime_type, application, user_identity, description, actual_filename, human_filename, human_path, size)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10)",
params![
asset.timestamp,
asset.hash,
asset.mime_type,
asset.application,
asset.user_identity,
asset.description,
asset.actual_filename,
asset.human_filename,
asset.human_path,
asset.size,
],
)?;
Ok(conn.last_insert_rowid())
}
/// Look up an asset by its hash.
pub fn get_asset_by_hash(conn: &Connection, hash: &str) -> rusqlite::Result<Option<Asset>> {
conn.query_row(
"SELECT id, timestamp, hash, mime_type, application, user_identity, description,
actual_filename, human_filename, human_path, is_trashed, is_corrupted, size
FROM assets WHERE hash = ?1",
params![hash],
|row| {
Ok(Asset {
id: row.get(0)?,
timestamp: row.get(1)?,
hash: row.get(2)?,
mime_type: row.get(3)?,
application: row.get(4)?,
user_identity: row.get(5)?,
description: row.get(6)?,
actual_filename: row.get(7)?,
human_filename: row.get(8)?,
human_path: row.get(9)?,
is_trashed: row.get(10)?,
is_corrupted: row.get(11)?,
size: row.get(12)?,
})
},
)
.optional()
}
/// Get tags for an asset.
pub fn get_asset_tags(conn: &Connection, asset_id: i64) -> rusqlite::Result<Vec<String>> {
let mut stmt = conn.prepare(
"SELECT t.name FROM tags t
JOIN asset_tags at ON at.tag_id = t.id
WHERE at.asset_id = ?1
ORDER BY t.name",
)?;
let tags = stmt.query_map(params![asset_id], |row| row.get(0))?;
tags.collect()
}
/// Upsert a tag and return its id.
pub fn upsert_tag(conn: &Connection, name: &str) -> rusqlite::Result<i64> {
conn.execute(
"INSERT OR IGNORE INTO tags (name) VALUES (?1)",
params![name],
)?;
conn.query_row("SELECT id FROM tags WHERE name = ?1", params![name], |row| {
row.get(0)
})
}
/// Replace all tags for an asset within a transaction.
pub fn set_asset_tags(conn: &Connection, asset_id: i64, tags: &[String]) -> rusqlite::Result<()> {
conn.execute(
"DELETE FROM asset_tags WHERE asset_id = ?1",
params![asset_id],
)?;
for tag in tags {
let tag_id = upsert_tag(conn, tag)?;
conn.execute(
"INSERT OR IGNORE INTO asset_tags (asset_id, tag_id) VALUES (?1, ?2)",
params![asset_id, tag_id],
)?;
}
Ok(())
}
/// Build an AssetMeta from an Asset row + tags.
pub fn asset_to_meta(conn: &Connection, asset: &Asset) -> rusqlite::Result<AssetMeta> {
let tags = get_asset_tags(conn, asset.id)?;
Ok(AssetMeta {
hash: asset.hash.clone(),
mime_type: asset.mime_type.clone(),
application: asset.application.clone(),
user: asset.user_identity.clone(),
tags,
description: asset.description.clone(),
human_filename: asset.human_filename.clone(),
human_path: asset.human_path.clone(),
timestamp: asset.timestamp,
is_trashed: asset.is_trashed,
is_corrupted: asset.is_corrupted,
size: asset.size,
})
}
/// Update description and/or tags for an asset.
pub fn update_asset_metadata(
conn: &Connection,
hash: &str,
description: Option<&str>,
tags: Option<&[String]>,
) -> rusqlite::Result<()> {
let asset = get_asset_by_hash(conn, hash)?
.ok_or(rusqlite::Error::QueryReturnedNoRows)?;
if let Some(desc) = description {
conn.execute(
"UPDATE assets SET description = ?1 WHERE id = ?2",
params![desc, asset.id],
)?;
}
if let Some(tags) = tags {
set_asset_tags(conn, asset.id, tags)?;
}
Ok(())
}
/// Flag an asset as corrupted.
pub fn flag_corrupted(conn: &Connection, hash: &str, corrupted: bool) -> rusqlite::Result<()> {
conn.execute(
"UPDATE assets SET is_corrupted = ?1 WHERE hash = ?2",
params![corrupted, hash],
)?;
Ok(())
}
/// Update file size for an asset (used by verifier to backfill).
pub fn update_asset_size(conn: &Connection, hash: &str, size: i64) -> rusqlite::Result<()> {
conn.execute(
"UPDATE assets SET size = ?1 WHERE hash = ?2",
params![size, hash],
)?;
Ok(())
}
/// Soft-delete: mark as trashed.
pub fn trash_asset(conn: &Connection, hash: &str) -> rusqlite::Result<()> {
conn.execute(
"UPDATE assets SET is_trashed = 1 WHERE hash = ?1",
params![hash],
)?;
Ok(())
}
/// List assets with pagination and filtering.
pub fn list_assets(conn: &Connection, params: &ListParams) -> rusqlite::Result<(Vec<Asset>, i64)> {
let limit = params.limit.unwrap_or(50);
let offset = params.offset.unwrap_or(0);
let order = match params.order.as_deref() {
Some("asc") => "ASC",
_ => "DESC",
};
let include_trashed = params.include_trashed.unwrap_or(false);
let include_corrupted = params.include_corrupted.unwrap_or(false);
let mut conditions = Vec::new();
let mut bind_values: Vec<Box<dyn rusqlite::types::ToSql>> = Vec::new();
if !include_trashed {
conditions.push("is_trashed = 0");
}
if !include_corrupted {
conditions.push("is_corrupted = 0");
}
if let Some(ref app) = params.application {
conditions.push("application = ?");
bind_values.push(Box::new(app.clone()));
}
if let Some(offset_time) = params.offset_time {
if order == "DESC" {
conditions.push("timestamp < ?");
} else {
conditions.push("timestamp > ?");
}
bind_values.push(Box::new(offset_time));
}
let where_clause = if conditions.is_empty() {
String::new()
} else {
format!("WHERE {}", conditions.join(" AND "))
};
let count_sql = format!("SELECT COUNT(*) FROM assets {}", where_clause);
let refs: Vec<&dyn rusqlite::types::ToSql> = bind_values.iter().map(|b| b.as_ref()).collect();
let total: i64 = conn.query_row(&count_sql, refs.as_slice(), |row| row.get(0))?;
let query_sql = format!(
"SELECT id, timestamp, hash, mime_type, application, user_identity, description,
actual_filename, human_filename, human_path, is_trashed, is_corrupted, size
FROM assets {} ORDER BY timestamp {} LIMIT ? OFFSET ?",
where_clause, order
);
let mut all_binds: Vec<Box<dyn rusqlite::types::ToSql>> = bind_values;
all_binds.push(Box::new(limit));
all_binds.push(Box::new(offset));
let refs2: Vec<&dyn rusqlite::types::ToSql> = all_binds.iter().map(|b| b.as_ref()).collect();
let mut stmt = conn.prepare(&query_sql)?;
let assets = stmt
.query_map(refs2.as_slice(), |row| {
Ok(Asset {
id: row.get(0)?,
timestamp: row.get(1)?,
hash: row.get(2)?,
mime_type: row.get(3)?,
application: row.get(4)?,
user_identity: row.get(5)?,
description: row.get(6)?,
actual_filename: row.get(7)?,
human_filename: row.get(8)?,
human_path: row.get(9)?,
is_trashed: row.get(10)?,
is_corrupted: row.get(11)?,
size: row.get(12)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok((assets, total))
}
/// Search assets with various filters.
pub fn search_assets(
conn: &Connection,
params: &SearchParams,
) -> rusqlite::Result<(Vec<Asset>, i64)> {
let limit = params.limit.unwrap_or(50);
let offset = params.offset.unwrap_or(0);
let order = match params.order.as_deref() {
Some("asc") => "ASC",
_ => "DESC",
};
let include_trashed = params.include_trashed.unwrap_or(false);
let include_corrupted = params.include_corrupted.unwrap_or(false);
let mut conditions = Vec::new();
let mut bind_values: Vec<Box<dyn rusqlite::types::ToSql>> = Vec::new();
let mut needs_tag_join = false;
if !include_trashed {
conditions.push("a.is_trashed = 0".to_string());
}
if !include_corrupted {
conditions.push("a.is_corrupted = 0".to_string());
}
if let Some(ref hash) = params.hash {
conditions.push("a.hash LIKE ?".to_string());
bind_values.push(Box::new(format!("{}%", hash)));
}
if let Some(start) = params.start_time {
conditions.push("a.timestamp >= ?".to_string());
bind_values.push(Box::new(start));
}
if let Some(end) = params.end_time {
conditions.push("a.timestamp <= ?".to_string());
bind_values.push(Box::new(end));
}
if let Some(ref mime) = params.mime_type {
conditions.push("a.mime_type = ?".to_string());
bind_values.push(Box::new(mime.clone()));
}
if let Some(ref user) = params.user {
conditions.push("a.user_identity = ?".to_string());
bind_values.push(Box::new(user.clone()));
}
if let Some(ref app) = params.application {
conditions.push("a.application = ?".to_string());
bind_values.push(Box::new(app.clone()));
}
// Tag filtering: AND logic - asset must have ALL specified tags
let tag_names: Vec<String> = params
.tags
.as_deref()
.unwrap_or("")
.split(',')
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.collect();
if !tag_names.is_empty() {
needs_tag_join = true;
let placeholders: Vec<String> = tag_names.iter().map(|_| "?".to_string()).collect();
conditions.push(format!(
"a.id IN (
SELECT at.asset_id FROM asset_tags at
JOIN tags t ON t.id = at.tag_id
WHERE t.name IN ({})
GROUP BY at.asset_id
HAVING COUNT(DISTINCT t.id) = ?
)",
placeholders.join(", ")
));
for tag in &tag_names {
bind_values.push(Box::new(tag.clone()));
}
bind_values.push(Box::new(tag_names.len() as i64));
}
let _ = needs_tag_join; // subquery handles it
let where_clause = if conditions.is_empty() {
String::new()
} else {
format!("WHERE {}", conditions.join(" AND "))
};
let count_sql = format!("SELECT COUNT(*) FROM assets a {}", where_clause);
let refs: Vec<&dyn rusqlite::types::ToSql> = bind_values.iter().map(|b| b.as_ref()).collect();
let total: i64 = conn.query_row(&count_sql, refs.as_slice(), |row| row.get(0))?;
let query_sql = format!(
"SELECT a.id, a.timestamp, a.hash, a.mime_type, a.application, a.user_identity,
a.description, a.actual_filename, a.human_filename, a.human_path,
a.is_trashed, a.is_corrupted, a.size
FROM assets a {} ORDER BY a.timestamp {} LIMIT ? OFFSET ?",
where_clause, order
);
let mut all_binds = bind_values;
all_binds.push(Box::new(limit));
all_binds.push(Box::new(offset));
let refs2: Vec<&dyn rusqlite::types::ToSql> = all_binds.iter().map(|b| b.as_ref()).collect();
let mut stmt = conn.prepare(&query_sql)?;
let assets = stmt
.query_map(refs2.as_slice(), |row| {
Ok(Asset {
id: row.get(0)?,
timestamp: row.get(1)?,
hash: row.get(2)?,
mime_type: row.get(3)?,
application: row.get(4)?,
user_identity: row.get(5)?,
description: row.get(6)?,
actual_filename: row.get(7)?,
human_filename: row.get(8)?,
human_path: row.get(9)?,
is_trashed: row.get(10)?,
is_corrupted: row.get(11)?,
size: row.get(12)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok((assets, total))
}
/// Get all non-trashed asset records (for verifier startup scan).
pub fn get_all_active_assets(conn: &Connection) -> rusqlite::Result<Vec<Asset>> {
let mut stmt = conn.prepare(
"SELECT id, timestamp, hash, mime_type, application, user_identity, description,
actual_filename, human_filename, human_path, is_trashed, is_corrupted, size
FROM assets WHERE is_trashed = 0",
)?;
let assets = stmt
.query_map([], |row| {
Ok(Asset {
id: row.get(0)?,
timestamp: row.get(1)?,
hash: row.get(2)?,
mime_type: row.get(3)?,
application: row.get(4)?,
user_identity: row.get(5)?,
description: row.get(6)?,
actual_filename: row.get(7)?,
human_filename: row.get(8)?,
human_path: row.get(9)?,
is_trashed: row.get(10)?,
is_corrupted: row.get(11)?,
size: row.get(12)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(assets)
}
#[cfg(test)]
mod tests {
use super::*;
fn make_test_asset(ts: i64, hash: &str) -> Asset {
Asset {
id: 0,
timestamp: ts,
hash: hash.to_string(),
mime_type: "text/plain".to_string(),
application: Some("test_app".to_string()),
user_identity: Some("test_user".to_string()),
description: Some("test desc".to_string()),
actual_filename: format!("{}_{}.txt", ts, hash),
human_filename: Some("readme.txt".to_string()),
human_path: Some("/docs/".to_string()),
is_trashed: false,
is_corrupted: false,
size: 0,
}
}
#[test]
fn test_insert_and_get_asset() {
let db = open_in_memory().unwrap();
let conn = db.lock().unwrap();
let asset = make_test_asset(1000, "abc123");
let id = insert_asset(&conn, &asset).unwrap();
assert!(id > 0);
let found = get_asset_by_hash(&conn, "abc123").unwrap().unwrap();
assert_eq!(found.hash, "abc123");
assert_eq!(found.timestamp, 1000);
assert_eq!(found.mime_type, "text/plain");
}
#[test]
fn test_get_nonexistent_asset() {
let db = open_in_memory().unwrap();
let conn = db.lock().unwrap();
let found = get_asset_by_hash(&conn, "nonexistent").unwrap();
assert!(found.is_none());
}
#[test]
fn test_tags() {
let db = open_in_memory().unwrap();
let conn = db.lock().unwrap();
let asset = make_test_asset(2000, "def456");
let id = insert_asset(&conn, &asset).unwrap();
let tags = vec!["photo".to_string(), "vacation".to_string()];
set_asset_tags(&conn, id, &tags).unwrap();
let fetched = get_asset_tags(&conn, id).unwrap();
assert_eq!(fetched, vec!["photo", "vacation"]);
// Replace tags
let new_tags = vec!["work".to_string()];
set_asset_tags(&conn, id, &new_tags).unwrap();
let fetched2 = get_asset_tags(&conn, id).unwrap();
assert_eq!(fetched2, vec!["work"]);
}
#[test]
fn test_update_metadata() {
let db = open_in_memory().unwrap();
let conn = db.lock().unwrap();
let asset = make_test_asset(3000, "ghi789");
insert_asset(&conn, &asset).unwrap();
let new_tags = vec!["updated".to_string()];
update_asset_metadata(&conn, "ghi789", Some("new desc"), Some(&new_tags)).unwrap();
let found = get_asset_by_hash(&conn, "ghi789").unwrap().unwrap();
assert_eq!(found.description, Some("new desc".to_string()));
let tags = get_asset_tags(&conn, found.id).unwrap();
assert_eq!(tags, vec!["updated"]);
}
#[test]
fn test_flag_corrupted() {
let db = open_in_memory().unwrap();
let conn = db.lock().unwrap();
let asset = make_test_asset(4000, "corrupt1");
insert_asset(&conn, &asset).unwrap();
flag_corrupted(&conn, "corrupt1", true).unwrap();
let found = get_asset_by_hash(&conn, "corrupt1").unwrap().unwrap();
assert!(found.is_corrupted);
flag_corrupted(&conn, "corrupt1", false).unwrap();
let found2 = get_asset_by_hash(&conn, "corrupt1").unwrap().unwrap();
assert!(!found2.is_corrupted);
}
#[test]
fn test_trash_asset() {
let db = open_in_memory().unwrap();
let conn = db.lock().unwrap();
let asset = make_test_asset(5000, "trash1");
insert_asset(&conn, &asset).unwrap();
trash_asset(&conn, "trash1").unwrap();
let found = get_asset_by_hash(&conn, "trash1").unwrap().unwrap();
assert!(found.is_trashed);
}
#[test]
fn test_list_assets_basic() {
let db = open_in_memory().unwrap();
let conn = db.lock().unwrap();
for i in 0..5 {
let asset = make_test_asset(1000 + i, &format!("hash_{}", i));
insert_asset(&conn, &asset).unwrap();
}
let params = ListParams {
limit: Some(3),
offset: Some(0),
offset_time: None,
order: Some("desc".to_string()),
application: None,
include_trashed: None,
include_corrupted: None,
};
let (assets, total) = list_assets(&conn, &params).unwrap();
assert_eq!(total, 5);
assert_eq!(assets.len(), 3);
// DESC order: highest timestamp first
assert!(assets[0].timestamp > assets[1].timestamp);
}
#[test]
fn test_list_excludes_trashed_by_default() {
let db = open_in_memory().unwrap();
let conn = db.lock().unwrap();
let a1 = make_test_asset(100, "visible1");
insert_asset(&conn, &a1).unwrap();
let a2 = make_test_asset(200, "trashed1");
insert_asset(&conn, &a2).unwrap();
trash_asset(&conn, "trashed1").unwrap();
let params = ListParams {
limit: None, offset: None, offset_time: None,
order: None, application: None,
include_trashed: None, include_corrupted: None,
};
let (assets, total) = list_assets(&conn, &params).unwrap();
assert_eq!(total, 1);
assert_eq!(assets[0].hash, "visible1");
}
#[test]
fn test_search_by_hash_prefix() {
let db = open_in_memory().unwrap();
let conn = db.lock().unwrap();
let a1 = make_test_asset(100, "abcdef123");
let a2 = make_test_asset(200, "abcxyz789");
let a3 = make_test_asset(300, "zzz000111");
insert_asset(&conn, &a1).unwrap();
insert_asset(&conn, &a2).unwrap();
insert_asset(&conn, &a3).unwrap();
let params = SearchParams {
hash: Some("abc".to_string()),
start_time: None, end_time: None, tags: None,
mime_type: None, user: None, application: None,
limit: None, offset: None, order: None,
include_trashed: None, include_corrupted: None,
};
let (assets, total) = search_assets(&conn, &params).unwrap();
assert_eq!(total, 2);
assert!(assets.iter().all(|a| a.hash.starts_with("abc")));
}
#[test]
fn test_search_by_tags() {
let db = open_in_memory().unwrap();
let conn = db.lock().unwrap();
let a1 = make_test_asset(100, "tagged1");
let id1 = insert_asset(&conn, &a1).unwrap();
set_asset_tags(&conn, id1, &["red".to_string(), "blue".to_string()]).unwrap();
let a2 = make_test_asset(200, "tagged2");
let id2 = insert_asset(&conn, &a2).unwrap();
set_asset_tags(&conn, id2, &["red".to_string()]).unwrap();
// Search for both red AND blue -> only tagged1
let params = SearchParams {
hash: None, start_time: None, end_time: None,
tags: Some("red,blue".to_string()),
mime_type: None, user: None, application: None,
limit: None, offset: None, order: None,
include_trashed: None, include_corrupted: None,
};
let (assets, total) = search_assets(&conn, &params).unwrap();
assert_eq!(total, 1);
assert_eq!(assets[0].hash, "tagged1");
}
}

42
src/error.rs Normal file
View File

@ -0,0 +1,42 @@
use axum::http::StatusCode;
use axum::response::{IntoResponse, Response};
use crate::models::ErrorResponse;
#[derive(Debug, thiserror::Error)]
pub enum AppError {
#[error("Not found: {0}")]
NotFound(String),
#[error("Bad request: {0}")]
BadRequest(String),
#[error("Asset is corrupted: {0}")]
Corrupted(String),
#[error("Database error: {0}")]
Database(#[from] rusqlite::Error),
#[error("IO error: {0}")]
Io(#[from] std::io::Error),
#[error("Internal error: {0}")]
Internal(String),
}
impl IntoResponse for AppError {
fn into_response(self) -> Response {
let (status, message) = match &self {
AppError::NotFound(msg) => (StatusCode::NOT_FOUND, msg.clone()),
AppError::BadRequest(msg) => (StatusCode::BAD_REQUEST, msg.clone()),
AppError::Corrupted(msg) => (StatusCode::INTERNAL_SERVER_ERROR, msg.clone()),
AppError::Database(e) => (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()),
AppError::Io(e) => (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()),
AppError::Internal(msg) => (StatusCode::INTERNAL_SERVER_ERROR, msg.clone()),
};
tracing::error!(%status, error = %message, "request error");
let body = serde_json::to_string(&ErrorResponse::new(message)).unwrap_or_default();
(status, [(axum::http::header::CONTENT_TYPE, "application/json")], body).into_response()
}
}

48
src/hash.rs Normal file
View File

@ -0,0 +1,48 @@
use sha2::{Digest, Sha256};
/// Compute SHA-256 hash as: SHA256(timestamp_bytes + content_bytes).
/// The timestamp is serialized as its string representation's bytes,
/// matching the spec: `SHA256([timestamp_bytes] + [raw_file_content_bytes])`.
pub fn compute_hash(timestamp_ms: i64, content: &[u8]) -> String {
let mut hasher = Sha256::new();
hasher.update(timestamp_ms.to_be_bytes());
hasher.update(content);
hex::encode(hasher.finalize())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_deterministic_hash() {
let ts = 1773014400123i64;
let content = b"hello world";
let h1 = compute_hash(ts, content);
let h2 = compute_hash(ts, content);
assert_eq!(h1, h2);
assert_eq!(h1.len(), 64); // SHA-256 hex = 64 chars
}
#[test]
fn test_different_timestamp_different_hash() {
let content = b"same content";
let h1 = compute_hash(1000, content);
let h2 = compute_hash(2000, content);
assert_ne!(h1, h2);
}
#[test]
fn test_different_content_different_hash() {
let ts = 1234567890i64;
let h1 = compute_hash(ts, b"content A");
let h2 = compute_hash(ts, b"content B");
assert_ne!(h1, h2);
}
#[test]
fn test_empty_content() {
let h = compute_hash(0, b"");
assert_eq!(h.len(), 64);
}
}

20
src/lib.rs Normal file
View File

@ -0,0 +1,20 @@
pub mod config;
pub mod db;
pub mod error;
pub mod hash;
pub mod models;
pub mod routes;
pub mod storage;
pub mod verifier;
pub mod xattr;
use std::sync::Arc;
use crate::config::Config;
use crate::db::Db;
#[derive(Clone)]
pub struct AppState {
pub config: Arc<Config>,
pub db: Db,
}

64
src/main.rs Normal file
View File

@ -0,0 +1,64 @@
use std::net::SocketAddr;
use std::path::PathBuf;
use std::sync::Arc;
use axum::extract::DefaultBodyLimit;
use axum::Router;
use tower_http::cors::CorsLayer;
use tower_http::trace::TraceLayer;
use can_service::config::Config;
use can_service::{db, routes, verifier, AppState};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// Initialize tracing
tracing_subscriber::fmt()
.with_env_filter(
tracing_subscriber::EnvFilter::try_from_default_env()
.unwrap_or_else(|_| "can_service=info,tower_http=info".into()),
)
.init();
// Load config
let config_path = std::env::args()
.nth(1)
.map(PathBuf::from)
.unwrap_or_else(|| PathBuf::from("config.yaml"));
let config = Config::load(&config_path)?;
tracing::info!("Loaded config, storage_root: {:?}", config.storage_root);
// Ensure directories exist
config.ensure_dirs()?;
// Open database
let db = db::open(&config.db_path())?;
tracing::info!("Database initialized at {:?}", config.db_path());
let config = Arc::new(config);
// Start background verifier
verifier::start((*config).clone(), db.clone());
let state = AppState {
config: config.clone(),
db,
};
// Build router
let app = Router::new()
.merge(routes::router())
.layer(DefaultBodyLimit::max(100 * 1024 * 1024)) // 100 MB
.layer(TraceLayer::new_for_http())
.layer(CorsLayer::permissive())
.with_state(state);
let addr = SocketAddr::from(([0, 0, 0, 0], 3210));
tracing::info!("CAN service listening on {}", addr);
let listener = tokio::net::TcpListener::bind(addr).await?;
axum::serve(listener, app).await?;
Ok(())
}

164
src/models.rs Normal file
View File

@ -0,0 +1,164 @@
use serde::{Deserialize, Serialize};
/// Database representation of an asset.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Asset {
pub id: i64,
pub timestamp: i64,
pub hash: String,
pub mime_type: String,
pub application: Option<String>,
pub user_identity: Option<String>,
pub description: Option<String>,
pub actual_filename: String,
pub human_filename: Option<String>,
pub human_path: Option<String>,
pub is_trashed: bool,
pub is_corrupted: bool,
pub size: i64,
}
/// API-facing asset metadata response.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AssetMeta {
pub hash: String,
pub mime_type: String,
pub application: Option<String>,
pub user: Option<String>,
pub tags: Vec<String>,
pub description: Option<String>,
pub human_filename: Option<String>,
pub human_path: Option<String>,
pub timestamp: i64,
pub is_trashed: bool,
pub is_corrupted: bool,
pub size: i64,
}
/// Standard API response wrapper.
#[derive(Debug, Serialize, Deserialize)]
pub struct ApiResponse<T: Serialize> {
pub status: String,
pub data: T,
}
impl<T: Serialize> ApiResponse<T> {
pub fn success(data: T) -> Self {
Self {
status: "success".to_string(),
data,
}
}
}
/// Error response body.
#[derive(Debug, Serialize, Deserialize)]
pub struct ErrorResponse {
pub status: String,
pub error: String,
}
impl ErrorResponse {
pub fn new(msg: impl Into<String>) -> Self {
Self {
status: "error".to_string(),
error: msg.into(),
}
}
}
/// Ingest success response data.
#[derive(Debug, Serialize, Deserialize)]
pub struct IngestResult {
pub timestamp: i64,
pub hash: String,
pub filename: String,
}
/// Pagination metadata in list responses.
#[derive(Debug, Serialize, Deserialize)]
pub struct Pagination {
pub limit: i64,
pub offset: i64,
pub total: i64,
}
/// Paginated list response.
#[derive(Debug, Serialize, Deserialize)]
pub struct ListResponse {
pub items: Vec<AssetMeta>,
pub pagination: Pagination,
}
/// PATCH request body for metadata updates.
#[derive(Debug, Deserialize)]
pub struct MetadataUpdate {
pub tags: Option<Vec<String>>,
pub description: Option<String>,
}
/// OS-level file attribute metadata (for xattr / NTFS ADS).
#[derive(Debug, Clone, Default, PartialEq)]
pub struct FileAttributes {
pub mime_type: Option<String>,
pub application: Option<String>,
pub user: Option<String>,
pub tags: Option<String>,
pub description: Option<String>,
pub human_filename: Option<String>,
pub human_path: Option<String>,
}
/// JSON-based data ingest request (agent-friendly, no multipart needed).
///
/// `data` accepts any JSON value — object, array, string, number, etc.
/// It gets serialized to pretty JSON and stored as a `.json` file.
/// Minimal call: `{ "data": {"key": "value"} }`
#[derive(Debug, Deserialize)]
pub struct DataIngestRequest {
/// The payload to store. Any valid JSON value.
pub data: serde_json::Value,
/// Override MIME type. Defaults to `application/json`.
pub mime_type: Option<String>,
/// Logical filename (e.g. "agent_config.json").
pub human_file_name: Option<String>,
/// Logical folder path (e.g. "/configs/").
pub human_readable_path: Option<String>,
/// Application that produced this data.
pub application: Option<String>,
/// User / agent identity.
pub user: Option<String>,
/// Comma-separated tags.
pub tags: Option<String>,
/// Human-readable description.
pub description: Option<String>,
}
/// Query parameters for the list endpoint.
#[derive(Debug, Deserialize)]
pub struct ListParams {
pub limit: Option<i64>,
pub offset: Option<i64>,
pub offset_time: Option<i64>,
pub order: Option<String>,
pub application: Option<String>,
pub include_trashed: Option<bool>,
pub include_corrupted: Option<bool>,
}
/// Query parameters for the search endpoint.
#[derive(Debug, Deserialize)]
pub struct SearchParams {
pub hash: Option<String>,
pub start_time: Option<i64>,
pub end_time: Option<i64>,
pub tags: Option<String>,
pub mime_type: Option<String>,
pub user: Option<String>,
pub application: Option<String>,
pub limit: Option<i64>,
pub offset: Option<i64>,
pub order: Option<String>,
pub include_trashed: Option<bool>,
pub include_corrupted: Option<bool>,
}

101
src/routes/asset.rs Normal file
View File

@ -0,0 +1,101 @@
use axum::body::Body;
use axum::extract::{Path, State};
use axum::http::header;
use axum::response::{IntoResponse, Response};
use axum::routing::{get, patch};
use axum::{Json, Router};
use tokio::fs::File;
use tokio_util::io::ReaderStream;
use crate::error::AppError;
use crate::models::{ApiResponse, FileAttributes, MetadataUpdate};
use crate::{db, xattr, AppState};
pub fn router() -> Router<AppState> {
Router::new()
.route("/api/v1/can/0/asset/{hash}", get(get_asset))
.route("/api/v1/can/0/asset/{hash}", patch(patch_asset))
}
/// GET /api/v1/can/0/asset/{hash} - Stream the physical file.
async fn get_asset(
State(state): State<AppState>,
Path(hash): Path<String>,
) -> Result<Response, AppError> {
let asset = {
let conn = state.db.lock().unwrap();
db::get_asset_by_hash(&conn, &hash)?
.ok_or_else(|| AppError::NotFound(format!("Asset not found: {}", hash)))?
};
if asset.is_corrupted {
return Err(AppError::Corrupted(format!(
"Asset {} is flagged as corrupted",
hash
)));
}
let file_path = state.config.storage_root.join(&asset.actual_filename);
let file = File::open(&file_path).await.map_err(|e| {
AppError::Internal(format!("Failed to open file {}: {}", asset.actual_filename, e))
})?;
let stream = ReaderStream::new(file);
let body = Body::from_stream(stream);
let content_type = asset.mime_type.clone();
let disposition = match &asset.human_filename {
Some(name) => format!("attachment; filename=\"{}\"", name),
None => format!("attachment; filename=\"{}\"", asset.actual_filename),
};
Ok((
[
(header::CONTENT_TYPE, content_type),
(header::CONTENT_DISPOSITION, disposition),
],
body,
)
.into_response())
}
/// PATCH /api/v1/can/0/asset/{hash} - Update metadata (tags, description).
async fn patch_asset(
State(state): State<AppState>,
Path(hash): Path<String>,
Json(update): Json<MetadataUpdate>,
) -> Result<Json<ApiResponse<String>>, AppError> {
let asset = {
let conn = state.db.lock().unwrap();
db::get_asset_by_hash(&conn, &hash)?
.ok_or_else(|| AppError::NotFound(format!("Asset not found: {}", hash)))?
};
// Update DB
{
let conn = state.db.lock().unwrap();
db::update_asset_metadata(
&conn,
&hash,
update.description.as_deref(),
update.tags.as_deref(),
)?;
}
// Update OS attributes
let file_path = state.config.storage_root.join(&asset.actual_filename);
if file_path.exists() {
let mut attrs = FileAttributes::default();
if let Some(ref desc) = update.description {
attrs.description = Some(desc.clone());
}
if let Some(ref tags) = update.tags {
attrs.tags = Some(tags.join(","));
}
if let Err(e) = xattr::write_attributes(&file_path, &attrs) {
tracing::warn!("Failed to update OS attributes: {}", e);
}
}
Ok(Json(ApiResponse::success("updated".to_string())))
}

273
src/routes/ingest.rs Normal file
View File

@ -0,0 +1,273 @@
use axum::extract::{Multipart, State};
use axum::routing::post;
use axum::{Json, Router};
use std::time::{SystemTime, UNIX_EPOCH};
use crate::error::AppError;
use crate::models::{ApiResponse, Asset, DataIngestRequest, FileAttributes, IngestResult};
use crate::{db, hash, storage, xattr, AppState};
pub fn router() -> Router<AppState> {
Router::new()
.route("/api/v1/can/0/ingest", post(ingest_multipart))
.route("/api/v1/can/0/ingest/data", post(ingest_data))
}
// ── Shared ingest pipeline ──────────────────────────────────────────────
/// All the parsed fields needed to ingest an asset, regardless of source.
struct IngestInput {
content: Vec<u8>,
mime_type: String,
human_file_name: Option<String>,
human_readable_path: Option<String>,
application: Option<String>,
user: Option<String>,
tags: Vec<String>,
description: Option<String>,
}
/// Common pipeline: timestamp → hash → write file → xattr → DB insert.
fn do_ingest(state: &AppState, input: IngestInput) -> Result<IngestResult, AppError> {
let timestamp = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_millis() as i64;
let file_hash = hash::compute_hash(timestamp, &input.content);
let actual_filename =
storage::build_filename(timestamp, &file_hash, &input.tags, &input.mime_type);
let file_path =
storage::write_asset(&state.config.storage_root, &actual_filename, &input.content)?;
// OS-level attributes (best-effort)
let attrs = FileAttributes {
mime_type: Some(input.mime_type.clone()),
application: input.application.clone(),
user: input.user.clone(),
tags: if input.tags.is_empty() {
None
} else {
Some(input.tags.join(","))
},
description: input.description.clone(),
human_filename: input.human_file_name.clone(),
human_path: input.human_readable_path.clone(),
};
if let Err(e) = xattr::write_attributes(&file_path, &attrs) {
tracing::warn!("Failed to write OS attributes: {}", e);
}
// Database insert
let asset = Asset {
id: 0,
timestamp,
hash: file_hash.clone(),
mime_type: input.mime_type,
application: input.application,
user_identity: input.user,
description: input.description,
actual_filename: actual_filename.clone(),
human_filename: input.human_file_name,
human_path: input.human_readable_path,
is_trashed: false,
is_corrupted: false,
size: input.content.len() as i64,
};
{
let conn = state.db.lock().unwrap();
let asset_id = db::insert_asset(&conn, &asset)?;
if !input.tags.is_empty() {
db::set_asset_tags(&conn, asset_id, &input.tags)?;
}
}
Ok(IngestResult {
timestamp,
hash: file_hash,
filename: actual_filename,
})
}
/// Parse a comma-separated tag string into a clean Vec.
fn parse_tags(raw: Option<&str>) -> Vec<String> {
raw.unwrap_or("")
.split(',')
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.collect()
}
// ── POST /api/v1/can/0/ingest (multipart — file uploads) ──────────────
async fn ingest_multipart(
State(state): State<AppState>,
mut multipart: Multipart,
) -> Result<Json<ApiResponse<IngestResult>>, AppError> {
let mut file_data: Option<Vec<u8>> = None;
let mut mime_type: Option<String> = None;
let mut human_file_name: Option<String> = None;
let mut human_readable_path: Option<String> = None;
let mut application: Option<String> = None;
let mut user: Option<String> = None;
let mut tags_str: Option<String> = None;
let mut description: Option<String> = None;
let mut original_filename: Option<String> = None;
while let Some(field) = multipart
.next_field()
.await
.map_err(|e| AppError::BadRequest(e.to_string()))?
{
let name = field.name().unwrap_or("").to_string();
match name.as_str() {
"file" => {
if let Some(fname) = field.file_name() {
original_filename = Some(fname.to_string());
}
if let Some(ct) = field.content_type() {
if mime_type.is_none() {
mime_type = Some(ct.to_string());
}
}
file_data = Some(
field
.bytes()
.await
.map_err(|e| AppError::BadRequest(e.to_string()))?
.to_vec(),
);
}
"mime_type" => {
let val = field
.text()
.await
.map_err(|e| AppError::BadRequest(e.to_string()))?;
if !val.is_empty() {
mime_type = Some(val);
}
}
"human_file_name" => {
human_file_name = Some(
field
.text()
.await
.map_err(|e| AppError::BadRequest(e.to_string()))?,
);
}
"human_readable_path" => {
human_readable_path = Some(
field
.text()
.await
.map_err(|e| AppError::BadRequest(e.to_string()))?,
);
}
"application" => {
application = Some(
field
.text()
.await
.map_err(|e| AppError::BadRequest(e.to_string()))?,
);
}
"user" => {
user = Some(
field
.text()
.await
.map_err(|e| AppError::BadRequest(e.to_string()))?,
);
}
"tags" => {
tags_str = Some(
field
.text()
.await
.map_err(|e| AppError::BadRequest(e.to_string()))?,
);
}
"description" => {
description = Some(
field
.text()
.await
.map_err(|e| AppError::BadRequest(e.to_string()))?,
);
}
_ => {}
}
}
let content = file_data.ok_or_else(|| AppError::BadRequest("Missing 'file' field".into()))?;
let resolved_mime = mime_type.unwrap_or_else(|| {
original_filename
.as_deref()
.and_then(|name| mime_guess::from_path(name).first_raw().map(|s| s.to_string()))
.unwrap_or_else(|| "application/octet-stream".to_string())
});
let result = do_ingest(
&state,
IngestInput {
content,
mime_type: resolved_mime,
human_file_name,
human_readable_path,
application,
user,
tags: parse_tags(tags_str.as_deref()),
description,
},
)?;
Ok(Json(ApiResponse::success(result)))
}
// ── POST /api/v1/can/0/ingest/data (JSON — agent-friendly) ────────────
/// JSON data ingest. Accepts any JSON value in `data`, serializes it to
/// pretty-printed JSON bytes, and stores it as a `.json` asset.
///
/// Minimal agent call:
/// ```json
/// POST /api/v1/can/0/ingest/data
/// Content-Type: application/json
///
/// { "data": { "key": "value" } }
/// ```
///
/// All metadata fields (tags, application, user, etc.) are optional —
/// same semantics as the multipart endpoint.
async fn ingest_data(
State(state): State<AppState>,
Json(req): Json<DataIngestRequest>,
) -> Result<Json<ApiResponse<IngestResult>>, AppError> {
// Serialize the data payload to pretty JSON bytes
let content = serde_json::to_vec_pretty(&req.data)
.map_err(|e| AppError::Internal(format!("Failed to serialize data: {}", e)))?;
let mime = req
.mime_type
.unwrap_or_else(|| "application/json".to_string());
let result = do_ingest(
&state,
IngestInput {
content,
mime_type: mime,
human_file_name: req.human_file_name,
human_readable_path: req.human_readable_path,
application: req.application,
user: req.user,
tags: parse_tags(req.tags.as_deref()),
description: req.description,
},
)?;
Ok(Json(ApiResponse::success(result)))
}

35
src/routes/list.rs Normal file
View File

@ -0,0 +1,35 @@
use axum::extract::{Query, State};
use axum::routing::get;
use axum::{Json, Router};
use crate::error::AppError;
use crate::models::{ApiResponse, AssetMeta, ListParams, ListResponse, Pagination};
use crate::{db, AppState};
pub fn router() -> Router<AppState> {
Router::new().route("/api/v1/can/0/list", get(list_assets))
}
async fn list_assets(
State(state): State<AppState>,
Query(params): Query<ListParams>,
) -> Result<Json<ApiResponse<ListResponse>>, AppError> {
let conn = state.db.lock().unwrap();
let (assets, total) = db::list_assets(&conn, &params)?;
let items: Vec<AssetMeta> = assets
.iter()
.map(|a| db::asset_to_meta(&conn, a))
.collect::<Result<Vec<_>, _>>()?;
let response = ListResponse {
items,
pagination: Pagination {
limit: params.limit.unwrap_or(50),
offset: params.offset.unwrap_or(0),
total,
},
};
Ok(Json(ApiResponse::success(response)))
}

22
src/routes/meta.rs Normal file
View File

@ -0,0 +1,22 @@
use axum::extract::{Path, State};
use axum::routing::get;
use axum::{Json, Router};
use crate::error::AppError;
use crate::models::{ApiResponse, AssetMeta};
use crate::{db, AppState};
pub fn router() -> Router<AppState> {
Router::new().route("/api/v1/can/0/asset/{hash}/meta", get(get_meta))
}
async fn get_meta(
State(state): State<AppState>,
Path(hash): Path<String>,
) -> Result<Json<ApiResponse<AssetMeta>>, AppError> {
let conn = state.db.lock().unwrap();
let asset = db::get_asset_by_hash(&conn, &hash)?
.ok_or_else(|| AppError::NotFound(format!("Asset not found: {}", hash)))?;
let meta = db::asset_to_meta(&conn, &asset)?;
Ok(Json(ApiResponse::success(meta)))
}

19
src/routes/mod.rs Normal file
View File

@ -0,0 +1,19 @@
pub mod ingest;
pub mod asset;
pub mod meta;
pub mod list;
pub mod search;
pub mod thumb;
use axum::Router;
use crate::AppState;
pub fn router() -> Router<AppState> {
Router::new()
.merge(ingest::router())
.merge(asset::router())
.merge(meta::router())
.merge(list::router())
.merge(search::router())
.merge(thumb::router())
}

38
src/routes/search.rs Normal file
View File

@ -0,0 +1,38 @@
use axum::extract::{Query, State};
use axum::routing::get;
use axum::{Json, Router};
use crate::error::AppError;
use crate::models::{ApiResponse, AssetMeta, ListResponse, Pagination, SearchParams};
use crate::{db, AppState};
pub fn router() -> Router<AppState> {
Router::new().route("/api/v1/can/0/search", get(search_assets))
}
async fn search_assets(
State(state): State<AppState>,
Query(params): Query<SearchParams>,
) -> Result<Json<ApiResponse<ListResponse>>, AppError> {
let conn = state.db.lock().unwrap();
let limit = params.limit.unwrap_or(50);
let offset = params.offset.unwrap_or(0);
let (assets, total) = db::search_assets(&conn, &params)?;
let items: Vec<AssetMeta> = assets
.iter()
.map(|a| db::asset_to_meta(&conn, a))
.collect::<Result<Vec<_>, _>>()?;
let response = ListResponse {
items,
pagination: Pagination {
limit,
offset,
total,
},
};
Ok(Json(ApiResponse::success(response)))
}

97
src/routes/thumb.rs Normal file
View File

@ -0,0 +1,97 @@
use axum::body::Body;
use axum::extract::{Path, State};
use axum::http::header;
use axum::response::{IntoResponse, Response};
use axum::routing::get;
use axum::Router;
use image::imageops::FilterType;
use image::ImageFormat;
use std::io::Cursor;
use crate::error::AppError;
use crate::{db, AppState};
pub fn router() -> Router<AppState> {
Router::new().route(
"/api/v1/can/0/asset/{hash}/thumb/{max_width}/{max_height}",
get(get_thumb),
)
}
/// Static fallback SVG icon for non-image assets.
const FALLBACK_SVG: &str = r##"<svg xmlns="http://www.w3.org/2000/svg" width="128" height="128" viewBox="0 0 128 128">
<rect width="128" height="128" rx="8" fill="#e0e0e0"/>
<text x="64" y="72" text-anchor="middle" font-family="sans-serif" font-size="40" fill="#888">?</text>
</svg>"##;
async fn get_thumb(
State(state): State<AppState>,
Path((hash, max_width, max_height)): Path<(String, u32, u32)>,
) -> Result<Response, AppError> {
let asset = {
let conn = state.db.lock().unwrap();
db::get_asset_by_hash(&conn, &hash)?
.ok_or_else(|| AppError::NotFound(format!("Asset not found: {}", hash)))?
};
// Check if MIME type is an image we can resize
let is_image = asset.mime_type.starts_with("image/")
&& !asset.mime_type.contains("svg");
if !is_image {
// Return fallback SVG
return Ok((
[(header::CONTENT_TYPE, "image/svg+xml".to_string())],
FALLBACK_SVG.to_string(),
)
.into_response());
}
// Check thumbnail cache
if state.config.enable_thumbnail_cache {
let cache_name = format!("{}_{}x{}.jpg", hash, max_width, max_height);
let cache_path = state.config.thumbs_dir().join(&cache_name);
if cache_path.exists() {
let data = tokio::fs::read(&cache_path).await?;
return Ok((
[(header::CONTENT_TYPE, "image/jpeg".to_string())],
Body::from(data),
)
.into_response());
}
}
// Read original file
let file_path = state.config.storage_root.join(&asset.actual_filename);
let data = tokio::fs::read(&file_path).await.map_err(|e| {
AppError::Internal(format!("Failed to read file: {}", e))
})?;
// Decode and resize
let img = image::load_from_memory(&data)
.map_err(|e| AppError::Internal(format!("Failed to decode image: {}", e)))?;
let thumb = img.resize(max_width, max_height, FilterType::Lanczos3);
// Encode as JPEG
let mut buf = Cursor::new(Vec::new());
thumb
.write_to(&mut buf, ImageFormat::Jpeg)
.map_err(|e| AppError::Internal(format!("Failed to encode thumbnail: {}", e)))?;
let jpeg_bytes = buf.into_inner();
// Cache the thumbnail
if state.config.enable_thumbnail_cache {
let cache_name = format!("{}_{}x{}.jpg", hash, max_width, max_height);
let cache_path = state.config.thumbs_dir().join(&cache_name);
if let Err(e) = tokio::fs::write(&cache_path, &jpeg_bytes).await {
tracing::warn!("Failed to cache thumbnail: {}", e);
}
}
Ok((
[(header::CONTENT_TYPE, "image/jpeg".to_string())],
Body::from(jpeg_bytes),
)
.into_response())
}

192
src/storage.rs Normal file
View File

@ -0,0 +1,192 @@
use std::path::{Path, PathBuf};
/// Build the physical filename per the spec:
/// `{timestamp}_{sha256}_{truncated_tags}.{extension}`
pub fn build_filename(
timestamp: i64,
hash: &str,
tags: &[String],
mime_type: &str,
) -> String {
let extension = mime_to_extension(mime_type);
let base = format!("{}_{}", timestamp, hash);
if tags.is_empty() {
return format!("{}.{}", base, extension);
}
// Sanitize tags: strip non-alphanumeric, join with underscore
let sanitized_tags: Vec<String> = tags
.iter()
.map(|t| t.chars().filter(|c| c.is_alphanumeric()).collect::<String>())
.filter(|t| !t.is_empty())
.collect();
if sanitized_tags.is_empty() {
return format!("{}.{}", base, extension);
}
let tag_part = sanitized_tags.join("_");
// Truncate to keep total filename under ~200 chars (safely under 255)
let max_tag_len = 200usize.saturating_sub(base.len() + extension.len() + 2); // 2 for _ and .
let truncated = if tag_part.len() > max_tag_len {
&tag_part[..max_tag_len]
} else {
&tag_part
};
format!("{}_{}. {}", base, truncated, extension)
.replace(". ", ".")
}
/// Derive file extension from MIME type.
pub fn mime_to_extension(mime: &str) -> &str {
match mime {
"application/pdf" => "pdf",
"application/json" => "json",
"application/xml" | "text/xml" => "xml",
"application/zip" => "zip",
"application/gzip" => "gz",
"application/octet-stream" => "bin",
"text/plain" => "txt",
"text/html" => "html",
"text/css" => "css",
"text/csv" => "csv",
"text/javascript" | "application/javascript" => "js",
"image/jpeg" => "jpg",
"image/png" => "png",
"image/gif" => "gif",
"image/webp" => "webp",
"image/svg+xml" => "svg",
"image/bmp" => "bmp",
"audio/mpeg" => "mp3",
"audio/wav" => "wav",
"audio/ogg" => "ogg",
"video/mp4" => "mp4",
"video/webm" => "webm",
_ => {
// Try to extract from mime_guess
mime_guess::get_mime_extensions_str(mime)
.and_then(|exts| exts.first().copied())
.unwrap_or("bin")
}
}
}
/// Write asset bytes to the storage root. Returns the full path.
pub fn write_asset(root: &Path, filename: &str, data: &[u8]) -> std::io::Result<PathBuf> {
let path = root.join(filename);
std::fs::write(&path, data)?;
Ok(path)
}
/// Read asset bytes from the storage root.
pub fn read_asset(root: &Path, filename: &str) -> std::io::Result<Vec<u8>> {
let path = root.join(filename);
std::fs::read(path)
}
/// Move an asset file to the .trash directory.
pub fn trash_asset_file(root: &Path, filename: &str) -> std::io::Result<()> {
let src = root.join(filename);
let trash_dir = root.join(".trash");
std::fs::create_dir_all(&trash_dir)?;
let dst = trash_dir.join(filename);
std::fs::rename(src, dst)?;
Ok(())
}
/// Parse a physical filename to extract the hash component.
/// Format: `{timestamp}_{sha256}_{tags}.{ext}` or `{timestamp}_{sha256}.{ext}`
pub fn parse_hash_from_filename(filename: &str) -> Option<String> {
// Remove extension
let stem = filename.rsplit_once('.')?.0;
// Split by underscore: first part is timestamp, second is hash (64 hex chars)
let parts: Vec<&str> = stem.splitn(3, '_').collect();
if parts.len() >= 2 && parts[1].len() == 64 {
Some(parts[1].to_string())
} else {
None
}
}
/// Parse a physical filename to extract the timestamp component.
pub fn parse_timestamp_from_filename(filename: &str) -> Option<i64> {
let stem = filename.rsplit_once('.')?.0;
let ts_str = stem.split('_').next()?;
ts_str.parse().ok()
}
#[cfg(test)]
mod tests {
use super::*;
use tempfile::TempDir;
#[test]
fn test_build_filename_no_tags() {
let name = build_filename(1773014400123, "a3b2c4d5e6f7", &[], "application/pdf");
assert_eq!(name, "1773014400123_a3b2c4d5e6f7.pdf");
}
#[test]
fn test_build_filename_with_tags() {
let tags = vec!["photo".to_string(), "vacation".to_string()];
let name = build_filename(1773014400123, "a3b2c4d5e6f7", &tags, "image/jpeg");
assert_eq!(name, "1773014400123_a3b2c4d5e6f7_photo_vacation.jpg");
}
#[test]
fn test_build_filename_strips_special_chars_from_tags() {
let tags = vec!["hello world!".to_string(), "test@123".to_string()];
let name = build_filename(100, "abc", &tags, "text/plain");
assert_eq!(name, "100_abc_helloworld_test123.txt");
}
#[test]
fn test_mime_to_extension() {
assert_eq!(mime_to_extension("image/png"), "png");
assert_eq!(mime_to_extension("application/pdf"), "pdf");
assert_eq!(mime_to_extension("text/plain"), "txt");
assert_eq!(mime_to_extension("unknown/thing"), "bin");
}
#[test]
fn test_write_and_read_asset() {
let dir = TempDir::new().unwrap();
let data = b"hello world";
let path = write_asset(dir.path(), "test_file.txt", data).unwrap();
assert!(path.exists());
let read_back = read_asset(dir.path(), "test_file.txt").unwrap();
assert_eq!(read_back, data);
}
#[test]
fn test_trash_asset_file() {
let dir = TempDir::new().unwrap();
write_asset(dir.path(), "to_trash.txt", b"bye").unwrap();
trash_asset_file(dir.path(), "to_trash.txt").unwrap();
assert!(!dir.path().join("to_trash.txt").exists());
assert!(dir.path().join(".trash").join("to_trash.txt").exists());
}
#[test]
fn test_parse_hash_from_filename() {
let hash_64 = "a".repeat(64);
let filename = format!("1773014400123_{}.pdf", hash_64);
assert_eq!(parse_hash_from_filename(&filename), Some(hash_64.clone()));
let filename_tags = format!("1773014400123_{}_photo_vacation.jpg", hash_64);
assert_eq!(parse_hash_from_filename(&filename_tags), Some(hash_64));
}
#[test]
fn test_parse_timestamp_from_filename() {
let hash_64 = "b".repeat(64);
let filename = format!("1773014400123_{}.pdf", hash_64);
assert_eq!(parse_timestamp_from_filename(&filename), Some(1773014400123));
}
}

308
src/verifier.rs Normal file
View File

@ -0,0 +1,308 @@
use notify::{Event, EventKind, RecursiveMode, Watcher};
use std::path::PathBuf;
// Mutex used via Db type alias
use std::time::Duration;
use tokio::sync::mpsc;
use crate::config::Config;
use crate::db::{self, Db};
use crate::hash::compute_hash;
use crate::models::FileAttributes;
use crate::storage::{parse_hash_from_filename, parse_timestamp_from_filename};
use crate::xattr;
/// Start the background verifier subsystem.
/// - Runs an initial full scrub
/// - Watches for filesystem changes
/// - Runs periodic scrubs
pub fn start(config: Config, db: Db) {
let config2 = config.clone();
let db2 = db.clone();
// Initial scrub
let config3 = config.clone();
let db3 = db.clone();
tokio::spawn(async move {
tracing::info!("Verifier: starting initial scrub...");
if let Err(e) = run_scrub(&config3, &db3).await {
tracing::error!("Verifier: initial scrub failed: {}", e);
}
tracing::info!("Verifier: initial scrub complete");
});
// Periodic scrub
let interval_hours = config.verify_interval_hours;
tokio::spawn(async move {
let mut interval =
tokio::time::interval(Duration::from_secs(interval_hours * 3600));
interval.tick().await; // Skip first immediate tick
loop {
interval.tick().await;
tracing::info!("Verifier: starting periodic scrub...");
if let Err(e) = run_scrub(&config2, &db2).await {
tracing::error!("Verifier: periodic scrub failed: {}", e);
}
tracing::info!("Verifier: periodic scrub complete");
}
});
// Filesystem watcher
tokio::spawn(async move {
if let Err(e) = run_watcher(config3_for_watcher(config), db).await {
tracing::error!("Verifier: filesystem watcher failed: {}", e);
}
});
}
fn config3_for_watcher(config: Config) -> Config {
config
}
async fn run_watcher(config: Config, db: Db) -> anyhow::Result<()> {
let (tx, mut rx) = mpsc::channel::<PathBuf>(100);
let storage_root = config.storage_root.clone();
// Spawn blocking watcher in a separate thread
let watcher_root = storage_root.clone();
std::thread::spawn(move || {
let tx_clone = tx.clone();
let mut watcher = notify::recommended_watcher(move |res: Result<Event, _>| {
if let Ok(event) = res {
match event.kind {
EventKind::Modify(_) | EventKind::Create(_) => {
for path in event.paths {
// Ignore hidden dirs (.trash, .thumbs, .can.db)
let filename = path
.file_name()
.and_then(|f| f.to_str())
.unwrap_or("");
if filename.starts_with('.') {
continue;
}
let _ = tx_clone.blocking_send(path);
}
}
_ => {}
}
}
})
.expect("Failed to create filesystem watcher");
watcher
.watch(&watcher_root, RecursiveMode::NonRecursive)
.expect("Failed to watch storage root");
// Keep watcher alive
loop {
std::thread::sleep(Duration::from_secs(3600));
}
});
// Process file change events
while let Some(path) = rx.recv().await {
let filename = match path.file_name().and_then(|f| f.to_str()) {
Some(f) => f.to_string(),
None => continue,
};
tracing::debug!("Verifier: checking modified file: {}", filename);
if let Err(e) = verify_single_file(&config, &db, &filename).await {
tracing::warn!("Verifier: error checking {}: {}", filename, e);
}
}
Ok(())
}
/// Run a full scrub: verify every active asset's hash.
async fn run_scrub(config: &Config, db: &Db) -> anyhow::Result<()> {
let assets = {
let conn = db.lock().unwrap();
db::get_all_active_assets(&conn)?
};
let mut corrupted_count = 0u32;
for asset in &assets {
let file_path = config.storage_root.join(&asset.actual_filename);
if !file_path.exists() {
tracing::warn!(
"Verifier: file missing for asset {}: {}",
asset.hash,
asset.actual_filename
);
continue;
}
match tokio::fs::read(&file_path).await {
Ok(content) => {
let expected_hash = compute_hash(asset.timestamp, &content);
if expected_hash != asset.hash {
tracing::warn!(
"Verifier: CORRUPTION detected for {} (expected {}, got {})",
asset.actual_filename,
asset.hash,
expected_hash
);
let conn = db.lock().unwrap();
db::flag_corrupted(&conn, &asset.hash, true)?;
corrupted_count += 1;
} else if asset.is_corrupted {
// File was previously marked corrupted but now passes - clear flag
let conn = db.lock().unwrap();
db::flag_corrupted(&conn, &asset.hash, false)?;
tracing::info!(
"Verifier: asset {} is no longer corrupted",
asset.hash
);
}
}
Err(e) => {
tracing::warn!(
"Verifier: cannot read {}: {}",
asset.actual_filename,
e
);
}
}
}
if corrupted_count > 0 {
tracing::warn!(
"Verifier: scrub found {} corrupted assets out of {}",
corrupted_count,
assets.len()
);
} else {
tracing::info!(
"Verifier: scrub passed for all {} assets",
assets.len()
);
}
// Sync DB metadata → OS-level file attributes
let mut attrs_synced = 0u32;
for asset in &assets {
let file_path = config.storage_root.join(&asset.actual_filename);
if !file_path.exists() {
continue;
}
// Build expected attributes from DB
let tags = {
let conn = db.lock().unwrap();
db::get_asset_tags(&conn, asset.id).unwrap_or_default()
};
let expected = FileAttributes {
mime_type: Some(asset.mime_type.clone()),
application: asset.application.clone(),
user: asset.user_identity.clone(),
tags: if tags.is_empty() {
None
} else {
Some(tags.join(","))
},
description: asset.description.clone(),
human_filename: asset.human_filename.clone(),
human_path: asset.human_path.clone(),
};
// Read current file attributes
let current = xattr::read_attributes(&file_path).unwrap_or_default();
if current != expected {
if let Err(e) = xattr::write_attributes(&file_path, &expected) {
tracing::warn!(
"Verifier: failed to sync attributes for {}: {}",
asset.actual_filename,
e
);
} else {
attrs_synced += 1;
}
}
}
if attrs_synced > 0 {
tracing::info!(
"Verifier: synced file attributes for {} assets",
attrs_synced
);
}
// Backfill missing file sizes
let mut sizes_backfilled = 0u32;
for asset in &assets {
if asset.size > 0 {
continue;
}
let file_path = config.storage_root.join(&asset.actual_filename);
if !file_path.exists() {
continue;
}
match file_path.metadata() {
Ok(meta) => {
let len = meta.len() as i64;
if len > 0 {
let conn = db.lock().unwrap();
if let Err(e) = db::update_asset_size(&conn, &asset.hash, len) {
tracing::warn!(
"Verifier: failed to backfill size for {}: {}",
asset.hash,
e
);
} else {
sizes_backfilled += 1;
}
}
}
Err(e) => {
tracing::warn!(
"Verifier: cannot stat {}: {}",
asset.actual_filename,
e
);
}
}
}
if sizes_backfilled > 0 {
tracing::info!(
"Verifier: backfilled file sizes for {} assets",
sizes_backfilled
);
}
Ok(())
}
/// Verify a single file by its physical filename.
async fn verify_single_file(
config: &Config,
db: &Db,
filename: &str,
) -> anyhow::Result<()> {
let hash = match parse_hash_from_filename(filename) {
Some(h) => h,
None => return Ok(()), // Not a CAN-managed file
};
let timestamp = match parse_timestamp_from_filename(filename) {
Some(t) => t,
None => return Ok(()),
};
let file_path = config.storage_root.join(filename);
let content = tokio::fs::read(&file_path).await?;
let computed = compute_hash(timestamp, &content);
if computed != hash {
tracing::warn!(
"Verifier: corruption detected on change for {}",
filename
);
let conn = db.lock().unwrap();
db::flag_corrupted(&conn, &hash, true)?;
}
Ok(())
}

187
src/xattr.rs Normal file
View File

@ -0,0 +1,187 @@
use crate::models::FileAttributes;
use std::path::Path;
/// Write CAN metadata as OS-level file attributes.
/// - Unix/macOS: Extended Attributes (xattr)
/// - Windows: NTFS Alternate Data Streams
pub fn write_attributes(path: &Path, attrs: &FileAttributes) -> std::io::Result<()> {
#[cfg(unix)]
{
write_xattr(path, attrs)
}
#[cfg(windows)]
{
write_ntfs_ads(path, attrs)
}
}
/// Read CAN metadata from OS-level file attributes.
pub fn read_attributes(path: &Path) -> std::io::Result<FileAttributes> {
#[cfg(unix)]
{
read_xattr(path)
}
#[cfg(windows)]
{
read_ntfs_ads(path)
}
}
// ── Unix implementation using xattr crate ──
#[cfg(unix)]
fn write_xattr(path: &Path, attrs: &FileAttributes) -> std::io::Result<()> {
use xattr::FileExt;
let file = std::fs::File::open(path)?;
if let Some(ref v) = attrs.mime_type {
file.set_xattr("user.can.mime_type", v.as_bytes())?;
}
if let Some(ref v) = attrs.application {
file.set_xattr("user.can.application", v.as_bytes())?;
}
if let Some(ref v) = attrs.user {
file.set_xattr("user.can.user", v.as_bytes())?;
}
if let Some(ref v) = attrs.tags {
file.set_xattr("user.can.tags", v.as_bytes())?;
}
if let Some(ref v) = attrs.description {
file.set_xattr("user.can.description", v.as_bytes())?;
}
if let Some(ref v) = attrs.human_filename {
file.set_xattr("user.can.human_filename", v.as_bytes())?;
}
if let Some(ref v) = attrs.human_path {
file.set_xattr("user.can.human_path", v.as_bytes())?;
}
Ok(())
}
#[cfg(unix)]
fn read_xattr(path: &Path) -> std::io::Result<FileAttributes> {
use xattr::FileExt;
let file = std::fs::File::open(path)?;
let read_attr = |name: &str| -> Option<String> {
file.get_xattr(name)
.ok()
.flatten()
.and_then(|bytes| String::from_utf8(bytes).ok())
};
Ok(FileAttributes {
mime_type: read_attr("user.can.mime_type"),
application: read_attr("user.can.application"),
user: read_attr("user.can.user"),
tags: read_attr("user.can.tags"),
description: read_attr("user.can.description"),
human_filename: read_attr("user.can.human_filename"),
human_path: read_attr("user.can.human_path"),
})
}
// ── Windows implementation using NTFS Alternate Data Streams ──
#[cfg(windows)]
fn write_ntfs_ads(path: &Path, attrs: &FileAttributes) -> std::io::Result<()> {
let base = path.to_string_lossy();
if let Some(ref v) = attrs.mime_type {
std::fs::write(format!("{}:can.mime_type", base), v)?;
}
if let Some(ref v) = attrs.application {
std::fs::write(format!("{}:can.application", base), v)?;
}
if let Some(ref v) = attrs.user {
std::fs::write(format!("{}:can.user", base), v)?;
}
if let Some(ref v) = attrs.tags {
std::fs::write(format!("{}:can.tags", base), v)?;
}
if let Some(ref v) = attrs.description {
std::fs::write(format!("{}:can.description", base), v)?;
}
if let Some(ref v) = attrs.human_filename {
std::fs::write(format!("{}:can.human_filename", base), v)?;
}
if let Some(ref v) = attrs.human_path {
std::fs::write(format!("{}:can.human_path", base), v)?;
}
Ok(())
}
#[cfg(windows)]
fn read_ntfs_ads(path: &Path) -> std::io::Result<FileAttributes> {
let base = path.to_string_lossy();
let read_stream = |name: &str| -> Option<String> {
std::fs::read_to_string(format!("{}:{}", base, name)).ok()
};
Ok(FileAttributes {
mime_type: read_stream("can.mime_type"),
application: read_stream("can.application"),
user: read_stream("can.user"),
tags: read_stream("can.tags"),
description: read_stream("can.description"),
human_filename: read_stream("can.human_filename"),
human_path: read_stream("can.human_path"),
})
}
#[cfg(test)]
mod tests {
use super::*;
use tempfile::NamedTempFile;
#[test]
fn test_write_and_read_attributes() {
let file = NamedTempFile::new().unwrap();
std::fs::write(file.path(), b"test content").unwrap();
let attrs = FileAttributes {
mime_type: Some("image/jpeg".to_string()),
application: Some("TestApp".to_string()),
user: Some("jason".to_string()),
tags: Some("photo,vacation,2024".to_string()),
description: Some("A test file".to_string()),
human_filename: Some("my_photo.jpg".to_string()),
human_path: Some("/photos/trip/".to_string()),
};
write_attributes(file.path(), &attrs).unwrap();
let read_back = read_attributes(file.path()).unwrap();
assert_eq!(read_back.mime_type, Some("image/jpeg".to_string()));
assert_eq!(read_back.application, Some("TestApp".to_string()));
assert_eq!(read_back.user, Some("jason".to_string()));
assert_eq!(read_back.tags, Some("photo,vacation,2024".to_string()));
assert_eq!(read_back.description, Some("A test file".to_string()));
assert_eq!(read_back.human_filename, Some("my_photo.jpg".to_string()));
assert_eq!(read_back.human_path, Some("/photos/trip/".to_string()));
}
#[test]
fn test_partial_attributes() {
let file = NamedTempFile::new().unwrap();
std::fs::write(file.path(), b"data").unwrap();
let attrs = FileAttributes {
mime_type: None,
application: Some("App".to_string()),
user: None,
tags: None,
description: None,
human_filename: None,
human_path: None,
};
write_attributes(file.path(), &attrs).unwrap();
let read_back = read_attributes(file.path()).unwrap();
assert_eq!(read_back.application, Some("App".to_string()));
assert_eq!(read_back.user, None);
assert_eq!(read_back.tags, None);
}
}

741
tests/integration.rs Normal file
View File

@ -0,0 +1,741 @@
use reqwest::multipart;
use std::sync::Arc;
use tempfile::TempDir;
use tokio::net::TcpListener;
// We need to import the binary crate's internals.
// Since integration tests can't access `mod` items directly, we'll spin up
// the server using the library-like approach by duplicating setup logic.
// A cleaner approach is to test through HTTP.
/// Helper: spin up a test server and return its base URL + temp dir handle.
async fn spawn_test_server() -> (String, TempDir) {
let tmp = TempDir::new().unwrap();
let storage_root = tmp.path().to_path_buf();
// Create config.yaml in tempdir
let config_content = format!(
r#"storage_root: "{}"
admin_token: "test_token"
enable_thumbnail_cache: true
rebuild_error_threshold: 50
verify_interval_hours: 999
"#,
storage_root.to_string_lossy().replace('\\', "/")
);
let config_path = tmp.path().join("config.yaml");
std::fs::write(&config_path, &config_content).unwrap();
// Load config
let config: can_service::config::Config =
serde_yaml::from_str(&config_content).unwrap();
config.ensure_dirs().unwrap();
// Open DB
let db = can_service::db::open(&config.db_path()).unwrap();
let config = Arc::new(config);
let state = can_service::AppState {
config: config.clone(),
db,
};
// Build router
let app = axum::Router::new()
.merge(can_service::routes::router())
.with_state(state);
// Bind to random port
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let base_url = format!("http://{}", addr);
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
// Give the server a moment to start
tokio::time::sleep(std::time::Duration::from_millis(50)).await;
(base_url, tmp)
}
#[tokio::test]
async fn test_ingest_and_retrieve_metadata() {
let (base_url, _tmp) = spawn_test_server().await;
let client = reqwest::Client::new();
// Ingest a file
let file_part = multipart::Part::bytes(b"hello world".to_vec())
.file_name("hello.txt")
.mime_str("text/plain")
.unwrap();
let form = multipart::Form::new()
.part("file", file_part)
.text("application", "TestApp")
.text("user", "jason")
.text("tags", "greeting,test")
.text("description", "A test file")
.text("human_file_name", "hello.txt")
.text("human_readable_path", "/docs/");
let resp = client
.post(format!("{}/api/v1/can/0/ingest", base_url))
.multipart(form)
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let body: serde_json::Value = resp.json().await.unwrap();
assert_eq!(body["status"], "success");
let hash = body["data"]["hash"].as_str().unwrap().to_string();
let timestamp = body["data"]["timestamp"].as_i64().unwrap();
assert!(!hash.is_empty());
assert!(timestamp > 0);
// Retrieve metadata
let resp = client
.get(format!("{}/api/v1/can/0/asset/{}/meta", base_url, hash))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let body: serde_json::Value = resp.json().await.unwrap();
assert_eq!(body["status"], "success");
assert_eq!(body["data"]["hash"], hash);
assert_eq!(body["data"]["mime_type"], "text/plain");
assert_eq!(body["data"]["application"], "TestApp");
assert_eq!(body["data"]["user"], "jason");
assert_eq!(body["data"]["description"], "A test file");
assert_eq!(body["data"]["human_filename"], "hello.txt");
assert_eq!(body["data"]["human_path"], "/docs/");
let tags = body["data"]["tags"].as_array().unwrap();
assert!(tags.contains(&serde_json::json!("greeting")));
assert!(tags.contains(&serde_json::json!("test")));
}
#[tokio::test]
async fn test_retrieve_physical_asset() {
let (base_url, _tmp) = spawn_test_server().await;
let client = reqwest::Client::new();
let file_content = b"binary content here";
let file_part = multipart::Part::bytes(file_content.to_vec())
.file_name("data.bin")
.mime_str("application/octet-stream")
.unwrap();
let form = multipart::Form::new().part("file", file_part);
let resp = client
.post(format!("{}/api/v1/can/0/ingest", base_url))
.multipart(form)
.send()
.await
.unwrap();
let body: serde_json::Value = resp.json().await.unwrap();
let hash = body["data"]["hash"].as_str().unwrap();
// Download the asset
let resp = client
.get(format!("{}/api/v1/can/0/asset/{}", base_url, hash))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let downloaded = resp.bytes().await.unwrap();
assert_eq!(downloaded.as_ref(), file_content);
}
#[tokio::test]
async fn test_patch_metadata() {
let (base_url, _tmp) = spawn_test_server().await;
let client = reqwest::Client::new();
// Ingest
let file_part = multipart::Part::bytes(b"patch me".to_vec())
.file_name("patch.txt")
.mime_str("text/plain")
.unwrap();
let form = multipart::Form::new()
.part("file", file_part)
.text("tags", "original")
.text("description", "original desc");
let resp = client
.post(format!("{}/api/v1/can/0/ingest", base_url))
.multipart(form)
.send()
.await
.unwrap();
let body: serde_json::Value = resp.json().await.unwrap();
let hash = body["data"]["hash"].as_str().unwrap().to_string();
// Patch
let patch_body = serde_json::json!({
"tags": ["updated", "new_tag"],
"description": "updated description"
});
let resp = client
.patch(format!("{}/api/v1/can/0/asset/{}", base_url, hash))
.json(&patch_body)
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
// Verify
let resp = client
.get(format!("{}/api/v1/can/0/asset/{}/meta", base_url, hash))
.send()
.await
.unwrap();
let body: serde_json::Value = resp.json().await.unwrap();
assert_eq!(body["data"]["description"], "updated description");
let tags = body["data"]["tags"].as_array().unwrap();
assert!(tags.contains(&serde_json::json!("updated")));
assert!(tags.contains(&serde_json::json!("new_tag")));
assert!(!tags.contains(&serde_json::json!("original")));
}
#[tokio::test]
async fn test_list_assets_pagination() {
let (base_url, _tmp) = spawn_test_server().await;
let client = reqwest::Client::new();
// Ingest 5 files
for i in 0..5 {
let content = format!("file content {}", i);
let file_part = multipart::Part::bytes(content.into_bytes())
.file_name(format!("file_{}.txt", i))
.mime_str("text/plain")
.unwrap();
let form = multipart::Form::new()
.part("file", file_part)
.text("application", "ListTest");
client
.post(format!("{}/api/v1/can/0/ingest", base_url))
.multipart(form)
.send()
.await
.unwrap();
// Small delay so timestamps differ
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
}
// List with limit=2
let resp = client
.get(format!(
"{}/api/v1/can/0/list?limit=2&offset=0",
base_url
))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let body: serde_json::Value = resp.json().await.unwrap();
let items = body["data"]["items"].as_array().unwrap();
assert_eq!(items.len(), 2);
assert_eq!(body["data"]["pagination"]["total"], 5);
assert_eq!(body["data"]["pagination"]["limit"], 2);
// List with offset=3
let resp = client
.get(format!(
"{}/api/v1/can/0/list?limit=10&offset=3",
base_url
))
.send()
.await
.unwrap();
let body: serde_json::Value = resp.json().await.unwrap();
let items = body["data"]["items"].as_array().unwrap();
assert_eq!(items.len(), 2); // 5 total, offset 3 = 2 remaining
// List with application filter
let resp = client
.get(format!(
"{}/api/v1/can/0/list?application=ListTest",
base_url
))
.send()
.await
.unwrap();
let body: serde_json::Value = resp.json().await.unwrap();
assert_eq!(body["data"]["pagination"]["total"], 5);
}
#[tokio::test]
async fn test_search_assets() {
let (base_url, _tmp) = spawn_test_server().await;
let client = reqwest::Client::new();
// Ingest files with different tags and metadata
let ingest = |name: &str, tags: &str, mime: &str| {
let base = base_url.clone();
let client = client.clone();
let name = name.to_string();
let tags = tags.to_string();
let mime = mime.to_string();
async move {
let file_part = multipart::Part::bytes(format!("content of {}", name).into_bytes())
.file_name(name.clone())
.mime_str(&mime)
.unwrap();
let form = multipart::Form::new()
.part("file", file_part)
.text("tags", tags)
.text("user", "tester");
let resp = client
.post(format!("{}/api/v1/can/0/ingest", base))
.multipart(form)
.send()
.await
.unwrap();
let body: serde_json::Value = resp.json().await.unwrap();
body["data"]["hash"].as_str().unwrap().to_string()
}
};
let _h1 = ingest("photo.jpg", "nature,landscape", "image/jpeg").await;
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
let _h2 = ingest("doc.pdf", "work,report", "application/pdf").await;
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
let _h3 = ingest("nature.png", "nature,macro", "image/png").await;
// Search by tags
let resp = client
.get(format!(
"{}/api/v1/can/0/search?tags=nature",
base_url
))
.send()
.await
.unwrap();
let body: serde_json::Value = resp.json().await.unwrap();
assert_eq!(body["data"]["pagination"]["total"], 2);
// Search by mime_type
let resp = client
.get(format!(
"{}/api/v1/can/0/search?mime_type=application/pdf",
base_url
))
.send()
.await
.unwrap();
let body: serde_json::Value = resp.json().await.unwrap();
assert_eq!(body["data"]["pagination"]["total"], 1);
// Search by user
let resp = client
.get(format!(
"{}/api/v1/can/0/search?user=tester",
base_url
))
.send()
.await
.unwrap();
let body: serde_json::Value = resp.json().await.unwrap();
assert_eq!(body["data"]["pagination"]["total"], 3);
}
#[tokio::test]
async fn test_asset_not_found() {
let (base_url, _tmp) = spawn_test_server().await;
let client = reqwest::Client::new();
let resp = client
.get(format!(
"{}/api/v1/can/0/asset/nonexistent_hash",
base_url
))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 404);
let resp = client
.get(format!(
"{}/api/v1/can/0/asset/nonexistent_hash/meta",
base_url
))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 404);
}
#[tokio::test]
async fn test_thumbnail_fallback_svg() {
let (base_url, _tmp) = spawn_test_server().await;
let client = reqwest::Client::new();
// Ingest a non-image file
let file_part = multipart::Part::bytes(b"not an image".to_vec())
.file_name("doc.txt")
.mime_str("text/plain")
.unwrap();
let form = multipart::Form::new().part("file", file_part);
let resp = client
.post(format!("{}/api/v1/can/0/ingest", base_url))
.multipart(form)
.send()
.await
.unwrap();
let body: serde_json::Value = resp.json().await.unwrap();
let hash = body["data"]["hash"].as_str().unwrap();
// Request thumbnail - should get SVG fallback
let resp = client
.get(format!(
"{}/api/v1/can/0/asset/{}/thumb/128/128",
base_url, hash
))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let content_type = resp.headers().get("content-type").unwrap().to_str().unwrap();
assert!(content_type.contains("svg"));
}
#[tokio::test]
async fn test_list_order() {
let (base_url, _tmp) = spawn_test_server().await;
let client = reqwest::Client::new();
// Ingest 3 files with delays
for i in 0..3 {
let file_part = multipart::Part::bytes(format!("order test {}", i).into_bytes())
.file_name(format!("order_{}.txt", i))
.mime_str("text/plain")
.unwrap();
let form = multipart::Form::new().part("file", file_part);
client
.post(format!("{}/api/v1/can/0/ingest", base_url))
.multipart(form)
.send()
.await
.unwrap();
tokio::time::sleep(std::time::Duration::from_millis(15)).await;
}
// List descending (default)
let resp = client
.get(format!("{}/api/v1/can/0/list?order=desc", base_url))
.send()
.await
.unwrap();
let body: serde_json::Value = resp.json().await.unwrap();
let items = body["data"]["items"].as_array().unwrap();
assert_eq!(items.len(), 3);
let ts0 = items[0]["timestamp"].as_i64().unwrap();
let ts1 = items[1]["timestamp"].as_i64().unwrap();
let ts2 = items[2]["timestamp"].as_i64().unwrap();
assert!(ts0 > ts1);
assert!(ts1 > ts2);
// List ascending
let resp = client
.get(format!("{}/api/v1/can/0/list?order=asc", base_url))
.send()
.await
.unwrap();
let body: serde_json::Value = resp.json().await.unwrap();
let items = body["data"]["items"].as_array().unwrap();
let ts0 = items[0]["timestamp"].as_i64().unwrap();
let ts1 = items[1]["timestamp"].as_i64().unwrap();
assert!(ts0 < ts1);
}
// ── JSON data ingest tests ──────────────────────────────────────────────
#[tokio::test]
async fn test_data_ingest_minimal() {
let (base_url, _tmp) = spawn_test_server().await;
let client = reqwest::Client::new();
// Minimal call: just data
let resp = client
.post(format!("{}/api/v1/can/0/ingest/data", base_url))
.json(&serde_json::json!({
"data": { "key": "value", "count": 42 }
}))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let body: serde_json::Value = resp.json().await.unwrap();
assert_eq!(body["status"], "success");
let hash = body["data"]["hash"].as_str().unwrap();
let filename = body["data"]["filename"].as_str().unwrap();
assert!(!hash.is_empty());
assert!(filename.ends_with(".json"));
// Retrieve and verify it's stored as pretty JSON
let resp = client
.get(format!("{}/api/v1/can/0/asset/{}", base_url, hash))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let stored: serde_json::Value = resp.json().await.unwrap();
assert_eq!(stored["key"], "value");
assert_eq!(stored["count"], 42);
// Verify metadata defaults to application/json
let resp = client
.get(format!("{}/api/v1/can/0/asset/{}/meta", base_url, hash))
.send()
.await
.unwrap();
let meta: serde_json::Value = resp.json().await.unwrap();
assert_eq!(meta["data"]["mime_type"], "application/json");
}
#[tokio::test]
async fn test_data_ingest_with_all_metadata() {
let (base_url, _tmp) = spawn_test_server().await;
let client = reqwest::Client::new();
let resp = client
.post(format!("{}/api/v1/can/0/ingest/data", base_url))
.json(&serde_json::json!({
"data": {
"agent_id": "planner-v2",
"session": "abc-123",
"output": ["step1", "step2", "step3"]
},
"application": "AgentOrchestrator",
"user": "agent_planner",
"tags": "agent,plan,session",
"description": "Planning agent output for session abc-123",
"human_file_name": "plan_output.json",
"human_readable_path": "/agents/planner/"
}))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let body: serde_json::Value = resp.json().await.unwrap();
let hash = body["data"]["hash"].as_str().unwrap();
// Verify all metadata persisted
let resp = client
.get(format!("{}/api/v1/can/0/asset/{}/meta", base_url, hash))
.send()
.await
.unwrap();
let meta: serde_json::Value = resp.json().await.unwrap();
assert_eq!(meta["data"]["application"], "AgentOrchestrator");
assert_eq!(meta["data"]["user"], "agent_planner");
assert_eq!(meta["data"]["description"], "Planning agent output for session abc-123");
assert_eq!(meta["data"]["human_filename"], "plan_output.json");
assert_eq!(meta["data"]["human_path"], "/agents/planner/");
assert_eq!(meta["data"]["mime_type"], "application/json");
let tags = meta["data"]["tags"].as_array().unwrap();
assert!(tags.contains(&serde_json::json!("agent")));
assert!(tags.contains(&serde_json::json!("plan")));
assert!(tags.contains(&serde_json::json!("session")));
}
#[tokio::test]
async fn test_data_ingest_various_json_types() {
let (base_url, _tmp) = spawn_test_server().await;
let client = reqwest::Client::new();
// Store a plain string
let resp = client
.post(format!("{}/api/v1/can/0/ingest/data", base_url))
.json(&serde_json::json!({
"data": "just a plain string log entry",
"tags": "log"
}))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let body: serde_json::Value = resp.json().await.unwrap();
let hash_str = body["data"]["hash"].as_str().unwrap();
// Retrieve and verify the string was stored
let resp = client
.get(format!("{}/api/v1/can/0/asset/{}", base_url, hash_str))
.send()
.await
.unwrap();
let stored: serde_json::Value = resp.json().await.unwrap();
assert_eq!(stored, "just a plain string log entry");
// Store an array
let resp = client
.post(format!("{}/api/v1/can/0/ingest/data", base_url))
.json(&serde_json::json!({
"data": [1, 2, 3, "four", null, true]
}))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let body: serde_json::Value = resp.json().await.unwrap();
let hash_arr = body["data"]["hash"].as_str().unwrap();
let resp = client
.get(format!("{}/api/v1/can/0/asset/{}", base_url, hash_arr))
.send()
.await
.unwrap();
let stored: serde_json::Value = resp.json().await.unwrap();
assert_eq!(stored, serde_json::json!([1, 2, 3, "four", null, true]));
// Store a number
let resp = client
.post(format!("{}/api/v1/can/0/ingest/data", base_url))
.json(&serde_json::json!({
"data": 99.5
}))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
}
#[tokio::test]
async fn test_data_ingest_shows_up_in_list_and_search() {
let (base_url, _tmp) = spawn_test_server().await;
let client = reqwest::Client::new();
// Ingest via JSON data endpoint
client
.post(format!("{}/api/v1/can/0/ingest/data", base_url))
.json(&serde_json::json!({
"data": { "sensor": "temperature", "value": 22.5 },
"application": "IoTAgent",
"tags": "sensor,temperature"
}))
.send()
.await
.unwrap();
// Also ingest via multipart
let file_part = multipart::Part::bytes(b"binary sensor log".to_vec())
.file_name("sensor.bin")
.mime_str("application/octet-stream")
.unwrap();
let form = multipart::Form::new()
.part("file", file_part)
.text("application", "IoTAgent")
.text("tags", "sensor,binary");
client
.post(format!("{}/api/v1/can/0/ingest", base_url))
.multipart(form)
.send()
.await
.unwrap();
tokio::time::sleep(std::time::Duration::from_millis(20)).await;
// Both should show up in list
let resp = client
.get(format!("{}/api/v1/can/0/list?application=IoTAgent", base_url))
.send()
.await
.unwrap();
let body: serde_json::Value = resp.json().await.unwrap();
assert_eq!(body["data"]["pagination"]["total"], 2);
// Search by tag should find the JSON one
let resp = client
.get(format!("{}/api/v1/can/0/search?tags=temperature", base_url))
.send()
.await
.unwrap();
let body: serde_json::Value = resp.json().await.unwrap();
assert_eq!(body["data"]["pagination"]["total"], 1);
assert_eq!(body["data"]["items"][0]["mime_type"], "application/json");
}
#[tokio::test]
async fn test_data_ingest_custom_mime_type() {
let (base_url, _tmp) = spawn_test_server().await;
let client = reqwest::Client::new();
// Agent stores data but overrides mime_type to text/plain
let resp = client
.post(format!("{}/api/v1/can/0/ingest/data", base_url))
.json(&serde_json::json!({
"data": "This is a plain text log line from the agent",
"mime_type": "text/plain",
"human_file_name": "agent.log"
}))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let body: serde_json::Value = resp.json().await.unwrap();
let filename = body["data"]["filename"].as_str().unwrap();
assert!(filename.ends_with(".txt"), "Expected .txt extension, got {}", filename);
let hash = body["data"]["hash"].as_str().unwrap();
let resp = client
.get(format!("{}/api/v1/can/0/asset/{}/meta", base_url, hash))
.send()
.await
.unwrap();
let meta: serde_json::Value = resp.json().await.unwrap();
assert_eq!(meta["data"]["mime_type"], "text/plain");
assert_eq!(meta["data"]["human_filename"], "agent.log");
}