vsecoder/dchain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(media): mandatory metadata scrubbing on /feed/publish + FFmpeg sidecar

Browse Source 
Every photo from a phone camera ships with an EXIF block that leaks:
GPS coordinates, camera model + serial, original timestamp, software
name, author/copyright fields, sometimes an embedded thumbnail that
survives cropping. For a social feed positioned as privacy-friendly
we can't trust the client alone to scrub — a compromised build,
a future plugin, or a hostile fork would simply skip the step and
leak authorship data.

So: server-side scrub is mandatory for every /feed/publish upload.

New package: media

  media/scrub.go
    - Scrubber type with Scrub(ctx, bytes, claimedMIME) → (clean, actualMIME)
    - ScrubImage handles JPEG/PNG/GIF/WebP in-process: decodes, optionally
      downscales to 1080px max-dim, re-encodes as JPEG Q=75. Stdlib
      jpeg.Encode emits ZERO metadata → scrub is complete by construction.
    - Sidecar client (HTTP): posts video/audio bytes to an external
      FFmpeg worker at DCHAIN_MEDIA_SIDECAR_URL
    - Magic-byte MIME detection: rejects uploads where declared MIME
      doesn't match actual bytes (prevents a PDF dressed as image/jpeg
      from bypassing the scrubber)
    - ErrSidecarUnavailable: explicit error when video arrives but no
      sidecar is wired; operator opts in to fallback via
      --allow-unscrubbed-video (default: reject)

  media/scrub_test.go
    - Crafted EXIF segment with "SECRETGPS-…Canon-EOS-R5" canary —
      verifies the string is gone after ScrubImage
    - Downscale test (2000×1000 → 1080×540, aspect preserved)
    - MIME-mismatch rejection
    - Magic-byte detector sanity table

FFmpeg sidecar — new docker/media-sidecar/

  Tiny Go HTTP service (~180 LOC, no non-stdlib deps) that shells out
  to ffmpeg with -map_metadata -1 + -map 0:v -map 0:a? to guarantee
  only video + audio streams survive (no subtitles, attached pictures,
  or data channels that could carry hidden info).

  Re-encode profile:
    video → H.264 CRF 28 preset=fast, Opus 64k, MP4 faststart
    audio → Opus 64k, Ogg container

  Dockerfile: two-stage build (Go → alpine+ffmpeg), ~90 MB image, non-
  root user, /healthz endpoint for compose probes.

  Node reaches it via DCHAIN_MEDIA_SIDECAR_URL. Without it, video uploads
  are rejected with 503 unless operator sets DCHAIN_ALLOW_UNSCRUBBED_VIDEO.

/feed/publish wiring

  - cfg.Scrubber is a required dependency
  - Before storing post body we call scrubber.Scrub(); attachment bytes
    + MIME are replaced with the cleaned version
  - content_hash is computed over the SCRUBBED bytes — so the on-chain
    CREATE_POST tx references exactly what readers will fetch
  - EstimatedFeeUT uses the scrubbed size, so author's fee reflects
    actual on-disk cost
  - Content-type mismatches → 400; sidecar unavailable for video → 503

Flags / env vars

  --feed-db / DCHAIN_FEED_DB            (existing)
  --feed-ttl-days / DCHAIN_FEED_TTL_DAYS (existing)
  --media-sidecar-url / DCHAIN_MEDIA_SIDECAR_URL   (NEW)
  --allow-unscrubbed-video / DCHAIN_ALLOW_UNSCRUBBED_VIDEO (NEW; default false)

Client responsibilities (for reference — client work lands in Phase C)

  Even with server-side scrub, the client should still compress aggressively
  BEFORE upload, because:
    - upload time is ~N× larger for unscrubbed media (mobile networks)
    - the server's 256 KiB MaxPostSize is a HARD cap — oversized uploads
      are rejected, not silently truncated
    - the on-chain fee is size-based, so users pay for every byte the
      client didn't bother to shrink

  Recommended client pipeline:
    images → expo-image-manipulator: resize max-dim 1080px, WebP or
             JPEG quality 50-60
    videos → react-native-compressor: H.264 CRF 28, 720p max, 64k audio
    audio  → expo-audio's default Opus 32k (already compressed)

  Documented in docs/media-sidecar.md (added later with Phase C PR).

Tests
  - go test ./... green across 6 packages (blockchain consensus identity
    media relay vm)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
vsecoder
2026-04-18 19:15:14 +03:00
parent 126658f294
commit f885264d23
8 changed files with 830 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files

30
cmd/node/main.go
View File

@@ -41,6 +41,7 @@ import (
"go-blockchain/consensus"
"go-blockchain/economy"
"go-blockchain/identity"
"go-blockchain/media"
"go-blockchain/node"
"go-blockchain/node/version"
"go-blockchain/p2p"
@@ -79,6 +80,8 @@ func main() {
mailboxDB := flag.String("mailbox-db", envOr("DCHAIN_MAILBOX_DB", "./mailboxdata"), "BadgerDB directory for relay mailbox (env: DCHAIN_MAILBOX_DB)")
feedDB := flag.String("feed-db", envOr("DCHAIN_FEED_DB", "./feeddata"), "BadgerDB directory for social-feed post bodies (env: DCHAIN_FEED_DB)")
feedTTLDays := flag.Int("feed-ttl-days", int(envUint64Or("DCHAIN_FEED_TTL_DAYS", 30)), "how long feed posts are retained before auto-eviction (env: DCHAIN_FEED_TTL_DAYS)")
mediaSidecarURL := flag.String("media-sidecar-url", envOr("DCHAIN_MEDIA_SIDECAR_URL", ""), "URL of the media scrubber sidecar (FFmpeg-based video/audio re-encoder). Empty = images only (env: DCHAIN_MEDIA_SIDECAR_URL)")
allowUnscrubbedVideo := flag.Bool("allow-unscrubbed-video", envBoolOr("DCHAIN_ALLOW_UNSCRUBBED_VIDEO", false), "accept video uploads without server-side metadata scrubbing (only when no sidecar is configured). DANGEROUS — leaves EXIF/GPS/author tags intact (env: DCHAIN_ALLOW_UNSCRUBBED_VIDEO)")
govContractID := flag.String("governance-contract", envOr("DCHAIN_GOVERNANCE_CONTRACT", ""), "governance contract ID for dynamic chain parameters (env: DCHAIN_GOVERNANCE_CONTRACT)")
joinSeedURL := flag.String("join", envOr("DCHAIN_JOIN", ""), "bootstrap from a running node: comma-separated HTTP URLs (env: DCHAIN_JOIN)")
// Observer mode: the node participates in the P2P network, applies
@@ -938,14 +941,27 @@ func main() {
},
}
// Media scrubber — strips EXIF/GPS/author/camera metadata from every
// uploaded image in-process, and forwards video/audio to the FFmpeg
// sidecar when configured. Mandatory for all /feed/publish traffic.
scrubber := media.NewScrubber(media.SidecarConfig{URL: *mediaSidecarURL})
if *mediaSidecarURL != "" {
log.Printf("[NODE] media sidecar: %s", *mediaSidecarURL)
} else {
log.Printf("[NODE] media sidecar: not configured (images scrubbed in-process; video/audio %s)",
map[bool]string{true: "stored unscrubbed (DANGEROUS)", false: "rejected"}[*allowUnscrubbedVideo])
}
feedConfig := node.FeedConfig{
Mailbox: feedMailbox,
HostingRelayPub: id.PubKeyHex(),
GetPost: chain.Post,
LikeCount: chain.LikeCount,
HasLiked: chain.HasLiked,
PostsByAuthor: chain.PostsByAuthor,
Following: chain.Following,
Mailbox: feedMailbox,
HostingRelayPub: id.PubKeyHex(),
Scrubber: scrubber,
AllowUnscrubbedVideo: *allowUnscrubbedVideo,
GetPost: chain.Post,
LikeCount: chain.LikeCount,
HasLiked: chain.HasLiked,
PostsByAuthor: chain.PostsByAuthor,
Following: chain.Following,
}
go func() {