feat(media): mandatory metadata scrubbing on /feed/publish + FFmpeg sidecar
Every photo from a phone camera ships with an EXIF block that leaks:
GPS coordinates, camera model + serial, original timestamp, software
name, author/copyright fields, sometimes an embedded thumbnail that
survives cropping. For a social feed positioned as privacy-friendly
we can't trust the client alone to scrub — a compromised build,
a future plugin, or a hostile fork would simply skip the step and
leak authorship data.
So: server-side scrub is mandatory for every /feed/publish upload.
New package: media
media/scrub.go
- Scrubber type with Scrub(ctx, bytes, claimedMIME) → (clean, actualMIME)
- ScrubImage handles JPEG/PNG/GIF/WebP in-process: decodes, optionally
downscales to 1080px max-dim, re-encodes as JPEG Q=75. Stdlib
jpeg.Encode emits ZERO metadata → scrub is complete by construction.
- Sidecar client (HTTP): posts video/audio bytes to an external
FFmpeg worker at DCHAIN_MEDIA_SIDECAR_URL
- Magic-byte MIME detection: rejects uploads where declared MIME
doesn't match actual bytes (prevents a PDF dressed as image/jpeg
from bypassing the scrubber)
- ErrSidecarUnavailable: explicit error when video arrives but no
sidecar is wired; operator opts in to fallback via
--allow-unscrubbed-video (default: reject)
media/scrub_test.go
- Crafted EXIF segment with "SECRETGPS-…Canon-EOS-R5" canary —
verifies the string is gone after ScrubImage
- Downscale test (2000×1000 → 1080×540, aspect preserved)
- MIME-mismatch rejection
- Magic-byte detector sanity table
FFmpeg sidecar — new docker/media-sidecar/
Tiny Go HTTP service (~180 LOC, no non-stdlib deps) that shells out
to ffmpeg with -map_metadata -1 + -map 0:v -map 0:a? to guarantee
only video + audio streams survive (no subtitles, attached pictures,
or data channels that could carry hidden info).
Re-encode profile:
video → H.264 CRF 28 preset=fast, Opus 64k, MP4 faststart
audio → Opus 64k, Ogg container
Dockerfile: two-stage build (Go → alpine+ffmpeg), ~90 MB image, non-
root user, /healthz endpoint for compose probes.
Node reaches it via DCHAIN_MEDIA_SIDECAR_URL. Without it, video uploads
are rejected with 503 unless operator sets DCHAIN_ALLOW_UNSCRUBBED_VIDEO.
/feed/publish wiring
- cfg.Scrubber is a required dependency
- Before storing post body we call scrubber.Scrub(); attachment bytes
+ MIME are replaced with the cleaned version
- content_hash is computed over the SCRUBBED bytes — so the on-chain
CREATE_POST tx references exactly what readers will fetch
- EstimatedFeeUT uses the scrubbed size, so author's fee reflects
actual on-disk cost
- Content-type mismatches → 400; sidecar unavailable for video → 503
Flags / env vars
--feed-db / DCHAIN_FEED_DB (existing)
--feed-ttl-days / DCHAIN_FEED_TTL_DAYS (existing)
--media-sidecar-url / DCHAIN_MEDIA_SIDECAR_URL (NEW)
--allow-unscrubbed-video / DCHAIN_ALLOW_UNSCRUBBED_VIDEO (NEW; default false)
Client responsibilities (for reference — client work lands in Phase C)
Even with server-side scrub, the client should still compress aggressively
BEFORE upload, because:
- upload time is ~N× larger for unscrubbed media (mobile networks)
- the server's 256 KiB MaxPostSize is a HARD cap — oversized uploads
are rejected, not silently truncated
- the on-chain fee is size-based, so users pay for every byte the
client didn't bother to shrink
Recommended client pipeline:
images → expo-image-manipulator: resize max-dim 1080px, WebP or
JPEG quality 50-60
videos → react-native-compressor: H.264 CRF 28, 720p max, 64k audio
audio → expo-audio's default Opus 32k (already compressed)
Documented in docs/media-sidecar.md (added later with Phase C PR).
Tests
- go test ./... green across 6 packages (blockchain consensus identity
media relay vm)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
vsecoder
@@ -41,6 +41,7 @@ import (
|
||||
"go-blockchain/consensus"
|
||||
"go-blockchain/economy"
|
||||
"go-blockchain/identity"
|
||||
"go-blockchain/media"
|
||||
"go-blockchain/node"
|
||||
"go-blockchain/node/version"
|
||||
"go-blockchain/p2p"
|
||||
@@ -79,6 +80,8 @@ func main() {
|
||||
mailboxDB := flag.String("mailbox-db", envOr("DCHAIN_MAILBOX_DB", "./mailboxdata"), "BadgerDB directory for relay mailbox (env: DCHAIN_MAILBOX_DB)")
|
||||
feedDB := flag.String("feed-db", envOr("DCHAIN_FEED_DB", "./feeddata"), "BadgerDB directory for social-feed post bodies (env: DCHAIN_FEED_DB)")
|
||||
feedTTLDays := flag.Int("feed-ttl-days", int(envUint64Or("DCHAIN_FEED_TTL_DAYS", 30)), "how long feed posts are retained before auto-eviction (env: DCHAIN_FEED_TTL_DAYS)")
|
||||
mediaSidecarURL := flag.String("media-sidecar-url", envOr("DCHAIN_MEDIA_SIDECAR_URL", ""), "URL of the media scrubber sidecar (FFmpeg-based video/audio re-encoder). Empty = images only (env: DCHAIN_MEDIA_SIDECAR_URL)")
|
||||
allowUnscrubbedVideo := flag.Bool("allow-unscrubbed-video", envBoolOr("DCHAIN_ALLOW_UNSCRUBBED_VIDEO", false), "accept video uploads without server-side metadata scrubbing (only when no sidecar is configured). DANGEROUS — leaves EXIF/GPS/author tags intact (env: DCHAIN_ALLOW_UNSCRUBBED_VIDEO)")
|
||||
govContractID := flag.String("governance-contract", envOr("DCHAIN_GOVERNANCE_CONTRACT", ""), "governance contract ID for dynamic chain parameters (env: DCHAIN_GOVERNANCE_CONTRACT)")
|
||||
joinSeedURL := flag.String("join", envOr("DCHAIN_JOIN", ""), "bootstrap from a running node: comma-separated HTTP URLs (env: DCHAIN_JOIN)")
|
||||
// Observer mode: the node participates in the P2P network, applies
|
||||
@@ -938,14 +941,27 @@ func main() {
|
||||
},
|
||||
}
|
||||
|
||||
// Media scrubber — strips EXIF/GPS/author/camera metadata from every
|
||||
// uploaded image in-process, and forwards video/audio to the FFmpeg
|
||||
// sidecar when configured. Mandatory for all /feed/publish traffic.
|
||||
scrubber := media.NewScrubber(media.SidecarConfig{URL: *mediaSidecarURL})
|
||||
if *mediaSidecarURL != "" {
|
||||
log.Printf("[NODE] media sidecar: %s", *mediaSidecarURL)
|
||||
} else {
|
||||
log.Printf("[NODE] media sidecar: not configured (images scrubbed in-process; video/audio %s)",
|
||||
map[bool]string{true: "stored unscrubbed (DANGEROUS)", false: "rejected"}[*allowUnscrubbedVideo])
|
||||
}
|
||||
|
||||
feedConfig := node.FeedConfig{
|
||||
Mailbox: feedMailbox,
|
||||
HostingRelayPub: id.PubKeyHex(),
|
||||
GetPost: chain.Post,
|
||||
LikeCount: chain.LikeCount,
|
||||
HasLiked: chain.HasLiked,
|
||||
PostsByAuthor: chain.PostsByAuthor,
|
||||
Following: chain.Following,
|
||||
Mailbox: feedMailbox,
|
||||
HostingRelayPub: id.PubKeyHex(),
|
||||
Scrubber: scrubber,
|
||||
AllowUnscrubbedVideo: *allowUnscrubbedVideo,
|
||||
GetPost: chain.Post,
|
||||
LikeCount: chain.LikeCount,
|
||||
HasLiked: chain.HasLiked,
|
||||
PostsByAuthor: chain.PostsByAuthor,
|
||||
Following: chain.Following,
|
||||
}
|
||||
|
||||
go func() {
|
||||
|
||||
Reference in New Issue
Block a user