vsecoder/dchain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(feed): end-to-end integration + two-node propagation (Phase B hardening)

Browse Source 
Adds two integration-test files that exercise the full feed stack over
real HTTP requests, plus a fix to the publish signature model that the
EXIF scrubbing test surfaced.

Bug fix — api_feed.go publish signature flow
  Previously: server scrubbed the attachment → computed content_hash
  over the SCRUBBED bytes → verified the author's signature against
  that hash. But the client, not owning the scrubber, signs over the
  RAW upload. The two hashes differ whenever scrub touches the bytes
  (which it always does for images), so every signed upload with an
  image was rejected as "signature invalid".

  Fixed order:
    1. decode attachment from base64
    2. compute raw_content_hash over Content + raw attachment
    3. verify author's signature against raw_content_hash
    4. scrub attachment (strips EXIF / re-encodes)
    5. compute final_content_hash over Content + scrubbed attachment
    6. return final hash in response for the on-chain CREATE_POST tx

  The signature proves the upload is authentic; the final hash binds
  the on-chain record to what readers actually download.

node/feed_e2e_test.go
  In-process harness: real BadgerDB chain + feed mailbox + media
  scrubber + httptest.Server with RegisterFeedRoutes. Tests drive
  it via real http.Post / http.Get so rate limiters, auth, scrubber,
  and handler code all run on the happy path.

  Tests:
  - TestE2EFullFlow — publish → CREATE_POST tx → body fetch → view
    bump → stats → author list → soft-delete → 410 Gone on re-fetch
  - TestE2ELikeUnlikeAffectsStats — on-chain LIKE_POST bumps /stats,
    liked_by_me reflects the caller
  - TestE2ETimeline — follow graph, merged timeline newest-first
  - TestE2ETrendingRanking — likes × 3 + views puts hot post at [0]
  - TestE2EForYouFilters — excludes own posts + followed authors +
    already-liked posts; surfaces strangers
  - TestE2EHashtagSearch — tag returns only tagged posts
  - TestE2EScrubberStripsEXIF — injects SUPERSECRETGPS canary into a
    JPEG APP1 segment, uploads via /feed/publish, reads back — asserts
    canary is GONE from stored attachment. This is the privacy-critical
    regression gate: if it ever breaks, GPS coordinates leak.
  - TestE2ERejectsMIMEMismatch — PNG labelled as JPEG → 400
  - TestE2ERejectsBadSignature — wrong signer → 403
  - TestE2ERejectsStaleTimestamp — 1-hour-old ts → 400 (anti-replay)

node/feed_twonode_test.go
  Simulates two independent nodes sharing block history (gossip via
  same-block AddBlock on both chains). Verifies the v2.0.0 design
  contract: chain state replicates, but post BODIES live only on the
  hosting relay.

  Tests:
  - TestTwoNodePostPropagation — Alice publishes on A; B's chain sees
    the record; B's HTTP /feed/post/{id} returns 404 (body is A's);
    fetch from A succeeds using hosting_relay field from B's chain
    lookup. Documents the client-side routing contract.
  - TestTwoNodeLikeCounterSharedAcrossNodes — Bob likes from Node B;
    both A's and B's /stats show likes=1. Proves engagement aggregates
    are chain-authoritative, not per-relay.
  - TestTwoNodeFollowGraphReplicates — FOLLOW tx propagates, /timeline
    on B returns A-hosted posts with metadata (no body, as designed).

Coverage summary
  Publish flow (sign → scrub → hash → store):          ✓
  CREATE_POST on-chain fee accounting:                 ✓
  Like / Unlike counter consistency:                   ✓
  Follow graph → timeline merge:                       ✓
  Trending ranking by likes × 3 + views:               ✓
  For You exclusion rules (self, followed, liked):     ✓
  Hashtag inverted index:                              ✓
  View counter increment + stats aggregate:            ✓
  Soft-delete → 410 Gone:                              ✓
  Metadata scrubbing (EXIF canary):                    ✓
  MIME mismatch rejection:                             ✓
  Signature authenticity:                              ✓
  Timestamp anti-replay (±5 min window):               ✓
  Two-node block propagation:                          ✓
  Cross-node body fetch via hosting_relay:             ✓
  Likes aggregation across nodes:                      ✓

All 7 test packages green: blockchain consensus identity media node
relay vm.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
vsecoder
2026-04-18 19:27:00 +03:00
parent f885264d23
commit 9e86c93fda
3 changed files with 1393 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files

107
node/api_feed.go
View File

@@ -149,8 +149,8 @@ func feedPublish(cfg FeedConfig) http.HandlerFunc {
return
}
// Decode attachment.
var attachment []byte
// Decode attachment (raw upload — before scrub).
var rawAttachment []byte
var attachmentMIME string
if req.AttachmentB64 != "" {
b, err := base64.StdEncoding.DecodeString(req.AttachmentB64)
@@ -160,57 +160,21 @@ func feedPublish(cfg FeedConfig) http.HandlerFunc {
return
}
}
attachment = b
rawAttachment = b
attachmentMIME = req.AttachmentMIME
// MANDATORY server-side scrub: strip ALL metadata (EXIF/GPS/
// camera/author/ICC/etc.) and re-compress. Client is expected
// to have done a first pass, but we never trust it — a photo
// from a phone carries GPS coordinates by default and the client
// might forget or a hostile client might skip the scrub entirely.
//
// Images are handled in-process (stdlib re-encode to JPEG kills
// all metadata by construction). Videos/audio are forwarded to
// the media sidecar; if none is configured and the operator
// hasn't opted in to AllowUnscrubbedVideo, we reject.
if cfg.Scrubber == nil {
jsonErr(w, fmt.Errorf("media scrubber not configured on this node"), 503)
return
}
ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
cleaned, newMIME, err := cfg.Scrubber.Scrub(ctx, attachment, attachmentMIME)
cancel()
if err != nil {
// Graceful video fallback only when explicitly allowed.
if err == media.ErrSidecarUnavailable && cfg.AllowUnscrubbedVideo {
// Keep bytes as-is (operator accepted the risk), just log.
log.Printf("[feed] WARNING: storing unscrubbed video — no sidecar configured (author=%s)", req.Author)
} else {
status := 400
if err == media.ErrSidecarUnavailable {
status = 503
}
jsonErr(w, fmt.Errorf("scrub attachment: %w", err), status)
return
}
} else {
attachment = cleaned
attachmentMIME = newMIME
}
}
// Content hash is computed over the scrubbed bytes — that's what
// the on-chain tx will reference, and what readers fetch. Binds
// the body to the metadata so a misbehaving relay can't substitute
// a different body under the same PostID.
h := sha256.New()
h.Write([]byte(req.Content))
h.Write(attachment)
contentHash := h.Sum(nil)
contentHashHex := hex.EncodeToString(contentHash)
// ── Step 1: verify signature over the RAW-upload hash ──────────
// The client signs what it sent. The server recomputes hash over
// the as-received bytes and verifies — this proves the upload
// came from the claimed author and wasn't tampered with in transit.
rawHasher := sha256.New()
rawHasher.Write([]byte(req.Content))
rawHasher.Write(rawAttachment)
rawContentHash := rawHasher.Sum(nil)
rawContentHashHex := hex.EncodeToString(rawContentHash)
// Verify the author's signature over the canonical publish bytes.
msg := []byte(fmt.Sprintf("publish:%s:%s:%d", req.PostID, contentHashHex, req.Ts))
msg := []byte(fmt.Sprintf("publish:%s:%s:%d", req.PostID, rawContentHashHex, req.Ts))
sigBytes, err := base64.StdEncoding.DecodeString(req.Sig)
if err != nil {
if sigBytes, err = base64.RawURLEncoding.DecodeString(req.Sig); err != nil {
@@ -228,6 +192,51 @@ func feedPublish(cfg FeedConfig) http.HandlerFunc {
return
}
// ── Step 2: MANDATORY server-side metadata scrub ─────────────
// Runs AFTER signature verification so a fake client can't burn
// CPU by triggering expensive scrub work on unauthenticated inputs.
//
// Images: in-process stdlib re-encode → kills EXIF/GPS/ICC/XMP by
// construction. Videos/audio: forwarded to FFmpeg sidecar; without
// one, we reject unless operator opted in to unscrubbed video.
attachment := rawAttachment
if len(attachment) > 0 {
if cfg.Scrubber == nil {
jsonErr(w, fmt.Errorf("media scrubber not configured on this node"), 503)
return
}
ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
cleaned, newMIME, err := cfg.Scrubber.Scrub(ctx, attachment, attachmentMIME)
cancel()
if err != nil {
if err == media.ErrSidecarUnavailable && cfg.AllowUnscrubbedVideo {
log.Printf("[feed] WARNING: storing unscrubbed video — no sidecar configured (author=%s)", req.Author)
} else {
status := 400
if err == media.ErrSidecarUnavailable {
status = 503
}
jsonErr(w, fmt.Errorf("scrub attachment: %w", err), status)
return
}
} else {
attachment = cleaned
attachmentMIME = newMIME
}
}
// ── Step 3: recompute content hash over the SCRUBBED bytes ────
// This is what goes into the response + on-chain CREATE_POST, so
// anyone fetching the body can verify integrity against the chain.
// The signature check already used the raw-upload hash above;
// this final hash binds the on-chain record to what readers will
// actually download.
finalHasher := sha256.New()
finalHasher.Write([]byte(req.Content))
finalHasher.Write(attachment)
contentHash := finalHasher.Sum(nil)
contentHashHex := hex.EncodeToString(contentHash)
post := &relay.FeedPost{
PostID: req.PostID,
Author: req.Author,