Motion transfer on self-hosted H3: a character performs a driving clip's dance or action, or replaces the person in that video
Kling-style motion control on H3 from one driving clip and one character still: Viggle-Animate (H3 ref2va finetune) character replacement from a repainted reference frame, with windowed chaining for long takes; H3 Fun ControlNet Union 2.0 driven by SDPose skeletons; and stock H3 Ref2VA with the clip as <Video 1> attribute_transfer. Includes clip conforming, reference-frame extraction, repaint, keypoint extraction, side-by-side sheets and a bone-direction pose-adherence score, plus the pinned runtime layer and weights. Generation profiles h3-motion-viggle, h3-motion-pose and h3-motion-prompt bind an owned driving clip and still through generation.prepare, which measures the clip (frames, grid, canvas, cuts) and refuses rather than conforms; self-hosted character (Krea2 Turbo) and repaint (Qwen-Image-Edit 2511) stills replace the Gemini steps, which stay as partner options.
Discover · Wavemaker catalog
When to use it
- Kling-style motion control on H3 from one driving clip and one character still: Viggle-Animate (H3 ref2va finetune) character replacement from a repainted reference frame, with windowed chaining for long takes; H3 Fun ControlNet Union 2.0 driven by SDPose skeletons; and stock H3 Ref2VA with the clip as <Video 1> attribute_transfer. Includes clip conforming, reference-frame extraction, repaint, keypoint extraction, side-by-side sheets and a bone-direction pose-adherence score, plus the pinned runtime layer and weights. Generation profiles h3-motion-viggle, h3-motion-pose and h3-motion-prompt bind an owned driving clip and still through generation.prepare, which measures the clip (frames, grid, canvas, cuts) and refuses rather than conforms; self-hosted character (Krea2 Turbo) and repaint (Qwen-Image-Edit 2511) stills replace the Gemini steps, which stay as partner options.
What you get
- A creative method with its instructions and supporting source files.
Setup and requirements
- Follow the included guide to bind your inputs and select the appropriate runtime.
Things to know
- Measured on four driving clips (5.17 s and 9.42 s street dance, two 4.46 s Comfy demo clips, one stylised CGI fight), three characters and mostly one seed per variant on B200 sandboxes (evidence READMEs). Multiple people, re-entry and seed variance are untested.
- Needs runtime/profiles/motion-transfer-20260929.json (sandboxes) or the prepared fleet variant catalog-b200-motion-transfer-20260929 (runtime/profiles/motion-transfer-viggle-fleet-20260929.json, not yet published) plus the seven weights pinned in services/comfy-runner/motion_transfer_weights.json. The variant preparation proposes a 540 s job bound (base 300 s); until it is published with it, the prompt route beyond about 124 frames and the pose route at 226 frames may not fit. Profiles cap viggle and prompt at 124 frames and pose at 226.
- Viggle-Animate needs a reference frame showing the performer's whole body and fails across cuts and whip zooms; it ghosted limbs on one clip regardless of reference or seed, cause unknown. Review every output and fall back to the pose route.
- The pose route is the most robust (including cuts and back turns) but arms followed the skeleton in 54-75% of bones on dance clips and it did not draw props; strength 2.0 degraded the image.
- The prompt route shifts timing by several frames, reordered a cut and takes about 10 minutes for a 9.4 s clip on B200. Fast stylised weapon choreography did not qualify on any route.
- The Gemini character and repaint steps need an egress runtime with credentials; the self-hosted character (Krea2 Turbo) and repaint (Qwen-Image-Edit 2511 Lightning) stills are unmeasured for this task: compare the repaint with the reference frame before every Viggle run. The pose score measures 2D body pose only; frames and full-speed viewing remain required. Keypoint extraction is not a fleet route (SavePoseKpsAsJsonFile is not on the fleet image); review.py stays local. The Viggle pack installs from its Comfy Registry archive by sha256 (its GitHub repository is no longer public).
Source licenses and model terms continue to apply. Check runtime availability before generation.
Download source · Learn how to reuse resources