EBSynth Tutorial - Transfer Your Art Style to Footage

Ebsynth can convert video clip to new style. Efficient to stylize & obscure footage
Efficient way to stylize painting/drawing style to video
Animate your footage, such as filming yourself speaking or moving, as if it were rotoscoped
Experiment to exploit the AI to unpredictably transform footage

Works with consistent, undynamic shots: static camera, figures dont move or turn wildly, elements arent leaving or entering shots
3D movement looks really cool
Things that work well:
o Gradual movement of a figure within the frame, slight turns, moving closer/further or changing size.
o Consistent moving patterns like snowfall over sky
Things that kind of work:
o Speaking/facial expressions. Kind of uncanny especially if far away and undetailed
o Panning backgrounds: New information is made procedurally, which can look okay for simple footage/abstract styles, but otherwise falls apart
Things that dont work well:
o Fast and dramatic movements well have lots of obvious artifacts, or sweep/smear what it moves over. Paint-looking and abstract styles make it better, and it snaps back on keyframes
More abstract styles make more footage viable

Its to stylize footage, not to do animation for youe.g. it doesnt create new information and doesnt know principles of motion

Endless possibilities & room to experiment/break rules.
You can edit footage before stylizing, create original footage to stylize, stylize multiple ways, stylize different layers, edit the stylized footage, stylize twice over, etc.


Process: Convert video clip to new style
1. Convert footage to PNG sequence: After Effects render -> Render Settings (custom) -> As PNG sequence
2. Select frames to use as keyframes.
If the shot is very static, you might only need 1.
Divide the shot into sections that need their own keyframe:
o New keyframe when any object, surface, etc. appears or disappears
o New keyframe when a figures angle is different
o Break up a shot if change is gradual and linear, but over a long enough time that early and later frames are very different
Choose a frame in the middle of the section, representing every element clearly
o By ?element I mean an objecta homogenously colored shape that should move and keep its integrity.
o E.g. a persons skin, their T shit, their eyes, their hair, another persons arm, a couch, a wall)
Elements in the frame should be similarly colored within, and differently colored compared to others. If theyre moving, choose a frame where theyre between the extremes.
o E.g. if part of an arm is brightened by the light against a white wall, choose a frame where the arm contrasts better against the wall.
3. Make your stylized version of the keyframe(s), matched as close to the real frame as possible. See tips section for style & original footage treatment notes
The AI reads areas of color and the borders between them, and how they moveitll transform the stylized areas and borders 1 to 1so they need to match
4. Optional: To isolate a figure from background, or if you used greenscreen, you can make an alpha mask version of footage and make a PNG sequence from that too
5. Organize project directory. Files within each folder need identical naming conventions and filetypes, and files between folders need matching frame numbers.
1. A folder of the PNG sequence
2. A folder of the stylized keyframes. Name them the same thing as the corresponding frame--the number must match exactly.
3. Optional: A folder of the mask PNG sequencethe number in the file names must match the corresponding frame exactly.
E.g. PNGs can be named ?MyComposition_0000.png, ?MyComposition_0001.png, etc. and ?Mask_0000.png
Doesnt matter if it starts at 0 or 1
6. Open EBSynth. Drag the stylized keyframes folder to ?Keyframes field & PNG sequence folder to ?Video field. (Optional) Turn Mask field on & drag the mask folder to it
Or hit ?select and dragon a file from the folder. It just needs to import the filepath and the file name template, replacing the frame numbers with #s
7. Specify frames per keyframe section
Under ?Keyframe: put the frame number of the first keyframe. E.g. if the keyframe is marked ?000030 and you type that, itll change to 30.
Under the first ?Stop: put the frame number of the earliest frame of the keyframes sectionprobably 0 or 1 for the first keyframe.
o (EBSynth will go backwards from the keyframe to this value, making a stylized version of each frame by interpreting the keyframe.)
Under the second ?Stop: put the frame number of the last frame of the keyframes section (if you only have one keyframe, the last frame of the sequence.)
o (EBSynth will go forwards from the keyframe to this value, after it creates the earlier ones.)
If you have more than one keyframe, hit the + and repeat this process for each.
8. Optional: Settings
Adjust weightse.g. increase style (e.g. to 10) to give it more influence than original. You can play with values to see what you like.
If you used a mask, raise the mask value to edge out artifacts that are supposed to be masked
Advanced settings are key:
o Mapping (5-50): More mapping keeps strokes in the same location as the keyframe. Less mapping is better for footage with more motion and new information.
o Deflicker (0.3-2.0): More deflicker compares frames more to earlier frames, for consistency. Less deflicker compares frames more to the keyframe, making each more independentlow values are good to clean artifacts and smearing. Setting to 0 makes each frame independently.
o Diversity (1,000-20,000): Smooths/blends footage to stick better to style, removing details and sharpness. Stick to low values unless it looks too sharp
o Synthesis: Output quality. Set to high, unless youre rushed or experimenting
9. Hit Synth to run it for a keyframe, or ?Run All to run it for all keyframes. It will take a long time, so plug in the computer and do something else
10. Import the newly stylized PNG sequence in the output folder(s) as footage into After Effects
Optional: Specify location for an output folder, where the newly stylized sequence will go. Default is a folder named ?Out in the project directory folder.
Import File -> Select the first image in the folder--Check ?PNG sequence under Sequence Options. (If possible, choose "footage under Import Asmy AE does this itself)
o If you import it as a new composition, itll put each PNG one frame apart using the default composition frame rate

Tips & Experiments
Treat original footage to make it easier for EBSynth to interpret. Basically, to stylize elements as one object, make their color similar to each other and different from others
Composition consistency: If the camera moves, you can zoom and correct for the position to keep the same elements in roughly the same part of the frame.
Lighting consistency: If part of a figure gets brighter/darker, it could change style and break off/mergecould be cool if intended, but if not keep the lighting the same throughout the shot
Add contrast to differentiate connected elements, especially foreground/background. But think through the effectsa dark outline that also divides a figure in half could end up splitting it
Lower contrast/hue difference of connected elements
If there are nearby regions of different elements, that are similarly light or dark, increase their difference in color (e.g. by saturating, or totally desaturating one region)
If there are regions of the same color (value & hue) that you want to stylize differently, try to edit one region to have a different color
o (E.g. masking the part of the frame and lowering bright point)

How the AI interprets a frame: It accounts for multiple things and is pretty smart
It seems to take into account BOTH the keyframe and the prior frames, to decide what element is repositioned from what other element. Thats cool and has these implications:
o Pro: A single frame can be totally glitched out, and it will scramble the style, but it wont affect the rest of the sequence
o Pro: If parts of an element start to disconnect or merge with another, it can recover and go back to normal
o Limit: If parts of an element are disconnected or merged long enough
It tries to keep each element consistent, and if it doesnt know what element a region belongs to, it tries to match its color to a nearby, similar color
o Limit: If an element leaves the frame completely and comes back in, and it was styled uniquely from other regions of the same color, it loses the unique style
It both tracks moving elements and procedurally creates elements. Details tell it to procedurally do similar things to dynamic details nearby
o E.g. bright spots on a moving pattern can jump around
It matches colors, but also notices how the same source color and styled different

Tips & style choices
You can change colors, and even vary them between like-colored regions. They should be far apart or distinct elements. (e.g. you can make two green figures blue and yellow
You can put full designs, patterns, colors over an element. Great for the Psychedelic/Ornamental style.
o Issue: If you add little baubles/designs that arent there (e.g. sparkles around a figure), theyll dance around a bit then get lost/distorted
You can vary textures, even between like-colored regions. They should be far apart or distinct elements. (e.g. noisy texture on one figure, smooth fill on another)
o Issue: A distinct texture will become messy, and if elements arent distinguished well they can take on the wrong texture
You can have complex groups of elements totally appear/disappear from blankness. E.g. in this video where a figure opens a door, and a full background appears
You can change/reshape content, like photoshopping video. E.g. turn a smile to frown, or ?white out an element. Adding new elements will get distorted. Great examples
o Good values: Mapping 20, De-Flicker 3.0, Diversity 1,000
Style: Oil-paint style is what official examples and most experiments usethey look good
Style: Ink & Fill - Joel Haver videos. Everything has an ink outline and solid color fill like a comic, plus colored highlights.
o Also uses some lighting and textures
Style: Psychedelic/Ornamental. Mapping to clear elements is important, but the more abstract the style, the better. These ones are very impressive
Style: Digital editing. Just transforming a keyframe image makes it very detailed, looking more interesting than just doing the same transformation to the footage
o Examples of combining EBSynth with neural transformations
Issue: When one part of the composition is moving and one part is static, a border between them becomes noticeable (from the expansion/contraction of overlap)
When making multiple keyframes, use multiple layers and use the same source for similar parts of the keyframes. Saves time and makes it look consistent
If elements are shaded with gradients, make each elements gradient uniqueunless you want shadows and highlights to merge.
o If two elements have similarly-colored highlights and shadows, consider differentiating the hue and value range of the elements
Issue: Some colors tend to grow and dominate, especially within gradients
Issue: If you use an exaggerated stylelike a clean graphic ?posterized lookit will get a bit messy. That means the keyframe will stick out.
Issue: Little leftover pieces of color will come off from moving figures, and hopefully disappear quick. Especially when moving parts go out of frame.

so interesting, thankyou so much

great tutorial thank you for sharing

improve little by little with this since we all do not have the facility to buy things