Merge / matte operation / channel / key - question

[KMKaine]

Hi there,

I'm working on a composite for a shot for a short film I'm making. It's a broken (burning) tank on a deserted street.

I've come quite far already, but I'm getting stuck (and very very confused) with the next step.

I have attached a still so you know what I'm talking about. I'm now in the process of compositing a fire and smoke on the tank, and it should run behind the front tree. So what I intend to do is key on the tree and invert that alpha so that I have a tree alpha / matte. That key is quite bad (because I'm unfortunatly working on XDcamEX 35mbit HD). Nothing we can do to change that, so... I pulled a key, and I want to put that in front of the fire and smoke (wich is in front of the same background). There are some dirty edges on the key (something else I noticed, but that's point B), and I wanted to overcome that by lightwrapping the fire and smoke over the new keyed / extracted front tree. But I can't nail it... :S

I'm running into very confusing stuff with the layers and channels. So I have a couple of questions:

1. Can I (and if so, how?!) use a rotopaint node or other mask source such as the keyer output to mask out just 1 source. Not as is ?necessary? in the merge node 2 sources, but use just one source, and output the result (the matted result) aswell as the alpha channel to the next step?
I got it to work on using 2 inputs (A & B) and then using a matte to merge them, but then I won't have an alpha channel output.

1b. Or perhaps I'm going about this the wrong way, so If there is another (right way) I like to know that one too ;-)

2. If you check out the second attached picture, you see my picture all bluey. How does that come? I'm using a constant in an over mode over a piece of video? Is the over mode not the default (hard)overlay mode? Should I be using a different mode?!

3. Are there already in depth tutorials available on the merge node, or perhapse better said: the image channels in the mergenode sense of the word? What I mean is: I have seen some in depth tutorials on using shuffle and so on to do some channel magic, but what I'm really curious about is how the channel controls work exactly in the merge node. Because I think i'm doing something wrong wich is making me not understand or getting the right result.

4. In keylight you can do a soften of the matte (wich is very handy in case of my tree key thingy) but it seems to soften on the initial border, not the one obtained after for example dilating, rendering it sort of useless in my case. What I want is to get a matte of the leafs, with a little softening on the edge so that it merges with the background.

I hope someone can give me a push in the right direction, because it's becoming a little frustrating :-P

PS: Iv'e watched this course among some others
http://www.digitaltutors.com/11/training.php?cid=203&vid=18929
Although nice, this isn't helping me any further with my current issue ;-)

[chrisg]

For your composite, it looks like Merge9 should be screen, and Merge 7 should be over.

This is close to what your order of merges should be.

Fire + Smoke Combined how you have now.

FireSmoke composited over background with inverted tree alpha cutting it out.

lightwrap composited on top of background.

1. Yes. You can put the output of the RotoPaint node into a different channel like mask.a, then premult your rgb by the mask.a channel. I think this is what you're looking for, if not could you give an example?


2. This is happening because your A pipe constant is not outputting an alpha channel, making the over into an add operation. To fix this, go to your constant and change channels to rgba and change a to something like 1. You should now see all blue.

3. You might try looking into the Introduction to Channels in NUKE (http://www.digitaltutors.com/11/training.php?pid=192) and Multi-channel Compositing in NUKE (http://www.digitaltutors.com/11/training.php?pid=194) courses.

4. Screen softness should blur the screen after the dilate. Screen PreBlur blurs it before the dilate. You can always just blur the alpha channel after you extract the alpha.

Hope this helps!

[KMKaine]

Hi Chris!

Thanks for your quick reply! When I get home tonight I will have another go at it with your input. Although I don't really get what you mean at point 1.

1. I have attached a new example of what I mean. For example, I want to use the soccer source file, and mask it out, so that just the inside of the rotopaint shape is visible. That I want to composite over the red background. But I want the cut-out to happen on another node, because I want to use the cut out of the soccer players also for the lightwrap for example. So doing that in a single merge node is not handy right? I got it to work on a single merge node with 2 inputs and using the roto as a mask, but when using only one input it seems to behave different. I now have that working (in the attached example), where there is only the inside of the roto is visible and where there is an actual alpha present in the same shape. But pluging that result in to the second merge node to add the cut out to the red background I get some weird results again. I have checked that alpha setting in the in the constant, but I think it is not necessary because it is in the background right?

2. Is working like a charm now. How come that you have to work this way? Why does it turn into an Add when missing an alpha channel? Should it not asume there is no alpha and thus making it all black?

3. I have watched the Multi Channel course, wich got me this far ;-) But it didn't really help me got unconfused. I guess i'm in a different mindset about channels and layers than is necessary in Nuke since I have only worked in After Effects and AvidDS before. I'll go and have a look at the Intro to Channels.

4. Thanks for the pointer!

Thanks again!

PS: I have an attachment problem: I have about 320 kb attached in 2 files, but it says i'm nearly on 512 kb... guess there is a ghost file or 2 somewhere in the system? I have removed the former 2 example pics from the post above to be able to attach a new one... Perhaps it's an idea to upgrade the 512 limit to 1mb or 1.5 mb in the future, that would give us just a little bit more flexibility for attaching jpgs.

[chrisg]

Hi KMKaine,

1. Instead of a Merge, you want to Output the RotoPaint into a new channel. Think of channels as just places to store your images while your working with them. So, we can store our RotoPaint image into a channel called "CustomAlpha" and then use that channel wherever we want later in the chain. However, channels are only kept or passed along in the B pipe, so we would actually want to roto the background. Here, copy and paste the following code into Nuke's Node Graph.


set cut_paste_input [stack 0]
version 6.2 v4
ColorWheel {
inputs 0
gamma 0.45
rotate 180
name ColorWheel1
selected true
xpos 731
ypos -277
}
Shuffle {
alpha white
name Shuffle_MakeAlphaSolid
selected true
xpos 731
ypos -193
}
Constant {
inputs 0
color {1 0 0 0.5}
name Constant2
selected true
xpos 846
ypos -177
}
add_layer {customMask customMask.mask1 customMask.mask2 customMask.mask3}
RotoPaint {
output customMask
curves {AnimTree: "" {
Version: 1.2
Flag: 0
RootNode: 1
Node: {
NodeName: "Root" {
Flag: 512
NodeType: 1
Transform: 0 0 S 0 0 S 0 0 S 0 0 S 0 1 S 0 1 S 0 0 S 0 480 S 0 270
NumOfAttributes: 10
"vis" S 0 1 "opc" S 0 1 "mbo" S 0 1 "mb" S 0 1 "mbs" S 0 0.5 "fo" S 0 1 "fx" S 0 0 "fy" S 0 0 "ff" S 0 1 "ft" S 0 0
}
NumOfChildren: 1
Node: {
NodeName: "Bezier1" {
Flag: 576
NodeType: 3
CurveGroup: "" {
Transform: 0 0 S 1 1 0 S 1 1 0 S 1 1 0 S 1 1 1 S 1 1 1 S 1 1 0 S 1 1 713.8 S 1 1 366.4
Flag: 0
NumOfCubicCurves: 2
CubicCurve: "" {
Type: 0 Flag: 8192 Dim: 2
NumOfPoints: 15
0 S 1 1 68 S 1 1 18 0 0 S 1 1 677 S 1 1 470 0 0 S 1 1 -142.948 S 1 1 -37.8392 0 0 S 1 1 5 S 1 1 58 0 0 S 1 1 530 S 1 1 384 0 0 S 1 1 -5 S 1 1 -58 0 0 S 1 1 -116 S 1 1 -2 0 0 S 1 1 586 S 1 1 221 0 0 S 1 1 116 S 1 1 2 0 0 S 1 1 -15 S 1 1 -68 0 0 S 1 1 855 S 1 1 292 0 0 S 1 1 15 S 1 1 68 0 0 S 1 1 48 S 1 1 -30 0 0 S 1 1 850 S 1 1 461 0 0 S 1 1 -48 S 1 1 30 0
}
CubicCurve: "" {
Type: 0 Flag: 8192 Dim: 2
NumOfPoints: 15
0 S 1 1 68 S 1 1 18 0 0 S 1 1 0 S 1 1 0 0 0 S 1 1 -142.948 S 1 1 -37.8392 0 0 S 1 1 5 S 1 1 58 0 0 S 1 1 0 S 1 1 0 0 0 S 1 1 -5 S 1 1 -58 0 0 S 1 1 -116 S 1 1 -2 0 0 S 1 1 0 S 1 1 0 0 0 S 1 1 116 S 1 1 2 0 0 S 1 1 -15 S 1 1 -68 0 0 S 1 1 0 S 1 1 0 0 0 S 1 1 15 S 1 1 68 0 0 S 1 1 48 S 1 1 -30 0 0 S 1 1 0 S 1 1 0 0 0 S 1 1 -48 S 1 1 30 0
}
NumOfAttributes: 43
"vis" S 0 1 "r" S 0 1 "g" S 0 1 "b" S 0 1 "a" S 0 1 "ro" S 0 0 "go" S 0 0 "bo" S 0 0 "ao" S 0 0 "opc" S 0 1 "bm" S 0 0 "inv" S 0 0 "mbo" S 0 0 "mb" S 0 1 "mbs" S 0 0.5 "mbsot" S 0 0 "mbso" S 0 0 "fo" S 0 1 "fx" S 0 0 "fy" S 0 0 "ff" S 0 1 "ft" S 0 0 "src" S 0 0 "stx" S 0 0 "sty" S 0 0 "str" S 0 0 "sr" S 0 0 "ssx" S 0 1 "ssy" S 0 1 "ss" S 0 0 "spx" S 0 480 "spy" S 0 270 "stot" S 0 0 "sto" S 0 0 "sv" S 0 0 "sf" S 0 1 "sb" S 0 1 "nv" S 0 1 "view1" S 0 1 "ltn" S 0 1 "ltm" S 0 1 "ltt" S 0 0 "tt" S 0 4
}
}
NumOfChildren: 0
}
}
}
}
toolbox {selectAll {
{ selectAll ssx 1 ssy 1 sf 1 }
{ createBezier ssx 1 ssy 1 sf 1 sb 1 tt 4 }
{ createBSpline ssx 1 ssy 1 sf 1 sb 1 }
{ createEllipse ssx 1 ssy 1 sf 1 sb 1 }
{ createRectangle ssx 1 ssy 1 sf 1 sb 1 }
{ brush ssx 1 ssy 1 sf 1 sb 1 }
{ eraser src 2 ssx 1 ssy 1 sf 1 sb 1 }
{ clone src 1 ssx 1 ssy 1 sf 1 sb 1 }
{ reveal src 3 ssx 1 ssy 1 sf 1 sb 1 }
{ dodge src 1 ssx 1 ssy 1 sf 1 sb 1 }
{ burn src 1 ssx 1 ssy 1 sf 1 sb 1 }
{ blur src 1 ssx 1 ssy 1 sf 1 sb 1 }
{ sharpen src 1 ssx 1 ssy 1 sf 1 sb 1 }
{ smear src 1 ssx 1 ssy 1 sf 1 sb 1 }
} }
toolbar_brush_hardness 0.200000003
toolbar_lifetime_type all
toolbar_source_transform_scale {1 1}
toolbar_source_transform_center {320 240}
colorOverlay 0
lifetime_type "all frames"
motionblur_shutter_offset_type centred
source_black_outside true
name RotoPaint1
selected true
xpos 846
ypos -82
}
Merge2 {
inputs 2
maskChannelInput customMask.mask1
name Merge2
selected true
xpos 731
ypos -23
}


This is one way you can set-up your RotoPaint to output a custom channel and use it in other nodes down the pipeline. Switch your viewer from rgba to CustomMask to see the 'alpha' we are carrying along with us.

In the example you provided, we should not be using the Merge node. With the Merge node, NUKE tries to combine the channels if there is data or if there is not. So you can see in your Merge 1, that the center of your roto shape is turning black, not what we wanted. We just wanted to copy the alpha channel in over our current alpha channel. So, let's use the copy node instead, which you can get with the 'k' hotkey. It, by default, is set up to copy the alpha channel from the A pipe into the B pipe. So if you switch the Merge with a Copy, half of our issues are fixed. We still get the front of our image added to our background. This leads us into your question 2.

2. The over mode turns into an Add when missing an alpha channel because of how the over math is working. All compositing is math, but it's the easy math. So let's take a look at the Over math, which you can see if you hover your mouse over the 'Over' box. Over is defined as A + B(1-a). Capital A and B are the color channels (RGB) of our A and B pipes. The lower case letters, a and b (not in this example) are the alpha channels of the A and B pipes. So to find the final color, you would take the color of the A pipe, then add it to the color of the B pipe multiplied by one minus the A alpha. One minus an alpha is essentially inverting it, so white becomes black, and black becomes white. So when A has no alpha (a is black) the over operation will turn it white, making the operation A + B (1-0) or A + B (1) or A + B, which is the Plus operation. Now we are getting the same issue with our edited composition, which I have here:


set cut_paste_input [stack 0]
version 6.2 v4
push $cut_paste_input
RotoPaint {
output alpha
curves {AnimTree: "" {
Version: 1.2
Flag: 0
RootNode: 1
Node: {
NodeName: "Root" {
Flag: 512
NodeType: 1
Transform: 0 0 S 0 0 S 0 0 S 0 0 S 0 1 S 0 1 S 0 0 S 0 480 S 0 270
NumOfAttributes: 10
"vis" S 0 1 "opc" S 0 1 "mbo" S 0 1 "mb" S 0 1 "mbs" S 0 0.5 "fo" S 0 1 "fx" S 0 0 "fy" S 0 0 "ff" S 0 1 "ft" S 0 0
}
NumOfChildren: 1
Node: {
NodeName: "Bezier1" {
Flag: 576
NodeType: 3
CurveGroup: "" {
Transform: 0 0 L 0 4096 0.5 2 0 0 "" 0 1 "" 1 0 0 1 0 1 256 0 L 0 4096 0.5 2 0 0 "" 0 1 "" 1 0 0 1 0 1 256 0 L 0 4096 0.5 2 0 0 "" 0 1 "" 1 0 0 1 0 1 256 0 L 0 4096 0.5 2 0 0 "" 0 1 "" 1 1 0 1 0 1 256 0 L 0 4096 0.5 2 0 0 "" 0 1 "" 1 1 0 1 0 1 256 0 L 0 4096 0.5 2 0 0 "" 0 1 "" 1 0 0 1 0 1 256 0 L 0 4096 0.5 2 0 0 "" 0 1 "" 1 569 0 1 0 1 256 0 L 0 4096 0.5 2 0 0 "" 0 1 "" 1 321.667 0 1 0 1 256 0
Flag: 0
NumOfCubicCurves: 2
CubicCurve: "" {
Type: 0 Flag: 8192 Dim: 2
NumOfPoints: 18
0 S 1 1 78 S 1 1 16 0 0 S 1 1 784 S 1 1 492 0 0 S 1 1 -78 S 1 1 -16 0 0 S 1 1 18 S 1 1 96 0 0 S 1 1 618 S 1 1 372 0 0 S 1 1 -18 S 1 1 -96 0 0 S 1 1 70 S 1 1 14 0 0 S 1 1 314 S 1 1 410 0 0 S 1 1 -70 S 1 1 -14 0 0 S 1 1 -72 S 1 1 38 0 0 S 1 1 272 S 1 1 160 0 0 S 1 1 72 S 1 1 -38 0 0 S 1 1 -86 S 1 1 -78 0 0 S 1 1 612 S 1 1 108 0 0 S 1 1 86 S 1 1 78 0 0 S 1 1 -70 S 1 1 -40 0 0 S 1 1 814 S 1 1 388 0 0 S 1 1 70 S 1 1 40 0
}
CubicCurve: "" {
Type: 0 Flag: 8192 Dim: 2
NumOfPoints: 18
0 S 1 1 78 S 1 1 16 0 0 S 1 1 0 S 1 1 0 0 0 S 1 1 -78 S 1 1 -16 0 0 S 1 1 18 S 1 1 96 0 0 S 1 1 0 S 1 1 0 0 0 S 1 1 -18 S 1 1 -96 0 0 S 1 1 70 S 1 1 14 0 0 S 1 1 0 S 1 1 0 0 0 S 1 1 -70 S 1 1 -14 0 0 S 1 1 -72 S 1 1 38 0 0 S 1 1 0 S 1 1 0 0 0 S 1 1 72 S 1 1 -38 0 0 S 1 1 -86 S 1 1 -78 0 0 S 1 1 0 S 1 1 0 0 0 S 1 1 86 S 1 1 78 0 0 S 1 1 -70 S 1 1 -40 0 0 S 1 1 0 S 1 1 0 0 0 S 1 1 70 S 1 1 40 0
}
NumOfAttributes: 43
"vis" S 0 1 "r" S 0 1 "g" S 0 1 "b" S 0 1 "a" S 0 1 "ro" S 0 0 "go" S 0 0 "bo" S 0 0 "ao" S 0 0 "opc" S 0 1 "bm" S 0 0 "inv" S 0 0 "mbo" S 0 0 "mb" S 0 1 "mbs" S 0 0.5 "mbsot" S 0 0 "mbso" S 0 0 "fo" S 0 1 "fx" S 0 0 "fy" S 0 0 "ff" S 0 1 "ft" S 0 0 "src" S 0 0 "stx" S 0 0 "sty" S 0 0 "str" S 0 0 "sr" S 0 0 "ssx" S 0 1 "ssy" S 0 1 "ss" S 0 0 "spx" S 0 480 "spy" S 0 270 "stot" S 0 0 "sto" S 0 0 "sv" S 0 0 "sf" S 0 1 "sb" S 0 1 "nv" S 0 1 "view1" S 0 1 "ltn" S 0 1 "ltm" S 0 1 "ltt" S 0 0 "tt" S 0 4
}
}
NumOfChildren: 0
}
}
}
}
toolbox {selectAll {
{ selectAll ssx 1 ssy 1 sf 1 }
{ createBezier ssx 1 ssy 1 sf 1 sb 1 tt 4 }
{ createBSpline ssx 1 ssy 1 sf 1 sb 1 }
{ createEllipse ssx 1 ssy 1 sf 1 sb 1 }
{ createRectangle ssx 1 ssy 1 sf 1 sb 1 }
{ brush ssx 1 ssy 1 sf 1 sb 1 }
{ eraser src 2 ssx 1 ssy 1 sf 1 sb 1 }
{ clone src 1 ssx 1 ssy 1 sf 1 sb 1 }
{ reveal src 3 ssx 1 ssy 1 sf 1 sb 1 }
{ dodge src 1 ssx 1 ssy 1 sf 1 sb 1 }
{ burn src 1 ssx 1 ssy 1 sf 1 sb 1 }
{ blur src 1 ssx 1 ssy 1 sf 1 sb 1 }
{ sharpen src 1 ssx 1 ssy 1 sf 1 sb 1 }
{ smear src 1 ssx 1 ssy 1 sf 1 sb 1 }
} }
toolbar_brush_hardness 0.200000003
toolbar_lifetime_type all
toolbar_source_transform_scale {1 1}
toolbar_source_transform_center {320 240}
colorOverlay 0
lifetime_type "all frames"
motionblur_shutter_offset_type centred
source_black_outside true
name RotoPaint3
selected true
xpos 1216
ypos -129
}
ColorBars {
inputs 0
name ColorBars3
selected true
xpos 1348
ypos -273
}
Copy {
inputs 2
from0 rgba.alpha
to0 rgba.alpha
name Copy1
selected true
xpos 1348
ypos -129
}
Constant {
inputs 0
color {1 0 0 1}
name Constant3
selected true
xpos 1460
ypos -142
}
Merge2 {
inputs 2
name Merge4
selected true
xpos 1348
ypos -29
}


So, how do we fix this? Well, the over operation never uses the A alpha (a) on the A pipe, so A's color are never changed. We need to pre-multiply A by it's alpha. We can do that with the Premult node, or with the Premultiplied Over operation (Matte). Matte's operation is defined as Aa + B(1-a). So in the black areas of our A pipe, a matte multiplies those colors to black, while still overing the B pipe correctly.

Whew, that was a lot of text. Let me know if you have any questions about anything I've said.

PS: I've cleaned up your other uploads, so you should have more space now.

[KMKaine]

Hi Chris!

Thanks for your thorough reply. It's becoming quite clear to me now. Some more practice and I'll guess I get the hang of it ;-) I have it properly comped now, so that's great. I just found out, that for he lightwrap there is no way of using all colors? It uses only the bright set of colors I came to realize... So that trick won't work I guess...

Still battleing the key tho, but I guess it's just crap: Keyeing a tree on a IMX35HD piece of footage is not really working... a lot, and I mean a lot of jittering in the key :(

Thanks again!