Pixels Are Vectors

How linear algebra, the maths of vectors and matrices, sits under the ColorMatrix node, the world position pass and deep compositing in Foundry Nuke, and why knowing it makes you a faster, braver compositor.

Compositing · Nuke · 13 min

Open a Nuke script from any feature film and you will find hundreds of nodes: grades, merges, masks, relights, deep merges. Under almost every one of them sits one small idea from linear algebra. A list of numbers is a **vector**, and a grid of numbers, a **matrix**, can move every vector in a picture at once. A pixel's colour is a vector. A pixel in a world position pass is a vector too, but this time it is a point in the 3D scene. A deep pixel is a whole list of them. This post shows how compositors use that one idea: in the ColorMatrix node to change saturation, mix channels and convert colour spaces; on the world position pass to build 3D masks, add light and move between camera and world space; and in deep compositing, where every pixel carries many samples and the files grow huge. Every figure is drawn live from one small 3D room rendered in your browser, so the numbers you see are the numbers being used.

A pixel is a list of numbers

In Nuke a pixel of an RGB image is three numbers: red, green and blue. Linear algebra calls a list of numbers like that a vector, and you can draw it as an arrow. Start at the black corner of a cube, where all three numbers are zero, and travel along the red, green and blue edges by the amount each channel says. White is the far corner, (1, 1, 1). A dark orange is a short arrow leaning towards red. Nuke works in 32-bit floating point and linear light, so the arrow is allowed to leave the cube: a bright highlight can be (4.2, 3.8, 3.1), and a value can even dip below zero. Thinking of colours as arrows is the step that unlocks everything else, because there is a whole branch of maths about moving arrows around.

A channel does not have to hold colour. A 3D renderer can write other vectors into the red, green and blue slots of extra passes, often called AOVs (arbitrary output variables). The most useful one for compositors is the **world position pass**, usually called P. Each pixel stores the x, y and z coordinates in the 3D scene where the camera's ray hit a surface. A normals pass, N, stores which way that surface faces. A depth pass stores how far away it was. If you have ever wondered how a compositor can grade just the wall without drawing a single roto shape, P is the answer: every pixel of the back wall in our room carries the wall's own position in the world. In Nuke these passes arrive as extra layers in a multi-channel EXR, and Foundry notes that in NukeX they can also be generated from footage with the DepthGenerator node.

A 3×3 grid that moves every colour

Foundry describes the ColorMatrix node simply: it multiplies the RGB values of every pixel by a 3×3 matrix, and "the output values of each primary color channel are calculated based on the input values from all three channels." Read the matrix one row at a time. The top row says how much of the old red, green and blue goes into the new red. The second row builds the new green and the third builds the new blue. That is all matrix multiplication is: three weighted sums, done on every pixel at once. The node also has the usual mask input, a mix slider, and an option to unpremultiply by alpha first, so you can use it like any other colour node in a script.

Because every colour is an arrow, a matrix does not really change colours one at a time. It bends the whole colour cube at once. The **identity matrix**, with ones down the diagonal and zeros everywhere else, leaves the cube alone. Numbers on the diagonal stretch the cube along one axis, which is exactly what a gain on a single channel does. Numbers off the diagonal lean one axis towards another, which mixes channels. Straight lines stay straight and black stays at black, and that rule is what makes it a *linear* operation. It is also why a matrix can never add an offset or a curve: for lift, gamma or a soft roll-off you still need a Grade or a ColorLookup.

The matrix compositors reach for most is saturation. First take the luminance of a pixel, a weighted sum using the Rec.709 weights: 0.2126 for red, 0.7152 for green and 0.0722 for blue. Then blend between that grey and the original colour. Written as a matrix, every row starts as the luminance weights times (1 − s), and s is added on the diagonal. At s = 1 you get the identity and nothing changes. At s = 0 all three rows are identical, so every channel gets the same value and the image turns grey. Above 1 the colours are pushed further apart. Because the three weights add up to exactly one, a neutral grey pixel stays the same grey whatever s is, which is why a saturation matrix never shifts your grey card.

Channel mixing is where the numbers off the diagonal earn their keep. Swapping red and blue is a matrix with ones in two opposite corners. A black and white image made from the red channel alone, like a red filter on black and white film, is a matrix whose three rows are all (1, 0, 0). Sending red into green, green into blue and blue into red rotates every hue by a third of the colour wheel. Compositors use mixes like these to fix footage that arrives with its channels in the wrong order, to pull a cleaner key from whichever channel has the most contrast, or to build a luminance matte from a custom blend of channels.

The job ColorMatrix does most often in a real pipeline is a **colour space conversion**. Two linear colour spaces that differ only in their primaries, such as linear Rec.709 and ACEScg, are linked by exactly one 3×3 matrix. ACEScg has much wider primaries, so the same orange is described by different, less extreme numbers. Push Rec.709 values through the conversion matrix and you get ACEScg values that mean the same colour. Forget the conversion and show those numbers as if they were still Rec.709, and the picture looks washed out, a classic mistake when a plate and a CG render meet in different spaces. Nuke's OCIO nodes normally handle this for you, but under the hood a change of primaries in linear light is this one matrix.

ColorMatrix has an **invert** checkbox. Foundry's reference says it applies the inverse of the matrix instead of the matrix itself, and the inverse is the one matrix that exactly undoes the first. Converting to ACEScg and back is the matrix followed by its inverse, and because Nuke keeps 32-bit floats, negative values included, the round trip returns the original numbers. Not every matrix can be undone, though. A single number called the **determinant** measures how much a matrix squashes the cube. If it is zero, the cube has been flattened and information is gone for good. A full desaturation is the classic case: once every pixel is grey, no matrix in the world can tell you what colour it used to be.

Masks and light that live in 3D

Back to the position pass. Point at the back wall and the z value stays the same everywhere on it, −4, because every pixel of that wall lies on the same plane in the world. Point at the floor and y is always zero. Each pixel is a vector, an address in the 3D scene, and that address does not depend on where the camera stands. This is what makes P so powerful for a compositor. A mask built from P is attached to the world rather than to the screen, so it sticks to the wall when the camera moves, and you never have to track it or animate a roto shape.

The simplest 3D mask is a sphere. Pick a point in the world, call it c. For every pixel, take the vector P − c and measure its length: that is the distance from the surface in that pixel to your point. Where the distance is smaller than a radius the mask is white, fading to black over a soft edge. Artists build this with a small Expression node or a shared gizmo, often called a P_Matte, and Foundry's PositionToPoints node, which "generates a 3D point cloud from an xyz image, such as a position pass", lets you see the cloud in the 3D viewer so you can place the sphere where you want it.

A sphere looks the same from every side, but a box does not. The teal crate in our room is turned 30 degrees, so a box mask built straight from P would cut across it. The fix is to move the world so the crate sits square. Subtract the crate's centre from P with an Add node, then rotate what is left with a rotation matrix. A rotation is just another 3×3 matrix made of cosines and sines, so a plain ColorMatrix applied to the P channels does it, even though the node was designed for colour. After that, one simple test, are x, y and z each between −0.5 and 0.5, gives a box mask that follows the crate's sides exactly. ColorMatrix only holds a 3×3, so it can rotate and scale but not move anything; the move always comes from an offset.

With P and N together you can add light to a render without going back to 3D. For every pixel, the vector from the surface to a lamp is L − P. Divide it by its length to get a direction, take the **dot product** with the normal N, and you know how squarely that surface faces the light. Divide by the squared distance and you have the light's natural falloff. Foundry describes the Relight node as taking "a 2D image containing normal and point position passes" and letting you light it with 3D lights, a camera and a material, and this arithmetic is the heart of it. It cannot cast new shadows, because a pass only knows the front surface, but for a quick extra kick, a moving practical light or a colour accent it is fast and interactive.

A point can be described from two places: from the world's origin, or from the camera. Moving between them needs a **4×4 matrix**. The extra row and column let one matrix hold a rotation and a move together, which a 3×3 cannot. The trick is to write every point as (x, y, z, 1), known as homogeneous coordinates. A camera's transform is exactly such a matrix, and Nuke's Camera node carries one. Multiply a world point by the inverse of the camera's matrix and you get the point in camera space, where the camera looks down its own −z axis. Divide x and y by the distance in front of the lens, scale by the focal length, and you know where the point lands in the frame. This is how a tracked camera lets you pin a 2D element exactly over a 3D point, and how a world position pass can be turned into a camera-space one.

Once P is in the comp, many everyday jobs shrink to a line of maths. Height fog is a ramp on P's y channel. Distance fog uses the length of P minus the camera position. A cool grade on only the back wall is a test on P's z, the "wall position" itself. All three stay correct as the camera moves, all three can be changed in seconds when the director asks, and none of them needs a roto shape or a re-render from the lighting department.

Many samples in every pixel

A normal flat image keeps one colour and one alpha per pixel. Foundry describes deep images as containing "multiple samples per pixel at varying depths", each with its own colour, opacity and camera-relative depth. Follow one pixel of our room, now filled with low fog, and its ray passes through slice after slice of thin smoke before it reaches the floor. A flat render squashes all of that into one number. A deep render keeps every slice as its own sample, with a front depth and a back depth, so each pixel becomes a list of vectors sorted by distance. Nuke loads these lists with the DeepRead node from deep OpenEXR files and Pixar's .dtex format.

The payoff comes when you combine elements. Foundry's DeepMerge "merges samples from deep images and handles holdouts": it interleaves the samples of both inputs by depth, so a CG element slots in at the right place inside the smoke, partly hidden by whatever lies in front of it and partly hiding whatever lies behind. With flat images you would need a holdout matte rendered for every element against every other one, and a new render whenever something moved. A flat merge by depth only has one distance per pixel, so its test is all or nothing: the orb either cuts straight through the fog or vanishes behind it, with hard, stepped edges. This is why Foundry says that with deep data you can "render the background once, and later move your objects to different places and depths".

Sometimes the flat world still needs a matte. DeepHoldout takes a main deep image and a holdout deep image and gives back a flat image of the main input "with holes where the objects in the holdout input have occluded parts of it", either as an alpha matte alone or as colour plus alpha. Compositors use it to hand a clean, already held-out element to the rest of a flat script, or to get an alpha for an effect that only exists in 2D, such as a glow or a lens flare that should dim where the CG element is buried in smoke.

Lighting teams render deep, but the look of a shot is built in flat comp. **DeepRecolor** joins the two. Foundry describes it as merging "standard 2D and deep image color information": it takes the colour of each pixel from a flat image and spreads it over every sample of the matching deep pixel. By default it unpremultiplies by the flat image's alpha and then multiplies again by each sample's own alpha. With *target input alpha* turned on, it shares the flat alpha out among the samples so the flattened result matches the flat image exactly, which helps when compression or a shader change has nudged the deep alphas. So you grade the plate as usual, recolour the deep data, and every DeepMerge after that carries your grade.

Because each deep sample has a depth along a known camera ray, it is also a point in 3D: the camera position plus the ray's direction vector times the depth. DeepToPoints does exactly this. Foundry says it transforms "deep pixel samples into points in 3D space that you can see in Nuke's 3D view, like a point cloud", and suggests using it as a position reference. You see the room, the fog as a cloud of pale points and your CG element inside it, and you can line up cards, particles or new 3D elements against them without asking for a new render.

All of this has a price, and the price is disk space. A flat RGBA pixel in half float is 8 bytes. A deep sample needs its colour and alpha plus a front and a back depth, six values usually stored as 32-bit floats, so 24 bytes, and every pixel also records how many samples it holds. Our little room averages about **six** samples per pixel; hair, smoke, foliage and motion blur can push real shots to dozens. Multiply that out at 4K and a single frame is many times larger than its flat version, before you add a single extra pass. Deep OpenEXR files can be compressed, but only losslessly, so they never shrink the way a flat beauty can.

So deep is used where it earns its keep: volumes, hair, heavy motion blur and shots where many elements cross each other in depth. Lighting teams tune how finely the renderer slices volumes, and compositors throw away what they do not need, for example with DeepCrop, which Foundry describes as cropping "deep data relative to a given depth". The heatmap shows where the cost really is. The foggy floor holds almost all of the samples, while each pixel of the walls behind is a single opaque sample.

Put together, a compositor's script is a chain of vectors and matrices. Colour comes in as vectors and is turned by ColorMatrix into the right space and the right look. P and N come in as vectors and become masks and light through dot products, lengths and rotations. The camera's 4×4 matrix links the world to the frame. Deep pixels are lists of vectors that DeepMerge sorts by depth. You never need to do the maths by hand, because Nuke does it for every pixel, but knowing that a node is really nine numbers or a dot product tells you what it can and cannot do, and lets you build the tool that is missing from the menu.

Sources and further reading

• Foundry, Nuke Reference Guide, "ColorMatrix": learn.foundry.com
• Foundry, Nuke Reference Guide, "Relight" and "PositionToPoints": Relight · PositionToPoints
• Foundry, Nuke User Guide, "Deep Compositing": learn.foundry.com
• Foundry, Nuke Reference Guide, "Deep Nodes", "DeepHoldout", "DeepRecolor", "DeepToPoints": Deep Nodes · DeepHoldout · DeepRecolor · DeepToPoints
• Academy of Motion Picture Arts and Sciences, "ACEScg: A Working Space for CGI Render and Compositing" (S-2014-004)
• ITU-R Recommendation BT.709, parameter values for HDTV (primaries and luma weights)
• OpenEXR documentation, "Interpreting OpenEXR Deep Pixels": openexr.com
• Gilbert Strang, *Introduction to Linear Algebra* (6th ed., Wellesley-Cambridge Press, 2023)

*All figures are illustrations computed in your browser from one simple ray-traced room, not real production renders or screenshots of Nuke. The deep data, merges and relighting are real calculations on that small scene. Node descriptions follow Foundry's own documentation as of October 2026.*