RL Post-Training in Diffusion Transformers Is Activation Steering

Tuna Han Salih Meral1, Kaan Oktay2, Alec Helbling3,4, Atakan Kara2, Duen Horng Chau3, Pinar Yanardag1

1Virginia Tech, 2fal, 3Georgia Tech, 4IBM Research

Paper (coming soon)Code (coming soon)Weights (coming soon)BibTeX

RL post-training improves image and video models. We show that most of that improvement is one fixed push inside the model, the same for every prompt.

Part 1

RL post-training rewards a model for its own good images.

An image model is first trained on many images. Afterwards it can be trained further with a reward. The model draws several images for a prompt, each image gets a score, and the model’s weights, the numbers it has learned, are changed so that the better-scoring images become more likely. This is reinforcement-learning (RL) post-training. Often the original weights stay fixed and an RL method called GRPO, which compares each image to the group’s average, trains only a LoRA, a small set of added weights.

Prompt

“an owl”

Image model Original weightsLoRA

After many rounds: the RL-tuned model

Four images it draws for this prompt, each with a score

Image 1 the original model drew for “an owl”Image 1 the RL-tuned model drew for “an owl”

score 23.99

more likely

Image 2 the original model drew for “an owl”Image 2 the RL-tuned model drew for “an owl”

score 23.47

less likely

Image 3 the original model drew for “an owl”Image 3 the RL-tuned model drew for “an owl”

score 23.33

less likely

Image 4 the original model drew for “an owl”Image 4 the RL-tuned model drew for “an owl”

score 23.70

more likely

Average score of the four: 23.62

The score here is PickScore, a model that predicts which image people prefer.

GRPO updates the LoRA

After many rounds, the model draws higher-scoring images

Part 2

After RL, the model draws differently for every prompt.

The reward can be for human preference (PickScore), text rendering (OCR) or text-to-image alignment (GenEval). Every prompt asks for something different, so you would expect the change inside the model to differ from prompt to prompt. We asked how much of it one fixed change could do.

Text rendering

Score: OCR accuracy, how closely the text read from the image matches the request (1.00 is exact)

Original modelRL-tuned
Original model: A detailed inventory list of a dragon's hoard, prominently displaying "Gold Yes" among a pile of shimmering treasures, ancient coins, and valuable artifacts, set in a vast, mystical cave with a majestic dragon guarding the wealth.

OCR accuracy 0.00

RL-tuned: A detailed inventory list of a dragon's hoard, prominently displaying "Gold Yes" among a pile of shimmering treasures, ancient coins, and valuable artifacts, set in a vast, mystical cave with a majestic dragon guarding the wealth.

OCR accuracy 1.00

Must show Gold Yes

A detailed inventory list of a dragon's hoard, prominently displaying "Gold Yes" among a pile of shimmering treasures, ancient coins, and valuable artifacts, set in a vast, mystical cave with a majestic dragon guarding the wealth.

Text-to-image alignment

Score: GenEval, an automatic check that the requested objects are there (pass or fail)

Original modelRL-tuned
Original model: a photo of a cow and a horse

GenEval fail

RL-tuned: a photo of a cow and a horse

GenEval pass

“a photo of a cow and a horse”

Human preference

Score: PickScore, a model that predicts which image people prefer (higher is better)

Original modelRL-tuned
Original model: A golden chair on top of a balcony, outer space

PickScore 24.02

RL-tuned: A golden chair on top of a balcony, outer space

PickScore 27.09

“A golden chair on top of a balcony, outer space”

Human preference

Score: PickScore, a model that predicts which image people prefer (higher is better)

Original modelRL-tuned
Original model: bitcoin mango hanging from a tree next to the ocean

PickScore 22.05

RL-tuned: bitcoin mango hanging from a tree next to the ocean

PickScore 25.38

“bitcoin mango hanging from a tree next to the ocean”

Each prompt changed in its own way

Part 3

A steering vector nudges the numbers inside the model while it draws.

An image model is a stack of layers, and each layer passes on a column of numbers for every token, a small piece of the image or the prompt. After each layer we add that layer’s own fixed column of numbers, a steering vector, to every one of those columns, the same for every prompt. The layers stay locked, but they work on the shifted numbers, so the push changes each image in its own way. If most of what RL changed were one fixed push, the right vectors could copy it.

Threeprompts“Burn After Reading”“AI Training Zone”“Group B Experiment 9”Original modelLayer 1the numbers passed on, one column per tokenLayer 2…Last layerOCR accuracy 0.19OCR accuracy 0.21OCR accuracy 0.24Original model + steering vectorsLayer 1++ the same vector to every columnLayer 2+…Last layer+OCR accuracy 1.00OCR accuracy 1.00OCR accuracy 1.00“Burn After Reading”“AI Training Zone”“Group B Experiment 9”Original modelLayer 1the numbers passed on,one column per tokenLayer 2…Last layerOCR accuracy 0.19OCR accuracy 0.21OCR accuracy 0.24+ steering vectorsLayer 1++ the same vectorto every columnLayer 2…Last layer++OCR accuracy 1.00OCR accuracy 1.00OCR accuracy 1.00“Burn After Reading”“AI Training Zone”“Group B Experiment 9”

Every layer stays locked; only the numbers passed between layers change.

The same vectors change all three images, each in its own way

Part 4

We find the vectors by copying an RL-tuned model, without any reward.

The simplest idea is to average what RL changed inside the model and add that. It recovers only part of the improvement. So we fit the vectors instead: we take an image the RL-tuned model made, add noise, and ask both models which way to move it. We adjust only the vectors until the two answers agree.

How the vectors are found

Made by theRL-tuned modeladd noiseNoisy imageRL-tuned modelOriginal model +steering vectorsWhich way each modelmoves the noisy imageRL-tunedoriginal + vectorsadjust only the vectorsMade by theRL-tuned modeladd noiseNoisy imageRL-tuned modelOriginal model +steering vectorsWhich wayeach modelmoves thenoisy imageRL-tunedoriginal +vectorsadjustonly thevectors

After fitting, the two agree

Averaging versus fitting, SD3.5-M

0%25%50%75%RL-tuned, 100%100%Human preference: averaging: 61.0% of the RL gain, 95% interval 61.0 to 61.0Human preference: averaging61.0%Human preference: fitting: 90.8% of the RL gain, 95% interval 89.8 to 91.7Human preference: fitting90.8%Text rendering: averaging: 21.5% of the RL gain, 95% interval 21.5 to 21.5Text rendering: averaging21.5%Text rendering: fitting: 94.4% of the RL gain, 95% interval 90.8 to 97.9Text rendering: fitting94.4%Text-to-image alignment: averaging: 12.2% of the RL gain, 95% interval 12.2 to 12.2Text-to-image alignment: averaging12.2%Text-to-image alignment: fitting: 95.9% of the RL gain, 95% interval 92.0 to 99.7Text-to-image alignment: fitting95.9%

100% is the improvement of the RL-tuned model. Averaging: take the average change RL made inside the model and add it as the vectors. Fitting: the method described here. Thin line: the range we are 95% sure the value lies in.

Part 5

With the vectors added, the original model gets most of RL’s reward gain.

Each row uses the same prompt and the same random start for all three models. Each group has its own RL-tuned model and its own steering vectors. Within a group, every prompt gets the same vectors.

Human preference

Score: PickScore, a model that predicts which image people prefer (higher is better)

Original model+ steering vectorsRL-tuned
Original model: a sailing ship in a bottle

20.91

+ steering vectors: a sailing ship in a bottle

23.74

RL-tuned: a sailing ship in a bottle

23.82

“a sailing ship in a bottle”

Original model+ steering vectorsRL-tuned
Original model: a samurai cat

21.47

+ steering vectors: a samurai cat

25.04

RL-tuned: a samurai cat

25.20

“a samurai cat”

Original model+ steering vectorsRL-tuned
Original model: enigmatic black square building on top of a purple hill, smoke stacks

21.93

+ steering vectors: enigmatic black square building on top of a purple hill, smoke stacks

25.54

RL-tuned: enigmatic black square building on top of a purple hill, smoke stacks

25.16

“enigmatic black square building on top of a purple hill, smoke stacks”

Original model+ steering vectorsRL-tuned
Original model: a miniature forest inside a crystal ball

23.46

+ steering vectors: a miniature forest inside a crystal ball

24.23

RL-tuned: a miniature forest inside a crystal ball

24.08

“a miniature forest inside a crystal ball”

Show 8 more examples
Original model+ steering vectorsRL-tuned
Original model: A golden chair on top of a balcony, outer space

24.02

+ steering vectors: A golden chair on top of a balcony, outer space

27.33

RL-tuned: A golden chair on top of a balcony, outer space

27.09

“A golden chair on top of a balcony, outer space”

Original model+ steering vectorsRL-tuned
Original model: A golden acorn next to an oak tree within a giant hexagon

22.26

+ steering vectors: A golden acorn next to an oak tree within a giant hexagon

26.56

RL-tuned: A golden acorn next to an oak tree within a giant hexagon

26.09

“A golden acorn next to an oak tree within a giant hexagon”

Original model+ steering vectorsRL-tuned
Original model: A collection is small bells and whistles, yellow, flat lay

22.67

+ steering vectors: A collection is small bells and whistles, yellow, flat lay

25.81

RL-tuned: A collection is small bells and whistles, yellow, flat lay

25.80

“A collection is small bells and whistles, yellow, flat lay”

Original model+ steering vectorsRL-tuned
Original model: portrait of a head made out of pumpkin, on a wooden table, candle-lit from behind

23.28

+ steering vectors: portrait of a head made out of pumpkin, on a wooden table, candle-lit from behind

24.37

RL-tuned: portrait of a head made out of pumpkin, on a wooden table, candle-lit from behind

24.20

“portrait of a head made out of pumpkin, on a wooden table, candle-lit from behind”

Original model+ steering vectorsRL-tuned
Original model: bitcoin mango hanging from a tree next to the ocean

22.05

+ steering vectors: bitcoin mango hanging from a tree next to the ocean

25.58

RL-tuned: bitcoin mango hanging from a tree next to the ocean

25.38

“bitcoin mango hanging from a tree next to the ocean”

Original model+ steering vectorsRL-tuned
Original model: a minimalistic style logo for a game theory research group that studies market and information design

19.46

+ steering vectors: a minimalistic style logo for a game theory research group that studies market and information design

23.74

RL-tuned: a minimalistic style logo for a game theory research group that studies market and information design

23.90

“a minimalistic style logo for a game theory research group that studies market and information design”

Original model+ steering vectorsRL-tuned
Original model: A sailing ship on a prismatic ocean

23.43

+ steering vectors: A sailing ship on a prismatic ocean

25.95

RL-tuned: A sailing ship on a prismatic ocean

25.95

“A sailing ship on a prismatic ocean”

Original model+ steering vectorsRL-tuned
Original model: Portrait of an alien woman. She is cute and wearing glasses. Digital art

21.33

+ steering vectors: Portrait of an alien woman. She is cute and wearing glasses. Digital art

25.10

RL-tuned: Portrait of an alien woman. She is cute and wearing glasses. Digital art

25.38

“Portrait of an alien woman. She is cute and wearing glasses. Digital art”

Text rendering

Score: OCR accuracy, how closely the text read from the image matches the request (1.00 is exact)

Original model+ steering vectorsRL-tuned
Original model: A realistic photograph of a modern city street, with a large, vibrant sign that says "Deep Learning" illuminated in neon lights, set against a backdrop of bustling city life and futuristic architecture.

0.00

+ steering vectors: A realistic photograph of a modern city street, with a large, vibrant sign that says "Deep Learning" illuminated in neon lights, set against a backdrop of bustling city life and futuristic architecture.

1.00

RL-tuned: A realistic photograph of a modern city street, with a large, vibrant sign that says "Deep Learning" illuminated in neon lights, set against a backdrop of bustling city life and futuristic architecture.

1.00

Must show Deep Learning

A realistic photograph of a modern city street, with a large, vibrant sign that says "Deep Learning" illuminated in neon lights, set against a backdrop of bustling city life and futuristic architecture.

Original model+ steering vectorsRL-tuned
Original model: A vibrant autumn leaf collage titled "Fall Colors", featuring a harmonious mix of red, orange, and yellow leaves arranged in a natural, artistic pattern against a subtle white background.

0.00

+ steering vectors: A vibrant autumn leaf collage titled "Fall Colors", featuring a harmonious mix of red, orange, and yellow leaves arranged in a natural, artistic pattern against a subtle white background.

1.00

RL-tuned: A vibrant autumn leaf collage titled "Fall Colors", featuring a harmonious mix of red, orange, and yellow leaves arranged in a natural, artistic pattern against a subtle white background.

1.00

Must show Fall Colors

A vibrant autumn leaf collage titled "Fall Colors", featuring a harmonious mix of red, orange, and yellow leaves arranged in a natural, artistic pattern against a subtle white background.

Original model+ steering vectorsRL-tuned
Original model: A close-up photograph of an engraved silver ring with the inscription "Forever Yours" delicately etched into its surface, set against a soft, blurred background of romantic, warm tones.

1.00

+ steering vectors: A close-up photograph of an engraved silver ring with the inscription "Forever Yours" delicately etched into its surface, set against a soft, blurred background of romantic, warm tones.

1.00

RL-tuned: A close-up photograph of an engraved silver ring with the inscription "Forever Yours" delicately etched into its surface, set against a soft, blurred background of romantic, warm tones.

1.00

Must show Forever Yours

A close-up photograph of an engraved silver ring with the inscription "Forever Yours" delicately etched into its surface, set against a soft, blurred background of romantic, warm tones.

Original model+ steering vectorsRL-tuned
Original model: A vibrant, realistic photograph of a DJ turntable with a colorful sticker on its side declaring "Spin Masters", set against a backdrop of a bustling, neon-lit nightclub.

0.00

+ steering vectors: A vibrant, realistic photograph of a DJ turntable with a colorful sticker on its side declaring "Spin Masters", set against a backdrop of a bustling, neon-lit nightclub.

1.00

RL-tuned: A vibrant, realistic photograph of a DJ turntable with a colorful sticker on its side declaring "Spin Masters", set against a backdrop of a bustling, neon-lit nightclub.

1.00

Must show Spin Masters

A vibrant, realistic photograph of a DJ turntable with a colorful sticker on its side declaring "Spin Masters", set against a backdrop of a bustling, neon-lit nightclub.

Show 8 more examples
Original model+ steering vectorsRL-tuned
Original model: A vibrant food truck parked on a bustling city street, its side panel airbrushed with the bold slogan "Tacos Save Lives", surrounded by eager customers and the aroma of sizzling tacos.

0.14

+ steering vectors: A vibrant food truck parked on a bustling city street, its side panel airbrushed with the bold slogan "Tacos Save Lives", surrounded by eager customers and the aroma of sizzling tacos.

1.00

RL-tuned: A vibrant food truck parked on a bustling city street, its side panel airbrushed with the bold slogan "Tacos Save Lives", surrounded by eager customers and the aroma of sizzling tacos.

1.00

Must show Tacos Save Lives

A vibrant food truck parked on a bustling city street, its side panel airbrushed with the bold slogan "Tacos Save Lives", surrounded by eager customers and the aroma of sizzling tacos.

Original model+ steering vectorsRL-tuned
Original model: An ancient Pharaoh’s tomb interior, dimly lit by torches, revealing intricate hieroglyphs on the walls that spell "Cursed WiFi Password" in a blend of traditional Egyptian symbols and modern text.

0.06

+ steering vectors: An ancient Pharaoh’s tomb interior, dimly lit by torches, revealing intricate hieroglyphs on the walls that spell "Cursed WiFi Password" in a blend of traditional Egyptian symbols and modern text.

1.00

RL-tuned: An ancient Pharaoh’s tomb interior, dimly lit by torches, revealing intricate hieroglyphs on the walls that spell "Cursed WiFi Password" in a blend of traditional Egyptian symbols and modern text.

1.00

Must show Cursed WiFi Password

An ancient Pharaoh’s tomb interior, dimly lit by torches, revealing intricate hieroglyphs on the walls that spell "Cursed WiFi Password" in a blend of traditional Egyptian symbols and modern text.

Original model+ steering vectorsRL-tuned
Original model: A whimsical children’s storybook cover titled "The Brave Little Snail", featuring a tiny snail in a knight’s helmet, standing heroically on a leaf against a backdrop of a magical forest, with soft, warm lighting and vibrant, colorful illustrations.

0.00

+ steering vectors: A whimsical children’s storybook cover titled "The Brave Little Snail", featuring a tiny snail in a knight’s helmet, standing heroically on a leaf against a backdrop of a magical forest, with soft, warm lighting and vibrant, colorful illustrations.

1.00

RL-tuned: A whimsical children’s storybook cover titled "The Brave Little Snail", featuring a tiny snail in a knight’s helmet, standing heroically on a leaf against a backdrop of a magical forest, with soft, warm lighting and vibrant, colorful illustrations.

1.00

Must show The Brave Little Snail

A whimsical children’s storybook cover titled "The Brave Little Snail", featuring a tiny snail in a knight’s helmet, standing heroically on a leaf against a backdrop of a magical forest, with soft, warm lighting and vibrant, colorful illustrations.

Original model+ steering vectorsRL-tuned
Original model: A dramatic scene with a volcano warning siren flashing the message "Eruption Sale Today" in a bustling town square, where panicked citizens and curious onlookers gather, blending realism with surrealism.

0.71

+ steering vectors: A dramatic scene with a volcano warning siren flashing the message "Eruption Sale Today" in a bustling town square, where panicked citizens and curious onlookers gather, blending realism with surrealism.

1.00

RL-tuned: A dramatic scene with a volcano warning siren flashing the message "Eruption Sale Today" in a bustling town square, where panicked citizens and curious onlookers gather, blending realism with surrealism.

1.00

Must show Eruption Sale Today

A dramatic scene with a volcano warning siren flashing the message "Eruption Sale Today" in a bustling town square, where panicked citizens and curious onlookers gather, blending realism with surrealism.

Original model+ steering vectorsRL-tuned
Original model: A plant nursery shelf filled with seed packets, prominently labeled "Jurassic Park Ferns", surrounded by lush, prehistoric-looking ferns and other ancient plants, with a subtle mist adding an ethereal atmosphere.

0.00

+ steering vectors: A plant nursery shelf filled with seed packets, prominently labeled "Jurassic Park Ferns", surrounded by lush, prehistoric-looking ferns and other ancient plants, with a subtle mist adding an ethereal atmosphere.

1.00

RL-tuned: A plant nursery shelf filled with seed packets, prominently labeled "Jurassic Park Ferns", surrounded by lush, prehistoric-looking ferns and other ancient plants, with a subtle mist adding an ethereal atmosphere.

1.00

Must show Jurassic Park Ferns

A plant nursery shelf filled with seed packets, prominently labeled "Jurassic Park Ferns", surrounded by lush, prehistoric-looking ferns and other ancient plants, with a subtle mist adding an ethereal atmosphere.

Original model+ steering vectorsRL-tuned
Original model: A weathered cave explorer's journal page, with the phrase "Lost City Near" prominently written in faded ink, surrounded by sketches of ancient ruins and cryptic symbols, under a dim, mystical light.

0.00

+ steering vectors: A weathered cave explorer's journal page, with the phrase "Lost City Near" prominently written in faded ink, surrounded by sketches of ancient ruins and cryptic symbols, under a dim, mystical light.

1.00

RL-tuned: A weathered cave explorer's journal page, with the phrase "Lost City Near" prominently written in faded ink, surrounded by sketches of ancient ruins and cryptic symbols, under a dim, mystical light.

1.00

Must show Lost City Near

A weathered cave explorer's journal page, with the phrase "Lost City Near" prominently written in faded ink, surrounded by sketches of ancient ruins and cryptic symbols, under a dim, mystical light.

Original model+ steering vectorsRL-tuned
Original model: A detailed ski resort trail map with a prominent marker labeled "Black Diamond Run", set against a snowy backdrop with pine trees and skiers in the distance, capturing the thrill and challenge of the advanced slope.

0.13

+ steering vectors: A detailed ski resort trail map with a prominent marker labeled "Black Diamond Run", set against a snowy backdrop with pine trees and skiers in the distance, capturing the thrill and challenge of the advanced slope.

1.00

RL-tuned: A detailed ski resort trail map with a prominent marker labeled "Black Diamond Run", set against a snowy backdrop with pine trees and skiers in the distance, capturing the thrill and challenge of the advanced slope.

1.00

Must show Black Diamond Run

A detailed ski resort trail map with a prominent marker labeled "Black Diamond Run", set against a snowy backdrop with pine trees and skiers in the distance, capturing the thrill and challenge of the advanced slope.

Original model+ steering vectorsRL-tuned
Original model: An astronaut standing on the gray surface of the moon with the blue Earth rising behind, holding up a white sign that reads "All That RL for a Constant Shift?" in bold black letters.

0.85

+ steering vectors: An astronaut standing on the gray surface of the moon with the blue Earth rising behind, holding up a white sign that reads "All That RL for a Constant Shift?" in bold black letters.

1.00

RL-tuned: An astronaut standing on the gray surface of the moon with the blue Earth rising behind, holding up a white sign that reads "All That RL for a Constant Shift?" in bold black letters.

1.00

Must show All That RL for a Constant Shift?

An astronaut standing on the gray surface of the moon with the blue Earth rising behind, holding up a white sign that reads "All That RL for a Constant Shift?" in bold black letters.

Text-to-image alignment

Score: GenEval, an automatic check that the requested objects are there (pass or fail)

Original model+ steering vectorsRL-tuned
Original model: a photo of a toothbrush and a snowboard

fail

+ steering vectors: a photo of a toothbrush and a snowboard

pass

RL-tuned: a photo of a toothbrush and a snowboard

pass

“a photo of a toothbrush and a snowboard”

Original model+ steering vectorsRL-tuned
Original model: a photo of four dogs

fail

+ steering vectors: a photo of four dogs

pass

RL-tuned: a photo of four dogs

pass

“a photo of four dogs”

Original model+ steering vectorsRL-tuned
Original model: a photo of a frisbee and an apple

pass

+ steering vectors: a photo of a frisbee and an apple

pass

RL-tuned: a photo of a frisbee and an apple

pass

“a photo of a frisbee and an apple”

Original model+ steering vectorsRL-tuned
Original model: a photo of three kites

fail

+ steering vectors: a photo of three kites

pass

RL-tuned: a photo of three kites

pass

“a photo of three kites”

Show 8 more examples
Original model+ steering vectorsRL-tuned
Original model: a photo of a yellow skateboard and an orange computer mouse

fail

+ steering vectors: a photo of a yellow skateboard and an orange computer mouse

pass

RL-tuned: a photo of a yellow skateboard and an orange computer mouse

pass

“a photo of a yellow skateboard and an orange computer mouse”

Original model+ steering vectorsRL-tuned
Original model: a photo of a knife and a zebra

fail

+ steering vectors: a photo of a knife and a zebra

pass

RL-tuned: a photo of a knife and a zebra

pass

“a photo of a knife and a zebra”

Original model+ steering vectorsRL-tuned
Original model: a photo of a baseball bat

fail

+ steering vectors: a photo of a baseball bat

pass

RL-tuned: a photo of a baseball bat

pass

“a photo of a baseball bat”

Original model+ steering vectorsRL-tuned
Original model: a photo of an elephant

pass

+ steering vectors: a photo of an elephant

pass

RL-tuned: a photo of an elephant

pass

“a photo of an elephant”

Original model+ steering vectorsRL-tuned
Original model: a photo of a hair drier and a bear

fail

+ steering vectors: a photo of a hair drier and a bear

pass

RL-tuned: a photo of a hair drier and a bear

pass

“a photo of a hair drier and a bear”

Original model+ steering vectorsRL-tuned
Original model: a photo of a microwave and a truck

fail

+ steering vectors: a photo of a microwave and a truck

pass

RL-tuned: a photo of a microwave and a truck

pass

“a photo of a microwave and a truck”

Original model+ steering vectorsRL-tuned
Original model: a photo of a tv and a carrot

fail

+ steering vectors: a photo of a tv and a carrot

pass

RL-tuned: a photo of a tv and a carrot

pass

“a photo of a tv and a carrot”

Original model+ steering vectorsRL-tuned
Original model: a photo of three oranges

fail

+ steering vectors: a photo of three oranges

pass

RL-tuned: a photo of three oranges

pass

“a photo of three oranges”

Part 6

How much of RL’s improvement do the vectors give? Between 89.8% and 111.9%, on eight RL-tuned models.

Bar: the improvement from the vectors as a share of the improvement from RL. 100% means the same improvement as the RL-tuned model. Thin line: the range we are 95% sure the value lies in.

A value above 100% means the vectors scored higher than the RL-tuned model on that test.

PickScore, HPSv2.1 and HPSv3 predict which result people prefer. The rule score checks whether a visual puzzle was solved. “No KL” is a variant of the RL training that lets the model move further from the original.

SD3.5-M and FLUX.1-dev generate images. Wan2.1 and Wan2.2 generate video.

Pick a model to see its examples below.

0%25%50%75%RL-tuned, 100%100%SD3.5-M · Human preference (PickScore): 90.8% of the RL gain, 95% interval 89.8 to 91.7SD3.5-M · Human preference (PickScore)90.8%SD3.5-M · Human preference (PickScore), no KL: 95.2% of the RL gain, 95% interval 94.4 to 96.1SD3.5-M · Human preference (PickScore), no KL95.2%SD3.5-M · Text rendering (OCR): 94.4% of the RL gain, 95% interval 90.8 to 97.9SD3.5-M · Text rendering (OCR)94.4%SD3.5-M · Text-to-image alignment (GenEval): 95.9% of the RL gain, 95% interval 92.0 to 99.7SD3.5-M · Text-to-image alignment (GenEval)95.9%FLUX.1-dev · Human preference (PickScore): 90.0% of the RL gain, 95% interval 88.9 to 91.1FLUX.1-dev · Human preference (PickScore)90.0%FLUX.1-dev · Human preference (HPSv2.1): 96.3% of the RL gain, 95% interval 95.5 to 97.2FLUX.1-dev · Human preference (HPSv2.1)96.3%Wan2.1 · Human preference (HPSv3): 111.9% of the RL gain, 95% interval 106.5 to 118.0Wan2.1 · Human preference (HPSv3)111.9%Wan2.2 · Video reasoning (rule score): 89.8% of the RL gain, 95% interval 76.2 to 105.6Wan2.2 · Video reasoning (rule score)89.8%

Examples for SD3.5-M · Human preference (PickScore)

Original model+ steering vectorsRL-tuned
Original model: A sailing ship on a prismatic ocean

23.43

+ steering vectors: A sailing ship on a prismatic ocean

25.95

RL-tuned: A sailing ship on a prismatic ocean

25.95

“A sailing ship on a prismatic ocean”

Original model+ steering vectorsRL-tuned
Original model: A golden acorn next to an oak tree within a giant hexagon

22.26

+ steering vectors: A golden acorn next to an oak tree within a giant hexagon

26.56

RL-tuned: A golden acorn next to an oak tree within a giant hexagon

26.09

“A golden acorn next to an oak tree within a giant hexagon”

Original model+ steering vectorsRL-tuned
Original model: a samurai cat

21.47

+ steering vectors: a samurai cat

25.04

RL-tuned: a samurai cat

25.20

“a samurai cat”

Original model+ steering vectorsRL-tuned
Original model: A golden chair on top of a balcony, outer space

24.02

+ steering vectors: A golden chair on top of a balcony, outer space

27.33

RL-tuned: A golden chair on top of a balcony, outer space

27.09

“A golden chair on top of a balcony, outer space”

More examples for every model in the gallery

Part 7

Steering has a strength.

Turn it up and the image moves from the original toward the RL-tuned one.

Pick a prompt

Original model
Original model: a samurai cat

PickScore 21.47

Strength 0: a samurai catStrength 0.25: a samurai catStrength 0.5: a samurai catStrength 0.75: a samurai catStrength 1: a samurai catStrength 1.25: a samurai catStrength 1.5: a samurai catStrength 2: a samurai cat

PickScore 21.4723.6424.3424.1825.0425.1725.3125.90

RL-tuned
RL-tuned: a samurai cat

PickScore 25.20

“a samurai cat”

Original model
Original model: a sailing ship in a bottle

PickScore 20.91

Strength 0: a sailing ship in a bottleStrength 0.25: a sailing ship in a bottleStrength 0.5: a sailing ship in a bottleStrength 0.75: a sailing ship in a bottleStrength 1: a sailing ship in a bottleStrength 1.25: a sailing ship in a bottleStrength 1.5: a sailing ship in a bottleStrength 2: a sailing ship in a bottle

PickScore 20.9123.1623.4823.6623.7423.9123.7823.62

RL-tuned
RL-tuned: a sailing ship in a bottle

PickScore 23.82

“a sailing ship in a bottle”

Original model
Original model: enigmatic black square building on top of a purple hill, smoke stacks

PickScore 21.93

Strength 0: enigmatic black square building on top of a purple hill, smoke stacksStrength 0.25: enigmatic black square building on top of a purple hill, smoke stacksStrength 0.5: enigmatic black square building on top of a purple hill, smoke stacksStrength 0.75: enigmatic black square building on top of a purple hill, smoke stacksStrength 1: enigmatic black square building on top of a purple hill, smoke stacksStrength 1.25: enigmatic black square building on top of a purple hill, smoke stacksStrength 1.5: enigmatic black square building on top of a purple hill, smoke stacksStrength 2: enigmatic black square building on top of a purple hill, smoke stacks

PickScore 21.9322.6923.7224.8925.5425.7325.5825.51

RL-tuned
RL-tuned: enigmatic black square building on top of a purple hill, smoke stacks

PickScore 25.16

“enigmatic black square building on top of a purple hill, smoke stacks”

Original model
Original model: A golden chair on top of a balcony, outer space

PickScore 24.02

Strength 0: A golden chair on top of a balcony, outer spaceStrength 0.25: A golden chair on top of a balcony, outer spaceStrength 0.5: A golden chair on top of a balcony, outer spaceStrength 0.75: A golden chair on top of a balcony, outer spaceStrength 1: A golden chair on top of a balcony, outer spaceStrength 1.25: A golden chair on top of a balcony, outer spaceStrength 1.5: A golden chair on top of a balcony, outer spaceStrength 2: A golden chair on top of a balcony, outer space

PickScore 24.0225.8626.9926.3627.3327.2927.4427.25

RL-tuned
RL-tuned: A golden chair on top of a balcony, outer space

PickScore 27.09

“A golden chair on top of a balcony, outer space”

Original model
Original model: A golden acorn next to an oak tree within a giant hexagon

PickScore 22.26

Strength 0: A golden acorn next to an oak tree within a giant hexagonStrength 0.25: A golden acorn next to an oak tree within a giant hexagonStrength 0.5: A golden acorn next to an oak tree within a giant hexagonStrength 0.75: A golden acorn next to an oak tree within a giant hexagonStrength 1: A golden acorn next to an oak tree within a giant hexagonStrength 1.25: A golden acorn next to an oak tree within a giant hexagonStrength 1.5: A golden acorn next to an oak tree within a giant hexagonStrength 2: A golden acorn next to an oak tree within a giant hexagon

PickScore 22.2624.4324.8226.0626.5625.7225.6825.66

RL-tuned
RL-tuned: A golden acorn next to an oak tree within a giant hexagon

PickScore 26.09

“A golden acorn next to an oak tree within a giant hexagon”

Original model
Original model: bitcoin mango hanging from a tree next to the ocean

PickScore 22.05

Strength 0: bitcoin mango hanging from a tree next to the oceanStrength 0.25: bitcoin mango hanging from a tree next to the oceanStrength 0.5: bitcoin mango hanging from a tree next to the oceanStrength 0.75: bitcoin mango hanging from a tree next to the oceanStrength 1: bitcoin mango hanging from a tree next to the oceanStrength 1.25: bitcoin mango hanging from a tree next to the oceanStrength 1.5: bitcoin mango hanging from a tree next to the oceanStrength 2: bitcoin mango hanging from a tree next to the ocean

PickScore 22.0523.1323.1525.5325.5825.6825.6425.79

RL-tuned
RL-tuned: bitcoin mango hanging from a tree next to the ocean

PickScore 25.38

“bitcoin mango hanging from a tree next to the ocean”

Original model
Original model: A realistic photograph of a modern city street, with a large, vibrant sign that says "Deep Learning" illuminated in neon lights, set against a backdrop of bustling city life and futuristic architecture.

OCR accuracy 0.00

Strength 0: A realistic photograph of a modern city street, with a large, vibrant sign that says "Deep Learning" illuminated in neon lights, set against a backdrop of bustling city life and futuristic architecture.Strength 0.25: A realistic photograph of a modern city street, with a large, vibrant sign that says "Deep Learning" illuminated in neon lights, set against a backdrop of bustling city life and futuristic architecture.Strength 0.5: A realistic photograph of a modern city street, with a large, vibrant sign that says "Deep Learning" illuminated in neon lights, set against a backdrop of bustling city life and futuristic architecture.Strength 0.75: A realistic photograph of a modern city street, with a large, vibrant sign that says "Deep Learning" illuminated in neon lights, set against a backdrop of bustling city life and futuristic architecture.Strength 1: A realistic photograph of a modern city street, with a large, vibrant sign that says "Deep Learning" illuminated in neon lights, set against a backdrop of bustling city life and futuristic architecture.Strength 1.25: A realistic photograph of a modern city street, with a large, vibrant sign that says "Deep Learning" illuminated in neon lights, set against a backdrop of bustling city life and futuristic architecture.Strength 1.5: A realistic photograph of a modern city street, with a large, vibrant sign that says "Deep Learning" illuminated in neon lights, set against a backdrop of bustling city life and futuristic architecture.Strength 2: A realistic photograph of a modern city street, with a large, vibrant sign that says "Deep Learning" illuminated in neon lights, set against a backdrop of bustling city life and futuristic architecture.

OCR accuracy 0.000.921.001.001.001.001.001.00

RL-tuned
RL-tuned: A realistic photograph of a modern city street, with a large, vibrant sign that says "Deep Learning" illuminated in neon lights, set against a backdrop of bustling city life and futuristic architecture.

OCR accuracy 1.00

Must show Deep Learning

A realistic photograph of a modern city street, with a large, vibrant sign that says "Deep Learning" illuminated in neon lights, set against a backdrop of bustling city life and futuristic architecture.

Original model
Original model: A vibrant autumn leaf collage titled "Fall Colors", featuring a harmonious mix of red, orange, and yellow leaves arranged in a natural, artistic pattern against a subtle white background.

OCR accuracy 0.00

Strength 0: A vibrant autumn leaf collage titled "Fall Colors", featuring a harmonious mix of red, orange, and yellow leaves arranged in a natural, artistic pattern against a subtle white background.Strength 0.25: A vibrant autumn leaf collage titled "Fall Colors", featuring a harmonious mix of red, orange, and yellow leaves arranged in a natural, artistic pattern against a subtle white background.Strength 0.5: A vibrant autumn leaf collage titled "Fall Colors", featuring a harmonious mix of red, orange, and yellow leaves arranged in a natural, artistic pattern against a subtle white background.Strength 0.75: A vibrant autumn leaf collage titled "Fall Colors", featuring a harmonious mix of red, orange, and yellow leaves arranged in a natural, artistic pattern against a subtle white background.Strength 1: A vibrant autumn leaf collage titled "Fall Colors", featuring a harmonious mix of red, orange, and yellow leaves arranged in a natural, artistic pattern against a subtle white background.Strength 1.25: A vibrant autumn leaf collage titled "Fall Colors", featuring a harmonious mix of red, orange, and yellow leaves arranged in a natural, artistic pattern against a subtle white background.Strength 1.5: A vibrant autumn leaf collage titled "Fall Colors", featuring a harmonious mix of red, orange, and yellow leaves arranged in a natural, artistic pattern against a subtle white background.Strength 2: A vibrant autumn leaf collage titled "Fall Colors", featuring a harmonious mix of red, orange, and yellow leaves arranged in a natural, artistic pattern against a subtle white background.

OCR accuracy 0.000.000.001.001.001.001.001.00

RL-tuned
RL-tuned: A vibrant autumn leaf collage titled "Fall Colors", featuring a harmonious mix of red, orange, and yellow leaves arranged in a natural, artistic pattern against a subtle white background.

OCR accuracy 1.00

Must show Fall Colors

A vibrant autumn leaf collage titled "Fall Colors", featuring a harmonious mix of red, orange, and yellow leaves arranged in a natural, artistic pattern against a subtle white background.

Original model
Original model: A vibrant, realistic photograph of a DJ turntable with a colorful sticker on its side declaring "Spin Masters", set against a backdrop of a bustling, neon-lit nightclub.

OCR accuracy 0.00

Strength 0: A vibrant, realistic photograph of a DJ turntable with a colorful sticker on its side declaring "Spin Masters", set against a backdrop of a bustling, neon-lit nightclub.Strength 0.25: A vibrant, realistic photograph of a DJ turntable with a colorful sticker on its side declaring "Spin Masters", set against a backdrop of a bustling, neon-lit nightclub.Strength 0.5: A vibrant, realistic photograph of a DJ turntable with a colorful sticker on its side declaring "Spin Masters", set against a backdrop of a bustling, neon-lit nightclub.Strength 0.75: A vibrant, realistic photograph of a DJ turntable with a colorful sticker on its side declaring "Spin Masters", set against a backdrop of a bustling, neon-lit nightclub.Strength 1: A vibrant, realistic photograph of a DJ turntable with a colorful sticker on its side declaring "Spin Masters", set against a backdrop of a bustling, neon-lit nightclub.Strength 1.25: A vibrant, realistic photograph of a DJ turntable with a colorful sticker on its side declaring "Spin Masters", set against a backdrop of a bustling, neon-lit nightclub.Strength 1.5: A vibrant, realistic photograph of a DJ turntable with a colorful sticker on its side declaring "Spin Masters", set against a backdrop of a bustling, neon-lit nightclub.Strength 2: A vibrant, realistic photograph of a DJ turntable with a colorful sticker on its side declaring "Spin Masters", set against a backdrop of a bustling, neon-lit nightclub.

OCR accuracy 0.000.000.821.001.001.001.001.00

RL-tuned
RL-tuned: A vibrant, realistic photograph of a DJ turntable with a colorful sticker on its side declaring "Spin Masters", set against a backdrop of a bustling, neon-lit nightclub.

OCR accuracy 1.00

Must show Spin Masters

A vibrant, realistic photograph of a DJ turntable with a colorful sticker on its side declaring "Spin Masters", set against a backdrop of a bustling, neon-lit nightclub.

Original model
Original model: A vibrant food truck parked on a bustling city street, its side panel airbrushed with the bold slogan "Tacos Save Lives", surrounded by eager customers and the aroma of sizzling tacos.

OCR accuracy 0.14

Strength 0: A vibrant food truck parked on a bustling city street, its side panel airbrushed with the bold slogan "Tacos Save Lives", surrounded by eager customers and the aroma of sizzling tacos.Strength 0.25: A vibrant food truck parked on a bustling city street, its side panel airbrushed with the bold slogan "Tacos Save Lives", surrounded by eager customers and the aroma of sizzling tacos.Strength 0.5: A vibrant food truck parked on a bustling city street, its side panel airbrushed with the bold slogan "Tacos Save Lives", surrounded by eager customers and the aroma of sizzling tacos.Strength 0.75: A vibrant food truck parked on a bustling city street, its side panel airbrushed with the bold slogan "Tacos Save Lives", surrounded by eager customers and the aroma of sizzling tacos.Strength 1: A vibrant food truck parked on a bustling city street, its side panel airbrushed with the bold slogan "Tacos Save Lives", surrounded by eager customers and the aroma of sizzling tacos.Strength 1.25: A vibrant food truck parked on a bustling city street, its side panel airbrushed with the bold slogan "Tacos Save Lives", surrounded by eager customers and the aroma of sizzling tacos.Strength 1.5: A vibrant food truck parked on a bustling city street, its side panel airbrushed with the bold slogan "Tacos Save Lives", surrounded by eager customers and the aroma of sizzling tacos.Strength 2: A vibrant food truck parked on a bustling city street, its side panel airbrushed with the bold slogan "Tacos Save Lives", surrounded by eager customers and the aroma of sizzling tacos.

OCR accuracy 0.140.430.000.931.001.001.001.00

RL-tuned
RL-tuned: A vibrant food truck parked on a bustling city street, its side panel airbrushed with the bold slogan "Tacos Save Lives", surrounded by eager customers and the aroma of sizzling tacos.

OCR accuracy 1.00

Must show Tacos Save Lives

A vibrant food truck parked on a bustling city street, its side panel airbrushed with the bold slogan "Tacos Save Lives", surrounded by eager customers and the aroma of sizzling tacos.

Original model
Original model: A whimsical children’s storybook cover titled "The Brave Little Snail", featuring a tiny snail in a knight’s helmet, standing heroically on a leaf against a backdrop of a magical forest, with soft, warm lighting and vibrant, colorful illustrations.

OCR accuracy 0.00

Strength 0: A whimsical children’s storybook cover titled "The Brave Little Snail", featuring a tiny snail in a knight’s helmet, standing heroically on a leaf against a backdrop of a magical forest, with soft, warm lighting and vibrant, colorful illustrations.Strength 0.25: A whimsical children’s storybook cover titled "The Brave Little Snail", featuring a tiny snail in a knight’s helmet, standing heroically on a leaf against a backdrop of a magical forest, with soft, warm lighting and vibrant, colorful illustrations.Strength 0.5: A whimsical children’s storybook cover titled "The Brave Little Snail", featuring a tiny snail in a knight’s helmet, standing heroically on a leaf against a backdrop of a magical forest, with soft, warm lighting and vibrant, colorful illustrations.Strength 0.75: A whimsical children’s storybook cover titled "The Brave Little Snail", featuring a tiny snail in a knight’s helmet, standing heroically on a leaf against a backdrop of a magical forest, with soft, warm lighting and vibrant, colorful illustrations.Strength 1: A whimsical children’s storybook cover titled "The Brave Little Snail", featuring a tiny snail in a knight’s helmet, standing heroically on a leaf against a backdrop of a magical forest, with soft, warm lighting and vibrant, colorful illustrations.Strength 1.25: A whimsical children’s storybook cover titled "The Brave Little Snail", featuring a tiny snail in a knight’s helmet, standing heroically on a leaf against a backdrop of a magical forest, with soft, warm lighting and vibrant, colorful illustrations.Strength 1.5: A whimsical children’s storybook cover titled "The Brave Little Snail", featuring a tiny snail in a knight’s helmet, standing heroically on a leaf against a backdrop of a magical forest, with soft, warm lighting and vibrant, colorful illustrations.Strength 2: A whimsical children’s storybook cover titled "The Brave Little Snail", featuring a tiny snail in a knight’s helmet, standing heroically on a leaf against a backdrop of a magical forest, with soft, warm lighting and vibrant, colorful illustrations.

OCR accuracy 0.001.001.001.001.001.001.000.95

RL-tuned
RL-tuned: A whimsical children’s storybook cover titled "The Brave Little Snail", featuring a tiny snail in a knight’s helmet, standing heroically on a leaf against a backdrop of a magical forest, with soft, warm lighting and vibrant, colorful illustrations.

OCR accuracy 1.00

Must show The Brave Little Snail

A whimsical children’s storybook cover titled "The Brave Little Snail", featuring a tiny snail in a knight’s helmet, standing heroically on a leaf against a backdrop of a magical forest, with soft, warm lighting and vibrant, colorful illustrations.

Original model
Original model: A detailed inventory list of a dragon's hoard, prominently displaying "Gold Yes" among a pile of shimmering treasures, ancient coins, and valuable artifacts, set in a vast, mystical cave with a majestic dragon guarding the wealth.

OCR accuracy 0.00

Strength 0: A detailed inventory list of a dragon's hoard, prominently displaying "Gold Yes" among a pile of shimmering treasures, ancient coins, and valuable artifacts, set in a vast, mystical cave with a majestic dragon guarding the wealth.Strength 0.25: A detailed inventory list of a dragon's hoard, prominently displaying "Gold Yes" among a pile of shimmering treasures, ancient coins, and valuable artifacts, set in a vast, mystical cave with a majestic dragon guarding the wealth.Strength 0.5: A detailed inventory list of a dragon's hoard, prominently displaying "Gold Yes" among a pile of shimmering treasures, ancient coins, and valuable artifacts, set in a vast, mystical cave with a majestic dragon guarding the wealth.Strength 0.75: A detailed inventory list of a dragon's hoard, prominently displaying "Gold Yes" among a pile of shimmering treasures, ancient coins, and valuable artifacts, set in a vast, mystical cave with a majestic dragon guarding the wealth.Strength 1: A detailed inventory list of a dragon's hoard, prominently displaying "Gold Yes" among a pile of shimmering treasures, ancient coins, and valuable artifacts, set in a vast, mystical cave with a majestic dragon guarding the wealth.Strength 1.25: A detailed inventory list of a dragon's hoard, prominently displaying "Gold Yes" among a pile of shimmering treasures, ancient coins, and valuable artifacts, set in a vast, mystical cave with a majestic dragon guarding the wealth.Strength 1.5: A detailed inventory list of a dragon's hoard, prominently displaying "Gold Yes" among a pile of shimmering treasures, ancient coins, and valuable artifacts, set in a vast, mystical cave with a majestic dragon guarding the wealth.Strength 2: A detailed inventory list of a dragon's hoard, prominently displaying "Gold Yes" among a pile of shimmering treasures, ancient coins, and valuable artifacts, set in a vast, mystical cave with a majestic dragon guarding the wealth.

OCR accuracy 0.000.140.861.001.001.001.001.00

RL-tuned
RL-tuned: A detailed inventory list of a dragon's hoard, prominently displaying "Gold Yes" among a pile of shimmering treasures, ancient coins, and valuable artifacts, set in a vast, mystical cave with a majestic dragon guarding the wealth.

OCR accuracy 1.00

Must show Gold Yes

A detailed inventory list of a dragon's hoard, prominently displaying "Gold Yes" among a pile of shimmering treasures, ancient coins, and valuable artifacts, set in a vast, mystical cave with a majestic dragon guarding the wealth.

Part 8

RL can also train the vectors directly.

So far we built the vectors by copying a model that RL had already trained. Can RL find such vectors on its own? We ran RL on the original model with every weight frozen, training only the vectors, starting from zeros. It ends up slightly ahead of the RL-tuned models: 103.6% of their improvement for human preference and 104.6% for text rendering. In a user study, people picked these images over the original model’s about 6 times in 10 (62% to 65%, a tie counted as half), and as often as the RL-tuned models’ (52%).

Original model

vectors

draw images

score them

update only the vectors

Start: the vectors are zero
Start: the vectors are zero: Beautifully strange painting of an alien tropical island. Glowing trees and flora. Hyperdetailed matte painting by by Benoit B. Mandelbrot, Steven Belledin, Martin Johnson Heade, Lee Madgwick, and Caspar David Friedrich. Alien flora and fauna. Glowing orbs and tropics. Alien moonrise.
After training
After training: Beautifully strange painting of an alien tropical island. Glowing trees and flora. Hyperdetailed matte painting by by Benoit B. Mandelbrot, Steven Belledin, Martin Johnson Heade, Lee Madgwick, and Caspar David Friedrich. Alien flora and fauna. Glowing orbs and tropics. Alien moonrise.
0%25%50%75%RL-tuned, 100%100%Vectors trained by RL (PickScore): 103.6% of the RL gain, 95% interval 102.0 to 105.1Vectors trained by RL (PickScore)103.6%Vectors trained by RL (OCR): 104.6% of the RL gain, 95% interval 102.1 to 107.3Vectors trained by RL (OCR)104.6%

Trained by RL alone, the vectors give slightly more than the RL-tuned model’s improvement

Bar: the improvement from the vectors as a share of the improvement from RL. 100% means the same improvement as the RL-tuned model. Thin line: the range we are 95% sure the value lies in.

Original model+ vectors trained by RLRL-tuned
Original model: Beautifully strange painting of an alien tropical island. Glowing trees and flora. Hyperdetailed matte painting by by Benoit B. Mandelbrot, Steven Belledin, Martin Johnson Heade, Lee Madgwick, and Caspar David Friedrich. Alien flora and fauna. Glowing orbs and tropics. Alien moonrise.

PickScore 22.45

+ steering vectors: Beautifully strange painting of an alien tropical island. Glowing trees and flora. Hyperdetailed matte painting by by Benoit B. Mandelbrot, Steven Belledin, Martin Johnson Heade, Lee Madgwick, and Caspar David Friedrich. Alien flora and fauna. Glowing orbs and tropics. Alien moonrise.

PickScore 25.10

RL-tuned: Beautifully strange painting of an alien tropical island. Glowing trees and flora. Hyperdetailed matte painting by by Benoit B. Mandelbrot, Steven Belledin, Martin Johnson Heade, Lee Madgwick, and Caspar David Friedrich. Alien flora and fauna. Glowing orbs and tropics. Alien moonrise.

PickScore 24.78

“Beautifully strange painting of an alien tropical island. Glowing trees and flora. Hyperdetailed matte painting by by Benoit B. Mandelbrot, Steven Belledin, Martin Johnson Heade, Lee Madgwick, and Caspar David Friedrich. Alien flora and fauna. Glowing orbs and tropics. Alien moonrise.”

Original model+ vectors trained by RLRL-tuned
Original model: spectral asteroid colliding with a broken planet

PickScore 20.84

+ steering vectors: spectral asteroid colliding with a broken planet

PickScore 23.87

RL-tuned: spectral asteroid colliding with a broken planet

PickScore 23.68

“spectral asteroid colliding with a broken planet”

Original model+ vectors trained by RLRL-tuned
Original model: A vintage library stamp, prominently marked "Forbidden Knowledge", is centered on an aged, parchment-like paper. The stamp features intricate, almost mystical designs around the text, suggesting a wealth of secret wisdom. The paper shows signs of wear, with faded edges and a few small tears, enhancing the sense of antiquity and mystery.

OCR accuracy 0.00

+ steering vectors: A vintage library stamp, prominently marked "Forbidden Knowledge", is centered on an aged, parchment-like paper. The stamp features intricate, almost mystical designs around the text, suggesting a wealth of secret wisdom. The paper shows signs of wear, with faded edges and a few small tears, enhancing the sense of antiquity and mystery.

OCR accuracy 1.00

RL-tuned: A vintage library stamp, prominently marked "Forbidden Knowledge", is centered on an aged, parchment-like paper. The stamp features intricate, almost mystical designs around the text, suggesting a wealth of secret wisdom. The paper shows signs of wear, with faded edges and a few small tears, enhancing the sense of antiquity and mystery.

OCR accuracy 1.00

Must show Forbidden Knowledge

A vintage library stamp, prominently marked "Forbidden Knowledge", is centered on an aged, parchment-like paper. The stamp features intricate, almost mystical designs around the text, suggesting a wealth of secret wisdom. The paper shows signs of wear, with faded edges and a few small tears, enhancing the sense of antiquity and mystery.

Original model+ vectors trained by RLRL-tuned
Original model: A weathered treasure map with a prominent "X  Gold Here" marking, laid on an old wooden table. Sunlight filters through the leaves, casting dappled shadows. A compass and antique lantern rest beside the map, hinting at the adventure to come.

OCR accuracy 0.00

+ steering vectors: A weathered treasure map with a prominent "X  Gold Here" marking, laid on an old wooden table. Sunlight filters through the leaves, casting dappled shadows. A compass and antique lantern rest beside the map, hinting at the adventure to come.

OCR accuracy 1.00

RL-tuned: A weathered treasure map with a prominent "X  Gold Here" marking, laid on an old wooden table. Sunlight filters through the leaves, casting dappled shadows. A compass and antique lantern rest beside the map, hinting at the adventure to come.

OCR accuracy 1.00

Must show X Gold Here

A weathered treasure map with a prominent "X Gold Here" marking, laid on an old wooden table. Sunlight filters through the leaves, casting dappled shadows. A compass and antique lantern rest beside the map, hinting at the adventure to come.

Part 9

This holds for RL, not for every kind of fine-tuning.

Is this true of any fine-tuning, or of RL in particular? We tested another kind. Teaching a model a new subject from a few photos, a method called DreamBooth, is different. There fixed vectors recover only 68% to 83% of the improvement for new people and 73% to 95% for other new subjects.

Photos of the subject

Real photos used for fine-tuning, not generated

A photo of the subject used for fine-tuningA photo of the subject used for fine-tuning
Fine-tuned modelOriginal model + steering vectors
Fine-tuned model: the toy in the snow
Original model + steering vectors: the toy in the snow

“the toy in the snow”

Fine-tuned modelOriginal model + steering vectors
Fine-tuned model: the toy with a city in the background
Original model + steering vectors: the toy with a city in the background

“the toy with a city in the background”

The steering vectors move the original model toward the subject but do not match the fine-tuned model.

Part 10

Most of the effect sits in a small part of the vectors.

Where inside the model does the effect sit? We kept only part of the vectors. On SD3.5-M there are 188 vectors in all, one for each place and stage inside the model where numbers are added. Keep only the vectors that act on the prompt’s text, early in the drawing process: 23 of the 188. That is 531 times fewer numbers than the RL update, and it still recovers 89.5% of the improvement for text rendering and 73.7% for human preference.

188 → 23 vectors

0%25%50%75%RL-tuned, 100%100%Text rendering: all 188 vectors: 94.4% of the RL gain, 95% interval 90.8 to 97.9Text rendering: all 188 vectors94.4%Text rendering: 23 vectors: 89.5% of the RL gain, 95% interval 86.0 to 93.1Text rendering: 23 vectors89.5%Human preference: all 188 vectors: 90.8% of the RL gain, 95% interval 89.8 to 91.7Human preference: all 188 vectors90.8%Human preference: 23 vectors: 73.7% of the RL gain, 95% interval 72.3 to 75.1Human preference: 23 vectors73.7%

With 23 vectors, most of the improvement stays

Bar: the improvement from the vectors as a share of the improvement from RL. 100% means the same improvement as the RL-tuned model. Thin line: the range we are 95% sure the value lies in.

Part 11

Why can the same vectors help every prompt?

While the model draws, the image part looks at the prompt’s text. The vectors change the text part, and the image part sees that change. In the second row we undo only this: the image part looks at the text as the original model would have it. Then most of the improvement is gone, for the vectors and for the RL-tuned model alike.

Original model+ steering vectorsRL-tuned

Normal

Original model: An ancient, weathered stone tablet, partially covered in moss, with the intricate inscription "Kingdom of the Sun" prominently carved into its surface, set against a backdrop of dense, overgrown forest.
+ steering vectors: An ancient, weathered stone tablet, partially covered in moss, with the intricate inscription "Kingdom of the Sun" prominently carved into its surface, set against a backdrop of dense, overgrown forest.

Improvement kept 100%

RL-tuned: An ancient, weathered stone tablet, partially covered in moss, with the intricate inscription "Kingdom of the Sun" prominently carved into its surface, set against a backdrop of dense, overgrown forest.

Improvement kept 100%

The image part looks at the original model’s text

Unchanged

+ steering vectors, the image part looks at the original model’s text: An ancient, weathered stone tablet, partially covered in moss, with the intricate inscription "Kingdom of the Sun" prominently carved into its surface, set against a backdrop of dense, overgrown forest.

Improvement kept 20.5%

RL-tuned, the image part looks at the original model’s text: An ancient, weathered stone tablet, partially covered in moss, with the intricate inscription "Kingdom of the Sun" prominently carved into its surface, set against a backdrop of dense, overgrown forest.

Improvement kept 26.4%

Most of the improvement is gone, for the vectors and for the RL-tuned model alike

Must show Kingdom of the Sun

Measured on the text-rendering test set.

Part 12

Our reading: RL post-training mostly steers a model toward what it can already do.

The vectors add nothing that depends on the prompt. The unchanged model turns the same push into the right change for each prompt. Whether RL can teach these models something they cannot already do is still open.

BibTeX

BibTeX will be added with the arXiv version.