Notes on making AI-generated pixel art usable

I’ve been building a small desktop companion — a pixel-art creature that sits on your Mac and reacts to what you’re doing. Four creatures, actually, one per element. All of the art is generated: PixelLab’s create_image_pro for the static poses, animate_image for the frames in between. I’m putting that in the first paragraph because the generation is the boring part of this story.

Generating everything took an afternoon or two. Getting it to a state where I could put it in a product took considerably longer, and most of that time went into finding out how the output was wrong. Some of the failures were obvious the moment I looked. Several were not, and one of them I only caught after building a metric specifically to catch it, then discovering the metric couldn’t.

These are my notes. Numbers where I have them.

4

creatures

52

static sprites

128×128

every frame, RGBA

91%

of the time, doing nothing

The contract

The thing I got right early, mostly by accident, was deciding what “correct” meant before generating anything at scale.

Every sprite is 128×128 RGBA. The feet line sits at y=123. The horizontal center is x=64. No pixel touches the top edge of the frame. Those four properties are checked by a script, and if any of them fails the build fails.

y=123 x=64 no pixel here

for p, im in ims.items():
    a = im.getchannel("A"); l, t, r, b = a.getbbox()
    errs = []
    if im.size != (128, 128): errs.append("size")
    if b != 123: errs.append(f"feet y={b}")
    if abs((l + r) // 2 - 64) > 1: errs.append(f"center x={(l + r) // 2}")
    if any(a.getpixel((x, 0)) > 0 for x in range(128)): errs.append("top edge")
    h = holes(im)
    if h: errs.append(f"hole {h}px")
    if errs: struct.append(f"{e}/{p}: {', '.join(errs)}")

That sounds like bureaucracy for a hobby project. It isn’t. It means pose and creature are interchangeable at runtime — I can swap the fire creature’s idle for the water creature’s wave and nothing shifts by a pixel. Without it you spend the rest of the project hand-nudging files, and every new pose is a new alignment problem. With 52 static sprites and 412 animation frames, that’s not a thing you fix later.

Generative models do not respect a contract like this. They produce something roughly centered, roughly the right size, roughly consistent. All of the work described below exists to close the gap between “roughly” and “exactly.”

Quantizing the input costs you the whole sequence

This one cost me the most and I’d never seen it written down.

animate_image takes a first frame and a last frame and interpolates between them. I was sending both as base64. There’s a size limit on the argument, and to stay under it I quantized the two images down to 6 colors. My assumption — reasonable, wrong — was that this degraded the two frames I was sending, and that the frames in between would be generated fresh.

They aren’t generated fresh. The model interpolates from what you hand it, and the poverty of the input propagates through everything:

For context, the static sprite those frames have to blend into has 50 colors. So the animation was measurably flatter than the still image sitting next to it, which is exactly the kind of defect you feel before you can name it. Looking closely, what’s gone is the speckling on the stone creature’s cheeks and forehead, and the half-tones in its leaves, which posterize into flat bands.

The water creature had the same problem in mirror image: 8 of its 13 sequences came back at 31–40 colors from 12-color inputs, against 57–64 for the five I’d sent intact. I hadn’t noticed until I started counting.

Color counting is the cheapest defect detector I found

That table is the whole lesson. Counting unique colors per frame is two lines of Pillow, and it surfaced a defect that I had been looking directly at without seeing.

from PIL import Image
len(set(Image.open(path).convert("RGBA").getdata()))

I now count colors on every batch. Current floor per creature, across all intermediate frames: fire 49 (a yawn frame), water 54, earth 64. When something drops well below its neighbors, something went wrong upstream.

The related metric that also earns its keep: percentage of pixels differing by more than 40/255 from the creature’s idle sprite. It tells you whether a pose is actually a different pose.

def diff(a, b):
    d = ImageChops.difference(a.convert("RGB"), b.convert("RGB")).convert("L")
    return 100 * sum(1 for v in d.getdata() if v > 40) / 16384

Mine now sit between 2% and 28%, with the low end being the blink poses, where only the eyelids change, which is correct. Three poses originally came back at 3–7% — technically different images, visually the same drawing. Regenerated with prompts insisting on the amplitude of the gesture (arms fully extended, clear space between arm and body) they moved to 21–23%.

The fix was transport, not compression

I burned a lot of time solving the size limit the wrong way.

The measured threshold is about 7300 cumulative base64 characters in one call. Past that the argument truncates in transit; the server notices the image is incomplete and refuses. Two full 128×128 images exceed it comfortably.

So I cropped both images to a shared window (the union of their bounding boxes), dropped the palette to 16 colors, and that worked for most poses. happy at 7428 and listening at 7460 were still over. I moved to computing the window per pair instead of globally and dropped to 12 colors, which brought everything into 6300–6950.

All of that was wasted. animate_image also accepts first_frame_url and last_frame_url.

# Two encoded images. ~7300 characters, truncated in transit.
animate_image(
    first_frame=b64(crop(quantize(start, 12))),
    last_frame=b64(crop(quantize(end, 12))),
)

# Two URLs. 120 characters each, and the images go out at full size.
animate_image(
    first_frame_url="https://gist.githubusercontent.com/.../idle.png",
    last_frame_url="https://gist.githubusercontent.com/.../wave.png",
)

I pushed the sprites to a temporary public gist, passed the raw URLs, deleted the gist afterward. Two 120-character strings instead of two encoded images. No truncation, no quantization, no cropping — and because the images now went out at full size, the realignment offset computed afterward came out to (0,0) on seven sequences out of eight.

I optimized the payload for two days before checking whether I needed to send a payload at all.

Bounding boxes lie when something hangs below the feet

Generated frames come back off-register, so they get realigned. My realigner compares bounding boxes: find the box in the generated frame, find the box in the target sprite, shift by the difference.

def anchor_of(im):
    """(center_x, bottom_y) of the bounding box — the registration point."""
    l, t, r, b = im.getchannel("A").getbbox()
    return ((l + r) // 2, b)

if dest:      # transition: the last frame has to land on the static pose
    ref, src = Image.open(f"sprites/{element}/{dest}.png"), frames[-1]
else:         # loop: the first frame registers against the delivered idle
    ref, src = Image.open(f"sprites/{element}/idle.png"), frames[0]
(rx, ry), (ax, ay) = anchor_of(ref), anchor_of(src)
dx, dy = rx - ax, ry - ay

The air creature’s grab sprite trails thin wind wisps down to y=122, while its body stops at y=117. The generated frames don’t draw those wisps. On top of that the model placed the whole scene about 8px too high. Net result: the script corrected by +13px and pushed the start of the sequence 5px below the floor.

I regenerated three times, twice with prompts constraining the motion, and got the same +13 every time. Not noise. Structural.

Anchoring on the first frame instead of the last just relocates the problem: the creature then drops 18px onto the held pose, which is the frame displayed for the entire time you’re dragging it. Worse, because it’s the one you look at.

I dropped the two sunken frames. The sequence went from 7 frames to 5 and now rises cleanly, 122 → 118 → 116 → pose. Nobody will ever know two frames are missing.

Interpolate from the state the engine actually passes through

My drop sequence was generated from grab → drop. You pick the creature up, you let go, it falls. Obvious.

It was wrong for a reason that has nothing to do with the model. My animation engine always returns to idle before playing a new sequence. So on release you’d see: grab, rewind to idle, then the first frame of drop — which is the grab — then the landing. The grab played twice, with a rewind in the middle.

Measured, drop_00 was 10.7% different from grab.png and 23.6% different from idle.png. Three of the five frames were visually just the grab.

Regenerated as idle → drop, frame 0 came down to 0% from idle — it doesn’t resemble the idle sprite, it is the idle sprite, pixel for pixel — and the divergence between the two sequences now widens to 26% across the frames, which is what you want: they start together and separate.

grab Pause 1/5

drop Pause 1/7

Some defects have no metric

The listening sequence came back with a pointed cone pushed up above the creature’s head on 2 frames out of 7, with the cloud spiral erased. A different creature, briefly.

Pixel difference didn’t flag it. The anchor didn’t. Luminance didn’t. I tried building a discriminator on “how much does the sprite grow upward,” which sounds promising until you check the other creatures: the air creature grows 12px on this defect, and the earth creature grows 17–20px with nothing wrong at all, because its floating pebbles pass above its head. There is no threshold that separates those two cases. I looked.

I found it by eye, on a contact sheet, which I now generate for every batch.

The fix was in the prompt, and the phrasing mattered more than I expected. Describing the shape to keep worked; describing the shape to avoid didn’t. The line that fixed it: “The head stays a round fluffy cloud with its spiral, exactly the same shape as in the first frame.”

Small things that cost me a regeneration each

Asking animate_image for “slow breathing” produces a character that closes its eyes and never opens them again. You have to state that the eyes stay open in every frame.

My four blink poses were prompted with “curved closed-eye lines, gentle smile.” That is the same eye drawing as the happy pose, so the two poses were indistinguishable — 2–4% apart. Regenerated with flat closed lines and a neutral mouth.

For a “change only X” edit on an existing sprite, edit_image beats a full regeneration, which redraws things you didn’t ask about (in my case the crest, every time).

I removed a stray droplet from one sprite’s forehead with a color-select filter. The filter left the white highlight and punched a 39px hole through the character’s head. Regenerating costs one API call. Patching a generative artifact by hand costs you a worse artifact.

And a genuinely stupid one: my frame-fetching script had the element name hardcoded. Fetching the water frames silently overwrote the fire frames, which I then had to re-download. The prefix is now the first positional argument and there is no default.

"""
The element is MANDATORY and prefixes every file written. An early version had
it hardcoded to "fire": fetching the water frames silently overwrote the fire
ones. Never go back to an implicit prefix.
"""

element = sys.argv[1]
if element not in ELEMENTS:
    sys.exit(f"unknown element: {element!r} (expected: {', '.join(ELEMENTS)})")

What it looks like now

Six scripts, everything reproducible from the raw generated output, no manual steps:

• process.py — raw output to aligned sprites, per-element tone curve, extracts the ”?” and “Zzz” decorations onto their own layers
• fetch_anim.py — pulls a job’s frames and re-registers them
• clean_anim.py — removes decoration the model baked into frames that already carry it on a layer, and sidelines bad frames
• qa.py — checks the four guarantees, plus internal holes and collisions between poses
• a contact-sheet builder and a manifest generator

One implementation detail I’d repeat anywhere: frames to discard get renamed to .skip, never deleted. Delete-and-renumber isn’t idempotent — the next frame inherits the freed number and gets destroyed on the following run. I found that out the way you’d expect.

# We RENAME them to .skip instead of deleting them: deleting and then
# renumbering isn't idempotent — a second run destroyed the following frame,
# which had taken the freed number.
SKIP = {"air/grab_01.png", "air/grab_02.png"}

for name in sorted(SKIP):
    path = os.path.join("anim", name)
    if os.path.exists(path):
        os.rename(path, path + ".skip")
        print(f"{name} dropped (renamed to .skip)")

The part I didn’t expect to be hard

None of the above was the actual problem.

Once the sprites worked, the creature was on screen all day, and it was exhausting. Not because the art was bad. Because it kept doing things.

The fix wasn’t slowing the animations down — that was my first attempt and it made things worse. A slow gesture isn’t restful, it just occupies the screen for longer. Measured, my “contemplative” setting produced 37% idle time against 68% for the faster one, which is precisely backwards from the intent.

What actually works is spacing. Same gesture speed, longer silences between gestures. The setting now multiplies the interval between actions rather than their duration, and the contemplative mode sits at 91% idle, 1.3 gestures per minute. Blinking turned out to matter too: a blink runs at 30ms per frame, and slowing it to 46ms turned it into a visible event happening fifteen times a minute, which is its own kind of awful.

91%

idle, contemplative mode

1.3

gestures per minute

30ms

per blink frame

91% doing nothing. That’s the number I’d have gotten most wrong if I’d guessed.

The creatures are for Tokimon, a desktop companion I’m building — the idea being that talking to an AI all day through a text field on a white page is a fairly bleak way to spend a career, and it doesn’t have to be. It isn’t finished. The art is, and this is what the art cost.

Happy to answer questions about any of the above.