A practical workflow for cleaning an image dataset

A folder of images collected from different places isn't a dataset yet. It has copies, off-topic pictures, odd color casts and no captions. The four ImgTagger tools can fix all of that in your browser, but the order matters: each step changes what the next one sees, and some, like re-encoding, can't be undone.

Below is the order we recommend, why each stage sits where it does, and what to check along the way.

The workflow at a glance

The logic: steps that remove images come before steps that work on them, and steps that change pixels come before the step that describes them.

Set up folders and keep the originals

First, copy the raw images into a folder you never edit, and back it up elsewhere too. Every ImgTagger tool leaves your files alone and returns results as a new ZIP, so a numbered layout keeps the stages separate:

my-dataset/
  00-originals/    untouched, also backed up elsewhere
  01-deduped/      keepers from the duplicate finder
  02-filtered/     after face or expression sorting
  03-color/        after color correction, if any
  04-captioned/    images + .txt files from the tagger

Keeping every stage lets you step back: if the color pass was a mistake, restart from 02-filtered.

Step 1: Gather

Collect more than you need, since later stages only remove images. While gathering:

• Use formats your browser can open. JPEG, PNG and WebP are safe. Phone photos are often HEIC, which many browsers can't display; convert those first.
• Get the largest, most original version. A thumbnail or re-posted copy has already lost detail, and the duplicate finder can only pick from what you collected.
• Note each image's source if licensing or credit matters. Nothing later records it.

Step 2: Remove exact and near duplicates

Deduplicate first because it's the cheapest filter and it shrinks every later stage. It also protects training: an image that appears three times is learned three times, which pulls a model toward that one picture.

Load the working folder into the duplicate image finder. Byte-identical files are dropped automatically, and the rest are grouped by visual similarity. The default DCT mode at its default threshold of 3 catches re-saves and resized copies. If you also want lightly edited copies, raise it one step at a time; if it groups different pictures, drop to 2. Our guide to how perceptual hashing works explains what that threshold measures.

What to check: in each group, keep the copy with the largest pixel size and no watermark or border. Decide on purpose about near-duplicates such as alternate expressions. One or two can add variety, but a run of near-identical frames over-weights one pose and background. The finder can't catch mirrored copies, so look for those by eye.

Common mistakes:

• Raising the threshold too far without looking. From about 4 or 5 upward, unrelated images with a similar layout start to group.
• Keeping the first image of each group by habit. It's just the earliest file.
• Deduplicating after color correction. You've corrected copies you'll delete, and once every file has been re-encoded, file size no longer tells you which copy was the best.

Step 3: Filter by faces or expressions

If your dataset is about a person or character, the next question is whether each image actually shows them. The face and expression sorter splits a batch into "Faces Detected" and "No Faces Detected" and lets you filter by face count (1, 2, 3, 4 or 5+). It can also score seven expressions: neutral, happy, sad, angry, surprised, fearful and disgusted.

It runs after deduplication because face detection runs a neural network on every image, and before color and tagging so you don't spend effort on images you'll drop.

What to check: the "No Faces Detected" group starts unchecked, so those images are left out of the download unless you select them. Look through it anyway. Back views, profiles, small faces in wide shots and drawn or stylized faces are easy to miss, and those shots often give a dataset its variety. For a single-subject set, use the face-count filter to find group shots. Our guide to curating images with face detection covers the input size and score threshold settings.

When to skip it: for style, scenery, object or architecture datasets, faces don't matter.

Common mistakes: treating "no face detected" as "no person", and filtering on expressions so tightly that every image has the same smile, which teaches the model a single expression.

Step 4: Even out color, or leave it alone

Color correction fixes faults that would otherwise be learned, such as a yellow cast from indoor lighting or a washed-out scan. The batch color fixer applies white balance, histogram normalization, brightness, contrast, saturation and gamma to every image at once, with per-image overrides.

It comes after filtering, since there's no point correcting images you'll delete. It comes before tagging because the tagger describes the pixels it sees. Hair and eye colors, monochrome and greyscale all come from the image, so a strong cast can change the tags you get.

For a style dataset, skip this stage. An artist's palette or a deliberate tint is the style, and "fixing" it teaches the model a look the artist never used. Be careful with Auto white balance on images dominated by one color: it uses the gray-world idea, scaling red, green and blue until their averages match, so a character in a blue outfit against a blue sky gets pushed toward gray.

Histogram normalization is gentler on illustrations than it sounds. It stretches each color channel from its darkest to its lightest value, and an image with black line art on a white background already spans the full range, so it changes little or nothing. The white balance and histogram guide covers both methods and lists the images to leave alone.

What to check: the fixer re-encodes every image you loaded when it exports. JPEGs come back with a .jpeg extension, WebP stays WebP, and everything else is saved as PNG. Base filenames are kept, so captions still pair up later. But each JPEG goes through another round of lossy compression, so correct once, from the cleanest copies.

Common mistakes: applying one automatic correction to a mixed batch without checking individual results, and correcting a style dataset into blandness.

Step 5: Auto-tag and review

With the final images in place, run the auto tagger. It runs a WD14 model in your browser and writes Danbooru-style tags for each image, the most confident ones first. The default confidence threshold of 0.35 is a reasonable place to start.

Tagging comes this late for two reasons. Tags describe pixels, so they should come from the final pixels. And every image runs through a model of several hundred megabytes, the slowest step per image in this workflow, so it should run on the smallest set.

What to review:

• Count tags. solo, 1girl, multiple_girls and no_humans carry a lot of weight in a caption, so make sure each one matches its image.
• Quality flags. Use the Filter panel to pull up images tagged watermark, signature or blurry. Each one is a reason to drop or replace an image, not just a tag to delete. ImgTagger only outputs blurry at 0.9 confidence or higher, so it flags clear cases only.
• Resolution. The models can't output lowres or jpeg_artifacts, so small or heavily compressed images won't flag themselves. Check the pixel size shown in the preview pane.
• Rare tags. Sort the filter list from low to high count. A tag on only one or two images often points to a wrong guess or an off-topic picture.

Step 6: Prune tags and export captions

Raw tagger output is a starting point. Pruning makes captions consistent and steers what the model learns:

• Drop tags for things that never change. In a character set, many trainers remove tags such as a fixed hair or eye color so the trigger word absorbs them. It's a common convention with real trade-offs, covered in our tag pruning guide.
• Add a trigger word with the prepend option, so it comes first in every caption.
• Add tags you never want to the "Never add these tags" box, such as the rating tags if you don't use them.
• Fix batch-wide mistakes in one click from the Filter panel or the post-run summary.

Then download a ZIP with the images and their .txt files. Each caption is saved under the image's base name, with tags separated by a comma and a space. The guide to caption .txt files covers what training tools expect.

Common mistakes: pruning so hard that visibly different images get identical captions, and removing or renaming images after export without checking that every image still has its .txt file.

How many images, and how good?

There's no single right dataset size. It depends on what you're training, the base model and your settings, and published advice varies widely. A few principles hold regardless:

• Quality beats quantity. Every image teaches something, including blur, watermarks and off-model drawings, and in a small set each one carries more weight.
• Vary what should vary. For a character, different poses, angles, outfits, backgrounds and lighting help the model separate the character from the setting. If every image has the same background, the model may treat it as part of the character.
• Keep constant what shouldn't vary. For a style, every image should really be in that style. A few off-style images dilute it.
• Watch the balance. If most images share one outfit, expect the model to default to it. Tag counts in the Filter panel make imbalances easy to spot.
• Mind the pixels. Images much smaller than your training resolution get upscaled and look soft, so dropping them is usually better.

When in doubt, start with a smaller, carefully checked set, train, study the results, and add images that fix specific weaknesses.

Checklist

Before you start

• Originals copied to a folder you never edit, and backed up elsewhere
• Filenames unique across all sources
• Every file in a format your browser can open

Duplicates

• Exact and visual duplicates removed, starting from a strict threshold
• Best copy kept in each group: largest size, no watermark or border
• Mirrored copies checked by eye

Faces (people and character sets only)

• "No Faces Detected" group reviewed for back views, profiles and small faces
• Group shots removed or kept on purpose

Color

• Corrected only where color is a fault, not a style
• Individual results checked after any automatic correction

Tags and captions

• Count tags correct on every image
• Images tagged watermark, signature or blurry reviewed
• Small images found and dropped or replaced
• Trigger word prepended, unwanted tags blacklisted or removed
• Every image in the export has a matching .txt file