Using face detection to curate image collections

When you have a folder of thousands of mixed images, the first question is often simple: which ones have people in them, and how many? A small face detector answers that quickly, right in your browser. It is a blunt instrument, though. Know what it does and where it fails, or you can end up with a dataset quietly skewed by the images it missed.

This guide explains how small face detectors work, what their two main settings trade off, and how to use face counts and expressions to curate photo collections and training sets. The examples use ImgTagger's face and expression sorter, which runs face-api.js in the browser, but the ideas apply to any detector.

Detection is not recognition

These terms often get mixed up, but the difference matters.

Recognition turns each face into a list of numbers (face-api.js uses 128) and measures how close two lists are. face-api.js ships such a model, but ImgTagger never loads it. The face sorter can tell you an image contains two faces. It cannot tell you whether the same person appears in two different images.

How a tiny face detector works

ImgTagger uses face-api.js's TinyFaceDetector. The face-api.js README describes it as an even smaller take on the Tiny YOLO v2 object detector. It uses depthwise separable convolutions to stay light, weighs about 190 KB once quantized, and was trained on roughly 14,000 images labeled with face boxes. Each image goes through four steps:

1. Resize. The image is scaled so its longer side equals the input size, then padded to a square. face-api.js requires a size divisible by 32. ImgTagger's slider runs from 128 to 768 in steps of 64.
2. Predict on a grid. The network divides the square into a grid of cells. Each cell proposes a few candidate boxes, and each box gets a confidence score between 0 and 1.
3. Threshold. Candidates scoring below the threshold are dropped.
4. Remove duplicates. Neighboring cells often find the same face. A step called non-maximum suppression keeps the highest-scoring box and drops boxes that overlap it heavily.

What's left is a list of boxes, most confident first. The face count is simply the length of that list. With expression analysis on, each box is cropped and passed to a second network of about 310 KB, which returns probabilities for seven expressions: neutral, happy, sad, angry, fearful, disgusted and surprised. Those seven numbers always add up to 1.

Score threshold vs input size

Both settings move you along the same trade-off between missed faces and false alarms, but in different ways.

Input size: how small a face can be

Because the image is scaled to the input size, what matters is how many pixels a face covers after scaling. Take a face 120 pixels wide in a 4000×3000 photo, which is typical of a group shot from a modern camera:

A face of 4 to 8 pixels is a smudge, and the README notes that this detector does slightly worse on small faces than the library's larger SSD MobileNet detector. Larger input sizes help, but the work grows with the square of the size: 768 means about 2.25 times as many pixels as 512. In a typical 1024-pixel training image with one main subject, the face is already large, and 256 or 320 is often enough.

Score threshold: how sure is sure enough

A low threshold keeps more real faces but lets in false positives, such as patterned fabric or a statue. A high threshold gives a cleaner Faces bucket but sends more real faces, especially small or turned ones, to No Faces. face-api.js defaults to 0.5. ImgTagger starts lower, at 0.3, which leans toward catching more faces. That is often the right bias for curation: a false positive in a few hundred face images is easy to spot, while a missed face buried among thousands of no-face images is not.

ImgTagger detects once at 0.1, keeps every candidate, and re-filters when you move the slider, so you can sweep the threshold without rescanning. To calibrate it:

1. Load a sample of 50–100 images that represents the whole collection.
2. At the default 0.3, look through both buckets and note the misses and false positives.
3. Step the threshold by 0.05 until the error you care about most is rare.
4. Use that value for the full run, and raise the input size only if small faces are the problem.

Curation recipes

These recipes assume you've already removed duplicates (see how perceptual hashing finds near-duplicates). One quirk of ImgTagger's sorter shapes them: a filter chip only hides images, and hidden images that are still checked go into the download. The bucket's header checkbox, on the other hand, only changes the images on screen. So clear the whole bucket first, then let a filter decide what you tick.

Keep single-subject portraits

In Faces Detected, leave only the 1 face-count chip on. That shows images with exactly one detected face, a good first cut for a person-focused training set. Be aware that a background crowd too small to detect still counts as "1", and so does a portrait with a second, partly hidden face. Skim the results with face boxes on. For illustrated images, the auto tagger's solo and portrait tags make a useful cross-check.

Remove crowd and group shots

Leave only the 4 and 5+ chips on, untick the bucket's header checkbox to deselect what's showing, then turn the other chips back on. Every image with four or more detected faces is now out of the download. Raise the input size for this pass. Crowd faces are usually small, and an undercounted crowd slips into your "2" or "3" group.

Separate people from everything else

The No Faces Detected bucket quickly gathers landscapes, objects and product shots. Don't trust it blindly: people seen from behind, far away or in profile often land there too. For this pass, lower the threshold to 0.2 and check which images move across.

Balance expressions

Many collections are dominated by neutral or smiling faces, and a model trained on them can lean the same way. Turn on emotion tagging, reprocess, and click each expression chip in turn to count the labels. If one dominates, keep more of the others, or caption expressions so they can be controlled at generation time. Our guide to LoRA dataset captions covers that side.

Two details shape the results. First, with every expression slider at its default of 0.60, each face can carry at most one label, because the seven probabilities sum to 1. Second, labels come from a single face per image, the one the detector was most confident about. A group shot is labeled by one person's expression.

Illustrations, anime and stylized faces

face-api.js was built for photographs of real people. Drawn faces break many of the cues a photo-trained detector relies on: anime faces often have very large eyes, tiny noses and mouths, and flat shading. Results on illustrations are inconsistent and depend heavily on the art style; the closer a style is to realistic proportions, the better your odds. Expression scores on drawn faces are even less reliable.

For anime and illustration collections, a tagger trained on that kind of art is usually the better tool. WD14 models predict count tags such as solo, multiple_girls and no_humans, framing tags like portrait and upper_body, and expression tags such as smile. You can filter on any of them in the tagger's filter panel. Our guide to Danbooru tags explains how the count tags work.

Bias and fairness

A face detector's error rate is not the same for every face. Every training set covers some faces better than others, and lighting, image quality, age, skin tone, glasses, head coverings and makeup can all change how reliably a face is found. The face-api.js README itself warns that glasses may reduce expression accuracy. Uneven error rates are common across face analysis: NIST's 2019 report on demographic effects (NISTIR 8280) tested 189 face recognition algorithms, not detectors, and found error rates that differed by demographic group in the majority of them.

For curation, the consequence is concrete: a detector's misses are not random, so a filtered dataset can be skewed in ways you never chose. If one group's faces are missed more often, filtering to Faces Detected quietly under-represents that group. Two habits help:

• Spot-check the No Faces bucket, not just the Faces bucket, before you discard anything.
• Compare the makeup of your collection before and after filtering. If one kind of image shrinks much more than the rest, review it by hand.

Expression labels deserve extra caution. The model labels facial movements, which vary between people and cultures and don't reliably reveal what someone feels. Use the labels to describe how an image looks, never to draw conclusions about a person.

Ethics and privacy

• Have the right to use the images. Photos of real people carry privacy and image rights. Training a model on someone's face without permission can harm them even if you never publish the images.
• Think about where personal photos go. Any tool that handles photos of people creates copies somewhere, so choose tools whose data practices you're comfortable with. ImgTagger's face sorter runs detection in your browser. Our privacy policy describes what data we may collect, and our guide to how in-browser AI works explains the technology.
• Detect, don't identify. Use face counts to organize images, not to track, single out or profile people.
• Know the rules where you are. Some laws restrict facial analysis. Since February 2025, for example, the EU's AI Act has prohibited using AI to infer people's emotions in workplaces and educational institutions, except for medical or safety reasons.

Face filtering is one step in a larger cleanup. The image dataset workflow shows where it fits between removing duplicates, fixing color and tagging.