Object detection explained: what it is and what it returns
Object detection is the automated location and labelling of things inside an image, returning for each item a bounding box, a class label, and a confidence score. It answers two questions at once, where something is and what class it belongs to, which is what separates it from plain image classification.
Classification tells you an image contains a wheel. Detection tells you there are three wheel-shaped regions and marks them. Segmentation, a third relative, outlines the object pixel by pixel instead of drawing a box.
How does object detection work?
A detector proposes candidate regions across the image, scores each region against every class it was trained on, and suppresses overlapping duplicates so one object yields one box. What it can find is bounded absolutely by its training classes. A model trained on 80 everyday object categories will never report a hairline weld crack, because "hairline weld crack" is not in its vocabulary. It will report the nearest thing it does know, or report nothing.
What the confidence score actually is
This is the point where most evaluations go wrong, so it is worth stating plainly: a confidence score is not a probability that the detection is correct. It is the model's output activation for that class, normally the top value of a softmax, and it has no guaranteed relationship to real-world correctness.
Guo and colleagues demonstrated this at ICML in 2017, in a paper titled On Calibration of Modern Neural Networks. Their finding was that modern networks are accurate yet systematically overconfident, a property they called mis-calibration, and that the maximum softmax value does not represent a true probability of correctness. Older, smaller networks were better calibrated than the newer, more accurate ones.
What that means operationally:
- 0.94 does not mean 94 percent right. Across a batch of detections scored at 0.94, the share that are actually correct can be considerably lower.
- Scores are not comparable across models. A 0.7 from one detector and a 0.7 from another mean different things.
- Scores are not comparable across conditions. The same model on darker, blurrier, or unusual-angle images stays confident while becoming less accurate. Confidence does not fall to warn you.
- Thresholds must be calibrated on your own data. A threshold tuned on the vendor's benchmark tells you nothing about your images.
How is object detection accuracy measured?
The standard benchmark metric is mean average precision, or mAP, computed on the COCO dataset across intersection-over-union thresholds from 0.50 to 0.95. Intersection over union measures how well the predicted box overlaps the true box, so a detection can be counted wrong purely for being poorly positioned even when the label is right.
The headline number is a useful sanity check on expectations. Leading detectors on COCO report mAP in the region of 60, and accuracy is consistently worse for small objects than for large ones. This is the state of the art on a curated benchmark of everyday objects photographed reasonably well. It is not a promise about your plant room.
Object detection: a practical example
A detector is pointed at a photo of a damaged bumper. It returns "car, 0.98" and "wheel, 0.91" and no box at all around the crack, because damage was never a training class. A reviewer skimming the output sees two high-confidence detections and reads the image as processed. Nothing in the output says "I was not asked to look for damage."
Absence of a detection is the single most misread signal in this field. It means the model did not find something it recognises. It does not mean the thing is not there.
When is object detection the right tool?
It works well as a routing and triage aid, not as a decision-maker. Good uses look like counting items on a pallet, flagging that no vehicle appears in a photo that should contain one, or pre-sorting a queue so a reviewer sees the likely-relevant images first. In every one of those, a wrong answer costs a small amount of wasted attention.
Bad uses look like approving a claim, closing a job, or triggering a payment on a label and a score alone. The correct framing for anything consequential is that a detection is a reason to look, never a verdict. The same discipline applies to any automated control point: it points a human at something, and the human decides. Detection also sits downstream of capture quality, so pairing it with image quality validation and guided capture raises its hit rate more reliably than swapping the model does.