Understanding the mAP (mean Average Precision) Evaluation Metric for Object Detection

In this tutorial, you will figure out how to use the mAP (mean Average Precision) metric to evaluate the performance of an object detection model. I will cover in detail what is mAP, how to calculate it, and give you an example of how I use it in my YOLOv3 implementation.

The mean average precision (mAP) or sometimes just referred to as AP. That's a popular metric used to measure the performance of models doing document/information retrieval and object detection tasks. So if from time to time you read new object detection papers, you may always see that authors compare mAP of their offered methods to the most popular ones.

Multiple deep learning object detection algorithms exist like RCNN's: Fast RCNN, Faster RCNN, YOLO, Mask RCNN, etc. All of these models solve two significant problems: Classification and Localization:

• Classification: Identify if an object is present in the image and the class of the object;
• Localization: Predict the coordinates of the bounding box around the object when an object is present in the image. Here we compare the coordinates of ground truth and predicted bounding boxes.

While measuring mAP, we need to evaluate the performance of both classifications and localization using bounding boxes in the image.

For object detection, we use the concept of Intersection over Union (IoU). IoU measures the overlap between 2 boundaries. We use that to estimate how much our predicted boundary overlaps with the ground truth (the actual object boundary):

• Red - ground truth bounding box;
• Green - predicted bounding box.

In simple terms, IoU tells us how well predicted and the ground truth bounding box overlap. You'll see that in code, we can set a threshold value for the IoU to determine if the object detection is valid or not.

For COCO, AP is the average over multiple IoU (the minimum IoU to consider a positive match). AP@[.5:.95] corresponds to the average AP for IoU from 0.5 to 0.95 with a step size of 0.05. For the COCO competition, AP is the average over 9 IoU levels on 80 categories (AP@[.50:.05:.95]: start from 0.5 to 0.95 with a step size of 0.05). The following are some other metrics collected for the COCO dataset:

And, because my tutorial series is related to the YOLOv3 object detector, here is AP results from the author's paper:

In the figure above, AP@.75 means the AP with IoU=0.75.

mAP (mean average precision) is the average of AP. In some contexts, we compute the AP for each class and average them. But in some context, they mean the same thing. For example, under the COCO context, there is no difference between AP and mAP. Here is the direct quote from COCO:

AP is averaged overall categories. Traditionally, this is called "mean average precision" (mAP). We make no distinction between AP and mAP (and likewise AR and mAR) and assume the difference is clear from the context.

Ok, let's get back to the beginning, where we need to calculate mAP. First, we need to set a threshold value for the IoU to determine if the object detection is valid or not. Let's say we set IoU to 0.5, in that case:

• If IoU ≥0.5, classify the object detection as True Positive(TP);
• If Iou <0.5, then it is a wrong detection and classifies it as False Positive(FP);
• When ground truth is present in the image, and the model fails to detect the object, we classify it as False Negative(FN);
• True Negative (TN): TN is every part of the image where we did not predict an object. This metrics is not helpful for object detection. Hence we ignore TN.

If we set the IoU threshold value to 0.5, then we'll calculate mAP50. If IoU=0.75, then we calculate mAP75. Sometimes we can see these as mAP@0.5 or mAP@0.75, but this is the same.

We use Precision and Recall as the metrics to evaluate the performance. Precision and Recall are calculated using true positives(TP), false positives(FP), and false negatives(FN):

To get mAP, we should calculate precision and recall for all the objects presented in the images.

It also needs to consider the confidence score for each object detected by the model in the image. Consider all of the predicted bounding boxes with a confidence score above a certain threshold. Bounding boxes above the threshold value are considered positive boxes, and all expected bounding boxes below the threshold value are considered harmful. So, the higher the confidence threshold is, the lower the mAP will be, but we'll be more confident with accuracy.

So, how to calculate general AP? It's pretty simple. For each query, we can calculate a corresponding AP. A user can have as many questions as they like against his labeled database. The mAP is simply the mean of all the queries that the use made.

To see, how we get an AP you can check the voc_ap function on my GitHub repository. When we have Precision(pre) and Recall(rec) lists, we use the following formula:

for i in list:
    ap += ((rec[i]-rec[i-1])*pre[i])

We should run this above function for all classes we use.

To calculate the general AP for the COCO dataset, we must loop the evaluation function for IoU[.50:.95] 9 times. Here is the formula from Wikipedia:

$mAP=\frac{1}{N}\sum _{i=1}^{N}A{P}_i$

Here N will be nine, and AP will be the sum of AP50, AP55, …, AP95. This may take a while to calculate these results, but this is how we need to calculate the mAP.

Practical YOLOv3 mAP implementation:

First, you should move to my YOLOv3 TensorFlow 2 implementation on GitHub. There is a file called evaluate_mAP.py. The complete evaluation is done in this script.

While writing this evaluation script, I focused on the COCO dataset to ensure it would work on it. So in this tutorial, I will explain how to run this code to evaluate the YOLOv3 model on the COCO dataset.

First, you should download the COCO validation dataset from the following link: http://images.cocodataset.org/zips/val2017.zip. Also, in the case for some reason you want to train the model on the COCO dataset, you can download and train the dataset: http://images.cocodataset.org/zips/train2017.zip. But it's already 20GB, and it would take a lot of time to retrain the model on the COCO dataset.

In TensorFlow-2.x-YOLOv3/model_data/coco/ is 3 files, coco.names, train2017.txt, and val2017.txt files. Here I already placed annotation files, that you won't need to twist your head where to get these files. Next, you should unzip the dataset file and place the val2017 folder in the same directory. It should look following: TensorFlow-2.x-YOLOv3/model_data/coco/val2017/images...

Ok, next we should change a few lines in our yolov3/configs.py:

• You should link TRAIN_CLASSES to 'model_data/coco/coco.names';
• If you wanna train on the COCO dataset, change TRAIN_ANNOT_PATH to 'model_data/coco/train2017.txt';
• To validate the model on the COCO dataset, change TEST_ANNOT_PATH to 'model_data/coco/val2017.txt';
• To change the input_size of the model, change YOLO_INPUT_SIZE and TEST_INPUT_SIZE to your need, for example, to 512.

Now we have all settings set for evaluation. Now I will explain the evaluation process in a few sentences. The whole evaluation process can be divided into three parts:

• In the first part, the script creates an mAP folder in the local directory, in which it establishes another ground-truth folder. Here it makes a .json file for every ground-truth image bounding box;
• In the second part, most part is done by our YOLOv3 model, and it runs predictions on every image. Similar way as in the first parts, it creates a .json file for every class we have and puts the detection bounding box accordingly;
• In the third part, we already have detected and ground-truth bounding boxes. We calculate the AP for each class with a voc_ap function. When we have AP of each class, we average it, and we receive the mAP.

Here is the output of evaluate_mAP.py script, when we call it with score_threshold=0.05 and iou_threshold=0.50 parameters:

81.852% = aeroplane AP  
12.615% = apple AP  
31.034% = backpack AP  
27.652% = banana AP  
60.591% = baseball-bat AP  
52.699% = baseball-glove AP  
91.895% = bear AP  
73.863% = bed AP  
43.250% = bench AP  
52.618% = bicycle AP  
47.743% = bird AP  
42.745% = boat AP  
23.295% = book AP  
52.714% = bottle AP  
55.977% = bowl AP  
31.840% = broccoli AP  
81.399% = bus AP  
49.839% = cake AP  
52.939% = car AP  
2.849% = carrot AP  
87.444% = cat AP  
46.828% = cell-phone AP  
52.618% = chair AP  
74.931% = clock AP  
57.715% = cow AP  
58.847% = cup AP  
47.931% = diningtable AP  
79.716% = dog AP  
24.626% = donut AP  
80.199% = elephant AP  
79.170% = fire-hydrant AP  
46.861% = fork AP  
82.098% = frisbee AP  
80.181% = giraffe AP  
4.545% = hair-drier AP  
27.715% = handbag AP  
79.503% = horse AP  
18.734% = hot-dog AP  
73.431% = keyboard AP  
39.522% = kite AP  
27.280% = knife AP  
75.999% = laptop AP  
69.728% = microwave AP  
67.991% = motorbike AP  
79.698% = mouse AP  
4.920% = orange AP  
58.388% = oven AP  
81.872% = parking-meter AP  
71.647% = person AP  
33.640% = pizza AP  
53.462% = pottedplant AP  
72.852% = refrigerator AP  
37.403% = remote AP  
23.826% = sandwich AP  
43.898% = scissors AP  
62.098% = sheep AP  
67.579% = sink AP  
73.147% = skateboard AP  
46.783% = skis AP  
56.955% = snowboard AP  
64.119% = sofa AP  
27.465% = spoon AP  
47.323% = sports-ball AP  
68.481% = stop-sign AP  
60.784% = suitcase AP  
64.247% = surfboard AP  
60.950% = teddy-bear AP  
78.744% = tennis-racket AP  
55.898% = tie AP  
31.905% = toaster AP  
83.340% = toilet AP  
39.502% = toothbrush AP  
44.792% = traffic-light AP  
88.871% = train AP  
48.374% = truck AP  
73.030% = tvmonitor AP  
67.164% = umbrella AP  
59.283% = vase AP  
54.945% = wine-glass AP  
84.508% = zebra AP  
mAP = 55.311%

Conclusion:

I did this tutorial because it's valuable to know how to calculate the mAP of your model. You can use this metric to check how accurate is your custom-trained model with the validation dataset. You can check how mAP changes when you add more images to your dataset, change threshold, or IoU parameters. This is mostly used when you want to squeeze as much as possible from your custom model. I thought about implementing mAP into the training process to track it on Tensorboard. Still, I couldn't find an effective way to do that, so if someone finds a way to do that effectively, I would accept pull requests on my GitHub, see you in the next tutorial part! That's it for this tutorial part.