notebooks/evaluation_tap_vit_h_v1_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation Guide for TAP-H\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "We provide three simple scripts to evaluate **TAP** on ***Instance Segmentation***, ***Instance Classification*** and ***Region Caption*** tasks respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Necessary datasets, imports and models for evaluation.\n",
    "\n",
    "```\n",
    "datasets\n",
    "|_ coco\n",
    "|  |_ train2017\n",
    "|  |  |_ 000000000009.jpg\n",
    "|  |  |_ ...\n",
    "|  |_ val2017\n",
    "|  |  |_ 000000000139.jpg\n",
    "|  |  |_ ...\n",
    "|  |_ annotations\n",
    "|  |  |_ coco_instances_val2017.json\n",
    "|  |_ results\n",
    "|  |  |_ coco_instances_val2017_vitdet_h_cascade.json  # Run detectron2 to generate this file.\n",
    "|- lvis\n",
    "|  |_ annotations\n",
    "|  |  |_ lvis_val_v1.json\n",
    "|  |_ results\n",
    "|  |  |_ lvis_val_v1_vitdet_h_cascade.json  # Run detectron2 to generate this file.\n",
    "|_ vg\n",
    "|  |_ images\n",
    "|  |  |_ 1.jpg\n",
    "|  |  |_ ...\n",
    "|  |_ annotations\n",
    "|  |  |_ test.json  # https://datarelease.blob.core.windows.net/grit/VG_preprocessed_annotations/test.json\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(\"..\")\n",
    "\n",
    "class AttrDict(dict):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super(AttrDict, self).__init__(*args, **kwargs)\n",
    "        self.__dict__ = self\n",
    "\n",
    "# global arguments.\n",
    "args = AttrDict(read_every=100, prompt_size=256)\n",
    "args.model_type = \"tap_vit_h\"\n",
    "args.checkpoint = \"../models/tap_vit_h_v1_1.pkl\"\n",
    "args.device = [0, 1, 2, 3, 4, 5, 6, 7]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation: Instance Segmentation on COCO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "92857 instances in 5000 images.\n",
      "im_process: 5000/5000 [0.137s + 0.014s] (eta: 0:00:00)\n",
      "Writing segmentations to /data/workspace/models/tokenize-anything/scripts/../outputs/coco_segmentations.json\n",
      "\n",
      "Evaluating COCO segmentations...\n",
      "loading annotations into memory...\n",
      "Done (t=2.18s)\n",
      "creating index...\n",
      "index created!\n",
      "Loading and preparing results...\n",
      "DONE (t=4.52s)\n",
      "creating index...\n",
      "index created!\n",
      "Running per image evaluation...\n",
      "Evaluate annotation type *segm*\n",
      "DONE (t=36.01s).\n",
      "Accumulating evaluation results...\n",
      "DONE (t=5.46s).\n",
      "Summary:\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.463\n",
      " Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.720\n",
      " Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.500\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.295\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.510\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.626\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.347\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.551\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.574\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.432\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.628\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.705\n"
     ]
    }
   ],
   "source": [
    "from scripts.eval_seg import main\n",
    "\n",
    "args.images_dir = \"../datasets/coco/val2017\"\n",
    "args.gt_json_file = \"../datasets/coco/annotations/coco_instances_val2017.json\"\n",
    "args.det_json_file = \"../datasets/coco/results/coco_instances_val2017_vitdet_h_cascade.json\"\n",
    "main(args)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation: Instance Segmentation on LVIS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3293288 instances in 19809 images.\n",
      "im_process: 19809/19809 [0.267s + 0.156s] (eta: 0:00:00)\n",
      "Writing segmentations to /data/workspace/models/tokenize-anything/scripts/../outputs/lvis_segmentations.json\n",
      "\n",
      "Evaluating LVIS segmentations...\n",
      "Summary:\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=all] = 0.432\n",
      " Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=300 catIds=all] = 0.613\n",
      " Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=300 catIds=all] = 0.463\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=     s | maxDets=300 catIds=all] = 0.302\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=     m | maxDets=300 catIds=all] = 0.560\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=     l | maxDets=300 catIds=all] = 0.654\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  r] = 0.337\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  c] = 0.441\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  f] = 0.463\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=all] = 0.519\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=     s | maxDets=300 catIds=all] = 0.375\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=     m | maxDets=300 catIds=all] = 0.648\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=     l | maxDets=300 catIds=all] = 0.714\n"
     ]
    }
   ],
   "source": [
    "from scripts.eval_seg import main\n",
    "\n",
    "args.images_dir = \"../datasets/coco/val2017\"\n",
    "args.gt_json_file = \"../datasets/lvis/annotations/lvis_val_v1.json\"\n",
    "args.det_json_file = \"../datasets/lvis/results/lvis_val_v1_vitdet_h_cascade.json\"\n",
    "main(args)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation: Instance Classification on LVIS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "244707 instances in 19809 images.\n",
      "im_process: 19809/19809 [0.200s + 0.021s] (eta: 0:00:00)\n",
      "Writing detections to /data/workspace/models/tokenize-anything/scripts/../outputs/lvis_detections.json\n",
      "\n",
      "Evaluating LVIS detections...\n",
      "Summary:\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=all] = 0.596\n",
      " Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=300 catIds=all] = 0.602\n",
      " Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=300 catIds=all] = 0.596\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=     s | maxDets=300 catIds=all] = 0.456\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=     m | maxDets=300 catIds=all] = 0.722\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=     l | maxDets=300 catIds=all] = 0.830\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  r] = 0.616\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  c] = 0.600\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=  f] = 0.582\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 catIds=all] = 0.670\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=     s | maxDets=300 catIds=all] = 0.518\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=     m | maxDets=300 catIds=all] = 0.783\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=     l | maxDets=300 catIds=all] = 0.890\n"
     ]
    }
   ],
   "source": [
    "from scripts.eval_cls import main\n",
    "\n",
    "args.images_dir = \"../datasets/coco/val2017\"\n",
    "args.gt_json_file = \"../datasets/lvis/annotations/lvis_val_v1.json\"\n",
    "args.concept = \"../concepts/lvis_1203.pkl\"\n",
    "args.max_dets = 300\n",
    "main(args)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation: Region Caption on Visual Genome"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "232935 instances in 5000 images.\n",
      "im_process: 5000/5000 [0.241s] (eta: 0:00:00)\n",
      "Evaluating captions...\n",
      "Bleu [0.3766071779004523, 0.23749450923369872, 0.1669849283973828, 0.12375479806651177]\n",
      "METEOR 0.17804049451301138\n",
      "Rouge 0.3620181333217273\n",
      "CIDEr 1.5423605268978664\n"
     ]
    }
   ],
   "source": [
    "from scripts.eval_cap import main\n",
    "\n",
    "args.images_dir = \"../datasets/vg/images\"\n",
    "args.gt_json_file = \"../datasets/vg/annotations/test.json\"\n",
    "main(args)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}