The Effect of Improving Annotation Quality on Object Detection Datasets: A Preliminary Study

Jiaxin Ma OMRON SINIC X Corp.

Yoshitaka Ushiku OMRON SINIC X Corp.

Miori Sagara Baobab Inc.

Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR) Workshops, 2022, pp. 4850-4859

Download Paper

In this study, we partially reannotate conventional benchmark datasets for object detection and check whether there is performance improvement/drop compared with the original annotations.

Recent studies on the annotation qualities of ImageNet for image classification revealed some issues of how to associate only a single label to each image accurately. Object detection, on the other hand, should have other nontrivial issues because there are multiple objects in a single image, and realizing consistency among bounding boxes is challenging.

A team of professional annotators was formed for MS COCO and Google Open Images datasets. To realize highly-consistent annotations, we prepared carefully designed guidelines for each category and selected quality inspectors who checked the annotation quality of each annotator. Finally, we applied conventional object detection methods for reannotated parts of each dataset. We found mixed results: whether the performance dropped or improved depended on each category and dataset.

Request Dataset

If you would like to receive our dataset, please fill in the form below and we will deliver it to your inbox.

Our services

BAOBAB

BAOBAB is a website designed to create translations and linguistic data by cooperating as a group and using machine learning in order to broaden communication and services all over the world. BAOBAB

Learning Data for Machine Translation

Our company began as a service that could offer the extremely large volumes of textual learning data needed for machine translation faster and at a more reasonable price than anywhere else.

Image annotation and tagging/
voice data collection

  • Image annotation and tagging
  • Image captioning
  • Voice transcription

We have created a special in-house tool for image annotation, ensuring speedy and accurate results.

moringa

Using Moringa, a mobile app developed by BAOBAB, staff all around the world can collect and tag images, and collect multilingual speech utterances/sounds.

  • Moringa-i, an image collection and tagging tool
  • Moringa-v, a voice data collection and tagging tool
moringa

Creating bilingual scenarios
in multiple languages

  • Data for dialogue scenarios, read aloud by native speakers.
  • Simulated conversations between 2 speakers, speaking freely on a predetermined setting.

We create transcribed voice data and written transcriptions of the above.
Click here for sample dialogue scenarios

Developing machine translation engines specialised for particular areas

We develop machine translation engines specialised for particular areas, and provide them as an API. We undertake everything from the creation of learning data, to the development of machine translation engines, and human-powered evaluation of the resulting translations.
Example:

  • A machine translation engine that specialises in recipes (Japanese ⇄ English)
Yoko's Yummy Recipes (iOS / Android)

News
2022.07.02

Baobab celebrated its 12th year in business.

2022.06.27

Paper co-authored by Baobab accepted at CVPR 2022, the world's leading conference on computer vision

2022.06.22

Baobab to begin an annotation team leader training program for women refugees staying in Japan

2022.06.19

Sponsored the IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR 2022).

2022.06.17

Baobab speaking at the Digital Skills Workshop & Careers Fair for Ukrainian Refugees featured on NHK's "Good Morning Japan"

2022.06.14

Sponsored the 36th Annual Conference of the Japanese Society for Artificial Intelligence (JSAI).

2022.03.17

Baobab data sets used in a paper from Nippon Telegraph and Telephone Corporation that received the Special Committee Award at the 28th Annual Meeting of the Association of Natural Language Processing (NLP 2022).

2022.03.15

Makaira invests in Baobab

2022.03.14

Sponsored the 28th Annual Meeting of the Association of Natural Language Processing (NLP 2022).

2021.12.17

Baobab data sets used in a paper from Nagoya University that received the Specially Selected Paper Award from the Journal of Information Processing

2021.12.09

Sponsored the 2nd Symposium on Industrial applications of Artificial Intelligence (SIAI 2021).

2021.11.01

President Sagara Recognised at the TOKYO Women CEO Awards

2021.09.30

Baobab Wins Corporate Award at 2021 Forbes Japan Women Awards

2021.07.02

Baobab celebrated its 11th year in business.

2021.06.08

Sponsored the 35th Annual Conference of the Japanese Society for Artificial Intelligence (JSAI).

2021.06.07

An employment support centre for people with disabilities that Baobab entrusts with annotation work was featured in a special report on the NHK Aomori show, "Apple Wide"

2021.03.27

Baobab featured in Google for Startups' Founder Story

2021.03.19

Baobab Inc, raises JPY 50 million

2021.03.18

The University of Tokyo wins Award at NPL2021 for a paper utilizing Baobab's data sets

2020.11.30

Baobab carried out a round of fundraising.

2020.11.19

Baobab CEO Miori Sagara was nominated to speak at the Women's Entrepreneurship Day Event 2020 by Google.

2020.10.28

Baobab CEO Miori Sagara was invited to be a guest lecturer at Meiji University.

2020.07.02

Baobab celebrated its 10th year in business.

2020.06.25

NHK, Japan's national broadcasting organization featured Baobab Inc. on their program "Tokorosan, Good Heavens!", an educational show.

2020.02.17

We were selected for Google for Startups Accelerator.

2019.10.04

Our CEO Miori Sagara will speak at DLLAB Engineer Days.

2019.06.22

Became a sponsor, and ran a booth at the CVPR2019.

2019.05.30

Our CEO Miori Sagara spoke at a panel session at Microsoft's de:code2019 conference.

2019.03.15

We will become a sponsor, and run a booth at the CVPR2019.

2019.03.12

Chiba Institute of Technology, AIST and NEDO together release world's largest video caption data set

2018.10.11

We will run a booth at CEATEC JAPAN 2018.

2018.06.05

Sponsored the 32th Annual Conference of the Japanese Society for Artificial Intelligence (JSAI).

2018.05.30

We have established our U.S. corporation "Baobab America Inc.".

2018.03.12

We have started a new service "Region definition and feature point annotation for images ".

2018.03.12

Became a sponsor, and ran a booth at the 24th Annual Conference of Natural Language Processing (NLP2018).

2018.02.03

Chinese (Simplified) version of our website has launched.

2017.05.23

Sponsored the 31st Annual Conference of the Japanese Society for Artificial Intelligence (JSAI).

2017.04.01

We are pleased to announce the appointment of Dr Graham Neubig from the Carnegie Mellon University Language Technology Institute as our adviser on April 1st, 2017.

2017.02.01

We have moved to a new office.

2016.12.11

We released image/voice data collection and tagging tool "moringa".

2016.07.10

Sponsoring the 26th International Conference on Computational Linguistics (Coling 2016).

2016.03.07

Sponsored the 22nd Annual Conference of Natural Language Processing (NLP2016).

2015.12.22

Baobab CEO Miori Sagara was appointed as a delegate of the Association for Natural Language Processing.

2015.07.27

Sponsored the 53rd Annual Meeting of the Association for Computational Linguistics (ACL2015).

2015.06.13

Baobab CEO Miori Sagara participated in the workshop at National Institute of Informatics as a guest speaker.

2015.04.01

Began an Image Gathering and Annotation Service

2014.08.23

Sponsored the 25th International Conference on Computational Linguistics (Coling 2014)

2014.07.15

Participated in "From Minato-ku! Latest Global Business Seminar" as a guest speaker.

2014.07.09

Aided in the 25th International Conference on Computational Linguistics (Coling 2014)

2014.06.09

Release of the updated version of “Yoko’s Yummy Recipes” An iPhone/Android translation application designed specifically for translating recipes

2013.11.27

Worked as a guest lecturer at Meiji University.

2013.03.12

Aided with the 19th Annual Language Processing Academic Convention (NLP2013)