\ud83d\udd11<\/strong> In 10 minutes you will learn:<\/strong><\/span><\/p>\n [\/vc_column_text]<\/div><\/div> This article is based on a speech given by David Kim, founder of DATUMO<\/strong>.[\/vc_column_text]<\/div><\/div>[\/vc_column][\/vc_row][vc_section full_width=”stretch_row” pix_over_visibility=”” css=”.vc_custom_1650444445523{padding-top: 80px !important;padding-bottom: 80px !important;background-color: #f8f9fa !important;}” el_id=”pix_section_program”][vc_row full_width=”stretch_row” pix_particles_check=””][vc_column content_align=”text-center” offset=”vc_col-lg-offset-0 vc_col-lg-12 vc_col-md-offset-1 vc_col-md-10″] AI models achieve intelligence from the manually annotated data. We call the annotated data \u201ctraining data.\u201d<\/p>\n AI models and training data are deeply implemented in our daily lives. For instance, automated vehicles are trained with manually annotated data to detect and recognize objects.<\/p>\n Now it\u2019s all about Data-centric AI.<\/p>\n [\/vc_column_text] The trend has changed from focusing on AI models to focusing on quality datasets.<\/p>\n -From model-centric to Data-centric AI<\/em><\/p>\n <\/p>\n What is \u201cquality data\u201d?<\/p>\n David points out four main characteristics of quality data.[\/vc_column_text] <\/p>\n <\/p>\n <\/p>\n Since the beginning of foundation, Datumo had a dedicated <User Guidelines Team> that solely focuses on writing easy and clear guidelines for the annotators.<\/p>\n <\/p>\n <\/p>\n <\/p>\n Easy and clear guidelines enhance the work efficiency.[\/vc_column_text] [\/vc_column_text]<\/div><\/div>[vc_column_text css=”.vc_custom_1653555277227{border-bottom-width: 0px !important;padding-bottom: 0px !important;border-bottom-color: rgba(0,0,0,0.2) !important;border-bottom-style: solid !important;}”]<\/p>\n <\/p>\n One of Datumo\u2019s methods for quality control is the exam system. All crowd-workers are required to pass complex exams based on guidelines in order to participate in data annotation.<\/p>\n <\/p>\n <\/p>\n Experienced crowd-workers and in-house inspectors cross-validate the annotated dataset to filter inaccurate data. Furthermore, Datumo is the only<\/span> platform in Korea to run strict validation based on the reliability of the inspectors.<\/p>\n \u2192 Highly dependent on the accuracy and attentiveness of the inspector[\/vc_column_text][vc_column_text css=”.vc_custom_1653659221558{border-bottom-width: 1px !important;padding-bottom: 40px !important;border-bottom-color: rgba(0,0,0,0.2) !important;border-bottom-style: solid !important;}”]Cross-validation based on majority rule<\/strong><\/p>\n \u2192 Inspectors of low accuracy may outnumber those of high accuracy and thus, spoil data validation[\/vc_column_text][vc_column_text css=”.vc_custom_1653659462389{border-bottom-width: 0px !important;padding-bottom: 0px !important;border-bottom-color: rgba(0,0,0,0.2) !important;}”]Data validation based on inspector reliability inference algorithm is more accurate than single inspection or cross-validation based on majority rule.[\/vc_column_text][vc_separator][vc_column_text css=”.vc_custom_1653659434318{border-bottom-width: 0px !important;padding-bottom: 0px !important;border-bottom-color: rgba(0,0,0,0.2) !important;}”]Datumo is the\u00a0only one<\/b>\u00a0to carry out validation based on inspector reliability.[\/vc_column_text]<\/div><\/div>[vc_column_text css=”.vc_custom_1651509922020{border-bottom-width: 1px !important;padding-bottom: 40px !important;border-bottom-color: rgba(0,0,0,0.2) !important;border-bottom-style: solid !important;}”][\/vc_column_text] <\/p>\n Datumo focuses on achieving consistent data by making sure the guidelines leave no room for ambiguity. For instance, when collecting or labeling dataset related to speech or emotion, we try to set standards in our guidelines so that everything falls under a certain category in order to avoid subjective interpretations depending on the annotator.[\/vc_column_text][vc_column_text]<\/p>\n <\/p>\n <\/p>\n <\/p>\n One of the most common issues in bounding box datasets is different box sizes depending on the annotator. As shown in the image above, without a specific guideline, Box 1 and Box 2 could be either correct of wrong depending on the inspector\u2019s perspective. In order to avoid such situation, Datumo came up with a solution: an inner guiding box(patented). An inner guiding box, which is Datumo\u2019s own-developed UI system, provides standards for both drawing and validating bounding boxes around the target object.[\/vc_column_text] <\/p>\n It is important to expand the \u201ccoverage\u201d of datasets by considering the environment of collecting data when planning the whole process.<\/p>\n <\/p>\n <\/p>\n <\/p>\n The essence of expanding coverage is to provide AI models with a variety of cases that AI could face in the real world.<\/p>\n For facial image datasets, Datumo collected 1,100 crowd-workers through our own developed crowd-sourcing platform Cash Mission and hypothesized 3 different lightings, 8 different situations, and 11 different angles, which resulted in 264 different cases.[\/vc_column_text][vc_column_text css=”.vc_custom_1653383169742{border-bottom-width: 0px !important;padding-bottom: 0px !important;border-bottom-color: rgba(0,0,0,0.2) !important;}”] <\/p>\n Similar, redundant data reduce the value of the whole dataset. Due to its quantity, it is almost impossible to manually filter such data. This is why we came up with our own data filtering system based on machine learning. As the only ones to implement such system, Datumo developed an AI model that automatically detects three image data that look similar to the newly submitted image. Then, the inspector manually validates whether the newly submitted image is redundant or not.[\/vc_column_text][vc_column_text css=”.vc_custom_1653554976343{border-bottom-width: 0px !important;padding-bottom: 80px !important;border-bottom-color: rgba(0,0,0,0.2) !important;}”] <\/p>\n For a text data collecting project regarding automobile issue reports, we wanted to avoid ending up with just a few most common issues reported repeatedly. Thus, we specified the types of issues into five different categories- inoperable\/visual\/auditory\/tactile\/olfactory – and made sure to collect similar amount of data for each category.[\/vc_column_text][vc_column_text]As a result, we were able to minimize data imbalance and provide our client with the variety of data they have asked for.<\/p>\n <\/p>\n <\/p>\n Based on such effort and technology, Datumo strives to make \u201cgood data.\u201d<\/p>\n Data quality was the core element of the business since foundation. We will continue to optimize and improve our technology and system to construct a smoother flywheel. In Data-centric perspective, consistent and high-quality data are of greatest value.<\/p>\n Datumo aims to drive impact in the AI industry by innovating the system of collecting AI training data, one of the worst bottlenecks in the industry, through revolutionary technology.[\/vc_column_text]\n
What <\/span><\/span> is <\/span><\/span> \u201cquality <\/span><\/span> data\u201d? <\/span><\/span> How <\/span><\/span> do <\/span><\/span> you <\/span><\/span> make <\/span><\/span> quality <\/span><\/span> data? <\/span><\/span> <\/h2><\/div>[vc_column_text css=”.vc_custom_1653382331843{border-bottom-width: 0px !important;padding-top: 40px !important;padding-bottom: 40px !important;}”]<\/p>\n
![\"\"](\"http:\/\/blog.datumo.com\/en\/wp-content\/uploads\/2020\/02\/samuele-errico-piccarini-MyjVReZ5GLQ-unsplash.jpg\")
<\/p>\n\u201cWithout <\/span><\/span> training <\/span><\/span> data, <\/span><\/span> there <\/span><\/span> would <\/span><\/span> be <\/span><\/span> no <\/span><\/span> AI <\/span><\/span> and <\/span><\/span> without <\/span><\/span> quality <\/span><\/span> data, <\/span><\/span> there <\/span><\/span> would <\/span><\/span> be <\/span><\/span> no <\/span><\/span> highly-performing <\/span><\/span> AI.\u201d <\/span><\/span> <\/h5><\/div>[\/vc_column][\/vc_row][\/vc_section][vc_row pix_particles_check=””][vc_column]
ACCURACY, <\/span><\/span> CONSISTENCY, <\/span><\/span> COVERAGE, <\/span><\/span> BALANCE <\/span><\/span> <\/h3><\/div>
1. Accuracy<\/h2><\/div><\/div><\/div>
Accuracy - Accurate data to fulfill the purpose<\/h3><\/div><\/div><\/div>[vc_column_text]
![\"\"](\"https:\/\/blog.datumo.com\/en\/wp-content\/uploads\/2022\/05\/Frame-2694.png\")
[\/vc_column_text]The key is to have guidelines with high readability.<\/strong><\/h3>\n
![\"\"](\"http:\/\/blog.datumo.com\/en\/wp-content\/uploads\/2020\/02\/3.jpg\")
<\/p>\n\u201cSet accurate guidelines\u00a0 \u2192\u00a0 Train crowd-workers\u00a0 \u2192\u00a0 Achieve high quality data\u201d\u00a0<\/strong>is an unchanging truth.<\/h5>\n
Mandatory exams in order to participate in data annotation<\/span><\/strong><\/h5>\n
![\"\"](\"https:\/\/blog.datumo.com\/en\/wp-content\/uploads\/2020\/02\/3.1-e1653555132700.png\")
<\/h5>\n
<\/h5>\n
Strict validation based on Datumo\u2019s experience & algorithm<\/span><\/strong><\/h5>\n
![\"\"](\"https:\/\/blog.datumo.com\/en\/wp-content\/uploads\/2022\/05\/Frame-2644.png\")
[\/vc_column_text]2. Consistency<\/h2><\/div><\/div><\/div>
Consistency - Consistent labeling results<\/h3><\/div><\/div><\/div>[vc_column_text]
![\"\"](\"https:\/\/blog.datumo.com\/en\/wp-content\/uploads\/2022\/05\/asdasd.png\")
Objective labeling rules to maintain consistency<\/strong><\/h5>\n
What technology does Datumo use in order to maintain consistency?<\/span><\/strong><\/h5>\n
![\"\"](\"https:\/\/blog.datumo.com\/en\/wp-content\/uploads\/2020\/02\/\u1100\u1161\u110b\u1175\u1103\u11732.jpg\")
<\/p>\n3. Coverage<\/h2><\/div><\/div><\/div>
Provide AI with enough case studies based on a variety of data<\/h3><\/div><\/div><\/div>
Dataset that covers a variety of cases that AI would face<\/span><\/strong><\/h5>\n
Data collecting process based on collecting environment<\/span><\/strong><\/h5>\n
![\"\"](\"https:\/\/blog.datumo.com\/en\/wp-content\/uploads\/2020\/02\/\u1100\u1161\u110b\u1175\u1103\u1173\u1105\u1161\u110b\u1175\u11ab4.png\")
[\/vc_column_text]Redundant data filtering system to ensure data variety<\/strong><\/h5>\n
![\"\"](\"https:\/\/blog.datumo.com\/en\/wp-content\/uploads\/2022\/05\/Frame-2726.png\")
[\/vc_column_text]4. Balance<\/h2><\/div><\/div><\/div>
Unbiased dataset<\/h3><\/div><\/div><\/div>
![\"\"](\"https:\/\/blog.datumo.com\/en\/wp-content\/uploads\/2022\/05\/1233-1.jpg\")
[\/vc_column_text][vc_column_text]<\/p>\nAvoid bias by various classification<\/strong><\/h5>\n
From Model-centric to Data-centric AI<\/strong><\/h5>\n