Entry Name: "PKU-Yuan-MC2"

VAST Challenge 2020
Mini-Challenge 2

Team Members:

Shaocong Tan, Peking University, tanshaocong0108@gmail.com PRIMARY

Sihang Li, Peking University, lisihang@pku.edu.cn

Jiacheng Yu, Peking University, jiachengyu@pku.edu.cn

Xiaoru Yuan, Peking University, xiaoru.yuan@pku.edu.cn ADVISOR

Student Team: YES

Tools Used:

D3.js

A visual analytics system for correcting detection errors in the challenge, developed by PKUVIS Group

TotemVis, a system that help users find the totem in the challenge, developed by PKUVIS Group

Approximately how many hours were spent working on this submission in total?

200

May we post your submission in the Visual Analytics Benchmark Repository after VAST Challenge 2020 is complete? YES

Video:

mc2.wmv or

http://vis.ucloud365.com/vast-challenge-2020/mc2.wmv

Questions

Given the images and text files as well as machine learning outputs, use visual analytics to answer the questions below to improve and understand machine learning outputs and track provenance, uncertainty, and confidence in machine learning results. Ultimately, you must link multiple data types to identify the group CGCS is seeking.

1 – Examine the outputs from the model – either from the detection results provided or the results from a model you chose. Which objects were identified well by the model and which were not? Please limit your answer to 5 images and 250 words.

We examine the detection results provided using manual classification as well as the visual analytics system in Question 2. We use the corrected classifications as ground truth. The scatterplot in Fig.1-1 shows the accuracy and recall(1 - miss rate) of the detection results of each category.

Fig.1-1 The scatterplot of the detection results of each category.

In Fig.1-1, we can see that the detection accuracy of each object category is relatively low, except paperPlate, which has the highest accuracy of 0.6. Besides, blueSunglasses has the highest recall, which means it is hard to miss during the detection. trophy has both the lowest accuracy and recall, which means the model does not perform well when detects trophies in the image. Moreover, 21 kinds of objects including rainbowPens and miniCards do not appear in the results but do appear in the given images. The circles of these categories are shown below the horizontal axis. It means that the model has wrong detecting these kinds of objects.

On this basis, we further draw the distribution of confidences of the given results using line charts. The line charts are shown in Fig.1-2. We can see that the confidences are mainly distributed around 0.3, while the correct results have more high confidence.

Fig.1-2 The distributions of confidences in the given results.

For the single category of objects, we can see that some categories have more high confidences in correct results than wrong results, like eyeball shown in Fig.1-3. This means that the detections of these kinds of objects with high confidences are reliable. However, some categories also have many wrong results with high confidences, like redWhistle shown in Fig.1-4. It means that the detections of these kinds of objects with high confidences are still unreliable.

Fig.1-3 The distributions of confidences of eyeball.

Fig.1-3 The distributions of confidences of redWhistle.

2 – Demonstrate your process for using visual analytics to correct for classification errors in the results. How do you represent confidence and uncertainty? How could the correction process be made more efficient? Please limit your answer to 10 images and 500 words.

The classification errors can be divided into two categories, object wrongly detected and object not detected. We provide a correction process to correct these two types of errors, including a rough process and a thorough process. In the rough correction process, we correct the classification errors that objects are wrongly detected of each category. In the thorough correction process, we correct both the two errors by deleting and adding classifications of one single image at once. The system is shown in Fig.2-1.

Fig.2-1 The visual analytics system to correct classification errors in the results.

In the rough correction process, we correct the images of a category at once. The process is shown in Fig.2-1(a). We first crop those images by the provided detection results, then calculate the cosine similarity between them and training images provided of each category. We use a scatterplot to show the average and variance of the similarities. Users can select several results using lasso, as shown in Fig.2-2. After selecting the results, the thumbnails of the cropped images will be shown under the scatterplot. Users can compare these thumbnails with the image above, and double click on each thumbnail to remove it from the selection. Then users can choose to delete or retain the selected classifications. This process allows users to correct many wrongly detected errors with a few interactions, which makes the correction process more efficient. However, this process cannot correct all errors, so we need a more detailed process.

Fig.2-2 Select results using lasso.

In the thorough correction process, we correct one single image at once. The process is shown in Fig.2-1(b). For each image, we show the detection results on the image using a red box. The thumbnails of the cropped images are shown on the bottom, next to the thumbnails of each category. It can help users compare the two images. Users can double click on the cropped thumbnails to remove the classifications from the results. Also, this process allows users to correct object not detected errors. Users can click on the thumbnail of one category, and then draw a box on the image above. It then becomes a new classification and will be added to the results. The process is shown in Fig.2-3. In order to make users better find the objects in the image that have not been detected yet, we also show the captions of the image. The words that may be related to the objects are highlighted, as shown in Fig.2-4. Furthermore, we find that the objects that appeared in one image are also likely to appear in other images of that person. So we use bar charts to count the objects in all images of the person. Objects that appear more often are placed in the front.

Fig.2-3 The process to add new detection results.

Fig.2-4 The captions of the image with keywords highlighted.

Besides, we also add the views used in Question 1 to our system, as shown in Fig.2-1(c). Users can have an overview of the model accuracy during the correction process. Moreover, we add a new line chart to show the distribution of IoU(Intersection over Union), cause we find that some detections are correct but the detected positions deviate from the correct targets.

3 –– Characterize the distribution of objects across the forty people. 

a. Which people have which objects?  Please limit your answer to 8 images and 250 words  

Matrix Overview

After correcting the classification errors by the visual analytics system in Question 2, we use a 40×43 matrix to display the objects each person has. The color saturation of cells encodes the confidence of the results. If there is an image that has a score over the threshold 0.6, we use the maximum of scores as confidence, else we use the average of the scores. Users can drag the bar on the bottom left to filter the results of low confidence.

Fig.3(a)-1 Matrix showing the distribution of items.

Revision

As shown in Fig.3(a)-2, when the cells in the matrix are chosen, the system provides users with the images as evidence of the person's procession of the object by the scores of the images. For example, Person 8 only has one image as the evidence of the procession of miniCard, which makes it hard for users to judge. Fig.3(a)-3 shows how to filter this possibility by decreasing the confidence of the cell.

Fig.3(a)-2 Evidence that Person 33 owns turtle.

Fig.3(a)-3 Decrease the confidence of uncertain possibility.

Object Display

Ultimately, when users select a particular person, as shown in Fig.3(a)-4, the cells in the corresponding row will be highlighted. The cells can be automatically sorted by confidence or manually aggregated by dragging. The sankey diagram shows the frequency of occurrence of objects where the width of the connection encodes the count of images and individuals are distinguished by color.

Fig.3(a)-4 Display of objects from Person3.

Take Person 3 as an example, the person has 6 objects of confidence over 0.9, namely cowbell, giftBag, glassBead, pumpkinNotes, rainbowPens and yellowBalloon. The images of these items are shown in Fig.3(a)-5.

Fig.3(a)-5 Images from Person3.

b.  Identify groups of people that have object(s) in common. Please limit your answer to 10 images and 500 words.

Object Distribution

The histogram under the matrix shows the amount distribution of objects. As shown in Fig.3(b)-1, there are the most owners of miniCards, up to 14 in total. Nine people own metalKey, pinkEraser, lavenderDie and blueSunglasses, while eight people own canadaPencil, noisemaker, rainbowPens and rubiksCube. We can recognize the number of owners of each object in the same way.

Fig.3(b)-1 The amount distribution of different objects.

When choosing a particular object in the sankey diagram, it will show the number of times the item appears in different sets of individuals. As shown in Fig.3(b)-2, the group of Person 2, 4, 6, 10, 16, 18, 21, 29 and 36 owns lavenderDie. Among them, Person 4, 16, 21 and 36 merely have one image showing the item.

Fig.3(b)-2 The individuals who own lavenderDie.

Object Co-occurrence

When choosing a particular object, the matrix will display its frequency of co-occurrence with other objects. The frequency of co-occurrence is calculated based on the size of each person's album. Take Person 36 as an example, as shown in Fig.3(b)-3, lavenderDie solely co-occurs with hairClip, thus the color saturation is higher. lavenderDie of Person 4, 6 and 21 co-occurs with miniCards and rubiksCube on the same image, which indicates a potential subgroup.

Fig.3(b)-3 The co-occurrence of lavenderDie and other objects.

People Aggregation

By dragging the bar on the bottom right of the matrix view, users can set the minimum number of individuals to cluster together in the matrix. The goal of rearrangement is to search for groups of people who own the same combination of objects. As shown in Fig.3(b)-4, when the minimum size of the cluster is greater than 5, multiple single items meet the condition of aggregation.

Fig.3(b)-4 Matrix view when the minimum size of the cluster is greater than 5.

We set the minimum number of people to 5. There are two groups of 5 people who own the same two objects. As shown in Fig.3(b)-5, Person 1, 8, 26, 34 and 39 in the first group own vancouverCards and miniCards, while Person 4, 6, 8, 21 and 34 in the second group own rubiksCube and miniCards.

Fig.3(b)-5 Matrix view when the minimum size of the cluster is equal to 5.

By adjusting the order of the matrix, it will show that every person in the third group of Person 8, 10, 24, 26 and 39 owns voiceRecorder and miniCards. The rearrangement is operated by an iterative algorithm. The maximum cluster is shown in the top left corner and it follows the same strategy when rearranging the rest of the matrix. As a result, it takes at most two steps for users to switch cluster schemes.

Then we set the minimum size of the cluster to 3. According to the matrix shown in Fig.3(b)-6 , we can find that Person 8, 26 and 39 own voiceRecorder, vancouverCards and miniCards, while Person 4, 6 and 21 all own miniCards, rubiksCube and lavenderDie.

Fig.3(b)-6 Matrix view when the minimum size of the cluster is equal to 3.

There are combinations containing 4 items at most when the minimum size of the cluster is equal to 2. Fig.3(b)-7 shows three of the six grouping schemes.

Fig.3(b)-7 Matrix view when the minimum size of the cluster is equal to 2.

4 –– Which group do you think is the most likely group with the “totem”? What is your rationale for that assessment? Please limit your response to 5 images and 300 words.

Based on the assumption, every person in the group of eight owns at least a common object. Thus, we focus on the objects that occur at least in 8 people's images when finding the “totem”. There are 9 objects that satisfy the condition, shown in Fig.4-1. Then we check the distribution and co-occurrence of each object one by one. We find a Suspicious phenomenon that rubiksCube occurs in the images of Person 4, 6 and 21 together with miniCards and lavenderDie.

Fig.4-1 The co-occurrence matrix and sankey diagram when selecting rubiksCube.

We further observe the co-occurrence image by clicking the corresponding cell in the matrix, and we are surprised to find that Person4_18 and Person6_1 are almost the same. It indicates that Person 4 and 6 may be related to each other, and these three objects are potential totems. Among them, eight people who own rubikscube have no record of text on Y*INT social media platform, which is likely the way that the white hat group hides its tracks.

Fig.4-2 Image Person4_18.

Fig.4-3 Image Person6_1.

To verify the hypothesis, we view the images that contain other items with the rubiksCube. In some images from Person 21 and 34, rubiksCube and the red joker different from minicards appear at the same time. Moreover, there is the concurrence of rubiksCube and rainbowPens in the images from Person 17, 30 and 38.

Fig.4-4 Image Person21_4.

Fig.4-5 Image Person34_12.

Therefore, we judge that the group that owns rubikscube is the most likely group with the "totem".

5 –– Process question: Did you choose to use the object recognition model results provided or use your own machine learning algorithm? Why did you make that choice? What was the biggest challenge you faced?  Please limit your response to 3 images and 300 words.

We choose to use the results provided for the following two reasons: The reasons for the training set and the reasons for the given images.

Training set

In the training set, each category only has around 12 images. The training set is too small to train an object detector. If we use image enhancement to enlarge the training set, it will be very time-consuming. So it is very hard to train a new model using the provided dataset.

Given images

Some given images also make it hard to detect objects. Objects in some images are very similar to their surroundings, like Fig.5-1. It is hard to tell the objects in the images, even do it manually.

Fig.5-1 An image that the object is hard to detect.

Also, some images own a lot of objects, like Fig.5-2. The objects overlap so much that they can hardly be all detected.

Fig.5-2 An image containing a lot of different categories of objects.

Besides, as the lighting conditions are very different while taking the photos, the colors of one specific object in different images can vary a lot, like Fig.5-3. So it is difficult for the model to detect the objects.

Fig.5-3 An image that the color of the object is quite different from that in the training set.