Entry Name:  "Webank-Qiangqiang-MC2"

VAST Challenge 2020
Mini-Challenge 2

 

 

Team Members:

Qiangqiang Liu, alfredliu@webank.com PRIMARY
Chunfeng Tang, jsontang@webank.com
Quan Li, forrestli@webank.com
Huanbin Lin, bindylin@webank.com
Haotian Xu, neoxu@webank.com
Tianjian Chen, tobychen@webank.com

Student Team: No

 

Tools Used:

D3.js

Python

Node.js

 

Approximately how many hours were spent working on this submission in total?

Approximately 168 hour

 

May we post your submission in the Visual Analytics Benchmark Repository after VAST Challenge 2020 is complete?
YES

 

Video

https://youtu.be/sz1GTgIzVAg

 

 

Questions

Given the images and text files as well as machine learning outputs, use visual analytics to answer the questions below to improve and understand machine learning outputs and track provenance, uncertainty, and confidence in machine learning results. Ultimately, you must link multiple data types to identify the group CGCS is seeking. 

1 – Examine the outputs from the model – either from the detection results provided or the results from a model you chose. Which objects were identified well by the model and which were not? Please limit your answer to 5 images and 250 words.

 

There are 43 images of training objects in the 'TrainingImages' file, but only 22 objects are found in image files. All 40 people share the same six images named 'pumpkinNotes', 'birdCall', 'pinkCandle', 'eyeball', 'cloudSign' and 'blueSunglasses'. We calculate the scores of each object in all the images, then we use boxplot to depict the distribution characteristics of each object. In the following chart, the row represents the object and the column represents the score. As we can see from the plot, 'paperPlate' of which the average score is 0.53, has the best performance. Besides, 'paperPlate', 'noisemake', 'redWhistle', 'hairClip', 'lavenderDie', paperPlate, noisemake, redWhistle, hairclip, lavenderDie and sign have better classification performance. We can compare the classification performance of different objects from the average value and get the stability of the value distribution from the individual boxplot’ scale. Thus, we can see that 'eyeball', 'pinkCandle', 'blueSunglasses', 'yellowBag', cupcakePaper, 'partyFavor', 'canadaPencil', 'trophy', 'vancouverCard' has poor classification performance.

Fig. 1 Score Boxplot View

"CupcakePaper" has poor classification performance, so we select 'cupcakePaper' in the boxplot to conduct further analysis. In the following 'Labeling View', users can select different images of persons and see the classified objects' zone. Then the user can compare the classified objects with the training images, and they can further relabel the classified results. We find that "cupcakePaper" of the 9th image of person 33 is some bricks. So, we check the classification results by the distribution of boxplot and artificial proofreading.

Fig. 2 Label View of cupcakePaper

2 – Demonstrate your process for using visual analytics to correct for classification errors in the results. How do you represent confidence and uncertainty? How could the correction process be made more efficient? Please limit your answer to 10 images and 500 words.

We can see the confidence and uncertainty distribution in boxplot. In the following 'Labeling View', users can browse images of persons and relabel them, and the results will be reflected in other views. For example, if we relabel a birdCall of a certain person as NO, then the numbers of birdCalls of this person will decrease by 1.

 

Fig. 3 Label View of yellowBalloon

When we relabel 'cupcakePaper' as NO, we modify the score of this object as 0. And this change will be reflected in the boxplot. Then we can see changes in the boxplot of 'cupcakePaper'. For a more efficient processing, we first look at some of the objects that the classification is not quite good through the distribution of boxplot and then select these objects. We select the images related to these objects, use the red box to mark the position of the object in the picture, and then manually compare the object with the 'TrainingImages', to judge whether it has been classified correctly or not and determine to label it as YES or NO.

Fig. 4 Comparison Score Boxplot View

3 –– Characterize the distribution of objects across the forty people. 

 

a.    Which people have which objects?  Please limit your answer to 8 images and 250 words  

 

In the following matrix view, the row represents a person ID and the column represents the objects. We use rectangles' colors to encode the number of objects that belong to certain persons. Persons with ids of 13, 17, 23, 32, 33, and 34 have larger number of objects. Most person have 'eyeball', 'pinkCandle', 'sign' and 'cloudSign'.

Fig. 5 Matrix View

 

In the following matrix view, the row represents a person ID and the column represents the objects. The rectangles' colors indicate that whether there is an object. We can see which objects belong to whom in this view. When users relabel some objects, the result will be also reflected in this view. Person 34 has the highest number of object (20) and person 34 has the least number of object(12).

Fig. 6 Matrix View

In the following view, we use the node-link view to demonstrate the relationships between persons and objects. The blue circles represent persons and the green circles represent objects. This view can reflect what objects a person owns and common objects to others.

Fig. 7 Person/Object Node-Link View

b.     Identify groups of people that have object(s) in common. Please limit your answer to 10 images and 500 words.

 

 

In the following projection view, each circle represents a person. 22 objects are found in image files. We count numbers of objects of each person, then we get a vector of 22 dimensions of each person. We use t-SNE to project persons to a two-dimensional space and use the scatter plot for visualization. The projection view represents the distribution similarity of the number of objects owned by different people. We brush a zone in the projection view, then we can see the common objects in 'Person/Object Node-Link View'. The selected persons will be also highlighted in the Matrix View. As we can see, persons of ids 7, 15, 17, 26, 34 and 38 have more objects that are in common.

Fig. 8 Similar Person Analysis

 

4 –– Which group do you think is the most likely group with the “totem”? What is your rationale for that assessment? Please limit your response to 5 images and 300 words.

5 –– Process question: Did you choose to use the object recognition model results provided or use your own machine learning algorithm? Why did you make that choice? What was the biggest challenge you faced?  Please limit your response to 3 images and 300 words.

We did not use our own model for image recognition. We analyzed the model results provided in the title. Due to the rush of time, we think that our existing system and analysis solutions still have a lot of room for improvement. First, for example, we use the comparison results of the manual comparison model and the reference picture to compare the classification results. Although this method is effective, pictures that require more user tags increase the workload of manual analysis. If you can provide some score filters, choose a lower score for manual labeling, or use a model to compare the classification objects and reference maps. Before manual labeling, do a preprocessing to filter out some unnecessary manual labeling workload. Second, we use a node-link view to illustrate the affiliation of objects and persons and show which person has the same object. However, it is difficult to observe if all 40 people are displayed. Techniques like edge bundling or changing the layout according to the data characteristics or adding some filtering functions can facilitate better observation.

                Fig. 9 Overview of Person/Object Node-Link View