Team Members:

Junhan Zhao, Computer Graphics Technology, Purdue University, zhao835@purdue.edu   PRIMARY
Dayu Wan, Computer Graphics Technology, Purdue University, wand@purdue.edu
Zongcheng Chu, Computer Graphics Technology, Purdue University, chu153@purdue.edu
Xiang Liu, Computer and Information Technology, Purdue University, xiang35@purdue.edu
Dr. Jieqiong Zhao, Electrical and Computer Engineering, Purdue University, zhao413@purdue.edu
Dr. Zhenyu Cheryl Qian, Art and Design, Purdue University, qianz@purdue.edu
Dr. Yingjie Victor Chen, Computer Graphics Technology, Purdue University, victorchen@purdue.edu

Student Team: YES

Tools Used:

YOLO v3
React, Flask, D3.js
i-Solitaire: Semi-automatic Interactive Object Detection on Small Datasets

Approximately how many hours were spent working on this submission in total?

230 hours

May we post your submission in the Visual Analytics Benchmark Repository after VAST Challenge 2019 is complete?YES

Video

https://va.tech.purdue.edu/vast2020mc2/video.mp4

---

Questions

Given the images and text files as well as machine learning outputs, use visual analytics to answer the questions below to improve and understand machine learning outputs and track provenance, uncertainty, and confidence in machine learning results. Ultimately, you must link multiple data types to identify the group CGCS is seeking.

1 – Examine the outputs from the model – either from the detection results provided or the results from a model you chose. Which objects were identified well by the model and which were not? Please limit your answer to 5 images and 250 words.

Figure 1-1. Confusion matrix visualizes VAST provided model predictions.


Figure 1-2. Parallel coordinates visualize the model predictions.


Figure 1-3. A subgroup of items are easy to be correctly predicted.


Figure 1-4. A subgroup of items are easy to be incorrectly predicted (model returns high-scores but wrongly predicted).


Figure 1-5. A subgroup of items are difficult to detect (model returns low-scores).

We firstly visualize VAST provided outputs in a confusion matrix (Figure 1-1). Grey cell means that a person owns a specific totem. Realizing that the given classifier results entirely miss predictions over 20 classes, we train our own classifier adapting to state-of-the-art YOLO v3 network. We follow the discipline of interactive machine learning, namely human-in-the-loop, and design i-Solitaire. The system allows the users to quickly confirm the correctness of model predictions with a glance and label the wrong predictions, regarding true positive (TP) and false positive (FP). We visualize model predictions of each iteration into a parallel coordinate matrix (Figure 1-2). The magenta and green line represent TP and FP respectively. The density of the lines means the predictions amount. A good classifier should have a clear cutoff separation between magenta and green line groups. Also, we expect to see a monotonic trend (either increasing or decreasing) of each line across iterations. FtE coordinate shows the user-verified ground truth, either 0 as wrong, or 1 as true. We also put VAST provided data on the right for comparison.

Based on the parallel coordinates matrix, we can find blueSunglasses, noisemaker, rainbowPens, paperPlate, redDart, turtle (Figure 1.3) were identified well by the model. Also, when checking their coordinate values on the Ft2 axis, our final round fine-tune, we can find the scores of the correct prediction are generally higher and the scores of the wrong are lower. This separation also means our model can well identify these items. However, we can find birdCall, hairClip, sliverStraw, trophy (Figure 1.4) are not identified well by our model, because most are green, even some with high prediction scores. PinkEraser, sign, vancouverCards (Figure 1.5) are also very difficult to identify, because the number of the pink lines is pretty small which means our model doesn't detect much of these items.

2 – Demonstrate your process for using visual analytics to correct for classification errors in the results. How do you represent confidence and uncertainty? How could the correction process be made more efficient? Please limit your answer to 10 images and 500 words.

Figure 2-1. The human-in-the-loop diagram of i-Solitaire. In the iterative traning process, users can verify true positive and correct false positive instances to enlarge the traning dataset.


Figure 2-2. The workflow of training and fine-tune process. Plain: training Yolo v3 neural network on training dataset (516 images in total, 12 images per class) provided by the vast committee for mini-challenge 2. Ft0: augmenting the training set (5 times data augmentation – rotation 90, 180, 270, mirror, crop pixels out, shift ) and re-traning Yolo v3 neural network. Ft1: fine-tune Yolo v3 network with first-round user verified lables. Ft2: fine-tune Yolo v3 network with second-round user verified lables. FtE: the final result that all objects in images are visually verified by human.


Figure 2-3. The interactive labeling panel – verifying True Positive (TP) patches. Users can filter by a specific label on the right side, and then click on a patch if the detected object matches the selected label, such as "yellowBag" in the image.


Figure 2-4. The interactive labeling panel – correcting Flase Positive (TP) patches. Users draw a bounding box on a image and then select a correct label.


Figure 2-5. An example of parallel coordinate for "blueSunglasses". Magenta lines indicate true positive cases and green lines indicate false positive cases. The axes from left to right shows the following stages: traing Yolo v3 with given training set (plain), augmented (Ft0), first-round fine-tune (Ft1), second-round fine-tune(Ft2), final results (FtE), and Yolo v2 model output provided by vast challenge community (vast).


Figure 2-6. The paralle coordinate matrix illustates the final human labeling results for all 43 objects.


Figure 2-7. An example of scatterplot for "brownDie". Each dot represents one bounding box predicted in a image. The left-side shows the confidence score distribution of false positive (FP) instances, and the right-side shows the confidence score distribtuion of true positive (TP) instances. Ideally, the confidence score should be low for FP cases and high for TP cases. The 45-degree soiled red line implies the ideal scenario. The dashed blue line shows the line-fitting result. The disparity between red line and blue line illustrates the difference between human judgments and machine predictions. The purple line represents the cut-off by maximizing accuracy.


Figure 2-8. The scatter matrix illustates the second-round fine-tune predictions vs. human labeled results for all 43 objects.

We adopted a dynamic human-machine collaborative training and labeling environment, i-Solitaire, to support the iterative training for object labels. We designed four visualization components to assist the identification, comparison, verification, and labeling of objects in daily life photos:

(a) an interactive labeling panel to enhance the user experience in labeling objects;
(b) a parallel coordinate matrix to track and visualize the iterative machine learning process and confidence scores output by models;
(c) a scatterplot matrix to track and visualize model prediction accuracy;
(d) a person&object confusion matrix to illustrate which person has which objects.

To correct the classification errors, we used the following iterative training workflow to improve model performance by increasing dataset (shown in Figure 2-1):

• maximize the number of categories of items present
• maximize the prediction scores above the cut-off threshold
• ensure the number of patches shown to users meet the minimum number requirement.

To visualize the provenance of the iterative training process, we use the parallel coordinate matrix (Figure 2-6). The axes represent the evolution of confidence scores of instances (i.e., patches) at different fine-tuning stages (Figure 2-5). The color of lines indicates a patch is labeled correctly or not. Magenta indicates that objects are correctly labeled (TF), and green indicates that objects are mislabeled (FP). The values underneath the axes denote how many patches are applied. As we can see in Figure 2-5 that the size of training patches is increasing. If the magenta and green lines are separated clearly and not blended in a messy, the model performs well on predicting this item. Specifically, the auto-identification of an object is more satisfying if the magenta folding lines are ascending monotonically, or the green lines are decreasing.

In addition, we designed a scatterplot to compare model confidence scores to human observations. As shown in Figure 2-7, the left-side displays the distribution of confidence scores over false positive instances, and the right-side shows the distribution over true positive instances. The purple line shows the recommended cut-off comuputed by maximizing the accuracy score. If the purple line is positioned higher (cut-off is high), it means a model is gullible, and it can be easily deceived by the variability of the data. Low cut-off means a model is hard to make decisions. Ideally, the cut-off should be around 0.5.

In summary, the grid layout, filtering based on label categories, object detection enhance with iterative machine learning, and post-hoc analysis with visualizations are components combined together to make the correction process more efficient.

3 – Characterize the distribution of objects across the forty people

a.   Which people have which objects?  Please limit your answer to 8 images and 250 words

Figure 3-1. A example of customized jitter plot to present the confidence score of "blueSunglasses" for person 9.


Figure 3-2. The confusion matrix to visualize the relationship between objects (row) and persons (columns). The objects are ordered based on the number of people who have the same object. The people are ordered alphabetically. A gray cell indicate that human verifies the person owns the object.

In the confusion matrix we can clear identify who has which objects (Figure 3-1 and 3-2). Each person owns 6 - 10 objects. The results are summaried as below:
Person1(7): miniCards, pinkEraser, noisemaker, rainbowPens, trophy, yellowBalloon, vancouverCards
Person2(7): blueSunglasses, lavenderDie, metalKey, noisemaker, yellowBalloon, brownDie, spiderRing
Person3(7): rainbowPens, voiceRecorder, giftBag, glassBead, pumpkinNotes, yellowBalloon, cowbell
Person4(8): miniCards, lavenderDie, canadaPencil, rubiksCube, hairClip, cupcakePaper, plaidPencil, yellowBalloon
Person5(7): pinkEraser, eyeball, legoBracelet, redWhistle, sign, stickerBox, yellowBalloon
Person6(9): miniCards, lavenderDie, rubiksCube, foamDart, yellowBag, pinkCandle, redBow, silverStraw, yellowBalloon
Person7(7): blueSunglasses, canadaPencil, redWhistle, cactusPaper, paperPlate, trophy, brownDie
Person8(7): metalKey, rubiksCube, voiceRecorder, cactusPaper, cloudSign, silverStraw, vancouverCards
Person9(7): blueSunglasses, metalKey, pinkEraser, cactusPaper, hairRoller, pinkCandle, stickerBox
Person10(7): miniCards, lavenderDie, voiceRecorder, cactusPaper, partyFavor, plaidPencil, sign
Person11(8): noisemaker, rainbowPens, eyeball, foamDart, hairClip, cactusPaper, cupcakePaper, giftBag
Person12(6): noisemaker, hairClip, legoBracelet, yellowBag, cactusPaper, pumpkinNotes
Person13(7): miniCards, blueSunglasses, legoBracelet, birdCall, cupcakePaper, paperPlate, partyFavor
Person14(7): miniCards, canadaPencil, birdCall, cloudSign, pinkCandle, pumpkinNotes, gyroscope
Person15(10): pinkEraser, canadaPencil, noisemaker, voiceRecorder, carabiner, foamDart, gClamp, birdCall, hairRoller, sign
Person16(7): lavenderDie, metalKey, yellowBag, birdCall, plaidPencil, trophy, cowbell
Person17(7): blueSunglasses, rainbowPens, rubiksCube, birdCall, glassBead, redBow, stickerBox
Person18(7): lavenderDie, metalKey, pinkEraser, redWhistle, birdCall, giftBag, silverStraw
Person19(7): pinkEraser, eyeball, gClamp, glassBead, hairRoller, paperPlate, gyroscope
Person20(8): metalKey, noisemaker, eyeball, gClamp, hairClip, cloudSign, partyFavor, redBow
Person21(6): lavenderDie, rubiksCube, gClamp, yellowBag, pumpkinNotes, sign
Person22(7): blueSunglasses, canadaPencil, foamDart, gClamp, giftBag, stickerBox, trophy
Person23(6): rainbowPens, gClamp, redWhistle, cupcakePaper, pinkCandle, brownDie
Person24(7): miniCards, voiceRecorder, gClamp, legoBracelet, plaidPencil, silverStraw, spiderRing
Person25(7): miniCards, canadaPencil, carabiner, paperPlate, partyFavor, sign, brownDie
Person26(7): voiceRecorder, carabiner, foamDart, cloudSign, cupcakePaper, gyroscope, vancouverCards
Person27(6): blueSunglasses, carabiner, legoBracelet, hairRoller, pinkCandle, pumpkinNotes
Person28(7): pinkEraser, carabiner, hairClip, yellowBag, stickerBox, cowbell, spiderRing
Person29(7): lavenderDie, carabiner, redWhistle, glassBead, plaidPencil, redBow, trophy
Person30(7): metalKey, rainbowPens, rubiksCube, carabiner, eyeball, giftBag, silverStraw
Person31(7): blueSunglasses, rainbowPens, cupcakePaper, glassBead, paperPlate, stickerBox, turtle
Person32(7): miniCards, metalKey, yellowBag, cloudSign, plaidPencil, pumpkinNotes, turtle
Person33(7): pinkEraser, foamDart, redWhistle, giftBag, hairRoller, turtle, brownDie
Person34(7): rubiksCube, legoBracelet, partyFavor, redBow, turtle, vancouverCards, spiderRing
Person35(7): canadaPencil, eyeball, pinkCandle, trophy, turtle, cowbell, gyroscope
Person36(7): lavenderDie, noisemaker, voiceRecorder, hairClip, sign, silverStraw, turtle
Person37(7): noisemaker, cloudSign, hairRoller, paperPlate, partyFavor, gyroscope, redDart
Person38(6): rainbowPens, rubiksCube, redWhistle, yellowBag, cowbell, redDart
Person39(8): miniCards, blueSunglasses, canadaPencil, voiceRecorder, foamDart, legoBracelet, vancouverCards, redDart
Person40(7): metalKey, pinkEraser, eyeball, hairClip, glassBead, redBow, redDart

b.   Identify groups of people that have object(s) in common. Please limit your answer to 10 images and 500 words.

Figure 3-3 hierarchical clustering.


Figure 3-4. 3 clusters.


Figure 3-5. 4 clusters.


Figure 3-6. 5 clusters.


Figure 3-7. 6 clusters.


Figure 3-8. 7 clusters.

After applying the hierarchical clustering with "Ward" distance measurements, the person with similar objects is gathered together. As shown in Figure 3-3, the matrix shows the objects in rows and the people in columns. A green cell with number "1" indicates a person has a specific object. Both the rows and columns are clustered using hierarchical clustering. If users are interested in objects shared by similar groups of people, they can inspect the rows in close positions. Similarly, people close in columns are sharing common objects. We compute five clusters initially since we assume eight people will be in the white hat group. Later we also examine the 3 - 7 clusters (Figure 3-4, 3-5, 3-6, 3-7, and 3-8).

Further investigating the clustering results in Figure 3-8 and inspecting common objects in Figure 3-3, we can see that "person 17" and "person 31" have the most similar set of objects. Specifically, they have 4 object in common including "blueSunglasses", "stickerBox", "glassBead", and "rainbowPens" (Figure 3-3). The second group two people ("person 20" and "person 40") is shareing four objects including "metalKey", "eyeball", "hairClip", and "redBow".

There are many groups of two people who share three objects, such as "person 9" and "person 27" also have three objects in common: "pinkCandle", "blueSunglasses", and "hairRoller".

Observing the matrix in Figure 3-3, we find no three persons share three objects in common. Therefore, we further search for multiple persons to share two objects. Only four groups of four people share two objects:

• There are four people sharing two objects "stcikBox" and "blueSunglasses": "person 17", "person 31", "person 22", "person 9".
• Another example of four people sharing two objects is "person 10", "person 24", "person 4", "person 32" shares "miniCards", "plaidPencil"
• The third example is "person 10", "person 29", "person 16", "person 4" shares "lavenderDie" and "plaidPencil".
• The last example is that "person 19", "person 9", "person 15", "person 33" shares pinkEaser and hairRoller.

There are multiple sets that three persons share two objects. Such as "person 1", "person 4", "person 6" shares "miniCards" and "yellowBallon".

4 – Which group do you think is the most likely group with the "totem"? What is your rationale for that assessment? Please limit your response to 5 images and 300 words.

Figure 4-1. Confusion matrix visualizes our final results verified by human perception on i-Solitaire.


Figure 4-2. Secrect totem candiates, which are owned by no less than 8 people.

We create a confusion matrix to track the model prediction and human verification in i-Solitaire (Figure 5-2 (D), Figure 4-1). Each row represents a totem and each column represents a specific person. Each cell visualizes final model prediction scores of each totem from all images that each person owns. If a cell’s background is grey out, it means the corresponding person owns this totem. This result is verified by human perception with i-Solitaire. We sort the rows according to the number of individuals (owners) for each totem. Since the only clue we have is that there are eight individuals in this subgroup with the totem, we check all totems with 8 persons. Additionally,we realize that “this item is unremarkable to those who are not part of the group; it could easily be confused for a free item handed out at a conference. ” Therefore, we assume there may be some random guy holding this totem as well (Figure 4-2). By enlarging the suspicious range, we focus on all totems with no less than 8 holders. There are 10 in total. The bar chart shows how many totem candidates each person owns (Figure 4-3). We can measure each person's likelyhood by this counts.

Additionally, after reviewing the captions, we suspect “turtle” could be the secret totem. The highlighted words have strong implications that “turtle” is the leader and fighting with the enemies. This inference pertains to the challenge context. However, “turtle” totem is only owned by 6 persons. Assuming some individuals in the subgroup are not showing their secret totem in the images, this could be a solution (Figure. 4.5).

5 – Process question: Did you choose to use the object recognition model results provided or use your own machine learning algorithm? Why did you make that choice? What was the biggest challenge you faced?  Please limit your response to 3 images and 300 words.

Figure 5-1. The quality of image data is not stable.


Figure 5-2. the Overview of i-Solitaire framework.

We decided to train a separated model for this object recognition task rather than using provided results. We firstly applied the provided model results to our “ Overview Matrix” (Figure 1-1). Apparently, there are 21 classes missing in the VAST provided results, which is nearly half of the total cases and way off the target. Moreover, many of the high-scoring predictions are incorrect. Inspired by the challenge description, we adapted a Yolo v3 model and trained it on this dataset. The detection performance (average precision) is significantly improved.

One of the biggest challenges comes from the small size of the training set, regarding the number of the available images for training. There are only 12 images per class in the training set. Moreover, after diagnosing the provided outputs, we realized that the inaccuracy also results from the unstable quality of the images, namely the separation of the objects in the images. Some of the images involve issues like, (1) multiple objects are in one image; (2) objects are obstructed; (3) random background; (4) the portion of the objects in the images is very small. Therefore, we firstly labeled the images with a bounding box. Then, we applied data augmentation to further enlarge the amount of the training set. As the parallel coordinates in Fig. 1-2 shows, after data augmentation, the accuracy of the predictions is improved significantly. Furthermore, our i-Solitaire system (Figure 5-2) allows the users to verify the correct predictions and freely label the incorrect predictions. After that, i-Solitaire will use these new labelled data and fine-tune the model until all images have been perceived (either labelled or verified) by the users. We design this high-efficient human-machine collaborative system to alleviate the tedious, repetitive and time-consuming process of manual detection. With the assistance of i-Solitaire, users can verify the existence of objects in the images quickly and accurately.