VAST Challenge 2020
Mini-Challenge 2
Yes
120 hours
Yes
Before we present the answers to the questions, we first explain the basic idea of our system, called the Visual Detective. Our system contains four views: a scatter plot (Fig.0-1 (a)), the BiSet visualization (Fig.0-1(b)), the sparkline viewer (Fig.0-1(c)), and the annotation window (Fig.0-1(d)).
The scatter plot visualizes both the images and text in the same 2D space. It enables the analyst to explore the semantics of images and text simultaneously and discover whether some people have met before. To analyze the two modals of data (images and text) in the same space, we adopt multiple machine learning techniques, as shown in Fig.0-2. We first obtain both the embeddings of images and text from pre-trained neural networks, including the backbone of YOLO v3 (DarkNet) [1,2] and the Transformer [3]. Then we project both the image embeddings and sentence embeddings into the same lower dimensional space using canonical correlation analysis (CCA) [4] technique by maximizing the correlation between the image and sentence pairs (more details will be discussed in question four).
The BiSet visualization shows the relationships between the persons and objects by modeling their relationship as a bipartite graph. The critical feature of BiSet visualization is that it explicitly bundles the edges using the biclustering algorithm. The BiSet enables us to examine the groups of persons who share the same objects.
The sparkline viewer summarizes the results of high-dimensional predictions as a list of parallel coordinates. It enables the analyst to get an overview of the quality of the predictions. Meanwhile, the analyst can click one object to examine the predictions and brush the images that are possiblily misclassified. The selected images will be highlighted in the scatterplot.
The annotation window enables the analyst to fix the prediction errors interactively. Once the analyst saves the corrections, the labeled images will be used as training data to update the underlying object detector. This technique, namely visual active learning[5,6], significantly improves the efficiency of the analyst.
Given the images and text files as well as machine learning outputs, use visual analytics to answer the questions below to improve and understand machine learning outputs and track provenance, uncertainty, and confidence in machine learning results. Ultimately, you must link multiple data types to identify the group CGCS is seeking.
Please limit your answer to 5 images and 250 words.
We retrain our model using the provided training data and give the reason in question-5. Specifically, we choose YOLO v3 [1,2], which is the state-of-art of object detection model and an updated version of YOLO v2.
For easy comparison, we quantify the identification goodness of each object by explicitly differentiating the confidence and uncertainty. The confidence is the maximum value in scores, while the uncertainty is defined using 1-vs-2 value. The reasons for doing so are described in Fig.1-2(a). Now we define that a bounding box is identified well only if the model is confident and certain, namely, goodness value defined in formula (2) in Fig.1-2(a). Then we calculate the object's goodness by averaging the goodness for all the bounding boxes labeled as this object.
Now we visualize each object as one sparkline coupled with a parallel coordinate (Fig.1-2(b)) and a histogram (Fig1-2.(c)). The parallel coordinate visualizes the score distribution of bounding boxes labeled as this class, while the histogram presents the distribution of the confidence values. Fig.1-2 presents the list of objects ranked by the goodness in reverse order (Top objects are identified well while the bottom ones are not). The analyst can select one object to check the parallel coordinate (Fig.1-2(d)). We present the parallel coordinates of top-3 and bottom-3 objects in Fig.1-2 on the left. Notice that the top objects usually have high scores, which result in a sensible peak in parallel coordinates while the objects at the bottom in the list have several low peaks in parallel coordinates.
Please limit your answer to 10 images and 500 words.
We leverage the visual active learning [5,6] strategy to correct classification errors. Visual active learning enables analysts to choose images with uncertain and unconfident bounding boxes based on their domain knowledge to correct the errors. Then the underlying model will be retrained using the corrected images. Once the model is updated, the visualization is updated. This strategy significantly improves the efficiency of analysts.
Specifically, the analyst can order the objects based on the goodness discussed in question one. Objects at the bottom in the list are likely to have more classification errors. The analyst can click one sparkline to examine the bounding boxes' classification results in the parallel coordinate righthand. Then the analyst interactively selects bounding boxes that are uncertain with low confidence values using brushes. The confidence of one bounding box is the maximum value of the scores for each class. And the uncertainty can be reflected in the parallel coordinate when there are multiple peaks, which means the object detector confuses between several objects. All the images that contain the selected bounding boxes will be highlighted in the scatterplot. Then the analyst can click the edit button to evoke the annotation window. The annotation window lets the analyst correct the errors by changing the coordinates and labels of bounding boxes, and drawing new bounding boxes or deleting them. Once the correction is saved, this image will be sent to the labeled training set. The underlying model is retrained using the newly added labeled images.
We demonstrate the whole process with an example.
Fig.2-1 is the initial stage of the Visual Detective. The analyst can click one sparkline (Fig.2-1(a)) or a person dot (Fig.2-1(b)) to select a subset of data. The selected samples will be highlighted in the scatterplot (Fig.2-1(c)). After selecting some samples, the analyst can click the edit button to evoke the annotation window (Fig.2-2).
For example, in Fig.2-2, the model detects the yellow balloon successfully. However, the bounding box is not precise enough. The analyst can directly adjust the bounding box interactively (Fig.2-3(a)). Moreover, the analyst can draw bounding boxes of the noisemaker and the lavender die that the model does not detect (Fig.2-3(b)-(c)). After finishing the correction, the analyst clicks the save button and further to correct other images.
In this example, we let each task as the process to correct images, retrain the model, and update the visualization. The number of samples for each task can be changed through the interface. We use a progress bar to reveal the progress of each task. The color of the progress bar indicates the state of the task. The blue color stands for the data correction process (Fig.2-4 (a)). The yellow color stands for the updating process after 20 images are corrected (Fig.2-4 (b)). Tasks can be conducted in parallel.
With several tasks completed until all the images either corrected or examined by the analyst, the final result is shown in Fig.2-5 (b).
Please limit your answer to 8 images and 250 words.
After several iterations of visual active learning, the analyst has checked all the images and makes sure that no errors in the classification. We present the final result using a matrix visualization to show which people have which objects.
Please limit your answer to 10 images and 500 words.
We model the relationships between persons and objects as a bipartite graph. Given two entity sets P and O, a (binary) relationship R (P, O) between P and O is a subset of P X O (the Cartesian product of P and O). We define the possibility that a person P1 has an object O1 as the maximum confidence value of all the bounding boxes labeled as object O1 by the object detector for the images of person P1. Meanwhile, let the confidence values of bounding boxes annotated by the analyst be one. Then we model the relationship R by binarizing the possibility. A person P1 has an object O1 only if its possibility is larger than a threshold. In this challenge, we choose the threshold as 0.6. The analyst can select a proper parameter through the interface.
We visualize the bipartite graph using the BiSet [8] technique. The BiSet explicitly groups the links between persons and objects as edge bundles using the biclustering algorithm, which significantly reduces the visual clutter. An edge bundle is a closed bicluster in which all the persons in it share the same objects, and there are no other persons or objects that can be included to maintain this relationship. In this challenge, we choose the BiMax [9] algorithm to mine all the biclusters in the bipartite graph.
More specifically, we visualize each person and object as one light-blue bar along the two sides. The length of the dark-blue bar at the left of the person bar stands for the number of objects this person has (Fig.3-3). Similarly, the length of the dark-blue bar at the left of the object bar stands for the number of persons this object belongs to (Fig.3-3). As there are more than 200 biclusters of the bipartite graph, we enable analysts to click person bars and object bars to filter the biclusters. For example, If the person P1 and the object O1 are selected, only the biclusters that contain P1 and O1 are displayed. Denote the number of persons and objects inside a bicluster as N1 (length of the light-blue bar at the left side of a bundle) and N2 (length of the light-gray bar at the right side of a bundle). The length of the edge bundle stands for the size of the bicluster (N1+N2). The analyst can hover any person or object bar to highlight the corresponding biclusters and links. Moreover, we draw a gray ruler on the list of objects in the BiSet to indicate the number eight. Objects that belong to eight persons are highlighted with red borders.
For example, the analyst can click the person five to examine all the biclusters containing this person (Fig.3-3(a)). Meanwhile, the analyst can check the specific relationships inside a bicluster by hovering an edge bundle. For example, the edge bundle highlighted in Fig.3-3(b) shows that persons 5, 9, and 28 share the same objects, including the stick box and the pink eraser.
Please limit your response to 5 images and 300 words.
We think the eight persons with the rainbow pen (totem) are behind the bug.
After several iterations to correct classification errors and improve the object detector until we are confident about the results, we find four candidate objects (voice recorder, rainbow pen, Canada pencil, and Rubik's cube) each belongs to eight different individuals. Now we want to exclude three objects by checking if persons in the same group have met in person.
Moreover, we find other similar relationships by exploring the scatterplot (Fig.4-4). This scatterplot is the key to find the clues discussed above, and we explain the technique in detail. Specifically, we project the image features extracted from the backbone of YOLO v3 (DarkNet) and the sentence vectors calculated by the Transformer into the same space using canonical correlation analysis (CCA). CCA learns two linear transforms of the image space and text space that maximize the correlation of image-text pairs. Then we visualize the shared space using the t-SNE [10] algorithm (Fig.4-5). Besides, we show the density contours over the scatterplot using kernel density estimation.
Please limit your response to 3 images and 300 words.
We choose to use our model YOLO v3 retrained on the training dataset for two reasons:
1) Model results provided can not support the task to evaluate the model in question one. The reason is that the data only provides one score for each bounding box, which hinders us from reasoning the decision's uncertainty. As the score only represents the detector's confidence that the bounding box contains the specific object, it does not tell us the uncertainty of this decision. Considering that most object detectors also provide the score distribution for each bounding box, we decide to train our model for this task. Specifically, we choose Yolo v3 [1,2], which is the state-of-art of object detection and an updated version of YOLO v2.
2) As shown in question two, we adopt the visual active learning strategy to update the underlying model every time the analyst makes corrections. Only the model results can not support this task.
The biggest challenge we faced is the scalability of visual active learning. In this challenge, we demonstrate the Visual Detective in Ubuntu 16.04 with GPU of GeForce RTX 2080 Ti. It takes about 10-15 minutes every time the YOLO v3 is retrained with 20 corrected images and updates the predictions of the rest images and visualization. A more promising solution is to adopt the strategy of progressive visual analytics [11,12]. That is, the analyst can steer the updating process based on his/her interests progressively. In the future, we will extend our work to progressive visual analytics to improve the scalability further.