Jinbin Huang, jhuan196@asu.edu (Primary)
Aditi Mishra, amishr45@asu.edu
Anjana Arunkumar, aarunku5@asu.edu
Dr. Chris Bryan, cbryan16@asu.edu (Advisor)
All team members are in the School of Computing, Informatics, and Decision Systems Engineering at Arizona State University.
Student Team: Yes
Python and Pandas were used for data cleansing and reformatting. The interfaces were built using d3.js and node.js.
Approximately how many hours were spent working on this submission in total?
~200 hrs by all team members.
May the submission be posted in the publicly accessible Visual Analytics Benchmark Repository?
Yes
Submission materials
Downloadable source code and installation instructions for the submission may be found at https://github.com/JakobWong/VAST2020/releases/tag/v1.0.
Video Demo
Question 1: Examine the outputs from the model – either from the detection results provided or the results from a model you chose. Which objects were identified well by the model and which were not? Please limit your answer to 5 images and 250 words.
Using the provided classifier, we measure performance on each object by the precision of its predictions (true positives / total positives). While other metrics such as recall were investigated, these are not incorporated in the finalized system as they offered little additional benefit in analyzing performance. To visually analyze precision, we design a “multiple beeswarm” Interface #1 (Fig 1.1). Each beeswarm plot shows individual predictions as circles, with vertical height representing confidence score. Reference images are shown as pictures below each beeswarm. (The interface is horizontally scrollable, see Fig 1.2.)
Using Interface #1, we efficiently inspected the model’s 4,300 predictions for 900 images from 40 people, interactively batch labeling predictions as true positives (green circles) or false positives (red circles). Fig 1.1 shows the correct and incorrect classification predictions after batch labeling. (The batch labeling process is described in Question 2.) Based on the large amount of false positives across all classes, the provided classifier exhibits very poor performance.
Figure 1.3 shows details. The classifier fails to recognize 21 of the 43 categories, classifying no objects in any of those 21 categories. Zero prediction of a category indicates especially poor performance. Among the 23 recognized classes, 13 have precision lower than 10.0% (another indicator for weak performance), 6 have precision between 10% and 30%, and 3 classes (noisemaker, yellowBalloon, paperPlate) have a true positive rate over 30.0%, with the peak performance of 50.0% true positive rate on yellowBalloon.
Question 2: Demonstrate your process for correcting classification errors using visual analytics. How do you represent confidence and uncertainty? How could the correction process be made more efficient? Please limit your answer to 10 images and 500 words.
Our error correction process follows Shneidermann’s Visual Information-Seeking Mantra (overview first, zoom and filter, details on demand). There are 4,412 total initial predictions (a bounding box on an image with predicted object class and confidence score). Each is shown using a circle in the appropriate beeswarm plot (for its predicted class) with vertical position representing confidence score. 23 classes initially had predictions, so there were 23 beeswarm plots (drawn using “neutral colors”) at load time (Fig 2.1). As Fig 2.1 shows, most predictions (even before error correction) have low confidence.
To support an efficient correction of classification errors, we employ a two-step process: (1) Using a design inspired by reCAPTCHA (Fig 2.2), users batch mark predictions as correct/incorrect. (2) Using a second interface (Interface #2), misclassified labels are relabeled to the correct object.
For (1), clicking a beeswarm’s reference image opens all prediction images for that class, ordered by decreasing confidence score. The user scans the images and clicks ones that are co correctly classified, turning their borders green. If needed, hovering on an image highlights it in the beeswarm and shows a magnified version for detailed review. When all correct images are clicked, a submit button batch labels them as “correct”; all other images for the object are labeled “incorrect.” Fig 2.3 shows this process.
After going through the 23 predicted classes, all predictions are either “correct” or “incorrect.” Incorrect labels need to be updated to the correct class (or deleted). To do this, we designed a second Interface #2 (Fig 2.4).
In interface #2, the user selects a person to load their images. At right, class labels are shown for each image either as green (correct) or red (incorrect). Clicking an image highlights it and lets the user update or add to its labels. (Like Interface #1, multiple images can be selected and batch relabeled.) Fig 2.5 walks through an example.
Interface #2 allows us to quickly relabel misclassified images for each person. Upon completing this relabeling process, every image is correctly labeled with no uncertainty (based on user review) and we have a complete network of person-object affiliations (i.e., the object shows up in a person’s picture). We use an Interface #3 to analyze this network and determine the totem and which subgroup of people it is affiliated with.
We can enhance the efficiency of the correction process by automatically determining whether a prediction is correct based on computed image similarity, measured by metrics such as normed correlation coefficient, between an example image of the predicted category and an image predicted to have an object in that category. The machine-determined correct results and incorrect results are then presented to human analysts for inspection. This automation shrinks the mental burden of carrying out a binary judgement on every image. Instead, the analyst only needs to select incorrect / correct results from those recommended as correct / incorrect.
Question 3: Characterize the distribution of objects across the forty people.
Question 3a: Which people have which objects? Please limit your answer to 8 images and 250 words
We review object distributions among people using an interactive scatter plot and Interface #2. We present two example cases of our analysis.
In Fig. 3.1, Person33 has 7 objects identified across 61 photos, on average 9 photos per object, which hints a potential strong affiliation between this person and some objects. To verify our hypothesis, we switch to Interface #2 and inspect Person33 (Fig 3.2). The object turtle stands out in the word cloud, as it appears 28 times in Person33’s 61 photos. Scrolling the label list and referencing the image gallery shows what other objects are present: foamDart, giftBag, brownDie, and redWhistle, hairRoller and pinkEraser. The same process can be replicated when we want to analyze in detail a person and his objects.
The scatter plot also reveals Person39 to be interesting, with 8 objects among 12 photos (1.5 photos per object). Using Interface #2, we see Person39 has 4 legoBracelet, 3 canadaPencil, 3 redDart, 3 miniCards, 2 blueSunglasses, 2 foamDart, 1 voiceRecorder, 1 vancouverCards. In contrast to Person33, this person only has a few photos and no “dominating” object they are strongly affiliated with.
Question 3b: Identify groups of people that have object(s) in common. Please limit your answer to 10 images and 500 words.
We introduce Interface #3 (Fig 3.4) to help answer this question. The primary visualization here is an ontology node-link diagram, where nodes are either persons (blue) and objects (gold). An edge between a person and object indicates an affiliation: the person has that object in at least one of their images. Edge width encodes the affiliation significance, measured as Number_of_Images_With_The_Object / Total_Images. When a person node is hovered on, the nodes of people having objects in common with this person are highlighted and so do those nodes of common objects. Other nodes are hidden to enhance focus. Similarly, hovering on an object node makes only nodes of people affiliated with the object visible. Two filter bars beside the graph provide min/max edge filtering for objects.
Other visualizations include a bar chart showing the number of people affiliated with each object; e.g. miniCards object has affiliations with 13 people. An adjacency matrix encodes affiliation similarity between people. Clicking a person’s label sorts the matrix rows and columns based on the number of shared affiliations (Fig 3.5). The sorting function enables us to clearly see the subgroup cliques each person is associated with.
Fig 2: When p6 is clicked, the matrix sorts accordingly. A strong similarity between p6 and p4 is then revealed.
As there are numerous subgroupings, fully answering the question “Which people have object(s) in common?” is impractical. We instead present two cases of subgroup exploration of people having common object(s).
(1) Scanning through the adjacency matrix (Fig 3.5 left), the cell (6,4) stands out. Sorting by Person6 shows a clique of three: Person6, Person4, Person34. We inspect individual images of Person6 and Person4 in Interface #2, which reveals the four shared objects: miniCards, yellowBalloon, hairClip and lavenderDie. We even noticed common scenes in images of these two people, indicating a strong connection (Fig 3.6).
(2) Because our ultimate goal is to identify the totem shared by a subgroup of eight people, we filter the network to only show objects with eight affiliations (eight edges to people). There are four object nodes remaining: rubiksCube, noisemaker, canadaPencil and rainbowPens. It is clear from the graph that Person39, Person14, Person15, Person25, Person 7, Person22, Person35, and Person4 have a common object canadaPencil.
Question 4: Which group do you think is the most likely group with the “totem”? What is your rationale for that assessment? Please limit your response to 5 images and 300 words.
We believe the totem is likely canadaPencil; the subgroup is the eight people affiliated with it. Our rationale is thus:
Since the totem relates to a subgroup of size eight, objects with fewer affiliations can be discarded. As the totem symbolizes secret affiliation, it should have an “important-but-stable presence” among the group. We take the affiliation significance equation from Question 3b to indicate the object’s importance to a person. Fig 4.1 shows average object importance (with standard deviation to indicate affiliation stability) for objects with 8 affiliations. We label average importance in the ontology graph, and visually inspect potential totems objects to discard those that are highly unstable. Fig 4.2 shows the inspection process for rainbowPencil; Fig 4.3 for other items.
This leaves canadaPencil, which has a high enough average importance and also the highest stability (Fig 4.3). Selecting the canadaPencil node in the graph loads eight columns of images (one per person) and any attached image captions. Interestingly, Person39 mentions the word incident in the caption for image Person39_3: “My souvenir from Canada is not very useful until I sharpen it. I had better grab my eye protection in case we have another ‘incident’ with the sharpener like last time.” At first, this caption seems to be an unfunny-but-benign pun. However, hints from the VAST Challenge description: the cyber event was “accidentally launched,” the totem is “unremarkable” and “could easily be confused for a free item handed out at a conference,” along with the fact that the VIS 2019 conference was held in Canada, provide strong circumstantial evidence of canadaPencil being the totem. Therefore, we conclude that the subgroup of eight people affiliated with canadaPencil is likely the subgroup behind the CGCS bug (Fig 4.4).
Question 5 (process question): Did you choose to use the object recognition model results provided or use your own machine learning algorithm? Why did you make that choice? What was the biggest challenge you faced? Please limit your response to 3 images and 300 words.
We started with the given model’s results, but implemented Interfaces #1 and #2 to correct misclassifications. While we could have trained our own ML model for classifying objects in images, this deviates from the primary task of the Mini-Challenge (using visual analytics to find the totem), and in a real world cyber-forensics scenario, training a new model might not be a feasible option. Since this was the VAST Challenge, we thought it would be an interesting and fun exercise to design visual analytics to interactively and efficiently relabel mispredicted images.
That said, our biggest challenge was ensuring that the image classifications could be efficiently reviewed for correctness and relabeled when incorrect. Our initial interface, a straightforward design where the user loaded and relabeled images one at a time, was stiflingly tedious and slow (an initial relabeling of one class averaged 1 minute per image, which would equate to 15 hours for the 900 images in the dataset). Our solution was to make the relabeling a two-part process. In Interface #1, we select the group of images for a single object and batch relabel them as correct/incorrect. This changes the problem from, “which objects does an image have?”---a multidimensional search problem---to one of, “which images have the target object?”---a binary decision problem that is well-suited to human perception (reCAPTCHA similarly takes advantage of this). Then, in Interface #2, we can focus on reviewing and fixing mislabeled images for each user (again using batch label updating), before doing the final analysis and totem identification with Interface #3. Using this approach and Interfaces #1 and #2, our team member finished reviewing and relabeling all 900 images in ~1 hour.
