Sarah
Mittenentzwei, Department of Simulation and Graphics, University of
Magdeburg, Germany, sarah.mittenentzwei@st.ovgu.de PRIMARY
Benedikt
Mayer, Department of Simulation and Graphics, University of
Magdeburg, Germany, benedikt@isg.cs.uni-magdeburg.de
Kai Lawonn,
Institute of Computer Science, University of Jena, Germany,
kai.lawonn@uni-jena.de
Bernhard Preim, Department of Simulation
and Graphics, University of Magdeburg, Germany,
bernhard@isg.cs.uni-magdeburg.de
Monique Meuschke, Department of
Simulation and Graphics, University of Magdeburg and Institute of
Computer Science, University of Jena, Germany,
meuschke@isg.cs.uni-magdeburg.de
Student Team: NO
MatLab 2020a
Approximately how many hours were spent working on this submission in total?
280
May we post your submission in the Visual Analytics Benchmark Repository after VAST Challenge 2020 is complete? YES
Video
Questions
Given the images and text files as well as machine learning outputs, use visual analytics to answer the questions below to improve and understand machine learning outputs and track provenance, uncertainty, and confidence in machine learning results. Ultimately, you must link multiple data types to identify the group CGCS is seeking.
1 – Examine the outputs from the model – either from the detection results provided or the results from a model you chose. Which objects were identified well by the model and which were not? Please limit your answer to 5 images and 250 words.
The provided classifier detected 22 of the 43 object classes. The results include a lot of false positive detections where objects are detected although they are not in the image (see Figure 1).
We calculate an average score per object class by taking the mean score of all detections of this class. The results are plotted as bar chart (see Figure 2) in our developed user interface (UI). The two lowest ranked classes are cupcakePaper and canadaPencil which both have a score of 0.31. The class paperPlate has the highest score with 0.53.
Even for the relatively high ranked object paperPlate only 13 of the 22 detection results were correct. It shows that a huge part of the detection results is not correct. This might be due to the small amount of training images. For some object classes the training images do not vary in scale (e.g., foamDart), rotation (e.g., cactusPaper), or translation (e.g., cloudSign). The training images are not representative for the images to be classified since in the latter objects are shown in much more variation and are sometimes hard to identify even for the human eye.
Similar to the score per object an average score per image is calculated (see Figure 3) and plotted in the UI. 13 images have a score of 0 because the detector did not find any objects in them. The other 894 images have scores between 0.25 and 0.85.
2 – Demonstrate your process for using visual analytics to correct for classification errors in the results. How do you represent confidence and uncertainty? How could the correction process be made more efficient? Please limit your answer to 10 images and 500 words.
Our UI has two main parts (see Figure 4). On the left side the user sees the images and bounding boxes of the detected objects. The bounding boxes can be edited, such as rescaled, repositioned, renamed, recolored, deleted or the user can create new bounding boxes. To simplify the renaming of the bounding boxes the user can choose a label from a list. This list has all currently used labels in it, but also offers the user to add new labels to the list. It is also possible to mark existing bounding boxes as correct to set their score to the current maximum. The image that should be displayed can be chosen from the list below. For each bounding box the label and the score are visualized.
On the right side meta information from the analysis of the dataset is shown to the user through different visualizations. The Score tab shows the score distribution over all the images. Below there is another list of the images sorted by score. This way the user can start the correction process with the images that have the lowest score and therefore are likely to have wrong classification results. If the user wants to focus on the empty images there is another list only showing images with score 0.
The user can choose a minimal score to remove uncertain detection results with a lower score than the selected threshold. In this way, many detections that have a low score and are therefore probably wrong can be deleted with one click.
In the History tab the user has an overview over the improvements reached so far. A graph shows the score improvement over the whole dataset in each interaction step which is calculated by taking the mean of all the scores of all detections (see Figure 5). Additionally, the user can see the overall score improvement in percent from the original data until the current moment. This way the user can improve a certain amount of the data, e.g., a 10% quantile of the score distribution.
For further improvement of the correction process similar images could be grouped and presented to the user. Similar images tend to have similar classifications as well as classification errors which might be easier to focus on for the user than to switch between strongly varying images.
Apart from showing images based on their score, it might also be helpful to present all images that contain a certain object class. In a similar way, this would allow the user to quickly scan images that were detected to include the same object. Like captcha tests, the user could be queried to click on all correct images or even double click on images to instantly correct their bounding box. The process could be finished by clicking on an accept button which will mark all selected images as being correctly classified and remove the remaining bounding boxes.
3 –– Characterize the distribution of objects across the forty people.
a. Which people have which objects? Please limit your answer to 8 images and 250 words
After doing 200 correction steps the average score of the whole dataset increased from 0.37 to 0.47 (see Figure 5). We used the corrected data for the further analysis.
An overview over the distribution of the objects among the persons is shown in a heatmap (see Figure 6). The axes of the heatmap are sorted by hierarchical clustering using complete linkage criterion so persons and objects that have a similar distribution are close together. The dendrogram (see Figure 7) of the hierarchical clustering gives further inside about the similarity of the different persons. We color-coded subtrees that are below a given linkage threshold to provide an easier interpretation of the dendrogram. The threshold can be adjusted by the user below the dendrogram.
A bar chart indicates the number of detected distinct object classes per person (see Figure 9). Similarly, a second bar chart shows the number of people for which the same object has been detected (see Figure 8). Given the detector’s output, the minimal number of persons for which the same object class has been detected is 10 (paperPlates and vancouverCards). In contrast, the object classes birdCall, blueSunglasses, cloudSign, eyeball, pinkCandle, and pumpkinNotes have been found to be present in images of all 40 persons. In terms of unique object classes per person the minimal number was 12 (cf. person 8) and the maximum number 22 (cf. person 23 and 5).
b. Identify groups of people that have object(s) in common. Please limit your answer to 10 images and 500 words.
To find out which persons have similar objects, the user can switch to a binary visualization of the heatmap (see Figure 10) and the dendrogram (see Figure 11). Instead of using the object counts, the binary view only considers if an object has been detected at least once per person.
The dendrograms in Figure 11 show that different groups of people can be found depending on the linkage threshold used. A small threshold produces a lot of small groups while a bigger threshold results in fewer large groups. These groups are formed of persons that have many objects in common. There are some objects that occur a lot of times: birdCall, cloudSign, blueSunglasses, pumpkinNotes and eyeball are detected in the images of every person. Therefore the groups are mainly formed by objects that co-occure less frequently.
In Figure 11 a) the upper red cluster consists of Person23, Person5 and Person13. These persons have more than half of their objects in common (see Figure 10). With the exceptions of Person23 having brownDie and rainbowPens, Person5 having pinkEraser and Person13 missing noisemaker. With continuously increasing size of the cluster more exceptions are accepted to form a group of persons that have objects in common, e.g. in Figure 11 b) the purple cluster.
To find groups of persons that have a single object in common, the user can specify a range of interesting person per object counts. The objects that are owned by such an amount of people are called candidates and are highlighted in light blue. In the persons tab all persons that own these objects are highlighted light blue as well (see Figure 12).
By clicking on one bar in the bar charts in the Object tab, the bars for this object are highlighted in red (see Figure 12). The plots are linked, so that in the other tabs the same object or the persons where this object was detected are highlighted as well. Since for the dendrogram, color is already used to enhance interesting subtrees, selected persons are highlighted by a dashed line (see Figure 13). After selecting the object class giftBag, Person3, Person11 and Person33 are highlighted which shows that the group of these three people have this specific object.
The user can also select a bar from the charts in the Person tab which highlights this person or the objects detected in the images of this person in the other plots.
4 –– Which group do you think is the most likely group with the “totem”? What is your rationale for that assessment? Please limit your response to 5 images and 300 words.
Since we are searching for a group of 8 people we searched in a candidate range from 6 to 10 people per object. We allow for some inaccuracies due to people carrying objects at random or the investigator missing an appropriate photo of a person carrying the totem object.
Candidates with a high score are considered more trustworthy and are therefore more likely to be the correct result than candidates with a lower score. Since the score per object is highly influenced by each detection result for objects with only a few occurrences we also look at the scores of the persons that own these objects.
It can be seen that objects miniCards, rainbowPens, turtle and vancouverCards are interesting candidates for being the totem. vancouverCards has a lower score than the other object classes and therefore might be less accurate. The score per person shows that persons with a lower but also persons with a higher score have this object.
However, vancouverCards is the only object class that appears in a group of exactly eight persons. miniCards, rainbowPens and turtle have groups of only six persons.
Therefore, we assume that the object class vancouverCards is the totem and persons 18, 20, 39, 13, 4, 5, 23 and 34 who have it are the searched group.
Due to the classifier’s limited detection rate, the confidence in the result is low. Looking at the score overview (see Figure 16) we see that there are still a lot of images with a relatively low score that are not corrected. Since the number of persons per object can easily change with the correction of just a few images, the result after more corrections might be different.
5 –– Process question: Did you choose to use the object recognition model results provided or use your own machine learning algorithm? Why did you make that choice? What was the biggest challenge you faced? Please limit your response to 3 images and 300 words.
Even though we trained our own model, we used the object detection results provided in the dataset to solve the presented tasks. We based our model on transfer learning. Therefore, we used the pre-trained vehicle detector from MatLab and the YOLOv3 network structure. Our model manages to detect more object classes (36 instead of 22) but either detects too many objects in a single image (see Figure 17) or no object at all in more than one hundred images. If the score threshold is set too low, the same object class even gets detected multiple times in one image.
Even though we tried to use data augmentation to cope with the small amount of training data it seems that the number and variation of the training images is too low for the algorithm to train a reliable object detector.
Concerning the correctly classified images, we did not see any major improvement compared to the given classification. Since our own results force the user to either delete a lot of incorrect bounding boxes or draw boxes for many empty images, two tasks that are both very time consuming, we used the given object detection results which provided a more convenient number of detections per image.
