Entry Name: "TUK-Victor-MC2"
VAST Challenge 2020
Mini-Challenge 2
Team Members:
Viny Saajan Victor, Technical University Kaiserslautern, v_victor18@cs.uni-kl.de PRIMARY (+49 179 260 4921)
Pramod Vadiraja, Technical University Kaiserslautern, p_vadiraja18@cs.uni-kl.de
Jan-Tobias Sohns, Technical University Kaiserslautern, j_sohns12@cs.uni-kl.de
Heike Leitte, Technical University Kaiserslautern, leitte@cs.uni-kl.de
Student Team: YES
Tools Used:
Plotly for Visualizations
DASH for Web Application
Pandas and Numpy for Data Processing and Computation
NLTK for Text Processing
Approximately how many hours were spent working on this submission in total?
240 Hours
May we post your submission in the Visual Analytics Benchmark Repository after VAST Challenge 2020 is complete? YES
Video
Questions
Given the images and text files as well as machine learning outputs, use visual analytics to answer the questions below to improve and understand machine learning outputs and track provenance, uncertainty, and confidence in machine learning results. Ultimately, you must link multiple data types to identify the group CGCS is seeking.
Our Design:
1 Examine the outputs from the model either from the detection results provided or the results from a model you chose. Which objects were identified well by the model and which were not? Please limit your answer to 5 images and 250 words.
Answer:
Our tool depicts the average confidence score of each object as a bar chart to the user. On interaction with the bar graph, the user is provided with the distribution of confidence scores for each of the objects. From the graph, it can be observed that paper plate objects are classified with higher confidence and cupcake paper is classified poorly compared to other objects. Since the prediction contains many False Positives and false negatives there is a high degree of uncertainty in the visualization which is addressed by including human correction which is explained in the next answers.
We also presented a graph of variance vs mean of the confidence scores for each object(to observe uncertainties), with size of the bubble indicating average size of the bounding box to the users which reveals that objects with smaller bounding boxes were classified poorly when compared to the objects with larger bounding boxes.
We calculated the density of objects per image as follows
Density = (number of objects predicted by the model / (max(x-values) - min(x-values) * (max(y-values) - min(y-values)))
[Note: Here x-values and y-values are with respect to the bounding boxes of the objects which are predicted for the corresponding image]
And plotted confidence score mean vs density of objects per image, which reveals that objects which are cluttered are classified poorly compared to the objects which are spread out.
2 Demonstrate your process for using visual analytics to correct for classification errors in the results. How do you represent confidence and uncertainty? How could the correction process be made more efficient? Please limit your answer to 10 images and 500 words.
Answer: We formulate two processes for correcting classification errors
i) Classifier Analysis: We try to evaluate the performance of the classifier by introspecting its predictions.
a) Correlating Mean and Variances of the confidence scores for each object with its average bounding box size: It is expected that the objects with larger bounding sizes will have higher confidences, we can leverage this analysis to identify and isolate outlier points and conclude how or why such cases exist by analyzing the corresponding images. We suggest having the test images with objects of bigger bounding boxes for better accuracy or one can use a model that performs well for the objects regardless of their bounding box sizes.
b) Correlating the degree of clutteredness of objects in the image with the average confidence score for that image: We can observe that confidence score is lower for images where objects are cluttered. We suspect this is a disadvantage of YOLO (the model which is provided for the analysis) because of using only two anchor boxes in a grid. We suggest using a model which uses more anchors in a single grid.
Overview of Classifier Analysis:
ii) Classifier Correction: Rather than just manipulating the models’ internal workings, we considered the perception of the user himself while improving the models.
a)Eliminating False Positives through user feedback: It is common that some objects will be more accurately predicted by the classifier than others. One of the many reasons for this could be due to similar ambiguous features/patterns in the objects .This leads to a lot of False Positives in the data. eg: An object that is circular, orange in color has a high probability of it being an Orange. At the same time, if the model was also given a ball that is orange in color, this leads to the creation of False Positives. We present the users with a list of objects (as a Funnel Chart) in the decreasing order of them having higher chances of being False Positives, such as the objects which are predicted to be contained in many images( through selective attention) and allow them to select and eliminate the same thereby increasing the certainty of the model.
Correct Bounding Boxes: For True Positives from the previous step we provide an interface for correcting bounding-boxes. We show the reference image overlayed with the classifier’s bounding-box and label predictions on one side. The user can correct the bounding boxes and reannotate the object if they are inaccurate and use this for training the images or also can be used by our model with greater confidence on the other side.
b) Reannotate the ambiguous images: We present the user, ground-truth vs prediction mapping(where limited ground-truths were obtained from image captions) and allow the user to reannotate the ambiguous predictions. The user through inspection can recognise the images with only False Negatives, and with the help of training images, he reannotates the most ambiguous False Negatives with the groundtruth.
3 Characterize the distribution of objects across the forty people.
a. Which people have which objects? Please limit your answer to 8 images and 250 words
For this purpose, we are providing a graph that shows the histogram for each object across 40 people as shown below. This can be visualized upon the selection of the required object from the “Average Confidence Score for each Object” graph as discussed earlier. This aids the user to understand how each object is distributed amongst 40 people. With this kind of setup, the user can get a holistic view that relates the average confidence score for an object with its distribution among 40 people.
In the “Totem Analysis” tab, we have provided a graph that depicts the connectivity between people with common tokens in their captions. These tokens are further filtered to match with the available objects. This in turn can be used as another waypoint to determine the objects associated with each person.
Here, Person3 and Person12 have pumpkinsticker, yellow bag and hair clip objects in common. The common words are filtered by the objects that can be present.
In addition to the above requirement, this analysis also sheds light on an insight that we stumbled upon. For eg: On selecting the object “cloud sign” from the “Average Confidence Scores of Objects” graph, we see that the object “cloud sign” has been distributed densely amongst 40 persons, in comparison with the object “paper plate” that has a sparse distribution. However, the average confidence score of paper plate is much higher (53%) than that of cloud sign (34%). This implies, there might not be a linear correlation between the density of distribution of an object among 40 people with its confidence score.
CloudSign:
Paperplate:
b. Identify groups of people that have object(s) in common. Please limit your answer to 10 images and 500 words.
Answer:We tackle this problem by providing visualizations in the “Totem Analysis” tab . We provide two ways of determining common objects between people.
The first approach corresponds to the heatmap that shows “Object Similarity between people”. Each entry in the heatmap corresponds to a similarity score. This entry reflects the common objects between a pair of people. Upon clicking/selecting this entry in the heatmap, the set of common objects between the pair is revealed as a word cloud visualization that shows “Common objects between people”.
The second approach corresponds to a network graph visualization titled - “Person Connectivity based on captions”. Each node represents a person and the connectivity between people indicates they have a set of common tokens in their associated set of captions. With this setup, we are able to extract the common objects between people by filtering the common tokens. One can use the lasso selection tool to select the users for whom he needs to obtain the common objects.
Calculation of the heat map:
The heatmap corresponds to a similarity matrix. This matrix gives the pairwise similarities between the 40 people in terms of the common objects between them. Each person is mapped to a vector with 40 dimensions where each dimension corresponds to 40 objects. The entry corresponding to each dimension will have the frequency of the respective object if the object is present else 0. Then we took cosine similarities (which also takes care of the magnitude) and colored them based on similarity. Brighter the color indicates stronger the similarity.
Calculation of the network graph
As there was no information provided to us about the true objects associated with each image, we realized we can rely on the captions associated with each image to contain information about the true objects in the image. This is mainly due to the reason that the classifier had a lot of uncertainties as discussed earlier.
The captions were subjected to the Natural Language Processing pipeline that included the steps of tokenization (extract words from a sentence), stop word removal (remove connecting words like articles, prepositions), lemmatization (reduce different forms of a word to its root form) before being considered for further processing. We built a graph structure that connected people based on the common tokens in their captions. These tokens were further filtered out to have only the objects. Hence we had a graph that connected people only if they had common objects. We believe this style of analysis is more reliable than the former approach as the captions mostly act like a ground-truth to the objects present in the images.
4 Which group do you think is the most likely group with the totem? What is your rationale for that assessment? Please limit your response to 5 images and 300 words.
Answer:
Group - 33, 37, 7, 13, 15, 27, 29, 3
Totem - red whistle
Rationale:
We initially constructed a network graph with nodes representing persons and edges representing the common tokens in their captions.
We analyzed the above graph by extracting cliques of various sizes and got the below results:
}(1)
We constructed a map between the text content of a person with person captions and got the results:
}(2)
As seen above we got the mapping between the text contents of <29, 35, 39> with the captions of all other persons.
After analyzing the above caption-2-caption cliques and text-2-caption mappings, we deduced the below strongly connected group of 6 people:
<33, 37, 7, 13, 15, 29> }(3)
Next, we selected the above persons in the network graph and found “red whistle” to be the common token between these people in their captions/texts that mapped to an object.
We also realized in the result (2), that “Person29” was the only person who had a mapping between his textual content with the captions of <33, 3, 37, 7, 12, 13, 15, 24, 26, 27, 28, 31>. Hence, we analyzed this group a bit deeper by leveraging on our perception skills by inspecting the images of this group. We figured “Person27” had a white-colored whistle in one of the images:
and hence added him to our result (3). As “Person3” was present in both result (1) and the result (2), and also one of his images :
was very similar to the shape of a whistle, we had a hunch that he is also part of the secret group <33, 37, 7, 13, 15, 29, 27>. We also ascertained this in the heatmap by extracting common objects between “Person29” and “Person3” and got the below result:
5 Process question: Did you choose to use the object recognition model results provided or use your own machine learning algorithm? Why did you make that choice? What was the biggest challenge you faced? Please limit your response to 3 images and 300 words.
Answer:We used the object recognition model provided. Because we wanted our correction process to be unbiased and to make it sort of a generic process for any object recognition model. Using our own model might end up creating a bias in our analysis.
Since the provided model had several uncertainties, we decided to incorporate user feedback in the correction process. This decision supported our choice of using the given model’s predictions for analysis and correction. As correction through human intervention is more impactful for models with uncertainties rather than a perfectly trained model.
The biggest challenge we faced was that we didn't have the ground-truth data. So a lot of our analysis which wasn't involving the images had uncertainties. And hence we had to rely on partial ground-truth obtained from image captions and also on the user to increase the certainty. The second major challenge was that we were not able to track the model-specific corrections such as increasing training epochs, training with a wide variety of training images, and changing network architecture to suit our requirements since we didn't have the model to train.
In the correction process involving the elimination of False Positives, as the model was relatively less accurate (model’s threshold specification was 25%), several objects were associated with a large number of False Positives. Hence, we faced delays while rendering several images associated with an object during the elimination process.
As seen in the above graph, eyeball is mapped to 501 images most of which are False Positives.
____________________________________________________________________________________