VisMe can be found here, developed by the VisMe team, University of Konstanz - Applied Visual analysis Course, taught Summer
2020 by Prof. Dr. Daniel Keim,
Eren Cakmak, Udo Schlegel, Juri Buchmuller, to adapt the challenge
Approximately how many hours were spent working on
this submission in total?
180
May we post your submission in the Visual Analytics
Benchmark Repository after VAST Challenge 2020 is complete?
With
your help, the Center for Global Cyber Strategy (CGCS) searched the vast graph
of offline records and narrowed its list of possible groups involved in the
accidental cyber event. Further analysis of the cyber event has given a strong
indication that a subgroup of eight individuals were behind the bug. The CGCS
has received a tip that this group rarely meets in person and instead they use
a special item - a totem - as a secret signal of their affiliation. This item is
unremarkable to those who are not part of the group; it could easily be
confused for a free item handed out at a conference. The CGCS gathered all
records associated with the group from their offline archives including text
and photos that potential group members posted on the Y*INT social media
platform. Investigators at the CGCS need your help analyzing the contents of
those records to look for clues to the identity of the eight individuals.
Declaration
To adapt the challenge, we have
developed a visualization tool named VisMe to fully support the provided dataset. Moreover, it can be used
to visual analytics any object recognition dataset that followed the same format of the provided dataset.
The tool has two parts: The visualization part and the Correction part. Regarding the Visualization part, we
provide from the general to the specific view of objects distribution in people and their important
attribute as confidence score. In the Correction part, we concentration on presenting the uncertainty of the
classifier results and providing the tool to allow the users to correct the misclassified cases with the
active learning approach with machine learning-based algorithm to make the process more
efficient.
1. Examine
the object classifier you used either from the classifier results provided or
the results from a classifier you chose. Which objects classified well and
which more poorly by the machine learning algorithm? Please limit your answer
to 5 images and 250 words.
Both the provided classifier and our classifier use the confidence score to evaluate how confident an
object's class is detected by the model at a certain position. Theoretically, an object with a high confidence
score should have a higher chance to be classified correctly than the one with a low confidence score. However, as we
don't have the ground truth to evaluate the performance of the classifier, they all have a base level of
uncertainty. As in the figure below, we show the true negative case, a wrong prediction
with high confidence score and the right prediction with a low confidence score:
Fig.1 - Uncertainty in confidence score with two typical cases: Wrong prediction with high
confidence score on the left and correct prediction with low confidence score on the right
Therefore, we assume the following well-classified definition that must satisfy two
conditions: Firstly, the object has to be recognized correctly by the classifier (the object is assigned the
correct label and its bounding box covers more than 70% of its surface); Secondly, the confidence score which
represents the classifier's confidence in its prediction should be larger than 0.5. If all of
the objects are not satisfying the above constraints, we will have poorly classified objects. There are two ways to
get the desired behaviour:
As we have 21 out of 43 recognizable objects, the first method is to go through each object (we can take the object
from the top highest confidence score as in figure 2 first) and apply the check
process: find the person where the object has the highest appearance frequency -> check with our
constraints.
The second method is to go through each person and check which object satisfies the constraints.
Fig.2 - Top highest confidence score objects
As the result, we have the list of well and non-well classified objects as follows:
Well classified objects: noise maker, blue sunglasses, pumpkin notes, sign, red whistle, yellow bag,
lavender die, eye ball
Poorly classified objects: the rest from list of objects referenced to the training corpus
Fig.2.1 - Well classified objects: noise maker, pumpkin notesFig.2.2 - Well classified objects: sign, red whistleFig.2.3 - Poorly classified objects: bird call, pink eraser
2. Demonstrate
your process for correcting classification errors using visual analytics. How
do you represent confidence and uncertainty? How could correction process be
made more efficient? Please limit your answer to 10 images and 500 words.
Confidence score and classifier uncertainty are the two important keywords that we have mentioned in the
previous question. To visualize them, we use data mining and visual analytics approaches to draw out their
characteristics. Regarding the confidence score, firstly we present the top of the highest and lowest
confidence score objects as well as additional information such as their name, the person they belong to quantity and highest / lowest score.
Fig.3 - Top highest and lowest confidence score objects
These tables give the user a quick overview of the extreme values of each object (highest and lowest). In
addition, to further investigate the confidence score of each object in a certain person, we provide an
interactive bar chart section to show the potential "totem" for each person as seen in Figure 4. In this section,
we present the
distribution of confidence score as a frequency of appearance for each object to get an overview of how confidence
score is different between each object in a person and how confidence score is different between each person
for a typical object. Moreover, we can arbitrarily select persons from the drop down menu to get a detailed
visualization for each person.
Fig.4 - Confidence Scores and Appearance Frequency distribution for Person 2 (left image) and
Person 1 (right image). On the right hand side of each image, we provide a statistic of the top 5 most
appearing objects for the selected person to quickly give users an overview.
To adapt to the classifier uncertainty, we apply the correction process firstly to check how uncertain the
classifier is and secondly to provide the ground truth for further enhancement. At the above-mentioned section
at the dashboard page of the application, we provide a "Detail" button which leads to the uncertainty checking
and manual annotation page as seen in Figure 5.
Fig.5 - Annotation Page
At this page, we check the uncertainty based on checking the label, confidence score
of an object and its corresponding bounding boxes. We can either approve or disapprove of the predicted result.
In the case of rejection, we can manually annotate and assign the label for the objects. The
annotation process is shown in Figure 6.
Fig.6 - Annotation Process. From left to right, click Reject -> draw bounding boxes -> assign
label
This approach can be used to correct the predicted result as well as to
provide new object classes which are not recognized but appear in the training set as in Figure 7 and the
new object
classes which do not appear in the training set as shown in Figure 8 further below.
Fig.7 - New object class (Failed to recognize but appears in the trainings set as
turtle)Fig.8 - New object class (a completely new object which isn't refereced in the
training set)
To make the correction process more efficient, we propose and apply an active learning approach. The structure
of this approach is presented in Figure 9 bellow. In this technique, we use the term semi-correction to apply
the semi-supervised machine learning algorithm. Instead of annotating the whole dataset, the users can choose to
correct a part of the falsely labeled data (in this case the rejected and newly labeled image). The manual annotation will be
combined with the training dataset as an input to the model when we click the "train model" button. After an
epoch, the predicted results will be updated and the users can then once again evaluate the result and provide annotated
samples. This recursive process continues until it satisfies the requirements as the example in Figure 10 shows.
Fig.9 - Active learning approachFig.10 - Active learning approach result
3. Characterize the
distribution of objects across the forty people.
Which people have which objects? Please limit your answer to 8 images and 250 words
In our application, we present the distribution of objects in people in the two following ways: On the one hand we have a more general
way and on the other hand a more detailed way. In the General way, we present all objects and people in a pairwise table. The row of
the table represents a certain person while the column represents a certain object and the cell has the
value of appearance frequency value. All values are normalized in the range of [0,1] in their respective local scales.
Moreover, all the values are also encoded with the color range from white to red to make it more
expressive. An example for this pairwise table is seen in Figure 11. To make the table more interactive,
we also make the table sortable by simply clicking on the column you want to sort with. In the
example in Figure 12, we have sorted the table in a descending manner for object 2 (canada pencil). With this table we
aim to provide a general overview of the object distribution among people and provide some interactivity to look
for the most probable person for every object to make further tasks more easy.
Fig.11 - Pairwise Table: All values are normalized in local scale and encoded with different shades of redFig.12 - Sorted Pairwise Table by item no. 2 (canada pencil) - Person23 has the highest chance to have the
canada pencilFig.13 - Sorted Pairwise Table by item no. 5 (eye ball) - Person23 again has the highest chance to have the eye
ballFig.14 - Sorted Pairwise Table by item no. 8 (metal key) - Person23 also has the highest chance to have metal
key
In the Detailed way, we present the distribution of objects for each specific person. As we are using the results of a machine
learning classifier, we indicate the result by using the term "high chance" because this result is always an
approximate result instead of an absolute one. The distribution of objects is mentioned as the caption
for the following examples:
Fig.15 - Person 4 has the highest chance to have the eyeball (item no. 5) - Person 8 has the highest chance
to have sign (item no. 15)Fig.16 - Person 14 has the highest chance to have sign (item no. 15) - Person 17 has the highest chance
to have cloud sign (item no. 3)
Identify groups of people that have object(s) in common. Please limit your answer
to 10 images and 500 words.
To answer this question, we divide it into two parts. In the first part, we answer the
question of identifying the group of common objects given a group of people. To answer this question, we
provide an interactive bipartite graph that allows us to arbitrarily select people to show their
relationship between objects and other people. Moreover, we also included an
example image for every object to make it more recognizable for the users. In the following picture, we
show some potential groups of people and their corresponding common objects:
Fig.17 - The group of people (1, 6, 9, 12, 14, 17) has the high chance to have the following common
objects: bird call, eye ball, pink candle, pumpkin notes, blue sunglasses, yellow bag, cloud sign, red
whistle, sign, hair clip, metal key, lavender die, cupcake paper (item no. 0, 1, 3, 4, 5, 6, 7, 8, 12, 13, 14, 15 and 20)Fig.18 - The group of people (25, 31, 21, 7, 38, 39) has the high chance to have the following common
objects: cloud sign, eye ball, pink candle, hair clip, bird call, pumpkin notes, red
whistle, blue sunglasses (item no. 0, 1, 3, 5, 6, 12, 13 and 14)
In the second part, we answer the question of identifying the group of peoples given the common objects. As in the first part,
we also provide an interactive bipartite graph that allows us to select arbitrary objects to show
their relationships to the people. This way you can see a group of people that share the same object.
Below are some examples:
Fig.19 - All of the following items blue sunglasses, cloud sign, hair clip, cupcake paper, metal
key, lavender die, red whistle (items no. 1, 3, 4, 6, 7, 8 and 14) are shared by the following group of people
(1,2,3,4,5,6,7,9,11,12,13,14,16,17,18,20,23,24,25,29,32,33,34,37,39)Fig.20 - Typical failure cases: catus paper and foam dart have not been recognized by the classifier,
therefore there aren't any people sharing this item
4. Which group do you think is the most likely group with the
"totem"? What is your rationale for that assessment? Please limit your response
to 5 images and 300 words.
Theoretically, we could show a group of people that share potential "totems". However, the
classifier has a high uncertainty in the predicted results. Therefore, we use a combination of the following
methods to figure out the potential group with the "totems". First we select the potential objects from the
list of objects that are well classified by the classifier because they have a high chance to be a "totem"
used by the people. Then we use the bipartite graph to figure out how many people there are that share this potential
"totem". As we don't have absolute ground truth to know the exact objects that belong to each person, we
propose the way to predict the objects to have the highest chance to be a "totem" for a certain person is based on their
appearance frequency. Therefore, we use the pairwise table one more time as in figure 22 to rank the
person. The higher the appearance frequency of an object is, the higher rank of the person increases
Therefore, the potential people for the "totems" in order are 23, 17, 34, 5,
37,4,3,2,25,15,36,14,12
Fig.21 - The group of people that share the potential "totems"Fig.22 - Pairwise object person table for ranking people based on the appearance frequency of
the potential "totems"
5. Process question: Did you choose to use the object recognition model results provided or
use your own machine learning algorithm?
Why did you make that choice? What was the biggest challenge you faced? Please limit your response to 3
images and 300 words.
We chose our own machine learning model. This is due to the fact that the provided output was more or less
random squares in random parts of the picture with random labels. Furthermore, there was only a subset of all classes detected
in the whole data set. This way analyzing the data and making (correct) deductions from it becomes impossible.
The biggest challenge for building a model is the limited amount of training images that are present (12 images per
class). This is by far not enough to build a working image detection model (like YOLO, ..) where one would
need at least 1k+ images per class to get a satisfactory result. This is why we chose a One-shot-learning
approach in combination with an active learning functionality. The plus side of the One-shot-learning approach
is that it needs far less examples per class to function in a sufficient manner. Basically the model just
compares two images and gives you a similarity score as an output. By comparing the similarity scores of all the different classes
one can easily create their own classifier.
As explained earlier, we
provide the user with some output, which can be correct or false, and if the user wishes, can correct the
output. In this case the user can draw a rectangle around a piece of the image and annotate the class. This is
then added to the training images and used in the background training.
Therefore the user iteratively increases the performance of the model which in return makes it easier for the
user to find the groups and / or totems.
Fig.23 - Classical outline of the one shot learning siamese network Fig.24 - Proposed interaction between the user and the model
