Entry Name: "OVGU-Chennaboina-MC2"

VAST Challenge 2020
Mini-Challenge 2

 

 

Team Members:

Ravi Mallikarjun, Otto-von Guericke University, Magdeburg, ravi.chennaboina@st.ovgu.de   PRIMARY
Niha Mohanty, Otto-von Guericke University, Magdeburg, niha.mohanty@ovgu.de

Student Team: YES

Tools Used:

Python

R Programming

Plotly

Bokeh library for interactive Visualisation and Networkx for Edge, Node graphs.

We also created an application for drawing the bounding box and giving the corresponding label to it. The basic structure was taken from a online source. We modified the application with more advanced features and we also tried integrated it with the python code which calculates the correct classification and incorrect classifications.

Referce to the source code of basic structure: https://github.com/tzutalin/labelImg

Referce to our source code ,you can find it in BBoxLabeling folder: https://github.com/ravi419/VAST-Mini-Challenge2

Approximately how many hours were spent working on this submission in total? 80 hrs

May we post your submission in the Visual Analytics Benchmark Repository after VAST Challenge 2020 is complete? YES

Video The link to the video are:

https://youtu.be/jUbEe1US5cw

https://github.com/ravi419/VAST-Mini-Challenge2/tree/master/ScreenCast

 

 

Questions:

Given the images and text files as well as machine learning outputs, use visual analytics to answer the questions below to improve and understand machine learning outputs and track provenance, uncertainty, and confidence in machine learning results. Ultimately, you must link multiple data types to identify the group CGCS is seeking.


1. Examine the outputs from the model - either from the detection results provided or the results from a model you chose. Which objects were identified well by the model and which were not? Please limit your answer to 5 images and 250 words.


We have used the YOLO algorithm output to examine the results:
Approach: To examine any outcome, there should exist a Ground Truth. In order to generate the ground truth, we created a bounding box Labeling application through which we have labeled all the 40 person's images manually ,the application outputs a csv file containing the BoundingBox mesurements i.e x, y points and width, height of the box along with the corresponding label and filename.

Firstly, we compare the distribution of the labels accross the YOLO algorithm and the manually labeled files.
Figure 1.1 Yolo prediction count of each object.
Figure 1.2 Manual prediction count of each object.
After comparing Figure 1.1 and Figure 1.2, we can clearly see that YOLO algorithm did not classify few object class labeles such as BrownDie, FoamDart, GiftBag etc.

Secondly, we check the number of times an object label is misclassified by the YOLO algorithm. Here, we use the confusion matrix to show the misclassification of an object.
Count of YOLO predicted Lables vs Count of Manually Assigned Lables
Figure 1.3 Confusion matrix representation of missclassifications.
From the above heatmap we can see the number of times an actual label is misclassified by the YOLO algorithm. The most misclassified objects by YOLO are paperPlate as pumpkinNotes and partyFavor as cloudSign. Lets inspect the reason behind the missclassifications:
paperPlate
Figure 1.4(a) paperPlate
pumpkinNotes
Figure 1.4(b) pumpkinNotes
partyFavor
Figure 1.5(a) partyFavor
cloudSign
Figure 1.5(b) cloudSign
Further, we tried to visualise the objects that are correctly and incorrectly classified in the images by the YOLO algorithm.


2. Demonstrate your process for using visual analytics to correct for classification errors in the results. How do you represent confidence and uncertainty? How could the correction process be made more efficient? Please limit your answer to 10 images and 500 words.


With the Groud Truth that we prepared earlier, we followed the followed certain analysis steps to determine the classification errors and correcting the missclassifications by our tool.
Snow
Figure 2.1 Yellow Baloon
  • Blue bounding box: Yolo Classified BBox.
  • Green bounding box: Manually Classified BBox.
Figure 2.2 Correct classified lables vs. Missclassified lables count.
Figure 2.3 Manual image files vs. Yolo image files count.
Figure 2.4 Pie Chart for bounding box detection distribution.
Figure 2.5 Good vs. Poor classification.


3. Characterize the distribution of objects across the forty people.

A) Which people have which objects? Please limit your answer to 8 images and 250 words.


To analyse the object to person's distribution we used wordcloud library in python. The process is discussed below.
  1. From the manually classified data we extracted persons data along with the objects (labels) and number of times the label linked to that person.
  2. Store the labels in a list along with multiple occurances.
  3. Join all the elements in the list and convert the joined elements into a single string.
  4. Pass the joined string into the WordCloud library and generate the WordCloud plot.
  5. Repeat the process from step one over all 40 persons and store it in a directory.
word cloud for first six person
Figure 3(a).1 WordClouds representations of labels accociated to Person1 - Person6.
Snow
Figure 3(a).2 WordClouds representations of labels accociated to Person7 - Person12.
Snow
Figure 3(a).3 WordClouds representations of labels accociated to Person13 - Person18.
Snow
Figure 3(a).4 WordClouds representations of labels accociated to Person19 - Person24.
Snow
Figure 3(a).5 WordClouds representations of labels accociated to Person25 - Person30.
Snow
Figure 3(a).6 WordClouds representations of labels accociated to Person31 - Person36.
Snow
Figure 3(a).7 WordClouds representations of labels accociated to Person37 - Person40.
Heatmap for Persons vs. Labels
Figure 3(a).8 Persons vs. Labels distribution.


B) Identify groups of people that have object(s) in common. Please limit your answer to 10 images and 500 words


We used networks library in python to create a ntework graph and also used NodesAndLinkedEdges library to link objects to correcponding persons.
Process:
Figure 3(b).1 Interactive Plot for Objects to Persons.
Figure 3(b).2 Interactive Plot for two Objects.
Figure 3(b).3 Interactive Plot for three Objects.
Figure 3(b).4 Interactive Plot for four Objects.


4. Which group do you think is the most likely group with the "totem"? What is your rationale for that assessment? Please limit your response to 5 images and 300 words.


From the hint given in the challenge, we came to know that the analysis of the cyber event gave a strong indication that a subgroup of eight individuals were behind the bug. We rule out the cases where less than 8 persons have an item.

Below image shows the count of person's per group who have object in common:
Snow
Figure 4.1 Count of Persons per object.
If an outsider person has the totem it does not serve the purpose of secret signal anymore. Thus we filter the scenarios where an item is present exactly among 8 persons. So, we have ruled out the cases where more than eight persons have a common item. We have five such items which are shown in Figure 4.2.
Snow
Figure 4.2 Potential totem's perperso.
Figure 4.3 Probability that a person belongs to crime group.
We see that there are 11 persons with a probability of at least 0.4. It is highly likely that the majority of members of the secret group are among these 11. 1, 2, 4, 11, 12, 15, 17, 20, 30, 36, 38. Based on these probabilities, we calculated how likely it is that an item is a totem, by simply multiplying the probabilities of persons who have the item.
We see it is more likely that the 'noisemaker' is the totem. Interestingly, among the 11 persons with high probability, the persons 1, 2, 11, 12, 15, 20, 36 all have the 'noisemaker' object. Further strengthening the idea that the 'noisemaker' is the totem and the group of 8 persons 1, 2, 11, 12, 15, 20, 36, 37 are the secret group.


5. Did you choose to use the object recognition model results provided or use your own machine learning algorithm? Why did you make that choice? What was the biggest challenge you faced? Please limit your response to 3 images and 300 words.

Yes, we chose to use the object recognition model results provided for the purpose of the visualization task. We decided to use the given model results as our primary task was to vizualize these results for better interpretation.

The biggest challenge that we faced was to perform the vizulization task. We developed an application which is used to re-classify and compare the classification results of the YOLO algorithm. The images that are classified by the YOLO algorithm had sufficient details for us to evalute its performance. With help of our application which is based on python, when an image is opened, there is a provision for drawing bounding boxes and labeling them with the available Label names within the training data. In this application when an image is opened, its corresponding bounding boxes (which are provided by the YOLO algorithm) are projected on them. Thus, this made it easier for us to know whether the image classification was accurate. If the classification is incorrect then we can delete or resize or re-label all the present bounding boxes. With the help of this application we are also able to draw a new bounding box on the image if required. After saving the changes we can immediately know the overall classification and mis-classication rate through our application.

The three images to justify our approach are:
Bounding boxes represented by YOLO Algorithm
Figure 5.1(a) Bounding boxes represented by the YOLO algorithm.
Bounding boxes rectified manually with the help of our Application
Figure 5.1(b) Bounding boxes rectified manually with the help of our application.
No bounding box identified by YOLO Algorithm
Figure 5.2(a) No bounding box identified by the YOLO algorithm.
Drew one correct bounding box manually with the help of our Application
Figure 5.2(b) Drew one correct bounding box manually with the help of our application.
Multiple bounding boxes are identified by YOLO Algorithm
Figure 5.3(a) Multiple bounding boxes are identified by the YOLO algorithm.
Rectified the correct bounding box and deleted the rest of the irrelevant ones manually with the help of our Application
Figure 5.3(b) Rectified the correct bounding box and deleted the rest of the irrelevant ones manually with the help of our application.