Entry Name: VAST-MC2-Kishan

VAST Challenge 2020
Mini-Challenge 2

Team Members:

Kishan Bharadwaj Shridhar, Singapore Management University, kishanbs.2016@mitb.smu.edu.sg PRIMARY

Student Team: NO

Tools Used:

Python for data processing

Plotly for graphs

Dash for front end

Power BI for quick exploratory Data Analysis

Approximately how many hours were spent working on this submission in total?

120

May we post your submission in the Visual Analytics Benchmark Repository after VAST Challenge 2020 is complete? YES

Video

https://drive.google.com/file/d/1S22GR24w5-o_iLqmUb5Q15Ap7PROjrRz/view?usp=sharing

(Might be better to download the MP4 file instead of playing in-browser)

Interface

The interface developed in response to this work, where the majority of the graphs below are from, can be viewed at https://mc2app-kish.herokuapp.com/. This will allow users to view the results presented below in an interactive manner.

Questions

Given the images and text files as well as machine learning outputs, use visual analytics to answer the questions below to improve and understand machine learning outputs and track provenance, uncertainty, and confidence in machine learning results. Ultimately, you must link multiple data types to identify the group CGCS is seeking.

1 – Examine the outputs from the model – either from the detection results provided or the results from a model you chose. Which objects were identified well by the model and which were not? Please limit your answer to 5 images and 250 words.

According to the logic commonly followed in Object Detection evaluations, the first goal was to isolate the error analysis into classification errors, and subsequently into localization errors. From the model detection results provided, it was found that only 22 of the 43 possible classes were detected by the classifier. Subsequent checks by a human annotator confirmed that there were more classes detected in the set of all test images. A confusion matrix view of the classifier performance is possible due to this extensive annotation process by a human. The process is laid out in more detail in Q2. The four typical categories with their interpretation are provided in the table below. In this case, classifier has produced a label for each of the file. For the 12 files with empty bounding boxes, analysis has been done to see which objects are found in the image as well. So, all these are bunched together into the category called ‘False Negatives’. Hence, the assumption is that there are no true negatives in this case as per the below definition.

Classifier vs Actual object presence matrix	Object is found in the image	Object is not found in the image
Classifier has produced the label	Category 1: True Positives (Classifier can identify the image, with a certain degree of confidence, and this is also validated by a human annotator who has checked the image and confirmed its presence)	Category 2: False Positives (Classifier has identified the object to be present in the image, but on validation by a human annotator, it is noted that it is not present in the image)
Classifier has not produced the label	Category 3: False Negatives (Classifier has not produced a list of labels, but on validation by human, it is found that there are other objects which are detected which the label has not produced)	Category 4: True Negatives (Empty images which do not have any relevant objects present in training set, and classifier has not produced the labels for the same)

Table 1: Confusion matrix used to interpret the given object detection model results

The count of objects falling under each category can be viewed interactively. As for the objects identified well by the system, which are the True Positives, it is seen that blueSunglasses, redWhistle and eyeball are detected well in majority of images across the set.

Figure 1: True Positives detected by the classifier (can be viewed in the UI for better clarity)

In terms of objects which were assumed to be detected, but are not actually present, the classifier has been confused the most when it comes to detecting eyeball, sign and cloudSign.

Figure 2: False Positives detected by the classifier

Figure 3: Objects that were not detected by the classifier, also termed as False Negatives in this case (view entire image in UI)

The analysis above shows the number of instances the classifier missed out on identifying a certain object, despite it being present in the image. The presence is confirmed by the human annotator, and this is essentially what we can term as the ones that were not identified and hence led to False Positives. The dominant category of images in this case belong to rainbowPens, turtle and rubiksCube.

Astoundingly, none of these objects form a part of True Positives, meaning regardless of the nature of the image, whether these objects in the image were visible/not, the classifier has failed to detect them completely!

Focusing on the part where the classifier performed well, the rank of each object in the list of all objects detected in an image is plotted on a box plot. Calculation of the rank is done as below. Once the rank is calculated, visualisation for the distribution of rank for objects identified as True Positives, are shown through a box plot as below.

File Name	Objects detected by classifier	Score	Rank
Person 1_7.jpg	eyeball	0.25584	6
	pumpkinNotes	0.50089	2
	Sign	0.53331	1
	silverStraw	0.41412	4
	yellowBalloon	0.41576	3
	hairClip	0.32095	5

Table 2: Calculation of rank to evaluate how well True Positives were detected

From the below, it is clear the median rank of paperPlate, pumpkinNotes, sign and redWhistle was 1, meaning that on average, they were detected the best by the classifier. In other words, there is higher chance that the detector allocates the highest confidence score to paperPlate as opposed to metalKey when both are found in the same image.

Figure 4: Box plot showing median rank of objects identified as True Positives.

Since a human annotator is able to draw bounding boxes over those objects identified as True Positives, the calculation of IoU(Intersection over Union) score is possible. From the below, it is clear that the detector was able to accurately localize yellowBalloon, paperPlate and cupcakePaper very well. Do note that this was based on a single iteration of drawing the ground truth box by an annotator, but the interface is able to provide such insights on localization accuracy.

Figure 5: IoU score for objects identified as True Positives

2 – Demonstrate your process for using visual analytics to correct for classification errors in the results. How do you represent confidence and uncertainty? How could the correction process be made more efficient? Please limit your answer to 10 images and 500 words.

While there have been improvements in object detection understanding common everyday objects such as fruit/trees/animals etc., when it comes to domain specific data, of uncommon objects such as those in the set provided, human in the loop learning can be effective in debugging and understanding the performance pitfalls of object detection models. The process for correcting classification errors follows the below pathway.

Figure 6: Leveraging human in the loop to understand performance pitfalls of object detection models

Drawing bounding boxes on images using detector results

A very good library to draw for drawing bounding boxes based on coordinates, confidence of detector, was utilized to draw bounding boxes on all images. These images were all uploaded to the cloud, from where they could be read into an interface, for a human to validate. The correction process follows the cycle below. The two starting points of the flow called ‘Annotation Explorer’ and ‘Checking IOU’ refer to the corresponding tab names in the interface developed for this challenge.

Figure 7: Architecture behind the annotation process offered to a human

Human input to validate identification

On selection of a person’s set of images, the score graph showing the list of objects sorted by the detection confidence, and the annotated image with bounding boxes is shown. The human annotator can use the checkboxes below to confirm the presence of a set of objects in the image.

Figure 8: Screengrab of the annotation explorer interface, which allows a human to validate a set of detections provided by an object detection model (please see video for a demo of annotation process)

Bounding boxes are powerful visual mechanisms which help to understand where in an image, an object has been identified and localized. Using these, the human annotator can quickly validate if the classes said to be detected, are found in the object. To ease the annotator into understanding the set of 43 images, a training explorer tab is provided. The radio button selection in the training explorer can help them to familiarize with the objects they are dealing with.

Figure 9: Training explorer tab

Incorporating an annotator’s subjectivity into the analysis

Additionally, an annotator’s input can be backed by some subjective facts or remarks. So, a region for annotator to makes notes on the rationale behind their judgement, is also added. Parsing these texts gave insights on which objects are being misclassified and identified wrongly. This data was used in understanding False Negatives above. (‘not_detected’ column below).

Figure 10: Notes on a particular image can help provide insights on what objects were misclassified (screengrab of the backend xlsx stored in cloud which captures the human inputs)

Calculating IoU for True Positives

While checkboxes focus on helping to understand classification performance, the IoU tab has a canvas, where images identified as True Positives in the Annotation Explorer are carried forward. Here, the annotator can draw a rectangle upon the objects which were identified as True Positives. A blank view of the image is shown to the annotator, and an illustration of the drawing process is as shown below. In case of presence of multiple classes, annotator is provided a radio button which can be used to draw for each class. These bounding box coordinates are saved in a file, from where the IoU scores can be computed. This output was summarized in Figure 5 above.

Figure 11: GIF showing the process to draw bounding boxes to calculate Intersection over Union for images identified as True Positives

Notes by the annotator is one way to identify gaps in detector analysis. In the absence of ground truth annotations and access to the YOLO v2 model layers, visualization of what each layer understood, like this, is not possible. Confidence in detections, once validated by the annotator, can be analyzed based on the median rank of True Positive images, as seen in Figure 4 above. Uncertainty can be deciphered from the model through two categories as below.

Uncertainty: Detector getting confused by similar colored objects

From the notes entered by the annotator (which in this case was me), we can see that the detector has identified redWhistle with a confidence of 0.84. But, from the notes it was understood that there was no actual redWhistle, but it was a red drum instead. One can fairly understand why the detector might have thought it was a redWhistle.

Figure 12: Confusion in understanding similar colored objects for the detector

Uncertainty: Detector getting confused by similar shaped objects:

Albeit with low confidence, it is seen below that the detector has picked up birdCall, due to the nature of the blanket, and the shape in which the black ribbon is super imposed upon a white surface.

Figure 13: Confusion in understanding similar geometries for the detector

While most of the interface is automated for the annotator to work on the most important set of images, a heatmap is added with status of the ingestion of the file into the system. Out of the total 907 image files across different people, there were about 13 files which have empty bounding boxes. And about 13 more with erroneous bounding box values (x and y), bring the total number of analyzable files to 881. The tool can tell quickly to CGCS analysts, which files can be read, and which cannot, as shown below. Files which cannot be read and indicated in red in the heatmap, show appropriate warnings to the annotator in the Annotation Explorer tab.

Figure 14: File status heatmap (view with interactivity in the UI)

Ideas to improve efficiency of annotation

Sometimes, annotations over a plethora of images can tend to be monotonous. That is the reason why previous annotations done on an image are stored in the database. This can allow multiple people to work together on the annotation process. This is something I also notice with industry players today, with increasing calls for annotators, in improving accuracy of object detection systems. The whole train of through behind human-in-the-loop learning is well illustrated in commercial establishments focusing on annotations such as Capestart.

The threshold sliders can also be made more interactive, where only bounding boxes above a certain threshold can be shown, but this will necessitate capturing a frame each for each object detected. A better way can be loading the image once and move the bounding box coordinates according to the object selected/slider threshold. This additional logic is not currently implemented in the UI, but certainly a possibility. On hover over the bounding box frame, an image similarity engine can be used to retrieve the best match to the training database, to fasten annotator’s work, and speed up annotator’s familiarity with the training set.

3 –– Characterize the distribution of objects across the forty people. 

a. Which people have which objects?  Please limit your answer to 8 images and 250 words  

The analysis for this section is done post the annotation process followed in Q2. This will give a clear illustration of all objects in possession of person, regardless of whether they were detected by the classifier. As explained above, the false negatives are captured through the annotator’s notes, providing a complete overview of the presence of an object with a person. This is represented as the below one hot encoding matrix, which can then be fed into an appropriate visualization.

Person/Object	miniCards	noisemaker	pinkEraser	rainbowPens	Trophy	vancouverCards	yellowBalloon	.	.	.	.	.
Presence for Person 1	1	1	1	1	1	1	1	0	0	0	0	0

Table 3: Presence matrix for persons-objects combinations. In this case, Person 1 has the objects marked True (1), and other objects in the set are not in possession of person 1

A Sankey chart with a person selection is provided for end user to visualize which persons are in possession of which objects. This helps to answer the question and helps to understand what common objects might be shared amongst people. In the below, for Person 15, they are in possession of the objects shown in the linked bars. Of interest is the capturing of voiceRecorder, which is from an image (15_13.jpg) unable to be parsed by the tool, due to an invalid starting byte.

Figure 15: Person 15’s list of objects represented through a Sankey diagram

On selection of a particular person/set of persons, the visualization below provides the spread of the images, in terms of the count of occurrences across the image set provided for the particular person.

Figure 16: Count of objects distribution per selected person (single selection)

This visualization also allows multiple selections such as the below, where Person 1 and Person 11 share miniCards in common.

Figure 17: Screengrab of the sankey diagram showing Person 1 and 11 have miniCards in common

The visualization developed here, formed the basis for the analysis in the next question, since we are able to observe patterns such as below, where the pair of pinkEraser-yellowBalloon is possessed between Person 1 and Person 5.

Figure 18: Person 1 and 5 possess both pinkEraser and yellowBalloon in common

The interactive nature of the interface will help in slicing and dicing further amongst the set of 40 individuals, and viewing multiple selections, as shown in the screengrab below. Few insights that can be drawn from below are

· 10,12 and 36 share cactusPaper and hairclip

· 1,10,12,36 all possess noisemaker

· Only P36 has voiceRecorder in this set

Figure 19: In detail: multiple selections can be made, and visualized in the Sankey diagram

b.  Identify groups of people that have object(s) in common. Please limit your answer to 10 images and 500 words.

Visualizing the combinations of objects is slightly tricky because of the sparsity of the object spread, and high dimensionality of the person-object matrix (40x43). By ordering each object present with person in a sequential format, it is easier to identify groups of objects shared amongst people. Groups are defined as pairs/triples/quadruples as seen below.

Referring to table 3, where person 1 is in possession of miniCards-noisemaker-pinkEraser-rainbowPens-trophy-vancouverCards-yellowBalloon, the below definition can be created for groups of objects.

Pairs (	Triples	Quadruples (
miniCards-noisemaker	miniCards-noisemaker-pinkEraser	miniCards-noisemaker-pinkEraser-rainbowPens
miniCards-trophy	miniCards-noisemaker-trophy	miniCards-noisemaker-pinkEraser-yellowBalloon
trophy-yellowBalloon	rainbowPens-trophy-yellowBalloon	noisemaker- -rainbowPens-trophy-yellowBalloon
vancouverCards-yellowBalloon	trophy-vancouverCards-yellowBalloon
noisemaker-vancouverCards	noisemaker-pinkEraser-trophy
miniCards-yellowBalloon
pinkEraser-rainbowPens

Table 4: Illustrative example of pairs, triples and quadruples used in the analysis (‘n’ in table refers to the total set of objects, which is in this case would be 43 across the total set)

The reason why this combination list stops with quadruples, is the maximum number of shared set of objects between two persons is found out to be at most 4.

Figure 20: Understanding which people have quadruples and triples, which ultimately map to pairs of objects

The user is also given the option to choose quadruples/triples individually.

Figure 21: Screengrab of quadruples captured individually

From the matrix of persons and objects, it is possible to calculate the distance in 2D space between any two persons. The metric used below is Cosine similarity. On the correlation matrix with cosine distance between persons (40x40), T-SNE is applied, which reduces the representation of persons into a 40x2 array.

Label	birdCall	blueSunglasses	brownDie	cactusPaper	canadaPencil	carabiner	cloudSign	cowbell
person
1	0	0	0	0	0	0	0	0
2	0	1	1	0	0	0	0	0
3	0	0	0	0	0	0	0	1
4	0	0	0	0	1	0	0	0
5	0	0	0	0	0	0	0	0
6	0	0	0	0	0	0	0	0
7	0	1	1	1	1	0	0	0
8	0	0	0	1	0	0	1	0

Table 5: Person Object matrix used to calculate cosine similarity and subsequently T-SNE is applied on the same

From the T-sne visualizations, one can get a sense for groups of persons with similar objects. This can be fed into a clustering algorithm to get clusters of similar people. But, since the focus of this piece was on matching groups of objects and groups of people, the analysis is best viewed through the Sankey chart above.

Figure 22: T-SNE visualization for the cosine similarity matrix generated based on the presence of objects with each person. This was starting to cause some memory issues in the UI, hence this visualization is not available in the final published UI

4 –– Which group do you think is the most likely group with the “totem”? What is your rationale for that assessment? Please limit your response to 5 images and 300 words.

The fundamental assumption is that the totem must be had by at least 8 persons, since that is the length of the group the CGCS is seeking. Post the human annotation process, we can confirm that the following are the objects possessed by at least 8 pax.

Label	1	2		3		4	5		6		7		8	9		10	11	cnt
lavenderDie	2	4		6		10	16		17		18		20	21		29	36	11
miniCards	1	4		6		10	11		13		14		24	25		32	39	11
blueSunglasses	2	7		9		13	17		22		27		31	39				9
metalKey	2	8		9		16	18		20		30		32	40				9
pinkEraser	1	5		9		15	18		19		28		33	40				9
eyeball	5	11		15		19	20		30		35		40					8
noisemaker	1	2		11		12	15		20		36		37					8
rainbowPens	1	3		11		17	23		30		31		38					8
rubiksCube	4	6		8		17	21		30		34		38					8
canadaPencil	4		7		14			22		25		35			39				7

Table 6: Objects that are in possession of at least 8 persons

I believe the totem is the pinkEraser and the group behind it as highlighted in Table 6.

Based on the clues provided by the CGCS investigators, it seems that this is an object which is not remarkable at all to the audience outside this network. Also, using a combination of the texts associated with images that have pinkEraser, and the file heatmap, which indicates that files which have pinkEraser such as Person28_6.jpg having an issue with parsing, led to strong suspicion towards this object. The relative count of pinkEraser was generally only 1 with most persons, indicating that this was their secret tool for communication. Also, if there is one person to eliminate from this 9 group, I would pick person 5. Person 5 only seems to be possessing pinkEraser for genuine reasons, since they have combined it with other objects in a lot of the cases. Only for the others, pinkEraser is generally found in isolation. Also, a particularly suspicious image of pinkEraser is Person 40_9.jpg, which says the below caption “Blocked. Snuffed. Erased”. This might be a strong hint that Person 40 is part of the group and is communicating that they were in trouble.

From the text notes, not much evidence can be gathered since most files belonging to this chain do not have accompanying text data. This analysis was done more using the insights derived in the previous question and combining it with the file status heatmap.

Figure 23: Suspicious image of pinkEraser with person 40

5 –– Process question: Did you choose to use the object recognition model results provided or use your own machine learning algorithm? Why did you make that choice? What was the biggest challenge you faced?  Please limit your response to 3 images and 300 words.

I believed it will be more practical to invest time in building a generic end-to-end human in the loop framework for interpreting and correcting object annotation errors, than training a new model.

I made the choice to retain the existing results because:

1. Bounding boxes were clearly provided, in a very ingestible format. Since this is a growing field of interest, several open source libraries helped to quickly visualize bounding boxes of the entire test data.

2. Training images did not have annotations for the objects, only had the tags which were folder names. So, there was an annotation procedure/effort involved. After an initial trial of preparing bounding boxes from training images, I went ahead with the existing model, with the key assumption that the framework presented in this work could be generic, and can be extended to models that provide bounding box data in the given format.

3. Also, a challenge-specific assumption I made was that the ‘totem’ could be identified regardless of the model chosen.

In terms of challenges faced, there was no option to retrain with a new model directly. So, besides building the training explorer, there can be extensions in the framework that allow annotators to go ahead, and annotate training images, so that it can be fed into any compatible models such as Yolo V2. An implementation very similar to the canvas used to measure IOU can be mimicked in parallel in the training explorer tab.

As seen below, there is a heavy clutter of labels thrown out by the detector, which of course can be filtered out by the threshold. Model that provides lesser false positives or clipping off false positives beyond a certain threshold can prevent the problem below, where multiple bounding boxes are clouding the annotator’s view.

Figure 24: An example of model with large false positives clouding the view of an annotator

Also, there were cases where bounding box coordinates are going outside the provided picture, as one can observe below.

Figure 25: The bounding box for yellowBag: Top left coordinates are (353,1035). Adding the width and the height, mean the bottom right coordinates are at (2460,4046). Given image size of (3024 x 4032), the detector bounding box is compliant with the width of the image, but not with the height of the image provided. Hence, this posed a challenge in the correction process

References:

I would like to thank the open source community, whose visualization ideas continued to inspire me through the journey of solving the questions of the challenge. Below are the key references/inspirations used in this project.

· https://dash-gallery.plotly.host/self-driving/

· https://medium.com/plotly/productionizing-object-detection-models-with-dash-enterprise-dba1c9402c2f

· https://github.com/nalepae/bounding-box

· https://github.com/plotly/dash-object-detection

· https://stackoverflow.com/questions/3438756/some-built-in-to-pad-a-list-in-python

· https://community.plotly.com/t/how-to-manage-the-layout-of-division-figures-in-dash/6484/3

· https://plotly.com/python/reference/#heatmap

· https://community.plotly.com/t/checklist-options-wont-update-on-callback/31525

· https://github.com/plotly/dash-core-components/issues/133

· https://stackoverflow.com/questions/57395546/cannot-get-multiple-inputs-and-outputs-to-work-in-dash/57475869#57475869

· https://pypi.org/project/azure-storage-blob/

· https://dash-bootstrap-components.opensource.faculty.ai/docs/components/modal/

· https://stackoverflow.com/questions/50014827/write-python-dataframe-as-csv-into-azure-blob

· https://towardsdatascience.com/evaluating-performance-of-an-object-detection-model-137a349c517b

· https://community.plotly.com/t/draw-polygon-in-mapbox-with-dash-python/34917/2

· https://dash.plotly.com/canvas

· https://dash-canvas.plotly.host/bounding-box/

· https://www.pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/

· https://www.codementor.io/@bertranddelvaux/how-to-build-a-quick-image-labelling-tool-to-train-a-neural-network-y4tpt6rhr

· https://www.pyimagesearch.com/2015/03/02/convert-url-to-image-with-python-and-opencv/

· https://stackoverflow.com/questions/50789834/parse-a-dataframe-column-by-comma-and-pivot-python

· https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/

· https://plotly.com/python/sunburst-charts/

· https://gist.github.com/jovianlin/ce13daf9e7c846da8fb50f03cd57be9e

· https://plotly.com/python/colorscales/#custom-discretized-heatmap-colorscale

· https://devcenter.heroku.com/articles/getting-started-with-python