estimating_variation - Gandhie/AICS-Project GitHub Wiki

Discussions of commit f0d590c3d384f8ca92dd2b4ece7a15f54bc78793 (Mehdi's code)

Amelie and Simon

• Where: variation of bounding boxes of targets and landmarks
  • Functional relations: (i) there will be a high variation of targets/landmarks geometrically; (ii) actually there will be low variation of targets/landmarks geometrically because relations are much more restricted to particular objects and therefore restricted to particular locations
  • Bounding boxes:
    • Currently, normalisation against the image dimensions (ensures that all images are of the same dimension); the same two objects would appear spatially very different if the image is taken from close and from afar
    • We project the x, y, w, h into a 100x100 mask, i.e. a matrix of 0 and 1
    • To do:
      • How similar are different relations in terms of landmarks and targets? Plot similar graphs for every target-landmark pair; what do they look like? This will be qualitative, observational evidence.
  • Estimate the similarity between two graphs using cosine?
  • Estimate the variation of targets/landmarks of a particular relation and then rank all prepositions by this variation.
  • What variation?
    • Currently stdev is calculated for the entire x, y, h, w: hence also on the height and widths of objects; but objects are the same between the relations; use prepositions_bboxes_mask[p].std(0).mean()? Mehdi made a mistake here?
    • Solution using cosine: create a mask with targets (or landmarks); compare targets pairwise with cosine; take the stdev of the resulting cosine similarities

     tar1  tar2 tar3
tar1  *    *    *
tar2       *    *
tar3            *

• What: visual similarity of targets and landmarks
  • For each relation extract visual features of targets and visual features of landmarks prepositions_bboxes[p].append([v_target, v_landmark])
  • For every preposition p calculate cosine similarity between the visual features of every v_target (and the same for for v_landmark)
  • Estimate the variation of cosine for v_tragets (and for v_landmarks separately)
  • Is there a difference between relations, i.e. are targets fo fuctional relations more similar than those of geometric ones?
  • Rank the relations by the resulting targets (or landmark) variation: do we get clustering of functional vs geometric relations?