complexobject_detection - Extended-Object-Detection-ROS/wiki_english GitHub Wiki

Complex Objects detection

!!! To use this part of the package, you need to install the library igraph.

1. Abstract

Complex (or compound) objects are a collection of Simple Objects with a given relationship between them. At the moment, relationships are comparative (more, less) or spatial (being inside, being at a distance). With this tool, you can greatly expand the set of objects that can be recognized by the system. Often in the scientific literature for such a definition of objects, the term "scene" or "situation" is used, sometimes supplementing it with various constructions, for example, context. The image below is an example of a well-known picture (in Russia) that shows how relationships can be defined between objects.

The task of the algorithms for recognizing Complex Objects in this example can be "find a person with a rope" or "find a person standing on the road in front of a bush", "find two people holding hands." In the first case, the answer will be one group of objects, in the second two, in the third also two. For the problem of recognizing complex objects, the theory of graphs is used. The observed scene and the description of a complex object are presented in the form of graphs, where simple objects are vertices, and relations are edges. Next, the problem of finding isomorphic subgraphs is solved.

2. Hard and soft recognition

Similar to the problem of recognizing Simple Objects, recognition of Complex Objects should be performed in two modes: hard and soft.

  • Hard - when the presence of all objects and connections is required for recognition.
  • Soft - when groups of objects are taken that partially fall under the given description, and the confidence coefficient for this group is calculated.

3. XML-description

Complex Object descriptions are located in the ComplexObjectBase tag, referring to Simple Objects from SimpleObjectBase and relationships from RelationLib.

3.1. Relationships

As mentioned, relationships are specified in the RelationLib tag. For each relationship, a RelationShip tag is created with the following parameters:

  1. Name (string, must be set) The unique name of the relationship.
  2. Type (string, must be set) The relationship type must be one of the table below.
  3. Other parameters are specific to each type of relationship.
Type Description Workspace* Bilinear Link
ImageRange Distance ratio between objects in an image I ✔✔❌ (depends on the mode) link
3DRange Distance relationship between objects in 3D space 3D ✔ link
LogicAnd Logical AND over two relationships Inherited** Inherited** link
LogicOr Logical OR over two relationships Inherited** Inherited** link
LogicNot Logical NOT to relation Inherited** Inherited** link
SpaceIn Checks that one object is completely inside another I ❌ link
SpaceOut Checks that one object is completely outside the other I ✔ link
SpaceUp Checks that the center of an object is higher than the center of another I ❌ link
SpaceDown Checks that the center of an object is lower than the center of another I ❌ link
SpaceLeft Checks that the center of an object is to the left of the center of another I ❌ link
SpaceRight Checks that the center of an object is to the right of the center of another I ❌ link
SizeSame Checks similarity of area of boxes I ✔ link
SameExtractedInfo Cheks similarity in extracted info - ✔ link

* - It can be either I, which means that the objects are compared in image coordinates, and 3D, which means that the relation operates in three-dimensional space, however, for this option, the objects must have attributes that determine the three-dimensional position.

** - Logical relationships take the parameters of the relationships to which they are applied.

3.2. Complex Objects

All complex objects are described inside the ComplexObjectBase tag, and each of them must be enclosed in the ComplexObject tag.

3.2.1. ComplexObject tag parameters

  1. ID (int, must be set) A unique identifier for a complex object. May be the same as Simple Object identifiers.
  2. Name (string, must be set) The unique name of the Complex Object.
  3. Mode (string, default: hard) Detection mode could be Hard and Soft.
  4. Probability (double, default: 0.75) Threshold of degree of confidence.

3.2.2. Internal tags with their own parameters

  1. SimpleObject - these tags denote which simple objects make up a complex
  • Class (string) The name of a simple object in the base.
  • InnerName (string) Some internal name that will be used when defining relations.
  1. Relation - these tags describe how simple objects are connected by relationships
  • Obj1 (string) The internal name of the first object connected by the relationship.
  • Obj2 (string) The internal name of the second object connected by the relationship.
  • Relationship (string) Relation name from RelationLib.
  • Weight (double, default: 1) Significance coefficient.

3.3. Example

An example of a complex object consisting of simple ones is a PC.

<?xml version="1.0" ?>

<AttributeLib>
    
    <Attribute Name="KeyboardDnn" Type="Dnn" Weight="4" framework="tensorflow" weights="ssd_mobilenet_v1_coco_2017_11_17/frozen_inference_graph.pb" config="ssd_mobilenet_v1_coco_2017_11_17/config.pbtxt" labels="ssd_mobilenet_v1_coco_2017_11_17/mscoco_label_map.pbtxt" inputWidth="300" inputHeight="300" Probability="0.75" obj_id="76"/>            
    
    <Attribute Name="ScreenDnn" Type="Dnn" Weight="4" framework="tensorflow" weights="ssd_mobilenet_v1_coco_2017_11_17/frozen_inference_graph.pb" config="ssd_mobilenet_v1_coco_2017_11_17/config.pbtxt" labels="ssd_mobilenet_v1_coco_2017_11_17/mscoco_label_map.pbtxt" inputWidth="300" inputHeight="300" Probability="0.75" obj_id="72"/>   
    
    <Attribute Name="MouseDnn" Type="Dnn" Weight="4" framework="tensorflow" weights="ssd_mobilenet_v1_coco_2017_11_17/frozen_inference_graph.pb" config="ssd_mobilenet_v1_coco_2017_11_17/config.pbtxt" labels="ssd_mobilenet_v1_coco_2017_11_17/mscoco_label_map.pbtxt" inputWidth="300" inputHeight="300" Probability="0.5" obj_id="74"/>    
    
    <Attribute Name="MouseBlob" Type="Blob" Weight="1" minThreshold="10" maxThreshold="297" blobColor="0" minArea="554" minCircularity="0.10" minConvexity="0.87" minInertiaRatio="0.01"/>
    
    <Attribute Name="MouseBlack" Type="HistColor" Histogram="histograms/Black.yaml" Probability="0.75" Contour="FALSE"/>    
        
</AttributeLib>

<SimpleObjectBase>
    
    <SimpleObject Name="Keyboard" ID="1">
        <Attribute Type="Detect">KeyboardDnn</Attribute>
    </SimpleObject>
    
    <SimpleObject Name="Mouse" ID="2" Mode="Soft" Probability="0.5">
        <Tracker IOU="0.3" decay="0.01" soft_prob="0.2">NONE</Tracker>
        <Attribute Type="Detect">MouseBlack</Attribute>
        <Attribute Type="Detect">MouseDnn</Attribute>
        
    </SimpleObject>
    
    <SimpleObject Name="Screen" ID="3">
        <Attribute Type="Detect">ScreenDnn</Attribute>
    </SimpleObject>
    
</SimpleObjectBase>

<RelationLib>    
    
    <RelationShip Type="SpaceLeft" Name="left"/>
    <RelationShip Type="SpaceUp" Name="up"/>
    <RelationShip Type="SpaceOut" Name="out"/>
    
    <RelationShip Type="LogicAnd" Name="out-left" A="out" B="left"/>
    
</RelationLib>

<ComplexObjectBase>
    
    <ComplexObject ID="1" Name="PC" Mode="Hard" Probability="0.2">
        <SimpleObject Class="Screen" InnerName="Screen" Weight="2"/>
        <SimpleObject Class="Mouse" InnerName="Mouse"/>
        <SimpleObject Class="Keyboard" InnerName="Keyboard"/>        
        
        <Relation Obj1="Keyboard" Obj2="Mouse" Relationship="out-left"/>        
        <Relation Obj1="Screen" Obj2="Keyboard" Relationship="up"/>
        <Relation Obj1="Screen" Obj2="Mouse" Relationship="up"/>
        
    </ComplexObject>
    
</ComplexObjectBase>

Complex PC example

⚠️ **GitHub.com Fallback** ⚠️