04. Graph Format Description - VitalyRomanov/method-embedding GitHub Wiki

The graph dataset has the following structure

graph_dataset    
│
└───no_ast
│   │───common_call_seq.bz2
│   │───common_edges.bz2
│   │───common_function_variable_pairs.bz2
│   │───common_nodes.bz2
│   │───common_source_graph_bodies.bz2
│   └───node_names.bz2
│   
└───with_ast
    │───common_call_seq.bz2
    │───common_edges.bz2
    │───common_function_variable_pairs.bz2
    │───common_nodes.bz2
    │───common_source_graph_bodies.bz2
    └───node_names.bz2

no_ast contains graph built from global relationships only. with_ast contains graph with AST nodes and edges. Two main files for building the graph are common_nodes.bz2 and common_edges.bz2.

Pandas DataFrame pickle containing node data. The columns are

id,type,serialized_name,mentioned_in,string 

The first column is global node id, the second is node type, third is a node name. Column mentioned_in stores ids of functions where AT nodes have appeared. The column string has string representation for some of AST nodes.

Pandas DataFrame pickle containing edge data. The columns are

id,type,source_node_id,target_node_id,file_id,mentioned_in 

As with nodes, the column mentined_in stores id of a function where a given edge has appeared. By grouping edges with identical mentioned_in id one can get subgraphs for separate functions.

Pandas DataFrame pickle containing edges for training Call Sequence Prediction Objective. These edges show which function is called after a given function.

Pandas DataFrame pickle containing data for Variable Use Objective. Each function is connected to a set of variable names used inside this function.

Pandas DataFrame pickle containing ids of functions, their source code, the list of spans that represent AST nodes appeared in the body of this function.

Pandas DataFrame pickle containing data used for Name Prediction Objective. Functions and variables are connected to their names.

The files are stored as pickled pandas table (read with pandas.read_pickle) and probably not portable between platforms. One can view the content by converting table into the csv format

python SourceCodeTools/code/data/sourcetrail/pandas_format_converter.py common_nodes.bz2 csv

Source Code Gaph is based on AST. However, we introduce several modification. Some AST nodes have simplified representations. For example, we do not reprsent strings using their content, but simply as a string constant. We argue that for program understanding nde types plays more important role. This allows us to minimize the number of unique nodes. Not all edges that are present in AST appear in our Source Code Graph. Some edges are omitted. The list of all edges take from AST can be found in table below.

Moreover, we introduce additional edges into AST to increase connection density of the graph. The list of additional edges can be found in table below.

Blue edges represent additional edges that do not appear in AST.

with open(a) as b:
   do_stuff(b)

a: int = 5

def m():
       pass

[i for i in list if i != 5]

for i in list:
   k = fn(i)
   if k == 4:
       fn2(k)
       break
else:
   fn2(0)

if d is True:
   a = b
else:
   a = c

a = 5 if b is True else 0

try:
   a = b
except Exception as e:
   a = c
else:
   a = d
finally:
   print(a)

def f(a: int = 5):
   return a

def __init__(self, argument: int):
        """
        Initialize. Инициализация
        :param argument:
        """
        self.field = argument

def method2(self) -> str:
        """
        Simple operations.
        Простые операции.
        :return:
        """
        variable1: int = self.field
        variable2: str = str(variable1)
        return variable2

def main():
    a = Number(4)
    b = Number(5)
    print(a+b)

class ExampleClass:
    def __init__(self, argument: int):
        """
        Initialize. Инициализация
        :param argument:
        """
        self.field = argument

    def method1(self) -> str:
        """
        Call another method. Вызов другого метода.
        :return:
        """
        return self.method2()

    def method2(self) -> str:
        """
        Simple operations.
        Простые операции.
        :return:
        """
        variable1: int = self.field
        variable2: str = str(variable1)
        return variable2

from Module import Number

def main():
    a = Number(4)
    b = Number(5)
    print(a+b)

main()