Meetings - WM-SEMERU/SemeruGuidelines GitHub Wiki

22 Dec 2022

We discussed the priority of current projects and also assigned some tasks, in particular:

Tasks:

Alejandro will finish the exploratory analysis for each of the selected models by Dec 30 (or before)-> https://docs.google.com/spreadsheets/d/1mgzJ3wZF7ewkOfT_2hjp_Ohuj23bYwAnVTwBdeS57MA/edit#gid=0
Daniel will review the current state and help to execute the experiments (Logits extractor) using GPUs of SEMERU.

Alejandro will participate in the evaluation of flamingo performance using the statistical analysis techniques that have been implemented so far in CodeSyntaxConcepts.

We will have a meeting by January 6th to discuss more in detail the Capabilities project and also include Dipin in the discussion.

We presented the idea of CodeSyntaxConcepts to Kevin. We discussed the approach, research questions, and some extensions on the proposed study.

Our target Conference would be ICSE instead of FSE, we will have more time to improve the results and also conduct a survey study.

Tasks:

Run the models and collect the predictions (David, Daniel)
Run the AST Analysis on the predictions for all the models (Alejandro)
Perform a survey study to determine how useful is the proposed approach (To be defined)

New Ideas:

Propose a new BPE for neural models to encode inputs based on the rules defined by ASTs.
Extend the AST evaluation from methods to classes and more complex structures to evaluate more complex concepts (e.g., Semantic Capabilities, Design Patterns, Anti-patterns)

We will coordinate a new meeting to show the full results.