Response to AMMBER paper - MotivationalModelling/mm-local-editor GitHub Wiki

Response to AMMBER paper Reviewer 1: The paper positions itself as presenting a set o f lessons learned from involving students in a software maintenance project. From the title and abstract, the reader gets the idea that the paper presents insights on involving students in software maintenance activities, and that those insights are based on well-described activities and results. That would be very useful for the academia involved in teaching that type of skills to the students. However, after reading the paper, it is my opinion that the paper does not live to the expectations. The following are the issues that I found:

Noted

The organization of the paper is not suited to the proposed goals of the paper. It start with some concepts about agents and complex systems (which can by anything, for instance, microservices based systems), and agents, and the relevance of this is not clear at all. Then the paper goes to discuss requirement elicitation, and, once again, there is no clear notion on how this is relevant. Sections 2.2 and 2.3 are not very useful at all. I would expect something amount concepts on software maintenance, current state of the art and related challenges.

The reason for agents was that was where the modelling came from. I will replace the section with a comment in the introduction. Software maintenance comments needs more work.

Concerning the presentation and English use, I notice three things: i) the tone is quite informal, and, at times, a bit imprecise (at least one tie the is the expression “I”, when the paper has 8 authors – who is “I” in this context?).

Noted

ii) The paper seems to restart several times, restating the goal and context at the start of several sections.

Noted, will modify

Iii) At times, it seems that one is reading a student report and not an academic paper. Instead of a purely chronological narrative, it would be more useful to present a top-down narrative with research questions, context, summary of difficulties and results, and a discussion of each result. The discussions about how a given diagram was incorrect seems pretty out of place, just to cite an example.

My style is non-standard. I need to explain better

A significant part of the paper is devoted to explaining the AMMBER project. However, it would be more interesting to describe the actions of the students involved in the project, and how each action impacted the project. This impact should be measured according to some software engineering methodology and metric (e.g., complexity, bug density, time needed, etc.). Results should be presented in an objective and qualifiable manner. Let’s discuss what metrics we have. The are some claims that need to be based on literature references. E.g., “student projects are not thoroughly tested”. Ok, I can accept that, but that should not be just an opinion – it should be based on something more tangible. There are several examples of this type of claim.

Again my style is informal. Will need to think how to address.

The paper mentions the use of github and it seems that part of the issues students have are a consequence of the use of github. I would say that the problems are derived from lack of knowledge on how to properly use guthub. In fact, it seems to me that what the authors describe concerning the use of branches does not align with how github should be used.

Can Ben comment

In section 5 there are many places where details of a specific softwa or library or API are used, without any context at all, leaving the reader completely dumbfolded. E.g., React context (is there a web app somewhere?), FileProvider (?), updateTextForGoalId… What is all that and what is the relevant of that?

Ben please comment

The lessons learned, presented in section 7, seems pretty obvious: control the repository review decisions, testing, ensure the involvement of real users, etc. All those seems to be already known in software engineering and agile methodologies. I would rather see lessons learned on how to involve students in maintenance projects in education contexts, as the title seems to indicate.

Probably need to do a survey

At the end of the paper, the reader is left without what the paper should have presented: background concepts on software maintenance, student actions and team leader actions regarding the project maintenance. Metrics and quantifiable results along the several years the project existed, conclusions about how to include maintenance skill in educational contexts. I would say the intended goal of the authors would merit publications, but in the current version, this paper does not fulfill the stated goals.

Work to do

Reviewer 2:

  • Paper claims "case study research methodology" (line 49) but lacks formal research design, data collection methods, or validity threats — reads as an experience report, not a research study

Perhaps should be described as an experience report.

  • Key claims unsupported by evidence (e.g., "interns work better than teams," "maintenance improved") — provide repository metrics, student feedback, or other data

A challenge for me to quantify, but I may try

  • Related work section overemphasizes agent-oriented modelling (Sections 2.1–2.2) but barely covers maintenance education literature — missing OSS onboarding studies, technical debt literature, ICSE-SEET/SIGCSE comparisons

Work needs to be done here.

  • Maintenance categories (corrective, adaptive, perfective) promised in abstract but never defined or mapped to AMMBER activities

That shouldn’t be hard to do.

  • Section 5 too React-specific (useReducer, Redux Toolkit) — focus on principles, not implementation details

Ben to comment

  • Lessons learned (Section 7) repeat earlier sections rather than synthesizing at a higher abstraction level

Need to think about

  • Inconsistent voice — switches between "we" and "I" (e.g., line 275)
  • Typos: "anti=goal" → "anti-goal" (line 194); "to have to fixed ideas" → "too fixed ideas" (line 128)
  • Reference [16] truncated — title is just "Bridging"
  • Figure 4 unreadable at current resolution
  • No discussion of generalisability limitations (single institution, single tool, masters students only)
  • "Unpaid internships" (lines 160, 647) — add note on ethical structuring
  • Section 3 (AMMBER description) too detailed for the paper's purpose — condense
  • Reframe as experience report or adopt proper case study framework (e.g., Yin, Runeson & Höst)
  • Mine GitHub for quantitative evidence (issues opened/closed, commit frequency, branch lifecycle)
  • Condense Sections 2.1–2.2 and 3; expand Section 2.3 significantly
  • Add a Limitations/Threats to Validity section
  • Present lessons in structured format: Context → Problem → Solution → Evidence
  • Fix incomplete references and proofread thoroughly