Judging the Hackathon Discussion Notes and Next Steps 10 Feb 2023 - OpenDataforWeb3/Resources GitHub Wiki

We held a two-hour open office hours discussion about the ongoing judging of the OpenData "Data Builders" Hackathon.

This note is intended to summarize that discussion, focusing on helpful tips for judges and lessons learned for future hackathons.

Hackathon Judging:

  • we agreed that it would be helpful for judges to both a) judge on their own and b) learn from the judging of other judges
  • obviously, there is a creative tension between those goals

As a next step, we agreed that epowell101 will share again the overall results sheet and this time encourage judges to link their own version of the spreadsheet on a tab; this way async collaboration could occur

Additionally - the suggestion to judges will be to put notes to the right for their fellow judges including focusing on which entries are most worthy of deeper analysis

ALSO - we discussed the overall ethos of the judging. As discussed, in hackathons it is important both to provide objective and in-depth analysis of the most impressive entries AND also important to encourage entrants that may be less experienced and newer to the skills; overall we want to screen out and not reward those entrants that are putting in the minimum of effort just to hopefully win something.

Transparency of scoring:

  • there are some judges that are also competing in the hackathon
  • as discussed in prior community meetings, we are both encouraging this as these are often our most advanced data scientists in the community, and expecting them to be objective in their analysis, and also we will publish the quadratic votes for the Hack on the Hackathon bounties to attempt to mitigate self-dealing

What makes a good submission?

  • we have criteria that have been shared and also now have a checklist that is useful for making more objective our scoring: Judging Criteria
  • additionally - judges expressed that they are looking for analysis that: a) clearly explains why it believes the analysis to be useful or interesting b) presents the hypotheses being tested clearly c) seeks to thoughtful test those hypotheses
  • As an example judges tend to rate highly exploratory data analysis that reasons about potential signs or symptoms of Sybils such as the gas network approach outlined by TrustaLabs and described elsewhere; and that then seek to confirm or disprove that analysis using a different approach - such as examining whether these apparently related wallets behave similarly when it comes to their granting behavior.

Immediate priorities and future direction of ODC:

  • a few of us also discussed the immediate priorities of the ODC including onboarding and supporting sandbox projects and standing up a data stack for the use of the community and, potentially, sponsors such as Gitcoin
  • for the next hackathon - to kick off in mid to late April and lasting long enough to ensure analysis is delivered for the April / May gitcoin round, we thought that it might make sense to Gitcoin for exploratory data analysis to possibly have a few specific tracks: passport & sybils; wallets & sybils; grants & sybils. While underlying graph analysis often touches on all of these - one idea is to have more specific and focused bounties, IF these resonate with the relevant projects in Gitcoin and with the broader community.
  • we also discussed the need to write up personas in a collaborative way so we are thoughtful about how to make sure we are serving these personas and can potentially go deeper in our thinking than simply: data scientists, project developers, sponsors
  • EVENTS: we are making some progress on the idea of a plug-fest in a few weeks at EthDenver; we talked about possibly in lieu of any bounties for the work done by Sarob on the community, subsidizing his travel to Eth Denver to participate in this and to assist in ODC matters. We will discuss transparently to get feedback. ALSO we discussed possibly attending BlackHat in mid August in Las Vegas and overall the need to collaborate on planning these and other events, presumably to include future Eth events.