Candidate Answer Selections - GeeUnit/hw5-team07 GitHub Wiki
Our pipeline uses a K-Candidate voting system to select the best answer given the best K candidate sentence scores for each answer. Each candidate sentence has 1 vote and the answer choice which receives the most votes is selected as the winner. The winning answer must receive a number of votes as least as high as the minimum threshold. Otherwise, no answer will be selected. After trying out several voting methods, the single vote method proved to be the most successful voting method overall. We attribute the much of the success of single vote to 'single-feature-encapsulation', meaning that assuming each candidate sentence focuses on 1 feature of the answer, as long as we have an array of features represented in the answers, the answer with highest number of 'strongest' features should receive the most votes.
Other Methods Tested
Aggregate score method - the score from all candidate sentences were combined to yield the final score for an answer.
Borda ranked voting method - each candidate ranks their top X answers, best average rank wins.
We have several configuration parameters to control exactly how the voting is carried out.
K_CANDIDATES - Represents the number of candidate sentences which each vote for an answer.
WEIGHT_BY_RELEVANCE_VALUES - If true, each candidate sentence's vote will be weighted by the relevance value of the candidate sentence itself.
MIN_SCORE_THRESHOLD - The minimum number of votes an answer must receive before it can be chosen as the correct answer.
MIN_TO_VOTE_THRESHOLD - The minimum score an answer must receive from a candidate sentence before the candidate sentence can cast a vote for the answer.
Not Voting/None of the Above Answers
Because we are trying to maximize c@1 score with our system, not guessing is better than guessing incorrectly. Thus often times our system retrieves null as the answer to a question it is not confident enough to answer. Interestingly, many of the questions in the data set have as the final answer 'None of the above'. In order to improve our system's performance on these questions, we added some functionality to the voting logic such that whenever 'None of the above' was an answer choice and the voting system came up with null as answer (our system was not confident enough to provide an answer), we changed the answer to 'None of the above.' It was difficult to implement this logic with PMI scoring because standardizing PMI score was not as simple as other scores who were scaled from 0 to 1. Because PMI score scale changes, deciding a
MIN_SCORE_THRESHOLD as well as
MIN_TO_VOTE_THRESHOLD would be different for nearly every question, so this logic is not included with runs based off of PMI score.