Development - PxYu/Hide-Seek GitHub Wiki
Prerequisites
Before reading this documentation, make sure you have prior knowledge in Chrome extension development and JavaScript. Specifically, you need to understand:
- basic elements of Chrome extension including
- manifest file
- background page
- content script
- communication between background page and content script
- basic knowledge of JavaScript including
- JQuery
- ajax
- grammar of Javascript
- callback mechanism of JS
For more information on Chrome extension development, I highly recommend google's official tutorial here. There are tons of tutorials on JS, like this one.
Understanding by Behaviors
The best way to understand an interactive program is to break it down by user behaviors, in my opinion. In the following section, I will explain code blocks that serves each individual behavior.
A summary of user behaviors
Here's a list of user behaviors I anticipate that our program to be capable to deal with.
- User installs the extension
- User searches a keyword
- the moment he/she hits "click"
- the moment the page is done loading
- User clicks a link on the result page
- User turns the off-switch
- the whole program
- re-ranking feature
- User requests visualization
1. User installs the program
The first time user installs the program, function generateUUID() in the background page will be triggered. This function generates a unique user ID, sends it to the servlet (who will store this id into database) and stores it locally in user's machine (using cookie by Store.js).
2. User searches a keyword
The key behavior Hide & Seek deals with.
2.1 the moment he hits "click"
Now, given that the extension is turned on, the program should capture user's search query, send it along with necessary information to the server, receive returned cover queries along with their topics if applicable, open a new tab for each cover query and randomly click a link on the simulated page. Let's see how it's done with JS code.
In manifest.json file:
"content_scripts": [{
"all_frames": false,
"css": [
"styles/fonts.css"
],
"js": [
"bower_components/jquery/dist/jquery.js",
"scripts/content.js"
],
"matches": [
"http://www.google.com.hk/*",
"https://www.google.com.hk/*",
"http://www.google.com/*",
"https://www.google.com/*"
]
}]
This means that content.js will be injected into the web page every time there is a google search. Notice that I only match google.com and google.com.hk just now.
Then, in scripts/content.js:
if (href.indexOf('www.google.com.hk/search') != -1 || href.indexOf('www.google.com/search') != -1) {
var q = decodeURIComponent(getQueryString(href, 'q'));
console.log('q', q);
if (q) {
chrome.extension.sendRequest({ handler: 'handle_search', q: q }, function(result) {
console.log('result', result);
This part gets the query word and sends it to background page (specifically the function requestHandlers.handle_search) to handle. Then, starting from line 244 in scripts/background.js, that function handles the keyword.
requestHandlers.handle_search = function(data, callback, sender) {
var q = data.q;
if (simulateTab && simulateTab.id === sender.tab.id) {
return callback({ simulate: true });
} else {
callback({ simulate: false });
}
if (lastSearch != q) {
lastSearch = q;
if (popupSettings.started) {
$.ajax({
type: 'POST',
url: encodeURI(apihost + '/QueryGenerator/QueryGenerator?action=Q&query=' + q + '&uid=' + popupSettings.uuid + '&numcover=4'),
success: function(keywords) {
last_generated_topics = [];
$.each(keywords, function(key, val) {
console.log(key + ", " + val);
if (key == "input") {
last_user_topic = val;
addTopic(userTopics, val);
addQuery(userQueries, q.replace(/[^A-Za-z0-9]/g, ' '));
} else {
keywordsPools = keywordsPools.concat(key);
last_generated_topics.push(val);
addTopic(generatedTopics, val);
addQuery(generatedQueries, key);
}
saveTopics();
saveLastTopics();
saveQueries();
});
}
});
}
}
}
This part sends the query, user id and number of cover queries to generated to the remote servlet. You do not have to worry about how this servlet handles your request. The rest of this part serves to save data into storage and keyword pool. As you can see above, cover queries are appended to the keyword pools. Then what? Well:
At around line 228:
setInterval(function() {
if (!popupSettings.started) {
return;
}
simulateSearch();
}, 5 * 1000);
And around line 173:
var simulateSearch = function() {
if (simulateTab) {
return;
}
if (!keywordsPools || !keywordsPools.length) {
return;
}
......
Every five seconds, the program attempts to run simulateSearch(). If keyword pool is empty, nothing happens; otherwise, background page creates a new tab and start function autoSearch() in content script starts to run. Every 15 * 1000 milliseconds, background page tries to close the new opening tab so that some slow-rendering page would not block the simulation queue.
Next step is randomly clicking in simulated page. Content script is injected in both user's search page and simulated pages, and it's important to distinguish since different actions take place. In content script:
chrome.extension.sendRequest({ handler: 'handle_search', q: q }, function(result) {
console.log('result', result);
......
The returned result tells if current google page is simulated or not.
If yes, then randomly click something. Very easy, but one problem I came across is that, some randomly clicked link is actually a url for a file, say, a PDF file. If the program clicks on it, the browser starts downloading the PDF file, which is not what we want. Also, a PDF file does not necessarily has url that ends with ".pdf". To address this issue, in the scripts/content.js:
// make sure random click does not trigger a download
chrome.runtime.sendMessage({
method: 'HEAD',
action: 'A',
url: alist[idx].href,
rank: idx
}, function(response) {
console.log(response.status);
if (response.status == "YES") {
alist[idx].click();
} else {}
});
I capture the link to be clicked and send it to the background page. Then in the bgp (short for background page):
if (request.action == 'A') {
$.ajax({
type: request.method,
url: request.url,
async: true
}).done(function(message, text, jqXHR) {
var type = jqXHR.getResponseHeader('Content-Type').split(";")[0];
if (type == "text/html") {
sendResponse({ status: "YES" });
rank = request.rank + 1;
} else {
console.log("%%%%%%%% Cannot open type: " + type + " %%%%%%%%");
sendResponse({ status: "NO" });
}
})
}
We use ajax to check the file type behind that certain url. Then the bgp returns the answer to the content script concerning whether that link is clickable. You might wonder why go through the bgp? Seems much trouble huh? Well, that is because content script cannot fire http request. It is called cross-origin requests, which are prohibited.
Note that simulated clicks ("SC") information will be sent to the servlet to be documented. Implemented in bgp.
2.2 the moment the page is done loading
On the other hand, if this google page is of user's genuine query, something completely different happens.
First thing is that we have to upload a snapshot of current page to the servlet and receive an array of integer numbers representing how items on this page should be re-organized.
var items = [];
var snippets = [];
var block = [];
var numInEachBlock = [0];
$.each($("div.srg"), function(index, value) {
console.log("=======");
// save the block
block.push($(this));
// save items in block in order
var tmpItem = $(this).find("div.g").toArray();
numInEachBlock.push(tmpItem.length);
$.each(tmpItem, function(idx, val) {
items.push($(this));
});
console.log(tmpItem);
$.each($(this).find("div.g span.st"), function(idx, val) {
console.log($(this));
snippets.push($(this).text());
})
});
One thing you might get confused is div.srg. This is actually a block of items. In most cases, on page means one block containing 10 items, but in not-so-infrequent cases, there're two or more blocks containing items to re-rank, so our program has to adapt well to those circumstances as well.
3. User clicks a link on the result page
When a user clicks a link, the extension sends what he/she clicks to the server.
In scripts/content.js:
$('#res .g .r a').click(function() {
var self = $(this);
var url = self.attr('href');
if (url.indexOf('/url?') == 0) {
url = decodeURIComponent(getQueryString(url, 'url'));
}
var snip = $(this).parent().closest('div').find(".st").text();
var title = self.text();
var keyword = $('#lst-ib').val();
chrome.runtime.sendMessage({
action: "UC",
content: snip,
url: url,
title: title,
keyword: keyword,
index: -1
});
});
This is abbreviated as "UC" (user click) similar in name and implementation as "SC" (simulated click).
4. User turns the off-switch
This part is fairly easy, since only to boolean values are concerned. The status of the program off-switch can be changed in the popup window and options page, while re-ranking feature could only be altered by the latter one. The bottom line is that they are all trying to manipulate the boolean variables in bgp.
5. User requests visualization
Visualization module is dependent on Highcharts.js. The trick of this part is how to organize the hierarchical topics. Each cover query has a topic, which is something like "xxa/xxb/xxc". You might be interested in reading the algorithms.
A few more things
- All data is stored with
store.js. User-id, program settings and user history. Because we do not provide cloud service, all the information is local. - Some permissions in
manifest.jsonare obsolete. I will delete extra ones, so do not get confused. - What's difficult about this program (and all other chrome-extensions) is how content scripts and bgp work together.