Tips for writing your own connector - Phyks/konnectors GitHub Wiki

Fetching data in your connector

How to find the best place to find data

⚠️ This section is still being written

Scrap a webpage

⚠️ This section is still being written

• Firefox networking tool, sites mobiles, apps

Now that you understand better how the konnector architecture works, we'll explain better the basics of web scrapping.

The first step is to login like if you were the user. For that you must go to the target login page and activate your browser network watcher (ctrl+shift+Q in Firefox). Then simply login into your target website. You will see in the network watcher query list a POST query that performs the login action. So to allow your konnector to fetch user information you will have to mimic this request. The analyze of the network watcher will give you the target url for login and the required fields required. Ex: For the Free website the login url is https://subscribe.free.fr/login/login.pl and the required fields are pass and login.

With that information, you can now use the request library to send a POST request to the login page. Don't forget to activate the cookie jar. It's an option that makes request store session information.

var options = {
  method: 'POST',
  jar: true,
  url: 'https://subscribe.free.fr/login/login.pl',
  form: { 
    pass: requiredFields.password,
    login: requiredFields.login
  }
};

request(options, function (err, res, body) {
  // Now we are logged in.
});

Once you are logged in you can fetch the page on the behalf of the user and parse it to grab his data. You will use request to fetch the page and cheerio to get the content of the part that interests you. Once again you will have to connect with your browser first. With the Inspector (ctrl+shift+C), you will see which part of the id and classes of the html tag of which the content should be extracted.

var billsToSave = [];
var url = 'https://adsl.free.fr/liste-factures.pl';
request.get(url, function (err, res, body) {

  var $ = cheerio.load(body);
  $('.pane li.bill-line').each(function () {
    var amount = $('.amount').html();
    amount = parseFloat(amount.replace(' Euros', ''));

    var month = $(this).find('.date');
    var date = moment(month.html(), 'YYYYMM');

    billsToSave.push({
      date: date,
      amount: amount,
      vendor: 'Free'
    });
  });
});

Once you're done with the parsing, you can save the bill data to database.

Request an API

If the target vendor did his job well, it provides an API. This makes things simpler to fetch the user data. Now you need to refer to the vendor API documentation to see what you could do. Most of API are JSON-based. For this job we recommend the use of request-json which is better to request APIs than request.

Many APIs simply requires a basic authentification. Here is an example to connect on Github that will show you how to set auth and query a page with the Github API.

var client = requestJson.createClient('https://api.github.com');

var username = requiredFields.login;
var pass = requiredFields.password;
client.setBasicAuth(username, pass);

var path = 'users';
client.get(path, function (err, res, users) {
   console.log(users);
});

Managing errors in your connector

⚠️ This section is still being written