FAQ - strohne/Facepager GitHub Wiki
Don't be afraid to have questions. Everyone has them. Here you find the frequently asked questions. There is German version as well. In case you have more questions: Check out our Facebook User Group.
General
Q: Where do I find error messages?
A: When fetching data the error codes are printed in the status view. Furthermore, you find the detailed error messages in the fetched data. Click onto the new node (expand the parent node first) and look at the data view. If the message is too long, hover your mouse cursor over the item and wait until the tooltip comes up. You can copy the data from the data view to a text editor to see everything. Or you can setup the columns to show the error message in the nodes view.
Q: Can I import local files with Facepager?
A: You can use file URIs in the base path or resource path field. For example, the URL file:///c:/users/powerwomen/myfile.json references a file on Windows systems. To assemble these URIs from existing files, in the Generic module, click the folder button next to the download or upload folder field and check the "Add selected files as nodes" option to import filenames.
Q: What kind of operating systems does Facepager support?
A: You can use Facepager with Mac and Windows devices out of the box. Linux-Users please have a look at the instructions for running from code: https://github.com/strohne/Facepager/tree/master/src
Q: Is it necessary to be registered at Facebook or other platforms to use Facepager?
A: Yes, it is necessary. You log in to Facebook or other platforms via Facepager. Facepager itself doesn’t get any information about you.
When you log in via Facepager, you will get an Access Token. This token gives you access to the API of the provider.
This procedure is called Open Authorization. This works until you delete the Access Token from the input field in Facepager.
If you set a tick at Clear settings when closing the token will automatically be deleted when you close Facepager.
Q: What is an Access Token?
A: When you log in to Facebook or the other platforms via Facepager , you will get an Access Token.
It is a newly generated password that you will get from the provider of the specific API.
In the Query Setup, this token will be encoded, whereas it will be visible in the Status View.
If you ask for help, you should black it in screenshots or in the messages from the Status View to prevent data misuse.
If you work on public devices, you should delete your Access Token before closing Facepager or just set a tick at Clear settings when closing, because this token is available until you will delete it yourself.
Q: What is the field "Client Id" for?
A: There are two different ways to get access to Facebook, Twitter and YouTube via Facepager. Firstly, you can log in with Facepager as it comes and you will get an Access Token which is connected to our registration of Facepager at Facebook, Twitter or YouTube. Secondly, it is also possible to register your own application, which gives you more control. For example, you can hand it in for review to get access to restricted areas of an API.
In both cases, Facepager will be an intermediary program between a registered application and the API. The registration (either way) is necessary, because the provider wants to know, that Facepager has the permission to access data. If you register your own application you will get some identifiers like a "Client Id" which you can then enter into the field in the Query Setup. Depending on the API, different types of identifiers are used:
Facebook: Client IdYouTube: Client Id and Client SecretTwitter: Consumer Key and Consumer Secret
If you want to register your own app you will find some hints for the settings in the source code. Don't bother about the first lines of the linked file, these are related to running Facepager from code. Skip to the sections starting with "# Create an app".
Q: What can you do with the Scope field?
The Scope field determines which type of permissions are requested at login. Sometimes you need specific permissions, e. g. to access privacy protected data. See the details in the documentation of the specific API.
Q: Is it possible to automate the extraction of data temporal?
A: Right next to the fetch button, you will find the timer function.
Q: I want to download information about other people. Why do I get different amounts of data for different people?
A: You only can download data about users on Facebook, Twitter or YouTube which the platform and the users give you access to. Facepager will not overcome privacy settings of users.
Q: How high are the rate limits at Facebook, Twitter and YouTube?
A: There are rate limits controlling the amount of data traffic in a time slot. The limits are different from API to API. Please see the documentations of the providers:
Facebook: https://developers.facebook.com/docs/graph-api/advanced/rate-limitingTwitter: https://developer.twitter.com/en/docs/basics/rate-limiting and https://developer.twitter.com/en/docs/basics/rate-limitsYouTube: https://developers.google.com/youtube/v3/getting-started
Q: Is it possible to run a paginated query?
A: Yes, it is. For this you use the combination of the limit parameter and the number of Maximum pages.
However, it depends on the specific API, how high you can set the limit parameter.
For example, if you extract posts from a Facebook page, you will have a limit of 100.
Please regularly look at the documentation of the API. There you find information about which queries are allowed and possible.
Q: Is it possible to accelerate the download progress in case of a big amount of data?
A: Yes, it is. Increase the number of Parallel Threads and the Requests per minute in the Query Setup.
Please note, that most platform providers limit the amount of queries in a specific time window.
Q: Which export-settings should I use?
A: If you export your database, you have different possibilities to create the csv-file. You can choose with which symbol you want to separate your columns. You choose whether you want to export only selected nodes or all of them.
If you have problems with special characters, play with the Use a BOM field. BOM means “Byte Order Mark” and is a marker for the character encoding in UTF-format. In this specific format, special characters are stored with two bytes. Since the order of the two bytes is arbitrary a BOM is used to mark the order. A BOM is not always neccessary.
You can also choose to export your database in wide-format. If you choose this, the single node levels will be attached next to each other instead of below each other.
If you want full control, instead of exporting you can open your SQLite-database with a database program like SQLite-Browser or R.
Q: Settings, parameters or presets I usually use do not work anymore. Why?
A: One explanation could be that the API of the used platform changed. Regularly look at the documentation of the specific API. Another explanation could be that you use an outdated version of Facepager. Look out for the newest program version: https://github.com/strohne/Facepager/releases
Q: I receive error reports. How I can find out, what they mean?
A: Most errors are common HTTP status codes and not specific Facepager errors. You can see them in the Status View and in the Data View as well. The Data View explains more detailed, what went wrong. Just click on the relevant node and you will get more information than just the error code. If the explanation is not completely readable, just copy this line to a common text program. Next to the general codes (look here: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) the documentations of the providers will also help you to understand the error reports:
Facebook: https://developers.facebook.com/docs/Twitter: https://developer.twitter.com/en/docsYouTube: https://developers.google.com/youtube/v3/
Q: Why does it take so long to delete all the nodes?
A: Well, the focus is on getting data, not on deleting it. ;) In case of doubt: Delete the whole database in your windows explorer or edit the created SQLite-database with a different program (e.g. SQLiteBrowser).
Q: Are there other resources about Facepager than this FAQ?
A: Yes, of course. Click through the articles of the wiki to learn more about the program.
In the Facebook User Group you find also a compilation of different manuals that have been written in the past years. Furthermore, this group is the first contact point for every question.
If you find a bug in Facepager itself, please report it in the GitHub Issuses.
Most important are the documentations of the APIs. Take a look there, to know which queries are possible and which are not. APIs are changing regularly, that’s why you should take a look at these documentations in case something doesn’t work anymore:
Facebook: https://developers.facebook.com/docs/Twitter: https://developer.twitter.com/en/docsYouTube: https://developers.google.com/youtube/v3/
Q: Is it possible to extract data about group members from private groups?
A: No, it is not. As of late, it is not possible to get data of group members, whether the group is private or open. Look here: https://developers.facebook.com/docs/graph-api/reference/v2.12/group/members
To be precise, it could be possible if you register your own app and hand it in for app review. See https://developers.facebook.com/docs/apps/review/
Q: Can I get the mail address from other Facebook users with Facepager?
A: No, the API does not allow this.
Q: Is it possible to extract data from Facebook by using key words?
A: Unfortunately not, because the API does not allowed this. However, you can fetch the data you want, like posts, export your results and filter the created csv-file by key words. Look also here: https://developers.facebook.com/docs/graph-api/reference
Q: Is it possible to fetch all the posts and comments of one Facebook page or group?
A: It depends how much there is. The API does not allow the extraction of very old posts or comments.
And it will take a lot of time to fetch huge amounts of data because of rate limits by Facebook.
Facepager itself doesn't allow you to set the Maximum pages parameter to infinity. This normally isn't necessary or useful.
Q: Can I also fetch the reactions to a Facebook post or comment?
A: Yes, see the preset or the documentation: https://developers.facebook.com/.../v2.11/object/reactions.
Q: Can I get the date of the likes or reactions?
A: Unfortunately not, because the API doesn't allow this.
Q: When I extract comments the emoticons do not look like as they should look like. Why is that?
A: Not every website and device is using the same codes for emoticons. That is why your results can look different compared to the original post or comment. Look here to understand these differences in the graphic representation: https://stackoverflow.com/questions/41541138/translation-and-mapping-of-emoticons-encoded-as-utf-8-code-in-text and https://github.com/today-is-a-good-day/emojis
Q: When I fetch Facebook posts for a selected period of time, I do not get any posts (or just a few), but there are definitely (more) posts. What does this mean?
A: There is Facebook's 600-post-limit, which means you cannot fetch all posts, when you fetch data from pages that publish more than 600 posts per year, even if you select a shorter period of time. See the study of Justin Chun-Ting Ho for further information about the sampling criteria. You can check for the limit by altering the period. It is an indicator for this limitation if you are able to fetch posts for a wider period and do not get posts for the dates you are looking for.
Q: Why are there differences when I fetch data from Twitter and Facebook, even if I use the same parameters?
A: Twitter and Facebook have completely different APIs. Not everything that is possible with the Facebook API is also possible with the one from Twitter - and vice versa. Please note the different types of APIs. Some queries are for free, for other you have to pay. Look here: https://developer.twitter.com/en/docs
Q: Is it possible to fetch tweets which are older than seven days?
A: To fetch older tweets you have to use a workaround or pay:
- You can try it with the free version of the Premium search API. You have to register your own App on Twitter to get a Consumer Key and Consumer Secret, which you can enter in Facepager. Look here: https://developer.twitter.com/en/docs/tweets/search/overview/premium
- Instead of using the search API you could also try to fetch the timelines of users. You have a limit here, as well. By the time of this writing you can fetch the last 3200 tweets of a user. Look here: https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html
- Use the website of Twitter or a webscraper to get the tweet IDs. Facepager can help you to fetch the content of them afterwards. Look here: https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-show-id
- You can use the paid version of the APIs of Twitter or services like GNIP. Facepager helps you to fetch the data after you paid for it.