Query Setup. Twitter and Twitter Streaming - strohne/Facepager GitHub Wiki
The Getting Started section gives you a quick introduction. For detailed information on specific options of the Twitter API have a look at the documentation provided by Twitter. Some information from the documentation also shows up in tooltips when hovering with the mouse cursor over options in the query settings or even over keys in the Data View.
In addition to the normal search API there is a “Twitter Streaming” tab, which allows you to fetch data from Twitter in real-time. See the documentation provided by Twitter for further information: https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter
Starting nodes for fetching data from Twitter usually are search terms, user ids or user names. For example, if you want to get the most recent tweets of someone, you would add his or her username as a node. Once you fetched some tweets, take a look at the detail data. Every tweet has a unique numerical id, which subsequently may serve as starting point or otherwise be used in the parameters.
Please remind: This screenshot was taken on 25th of June in 2018. If you try this out yourself, remind that APIs are changing constantly, so this approach may be outdated by the time you read this text.
| Query settings | Explanation |
|---|---|
| Base path | The base path should always be filled with the URL of the most current version of the API. Compared to other APIs the basebath of the Twitter API is very stable, it didn't change for years. But since nothing lasts forever, regularly check the documentation and look out for new versions. In case of an update of the base path, change it in the Query Setup, e. g. from https://api.twitter.com/1.0/ to https://api.twitter.com/1.1/. |
| Resource | The most basic resources are statuses/user_timeline to get the tweets of a specific user and search/tweets for searching tweets by terms, hashtags or mentioned users. Tooltips with detailed explanations are showing up when you hover an item in the drop down list. |
| Parameters | When selecting a resource, the drop down lists are filled automatically with possible parameters. Look out for tooltips. Some parameters are mandatory, depending on the selected resource. For example, you have to provide the parameter q when searching tweets. Simply add search terms or hashtags as nodes and set the parameter to <Object ID>. For getting tweets from a specific user, you either have to specify the user_id or the screen_name. Look below for some useful parameters. |
| Maximum pages | You can’t get all tweets of all times in a single request. But you may get bunches of tweets one after another, which is called pagination. Increase the maximum pages to get more results. Twitter provides different kinds of pagination and Facepager tries to automatically detect the best way. See the documentation of Twitter for further information. |
| Access Token | Crawling data from Twitter requires a login with your user credentials. Before starting your data collection process, you have to complete the login via the corresponding button. A login page directly served by Twitter will open and once the login is successful, Facepager gets an Access Token. With this Access Token Facepager acts on behalf of the logged in user. Note: The Access Token is stored locally on your computer. No personal data is submitted to the developers or any other authority. |
Twitter provides different API Versions. Each version comes with different access levels and limitations. See the versioning page of Twitter for more information. Depending on what you want to collect, e.g. historical data, you might need a specific access level.
Access to historical data is very limited in the Standard API v1.1. At the time of this writing you only get data for the last seven days. A workaround is to request tweets of specific users instead of searching. To limit the request to a certain time period use the parameters max_id or since_id:
- max_id: The Twitter API usually provides the most recent tweets. To access older tweets you can leverage that tweet IDs are counted up in the order of the time posted. With the max_id parameter you can then restrict the results to contain only tweets posted before a given ID (plus the tweet with the given ID).
- since_id: In analogy to max_id you may restrict the results by specifying a since_id. Notice that there are limits to the number of tweets accessible by the API. You can’t always go back in time as far as you want.
Alternatively, you can try out the Premium API or the more recent API version 2:
| API Version | Access Level | Login & Restrictions |
|---|---|---|
| 1.1 | Standard (free) |
The Facepager v1.1 presets provide access to the standard API. You can use the API by clicking the Login to Twitter button with your regular Twitter login data . An access token is automatically retrieved. Access to historical data is very limited with the standard API version. |
| 1.1 | Premium (isn't free) |
The Premium Version for API v1.1 grants access to the past 30 days of Twitter data or even to the full history of Twitter data. You have to register an own app on Twitter to use the premium API with Facepager. See the Facepager preset for details about the login process. |
| 2 | Essential | In general, you have to register your own app to use the Twitter API version 2 with Facepager. An app is nothing more than a specific user account. In the process, you can apply for different access levels. When you register your app, you receive an API key and an API secret key. Keep them safe and don't show them to anyone. To use those credentials with Facepager, click on Settings next to the login button and change the "Authentication Type" to "OAuth2 Client Credentials". Now you can enter your personal API key into the Consumer Key field and your API secret key into the Consumer Secret field. When you click Login to Twitter, an access token should be retrieved and you can start fetching data. |
| 2 | Academic Research & Elevated Access | As an academic researcher, you can apply for higher Tweet caps or free full-archive access. After you were granted elevated access, the login process is exactly the same as for the essential access level. |
Please read the following terms of services from Twitter to get more information about
- the developer agreement and policies: https://developer.twitter.com/en/developer-terms/agreement-and-policy
- and the Twitter terms of service (here you have to look for your specific region): https://twitter.com/en/tos
Searching for more answers and explanations? Just read our FAQ.