Setting Request Headers, Cookies and Body - xbapps/xbvr GitHub Wiki

Not yet implemented

You can specify Request Headers, Cookies, and the Request body when scraping scenes or viewing trailers via the Heresphere API. Trailers and scrapers have separate configurations, so you could use them to authenticate on the site for trailers but not when scraping. This allows you to scrape the expected public scene data.

This can be useful in scenarios such as bypassing age verification prompts or skipping redirects to advertising pages. The effectiveness of these settings depends entirely on how the site has been implemented.

This facility is a tool—like being given a hammer. However, if you want to build a house, that is up to you to figure out. Investigating what Headers or Cookies you require can be quite complex.

When the scrapers make requests, they check for any configured details and add them to the request. The same happens when a trailer is requested via the Heresphere API. For some sites, it is possible to authenticate by setting Headers and/or Cookies and view the entire scene rather than just the trailer.

What a site may require is usually very specific, and even slight variations from what it expects can result in an error. You should use tools such as your browser’s Developer Tools to check the contents of cookies used by a site. Postman is very useful for trapping and viewing requests, as well as testing them.

Cookies

The Domain and Host fields for setting a cookie are similar but separate.

  • The Domain field is the domain value stored in the cookie.
  • The Host field is the site address where the cookie should be stored.

For example, in the screenshot:

  • Domain: (from the Domain column) .naughtyamerica.com
  • Host: (from the left hand side) https://www.naughtyamerica.com

image

Authenticating on Sites

You can often provide details to authenticate yourself on a site. However, this is usually done via a token that expires. These tokens may last a day, a week, or even a month, but some expire as quickly as 10 minutes. This means you will need a way to update the config with new tokens.

For sites where their Heresphere API is used to view trailers (e.g., Czech Network, NaughtyAmerica, POVR), the auth_token does not seem to expire.

Note: It is not recommended to authenticate on a site when scraping scenes. Scrapers expect publicly available scene data, and sites will often present a different view when authenticated, which could break the scraper.

Setting up Headers, Cookies, and Request Body

There are several ways to set up the request parameters:

  • Use the UI at Options > Advanced > "Headers/Cookies".
  • Import settings from a JSON file.
  • Use Bundle Import/Export, turning on the "Include Config Settings" option (off by default).
  • Query the XBVR database.
  • Use the XBVR API used by the UI and backend server.

The key for the request parameters is the domain of the request, followed by -scraper or -trailers.

The correct name may not be obvious, and the scraper name and trailer name can differ.
Example:
naughtyamerica-scraper and api.naughtyapi-trailers.

To assist in finding the correct name:

  1. Set the DEBUG environment variable to true.
  2. Restart XBVR.
  3. Scrape a scene or view a trailer.
  4. Check the XBVR log for a message like: time="2025-02-27T07:09:07+13:00" level=debug msg="Using Header/Cookies from api.naughtyapi-trailers"

Loading request parameters via XBVR API or Import File

The endpoint is /api/options/save-collector-config

Example json body

{"domain_key":"test-trailer","body":"","cookies":[{"name":"nats","value":"nats","domain":".naughtyamerica.com","path":"/","host":"https://www.naughtyamerica.com"}],"headers":[{"key":"auth_token","value":"test token"}],"other":[]}

Heresphere Authentication

The token sent by Heresphere to authenticate to a site appears not to expire, though this likely depends on the site. This means you can set it up once, and it should continue to work without needing to change it.

Obtaining an auth_token

To get the auth_token for a site supporting the Heresphere API, make a POST request to their auth endpoint with your username and password.

Example Request:

POST to https://api.naughtyapi.com/heresphere/auth


{
 "username": "myusername",
 "password": "my password"
}

Example Response:

{
    "auth-token": "7564c613-aaa1-49a7-b9ad-3c23333332fb",
    "access": 1
}

You can use tools such as Postman or curl to make the request. Below is an example using curl on Windows:

curl --location "https://api.naughtyapi.com/heresphere/auth" --header "Content-Type: application/json" --data "{ \"username\" : \"myusername\", \"password\" : \"my password\"}"