Other example of host: config options, cookies, cloudflare protection and captchas - oe-mirrors/e2iplayer GitHub Wiki

Sites that want a login, keep a session in cookies, sit behind Cloudflare or show a captcha need a few more parts in a host. This page shows the building blocks E2iPlayer has for them.

A host's options are shown with the BLUE key in the host list. They are ordinary Enigma2 config entries, defined at the top of the host file and returned by GetConfigList():

from Components.config import config, ConfigSelection, ConfigYesNo, getConfigListEntry
from Plugins.Extensions.IPTVPlayer.components.configsecret import ConfigLogin, ConfigSecret

config.plugins.iptvplayer.mysite_login = ConfigLogin(default="", fixed_size=False)
config.plugins.iptvplayer.mysite_password = ConfigSecret(default="", fixed_size=False)
config.plugins.iptvplayer.mysite_language = ConfigSelection(default="en", choices=[("en", _("English")), ("de", _("German"))])
config.plugins.iptvplayer.mysite_show_trailers = ConfigYesNo(default=False)


def GetConfigList():
    optionList = []
    optionList.append(getConfigListEntry(_("login") + ":", config.plugins.iptvplayer.mysite_login))
    optionList.append(getConfigListEntry(_("password") + ":", config.plugins.iptvplayer.mysite_password))
    optionList.append(getConfigListEntry(_("Language") + ":", config.plugins.iptvplayer.mysite_language))
    optionList.append(getConfigListEntry(_("Show trailers"), config.plugins.iptvplayer.mysite_show_trailers))
    return optionList

• Name every option config.plugins.iptvplayer.<hostname>_<option> - all hosts share this namespace.
• Read the value where it is used (config.plugins.iptvplayer.mysite_language.value), not once in __init__ - the user can change it while the host is open.
• Passwords, API keys and tokens: ConfigSecret shows ***** instead of the text in every config list and the web interface never sends it out; ConfigLogin shows xxxxx while it is empty. Both are stored like a ConfigText. components/configsecret.py exists only in this fork - for a host that should also run on e2iplayer-zadmario, use ConfigText there.
• Options that belong to all hosts (watched flag, naming, metadata providers, captcha services ...) are global settings. A host does not add its own copy of them.

All requests go through self.cm.getPage(url, params, post_data) (libs/pCommon.py). A host keeps one parameter dict and passes a copy of it:

CBaseHostClass.__init__(self, {'history': 'mysite', 'cookie': 'mysite.cookie'})   # sets self.COOKIE_FILE
self.HEADER = self.cm.getDefaultHeader(browser='chrome')
self.defaultParams = {'header': self.HEADER, 'use_cookie': True, 'load_cookie': True, 'save_cookie': True, 'cookiefile': self.COOKIE_FILE}

def getPage(self, url, addParams=None, post_data=None):
    if addParams is None:
        addParams = dict(self.defaultParams)
    return self.cm.getPage(url, addParams, post_data)

Use one User-Agent for the whole host: session cookies and Cloudflare clearances are bound to it. getDefaultHeader(browser=...) knows chrome, firefox, edge, safari, android, iphone, vlc and more (common.USER_AGENTS); with randomUA=True it picks one of a group - call it once in __init__, not per request.

Useful parameters:

After a request self.cm.meta holds url (the final url after redirects), status_code and the response headers. Cookie helpers: self.cm.getCookieItem(self.COOKIE_FILE, 'name'), self.cm.getCookieHeader(self.COOKIE_FILE), self.cm.clearCookie(self.COOKIE_FILE, removeNames=['session']).

The usual pattern: log in once when the host needs it, and again only when the user changed the credentials.

def __init__(self):
    ...
    self.loggedIn = None
    self.login = ''
    self.password = ''

def tryTologin(self):
    login = config.plugins.iptvplayer.mysite_login.value
    password = config.plugins.iptvplayer.mysite_password.value
    if self.loggedIn is not None and login == self.login and password == self.password:
        return self.loggedIn
    self.login, self.password = login, password
    self.loggedIn = False
    if login.strip() == '' or password.strip() == '':
        return False

    sts, data = self.getPage(self.getFullUrl('/login'))          # form page: hidden fields, session cookie
    if not sts:
        return False
    post = {'username': login, 'password': password}
    token = self.cm.ph.getSearchGroups(data, r'''name=['"]_token['"][^>]+value=['"]([^'"]+)['"]''')[0]
    if token:
        post['_token'] = token
    params = dict(self.defaultParams)
    params['header'] = dict(self.HEADER, Referer=self.getFullUrl('/login'))
    sts, data = self.getPage(self.getFullUrl('/login'), params, post)
    if sts and '/logout' in data:
        self.loggedIn = True
    else:
        GetIPTVNotify().push(_('Login failed.'), 'error', 10)
    return self.loggedIn

Call it at the start of listMainMenu() (or before the first request that needs it). The session cookie lands in self.COOKIE_FILE and is sent with every later request. Never write a login, password or API key into the host file.

Use getPageCFProtection() instead of getPage():

def getPage(self, url, addParams=None, post_data=None):
    if addParams is None:
        addParams = dict(self.defaultParams)
    return self.cm.getPageCFProtection(url, addParams, post_data)

On a normal answer it behaves like getPage(). When the answer is a browser check (Cloudflare "Just a moment...", DDoS-Guard, Sucuri, Imperva ...), it identifies the protection (libs/botprotection.py) and hands the page to MyE2i: the user opens the shown address in the browser of a phone or PC with the MyE2i extension, solves the check there, and the clearance cookie and the browser's User-Agent come back into the host's cookie jar. Following requests pass without a new check until the site asks again. Setup for users: Solve Cloudflare / hCaptcha / reCAPTCHA with MyE2i.

For this to work, defaultParams needs cookiefile and the cookie flags (see above) - the clearance is stored there. A hard block (the site refuses the IP or the country) cannot be solved by a browser; then getPageCFProtection() returns sts = False without starting MyE2i.

curl-impersonate (Chrome fingerprint): some Cloudflare setups only look at the TLS / HTTP2 fingerprint of the client - urllib, curl and pycurl get "Just a moment...", a client with Chrome's fingerprint gets the page without any cookie or browser. When the box has curl-impersonate (see Install → Helper programs), getPageCFProtection() tries that first on a Cloudflare challenge, before MyE2i. If it passes, the domain is remembered for the session and every later request to it (pages, covers, subtitles, downloads) goes the same way. Nothing to do in the host for that.

A host that knows its site needs it can ask for it directly:

sts, data = self.cm.getPage(url, {'impersonate': True})   # True = as Chrome, False = never, missing = automatic

Without the binary the normal request is made. For HLS streams whose server refuses every normal client (HTTP 403 for hlsdl and the players' ffmpeg), the urlparser marks the links with iptv_impersonate_hls (urlparserhelper.getImpersonateM3U8Playlist()): they are played buffered and downloaded through the curl-impersonate hls downloader.

For a Google reCAPTCHA (also v3/Enterprise), hCaptcha or Turnstile inside the site (login form, "show links" button), derive the scraper from CaptchaHelper:

from Plugins.Extensions.IPTVPlayer.components.captcha_helper import CaptchaHelper


class MySite(GenericFolderWatchedScraperMixin, CBaseHostClass, CaptchaHelper):
    ...
    def getLinksForVideo(self, cItem):
        sts, data = self.getPage(cItem['url'])
        ...
        sitekey = self.cm.ph.getSearchGroups(data, r'''data-sitekey=['"]([^'"]+)['"]''')[0]
        if sitekey:
            token, errorMsgTab = self.processCaptcha(sitekey, self.cm.meta['url'])
            if token == '':
                SetIPTVPlayerLastHostError('\n'.join(errorMsgTab))
                return []
            sts, data = self.getPage(cItem['url'], dict(self.defaultParams), {'g-recaptcha-response': token})
        ...

processCaptcha(sitekey, refUrl, ..., captchaType='', captchaAction='') first tries Google's image challenge on the TV, then the service the user chose in Settings → Captcha configuration (MyE2i, 2captcha.com, 9kw.eu, DeathByCaptcha, or MyJDownloader when its login is set). It returns the token ('' on failure) and a list of messages for the user. Score-based reCAPTCHA (v3/Enterprise, pass captchaAction) and Turnstile with action/cdata work only with MyE2i; hCaptcha with 2captcha needs captchaType='h1'.

Based on the original text by Maxbambi, thank you!