A more complex example of host : building item lists from HTML - oe-mirrors/e2iplayer GitHub Wiki

E2iPlayer is basically a browser without a screen: a host loads the pages of a site, searches the HTML (or JSON) for titles, images and links, and turns them into lists. So the main skill is reading a page's source and finding stable markers in it.

This page walks through 🔗hostTemplate.py, a complete movie/series host. The site it is written for is made up, the HTML it expects is shown here and in the comments of the file - with a real site you replace the markers, the structure stays the same.

1. Start from the template

Copy hostTemplate.py to IPTVPlayer/hosts/hostmysite.py, replace template/Template by your host name and set the main url:

def gettytul():
    return 'https://www.mysite.com/'

self.getFullUrl('/movies/') turns relative links of the site into full urls based on this main url.

2. Look at the site first

Open the site in a desktop browser with the developer tools (F12):

  • Elements / view-source: the HTML of the lists - what surrounds one title, its link, its image, the pagination.
  • Network: many sites load their lists or the player with a separate request (XHR/fetch, often JSON). That request is usually easier and more stable to use than the HTML around it.
  • Compare the source of the page (Ctrl+U) with the elements view: what only appears in the elements view is created by JavaScript - the host does not run JavaScript, so it has to find where the data comes from.

On the box, printDBG(data) writes what the host really received into the debug log - sites often answer a box (other IP, no JavaScript, other User-Agent) with something else than your browser.

3. The main menu

self.MENU = [
    {'category': 'list_items', 'title': _('Movies'), 'url_tpl': self.getFullUrl('/movies/page/{page}/')},
    {'category': 'list_items', 'title': _('Series'), 'url_tpl': self.getFullUrl('/series/page/{page}/')},
] + self.searchItems()

Labels are written in English inside _(), so they can be translated. self.searchItems() adds Search, Search history, Edit search history and Delete search history (see Building a search function).

url_tpl is the url of the list with {page} for the page number - the list function and the paging rows use it (see Host standard → paging).

4. handleService: the router

Every time the user opens a row, handleService() runs with that row in self.currItem. Its category decides which function fills the new list:

def handleService(self, index, refresh=0, searchPattern='', searchType=''):
    CBaseHostClass.handleService(self, index, refresh, searchPattern, searchType)
    if isJumpItem is not None and isJumpItem(self.currItem):
        self.currItem = jumpTarget(self, self.currItem)
    name = self.currItem.get('name', '')
    category = self.currItem.get('category', '')
    self.currList = []

    if name is None:                       # the host was just opened
        self.listsTab(self.MENU, {'name': 'category'})
    elif category == 'list_items':
        self.listItems(self.currItem)
    elif category == 'list_seasons':
        self.listSeasons(self.currItem)
    elif category == 'list_episodes':
        self.listEpisodes(self.currItem)
    elif category in ('search', 'search_next_page'):
        cItem = dict(self.currItem)
        cItem.update({'search_item': False, 'name': 'category'})
        self.listSearchResult(cItem, searchPattern, searchType)
    elif category == 'search_history':
        self.listsHistory({'name': 'history', 'category': 'search'}, 'desc')
    else:
        printExc()

    CBaseHostClass.endHandleService(self, index, refresh)

The category names are yours. Video rows are not routed here - opening a video calls getLinksForVideo().

5. A list from HTML

The movie list of the site looks like this:

<div class="items">
<article class="item movie"><a href="/movie/the-general-1926/"><img src="/img/the-general-1926.png" alt="The General"></a>
<h3 class="title">The General</h3><span class="year">1926</span><p class="plot">Plot of The General &amp; more.</p></article>
<article class="item movie"> ... </article>
</div>
<div class="pagination"><span class="current">1</span><a class="page-numbers" href="/movies/page/2/">2</a>
<a class="page-numbers" href="/movies/page/3/">3</a><a class="next page-numbers" href="/movies/page/2/">Next</a></div>

First load the page. getPage() returns (sts, data): sts is False when the request failed (no connection, HTTP error) - then stop:

def listItems(self, cItem):
    page = int(cItem.get('page', 1) or 1)
    sts, data = self.getPage(cItem['url_tpl'].format(page=page))
    if not sts:
        return

Then cut the page into one piece per title. The parsing helpers are in self.cm.ph (libs/pCommon.py, class CParsingHelper):

for item in self.cm.ph.getAllItemsBeetwenMarkers(data, ('<article', '>', 'item'), '</article>'):

A marker can be a string or a tuple (start, end, must contain): ('<article', '>', 'item') is any <article ...> tag that contains item between <article and > - whatever the other attributes are. That is more robust than the exact string '<article class="item movie">', which breaks as soon as the site adds a class.

From each piece take what you need:

    itemUrl = self.getFullUrl(self.cm.ph.getSearchGroups(item, r'''href=['"]([^'"]+)['"]''')[0])
    title = self.cleanHtmlStr(self.cm.ph.getDataBeetwenNodes(item, ('<h3', '>'), ('</h3', '>'), False)[1])
    icon = self.getFullIconUrl(self.cm.ph.getSearchGroups(item, r'''src=['"]([^'"]+)['"]''')[0])
    year = self.cm.ph.getSearchGroups(item, r'class="year">\s*(\d{4})')[0]
    desc = self.cleanHtmlStr(self.cm.ph.getDataBeetwenNodes(item, ('<p', '>', 'plot'), ('</p', '>'), False)[1])
helper returns
getSearchGroups(data, regex) list of the regex groups, '' for each when nothing matched - so [0] is always safe
getDataBeetwenMarkers(data, m1, m2, withMarkers=True) / getDataBeetwenNodes(...) (found, text) of the first match - take [1]
getAllItemsBeetwenMarkers(data, m1, m2, withMarkers=True) / getAllItemsBeetwenNodes(...) list of all matches
rgetDataBeetwenMarkers2(...), rgetAllItemsBeetwenMarkers(...) the same, searching from the end
self.cleanHtmlStr(html) text without tags, entities decoded (&amp; → &), whitespace cleaned

withMarkers=False returns the text without the markers. The helpers do not understand nesting: the end marker is the first </div> after the start, also when it closes an inner <div>. Pick end markers that cannot appear inside the part you want, or cut in two steps.

Then add the row:

    isSeries = '/series/' in itemUrl
    params = {'name': 'category', 'good_for_fav': True, 'url': itemUrl, 'icon': icon, 'desc': desc,
              's_title': title, 'meta_type': 'tv' if isSeries else 'movie', 'meta_title': title, 'meta_year': year}
    if isSeries:
        params.update({'category': 'list_seasons', 'title': title})
        self.addDir(params)
    else:
        params.update({'category': 'video', 'title': '%s (%s)' % (title, year) if (normalize and year) else title})
        self.addVideo(params)

Why a new dict per row, what good_for_fav, s_title and the meta_* fields are for, and why the title gets the year: see Host standard. Movies are addVideo() rows that point to the movie page; the links are fetched only when the user opens the movie.

At the end of the list, the paging rows:

pagination = self.cm.ph.getDataBeetwenNodes(data, ('<div', '>', 'pagination'), ('</div', '>'), False)[1]
hasNext = 'class="next' in pagination
lastPage = max([int(x) for x in re.findall(r'>\s*(\d+)\s*<', pagination)] or [0])
self.addPaging(cItem, page, hasNext, lastPage)

6. Seasons and episodes

The page of a series:

<div class="season" data-season="1"><ul>
<li class="episode"><a href="/episode/sherlock-1x1/">A Study in Pink</a><span class="num">1x1</span></li>
<li class="episode"> ... </li>
</ul></div>
<div class="season" data-season="2"><ul> ... </ul></div>

listSeasons() adds one folder per data-season, listEpisodes() cuts out the block of the chosen season and adds one video per episode:

data = self.cm.ph.getDataBeetwenNodes(data, ('<div', '>', 'data-season="%s"' % season), ('</ul', '>'), False)[1]
for item in self.cm.ph.getAllItemsBeetwenMarkers(data, ('<li', '>', 'episode'), '</li>'):
    url = self.getFullUrl(self.cm.ph.getSearchGroups(item, r'''href=['"]([^'"]+)['"]''')[0])
    epName = self.cleanHtmlStr(self.cm.ph.getDataBeetwenNodes(item, ('<a', '>'), ('</a', '>'), False)[1])
    numLabel = self.cleanHtmlStr(self.cm.ph.getDataBeetwenNodes(item, ('<span', '>', 'num'), ('</span', '>'), False)[1])
    epNum = self.cm.ph.getSearchGroups(numLabel, r'\d+x(\d+)')[0]
    ...
    params = dict(cItem)
    params.update({'good_for_fav': True, 'category': 'video', 'title': title, 'url': url, 'episode': epNum})
    self.addVideo(params)

Here dict(cItem) is right: the episode should keep the fields of its series (s_title, meta_*, icon). The episode title follows the naming rules (Sherlock - S01E01 - A Study in Pink).

7. Links

The page of a movie or episode lists its mirrors:

<div class="plot">The story of ...</div>
<ul class="mirrors"><li data-link="https://voe.sx/e/abc123">VOE</li><li data-link="https://otherhoster.com/e/1">Other</li></ul>
def getLinksForVideo(self, cItem):
    sts, data = self.getPage(cItem['url'])
    if not sts:
        return []
    plot = self.cleanHtmlStr(self.cm.ph.getDataBeetwenNodes(data, ('<div', '>', 'plot'), ('</div', '>'), False)[1])
    linksTab = []
    for url in re.findall(r'''data-link=['"]([^'"]+)['"]''', data):
        url = self.getFullUrl(url)
        if self.up.checkHostSupport(url) != 1:
            continue
        linksTab.append({'name': self.up.getHostName(url, True), 'url': strwithmeta(url, {'Referer': cItem['url']}), 'need_resolve': 1})
    if not linksTab:
        SetIPTVPlayerLastHostError(_('This video is only on hosters E2iPlayer cannot play.'))
        return []
    return applySidecarToLinks(linksTab, buildSidecarFromItem(cItem, IsSidecarEnabled(), plot))


def getVideoLinks(self, videoUrl):
    if not self.cm.isValidUrl(videoUrl):
        return []
    sidecar = sidecarFromUrlMeta(videoUrl, IsSidecarEnabled())
    return decorateResolvedLinkItems(self.up.getVideoLinkExt(videoUrl), sidecar)
  • self.up.checkHostSupport(url) is 1 when the urlparser knows the hoster. Unknown hosters are left out instead of offering links that cannot play.
  • need_resolve: 1 means: this is a hoster page, not a stream. When the user picks it, E2iPlayer calls getVideoLinks(url), which asks the urlparser for the stream. A direct stream (.mp4, .m3u8) gets need_resolve: 0.
  • strwithmeta(url, {...}) attaches HTTP headers to a url - the urlparser and the players send them. Many hosters need the Referer of the page that embeds them.
  • The sidecar calls pass plot and poster on to downloads, see Host standard → sidecar.

How the urlparser finds the stream and how to add a hoster it does not know: Understanding Urlparser.

8. When the links are not in the HTML

Often the page only shows a player button, and the link appears after a click. The link is then built by JavaScript - from data that is somewhere: in data-* attributes, in a JSON block inside a <script>, or loaded with a request you can see in the Network tab of the developer tools.

A typical case: the episodes carry only ids

<div class="player" data-voe="abc123" data-dood="" data-filemoon="x9y8z7"></div>

and the developer tools show that clicking "VOE" loads https://voe.sx/e/abc123. So the host builds the urls itself:

for attr, urlTpl in (('data-voe', 'https://voe.sx/e/%s'), ('data-filemoon', 'https://filemoon.sx/e/%s')):
    videoId = self.cm.ph.getSearchGroups(item, attr + r'''=['"]([^'"]+)['"]''')[0]
    if videoId:
        linksTab.append({'name': attr[5:], 'url': urlTpl % videoId, 'need_resolve': 1})

For a JSON answer use json_loads from libs/e2ijson.py instead of searching the text with regular expressions.

Based on the original text by Maxbambi, thank you!

Add a custom footer
⚠️ **GitHub.com Fallback** ⚠️