Sets of edits - atauenis/webone GitHub Wiki

The WWW has undergone many technological changes in recent years. Initially, browsers older than 5 years stopped opening web sites due to changes in image formats, HTML tags, JavaScript and CSS changes. Then traffic encryption (HTTPS) became a new trouble to old browsers. Also some interactive services in software, which using HTTP for network tasks, are stopped working after their servers shutdown or protocol changes.

WebOne can make all corrections to make particular websites (services) usable in particular old browsers (applications). It only needs specific instructions. These instructions are called Sets of edits, and commands from instructions are the Edits. They are stored in WebOne configuration files (*.conf). Anyone can write them, and there also is some common set, included in default installation.

Edit Set Gallery

You may see examples of Edit Sets in pre-installed configuration files (webone.conf, codepage.conf). Also there is a gallery of custom Edit Sets, designed for patching particular websites. Feel free to post your custom Edit Sets in it.

Syntax

The Set of edits can contain detection rules (determining when the Set should be used) and edition rules (what should be edited to make the web site usable in particular browser).

Most of rules are using Regular Expressions (RegExp/RegEx). See WWW, books and magazines for more details about RegEx match masks, capture groups and replacement masks.

Some rules are using 1/0, y/n, yes/no, on/off, enable/disable or true/false as value. Similar to Configuration file options, the Edit Set syntax is case sensitive.

Detection rules

The section title contains first (or single) URL regular expression mask determining on which URLs the Set should be used.

Some rules can be checked for both HTTP client requests and HTTP server responses (like URL), some can be only for HTTP responses (like status code or content type). So if the set is containing detection rules which cannot be checked on request time, WebOne will check and probably use the Set at the time of processing of remote server response.

OnUrl

If the URL in the title is not single, additional URL masks can be added as OnUrl rules. Example:

[Edit:jquery.min.js]
OnUrl=jquery2.js

;Will fire on:
; http://example.com/0458/jquery2.js
; https://example.com/0458/jquery2.js
; http://example.com/assets/javascript/jquery.min.js?assets_version=102
; http://example.com/libs/jquery/jquery.min.js

Added in v0.10.0.

  • Hint: when the Set should apply on all URLs, use [Edit:.*] title or its shortcut (added in v0.10.2): [Edit].
  • Use ^http:// address mask start for both HTTP and HTTPS requests. Currently the only way to differ protocols are OnHttpsOnly and OnHttpOnly detection rules (added in v0.16.0).

IgnoreUrl

If the OnUrl masks (or the mask in the title) is not so quite detailed, you may add similar masks that will determine URL black list. Example:

[Edit:bootstrap.min.js]
IgnoreUrl=2.3.1

;Will fire on:
; http://maxcdn.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js
; http://maxcdn.bootstrapcdn.com/bootstrap/4.5.6/js/bootstrap.min.js
; http://maxcdn.bootstrapcdn.com/bootstrap/1.0.0/js/bootstrap.min.js
;But not on:
; http://maxcdn.bootstrapcdn.com/bootstrap/2.3.1/js/bootstrap.min.js

Hint: Add IgnoreUrl=webdav rule in every set to disable them on WebDAV servers. Most of them are running on webdav.company.org domains, so the rule will work fine.

Added in v0.10.0.

OnContentType

It is possible to run the Set only on replies with particular MIME content types.

Added in v0.10.0.

OnCode

It is possible to run the Set only on replies with particular response code. It may be a real code (e.g. 200 or 403) or a "abbreviation":

  • 0 = less than 300
  • 2 = 2XX
  • 3 = 3XX
  • 4 = 4XX

Added in v0.10.0.

OnHeader

Sometimes it is need to run the Set only when the client sends particular request header. E.g. only for browsers which sending concrete User-Agent string. For those cases there is OnHeader rule. Example:

OnHeader=User-Agent: Mozilla/0
OnHeader=User-Agent: Mozilla/1
OnHeader=User-Agent: Mozilla/2
OnHeader=User-Agent: Mozilla/3

; set with these rules will apply only to traffic with old web browsers like Netscape 3.0
; because it sends "User-Agent: Mozilla/3.04Gold (WinNT; U)" header to every request.

Added in v0.10.1.

OnHttpsOnly

To enable the Set only for requests sent via HTTPS protocol, add this detection rule: OnHttpsOnly=yes.

Added in v0.16.0.

OnHttpOnly

Similar to above, but means only plain HTTP protocol.

Added in v0.16.0.

OnHostOS

This detection rule is useful if Set rely on a format converter, available only on particular operating systems. This rule disables the Set if the proxy server's OS is incorrect. Example:

; YouTube downloading on Win32 and GNU/Linux
[Edit:^(http://www\.|http://)youtube.com/watch.*&dl]
OnHostOS=Windows
AddConvert=yt.bat
AddResponseHeader=Content-Type: video/MP2T

[Edit:^(http://www\.|http://)youtube.com/watch.*&dl]
OnHostOS=Linux
AddConvert=yt.sh
AddResponseHeader=Content-Type: video/MP2T

Possible values: Windows, Linux, macOS.

Added in v0.12.1.

Editing rules

If the Set is applicable for current HTTP request/response, all of Set's editing rules will be applied to the operation.

Available masks

Every editing rule can contain masks (e.g. %UrlNoQuery%), which will be replaced by actual content when processed by WebOne.

  • %URL% - original URL (non encoded). Added in v0.4.0.
  • %Url% - URL-encoded original URL (https://example.com -> https%3A%2F%2Fexample.com), don't confuse with %URL% which is not encoded. Added in v0.9.2.
  • %UrlNoQuery% - original URL without ?arguments=. Added in v0.9.2.
  • %UrlNoPort% - original URL without port number (if any). Added in v0.8.1.
  • %UrlNoDomain% - part of original URL without domain name. Added in v0.6.1.
  • %UrlDomain% - domain name of the original URL. Added in v0.10.5.
  • %UrlHttps% - original URL with forced HTTPS protocol. Added in v0.9.2.
  • %UrlHttp% - original URL with forced insecure HTTP protocol. Added in v0.9.2.
  • %ProxyHost% - proxy host name. May be useful for using of in-proxy converters. Added in v0.9.0.
  • %ProxyPort% - proxy working port. Added in v0.9.0.
  • %Proxy% - shortcode for %ProxyHost%:%ProxyPort% which also may be equal to DefaultHostName setting. Added in v0.9.2.
  • %Request.header-name% - content of client request header. Added in v0.13.0.
  • %Response.header-name% - content of server response header. Added in v0.13.0.
  • %Response.HttpStatusCode% - server response status code. Added in v0.13.0.

AddRedirect

Return a HTTP-302 redirect to a particular URL.

Since v0.10.3 it can utilize RegEx capture groups from Edit title. E.g. if there are single URL group in the section title (no OnUrl rules), it may look like [Edit:(^http://.*/styles/prosilver/theme/)(.*)]. This will hit on all phpBB forums with ProSilver theme. Then add a redirect to patched theme file: AddRedirect=http://atauenis.github.io/webone-webfixes/prosilver/theme/$2. So $2 will be replaced with file name.

Added in v0.10.0, enhanced in v0.10.3.

AddInternalRedirect

Same as AddRedirect, but return a page from that URL without saying to the browser that the requested page is really a one from the redirect destination. Ignored on response processing. Added in v0.10.0.

AddHeader

Add a header(s) to HTTP/HTTPS request. Ignored on response processing. Added in v0.10.0.

AddResponseHeader

Add a header(s) to HTTP response. May be useful for manual adding of cookies, plugin overriding, etc. Added in v0.10.0.

AddConvert

Convert the content using a external format converting utility. For example, convert a WebP image to GIF, compress a JPEG picture for slow modem line or trans-code a 8K h264 video to Intel Indeo with low bit rate and resolution.

The argument is name of converter utility.

This rule should be used together with additional parameter-setting rules: AddConvertDest, AddConvertArg1, AddConvertArg2, etc.

;Example: convert all PNGs and WEBPs to GIF.
[Edit:.*]
OnContentType=image/png
OnContentType=image/webp
AddConvert=convert
AddConvertDest=gif
AddResponseHeader=Content-Type: image/gif

Added in v0.10.0.

AddConvertDest, AddConvertArg1, AddConvertArg2

See above. All were added in v0.10.0.

AddFind, AddReplace

Add these rules to apply edits to text part of responses. E.g. replace too modern JavaScript functions to their older equivalents, hide inline scripts, edit CSS on fly.

One set can contain multiple AddFind/AddReplace edits, but count of both them must be similar.

;Example: JS patch for Old-DOS.ru
[Edit:^http://old-dos.ru]
AddFind=eval\(('|")(.*?)\1\)
AddReplace=$2
AddFind=var _tmr = _tmr .* \[\];
AddReplace=var _tmr = _tmr;

Added in v0.10.0.

AddResponseHeaderFind, AddResponseHeaderReplace

Similar to above, but for HTTP response headers (their content only). Added in v0.11.0.

AddRequestHeaderFind, AddRequestHeaderReplace.

Similar to AddFind, AddReplace, but for HTTP request headers (their content only). Added in v0.11.0.

AddOutputEncoding

Set output encoding (code page) for this request. See OutputEncoding in configuration file for syntax. Added in v0.11.0.

AddTranslit

Enable (AddTranslit=yes) or force disable (AddTranslit=no) replacing of non-English letters in response body to their equivalents in Latin alphabet (text transliteration, e.g. Cyrillic "У" to Latin "U" or Greek "Σ" to Latin "S".). The configuration file(s) must have [Translit] section with transliteration table. Added in v0.11.0.

AddDumping

This feature is intended only for debug or research purpose. When this rule is added in the Set, all HTTP traffic will be saved to specified file. The file name may contain %Url% mask. Example:

[Edit]
AddDumping=dump-%Url%.log

Saved dump of the HTTP traffic contains both request (including headers and, if any, POST method body) and response. Also WebOne puts into dump all applied Edit Sets and other information about request processing. If request or response body is not text-based or is larger than 1KB, it's omitted in the dump.

This feature can be used to debug websites in retro browsers without any Developer Tools available. Or even for some HTTP-based services.

If the specified file is exists, it will be appended. Note that enabling the sniffing will slow down the proxy, and can fill entire disk space. Added in v0.12.3.

AddHeaderDumping

Since WebOne v0.10.2 there was AddHeaderDumping and AddRequestDumping rules, which do almost same as the current AddDumping. But the first was without body and the second was without response. Now they have AddDumping behavior, and support has kept for backward compatibility only. Added in v0.10.2, removed in 0.12.3.

AddRequestDumping

See above. Added in v0.10.2, removed in 0.12.3.

Edits for non-HTTP(S) protocols

The only available edit for protocols other than HTTP/HTTPS (such as IRC or OSCAR) is connection redirect. Use [NonHttpConnectRedirect] sections instead of [Edit] to redirect non-HTTPS connections. But be careful - most applications, like ICQ (modern) or Windows Messenger, are working with HTTP gateways or with HTTP-based API. So regular edit sets are useful for them. The WebOne log is always displaying which protocol is really used.

Added in v0.16.0.