Transparency Scoring - bmixonba/vpn-osint GitHub Wiki
VPNs are security critical applications that people use to bypass geo-blocking, circumvent censorship, and protect their connections from adversaries in repressive environments. Many VPNs in repressive countries have well over 1,000,000 downloads and monthly active users. Their security is dependent on the owners, operators, and developers of these systems yet when the details about who owns, operates, and develops these systems are unavailable, it calls into question the intentions of these parties. The VPN-OSINT Transparency Report Project has three primary goals:
1. Identify the owners, operators, and developers of VPN software used by people in repressive countries to bring transparency to this space
2. Inform users about the degree to which VPN services are operating transparently, and
3. Encourage large organizations like Apple and Google to take the safety of individuals of repressive environments more seriously by clearly identifying VPN applications that operate transparently and those that do not.
The VPN-OSINT Transparency Report will achieve this by collecting and aggregating open source intelligence (OSINT) from disparate sources on the web, synthesizing this information using the Transparency Score, a metric similar to a credit score, and providing details about the transparent and non-transparent organizations in the VPN landscape as well as making recommendations on how organizations can increase their transparency. This scoring methodology is different from simply ranking VPNs from “best” to “worst” because providers that are open about their development, VPNs such as TunnelBear, Lantern, Psiphon, and Mullvad, who appear to operate transparently all score equally high (850), whereas other VPNs such as those listed in Table 1, score low (500). This is desirable as it does not create contentious situations where VPNs have to compete against each other for a single top spot. The preliminary analysis indicates that TunnelBear, Lantern, Psiphon, and Mullvad all score high on transparency and so no contention is created in the community - transparency is rewarded equally.
Data are collected from sites like the Google Play and Apple App store, company websites, WHOIS records, social media sites such as Reddit, Twitter, Telegram, Hackernews, public git repositories, as well as analytical techniques based on automated analysis to compare source code to detect repackaging of applications, and CrypotSluice to analyze transport layer security. Finally, for a selection of applications, manual analysis will be conducted to determine whether the applications have other weaknesses, such as those introduced by third-party libraries, that could undermine user security through data exposure or other issues
Table 1. Below are a set of VPN apps with tens to hundred of millions of total downloads that are also popular in Indonesia, Russia, India, Pakistan, Saudi Arabia, Brazil, and the UAE. I found several indicators for each app that they are likely candidates for further consideration and manual analysis. Each of these apps either had no website, a website that is poorly designed or only a few characters long, or the website listed on the Google Play store simply points back to the Play store. Some websites did have links to social media such as Twitter telegram, or Reddit, but the links do not point to anything and few references to these apps exist online. None of the applications appear to have code on Github, gitlab, or gitee suggesting they are not open source.
Automated data collection will be sourced from websites such as sensortower, Google, ChatGPT, Github, gitlab, gitee, Twitter, Reddit, Hacker News, etc. I will initially rely on US-based web services such as those listed (the exception is gitee which is a Chinese git service) to bootstrap data sourcing and add foreign data sources such as websites as they are discovered.
Data will be collected from websites such as sensortower.com, appfigures.com, Google, Github, gitlab, gitee, Twitter, Reddit, Hacker News, telegram, discord, etc. I will initially rely on US-based web services such as those listed (the exception is gitee which is a Chinese git service) to bootstrap data sourcing and add foreign data sources such as websites as they are discovered.
The current approach is a composite scoring system using the following equation:
Transparency Score = b * B + n * N + d * D + s * S + m * M
The upper case letters represent a numeric value associated with each Transparency Factor and the lower case represent factor weights. The following describes proposed information included in each transparency factor:
- Business Operations
- Social Media
- Network Operations
- Developer
- Miscellaneous
will consider factors such as whether the VPN has an associated web page, whether there is an “About” page with information about the company’s staff, developers, management team, and other individuals; email addresses, such as whether it appears to be a personal gmail account or a business email address; whether it has a privacy policy; what specific user-information the company collects based on the account creation process and what is stated in the privacy policy, and from where the organization is headquartered. Organizations with a functional website that actually exists, such as Mullvad, Psiphon, Lantern, and TunnelBear have such websites, would score well for Business Operations Transparency, whereas the apps listed in Table 1 by and large do not have functional websites score lower. Organizations that are headquartered from countries with high Reporters without Borders (https://rsf.org/en/index) and Freedom House (https://freedomhouse.org/countries/freedom-world/scores) indices will have higher scores than those with lower scores. An exception I will consider is infrastructure location. Different regions govern PII, data retention, and communications differently. While a provider may be located in a less repressive region, they will likely be required to collect specific information to comply with local laws. I will identify the VPN server locations as stated in their server list. I will factor in how the company handles data and provide details about how their policy differs across infrastructure geographies, which should be provided in the privacy policy or similar documentation.
Social Media & Marketing Transparency considers factors such as whether the VPN has a social media footprint (e.g., on Twitter, Reddit, telegram, facebook, instagram, Mastodon, irc, slack, discord and other social platforms), the size of their social network, how they market their product, such as through website ads, social media campaigns, through influencer advertisements, whether they provide localized materials for specific languages, and work with local actors to promote Internet freedom. Applications with at least one social media account will score high in this factor, whereas ones with none will score low. Social media interaction indicates the developer/organization engages with the community which is a sign of transparency. Mullvad, Psiphon, TunnelBear, and Lantern for example have at least a Twitter presence and score high, whereas the apps in Table 1 do not and receive a low score. Marketing identification will be challenging because specific VPNs may only market to smaller communities or influencers that I may not be able to identify. I will attempt to identify as much
Considers factors such as whether the app has a git repository, whether the project is open source, what parts of the project are open source, code documentation, and whether they provide security audit information. Projects with a public code repository and that are open source will receive a high score because they have demonstrably transparent development practices, while the lack of them indicates the opposite and hence a lower score. Developer Transparency will factor in client code and configuration, server code and configuration, code comments. Verifying that the code is present and runs as expected is likely a manual process as is reviewing code comments, which developers are notoriously bad at maintaining. The five low scoring and five high scoring apps will additionally include manual and dynamic client analysis such as consistency between the stated privacy policy, information listed in the AndroidManifest.xml file, and how that information is used in the app and transmitted, whether the app is repackaged and/or has compromising additions, and other analyses related to how Android implements its VPN service permission on Linux.
Network Operations Transparency considers factors such as WHOIS information, whether they use DNSSEC, Registrant Organization in the associated WHOIS record, emails and person of contact. This category is more informational than contributing to the overall score, but in some instances, such as Mullvad, there is additional information suggesting Mullvad operates transparently. Also, many developers either redact information for privacy or the WHOIS records point to a CDN such as CloudFlare. For the five low scoring and five high score apps, more technical information about the transport layer security may be derived from CryptoSluice output which could assess whether the application leaks sensitive client information. Apps with large volumes of invariant content would receive a lower score while those without would score high. Finally, analysis of the servers, such as whether they are susceptible to Port Shadowing or similar attacks as performed in the analysis phase will be included in this Transparency Factor. Apps that are susceptible to such attacks will score low and this would have a high impact on their score whereas lack of such attacks would contribute to a higher score. While the manual analysis portion of the score is more security focused as opposed to business, organizational, or developer transparency, such violations are relevant in the sense that there is an assumption that VPN products protect users and information leaks indicate a lack of transparency.
Miscellaneous Information considers other information that is not directly related to any of the other factors. For example, Mullvad has an onion service and lists a PGP key on their website. This information suggests that Mullvad cares about security and transparency in general.
Members of my support team and I will focus on a technical-heavy and comprehensive report detailing methodologies for data collection, app selection, analysis, scoring. I will use this as the basis for generating non-technical reports used for outreach and advocacy by working with the Usability Lab to tailor the output towards those audiences. I have also attached a mockup report based on preliminary analysis of high and low scoring VPN apps that depicts the envisioned reporting output for non-technical stakeholders. The mockup includes tables that rank VPN apps from high to low transparency scores so that at-a-glance users can assess which apps are more or less transparent. This is followed up with a further breakdown of each VPN’s composite score and the scores of each factor individually. This section may also optionally include details about why a particular factor was ranked in a particular way and provide recommendations to address the specific issue. This information will provide recommendations to technical audiences on how to address lack of transparency but can be excluded from reports to non-technical audiences