Automated Testing - 509dave16/react-native-prototypes GitHub Wiki

Index

  1. Summary
  2. Testing Solution Factors
  3. Backend Testing
  4. Frontend Testing

Summary

We definitely need coverage on both ends of the platform. So we need testing done for the frontend and the backend. We also have different factors to consider for each end as we seek to find solutions.

Testing Solution Factors

General

  • Speed
  • Cost
  • Ease of Use
  • Regression Testing
  • Limitations - For example, could we test that an SMS API verification code message was received and worked when entered/submitted

Frontend Specific

  • Device Variability - For example a user reports a specific UX issue that all texts are cut off on the OnePlus device. But this does not occur on other devices. We should be able to remotely verify that that behavior is occurring on that specific device.
  • UI/UX Testing

Backend Specific

  • GraphQL Support

Backend Testing

API Testing

We have a Postman account which has support defining GraphQL requests. This could be used for test coverage and stress testing of our GraphQL API. As well as for a few non-GraphQL endpoints that we have.

Unit Testing

I would say that at minimum for unit tests on the backend, we need coverage for the Stripe order fulfillment and connect oauth callbacks. Since we won’t necessary be able to mock the callback requests from Stripe for API testing. These callback endpoints can be found in server/lib/routers/payments.js

E2E Testing

The following could be done manually or automated:

  • UX/UI Testing
  • Performance Testing

Manual Overview

Just need a Device Farm for manual testing. Tends to priced by slots or minutes. With slot pricing being fairly high.

  • Browserstack is the cheapest option while being unlimited but has a limited device selection. And if we wanted to have more than 1 active user at a time we would need to go from $40 to $130 to have up to 5 active users.
  • Experitest is basically the same pricing as Browserstack. $59 month to month for 1 user. $179 for up to 5 users. But they have the device selection that we need, which I believe supports more brands than Kobiton.
  • AWS Device Farm is flexible for pricing type but is more expensive($250/slot or $0.17/minute) and has limited device selection.
  • Kobiton would be cheapest by the minute with good device selection. For $40/month Team Plan, initial 500 min are $0.08 apiece. With additional minutes being $0.14.
  • Waldo would be good for this in that they allow unlimited users. Would only be worth it at the stage where we would be acquiring QA engineer and would choose instead to replace them.

Automation Overview

Automation for functional mobile tests is achieved by writing or generating test scripts that are capable of driving the application. For our use case, we will most likely be generating Appium scripts. So first off we need an editor/recorder for writing the test scripts. Secondly we need a service that hosts the devices that these test scripts be executed against.

Recorders

  • Kobiton - Free. https://kobiton.com/scriptless-test-automation
  • Appium Studio - Free. Desktop App. With Enterprise paid month to month option for enterprise support and executing tests on remote devices from the app.
  • Appium Desktop - Free. Desktop App. Integrates with Sauce Labs, Headspin, BitBar, Perfecto, TestingBot, BrowserStack, Kobiton, and pCloudy.com for recording on device clouds.
  • Katalon Studio - Free. Desktop App. Paid Month to Month option for full feature set to include other types of testing. They also have volume and bundled licenses options. Integrates with Kobiton, SauceLabs, and Browserstack.
  • MoQuality Barista - Free. Runs in the browser while communicating with your localhost through an agent program. Integrates with MQ, Mobile Labs, Sauce Labs, and Genymotion.
  • TestProject - Free. Runs in the browser while communicating with your localhost through an agent program. Will integrate with Browserstack and SauceLabs by end of 2019.
  • ZapTest - Free. But only available on Windows.
  • Waldo - $499/month per app. Produces recorded sessions that can replayed without relying on any test code. This would be most ideally if we just want a stakeholder, developer, or manager to create the recorded flows as a part of their responsibilities to avoid hiring a QA engineer.
  • Ranorex - Very expensive perpetual licenses
  • Test Complete - Very expensive perpetual licenses with post 1 year subscription for maintenance costs
  • Squish - Very expensive perpetual licenses that require post 1 year subscription for support and updates

Kobiton does provide this functionality. But the SLA for the provision of the Appium scripts from manual test session is 5 business days. I’m assuming this is what they have been working to improve so that it can be provided within a few minutes of completing a manual test session: https://docs.kobiton.com/automation-testing/appium-script-generation/

Device Farms

Then we need to select a device farm that we would like the Appium scripts to be executed against. In essence, these device farms have an Appium Server running for your account that receives request to execute Appium scripts on a group of devices.

There are two aspects to device farm pricing models:

  1. Number of parallel tests
  2. Number of minutes
  • Sauce Labs - Month to Month about 50% more than Annually
    • Number of parallel tests - 2+ : $349 : Price doubles as number doubles
    • Number of minutes - 2000 : Number of parallel tests doubles minutes : Has option for add on to have unlimited minutes. Price for that add on also doubles as number of parallel tests doubles. If not unlimited pay $0.20 for every minute over.
    • Supported Test Automation: Appium, Selenium, Espresso, XCUITest
  • Browser Stack - Automate App plan - Month to Month about 20% more than Annually
    • Number of parallel tests - 1+ : $159 : Price doubles as number doubles
    • Number of minutes unlimited
    • Supported Test Automation: Appium, EarlGrey, Espresso, XCUITest
  • Experitest - Automate App plan - Month to Month about 25% more than Annually
    • Number of parallel tests - 1+ : $249 : Price doubles as number doubles
    • Number of minutes unlimited
    • Supported Test Automation: Appium
  • Kobiton - Month to Month
  • Bitbar - Month To Month
    • Number of parallel tests - Unlimited : $39/$390
    • Number of minutes - 300+/2600+ : $0.29 for every minute over : Enterprise pricing is available on contact : Other option is to do all pay as you go through AWS Marketplace for $0.16 a minute
    • Supported Test Automation: Appium, Calabash, Espresso, XCUITest
  • AWS Device Farm - Month To Month
    • Number of parallel tests - Either Unlimited for pay as you go or based on Device Slots purchased at $250 apiece
    • Number of minutes - $0.17 a minute for pay as you go or unlimited if Device Slots purchased
    • Supported Test Automation: Appium, Calabash, UI Automation, XCUITest
  • Waldo - Month to Month - $499/month per app(i.e. foodbarrioQA for iOS)
    • Number of parallel tests - 1 + $50/month for each additional device for app platform
    • Number of minutes : Unlimited
    • Number of users : Unlimited