Campaigns Testing Sequence - Automattic/newspack-popups GitHub Wiki

Campaigns functional tests (h/t/ @dkoo)

Test 1

Test 1 conditions

  • Campaign A: Overlay campaign, sitewide default, contains newsletter prompt
  • Campaign B: Inline campaign, displays every page, contains newsletter prompt
  • Campaign C: Inline campaign, displays every page, contains newsletter prompt
  • Campaign D: Inline campaign, displays every page, does NOT contain newsletter prompt
  • Suppress Newsletter campaigns if visitor is coming from email: ENABLED
  • Suppress all Newsletter campaigns if at least one Newsletter campaign was permanently dismissed: ENABLED

Test 1 results

  • When viewing the site for the first time in a new incognito window: all campaigns are displayed, as expected.
  • After refreshing the page, without interacting with any campaigns: all campaigns still show, as expected.
  • After dismissing Campaign D (no newsletter prompt), it does not reappear after refreshing the page. Campaigns A-C do appear as expected.
  • NOT WORKING: After dismissing Campaign A, B, or C (contains a newsletter prompt): I encounter the utm_suppression notice highlighted below on subsequent page refreshes, and the other newsletter prompt campaigns are not suppressed.
  • Loading a page in a new incognito window with the utm_medium=email param: Campaigns A-C are suppressed, while Campaign D is not, as expected.
  • After navigating to another page, without the utm_medium param: Campaigns A-C are still not displayed, as expected.

Test 2

Test 2 conditions

  • Campaigns A-D: Same as in Test 1
  • Suppress Newsletter campaigns if visitor is coming from email: DISABLED
  • Suppress all Newsletter campaigns if at least one Newsletter campaign was permanently dismissed: ENABLED

Test 2 results

  • Loading a page in a new incognito window with the utm_medium=email param: All campaigns are displayed, as expected.

Test 3

Test 3 conditions

  • Campaigns A-D: Same as in Tests 1-2
  • Suppress Newsletter campaigns if visitor is coming from email: ENABLED
  • Suppress all Newsletter campaigns if at least one Newsletter campaign was permanently dismissed: DISABLED

Test 3 results

  • Loading a page in a new incognito window with the utm_medium=email param: Campaigns A-C are not displayed, while Campaign D, as expected.
  • After navigating to another page, without the utm_medium param: Campaigns A-C are still not displayed, while Campaign D is, as expected.
  • Loading a page in a new incognito window without the utm_medium=email param: All campaigns are displayed, as expected.
  • After dismissing Campaign A, B, or C (contains a newsletter prompt): Only the dismissed campaign is suppressed on subsequent page loads, as expected.

Test 4

Test 4 conditions

  • Campaigns A-C: Same as in Tests 1-3
  • Campaign D: Inline campaign, displays once a day, no newsletter prompt

Test 4 results

  • In a new incognito window, all campaigns are displayed, as expected.
  • After navigating to a new page: Campaigns A-C are displayed, Campaign D is not, as expected.
  • NOT WORKING: If the first page visited is an AMP page and the second page is not (or vice versa), Campaign D is still displayed on the second page (but is not on subsequent pages—seems to be that the suppression is just not remembered across AMP vs. non-AMP sessions).

Test 5

Test 5 conditions

  • Campaign A: Overlay campaign, sitewide default, contains newsletter prompt, TEST MODE
  • Campaigns B-D: Same as in Tests 1-3

Test 5 results

  • In a new incognito window: Campaign A is not displayed, Campaigns B-D are displayed, as expected.
  • After logging into WP Admin: Campaign A is displayed, Campaigns B-D are not, as expected.

Test 6

Test 6 conditions:

  • Campaigns A-C: Same as in Tests 1-3
  • Campaign D: UTM Suppression field set to "suppress this"

Test 6 results:

  • In a new incognito window, all campaigns are displayed, as expected.
  • When visting the page with the utm_source=suppress+this or utm_source=suppress%20this param, Campaigns A-C are displayed, while Campaign D is not, as expected.
  • NOT WORKING (?): After navigating to a new page, without the utm_source param: Campaign D is displayed again. Expected: the suppression should be remembered across pages once it happens the first time. Marked (?) as I'm not sure whether my expectation is correct?