Gmail Mail Collection Developer Guide - edmozley/freeitsm GitHub Wiki

How Gmail mail collection works โ€” Developer Guide

Gmail is the odd one out among FreeITSM's three mailbox providers, and almost every mistake made against it comes from one fact: Gmail has no folders.

If you are touching includes/gmail.php, read this first. The model is genuinely different from Microsoft 365 and IMAP, and code that assumes otherwise fails quietly rather than loudly.

Related: Verifying a mail folder โ€” Developer Guide ยท Mailbox authentication ยท Architecture

1. The model, in one section

A Gmail message does not live in a place. It carries a set of labels, and it carries as many as you like at once.

What you would expect What Gmail actually does
A message is in one folder A message carries several labels at once
The Inbox is a folder INBOX is just another label
Archiving moves a message Archiving removes the INBOX label and changes nothing else
Moving to a folder Adding one label, usually removing another
Sent items is a folder SENT is a label, and a message can be both SENT and INBOX

The consequence that matters:

"Archived" is not a place. It is the absence of the INBOX label.

So a message that a Gmail filter sent straight to a Support label, skipping the Inbox, carries Support and does not carry INBOX. It is not hidden, not deleted, not in some other folder. It simply never had the one label that most code assumes everything has.

There is also a second, smaller trap: the labels a user creates have ids equal to their names, but Gmail's own labels have fixed ids (INBOX, SENT, CATEGORY_PROMOTIONS) that are not localised, while their display names are. Always match and store by id, never by display name.

2. The bug this page exists because of

gmailGetEmails() used to build its query like this:

$q = 'is:unread';
$labelId = 'INBOX';                                    // โ† always
$folder = strtoupper($mailbox['email_folder'] ?? 'INBOX');
if ($folder !== 'INBOX') {
    $q .= ' label:' . strtolower($mailbox['email_folder']);   // โ† an EXTRA term
}

$listUrl = 'https://gmail.googleapis.com/gmail/v1/users/me/messages?'
    . http_build_query(['q' => $q, 'maxResults' => $maxResults, 'labelIds' => $labelId]);

Read that carefully. labelIds=INBOX is unconditional. The configured folder is added on top of it, never instead of it. The effective question being asked was:

unread AND in the Inbox AND labelled Support

when the question the user thought they were asking was:

unread AND labelled Support

For a mailbox configured with a label, that meant FreeITSM collected only mail that had been left in the Inbox and labelled. Anyone using a Gmail filter to route support mail to a label โ€” which is the entire reason to have a label โ€” got nothing, forever, with no error, no warning, and a mailbox that reported itself as Connected and checked on schedule.

Why it survived so long: the failure mode is an empty result set, and an empty result set is exactly what a quiet mailbox looks like. There is nothing to alert on. "No new mail" and "this configuration can never collect mail" were the same observation.

That is the same shape as the Operational status bug and the Watchtower name lookups: a lookup that matches nothing returns a confident zero, never an error.

3. How it works now

The configured folder resolves to a label id, and that id becomes the scope:

function gmailResolveListLabelId(string $accessToken, array $mailbox): string {
    $folder = trim((string) ($mailbox['email_folder'] ?? '')) ?: 'INBOX';
    if (strcasecmp($folder, 'INBOX') === 0) {
        return 'INBOX';
    }

    $label = gmailFindLabel(gmailListLabels($accessToken), $folder);
    if ($label === null) {
        // Loudly, rather than quietly returning nothing forever.
        throw new Exception('Gmail label "' . $folder . '" was not found in this account.');
    }
    return (string) $label['id'];
}

and the reader is then simply:

$labelId = gmailResolveListLabelId($accessToken, $mailbox);

$listUrl = 'https://gmail.googleapis.com/gmail/v1/users/me/messages?'
    . http_build_query(['q' => 'is:unread', 'maxResults' => $maxResults, 'labelIds' => $labelId]);

Three things fell out of this that were not the point but are worth knowing:

  • Labels with spaces work. The old code built label:customer support, which Gmail's search syntax reads as two terms. Scoping by id sidesteps search syntax entirely.
  • A missing label is now an error. It used to produce zero results, indistinguishable from a quiet mailbox.
  • Matching is case-insensitive, so sent, Sent and SENT all resolve.

Resolution behaviour, probed against a live account:

email_folder Resolves to
NULL INBOX
'' INBOX
INBOX / inbox INBOX
SENT / Sent SENT
NoSuchLabel throws, naming the label

4. โš ๏ธ resultSizeEstimate is not a count

This one cost a round of rework, and it will catch you too.

The obvious way to count messages is messages.list and read resultSizeEstimate. Google documents it as an estimate. It is worse than that โ€” it is not usable per-label at all. Measured against a live account:

Label labels.get (true) resultSizeEstimate
INBOX 2 201
SENT 27 201
TRASH 1 201
DRAFT 0 0
SPAM 0 0

It returned the same number for three labels whose real sizes were 2, 27 and 1. Note that the zero cases agreed, which is exactly what makes this dangerous: a quick check against an empty label confirms the approach and tells you nothing.

Use labels.get, which returns exact figures for the label:

$detail = $get('https://gmail.googleapis.com/gmail/v1/users/me/labels/' . rawurlencode((string) $match['id']));
$total  = (int) ($detail['body']['messagesTotal'] ?? 0);
$unread = (int) ($detail['body']['messagesUnread'] ?? 0);

This is only correct because the label is now the collection scope. Under the old INBOX-AND-label behaviour, labels.get would have over-reported, which is why the first version of Verify deliberately avoided it. Fixing the reader is what made the simple count correct โ€” worth remembering when a workaround starts looking like the design.

5. The upgrade consequence, and why Verify warns

Collection takes unread mail from the label, wherever it sits. On an account where a label has been accumulating archived unread mail for months, that backlog is now eligible to become tickets, max_emails_per_check at a time.

That is the correct behaviour โ€” it is the mail the user asked to collect โ€” but it should not be a surprise. So gmailVerifyFolder() says so before it happens:

$note = null;
$perCheck = (int) ($mailbox['max_emails_per_check'] ?? 10);
if (!$isInbox && $perCheck > 0 && $unread > $perCheck) {
    $note = $unread . ' unread message(s) carry this label, including any that have been '
          . 'archived out of the Inbox. All of them are eligible to become tickets, '
          . $perCheck . ' per check. Mark the older ones as read first if you only want new mail.';
}

rendered in amber rather than green, because a green "found" reads as "nothing to think about".

Note the inversion: the previous version of this note warned that archived mail would be missed. Fixing the reader made that warning false, so it had to change in the same commit. A caveat that documents a bug has to be re-examined the moment the bug is fixed, or it becomes a confident statement of something untrue.

6. ๐Ÿ“ The files involved

File Role
includes/gmail.php gmailResolveListLabelId() (folder โ†’ scope), gmailListLabels(), gmailFindLabel(), gmailApiGet(); gmailGetEmails() reads through the resolved scope; gmailVerifyFolder() counts through labels.get
api/tickets/check_mailbox_email.php Calls gmailGetEmails(); a thrown missing-label now surfaces as a failed check rather than an empty one
api/tickets/verify_mailbox_folder.php The Google branch, above the OAuth-token gate

7. If you work on this next

  • Reading and verifying must share the resolver. gmailVerifyFolder() calls gmailListLabels() / gmailFindLabel(), the same functions the reader uses. A verifier cleverer than the reader is issue #77 again.
  • is:unread is the collection trigger, not a date. There is no "only mail since X" concept here, which is why the backlog warning matters.
  • Post-import actions are label operations. gmailMarkAsRead() removes UNREAD; gmailTrashMessage() moves to TRASH. Neither "moves" anything in the folder sense.
  • Testing without creating tickets: resolve the scope in isolation rather than running a collection. A short read-only script that loads the mailbox row, decrypts it, and calls gmailResolveListLabelId() against each candidate folder value proves the branch that matters without importing anything.

Related pages