Attachment Size Limits and Large Emails Developer Guide - edmozley/freeitsm GitHub Wiki

Attachment size limits and large emails - Developer Guide

How FreeITSM decides how big an attachment may be, and what happens to an inbound email that is too large (issue #170). This page is for someone changing the code. The settings themselves are explained in the in-app help: System โ†’ Security โ†’ Attachments and the mailbox admin guide's Large emails section.

Related: Mailbox Authentication ยท Basic IMAP mailboxes ยท Email send log ยท Outbound email attachments - Developer Guide

1. The short version

  • There are two limits: the largest single attachment (attachment_max_mb, default 10) and the largest total of one inbound email's attachments (attachment_max_total_mb, default 0 = no limit). Both live in system_settings and are set on System โ†’ Security.
  • The per-attachment limit replaced a hard-coded 10 MB. Every upload path that does not pass a limit of its own now gets the configured one.
  • An inbound email over either limit is either kept with the oversized attachments stripped (a note on the ticket lists them) or rejected (no ticket). When the attachments are each fine but too big together, a setting decides which go: largest first, all, or in order until the next would not fit.
  • Optionally an alert goes to chosen addresses and a reply goes to the sender. The reply cannot loop.
  • All of that has install-wide defaults (Tickets โ†’ Settings โ†’ Mailboxes) and a per-mailbox override, limits included, stored as JSON on the mailbox.
  • Everything is recorded in the mailbox activity log, and the alert and reply in the email send log.

The whole decision lives in one file, includes/attachment_limits.php. The mailbox checker only measures, asks, and acts on the answer.

2. The files involved

๐Ÿ—„๏ธ schema ยท ๐Ÿ“– read ยท โœ๏ธ write ยท ๐Ÿ”— cross-module ยท ๐Ÿ–ฅ๏ธ UI / help ยท ๐Ÿ“„ docs

๐ŸŽจ File What it does here
๐Ÿ“– includes/attachment_limits.php The rules. Reads the settings, merges a mailbox's override, decides what to strip or reject (oversizePlan()), sends the alert and reply (oversizeNotify()), and the loop guard. Also phpUploadLimits() for the settings screen.
โœ๏ธ api/tickets/check_mailbox_email.php The mailbox checker. Measures each email's attachments before importing it, then rejects, strips or carries on. oversizeMeasure(), oversizeLogStep() and oversizeSummary() live at the bottom.
๐Ÿ”— includes/uploads.php uploadStoreFile() and uploadStoreBytes() - a null limit now means "the configured one" (attachmentMaxBytes()), not the old constant.
๐Ÿ”— includes/mailbox_send.php mailboxSendSimple() - sends one plain email from a ticket mailbox through Microsoft Graph, Gmail or SMTP, and writes the send log. Used for the alert and the reply.
๐Ÿ”— includes/email_log.php Two send-log routes: oversize_alert and oversize_reply.
โœ๏ธ api/tickets/oversize_settings.php GET / POST the install-wide defaults (Cap::TICKETS_MAILBOXES).
โœ๏ธ api/tickets/save_mailbox.php Saves a mailbox's own rules into target_mailboxes.oversize_policy.
๐Ÿ“– api/tickets/get_mailboxes.php Returns oversize_policy with each mailbox.
โœ๏ธ api/settings/save_system_settings.php + includes/settings_keys.php Saves the two limits (registered keys, System โ†’ Security).
๐Ÿ–ฅ๏ธ system/security/index.php The Attachments card: the two limits, PHP's own limits, the warning, the link to Mailboxes.
๐Ÿ–ฅ๏ธ tickets/settings/index.php When an email is too large under the mailbox list, and Large emails in each mailbox's dialog. Both are drawn by one PHP function, renderOversizeFields($prefix, $withLimits), and read by one JS pair, setOversizeFields() / oversizeFields().
๐Ÿ–ฅ๏ธ system/help/security.php, tickets/help-mailbox-auth.php The two help sections.
๐Ÿ—„๏ธ includes/db_verify_schema.php, database/freeitsm.sql target_mailboxes.oversize_policy TEXT NULL.
๐Ÿ”— includes/feature_bingo/cards/tickets-channels.php The card for "somebody hears about an oversized email".

What you do not touch: the ingest notice (attachmentIngestNotice() in includes/uploads.php) is reused unchanged - a stripped attachment is reported exactly like any other attachment FreeITSM would not keep.

3. Where each setting lives, and why it is split

The owner asked for this split, with each screen pointing to the other:

Setting Key Screen Why there
Largest attachment attachment_max_mb System โ†’ Security It applies to every upload path, not just email.
Largest email attachment_max_total_mb System โ†’ Security Sits with its sibling. Only inbound email uses it.
Keep or reject oversize_mode (strip / reject) Tickets โ†’ Settings โ†’ Mailboxes Only email can be "too big" after the fact. A browser upload is simply refused on the spot.
What goes when over the total oversize_strip_order (largest / all / order) Mailboxes Admins disagree on the best answer, so it is a choice.
Alert oversize_alert (0/1), oversize_alert_to Mailboxes
Reply to the sender oversize_autoreply (0/1), oversize_autoreply_text Mailboxes

A mailbox's own rules are one JSON value in target_mailboxes.oversize_policy, in MB:

{"max_mb":50,"max_total_mb":200,"mode":"strip","strip_order":"largest",
 "alert":true,"alert_to":"[email protected]","autoreply":false,"autoreply_text":"..."}

NULL means "use the defaults". When it is set, all of it applies, the limits included - so a supplier mailbox can accept 50 MB drawings while everything else stays at 10.

4. Reading the rules - oversizeRules()

One function turns settings into the rules for one mailbox. Limits are in bytes from here on.

$rules = [
    'max_bytes'       => attachmentMaxBytes($conn),
    'max_total_bytes' => attachmentMb(attachmentSetting($conn, 'attachment_max_total_mb', '0'), 0, true) * 1024 * 1024,
    'mode'            => attachmentSetting($conn, 'oversize_mode', 'strip') === 'reject' ? 'reject' : 'strip',
    ...
    'source'          => 'defaults',
];
$own = $mailbox && !empty($mailbox['oversize_policy']) ? json_decode((string)$mailbox['oversize_policy'], true) : null;
if (is_array($own)) {
    $rules = array_merge($rules, oversizeNormalisePolicy($own, $rules));
    $rules['source'] = 'mailbox';
}

oversizeNormalisePolicy() is used both when a mailbox is saved and when it is read, so a hand-edited row ("max_mb": -5, "mode": "explode") can never break a mailbox check - it falls back to something sane. attachmentMb() clamps every number (1 to 100 GB; 0 allowed only where it means "no limit").

source is carried through so the alert can say whether the install's or the mailbox's limits applied.

5. One inbound email, step by step

The checker loops over the emails a mailbox returned. The size check sits after the sender allow-list check and before the email is imported:

fetch emails
for each email:
    allow-list check  โ”€โ”€ fail โ”€โ–ถ rejected (as before)
    oversizeMeasure()           fetch + measure attachments, once
    oversizePlan()              what breaks a limit?
    plan says reject  โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ถ handle like rejected mail, notify, log 'rejected', next
    saveEmailToDatabase(..., $prefetched, $strip, $maxBytes)
    if anything was stripped:   notify, put the reason on the log row
    log 'imported'

5.1 Why the attachments are fetched before the import

Before #170, Microsoft 365 attachments were fetched after the ticket and the email row had been written, inside saveEmailToDatabase(). "Reject the whole email" has to be decided before a ticket exists, so the fetch moved forward:

[$prefetched, $sizeAtts] = oversizeMeasure($conn, $email, $accessToken);
if ($sizeAtts) $sizePlan = oversizePlan($sizeAtts, $oversizeRules);

The same fetched list is handed on to saveEmailToDatabase() as $prefetched, so nothing is downloaded twice. For IMAP and Gmail the attachments already arrive with the message (attachments_inline), so nothing extra is fetched at all.

oversizeMeasure() checks the de-dup table first. An email already imported is skipped by saveEmailToDatabase() anyway, and measuring it would send the alert and the reply about the same email twice.

5.2 Measuring

Sizes are the decoded bytes when the content is in hand, otherwise the size the provider reports:

$b64  = (string)($a['contentBytes'] ?? '');
$size = $b64 !== '' ? (int)floor(strlen($b64) * 3 / 4) - substr_count(substr($b64, -2), '=')
                    : (int)($a['size'] ?? 0);

Graph's own size includes MIME overhead, so it reads a little high - fine as a fallback, not as the measure.

Only file attachments are counted (#microsoft.graph.fileAttachment, which the IMAP and Gmail connectors also produce). Inline images count, because they are files on disk like any other.

If measuring throws, the email is imported exactly as before and the failure goes into the check's error list. A size check must never cost somebody their ticket.

5.3 Deciding - oversizePlan()

Pure: a list of ['name', 'size'] in message order plus the rules in, a plan out. No database, no I/O, so it is easy to test.

  1. Any attachment over max_bytes is always removed.
  2. If what is left is still over max_total_bytes (and that is not 0), strip_order decides what else goes:
strip_order What happens
largest Sort what is left by size, remove from the top until the rest fit. Keeps the most files.
all Remove everything that is left.
order Walk in message order, keeping each one while the running total fits; anything that would overflow goes. 
return ['over' => $over, 'reject' => $over && $rules['mode'] === 'reject',
        'strip' => $rules['mode'] === 'reject' ? [] : $strip,
        'total' => $total, 'reasons' => $strip];

strip is what to leave out, keyed by file-attachment index. reasons keeps the same list even in reject mode, so the alert can say which attachments broke the rules.

5.4 Stripping - inside saveEmailToDatabase()

The attachment loop counts file attachments in the same order oversizeMeasure() did, and skips the ones in $strip:

$fileIndex++;
if (isset($strip[$fileIndex])) {
    $rejectedAttachments[] = ['name' => $attachment['name'] ?? 'attachment', 'reason' => $strip[$fileIndex]];
    continue;
}
$savedAttachment = saveAttachment($conn, $dbEmailId, $attachment, $maxBytes);

A stripped file goes into the same $rejectedAttachments list as a file whose type is not accepted, so the existing note at the foot of the message lists it, with the reason, where the attachment would have been. That note is the "stub".

โš ๏ธ The index alignment matters. Both loops skip anything that is not a file attachment in exactly the same way. If you add a filter to one, add it to the other, or the wrong attachment is stripped.

saveAttachment() passes $maxBytes (the mailbox's limit when it has its own) down to uploadStoreBytes(). Without that, a mailbox allowed 50 MB would still have its files refused at the global 10 MB on the way to disk.

5.5 Rejecting

A rejected email goes through handleEmailAfterProcessing() with the mailbox's Rejected emails action (delete, move to Deleted Items, or mark as read) - the same path as a sender who is not on the allow-list. No ticket, no email row, no attachments written.

6. Telling people - oversizeNotify()

Called once per oversized email with 'rejected' or 'stripped'. It returns processing-log steps, so the mailbox log shows whether each alert and reply was sent, skipped or failed.

  • Alert: one email to each address in alert_to, listing every attachment with its size and what happened to it, the limits and where they came from, and the ticket number when one was made.
  • Reply: autoreply_text with {subject}, {limit} and {total_limit} filled in, HTML-escaped and turned into paragraphs.

Both go through mailboxSendSimple(), so they leave from the mailbox the email arrived at, by the same route that mailbox uses for everything else, and land in the Email send log under their own routes.

6.1 Why the reply cannot loop

Two systems that each answer an oversized email with an email can bounce messages back and forth forever. oversizeReplySkipReason() refuses to reply:

  • to an address that is not a valid email address;
  • to the mailbox itself;
  • to an automated local part - no-reply, noreply, do-not-reply, mailer-daemon, postmaster, bounce, notifications, auto-reply and their variants;
  • to anyone it has already replied to from the same mailbox in the last hour.

The last check reads the mailbox activity log, which already records every reply in its processing log - no new table:

SELECT COUNT(*) FROM mailbox_activity_log
 WHERE mailbox_id = ? AND from_address = ? AND created_datetime > UTC_TIMESTAMP() - INTERVAL 1 HOUR
   AND processing_log LIKE '%"step":"oversize_reply"%"result":"sent"%'

A skipped reply is logged as skipped with the reason, so "why didn't they get a reply?" has an answer.

7. What gets recorded

Where What
Mailbox activity log, reason Too large: big.zip (40 MB), video.mp4 (30 MB) for a rejection, Attachments removed (too large): ... for a strip. Visible on the row without opening it.
Mailbox activity log, processing_log A size_check step (the limits, where they came from, the mode and order, every attachment and its fate), the post-action, and one step per alert and reply.
The ticket The note at the foot of the message, for a strip.
Email send log Each alert (oversize_alert) and reply (oversize_reply), sent or failed.

8. Uploads outside email

uploadStoreFile() and uploadStoreBytes() used to default to the constant UPLOAD_MAX_BYTES (10 MB). They now take ?int $maxBytes = null, and null resolves to attachmentMaxBytes():

function uploadStoreFile(array $file, string $destDir, array $allowed = UPLOAD_TYPES_ATTACHMENT, ?int $maxBytes = null): array
{
    $maxBytes = $maxBytes ?? attachmentMaxBytes();

So every caller that never passed a limit - the portal, chat channels, analyst replies, attached documents, change attachments - follows the setting with no change to the caller. Callers that pass their own limit (logos at 2 MB, RFP documents at 20 MB) are unaffected. UPLOAD_MAX_BYTES stays as a documented fallback only.

The total limit is not applied to browser uploads. A person uploading is told at once that a file is too big; the total exists for email, which arrives as one thing nobody can trim first.

9. PHP and the web server

FreeITSM's limit can only be lower than what PHP accepts. A file uploaded through a browser is refused by PHP before any FreeITSM code runs if it is over upload_max_filesize or post_max_size, and a web server or proxy in front (nginx client_max_body_size, IIS maxAllowedContentLength) can be lower again.

phpUploadLimits() reads all three relevant ini_get() values, turning 2M / 1G into bytes (-1 means unlimited), and the Security card shows them and warns when the configured limit is higher than PHP's.

Email does not pass through those two limits, because it is fetched, not uploaded. But the whole message is held in memory while it is decoded and measured, so memory_limit is shown too.

โš ๏ธ Apache and the command line often have different php.ini files. A WAMP install was found running 2 MB / 8 MB under Apache while the CLI said 10 MB. The card reads the values of the PHP that serves the page, which is the one that matters for browser uploads.

10. Traps

  • Measure before import, and only once. Fetching in the main loop and again in saveEmailToDatabase() would double the Graph traffic for every email with attachments. Pass $prefetched on.
  • De-dup before measuring, or a duplicate in reject mode alerts and replies twice.
  • The two loops count file attachments the same way (5.4). The strip list is keyed by that count.
  • Pass the mailbox's limit down to uploadStoreBytes(), or an override above the global limit is silently undone on the way to disk.
  • The reply must never loop (6.1). Do not add an "always reply" option.
  • oversizeNormalisePolicy() on read as well as save. The JSON is stored text, and a bad value must not break a mailbox check.
  • A failed measurement imports the email as before. Never let the size check lose mail.

11. Testing it

oversizePlan() and oversizeRules() are pure enough to test from the command line against a fixture list. The cases worth keeping:

  • within the limits: nothing happens;
  • one attachment over the per-attachment limit: always removed, whatever the order setting;
  • total over with each of largest, all and order, on a list where they give different answers;
  • total limit 0: only the per-attachment limit applies;
  • reject mode: strip empty, reasons still filled;
  • a mailbox override wins, limits included, and junk addresses and nonsense values are cleaned;
  • the reply guard refuses no-reply, mailer-daemon, postmaster, bounce and the mailbox itself, and allows a person.

The end-to-end path needs a real mailbox. Set a small limit (1 MB), switch on the alert and the reply, send the mailbox an email with a 2 MB attachment, then check the note on the ticket, the mailbox Logs, the send log and the two inboxes. Put the settings back afterwards.