ART Summary Writer - Trusted-AI/adversarial-robustness-toolbox Wiki

[wiki work in progress]

Default Summary Writer

ART 1.9 introduces a Summary Writer API which collects metrics about losses, gradients, adversarial patches and textures, etc. at strategic locations in the supported attacks. ART 1.9 provides a default implementation art.summary_writer.SummaryWriterDefault and allows users to implement custom summary writers by implementing base class art.summary_writer.SummaryWriter.

Attacks supporting Summary Writer

The attack’s Summary Writer is activated with argument summary_writer:

Default Metrics (so far)

1. Total Loss and available loss components per sample

2. Norms of gradients per mini-batch

3. Adversarial Patch and Texture

4. Indicators of Attack Failure per sample (Pintor et al., 2021)

Tracks if adversarial example switches back to benign.

Quantifies attack convergence.

Quantifies attack convergence rate.

Checks quality of loss gradients.

TensorBoard Analysis of Scalar Values

TensorBoard Analysis of Adversarial Patches

Custom Summary Writer

Users can create custom summary writers to collect custom metrics by implementing art.summary_writer.SummaryWriter and adding custom metrics in method update:

def update(self, batch_id, global_step, grad=None, patch=None, estimator=None, x=None, y=None, **kwargs):

providing access to current batch ID and global step, gradients, patches, estimator, input and label data.