FILE PROCESSING PLUGIN - nself-org/nchat GitHub Wiki

ɳChat File Processing Plugin

Plugin Name: file-processing Version: 1.0.0 Category: Infrastructure Status: Production Ready Priority: HIGH


Overview

The File Processing Plugin provides comprehensive file transformation and optimization capabilities for ɳChat. It handles image resizing, video thumbnail generation, document previews, EXIF metadata stripping, and optional virus scanning.


Features

Core Features

  • Image Processing - Resize, optimize, and convert images
  • Video Thumbnails - Generate preview thumbnails from videos
  • Document Previews - Create previews for PDF and Office documents
  • Metadata Stripping - Remove EXIF data for privacy
  • Format Conversion - Convert between image formats (PNG, JPEG, WebP, AVIF)
  • Batch Processing - Process multiple files concurrently
  • Virus Scanning - Optional ClamAV integration
  • S3 Integration - Store processed files in MinIO/S3

Advanced Features

  • Smart Compression - Automatic quality optimization
  • Responsive Images - Generate multiple sizes
  • Watermarking - Add branding to images
  • Text Extraction - Extract text from PDFs/documents
  • Image Analysis - Detect dimensions, format, quality

Installation

Prerequisites

  • Docker running
  • nself CLI v0.9.8+
  • MinIO service (provided by nself stack)
  • 512MB+ RAM available

Install Plugin

cd /Users/admin/Sites/nself-nchat/backend
nself plugin install file-processing

Configuration

Add to backend/.env.plugins:

# File Processing Plugin
FILE_PROCESSING_ENABLED=true
FILE_PROCESSING_PORT=3104
FILE_PROCESSING_ROUTE=files.${BASE_DOMAIN:-localhost}
FILE_PROCESSING_MEMORY=512M

# S3 Configuration
FILE_PROCESSING_S3_ENDPOINT=http://minio:9000
FILE_PROCESSING_S3_BUCKET=${S3_BUCKET:-nchat-files}
FILE_PROCESSING_S3_ACCESS_KEY=minioadmin
FILE_PROCESSING_S3_SECRET_KEY=minioadmin
FILE_PROCESSING_S3_REGION=us-east-1

# Image Processing
FILE_PROCESSING_IMAGE_MAX_WIDTH=2048
FILE_PROCESSING_IMAGE_MAX_HEIGHT=2048
FILE_PROCESSING_IMAGE_QUALITY=85
FILE_PROCESSING_THUMBNAIL_SIZE=200
FILE_PROCESSING_THUMBNAIL_QUALITY=80

# Video Processing
FILE_PROCESSING_VIDEO_THUMBNAIL_ENABLED=true
FILE_PROCESSING_VIDEO_THUMBNAIL_TIME=1
FILE_PROCESSING_VIDEO_THUMBNAIL_WIDTH=1280

# Document Processing
FILE_PROCESSING_DOCUMENT_PREVIEW_ENABLED=true
FILE_PROCESSING_DOCUMENT_PREVIEW_DPI=150
FILE_PROCESSING_PDF_MAX_PAGES=10

# Security
FILE_PROCESSING_STRIP_METADATA=true
FILE_PROCESSING_VIRUS_SCAN_ENABLED=false
FILE_PROCESSING_MAX_FILE_SIZE=104857600  # 100MB

# Performance
FILE_PROCESSING_CONCURRENCY=5
FILE_PROCESSING_QUEUE_SIZE=100
FILE_PROCESSING_TIMEOUT=30000

Start Service

nself restart

API Endpoints

Health Check

GET /health

Response:

{
  "status": "healthy",
  "service": "file-processing",
  "version": "1.0.0",
  "uptime": 86400,
  "dependencies": {
    "s3": {
      "status": "connected",
      "bucket": "nchat-files"
    }
  },
  "queue": {
    "waiting": 0,
    "processing": 2,
    "completed": 1542
  }
}

Image Processing

Resize Image

POST /image/resize
Content-Type: multipart/form-data

file: <image-file>
width: 800
height: 600
fit: cover  # Options: cover, contain, fill, inside, outside

Response:

{
  "success": true,
  "url": "https://storage.nchat.local/processed/image-123-800x600.jpg",
  "width": 800,
  "height": 600,
  "format": "jpeg",
  "size": 45678
}

Optimize Image

POST /image/optimize
Content-Type: multipart/form-data

file: <image-file>
quality: 85  # 1-100
progressive: true
stripMetadata: true

Generate Thumbnail

POST /image/thumbnail
Content-Type: multipart/form-data

file: <image-file>
size: 200  # px
crop: true

Convert Format

POST /image/convert
Content-Type: multipart/form-data

file: <image-file>
format: webp  # Options: jpeg, png, webp, avif, tiff
quality: 85

Strip Metadata

POST /image/strip-metadata
Content-Type: multipart/form-data

file: <image-file>

Get Image Info

POST /image/info
Content-Type: multipart/form-data

file: <image-file>

Response:

{
  "width": 1920,
  "height": 1080,
  "format": "jpeg",
  "size": 234567,
  "hasAlpha": false,
  "exif": {
    "Make": "Canon",
    "Model": "EOS 5D Mark IV",
    "DateTime": "2026:02:03 12:00:00"
  }
}

Video Processing

Generate Video Thumbnail

POST /video/thumbnail
Content-Type: multipart/form-data

file: <video-file>
time: 1  # seconds into video
width: 1280
format: jpeg

Response:

{
  "success": true,
  "url": "https://storage.nchat.local/thumbnails/video-123.jpg",
  "width": 1280,
  "height": 720,
  "timestamp": 1
}

Extract Video Metadata

POST /video/metadata
Content-Type: multipart/form-data

file: <video-file>

Response:

{
  "duration": 120.5,
  "width": 1920,
  "height": 1080,
  "codec": "h264",
  "fps": 30,
  "bitrate": 5000000,
  "size": 75600000
}

Document Processing

Generate PDF Preview

POST /document/preview
Content-Type: multipart/form-data

file: <pdf-file>
page: 1  # optional, defaults to first page
dpi: 150
format: png

Extract Document Text

POST /document/extract-text
Content-Type: multipart/form-data

file: <document-file>
pages: 1-5  # optional, all pages by default

Response:

{
  "text": "Extracted text content...",
  "pages": 5,
  "wordCount": 1234
}

Virus Scanning

POST /scan
Content-Type: multipart/form-data

file: <any-file>

Response:

{
  "clean": true,
  "scanner": "clamav",
  "signature": "2026-02-03",
  "scanTime": 123
}

Batch Processing

POST /batch/optimize
Content-Type: multipart/form-data

files: <multiple-image-files>
quality: 85
format: webp

Response:

{
  "processed": 10,
  "failed": 0,
  "results": [
    {
      "filename": "image1.jpg",
      "url": "https://storage.nchat.local/processed/image1.webp",
      "originalSize": 1234567,
      "processedSize": 456789,
      "savings": "63%"
    }
  ]
}

Frontend Integration

Environment Variables

# .env.local
NEXT_PUBLIC_FILE_PROCESSING_URL=http://files.localhost:3104
NEXT_PUBLIC_FILE_PROCESSING_ENABLED=true
NEXT_PUBLIC_MAX_FILE_SIZE=104857600  # 100MB

React Hook

import { useFileProcessing } from '@/hooks/use-file-processing'

function FileUpload() {
  const { uploadAndProcess, isProcessing, progress } = useFileProcessing()

  const handleUpload = async (file: File) => {
    const result = await uploadAndProcess(file, {
      resize: true,
      maxWidth: 2048,
      maxHeight: 2048,
      optimize: true,
      quality: 85,
      thumbnail: true,
      stripMetadata: true,
    })

    console.log('Processed file:', result.url)
  }

  return (
    <div>
      <input type="file" onChange={(e) => handleUpload(e.target.files[0])} />
      {isProcessing && <ProgressBar value={progress} />}
    </div>
  )
}

Service Layer

import { FileUploadService } from '@/services/files'

const fileService = new FileUploadService()

// Upload and process image
const result = await fileService.uploadAndProcess(file, {
  resize: true,
  optimize: true,
  thumbnail: true,
  stripMetadata: true,
})

// Generate thumbnail
const thumbnail = await fileService.generateThumbnail(file, 200)

// Convert format
const webp = await fileService.convertFormat(file, 'webp', 85)

Supported File Types

Images

  • JPEG/JPG
  • PNG
  • WebP
  • AVIF
  • TIFF
  • GIF
  • BMP
  • SVG

Videos

  • MP4
  • WebM
  • MOV
  • AVI
  • MKV

Documents

  • PDF
  • DOCX
  • XLSX
  • PPTX
  • TXT
  • Markdown

Performance

Metrics

  • Processing Speed: 10-50ms per image (typical)
  • Concurrent Jobs: Up to 5 simultaneous
  • Throughput: 100+ images/minute
  • Memory Usage: ~512MB baseline
  • CPU Usage: ~1 core per job

Optimization Tips

  1. Batch Processing: Use batch endpoints for multiple files
  2. Quality Settings: Use quality 85 for good compression with minimal loss
  3. Format Selection: WebP offers best compression, AVIF for cutting-edge
  4. Caching: Enable CDN caching for processed files
  5. Lazy Processing: Process on-demand vs. upload time

Testing

Health Check Test

curl http://files.localhost:3104/health

Integration Test

describe('File Processing Plugin', () => {
  it('should process image', async () => {
    const formData = new FormData()
    formData.append('file', imageFile)
    formData.append('width', '800')

    const response = await fetch('http://files.localhost:3104/image/resize', {
      method: 'POST',
      body: formData,
    })

    expect(response.ok).toBe(true)
    const data = await response.json()
    expect(data).toHaveProperty('url')
  })
})

Troubleshooting

High Memory Usage

Problem: Service using too much RAM

Solutions:

  1. Reduce FILE_PROCESSING_CONCURRENCY
  2. Lower FILE_PROCESSING_IMAGE_MAX_WIDTH/HEIGHT
  3. Increase service memory limit
  4. Enable image streaming for large files

Slow Processing

Problem: Files taking too long to process

Solutions:

  1. Check S3 latency
  2. Reduce quality settings
  3. Increase concurrency
  4. Use batch processing

Format Errors

Problem: Unsupported file format errors

Solutions:

  1. Check file extension matches content type
  2. Verify file isn't corrupted
  3. Update supported formats list
  4. Use format detection endpoint first

Security

Best Practices

  1. Virus Scanning: Enable for user uploads
  2. Metadata Stripping: Always strip EXIF for privacy
  3. File Size Limits: Enforce max upload size
  4. Format Validation: Verify file types
  5. S3 Permissions: Use bucket policies

Malware Protection

# Enable ClamAV
FILE_PROCESSING_VIRUS_SCAN_ENABLED=true
FILE_PROCESSING_CLAMAV_HOST=clamav
FILE_PROCESSING_CLAMAV_PORT=3310

Monitoring

Metrics Endpoint

curl http://files.localhost:3104/metrics

Key Metrics:

  • file_processing_jobs_total - Total jobs processed
  • file_processing_duration_ms - Processing duration histogram
  • file_processing_errors_total - Error count
  • file_processing_queue_size - Current queue size

Logs

nself logs file-processing --follow

Changelog

Version 1.0.0 (2026-02-03)

  • Initial release
  • Image processing (resize, optimize, convert)
  • Video thumbnail generation
  • Document preview generation
  • EXIF metadata stripping
  • S3 storage integration
  • Batch processing support
  • ClamAV virus scanning

Support


Related Documentation

⚠️ **GitHub.com Fallback** ⚠️