Fetchers Module Relationship Diagram

Overview

This document describes the call relationships between components in the fetchers module of the fetch-mcp project, providing a better understanding of data flow and code organization.

Module Relationship Diagram

graph TD
    subgraph "API Interface Layer"
        A[fetch.ts API Interface] --> B[Fetcher Unified Class]
    end

    subgraph "Core Abstraction Layer"
        B --> C[FetcherFactory Factory Class]
        C -->|Creates| D[IFetcher Abstract Interface]
    end

    subgraph "Base Implementation"
        E[BaseFetcher Basic Fetcher]
        E -->|Provides base method| F[createSuccessResponse]
        E -->|Provides base method| G[createErrorResponse]
        E -->|Provides base method| H[handleContentChunking]
        E -->|Provides base method| I[getChunkContent]
        E -->|Provides base method| J[processWithContentExtraction]
    end

    subgraph "Concrete Implementations"
        K[BrowserFetcher Browser Fetcher] -.->|Inherits| E
        L[NodeFetcher Node Fetcher] -.->|Inherits| E
        K -->|Implements| D
        L -->|Implements| D
        C -->|Creates| K
        C -->|Creates| L
    end

    subgraph "BrowserFetcher Components"
        K -->|Uses| M[BrowserInstance]
        K -->|Uses| N[PageOperations]
        K -->|Uses| O[CookieManager]
    end

    subgraph "NodeFetcher Components"
        L -->|Uses| P[HttpClient]
    end

    subgraph "Common Utilities"
        Q[ContentProcessor]
        R[ContentExtractor]
        S[ChunkManager]
        T[ContentSizeManager]
        U[ErrorHandler]
        V[Logger]
    end

    K -->|Calls| Q
    K -->|Calls| R
    K -->|Calls| S
    K -->|Calls| T

    L -->|Calls| Q
    L -->|Calls| R
    L -->|Calls| S
    L -->|Calls| T
    L -->|Calls| U

    E -->|Calls| S
    E -->|Calls| T
    E -->|Calls| R

    B -->|Calls| V
    C -->|Calls| V
    K -->|Calls| V
    L -->|Calls| V
    E -->|Calls| V

Data Flow Description

1. External API Call Flow:

   • External applications call API functions (such as fetchHtml, fetchJson, fetchTxt, fetchMarkdown) in fetch.ts
   • API functions call the corresponding static methods of the Fetcher class
   • The Fetcher class uses FetcherFactory to create appropriate fetcher instances
   • The fetcher instance performs specific fetching logic and returns results

2. Factory Pattern Implementation:

   • FetcherFactory decides which type of fetcher to create based on request parameters (such as useBrowser, useNodeFetch, autoDetectMode)
   • By default, NodeFetcher is preferred because it is more lightweight

3. Fetcher Internal Processing Flow:

   • All fetchers inherit from BaseFetcher, gaining common functionality such as result creation, content chunking, content extraction, etc.
   • BrowserFetcher uses Puppeteer to implement webpage fetching in browser mode
   • NodeFetcher uses node-fetch to implement webpage fetching in standard mode
   • Both fetchers implement methods defined by the IFetcher interface

4. Content Processing Flow:

   • Provides content in different formats (HTML, JSON, plain text, Markdown) based on user requirements
   • Supports content extraction, intelligently extracting the main content of web pages
   • Supports content chunking, handling segmented retrieval of large content

Main Component Responsibilities

API Layer

• fetch.ts: Provides concise external API interfaces
• Fetcher: Unified fetcher class, providing static methods to fetch content in different formats

Abstraction Layer

• FetcherFactory: Factory class, creating appropriate fetcher instances
• IFetcher: Fetcher interface, defining methods that all fetchers must implement

Implementation Layer

• BaseFetcher: Base fetcher class, providing common functionality
• BrowserFetcher: Browser mode fetcher, implemented using Puppeteer
• NodeFetcher: Node mode fetcher, implemented using node-fetch

Utility Components

• BrowserInstance: Manages browser instances
• PageOperations: Browser page operations
• CookieManager: Cookie management
• HttpClient: HTTP client
• ContentProcessor: Content processor
• ContentExtractor: Content extractor
• ChunkManager: Chunk manager
• ContentSizeManager: Content size manager
• ErrorHandler: Error handler
• Logger: Logger

Key Design Patterns

1. Factory Pattern: Using FetcherFactory to create fetcher instances
2. Strategy Pattern: Different fetchers implement the same interface, providing different implementation strategies
3. Template Method Pattern: BaseFetcher defines the basic algorithm framework, subclasses implement specific steps
4. Singleton Pattern: BrowserInstance manages a single browser instance

Summary

The fetchers module adopts a clear layered design and multiple design patterns, providing flexible and extensible web content fetching functionality. Through factory and strategy patterns, the system can choose appropriate fetching methods based on different requirements, while reducing redundant code by providing common functionality through base classes. Each component in the module has clear responsibilities, following the single responsibility principle, making it easy to maintain and extend.