DocETL - w4111/w4111.github.io GitHub Wiki

Names and UNI

  • yx2950 Lorraine Xu
    • What are the alternatives, and what are the pros and cons of the technology compared with alternatives? (what makes it unique?)
      • Alternative 1
      • Alternative 2
    • Example
  • zt2373 Zeyi Tong
    • Explain the problem that it solves.
    • How does the technology solve the problem?
    • How it relates to concpts from 4111.
    • Tutorials

DocETL

The Problem and Solution

  • Explain the problem that it solves.
  • How does the technology solve the problem?
  • What are the alternatives, and what are the pros and cons of the technology compared with alternatives? (what makes it unique?)
  • How it relates to concpts from 4111.
  • NOTE: illustrating the relationship with concepts from this class IS BY FAR THE MOST IMPORTANT ASPECT of this extra credit assignment

Remember, the more detailed and thorough you contrast the technology with the topics in class the better. We've discussed the relational model, constraints, SQL features, transactions, query execution and optimization, and recovery. Pick a subset of topics to really dive deep and compare and contrast the similarities and differences between our discussion in class and the technology.

Tutorial

Note: Installation is less relevant than a tutorial highlighting the main concepts as they relate to 4111.

The problem that DocETL solves.

DocETL addresses the challenge of processing complex, unstructured documents by providing a low-code, declarative system that integrates Large Language Models (LLMs) into data processing pipelines. Traditional methods often struggle with the intricacies of unstructured data, leading to inefficiencies and inaccuracies.

How does DocETL solve the problem?

  1. Declarative Low-Code Interface Traditional methods for processing unstructured data require extensive coding expertise and are hard to scale. DocETL uses a YAML-based declarative interface, allowing users to define processing pipelines without extensive coding. This simplifies the setup of data extraction, transformation, and loading tasks.

  2. Integration with Large Language Models (LLMs) Extracting meaningful information from unstructured data like text documents is difficult with traditional data processing tools. DocETL integrates LLMs to handle nuanced language understanding tasks, such as text summarization, entity recognition, and contextual analysis. This significantly improves the accuracy and efficiency of document processing.

  3. Specialized Operators for Document Processing Standard processing tools struggle with domain-specific requirements like preserving context or resolving ambiguities in complex documents. DocETL includes operators like:

    • resolve: For entity resolution to disambiguate information across documents.
    • gather: To maintain context while splitting or organizing text fragments. These operators allow users to handle complex workflows specific to fields like medicine, law, and social sciences.

  4. Scalability for Domain-Specific Tasks Domain-specific data often requires unique handling rules that are difficult to generalize with traditional tools. DocETL is adaptable across various domains, making it a versatile tool for processing medical records, legal documents, or scientific papers.

  5. Simplified Workflow Management Managing workflows for unstructured data processing can be tedious and error-prone. DocETL pipelines are modular and easy to configure, enabling seamless management of document ingestion, processing, and output generation.

  6. Enhanced Efficiency Processing large volumes of documents manually is time-consuming and prone to errors. By automating key aspects of document processing with its pipeline structure and LLM integration, DocETL reduces manual effort while ensuring consistent, high-quality results.

What are the alternatives?

DocETL is a specialized system for LLM-powered data processing pipelines, focusing on handling complex, unstructured datasets like long documents. Below, we compare DocETL with two prominent alternatives and outline its unique advantages.

Apache Tika

Apache Tika is an open-source library designed for metadata extraction and text parsing from a broad range of document formats, including PDFs, Word documents, images, and more. Its primary focus is content extraction, making it suitable for lightweight, straightforward workflows. Tika is particularly popular for its compatibility with Java and seamless integration into larger pipelines. However, it lacks advanced document processing features like contextual reasoning or multi-step workflow management. Tika is ideal for projects that need reliable extraction tools without extensive transformation capabilities.

AWS Textract

AWS Textract is a cloud-native OCR tool optimized for extracting text, tables, and structured data from scanned or digital documents. It integrates deeply into the AWS ecosystem, enabling users to incorporate extracted data into AWS workflows and applications. Textract is effective for structured document analysis, such as invoices or forms, and supports scalable processing for large document volumes. However, it focuses primarily on structured data extraction and lacks the flexibility needed for unstructured or complex multi-step tasks. This makes Textract suitable for environments where AWS services are already in use and where scalability is key.

What are the pros and cons of DocETL compared with alternatives?

Pros

1. Advanced Document Workflows
DocETL

DocETL is uniquely capable of handling advanced workflows, such as map-reduce operations, and supports intricate tasks requiring contextual consistency across documents. Its built-in operators, like “resolve” for entity resolution and “gather” for context preservation during document splitting, enable complex data manipulations that alternatives cannot match. This makes DocETL particularly effective for industries like law and medicine, where precision and multi-step reasoning are critical.

Apache Tika and AWS Textract

Neither Apache Tika nor AWS Textract supports advanced document workflows or contextual reasoning. Tika’s primary strength lies in lightweight text extraction, making it unsuitable for managing interdependent document splits or reductions. AWS Textract, while powerful for structured document analysis, lacks features for handling unstructured data and multi-step transformations. These limitations leave both alternatives less effective for use cases requiring high contextual accuracy and workflow complexity.

2. Accuracy and Optimization
DocETL

DocETL leverages an LLM-powered optimizer to ensure correctness by experimenting with pipeline rewrites and selecting the most accurate configuration. It can optimize batch sizes, retry tasks that fail validation, and adapt dynamically to workload demands, ensuring consistently high output quality. These features are vital for workflows where errors can have significant consequences, such as medical diagnoses or legal analyses.

Apache Tika and AWS Textract

Apache Tika does not include mechanisms for optimizing workflows or validating output, relying entirely on manual adjustments by developers. Similarly, AWS Textract lacks adaptive optimization and retry mechanisms, focusing instead on straightforward data extraction. Both tools may struggle to ensure high accuracy or recover from processing errors in complex workflows, which limits their effectiveness in high-stakes environments.

3. Low-Code Interface
DocETL

DocETL’s declarative YAML-based interface provides a low-code solution for designing pipelines while maintaining full control over the logic and LLM prompts. This balance of simplicity and flexibility allows users to define complex workflows without needing extensive programming knowledge. It is particularly useful for teams that want to create and modify pipelines quickly while retaining the ability to fine-tune them as needed.

Apache Tika and AWS Textract

Apache Tika requires extensive programming to configure workflows and lacks a low-code interface, making it less accessible for non-developers. AWS Textract, while offering a more user-friendly API, still requires users to write code and lacks DocETL’s fine-grained control over pipeline logic. These factors make both alternatives less efficient for teams that prioritize rapid iteration and minimal coding.

4. Multi-Domain Support
DocETL

DocETL is tailored for a wide range of domains, including law, medicine, and social sciences. Its ability to process complex, unstructured datasets and adapt to various domain-specific requirements makes it a versatile solution. It excels in scenarios where domain-specific context, entity resolution, and inter-document relationships are crucial for accurate outputs.

Apache Tika and AWS Textract

Apache Tika and AWS Textract are more limited in their domain adaptability. Tika focuses on general-purpose extraction without domain-specific enhancements, while Textract is optimized for structured data and may not perform well with unstructured datasets. These constraints make them less suitable for industries that require nuanced handling of specialized data types.

Cons

1. Complexity and Learning Curve
DocETL

DocETL’s advanced features, such as LLM-powered optimization and specialized operators, come with a steep learning curve. Although the YAML interface is low-code, the need to understand complex workflows, entity resolution, and optimization logic can be challenging for new users or those with limited technical expertise. This complexity makes onboarding and initial implementation slower compared to simpler tools.

Apache Tika and AWS Textract

Apache Tika and AWS Textract are comparatively easier to use for their intended purposes. Tika’s straightforward APIs make it a quick-start tool for developers, while AWS Textract’s documentation and cloud-native services simplify integration into AWS ecosystems. Teams prioritizing ease of use or rapid deployment may find these alternatives more accessible than DocETL.

2. Cost Considerations
DocETL

DocETL’s extensive capabilities come at a higher cost compared to its alternatives. The need for computational resources to run LLM-powered pipelines and optimize workflows can significantly increase operational expenses, especially for large-scale projects. For teams or projects with budget constraints, this can make DocETL less feasible despite its advanced capabilities.

Apache Tika and AWS Textract

Apache Tika is free and open-source, making it the most cost-effective option for basic extraction tasks. AWS Textract’s pay-as-you-go model provides predictable and scalable pricing, which can be more economical for straightforward workflows. Both alternatives are more budget-friendly for users whose requirements don’t extend to complex document processing.

3. Dependency on LLMs
DocETL

DocETL’s reliance on LLM-powered optimization introduces performance variability based on the underlying model and its configuration. Factors such as prompt design, LLM model updates, and API latency can affect processing times and accuracy. This dependency also makes DocETL susceptible to issues like high latency or output errors during transient LLM outages or failures.

Apache Tika and AWS Textract

Neither Apache Tika nor AWS Textract relies on LLMs, which makes them more predictable and consistent in terms of performance. Their deterministic algorithms ensure consistent results without the variability introduced by model updates or API interactions, providing reliability in environments where deterministic outputs are essential.

4. Limited Focus on Simple Workflows
DocETL

While DocETL excels in complex workflows, it can be overkill for simpler tasks, such as extracting text from PDFs or parsing structured data. The additional setup and optimization steps required to use DocETL effectively may add unnecessary overhead for tasks that could be handled by simpler tools.

Apache Tika and AWS Textract

Apache Tika and AWS Textract are better suited for straightforward workflows. Tika’s minimalistic approach is ideal for lightweight tasks, while Textract’s specialization in structured document processing allows users to achieve quick results without the complexity of a broader framework. These tools are more efficient when advanced features of DocETL are not required.

How Does DocETL Relate to COMS 4111 Concepts?

1. Data Modeling and Logical Design

  • Concept from COMS 4111: Understanding Entity-Relationship (ER) models and designing schemas for structured data.
  • Relation to DocETL:
    • DocETL effectively maps unstructured text data (e.g., debate transcripts) into structured themes using LLMs, which can be seen as an automated logical design process.
    • The pipeline’s ability to organize data into distinct themes mirrors the transformation of unstructured data into entities and attributes in an ER model.

2. SQL and Relational Query Processing

  • Concept from COMS 4111: Writing SQL queries, understanding joins, and executing group-by operations.
  • Relation to DocETL:
    • DocETL’s intermediate stages, such as "Unnest Themes" and "Deduplicate and Merge Themes," simulate SQL-like operations such as flattening nested data structures and removing duplicates. These operations align with relational query processing in databases.

3. Query Optimization

  • Concept from COMS 4111: Optimizing query execution plans to improve efficiency and performance.
  • Relation to DocETL:
    • By splitting large datasets into smaller, manageable components and only invoking LLMs where necessary, DocETL performs an optimization akin to minimizing costly operations in a query plan. This approach reduces costs (e.g., $0.29 to run) while preserving accuracy.

4. APIs and Integration

  • Concept from COMS 4111: Designing and using APIs to integrate applications with databases.
  • Relation to DocETL:
    • DocETL acts as a bridge between unstructured data sources (e.g., text documents) and analytical applications. Its declarative pipelines provide an API-like interface for integrating LLM-powered and non-LLM operations seamlessly.

Example: Cooking Social App

The Cooking Social App connects chefs and eaters, allowing chefs to showcase their meals through posts, and eaters to explore, interact, and leave reviews. In this example, we introduce an enhanced feature powered by DocETL: the automatic generation of profile tags from user reviews. Previously, the app only allowed users to create tags for their profiles during sign-up, reflecting their self-described skills or characteristics (e.g., "Creative Chef," "Vegetarian Specialist"). With DocETL, the system now dynamically extracts meaningful information from reviews left by others and generates new, contextually rich tags for user profiles. This feature highlights DocETL's ability to perform advanced data processing using LLM-powered pipelines, offering users a more dynamic and accurate profile representation.

Step 1: Extracting Review Data

When an eater leaves a review, they often comment on the chef's abilities, meal quality, and overall experience. Here are a few example reviews:

  • “The meal was exquisite! The dumplings were authentic, and the chef was very accommodating to my dietary needs.”
  • “Loved how prompt and organized the chef was. The presentation of the food was absolutely stunning!”
  • “Delicious vegan options! The chef clearly has a passion for healthy and sustainable cooking.”

DocETL’s LLM-powered map-reduce approach efficiently parses these reviews, identifying key phrases (e.g., "authentic dumplings," "accommodating dietary needs," "passion for healthy cooking"). DocETL’s powerful integration with natural language processing (NLP) ensures that even nuanced feedback is captured accurately.

Step 2: Generating Tags

After extracting these key attributes from the reviews, DocETL processes the text using LLM-powered map-reduce operations. This approach allows the system to aggregate and refine extracted data into actionable insights. It generates specific, contextually relevant tags, such as:

  • Authentic Dumpling Master
  • Dietary Needs Accommodator
  • Prompt & Organized
  • Vegan Food Enthusiast
  • Sustainable Cooking Advocate

These tags are automatically added to the chef’s profile, evolving as more reviews are accumulated. This process is automatic, requiring no manual input from the chef. DocETL’s declarative YAML pipeline is used to define the entire operation, streamlining the process with minimal code.

Step 3: Validating Tags

DocETL's validation and retry mechanisms ensure that only relevant, high-quality tags are created. If a review is overly generic (e.g., "good chef") or contains invalid data, DocETL can automatically retry the tag-generation process, refining the tags based on better insights or rejecting unhelpful data.

For example, if a review mentions the chef’s "good attitude," but the phrase doesn’t add value to the profile, DocETL would filter out tags like "Good Attitude" and avoid cluttering the profile with generic terms.

Step 4: Updating Profiles Dynamically

As new reviews are submitted, the chef's profile tags are updated in real-time, reflecting the most current feedback. This creates a dynamic, evolving profile that changes based on the chef’s reputation and interactions with eaters. Tags aren’t static like they were during sign-up but change and adapt based on reviews over time.

For example, a chef initially tagged as "Authentic Dumpling Master" may later gain additional tags like "Keto-Friendly Specialist" or "Outstanding Presentation" based on new reviews. This ensures the profile continuously evolves and accurately represents the chef's skills and specialties.

Tutorial

Prerequisites

We choose to use the Playground UI. Here is the tutorial of how to set up the playground.

[UI Playground](https://ucbepic.github.io/docetl/playground/) Here should be the home page once you connect to the playground. home

Once you finish building the pipelines, clicking on the top right run button and results will be generated.

However, we encountered a problem which might be related to the server side, so we didn't get our results. (will post an issue in docETL repo). error

Step1: Look At The Data

Here is a sample of our chefs' review data in reviews.json:

[
    {
        "comment": "The pasta event was fantastic! The flavors were amazing and the ambiance was cozy."
    },
    {
        "comment": "The desserts were underwhelming and not worth the price."
    },
    {
        "comment": "Great experience overall, but the main course could have been warmer."
    },
    {
        "comment": "Loved the creativity in preparing dishes. Highly recommended!"
    },
    {
        "comment": "The service was slow, and the food lacked flavor."
    },
    {
        "comment": "Unique dishes, but the portions were a bit small for the price."
    },
    {
        "comment": "The event was well-organized and fun, with delicious food."
    },
    {
        "comment": "The flavors were bland, and the presentation was lacking."
    },
    {
        "comment": "Impeccable presentation and delicious food. Highly enjoyable."
    },
    {
        "comment": "Extremely disappointing. The food was cold, and the chef was unresponsive to feedback."
    }
  ...
]

Step2: Extract common weaknesses across reviews from each document

Add a map operation

map operation

The initial output will be 3 columns: positive, negative, and content. They might not be the desirable results, might need some iterations.

Step3: Unnest/flatten the list of weaknesses

Add unnest operation(s)

unnest operation

After this step, the attributes inside the positive and negative dictionaries generated by the map operation will becomes individual columns (positive_shortname, positive_quotes, negative_shortname, negative_quotes)

Step4: Resolve/deduplicate weaknesses

Add resolve operation(s)

Here I added one for positive comments and we could add a identical one for negative comments.

resolve operation

Step5: Optimize the resolve operation so we don't have to run all pairwise comparisons (By clicking the optimize button)

optimize

Step6: Iterate on the resolve operation

Fine tune this step in the pipeline to derive the optimal results.

Here our final results can be used to shown as tags for each chef.