2024 07 06 lambda with typescript fetches recent s3 files troubleshooting undefined errors - bizkite-co/handterm GitHub Wiki

original file name: Codeium Chat - Y0sR5wPPnJqUEmkCbZlk3ZDLeMPPLyDw.md summary: The conversation involved the user requesting assistance in modifying a Lambda function (getLog.ts) to return the contents of the most recent 5 files in an S3 bucket. Codeium provided guidance on listing, sorting, and fetching the files based on their timestamps. The user encountered TypeScript errors related to possible undefined values ('a.Key', 'b.Key', 'sortedKeys') and sought advice on implementing guard clauses. Codeium suggested using filter operations to ensure keys were defined before sorting and mapping. The user then asked about handling a more informative error response if the S3 bucket was missing. Codeium advised checking for specific error codes, like 'NoSuchBucket', in AWS SDK errors and responding with appropriate status codes and messages. date: 2024-07-06T16:01

Codeium Chat Conversation

Note: This is purely the output of the chat conversation and does not contain any raw data, codebase snippets, etc. used to generate the output.

User at 4:01 pm, Jul 6, 2024

I want this to return the contents of the most recent 5 files in the S3 bucket.

The file are named withe the time integer created by new Date() in the client browser.

// cdk/lambda/userStorage/getLog.ts

import * as AWS from 'aws-sdk';
const s3 = new AWS.S3();

exports.handler = async (event: any) => {
  const authorizer = event.requestContext.authorizer;
  console.log('event:', event, 'lambda:', authorizer.lambda);
  const userId = authorizer.lambda.userId;
  const logDomain = event.queryStringParameters.wrt || '';
  console.log('userId:', userId);
  const bucketName = 'handterm';
  if (!userId) {
    return { statusCode: 401, body: JSON.stringify({ message: 'Unauthorized' }) };
  }

  try {
    const contents = await s3.listObjectsV2({
      Bucket: bucketName,
      Key: `user_data/${userId}/logs/${logDomain}`
    }).promise();

    return { statusCode: 200, body: JSON.stringify({ body: contents }) };
  } catch (err) {
    return { statusCode: 405, body: JSON.stringify(err) };
  }
};

Codeium at 4:01 pm, Jul 6, 2024

To modify your getLog.ts function to return the contents of the most recent 5 files in the S3 bucket, you'll need to adjust your implementation to:

  1. List the files in the specified S3 bucket and prefix.
  2. Sort the files based on the timestamp within their names.
  3. Fetch the content of the top 5 most recent files.
  4. Return these contents in your response.

Since your files are named with the time integer created by new Date() in the client browser, you can utilize this naming convention to sort the files. However, to perform sorting and fetching efficiently, you'll first list the files, then sort them based on the timestamp embedded in their names, and finally, retrieve the content of the top 5 files.

Here's how you can adjust your Lambda function:

import * as AWS from 'aws-sdk';
const s3 = new AWS.S3();

exports.handler = async (event: any) => {
    const authorizer = event.requestContext.authorizer;
    console.log('event:', event, 'lambda:', authorizer.lambda);
    const userId = authorizer.lambda.userId;
    const logDomain = event.queryStringParameters.wrt || '';
    console.log('userId:', userId);
    const bucketName = 'handterm';
    if (!userId) {
        return { statusCode: 401, body: JSON.stringify({ message: 'Unauthorized' }) };
    }

    try {
        // List all objects in the bucket with the specified prefix
        const listedObjects = await s3.listObjectsV2({
            Bucket: bucketName,
            Prefix: `user_data/${userId}/logs/${logDomain}`
        }).promise();
        
        // Sort the objects based on the timestamp embedded in the file names
        const sortedKeys = listedObjects.Contents
            ?.filter((item) => item.Key) // Ensure the item has a key
            .sort((a, b) => {
                // Extract timestamps from file names and convert to numbers
                const timeA = parseInt(a.Key.split('/').pop() || '0', 10);
                const timeB = parseInt(b.Key.split('/').pop() || '0', 10);
                return timeB - timeA; // Sort in descending order
            })
            .slice(0, 5); // Get the most recent 5 keys

        // Fetch the content of the top 5 most recent files
        const contents = await Promise.all(
            sortedKeys.map(async (keyItem) => {
                const s3Response = await s3.getObject({
                    Bucket: bucketName,
                    Key: keyItem.Key,
                }).promise();
                return s3Response.Body?.toString('utf-8');
            })
        );

        return { statusCode: 200, body: JSON.stringify({ contents }) };
    } catch (err) {
        console.error('Error:', err);
        return { statusCode: 500, body: JSON.stringify(err) };
    }
};

This updated function does the following:

  • Lists objects in the specified S3 bucket and prefix.
  • Filters out any items without keys and sorts the remaining items based on the timestamp extracted from their file names.
  • Retrieves the content of the top 5 most recent files.
  • Returns the contents of these files in the response.

Please note, this example assumes that the timestamps can be extracted as integers directly from the file names and sorted numerically. If your filename structure is different or more complex, you may need to adjust the sorting logic accordingly.

User at 4:04 pm, Jul 6, 2024

I am getting this error for a.Key, b.Key and sortedKeys.map()

'a.Key' is possibly 'undefined'.ts(18048)

I think I need a guard clause.

Codeium at 4:04 pm, Jul 6, 2024

To address the TypeScript error 'a.Key' is possibly 'undefined'.ts(18048), you can indeed use a guard clause or a type assertion to ensure that the Key property is not undefined before proceeding with operations that assume its existence. A straightforward way to handle this within your current code structure is to filter out any objects where the Key is undefined before attempting to sort or map over them.

Here's how you can adjust your code to include a check that only processes objects with a defined Key property:

import * as AWS from 'aws-sdk';
const s3 = new AWS.S3();

exports.handler = async (event: any) => {
    const authorizer = event.requestContext.authorizer;
    const userId = authorizer.lambda.userId;
    const logDomain = event.queryStringParameters.wrt || '';
    const bucketName = 'handterm';
    if (!userId) {
        return { statusCode: 401, body: JSON.stringify({ message: 'Unauthorized' }) };
    }

    try {
        const listedObjects = await s3.listObjectsV2({
            Bucket: bucketName,
            Prefix: `user_data/${userId}/logs/${logDomain}`
        }).promise();
        
        // Ensure items have a key before sorting
        const sortedKeys = listedObjects.Contents
            ?.filter((item) => item.Key !== undefined) // Filter out items without a Key
            .sort((a, b) => {
                // Assuming both a.Key and b.Key exist due to filter above
                const timeA = parseInt(a.Key!.split('/').pop() || '0', 10);
                const timeB = parseInt(b.Key!.split('/').pop() || '0', 10);
                return timeB - timeA; // Sort in descending order
            })
            .slice(0, 5); // Get the most recent 5 keys

        const contents = await Promise.all(
            sortedKeys.map(async (keyItem) => {
                // Key existence is guaranteed by the filter, but TypeScript doesn't infer this
                const s3Response = await s3.getObject({
                    Bucket: bucketName,
                    Key: keyItem.Key!,
                }).promise();
                return s3Response.Body?.toString('utf-8');
            })
        );

        return { statusCode: 200, body: JSON.stringify({ contents }) };
    } catch (err) {
        console.error('Error:', err);
        return { statusCode: 500, body: JSON.stringify(err) };
    }
};

In this updated code:

  • The .filter((item) => item.Key !== undefined) line ensures that only objects with a defined Key are processed. This removes the need for a guard clause later in the code by guaranteeing that a.Key and b.Key are not undefined.
  • The non-null assertion operator (!) is used when accessing a.Key and b.Key in the sort method and keyItem.Key in the map method. This tells TypeScript that you're certain Key will not be undefined at this point. It's safe here because of the preceding filter, but use this operator cautiously, as it bypasses TypeScript's type checks.

This code should eliminate the TypeScript error by ensuring that Key is always defined when accessed.

User at 4:06 pm, Jul 6, 2024

I am still getting:

'sortedKeys' is possibly 'undefined'.ts(18048)
⚠ Error (TS18048)  | 
sortedKeys is possibly  undefined

Codeium at 4:06 pm, Jul 6, 2024

The error indicating that 'sortedKeys' is possibly 'undefined' can arise if TypeScript thinks that the result of the chain starting with listedObjects.Contents?.filter(...).sort(...).slice(0, 5) could be undefined. This could happen because the optional chaining operator (?.) used with listedObjects.Contents?.filter introduces the possibility that the entire expression evaluates to undefined if Contents is undefined.

To address this, you can ensure that sortedKeys is always treated as an array, even in the case where listedObjects.Contents might be undefined. This can be achieved by providing a fallback empty array ([]) for cases where the optional chaining might result in undefined.

Here's how you can adjust the relevant part of your code:

// Ensure items have a key before sorting
const sortedKeys = (listedObjects.Contents || [])
    .filter((item) => item.Key !== undefined) // Filter out items without a Key
    .sort((a, b) => {
        // Assuming both a.Key and b.Key exist due to filter above
        const timeA = parseInt(a.Key!.split('/').pop() || '0', 10);
        const timeB = parseInt(b.Key!.split('/').pop() || '0', 10);
        return timeB - timeA; // Sort in descending order
    })
    .slice(0, 5); // Get the most recent 5 keys

// Proceed with the rest of your code...

By replacing the optional chaining operator (?.) with a logical OR (||) and providing an empty array as a fallback, you guarantee that the expression on the right-hand side of the assignment to sortedKeys evaluates to an array. This array may be empty if Contents is undefined or if no items have a defined Key, but it will not be undefined itself, thus resolving the TypeScript error.

This change ensures that sortedKeys is always an array, and subsequent operations on sortedKeys (like .map) can proceed without TypeScript fearing that sortedKeys could be undefined.

User at 4:08 pm, Jul 6, 2024

Isn't there a way to return a better error than 500 if the S3 bucket is missing?

Codeium at 4:08 pm, Jul 6, 2024

Yes, you can handle specific AWS SDK errors more gracefully by inspecting the error object returned when an operation fails. The AWS SDK errors contain a code property that represents the error type, which you can use to differentiate between various errors and respond accordingly.

For example, if an S3 operation fails because the bucket does not exist, the error code will be 'NoSuchBucket'. You can check for this specific error code and return a more informative error response instead of a generic 500 Internal Server Error.

Here's how you can modify your getLog.ts Lambda function to include this error handling:

import * as AWS from 'aws-sdk';
const s3 = new AWS.S3();

exports.handler = async (event: any) => {
    const authorizer = event.requestContext.authorizer;
    const userId = authorizer.lambda.userId;
    const logDomain = event.queryStringParameters.wrt || '';
    const bucketName = 'handterm';
    if (!userId) {
        return { statusCode: 401, body: JSON.stringify({ message: 'Unauthorized' }) };
    }

    try {
        const listedObjects = await s3.listObjectsV2({
            Bucket: bucketYourName,
            Prefix: `user_data/${userId}/logs/${logDomain}`
        }).promise();

        // Your existing logic to sort and fetch objects goes here

    } catch (err) {
        console.error('Error:', err);
        // Check if the error is because the S3 bucket does not exist
        if (err.code === 'NoSuchBucket') {
            return { statusCode: 404, body: JSON.stringify({ message: 'Bucket does not exist' }) };
        }
        // Handle other AWS SDK errors or unknown errors
        else if (err instanceof AWS.AWSError) {
            return { statusCode: 400, body: JSON.stringify({ message: `AWS error: ${err.message}` }) };
        } else {
            return { statusCode: 500, body: JSON.stringify({ message: 'Internal Server Error' }) };
        }
    }
};

In this updated code, when an error occurs in the try block, the catch block checks if the error is an instance of AWS.AWSError and then further checks the code property to see if the error is specifically 'NoSuchBucket'. If so, it returns a 404 status code with a more descriptive message. This approach provides clearer feedback to the client about the nature of the error.

For other AWS SDK errors, you might choose to handle them differently or log additional information. For errors not specific to AWS SDK operations, a generic 500 Internal Server Error response is returned, but you can extend this logic to handle other types of errors as needed.

⚠️ **GitHub.com Fallback** ⚠️