To fix the RecursionError: maximum recursion depth exceeded when iterating over objects in an S3 bucket, you can refactor your code to use an iterative approach instead of a recursive one. This way, you avoid deep recursion, which can hit Python's recursion limit.

Here's a guide to convert your recursive function into an iterative one using a stack:

Usual Recursive Function
The usual recursive function we are using:

import boto3

def list_objects(bucket_name, prefix=''):
    s3 = boto3.resource('s3')
    bucket = s3.Bucket(bucket_name)
    for obj in bucket.objects.filter(Prefix=prefix):
        print(obj.key)
        if obj.key.endswith('/'):
            list_objects(bucket_name, prefix=obj.key)

bucket_name = 'your-bucket-name'
list_objects(bucket_name)

Iterative Approach
To convert this to an iterative approach, you can use a stack to manage the prefixes to process:
import boto3

def list_objects(bucket_name, prefix=''):
    s3 = boto3.resource('s3')
    bucket = s3.Bucket(bucket_name)
    
    # Initialize a stack with the initial prefix
    stack = prefix
        while stack:
        current_prefix = stack.pop()
        for obj in bucket.objects.filter(Prefix=current_prefix):
            print(obj.key)
            # If the object key ends with '/', it indicates a "directory"
            if obj.key.endswith('/'):
                stack.append(obj.key)

# Call the function with your bucket name
bucket_name = 'your-bucket-name'
list_objects(bucket_name)

Explanation

Initialization:
  • Create a stack initialized with the starting prefix.
  • Set up the S3 resource and bucket.
Iteration Using a Stack:
  • Use a while loop to process prefixes from the stack until it is empty.
  • Pop the current prefix from the stack.
Filtering and Processing Objects:
  • List objects in the bucket with the current prefix using the filter method.
  • Print the key of each object.
Handling "Directories":
  • If an object's key ends with a '/', it is treated as a "directory".
  • Push these "directory" keys onto the stack for further processing.

Benefits
Avoids Recursion Limit: The iterative approach using a stack avoids hitting the recursion limit by not relying on function call stack for recursion.

Simple and Clear: The iterative method is often easier to understand and debug compared to deeply nested recursive calls.

This iterative approach ensures that you can handle large and deeply nested S3 bucket structures without encountering the RecursionError.