functionbeat - Could not execute the lambda function

[kaykhan]

We had functionbeat running and i had to remove and deploy it, however im getting the following errors:

{"log.level":"info","@timestamp":"2022-09-23T13:55:44.066+0100","log.logger":"aws","log.origin":{"file.name":"aws/op_cloudformation.go","file.line":97},"message":"Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbcloudwatchSFawslambdanfttransposeprodmain, ResourceStatus: CREATE_FAILED, ResourceStatusReason: Resource handler returned message: "Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function. (Service: CloudWatchLogs, Status Code: 400, Request ID: 56335ff6-e8d8-4536-bbfa-44f72d15426f)" (RequestToken: a0b9b8fc-e348-398b-dd38-dc91cabcb2ab, HandlerErrorCode: InternalFailure)","service.name":"functionbeat","ecs.version":"1.6.0"}

So i tried to add the log groups 1-by-1, but at some point it fails again (random points). I dont know why it fails, it suggests to check permissions but i dont believe that is the issue as im able to add log groups sometimes and other times i get this error. Once i get this error, the functionbeat is in an unrecoverable state and i can no longer rollback/update. I have to remove and redeploy and start again.

For confirmed bugs, please report:

• Version: functionbeat 8.2.2
• Operating System: functionbeat serverless
• Discuss Forum URL: https://discuss.elastic.co/t/functionbeat-fails-to-update-with-a-new-cloudwatch-log-group-could-not-execute-the-lambda-function/315020
• Steps to Reproduce:

###################### Functionbeat Configuration Example #######################

# This file is an example configuration file highlighting only the most common
# options. The functionbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/functionbeat/index.html
#

# ================================== Provider ==================================
# Configure functions to run on AWS Lambda, currently we assume that the credentials
# are present in the environment to correctly create the function when using the CLI.
#
# Configure which S3 endpoint should we use.
functionbeat.provider.aws.endpoint: "s3.amazonaws.com"
# Configure which S3 bucket we should upload the lambda artifact.
functionbeat.provider.aws.deploy_bucket: "functionbeat-deploy-geeiq"

functionbeat.provider.aws.functions:
  # Define the list of function availables, each function required to have a unique name.
  # Create a function that accepts events coming from cloudwatchlogs.
  - name: cloudwatch
    enabled: true 
    type: cloudwatch_logs

    # Description of the method to help identify them when you run multiples functions.
    description: "lambda function for cloudwatch logs"

    # Concurrency, is the reserved number of instances for that function.
    # Default is 5.
    #
    # Note: There is a hard limit of 1000 functions of any kind per account.
    #concurrency: 5

    # The maximum memory allocated for this function, the configured size must be a factor of 64.
    # There is a hard limit of 3008MiB for each function. Default is 128MiB.
    #memory_size: 128MiB

    # Dead letter queue configuration, this must be set to an ARN pointing to a SQS queue.
    #dead_letter_config.target_arn:

    # Execution role of the function.
    #role: arn:aws:iam::123456789012:role/MyFunction

    # Connect to private resources in an Amazon VPC.
    #virtual_private_cloud:
    #  security_group_ids: []
    #  subnet_ids: []

    # Optional fields that you can specify to add additional information to the
    # output. Fields can be scalar values, arrays, dictionaries, or any nested
    # combination of these.
    #fields:
    #  env: staging

    # List of cloudwatch log group registered to that function.
    triggers:
      #- log_group_name: /aws/eks/<redacted>
      - log_group_name: /aws/lambda/<redacted>
      - log_group_name: /aws/lambda/<redacted>
      - log_group_name: /aws/lambda/<redacted>
      - log_group_name: /aws/lambda/<redacted>
      - log_group_name: /aws/lambda/<redacted>
      - log_group_name: /aws/lambda/<redacted>
      - log_group_name: /aws/lambda/<redacted>
      - log_group_name: /aws/lambda/<redacted>
      - log_group_name: /aws/lambda/<redacted>
      - log_group_name: /aws/lambda/<redacted>
      - log_group_name: /aws/lambda/<redacted>
      - log_group_name: /aws/lambda/<redacted>
        #- log_group_name: /aws/lambda/<redacted>
        #- log_group_name: /aws/lambda/<redacted>
        #- log_group_name: /aws/lambda/<redacted>
        #- log_group_name: /aws/lambda/<redacted>
        #- log_group_name: /aws/lambda/<redacted>
        #- log_group_name: /aws/lambda/<redacted>


    # Define custom processors for this function.
    processors:
     - decode_json_fields:
        fields: ["message"]
        process_array: false
        max_depth: 5
        target: "acme"
        overwrite_keys: true
        add_error_key: true  # Create a function that accepts events from SQS queues.

  - name: sqs
    enabled: false
    type: sqs

    # Description of the method to help identify them when you run multiples functions.
    description: "lambda function for SQS events"

    # Concurrency, is the reserved number of instances for that function.
    # Default is 5.
    #
    # Note: There is a hard limit of 1000 functions of any kind per account.
    #concurrency: 5

    # The maximum memory allocated for this function, the configured size must be a factor of 64.
    # There is a hard limit of 3008MiB for each function. Default is 128MiB.
    #memory_size: 128MiB

    # Dead letter queue configuration, this must be set to an ARN pointing to a SQS queue.
    #dead_letter_config.target_arn:

    # Execution role of the function.
    #role: arn:aws:iam::123456789012:role/MyFunction

    # Connect to private resources in an Amazon VPC.
    #virtual_private_cloud:
    #  security_group_ids: []
    #  subnet_ids: []

    # Optional fields that you can specify to add additional information to the
    # output. Fields can be scalar values, arrays, dictionaries, or any nested
    # combination of these.
    #fields:
    #  env: staging

    # List of SQS queues.
    triggers:
        # Arn for the SQS queue.
      - event_source_arn: arn:aws:sqs:us-east-1:xxxxx:myevents

    # Define custom processors for this function.
    #processors:
    #  - decode_json_fields:
    #      fields: ["message"]
    #      process_array: false
    #      max_depth: 1
    #      target: ""
    #      overwrite_keys: false
    #

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[el-Joft]

@kaykhan have you been able to resolve this? I ran into the same problem updating my functiobeat code.

[kaykhan]

@kaykhan have you been able to resolve this? I ran into the same problem updating my functiobeat code.

No i was not able to resolve it :/ For now we stopped using the functionbeat code and started to manually update/remove the log groups from the generated lambda function.

[el-Joft]

@kaykhan have you been able to resolve this? I ran into the same problem updating my functiobeat code.

No i was not able to resolve it :/ For now we stopped using the functionbeat code and started to manually update/remove the log groups from the generated lambda function.

Ok. Thank you

[kaykhan]

@kaykhan have you been able to resolve this? I ran into the same problem updating my functiobeat code.

No i was not able to resolve it :/ For now we stopped using the functionbeat code and started to manually update/remove the log groups from the generated lambda function.

Ok. Thank you

Curious what version of functionbeat are you using ?

Perhaps a newer version will fix this issue? But I've not had the chance to try it

[el-Joft]

@kaykhan have you been able to resolve this? I ran into the same problem updating my functiobeat code.

No i was not able to resolve it :/ For now we stopped using the functionbeat code and started to manually update/remove the log groups from the generated lambda function.

Ok. Thank you

Curious what version of functionbeat are you using ?

Perhaps a newer version will fix this issue? But I've not had the chance to try it

Tried with version 8.1.2 and 8.5.0 same issue

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[nimarezainia]

Functionbeat has reached end-of-life. Please use Elastic Serverless Forwarder