Merge pull request #398 from aws-solutions/release/v2.1.2

Release v2.1.2

CHANGELOG.md

@@ -5,6 +5,15 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.1.2] - 2023-10-19
+
+### Changed
+
+* Upgraded to Media Insights on AWS v5.1.4
+* Consolidated policy statements into one S3 bucket policy for the website bucket
+* Updated instructions in README to opt out of metric reporting
+* Updated all Lambda functions Python Runtime version to 3.11
+
 ## [2.1.1] - 2023-07-13
 
 ### Changed

NOTICE

@@ -25,7 +25,7 @@ botocore under the Apache License Version 2.0
 cfn-lint under the Massachusetts Institute of Technology license
 cfnresponse under the Apache License Version 2.0
 chalice under the Apache License Version 2.0
-char.js under the Massachusetts Institute of Technology license
+chart.js under the Massachusetts Institute of Technology license
 diff under the BSD with attribution (BSD-3) license
 dropzone under the Massachusetts Institute of Technology license
 elasticsearch-py under the Apache License Version 2.0
@@ -37,11 +37,11 @@ number-to-words under the Massachusetts Institute of Technology license
 pytest under the Massachusetts Institute of Technology license
 register-service-worker under the Massachusetts Institute of Technology license
 requests under the Apache License Version 2.0
-requests_aws4auth under the Massachusetts Institute of Technology license
+requests-aws4auth under the Massachusetts Institute of Technology license
 routerjs under the Massachusetts Institute of Technology license
 selenium under the Apache License Version 2.0
 urllib3 under the Massachusetts Institute of Technology license
-videojs under the Apache License Version 2.0
+video.js under the Apache License Version 2.0
 videojs-flash under the Apache License Version 2.0
 videojs-hotkeys under the Apache License Version 2.0
 videojs-markers under the Massachusetts Institute of Technology license
@@ -51,5 +51,5 @@ vue-highlightjs under the Massachusetts Institute of Technology license
 vue-router under the Massachusetts Institute of Technology license
 vuex under the Massachusetts Institute of Technology license
 vuex-persistedstate under the Massachusetts Institute of Technology license
-wavesurfer.js under the Creative Commons Attribution 3.0 Unported License
-webdriver_manager under the Apache License Version 2.0
+wavesurfer.js under the BSD with attribution (BSD-3) license
+webdriver-manager under the Apache License Version 2.0

README.md

@@ -1,6 +1,3 @@
-[![scheduled-workflow](https://github.com/aws-solutions/content-localization-on-aws/actions/workflows/scheduled-workflow.yml/badge.svg)](https://github.com/aws-solutions/content-localization-on-aws/actions/workflows/scheduled-workflow.yml)
-
-
 # Content Localization on AWS 
 
 Welcome to the Content Localization on AWS project!   This project will help you extend the reach of your VOD content by quickly and efficiently creating accurate multi-language subtitles using AWS AI Services.  You can make manual corrections to the automatically created subtitles and use advanced AWS AI Service customization features to improve the results of the automation for your content domain. Content Localization is built on [Media Insights on AWS](https://github.com/aws-solutions/media-insights-on-aws), a framework that helps accelerate the development of serverless applications that process video, images, audio, and text with artificial intelligence services and multimedia services on AWS. 
@@ -326,7 +323,7 @@ aws s3 rb s3://<bucket-name> --force
 
 This solution collects anonymized operational metrics to help AWS improve the
 quality of features of the solution. For more information, including how to disable
-this capability, please see the [implementation guide](https://docs.aws.amazon.com/solutions/latest/content-localization-on-aws/operational-metrics.html).
+this capability, please see the [implementation guide](https://docs.aws.amazon.com/solutions/latest/content-localization-on-aws/reference.html#anonymized-data-collection).
 
 When enabled, the following information is collected and sent to AWS:
 
@@ -350,7 +347,7 @@ Example data:
 }
 ```
 
-To opt out of this reporting, edit [deployment/content-localization-on-aws.yaml](deployment/content-localization-on-aws.yaml) and change `AnonymizedData` in the `Mappings` section from:
+To opt out of this reporting on the Content Localization on AWS stack, edit [deployment/content-localization-on-aws.yaml](deployment/content-localization-on-aws.yaml) and change `AnonymizedData` in the `Mappings` section from:
 
 ```
 AnonymizedData:
@@ -366,6 +363,47 @@ AnonymizedData:
       Data: "No"
 ```
 
+To opt out of this reporting on the nested Media Insights on AWS stack, do one of the following options:
+
+### *Option 1:* Deploy Media Insights on AWS separately with metric reporting disabled
+
+1. Follow the instructions to disable metric reporting in the [Anonymized Data Collection](https://docs.aws.amazon.com/solutions/latest/media-insights-on-aws/reference.html#anonymized-data-collection) section of the Media Insights on AWS implementation guide, then deploy the solution
+2. Follow the instructions to [Install Content Localization on AWS over an existing Media Insights on AWS stack](#option-1-install-content-localization-on-aws-over-an-existing-media-insights-on-aws-stack) 
+
+### *Option 2:* Modify the source template that is used to deploy Media Insights on AWS as a nested stack
+
+1. Download the Media Insights on AWS template from `https://solutions-reference.s3.amazonaws.com/media-insights-on-aws/<version>/media-insights-on-aws-stack.template` and replace `<version>` to get the specific Media Insights on AWS version you would like to deploy
+2. Follow the instructions to disable metric reporting in the [Anonymized Data Collection](https://docs.aws.amazon.com/solutions/latest/media-insights-on-aws/reference.html#anonymized-data-collection) section of the Media Insights on AWS implementation guide, but do not deploy this template
+3. Upload the template to an S3 bucket
+4. Edit [deployment/content-localization-on-aws.yaml](deployment/content-localization-on-aws.yaml) and change `TemplateURL` within the `MieStack` properties to point to the template you've uploaded to S3
+
+    from
+
+    ```
+    MieStack:
+        Type: "AWS::CloudFormation::Stack"
+        Properties:
+          TemplateURL: !Join
+            - ""
+            - - "https://solutions-reference.s3.amazonaws.com/media-insights-on-aws/"
+              - !FindInMap
+                - MediaInsights
+                - Release
+                - Version
+              - "/media-insights-on-aws-stack.template"
+    ```
+
+    to
+
+    ```
+    MieStack:
+        Type: "AWS::CloudFormation::Stack"
+        Properties:
+          TemplateURL: "<new url>"
+    ```
+
+5. Use the instructions in previous sections to [build](#building-the-solution-from-source-code) and [deploy](#option-2-install-content-localization-on-aws-with-a-new-media-insights-on-aws-stack) Content Localization on AWS
+
 # Help
 
 Join our Gitter chat at https://gitter.im/awslabs/aws-media-insights-engine. This public chat forum was created to foster communication between Media Insights on AWS developers worldwide.

deployment/build-s3-dist.sh

@@ -306,11 +306,11 @@ cp "./dist/websitehelper.zip" "$regional_dist_dir/websitehelper.zip"
 rm -rf dist
 
 echo "------------------------------------------------------------------------------"
-echo "Creating deployment package for anonymous data logger"
+echo "Creating deployment package for anonymized data logger"
 echo "------------------------------------------------------------------------------"
 
-echo "Building anonymous data logger"
-cd "$source_dir/anonymous-data-logger" || exit 1
+echo "Building anonymized data logger"
+cd "$source_dir/anonymized-data-logger" || exit 1
 [ -e dist ] && rm -rf dist
 mkdir -p dist
 [ -e package ] && rm -rf package
@@ -325,14 +325,14 @@ echo "[install]" > ./setup.cfg
 echo "prefix= " >> ./setup.cfg
 pip3 install --quiet -r ../requirements.txt --target .
 cp -R ../lib .
-if ! [ -d ../dist/anonymous-data-logger.zip ]; then
-  zip -q -r9 ../dist/anonymous-data-logger.zip .
-elif [ -d ../dist/anonymous-data-logger.zip ]; then
+if ! [ -d ../dist/anonymized-data-logger.zip ]; then
+  zip -q -r9 ../dist/anonymized-data-logger.zip .
+elif [ -d ../dist/anonymized-data-logger.zip ]; then
   echo "Package already present"
 fi
 popd || exit 1
-zip -q -g -9 ./dist/anonymous-data-logger.zip ./anonymous-data-logger.py
-cp "./dist/anonymous-data-logger.zip" "$regional_dist_dir/anonymous-data-logger.zip"
+zip -q -g -9 ./dist/anonymized-data-logger.zip ./anonymized-data-logger.py
+cp "./dist/anonymized-data-logger.zip" "$regional_dist_dir/anonymized-data-logger.zip"
 
 # Clean up the build directories
 rm -rf dist

deployment/content-localization-on-aws-auth.yaml

@@ -1,5 +1,5 @@
 AWSTemplateFormatVersion: "2010-09-09"
-Description: "Content Localization on AWS %%VERSION%% - user authentication infrastructure"
+Description: "Content Localization on AWS user authentication infastructure %%VERSION%%"
 
 Parameters:
   AdminEmail:
@@ -92,7 +92,7 @@ Resources:
         - - index
           - .handler
       Role: !GetAtt CognitoRoleMapperLambdaExecutionRole.Arn
-      Runtime: python3.7
+      Runtime: python3.11
       Timeout: 30
 
   CognitoRoleMapperLambdaExecutionRole:

deployment/content-localization-on-aws-opensearch.yaml

@@ -1,5 +1,5 @@
 AWSTemplateFormatVersion: "2010-09-09"
-Description: "Content Localization on AWS %%VERSION%% - Opensearch Consumer stack"
+Description: "Content Localization on AWS Opensearch consumer stack %%VERSION%%"
 
 Parameters:
   NodeType:
@@ -92,7 +92,7 @@ Resources:
               "esconsumer.zip",
             ],
           ]
-      Runtime: "python3.7"
+      Runtime: "python3.11"
       Timeout: 900
       MemorySize: 2048
       Environment:

deployment/content-localization-on-aws-use-existing-mie-stack.yaml

@@ -1,5 +1,5 @@
 AWSTemplateFormatVersion: "2010-09-09"
-Description: "(SO0164) Content Localization on AWS Version %%VERSION%% - Deploys the Content Localization on AWS Application over an existing Media Insights on AWS stack."
+Description: "(SO0164) Content Localization on AWS using an existing Media Insights on AWS stack %%VERSION%%"
 
 Parameters:
   MieStackName:

deployment/content-localization-on-aws-video-workflow.yaml

@@ -1,5 +1,5 @@
 AWSTemplateFormatVersion: "2010-09-09"
-Description: "Content Localization on AWS %%VERSION%% - Deploys video processing workflow"
+Description: "Content Localization on AWS video processing workflow %%VERSION%%"
 
 Parameters:
   WorkflowCustomResourceArn:

deployment/content-localization-on-aws-web.yaml

@@ -1,5 +1,5 @@
 AWSTemplateFormatVersion: "2010-09-09"
-Description: "Content Localization on AWS %%VERSION%% - Deploys the Content Localization on AWS website resources"
+Description: "Content Localization on AWS website resources %%VERSION%%"
 
 Parameters:
   DataplaneEndpoint:
@@ -47,7 +47,7 @@ Resources:
               response_data = {'Data': stack_name.lower() + '-website'}
               cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data, "CustomResourcePhysicalID")
       Handler: index.handler
-      Runtime: python3.8
+      Runtime: python3.11
       Role: !GetAtt WebsiteBucketNameExecutionRole.Arn
 
   WebsiteBucketNameFunctionPermissions:
@@ -121,22 +121,6 @@ Resources:
             AbortIncompleteMultipartUpload:
               DaysAfterInitiation: 1
 
-  WebBucketPolicy:
-    Type: AWS::S3::BucketPolicy
-    Properties:
-      Bucket: !Ref ContentAnalysisWebsiteBucket
-      PolicyDocument:
-        Statement:
-          - Effect: Deny
-            Principal: "*"
-            Action: "*"
-            Resource:
-              - !Sub "arn:aws:s3:::${ContentAnalysisWebsiteBucket}/*"
-              - !Sub "arn:aws:s3:::${ContentAnalysisWebsiteBucket}"
-            Condition:
-              Bool:
-                aws:SecureTransport: false
-
   CopyWebSource:
     DependsOn: ContentAnalysisWebsiteBucket
     Type: Custom::WebsiteDeployHelper
@@ -171,6 +155,15 @@ Resources:
               - !Sub "arn:aws:s3:::${ContentAnalysisWebsiteBucket}/*"
             Principal:
               CanonicalUser: !GetAtt ContentAnalysisOriginAccessIdentity.S3CanonicalUserId
+          - Effect: Deny
+            Principal: "*"
+            Action: "*"
+            Resource:
+              - !Sub "arn:aws:s3:::${ContentAnalysisWebsiteBucket}/*"
+              - !Sub "arn:aws:s3:::${ContentAnalysisWebsiteBucket}"
+            Condition:
+              Bool:
+                aws:SecureTransport: false
 
   WebsiteDistribution:
     Type: AWS::CloudFront::Distribution
@@ -293,7 +286,7 @@ Resources:
       Handler: website_helper.lambda_handler
       MemorySize: 256
       Role: !GetAtt WebsiteHelperRole.Arn
-      Runtime: python3.9
+      Runtime: python3.11
       Timeout: 900
       Environment:
         Variables:
@@ -305,6 +298,7 @@ Resources:
           IdentityPoolId: !Ref IdentityPoolId
           AwsRegion: !Ref AWS::Region
           PoolClientId: !Ref PoolClientId
+          botoConfig: '{"user_agent_extra": "AwsSolution/SO0164/%%VERSION%%"}'
 
 Outputs:
   CloudfrontUrl:

deployment/content-localization-on-aws.yaml

@@ -1,5 +1,5 @@
 AWSTemplateFormatVersion: "2010-09-09"
-Description: "(SO0164) Content Localization on AWS Version %%VERSION%% - Deploys the Content Localization on AWS Application"
+Description: "(SO0164) Content Localization on AWS %%VERSION%%"
 
 Parameters:
   AdminEmail:
@@ -24,7 +24,7 @@ Conditions:
 Mappings:
   MediaInsights:
     Release:
-      Version: "v5.1.1"
+      Version: "v5.1.4"
   Application:
     SourceCode:
       GlobalS3Bucket: "%%GLOBAL_BUCKET_NAME%%"
@@ -61,7 +61,6 @@ Resources:
         DeployTestResources: "No"
         MaxConcurrentWorkflows: 5
         EnableXrayTrace: "Yes"
-        SendAnonymousData: !FindInMap [AnonymizedData,SendAnonymizedData,Data]
         SolutionId: SO0164
         SolutionVersion: "%%VERSION%%"
 
@@ -169,7 +168,7 @@ Resources:
             - MieStack
             - Outputs.OperatorLibraryStack
 
-  AnonymousDataCustomResourceRole:
+  AnonymizedDataCustomResourceRole:
     Type: AWS::IAM::Role
     Properties:
       AssumeRolePolicyDocument:
@@ -185,7 +184,7 @@ Resources:
       Path: "/"
       Policies:
         -
-          PolicyName: !Sub "${AWS::StackName}-anonymous-data-logger"
+          PolicyName: !Sub "${AWS::StackName}-anonymized-data-logger"
           PolicyDocument:
             Statement:
               -
@@ -203,7 +202,7 @@ Resources:
                 Resource:
                   - !Join ["", ["arn:aws:ssm:", Ref: "AWS::Region", ":", Ref: "AWS::AccountId", ":parameter/*"]]
 
-  AnonymousDataCustomResource:
+  AnonymizedDataCustomResource:
     Type: AWS::Lambda::Function
     Metadata:
       cfn_nag:
@@ -213,32 +212,32 @@ Resources:
           - id: W92
             reason: "This function does not require performance optimization, so the default concurrency limits suffice."
     Properties:
-      FunctionName: !Sub ${AWS::StackName}-anonymous-data
-      Description: Used to send anonymous data
-      Handler: anonymous-data-logger.handler
-      Role: !GetAtt AnonymousDataCustomResourceRole.Arn
+      FunctionName: !Sub ${AWS::StackName}-anonymized-data
+      Description: Used to send anonymized data
+      Handler: anonymized-data-logger.handler
+      Role: !GetAtt AnonymizedDataCustomResourceRole.Arn
       Code:
         S3Bucket: !Join [ "-", [ !FindInMap [ "Application", "SourceCode", "RegionalS3Bucket" ], Ref: "AWS::Region" ] ]
-        S3Key: !Join [ "/", [ !FindInMap [ "Application", "SourceCode", "CodeKeyPrefix" ], "anonymous-data-logger.zip" ] ]
-      Runtime:  python3.8
+        S3Key: !Join [ "/", [ !FindInMap [ "Application", "SourceCode", "CodeKeyPrefix" ], "anonymized-data-logger.zip" ] ]
+      Runtime:  python3.11
       Timeout: 180
 
-  # SendAnonymousData
-  AnonymousDataUuid:
+  # SendAnonymizedData
+  AnonymizedDataUuid:
     Condition: EnableAnonymizedData
     Type: "Custom::UUID"
     Properties:
-      ServiceToken: !GetAtt AnonymousDataCustomResource.Arn
+      ServiceToken: !GetAtt AnonymizedDataCustomResource.Arn
       Resource: UUID
 
-  AnonymousMetric:
+  AnonymizedMetric:
     Condition: EnableAnonymizedData
-    Type: "Custom::AnonymousMetric"
+    Type: "Custom::AnonymizedMetric"
     Properties:
-      ServiceToken: !GetAtt AnonymousDataCustomResource.Arn
-      Resource: AnonymousMetric
-      SolutionId: "SO0042"
-      UUID: !GetAtt AnonymousDataUuid.UUID
+      ServiceToken: !GetAtt AnonymizedDataCustomResource.Arn
+      Resource: AnonymizedMetric
+      SolutionId: "SO0164"
+      UUID: !GetAtt AnonymizedDataUuid.UUID
       Version: !FindInMap ["Application", "SourceCode", "Version"]
 
   # AppRegistry

source/anonymous-data-logger/anonymous-data-logger.py → source/anonymized-data-logger/anonymized-data-logger.py

@@ -16,8 +16,8 @@
 ##############################################################################
 #
 # PURPOSE:
-# This function sends anonymous performance data to the AWS
-# Solutions metrics API. This information is anonymous and helps improve the
+# This function sends anonymized performance data to the AWS
+# Solutions metrics API. This information is anonymized and helps improve the
 # quality of the solution.
 #
 ##############################################################################
@@ -48,7 +48,7 @@ def handler(event, context):
                 response_data = {'UUID': str(uuid.uuid4())}
                 unique_id = response_data['UUID']
                 cfn.send(event, context, 'SUCCESS', response_data, unique_id)
-            elif resource == 'AnonymousMetric':
+            elif resource == 'AnonymizedMetric':
                 Metrics.send_metrics(config)
                 unique_id = 'Metrics Sent'
                 cfn.send(event, context, 'SUCCESS', response_data, unique_id)

source/anonymized-data-logger/requirements.txt

@@ -0,0 +1,2 @@
+requests
+urllib3==1.26.18