Skip to content

Latest commit

 

History

History
141 lines (121 loc) · 7.2 KB

application-integration.md

File metadata and controls

141 lines (121 loc) · 7.2 KB

Queueing (SQS)

  • What is Messaging System ?

    • Used to provide asynchronous communication and decouple processes via messages / events from sender and receiver (producer and consumer)

  • What is Queuing System ?

    • A queueing system is a messaging system that generally will delete messages once they are consumed .
    • Simple Communication
    • Not Real-time
    • Have to pull
    • Not reactive

  • Simple Queuing System (SQS)

    • Fully managed queuing service that enables you to decouple and scale mircroservices, distributed systems, and serverless applications
    • Use Case: You need to queue up transaction emails to be sent
    • e.g. Signup, Reset Password
    • Default retention 4 Days and Max of 14 days
    • Limitation of 256 KB per message sent
    • Low Latency (<10ms on publish and receive)
    • Can have duplicate messages (at least once delivery, occasionally)
    • Unlimited Throughput
    • Encryption:
      • In-flight encryption using HTTPS API
      • At-rest encryption using KMS keys
      • Client-side encryption if the client wants to perform encryption/decryption itself

SQS Message Visibility Timeout

  • After a message is polled by a customer it becomes invisible to other consumers
  • By default the " message visibility timeout" is 30 seconds
  • That means the message has 30 seconds to process
  • If the message is not processed in the visibility timeout, it will be processed twice
  • A consumer could call the ChangeMessageVisibility API to get more time
  • If the visibility timeout is high(hours) and consumer crashes, reprocessing will take time
  • If visibility timeout is too low (seconds), we may get duplicates

Long Polling

  • When a consumer requests messages from the queue, it can optionally 'wait' for messages to arrive if there are none in the queue - Long Polling
  • Long Polling decreases the number of API calls made to SQS while increasing the latency and efficiency of the application
  • The wait time can be between 1 sec to 20 sec
  • Long Polling is preferable to Short Polling
  • Long Polling can be enabled at the queue level or at the API level using WaitTimeSeconds

Streaming and Kinesis

  • What is Streaming ?

    • Multiple consumers can react to events (messages)

    • Events live in the stream for long periods of time, so complex operations can be applied

    • Real-time

    • Amazon-Kinesis

      • Amazon Kinesis is the AWS fully managed solution for collecting, processing and analyzing streaming data in the cloud

    Kinesis Data Streams

    • Capture,process and store data streams

      • Security:

        • Control access/ authorization using IAM policies
        • Encryption in flight using HTTPS endpoints
        • Encryption at rest using KMS
        • You can implement encryption/decryption of data on client-side (harder)
        • VPC endpoints available for Kinesis to access within VPC
        • Monitor API calls using CLoudTrail

    Kinesis Data Firehose

    • load data streams into AWS data stores
    • Pay for only data that is going through Firehose
    • Supports many data formats, conversions, transformations, compression

    Kinesis Data Streams vs Firehose

    Kinesis Data Streams Kinesis Data Firehose
    - Streaming service for ingest at scale - Load streaming data into S3 /Redshift /OpenSearch / 3rd Party /custom HTTP
    write Custom code (producer/consumer) Fully managed
    Real-time (~200 ms) Near real-time (buffer time min 60 sec)
    Managed scaling (shard splitting / merging) Automatic scaling
    Data storage for 1 to 365 days No data storage
    Supports replay capability Doesn't support Capability

    Kinesis Data Analytics

    • analyze data streams with SQL or Apache Flink

    Kinesis Video Streams

    • Capture, process and store video streams

Pub-Sub and SNS

  • What is Pub / Sub ?

    • Publish-subscribe pattern commonly implemented in messaging systems.

    • In a pub/sub system the sender of messages (publishers) do not send their messages directly to receivers.

    • They instead send their messages to an event bus . The event bus categorizes their messages into groups.

    • The receivers of messages Subscribers subscribe to these groups

    • Whenever new messages appear within their subscription the messages are immediately delivered to them

    • Publisher have no knowledge of who their subscribers are

    • Subscribers do not pull for messages

    • Messages are instead automatically and immediately pushed to subscribers

    • Messages and events are interchangeable terms in pub/sub

    • Use case:

      • A real-time chat system
      • A web-hook system

  • Simple Notification Service

    • It is a highly available, durable, secure, fully managed pub/sub messaging service that enables you to decouple microservices, distributed systems and serverless applications

SQS and SNS - Fan Out Pattern

  • Push once in SNS, receive in all SQS queues that are subscribers
  • Fully decoupled : no data loss
  • SNS - Message Filtering
    • JSON policy used to filter messages sent to SNS topic's subscriptions
    • If a subscription doesn't have a filter policy, it receives every message