Skip to content

kumargautam02/stream_pipeline_public_s3

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

16 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Prerequisites

  • Docker Version - 24.0.6
  • Steps to run this script

  • Download the Zip file or Clone this repository.
  • Go to Project Directory and open a Terminal.
  • On Terminal use command- "docker compose up".
    [Note: make sure you are in the same Directory as of docker-compose.yml file]
  • Open jupyter notebook from docker at post 8888.
  • open a Terminal and write "python stream_processing.py"
  • Project Architecture

    Project Architecture

    Folder Architecture

    Folder Architecture

    [Note: Right now O/P mode is on Console, if you want to get output in files in storage. please include ".option("path", "/output/")" and change format --> "parquet"]

    About

    No description, website, or topics provided.

    Resources

    Stars

    Watchers

    Forks

    Releases

    No releases published

    Packages

    No packages published