Support running on Azure

[rsignell]

What would be the best way to run Beam pipelines on Azure?
I think @yuvipanda would say "Flink!".
If so, does this look like the best resource? https://github.com/microsoft/flink-on-azure

[rsignell]

@TomAugspurger, I thought you said something about running Beam on Azure somewhere, but can't find it (and neither can Bing!)

[rsignell]

Actually, I decided to just ask Bing if I could run Apache Beam pipelines on Azure using Apache Flink. Here's the response:

Yes, you can run Apache Beam pipelines on Azure using Apache Flink. There are different ways to do this, depending on your needs and preferences. Here are some possible options:

• You can use Azure Data Factory Managed Airflow to orchestrate your Apache Flink jobs. This allows you to create and manage workflows that run on a fully managed Apache Airflow service in Azure. You can use the Flink operator to submit your Beam pipelines to a Flink cluster running on Azure HDInsight or Azure Kubernetes Service³.
• You can use the Flink Kubernetes Operator to run your Hop pipelines on a Flink cluster that is automatically started on Kubernetes. Hop is a data orchestration tool that supports Apache Beam as a pipeline engine. You can use Hop to design and execute your Beam pipelines using a graphical user interface or a code editor⁴.
• You can use Azure Event Hubs for Apache Kafka to stream data from and to your Apache Flink jobs. Event Hubs is a fully managed, real-time data ingestion service that is compatible with the Kafka protocol. You can use the Flink Kafka connector to read and write data from Event Hubs as if it were a Kafka topic⁵.

I hope this helps you to run your Apache Beam pipelines on Azure using Apache Flink. If you have any more questions, feel free to ask me. 😊.

Source: Conversation with Bing, 12/4/2023
(1) Apache Flink® job orchestration using Azure Data Factory Managed Airflow. https://learn.microsoft.com/en-us/azure/hdinsight-aks/flink/flink-job-orchestration.
(2) Running a Hop pipeline using the Flink Kubernetes Operator. https://hop.apache.org/manual/latest/pipeline/beam/flink-k8s-operator-running-hop-pipeline.html.
(3) Use Apache Flink with Azure Event Hubs for Apache Kafka. https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-kafka-flink-tutorial.
(4) Running the Apache Beam samples With Apache Flink. https://hop.apache.org/manual/latest/pipeline/beam/beam-samples-flink.html.
(5) Using the Apache Flink Runner - The Apache Software Foundation. https://beam.apache.org/documentation/runners/flink/.
(6) en.wikipedia.org. https://en.wikipedia.org/wiki/Apache_Flink.

[TomAugspurger]

https://github.com/TomAugspurger/pangeo-forge-cloud-federation/tree/azure/terraform/azure (from https://discourse.pangeo.io/t/pangeo-forge-bakeries/3630/13?u=tomaugspurger). I haven't actually gotten to the Beam part, just Flink on AKS.

…

________________________________ From: rsignell ***@***.***> Sent: Monday, December 4, 2023 9:54 AM To: pangeo-forge/pangeo-forge-runner ***@***.***> Cc: Mention ***@***.***> Subject: Re: [pangeo-forge/pangeo-forge-runner] Support running on Azure (Issue #157) @TomAugspurger<https://github.com/TomAugspurger>, I thought you said something about running Beam on Azure somewhere, but can't find it (and neither can Bing!) — Reply to this email directly, view it on GitHub<#157 (comment)> or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAKAOIVIK5FTPXVNPTBYMZLYHXW2XBFKMF2HI4TJMJ2XIZLTSOBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOJIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFVE2TCOBWGM3DEMJXQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSGAZDIMJXHAZTAMVHORZGSZ3HMVZKMY3SMVQXIZI>. You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.