Server auth

[pbecotte]

Am very intrigued by prefect (just came across it over the weekend!).

There is one dealbreaker- I don't see any provision for having auth on the backend server. I know that this is something that could apply to the parent company business model so

• is there a way of doing it I just missed?
• if not, would you accept a PR?

[cicdw]

Hi @pbecotte - we do not intend to implement or accept any auth in the open source server at this time; see here for some explanation. From our experience most people who require auth either switch to Prefect Cloud or deploy server within a private network.

[pbecotte]

Unfortunately, neither of those would be an option. Not everyone at my company can be allowed to see all of our data- and we also couldn't store any of it on your servers.

Of course, we would only want the UI accessible to anyone, with agents and the other services running inside a vpc. I suppose I can use an API gateway or something, but it does seem like the kind of missing feature that will be a deal breaker to many.

[cicdw]

Prefect doesn't store sensitive user / customer data or code, and we don't have access to such data. This is an important feature for many of our more security conscious customers. See https://medium.com/the-prefect-blog/the-prefect-hybrid-model-1b70c7fd296 and https://docs.prefect.io/orchestration/faq/dataflow.html for more information.

[pbecotte]

I actually missed that you could disable the log storage- the logs were the part we were concerned about. I will mention that to the team

[MrJack91]

Hey there
there is also a problem, that everyone can run/abort your flows.. :/

[cicdw]

In Prefect Cloud you can set users to read only roles

[mikelogaciuk]

Yes, but there are situations were you can't use cloud version.

Of course I can hide Prefect with ngix reverse proxy but it is far from ideal.

[FlavourDave92]

is there any way to prevent public users to run and abort flows in non cloud prefect (without some nginx stunts)?

[giacomochiarella]

We also have this situation. We cannot use Prefect Cloud and we do not want anyone access to Prefect UI (anyone can delete/start/stop flows). It is a very good tool but this feature is essential to be also competitive on the market.

[zanieb]

Hi all! It is likely that our use of FastAPI in v2 will enable easy extension of the server to include basic authentication (https://fastapi.tiangolo.com/advanced/security/http-basic-auth/)

We are working hard on improving other features of v2 right now and are unlikely to pursue this ourselves, but I'd be really excited to see someone from the community investigate it.

[gaby]

@zanieb Was this ever implemented? I recently came back into Prefect after several years and notice it lacks auth which is a huge issue for self-hosted.

[zanieb]

I don't work at Prefect anymore, but I don't recall hearing of anyone doing it.

[jd-santos]

This is a frustrating thread. I understand prioritizing development effort on the paid product, but basic auth capability is table stakes for a tool designed to orchestrate data. The offered solution of every person in your organization having write access to your pipelines is a not something that any data or IT professional should consider.

It's one thing to reserve capabilities for your paid offering, it's quite another to recommend a huge security and infrastructure vulnerability while winking and pointing to your sales team.

[rkrn]

I have to echo @jd-santos .

We have a fairly large set of ETL pipelines and self-hosted infrastructure (20 PB storage, 3,000 physical CPUs). I presume we'd size up among the top decile of your paid users in usage and be the kind of customer you'd want. Being able to support our internal infrastructure and LDAP authentication would be a bare minimum requirement.

FWIW, we evaluated Prefect thrice over the past 2 years and rejected it each time, chiefly for two reasons:

• Locking down a production-critical service like this on a network level, as recommended in your FAQ, isn't suitable for any serious commercial environment. Every other tool we evaluated has LDAP support.
• Seems like you're giving Dask first-class treatment among options for distributed computing. Dask seems more appropriate for ephemeral jobs like exploratory data analysis, where you're working with a large dataframe over a single interactive session, not long-running production jobs where you're possibly running many smaller jobs in parallel. Most clusters these days will already run Kubernetes/Mesos or a HPC job queuing system like SGE/Condor, and maintaining bloated Dask dependencies over the cluster felt awkward for us.

[ghost]

Also would at least need some basic authentication for all methods to access prefect (API, GUI, whatever else there is).

The linked FAQ mentioned also refers to the option of using a FastAPI Plugin.
I wonder if this would only be possible for the GUI or also for the API without breaking the GUI?

[johnkangw]

Has anyone pursued using a FastAPI Plugin?

[Abraxos]

It feels like adding even HTTP basic auth would make the server way more usable and be very easy. Even on my homelab I consider it unacceptable to deploy something like this without auth.

I imagine there would be plenty of people willing to actually code it up for the open source server.

[gaby]

@Abraxos Agree, I gave up on using Prefect just because of this. Having this fully open makes it useless.

Even If I run NGINX in front, there's no way to limit what users can do to other users workflows.

[maitlandmarshall]

It would be nice if the Prefect cli (deploy) supported authentication challenges, so we could use an AAD oauth proxy to provide auth to the API.

[vkrot-innio]

we've solved the authentication part with oauth proxy. users authenticate against github in the sample below, both api and ui are only available once users have successfully authenticated. in addition basic auth is supported for calling prefect api from other services (to trigger flow runs).
auth-proxy runs in the same k8s namespace where server runs, installed by helm.

oauth-proxy-values.yaml:

namespaceOverride: "prefect-2"

# Oauth client configuration specifics
config:
  # OAuth client ID
  clientID: "..."
  # OAuth client secret
  clientSecret: "..."
  cookieSecret: "..."
  configFile: |-
    provider = "github"
    email_domains = [ "*" ]
    upstreams = [ "http://prefect2-server:4200/"]
    github_org = "myorg-io"
    github_team = "prefect-users"
    scope = "user:email"

htpasswdFile:
  enabled: true
  entries:
    - github:{SHA}...
    - security:...

service:
  type: NodePort

ingress:
  enabled: true
  className: alb
  path: /*
  hosts:
    - prefect2-dev.myorg.io
  annotations:
     .. additional annotations specific to our org

installation:

helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests
helm install -n prefect-2 --values oauth-proxy-values.yaml oauth2-proxy oauth2-proxy/oauth2-proxy

[zzstoatzz]

adding this here for visibility's sake

tldr

we are planning to incorporate a security layer for the open source package

most likely it will be a token-based authentication mechanism along with a password for UI access

#2238 (comment)

[Abraxos]

That's excellent news. Thank you for letting us know, and thank you for planning to add this feature.