What would it take train it?

[HoneyTauOverTwo]

What would it take to build my own, perhaps a minimal proof of concept with smaller data set for training?

I guess it is not feasible to assume a single machine could do it (in less than a couple of months) and distributed training is a must.

And not only hardware-wise, about the code, I know Open AI for a very long time but never actually got into it, from that I mean, is everything really open if I dig deep enough?

About the training data used, is it available somewhere?

This assuming the already trained parameters are not available somewhere for download. Which I guess are not.

And I am sorry if I asked something that is answered somewhere else, I did not dig into it too much, but got pretty excited to get back into AI from this!

[lucidrains]

that's the billion dollar question, quite literally.

just join Laion discord and start collaborating with researchers and other ML folks out there! many people want this answered.

[lucidrains]

https://discord.com/invite/mVcgxMPD7e

[HoneyTauOverTwo]

that's the billion dollar question, quite literally.

just join Laion discord and start collaborating with researchers and other ML folks out there! many people want this answered.

Well, I guess I will then ;D

But the results they present. I assume was done in some ridiculously big cluster of machines... But how big?

[Veldrovive]

I don't think we know exactly, but I believe I saw that it was estimated to be around 125k A100 GPU hours.

[mindplay-dk]

I see you're still actively working on this. Any idea yet how you'll get this trained? I mean, it's all for nothing unless you can actually find the resources to get it trained, isn't it? 😅

[mindplay-dk]

I'm thinking something like crowdsourcing a GPU cloud, like folding-at-home but for GPUs? Plenty of people would be willing to donate their idle GPU hours to this, I think.

Maybe talk to run.ai and see if they're interested in helping out? This could be a great opportunity for them to back one of the most exciting projects in AI right now, without investing tons of money or GPU resources.

[Anmol-B617]

Great idea
But how easy would it be to implement?

[mindplay-dk]

No idea. But with so much work going into this project, without GPU time, what's the point?

Either somebody has to donate the GPU time, or it needs to be crowdsourced somehow - crowdsourcing money for GPU time would be another option, but it's probably more difficult to raise money? And a lot interested developers and interested users probably have mostly idle GPUs sitting around anyway.

[Anmol-B617]

Also, I think the GPU's need to be on a single machine
I highly doubt that distributed training would work if connection between the computer and GPU is unreliable

[mindplay-dk]

Have you heard of this?

https://sites.research.google/trc/about/

[HoneyTauOverTwo]

I did not! Found this in there:
https://huggingface.co/spaces/dalle-mini/dalle-mini
https://github.com/borisdayma/dalle-mini
It seems like you can download the trained models and the code for it! I might experiment with that later.