Implement Proxmox VE (proxmoxve) Support

[b-]

Implement Proxmox VE (proxmoxve) Support

This issue is to track support for the Proxmox VE hypervisor, which is a free and open source KVM+Qemu platform.

The official reserved platform ID is proxmoxve, and the implementation is probably going to be mostly ripped from the ibmcloud provider to start with, as it uses the same standard for its own cloud-init data.

The format for this issue is taken from the Implementing a new emerging platform document.

During Development

Create PRs addressing the following:

• Ignition (PR)
  • Add userdata fetch
  • If the platform supports it (unlikely), add userdata deletion
• Afterburn (PR)
  • (Cloud Only) Add relevant attributes
  • (Cloud Only) Add SSH key support if available
  • (Cloud Only) Add hostname support if available
  • (Cloud Only) Add check-in if needed (unlikely)
• fedora-coreos-docs (example PR)
  • Add a provisioning-<platform>.adoc that walks through how to setup the new platform
  • Add an entry in the modules/ROOT/nav.adoc that points to new documentation
• (Optional but recommended) Add support for the platform to kola to simplify testing
• Create or ask for a new upstream releases for:
  • Ignition
  • Afterburn
• Wait for new images with updated Ignition and Afterburn to reach stable then
  merge documentation with guestfish commands:
  • fedora-coreos-docs

At Release

There are no "At Release" steps as we do not produce new boot images for
emerging platforms/

[b-]

Related issues: #736 and coreos/ignition#1790

[b-]

Afterburn implementation will have to be completely different from ibmcloud: they use a weird MIME-encoding format for SSH keys and such that seems pretty unique to them.

Since Ignition can provide ssh keys (and Proxmox doesn't actually have a UI for any of my proposed Ignition support), maybe we can ignore Afterburn for this platform? If using the custom cloud-init support as I am suggesting then I don't think any other/extra metadata (such as the hostname) will available to the platform.

If it means anything, my personal workflow for CoreOS on Proxmox so far is the following:

1. Deploy Fedora CoreOS VM using the ibmcloud image (or a proxmoxve-stamped image after providers/proxmoxve: Add Proxmox VE provider ignition#1790 is merged) and a very simple Ignition file that doesn't actually create any users or set any SSH keys or passwords, but does install the cloud-init package
2. Power off the VM and remove the ignition disk
3. Set the VM as a template

Then, if I clone the VM template in Proxmox, it's ready for a Proxmox-style cloud-init provisioning data to install any necessary packages, create users, set SSH keys, and any other provisioning. This can actually be set via the Proxmox GUI.

A negative side effect of doing the above and not providing any authentication information via Ignition or Afterburn (and then essentially replacing Afterburn with cloud-init) is that the VM will warn you that Ignition and Afterburn weren't provided with any provisioning data.

Another negative side effect is that, of course, Ignition is the RedHat/Fedora/CoreOS/Flatcar way, not cloud-init. But it works -- I've deployed CoreOS (as well as ublue-os/ucore) this way for simple homelab workloads, and they're solid as ever

[jlebon]

Re. Afterburn support, there's a pending PR for it already in coreos/afterburn#1023, so we can just link to that from the checklist. It hasn't been reviewed in depth yet, but I don't see anything about having to decode SSH keys there.

Re. the cloud-init UX, we definitely want to avoid users jumping through those hoops. Ignition and Afterburn should be able to read from the same metadata sources that cloud-init does to provision the node (i.e. files in attached config drive). From my understanding of #736, the Proxmox UI doesn't make it easy to pass !cloud-init userdata, but it is possible on the command line (see #736 (comment)).

[b-]

Indeed, Proxmox only allows you to pass arbitrary data using the command line.

coreos/afterburn#1023 is reading Proxmox-style cloud-init data anyway, rather than Ignition data shoehorned into place as I was suggesting. I think that this would make a much better user experience, at the expense of being more limited than arbitrary Ignition data.

The only reason I didn't suggest doing this was because I figured that interpreting the cloud-init data sources might be out-of-scope for the Ignition project, and of course it's more limiting, but I do think it would provide a better UX.

Perhaps both could be supported? In other words, if the first line of user-data is #cloud-config, interpret the data source as cloud-init yaml. Otherwise, try to interpret it as Ignition json data.

[jdoss]

I am looking to start using FCOS on a test Proxmox VM host. Is there anything I can do to help out with this endeavor? It looks like coreos/ignition#1790 needs testing and review. @dustymabe if you or @jlebon can point me in the right direction for doing the testing, I offer myself as tribute. I should have my test Proxmox server online in a few weeks.

[jlebon]

@b- I presume #736 (comment) is still the latest state of things? I.e. that there is no sane way for users to provide arbitrary user-data to the instance without jumping through various hoops?

So obviously, ideally we would enhance Proxmox to allow that more easily.

I'm confused though, putting aside the CLI/API workarounds, if Proxmox fully owns the generated cloud-init, how do users provide their own cloud-init to do whatever else they want? Or is there a section of the web UI where they can type in extra cloud-init configs that get merged in? (Or is the expectation that you basically always just follow up with some config mgmt tool?)

[jlebon]

So basically if I'm understanding this right:

• it's very difficult currently to set the Ignition config as the user-data
• by default, Proxmox will set a cloud-init user-data and use that as its mechanism for SSH keys and hostname configuration

So Proxmox essentially uses cloud-init user-data as its platform metadata mechanism. Which... isn't great but we can live with I guess.

Perhaps both could be supported? In other words, if the first line of user-data is #cloud-config, interpret the data source as cloud-init yaml. Otherwise, try to interpret it as Ignition json data.

I'm kinda thinking actually that we just keep this in Afterburn and don't add any Ignition support for this at all for now if we don't really expect users to jump through the hoops necessary to set an Ignition config. (Though if there's an RFE somewhere in progress to have Proxmox make it easier to pass your own user-data, that changes things.)

Afterburn would just parse the strict subset of the cloud-init YAML schema that Proxmox uses. If it's somehow possible for users to pass in some custom cloud-init snippets that get merged in, then we should error out so we don't pretend to understand all of cloud-init's schema.

[dustymabe]

maybe we could hook into the cicustom step?

https://pve.proxmox.com/wiki/Cloud-Init_Support#_custom_cloud_init_configuration

[b-]

So basically if I'm understanding this right:

• it's very difficult currently to set the Ignition config as the user-data

• by default, Proxmox will set a cloud-init user-data and use that as its mechanism for SSH keys and hostname configuration

So Proxmox essentially uses cloud-init user-data as its platform metadata mechanism. Which... isn't great but we can live with I guess.

That is correct.

I'm kinda thinking actually that we just keep this in Afterburn and don't add any Ignition support for this at all for now if we don't really expect users to jump through the hoops necessary to set an Ignition config. (Though if there's an RFE somewhere in progress to have Proxmox make it easier to pass your own user-data, that changes things.)

Afterburn would just parse the strict subset of the cloud-init YAML schema that Proxmox uses. If it's somehow possible for users to pass in some custom cloud-init snippets that get merged in, then we should error out so we don't pretend to understand all of cloud-init's schema.

I like this idea a lot. I think this probably would be the simplest path forward to making Proxmox-flavored CoreOS images that anybody could use.

[dustymabe]

also of interest - some discussion of support for fw_cfg upstream: https://bugzilla.proxmox.com/show_bug.cgi?id=4068#c5

[b-]

maybe we could hook into the cicustom step?

https://pve.proxmox.com/wiki/Cloud-Init_Support#_custom_cloud_init_configuration

This is what I was suggesting as an additional/alternative to parsing the Proxmox-flavored subset of the cloud-init yaml.

The nice things about this approach is that it could also work on bare metal, as well as any other hypervisor, if one just makes a disk with the right volume name, format, and a single file or two.

Perhaps maybe a separate "baremetal" target could be made that would accept an ignition file passed via cicustom or on a USB stick/CD drive with the proper volume name?

EDIT: (because I don't want to spam this topic)

also of interest - some discussion of support for fw_cfg upstream: https://bugzilla.proxmox.com/show_bug.cgi?id=4068#c5

I mean this with the utmost respect to the Proxmox team, but that issue and many other "nice to haves" on their bugzilla never really make much progress. I don't know if it's worth it to wait for them…

[b-]

@b- I presume #736 (comment) is still the latest state of things? I.e. that there is no sane way for users to provide arbitrary user-data to the instance without jumping through various hoops?

That is correct

I'm confused though, putting aside the CLI/API workarounds, if Proxmox fully owns the generated cloud-init, how do users provide their own cloud-init to do whatever else they want? Or is there a section of the web UI where they can type in extra cloud-init configs that get merged in? (Or is the expectation that you basically always just follow up with some config mgmt tool?)

Custom cloud-init metadata can be provided via one of two ways:

The first way, which is the cicustom way, is that a "snippet"^[1] can be created on/uploaded to the server's storage (or a mountpoint on it). Then a CLI/API command (-cicustom on the CLI) can be used to generate the cloudinit drive and mount it on the VM.

The second way is to manually create a cloud-init drive.

To be clear (and I'm sorry if I keep repeating myself), the way Proxmox actually supplies the cloud-init metadata is by creating an ISO9660 image with a given volume name and a few files with specific filenames on it.

I only mention this to make it clear why I think that a separate target format ("baremetal") supporting a custom ignition config provided that way as well would be nice:

You could manually (or with a simple script) build an image or physical disk following that format, and then provide that to either any generic VM or baremetal PC, without needing to also supply boot args.

[1]: Proxmox terminology for text file like an ignition file, or a shell script ("hook script") to be run on VM operations

[abuisine]

Hi everyone,
If I may (and I may not, just tell me :)), we've been playing with proxmoxve + afterburn / ignition oses for quite some time now.
There are several approches possible with various level of difficulty :

First, I would disregard anything based on hook scripts. This is our current setup in order to provide ignition configuration to the guest OS on proxmoxve, it is quite messy, the surface of modification to be applied on the proxmoxve cluster is non trivial, and updating scripts is cumbersome => This is why we started to work with @arcln on afterburn and ignition.

The afterburn modification (for instance coreos/afterburn#1023) which allows it to parse cloud-init metadata generated by Proxmoxve is from my standpoint the minimal viable modification necessary for smooth operations of afterburn based OSes. It does not specifically respect the intent of ignition based OSes, but at least render them usable on a proxmoxve setup.

Then, should you want to benefit from ignition (and we rather do !), it seems that the most logical/reasonable path is :

• use proxmoxve primitives (available through the API) to override user-data and inject ignition configuration
• modifiy ignition to look for such configuration at the right place : user data file within the iso generated by proxmoxve with a fixed cidata filesystem id
  An example for such modification is available here Proxmox VE provider ignition#1844

From our standpoint we would rather have the two PR validated / corrected and then merged as it would allow full control of the OS through proxmoxVE API (with proper terraform support btw), as well as basic OS configuration through GUI.

Happy to discuss / challenge this point of view ...

[jlebon]

OK, so it sounds like the --cicustom approach is quite advanced/off the beaten path and requires additional Proxmox privileges.

How does this compare to the "attach your own ISO" approach? Does that require any special privileges? Is it part of the regular UX of a Proxmox user? (I.e. how common is it to have to resort to generating your own ISO to attach to your VM?)

I think it'd be nice to support passing in Ignition configs, but it sounds to me like we need to optimize for the Afterburn cloud-init case instead to better match expectactions (i.e. that'd be what we primarily document, and the custom Ignition case would be an advanced section).

[abuisine]

--cicustom is the option flag for the qm binary accessible from a shell on a proxmoxve hypervisor, but you got a similar option through the JSON/HTTP API.

The main problems related to building a custom iso (for instance through terraform on a station) are the upload to the proxmoxve cluster as well as the storage medium configured at the cluster level to receive such file.

Should you want to stick with custom iso, it would require:

• configure a replicated storage backend (the iso is attached to the vm, and the vm can move from one hypervisor to another, so the iso must be accessible from any hypervisor)
• upload the iso, the preferred method is usually ssh/scp to one hypervisor on the replicated storage

Or you can leverage from proxmoxve snippets:

• snippets are, by default, replicated within a proxmoxve cluster
• big files are not supported
• upload of custom configuration file as a snippet can be done either through the HTTP based API or through ssh/scp or through the GUI
• through the cicustom configuration, proxmoxve handle by himself the iso generation

I am very much in favor of the later option for two reasons:

1. ease of use (no preliminary complex configuration)
2. it seems also more inline with the proxmoxve way of dealing with specific configurations (snippets)

but I admit that it requires a modification of ignition in order to look at the right location (as proposed in coreos/ignition#1844)

[b-]

@abuisine are you certain that snippets can be uploaded via the API? That's the snag I got caught on when working on this a few months ago — I could not find an API call to actually upload or create a snippet. I looked pretty thoroughly… and I'm 100% certain there's no GUI method to manage snippets.

(Additionally, snippet storage isn't configured by default last time I checked either, so I also am not sure what you mean by "replicated by default")

If there is such an API call (and by all means there should be one), I absolutely agree that snippets + cicustom are the right way for this. But as it stands it seems to be missing parts.

To be clear, the only reason I suggest actually making a custom iso is because driving the cicustom methods via API seems impossible as it stands. I don't think a custom ISO is ideal, but rather just a relatively reasonable way to work around the lack of a complete API.

[abuisine]

Hi, let me double check all this and get back to you.

[abuisine]

Hi @b-
Reporting as agreed : you are right and I am wrong 🤠
Upload of files was solved a couple of years ago but never implemented within the API. I guess we are all waiting for it to happen (you, me, the community of proxmoxve users).

Regarding the replication of snippets, we usually configure their path on /etc/pve/snippets which benefits from the replication configured by default on /etc/pve. Just to be clear you are right again : snippets are not replicated by default, you need to setup it as stated for it to work.

So in light of this, I guess the arguments to justify an ignition modification are minimal:
I still do agree that upload of custom iso + attachement to VM will work as expected.

I think we still prefer (at Enix, the company I am working at with @arcln) to upload a json file as a snippet and let proxmoxve generate the iso for us. The argument behind this is more related our usage of terraform. We would rather generate an ignition through a terraform provider like https://github.com/community-terraform-providers/terraform-provider-ignition, rather than an iso image ... because it requires some extra binaries like mkisofs and that it is cumbersome to ensure of its availabilty on linux, mac, win...

So in short the question would be :
"Is it ok to modify ignition in order to make it support a proxmoxve cicustom based configuration" ?
If not, it is ok, we would stick to specific iso upload.

[b-]

For the reasons mentioned above, I do think that supporting the Proxmox subset of cloud-init would still be more useful to more people than ignition-as-cicustom data. Ideally both would be supported (possibly in their own separate flavors), but _It Would Be Nice_™ to be able to spin up a CoreOS VM on Proxmox without messing around with anything, too…

[abuisine]

Can you confirm the following:

Support of the Promox subset of cloud-init only requires an afterburn modification.
An example is available here coreos/afterburn#1023

Support of ignition-as-cicustom data requires an ignition modification and is probably less useful.
Two PRs today https://github.com/coreos/ignition/pull/1790/files and coreos/ignition#1844.
I propose to dismiss the one we created and to contribute to yours ? Does it make sense ?

For the CoreOS VM on Proxmox without messing around with anything ... apart from DHCP I guess we are far from the goal :P

[jlebon]

Can someone provide an example of a Proxmox-provided cloud-init config? Is e.g. https://github.com/enix/afterburn/blob/proxmoxve/tests/fixtures/proxmoxve/dhcp/user-data (from coreos/afterburn#1023) an accurate example? (Weird that they would provide both user: and users:; the latter I think overriding the former.)

The thing is there are keys in there that Afterburn is not really designed to handle and I don't think we should add support for. Basically of the user-data we should just support hostname/fqin and ssh_authorized_keys and that's it. And then document this clearly.

Re. network-config, unfortunately Afterburn's network configuration story hasn't had a lot of effort put into it due to the fact that it outputs systemd-networkd units and FCOS doesn't ship that. It'd be great to add support for outputting nmstate config files though.

Anyway, short-term the problem there is that we'll simply not be able to apply the network config on FCOS, which means that... yeah just DHCP will work. We should document that limitation too.

I think for completeness we indeed should also add support in Ignition for advanced users. In that case, (1) Ignition should no-op if the user-data starts with #cloud-config (otherwise assume it's Ignition and error out as usual if it's not), and (2) Afterburn should no-op if the user-data doesn't start with #cloud-config.

[arcln]

The config you'll find in Afterburn tests (the one you linked) was directly copied from Proxmox VE and is accurate.

For the network-config, it's true that even after implementing support in Afterburn, it wasn't working and we had to add our own systemd unit to invoke afterburn --network-units.

Finally, our Afterburn PR looks for #cloud-config header and will ignore the file if it doesn't start with that. We only miss this behaviour in Ignition, and we should be good to go.

[b-]

(Weird that they would provide both user: and users:; the latter I think overriding the former.)

Actually the other way around, at least on an Ubuntu 22.04 VM I provisioned on my home server. This behavior is definitely weird and confusing, but it sort of makes sense?

I set a username in the Proxmox Web UI (e.g., dummy-user), and such a user was created that belongs to the sudo group with the attached SSH key.

If you leave the User field in the web UI empty, it shows Default in its place and the only difference in the actual cloud-init user-data is /user:/d, i.e., there's no user: dummy-user line in the user-data file.

I haven't tested it specifically in a while, but what's supposed to happen in such a situation (with no user-provided username) is that a distro-specific default username is chosen (e.g., ubuntu or core).

But if you do specify a username (e.g., dummy-user), then such a default user is not created. So on the aforementioned Ubuntu 22.04 VM that was provisioned with a username specified, I don't have a ubuntu user in /etc/passwd.

[jlebon]

OK gotcha. In that case, we can't support that either. I think we can basically say that we'll always create the core user regardless of the set User in the web UI.

Ideally, we could add metadata to the image somehow that Proxmox could query to know what is/isn't supported and reflect that in the web UI. But ideally ideally, Proxmox would have first-class support for Ignition and/or custom user-data. :)

[b-]

I think we can basically say that we'll always create the core user regardless of the set User in the web UI.

I think this is reasonable, as long as setting the User field to core (or empty) will cause attached SSH keys and passwd hashes to be applied to the core user.

Ideally, we could add metadata to the image somehow that Proxmox could query to know what is/isn't supported and reflect that in the web UI.

I imagine this might be possible in "the future™," at least because Proxmox's LXC support relies on a similar metadata to pick some distro-specific behavior.

But ideally ideally, Proxmox would have first-class support for Ignition and/or custom user-data. :)

Hard agree. Honestly, I wish they could at least do what Oracle Cloud's web UI does, which is a big text box whose contents becomes the user-data file. Oracle's Ignition and afterburn support are so simple, but also so close to (yet so far from) what Proxmox does… 😕

@jlebon would it be useful for me to attach a network-config with static settings?

[jlebon]

I think we can basically say that we'll always create the core user regardless of the set User in the web UI.

I think this is reasonable, as long as setting the User field to core (or empty) will cause attached SSH keys and passwd hashes to be applied to the core user.

Yeah, ignoring it will give that behaviour. (We always create the core user by default and you'd need to be able to specify an Ignition config in order to override that.)

@jlebon would it be useful for me to attach a network-config with static settings?

I think there's an example config in the Afterburn PR tests: https://github.com/enix/afterburn/blob/proxmoxve/tests/fixtures/proxmoxve/static/network-config

[jlebon]

Ignition support was merged in coreos/ignition#1910. So now we need a new Ignition release with that in FCOS, but it's not too early for someone to start working on the docs part.