Install from boot.iso doesn't work for aarch64

[dustymabe]

oz will try to do a direct boot by downloading the kernel and initrd, if that fails then it will try to boot from a boot.iso in the install tree. Booting from a boot.iso doesn't seem to work. We check the boot.iso by looking for a kernel under /isolinux/vmlinuz, which does not appear to exist any longer in the boot.iso files we have been creating.

Here are the contents of an aarch64 boot.iso:

+[dustymabe@media mnt]$ find . 
.
./boot.catalog
./EFI
./EFI/BOOT
./EFI/BOOT/BOOTAA64.EFI
./EFI/BOOT/fonts
./EFI/BOOT/fonts/TRANS.TBL
./EFI/BOOT/fonts/unicode.pf2
./EFI/BOOT/grub.cfg
./EFI/BOOT/grubaa64.efi
./EFI/BOOT/MokManager.efi
./EFI/BOOT/TRANS.TBL
./images
./images/efiboot.img
./images/install.img
./images/pxeboot
./images/pxeboot/initrd.img
./images/pxeboot/TRANS.TBL
./images/pxeboot/vmlinuz
./images/TRANS.TBL
./TRANS.TBL

Here are the contents of an x86_64 boot.iso:

+[dustymabe@media mnt]$ find .
.
./EFI
./EFI/BOOT
./EFI/BOOT/BOOTX64.EFI
./EFI/BOOT/fonts
./EFI/BOOT/fonts/TRANS.TBL
./EFI/BOOT/fonts/unicode.pf2
./EFI/BOOT/grub.cfg
./EFI/BOOT/grubx64.efi
./EFI/BOOT/MokManager.efi
./EFI/BOOT/TRANS.TBL
./images
./images/efiboot.img
./images/install.img
./images/macboot.img
./images/pxeboot
./images/pxeboot/initrd.img
./images/pxeboot/TRANS.TBL
./images/pxeboot/vmlinuz
./images/TRANS.TBL
./isolinux
./isolinux/boot.cat
./isolinux/boot.msg
./isolinux/grub.conf
./isolinux/initrd.img
./isolinux/isolinux.bin
./isolinux/isolinux.cfg
./isolinux/ldlinux.c32
./isolinux/libcom32.c32
./isolinux/libutil.c32
./isolinux/memtest
./isolinux/splash.png
./isolinux/TRANS.TBL
./isolinux/vesamenu.c32
./isolinux/vmlinuz

Here is some snipped content from a log (from this task) where this happened:

dispatching a baseimg builder
Creating directory (/var/tmp/koji/tasks/8375/19668375/output_image) for persistent storage
No SecondaryDispatcher present - Use of secondary factories is not enabled
Created file /var/tmp/koji/tasks/8375/19668375/output_image/084033e9-fc7b-45af-90b5-2bc8ecf936a1.meta
Created file /var/tmp/koji/tasks/8375/19668375/output_image/084033e9-fc7b-45af-90b5-2bc8ecf936a1.body
Getting metadata in class (<class 'imgfac.BaseImage.BaseImage'>) my metadata is (())
Executing metadata in class (<class 'imgfac.BaseImage.BaseImage'>) my metadata is (('identifier', 'data', 'template', 'icicle', 'status_detail', 'status', 'percent_complete', 'parameters', 'properties'))
Saved metadata for image (084033e9-fc7b-45af-90b5-2bc8ecf936a1): {'status': 'NEW', 'identifier': '084033e9-fc7b-45af-90b5-2bc8ecf936a1', 'parameters': {'install_script': '#version=DEVEL\n# Keyboard layouts\nkeyboard \'us\'\n# Root password\nrootpw --iscrypted --lock locked\n# Use network installation\nurl --url="http://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170521.n.0/compose/Server/aarch64/os"\n# System language\nlang en_US.UTF-8\n# Firewall configuration\nfirewall --enabled --service=mdns\nrepo --name="koji-override-0" --baseurl=http://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170521.n.0/compose/Server/aarch64/os\n# Shutdown after installation\nshutdown\n# Network information\nnetwork  --bootproto=dhcp --device=link --activate\n# System timezone\ntimezone US/Eastern\n# Use text mode install\ntext\n# System authorization information\nauth --useshadow --passalgo=sha512\n# Run the Setup Agent on first boot\nfirstboot --reconfig\n# SELinux configuration\nselinux --enforcing\n\n# System services\nservices --enabled="sshd,NetworkManager,chronyd,initial-setup"\n# System bootloader configuration\nbootloader --append="no_timer_check console=tty1 console=ttyS0,115200n8" --location=mbr --timeout=1\nautopart\n# Clear the Master Boot Record\nzerombr\n# Partition clearing information\nclearpart --all --initlabel --disklabel=msdos\n\n%post\n\n# Setup Raspberry Pi firmware\ncp -Pr /usr/share/bcm283x-firmware/* /boot/efi/\nmv -f /boot/efi/config-64.txt /boot/efi/config.txt\ncp -P /usr/share/uboot/rpi_3/u-boot.bin /boot/efi/rpi3-u-boot.bin\n\nreleasever=$(rpm -q --qf \'%{version}\\n\' fedora-release)\nrpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-primary\necho "Packages within this disk image"\nrpm -qa\n# Note that running rpm recreates the rpm db files which aren\'t needed or wanted\nrm -f /var/lib/rpm/__db*\n\n# remove random seed, the newly installed instance should make it\'s own\nrm -f /var/lib/systemd/random-seed\n\ndnf -y remove dracut-config-generic\n\n# Disable network service here, as doing it in the services line\n# fails due to RHBZ #1369794\n/sbin/chkconfig network off\n\n# Remove machine-id on pre generated images\nrm -f /etc/machine-id\ntouch /etc/machine-id\n\n%end\n\n%post\n\n# setup systemd to boot to the right runlevel\necho -n "Setting default runlevel to multiuser text mode"\nrm -f /etc/systemd/system/default.target\nln -s /lib/systemd/system/multi-user.target /etc/systemd/system/default.target\necho .\n\n%end\n\n%packages\n@arm-tools\n@container-management\n@core\n@domain-client\n@hardware-support\n@headless-management\n@server-product\n@standard\nbcm283x-firmware\nchrony\ndracut-config-generic\nglibc-all-langpacks\ninitial-setup\nkernel\nlibcrypt-nss\nrng-tools\nuboot-images-armv8\n-dracut-config-rescue\n-generic-release*\n-initial-setup-gui\n-ipw*\n-iwl*\n-libcrypt\n-trousers-lib\n-uboot-images-armv7\n\n%end\n', 'offline_icicle': True}, 'icicle': None, 'data': '/var/tmp/koji/tasks/8375/19668375/output_image/084033e9-fc7b-45af-90b5-2bc8ecf936a1.body', 'status_detail': {'error': None, 'activity': 'Initializing image prior to Cloud/OS customization'}, 'template': "<template>\n    <name>Fedora-Server-26-20170521.n.0.aarch64</name>\n        <os>\n            <name>Fedora</name>\n            <version>22</version>\n            <arch>aarch64</arch>\n            <install type='url'>\n                <url>http://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170521.n.0/compose/Server/aarch64/os</url>\n            </install>\n        <icicle>\n                <extra_command>rpm -qa --qf '%{NAME},%{VERSION},%{RELEASE},%{ARCH},%{EPOCH},%{SIZE},%{SIGMD5},%{BUILDTIME}\n'</extra_command>\n            </icicle>\n        </os>\n    <description>Fedora-Server-26-20170521.n.0.aarch64 OS</description>\n    <disk>\n        <size>7G</size>\n    </disk>\n</template>\n", 'percent_complete': 0, 'type': 'BaseImage', 'properties': {}}
Attempting to match list target (['Fedora', '22', 'aarch64'])
create_base_image() called for TinMan plugin - creating a BaseImage
libvirt bridge name is virbr0
Libvirt type is kvm
Starting new HTTP connection (1): kojipkgs.fedoraproject.org
"POST /compose/branched/Fedora-26-20170521.n.0/compose/Server/aarch64/os HTTP/1.1" 302 393
Name: factory-build-084033e9-fc7b-45af-90b5-2bc8ecf936a1, UUID: 1a3e330a-73f1-4223-abfb-1a84550667c6
MAC: 52:54:00:d4:10:d8, distro: Fedora
update: 22, arch: aarch64, diskimage: /var/tmp/koji/tasks/8375/19668375/scratch_images/factory-build-084033e9-fc7b-45af-90b5-2bc8ecf936a1.dsk
nicmodel: virtio, clockoffset: utc
mousetype: ps2, disk_bus: virtio, disk_dev: vda
icicletmp: /var/tmp/koji/tasks/8375/19668375/oz_data/icicletmp/factory-build-084033e9-fc7b-45af-90b5-2bc8ecf936a1, listen_port: 58197
Original ISO path: /var/tmp/koji/tasks/8375/19668375/oz_data/isos/Fedora22aarch64-url.iso
Modified ISO cache: /var/tmp/koji/tasks/8375/19668375/oz_data/isos/Fedora22aarch64-url-oz.iso
Output ISO path: /var/tmp/koji/tasks/8375/19668375/scratch_images/factory-build-084033e9-fc7b-45af-90b5-2bc8ecf936a1-url-oz.iso
ISO content path: /var/tmp/koji/tasks/8375/19668375/oz_data/isocontent/factory-build-084033e9-fc7b-45af-90b5-2bc8ecf936a1-url
Cleaning up any old Oz guest
Cleaning up guest named factory-build-084033e9-fc7b-45af-90b5-2bc8ecf936a1
Generating JEOS install media
Grabbing named lock (Fedora-22-aarch64-url)
Got named lock (Fedora-22-aarch64-url)
Installtype is URL, trying to do direct kernel boot
Starting new HTTPS connection (1): kojipkgs.fedoraproject.org
"POST /compose/branched/Fedora-26-20170521.n.0/compose/Server/aarch64/os/.treeinfo HTTP/1.1" 200 1438
Going to write treeinfo to /var/tmp/koji/tasks/8375/19668375/oz_data/icicletmp/factory-build-084033e9-fc7b-45af-90b5-2bc8ecf936a1/treeinfo
Trying to get treeinfo from https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170521.n.0/compose/Server/aarch64/os/.treeinfo
Starting new HTTPS connection (1): kojipkgs.fedoraproject.org
"GET /compose/branched/Fedora-26-20170521.n.0/compose/Server/aarch64/os/.treeinfo HTTP/1.1" 200 1438
Got treeinfo, parsing
Returning kernel images/pxeboot/vmlinuz and initrd images/pxeboot/initrd.img
Attempting to get the lock for /var/tmp/koji/tasks/8375/19668375/oz_data/kernels/Fedora22aarch64-kernel
Got the lock for /var/tmp/koji/tasks/8375/19668375/oz_data/kernels/Fedora22aarch64-kernel
Fetching the original media
Starting new HTTPS connection (1): kojipkgs.fedoraproject.org
"POST /compose/branched/Fedora-26-20170521.n.0/compose/Server/aarch64/os/images/pxeboot/vmlinuz HTTP/1.1" 200 6529431
Fetching the original install media from https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170521.n.0/compose/Server/aarch64/os/images/pxeboot/vmlinuz
Starting new HTTPS connection (1): kojipkgs.fedoraproject.org
"GET /compose/branched/Fedora-26-20170521.n.0/compose/Server/aarch64/os/images/pxeboot/vmlinuz HTTP/1.1" 200 6529431
15252kB of 6376kB
Could not do direct boot, fetching boot.iso instead (the following error message is useful for bug reports, but can be ignored)
Expected to download 6529431 bytes, downloaded 15618560
Generating install media
Attempting to get the lock for /var/tmp/koji/tasks/8375/19668375/oz_data/isos/Fedora22aarch64-url.iso
Got the lock for /var/tmp/koji/tasks/8375/19668375/oz_data/isos/Fedora22aarch64-url.iso
Fetching the original media
Starting new HTTPS connection (1): kojipkgs.fedoraproject.org
"POST /compose/branched/Fedora-26-20170521.n.0/compose/Server/aarch64/os/images/boot.iso HTTP/1.1" 200 450578432
Fetching the original install media from https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170521.n.0/compose/Server/aarch64/os/images/boot.iso
Starting new HTTPS connection (1): kojipkgs.fedoraproject.org
"GET /compose/branched/Fedora-26-20170521.n.0/compose/Server/aarch64/os/images/boot.iso HTTP/1.1" 200 450578432
10240kB of 440018kB
20480kB of 440018kB
30720kB of 440018kB
40960kB of 440018kB
51200kB of 440018kB
61440kB of 440018kB
71680kB of 440018kB
81920kB of 440018kB
92160kB of 440018kB
102400kB of 440018kB
112640kB of 440018kB
122880kB of 440018kB
133120kB of 440018kB
143360kB of 440018kB
153600kB of 440018kB
163840kB of 440018kB
174080kB of 440018kB
184320kB of 440018kB
194560kB of 440018kB
204800kB of 440018kB
215040kB of 440018kB
225280kB of 440018kB
235520kB of 440018kB
245760kB of 440018kB
256000kB of 440018kB
266240kB of 440018kB
276480kB of 440018kB
286720kB of 440018kB
296960kB of 440018kB
307200kB of 440018kB
317440kB of 440018kB
327680kB of 440018kB
337920kB of 440018kB
348160kB of 440018kB
358400kB of 440018kB
368640kB of 440018kB
378880kB of 440018kB
389120kB of 440018kB
399360kB of 440018kB
409600kB of 440018kB
419840kB of 440018kB
430080kB of 440018kB
440018kB of 440018kB
Copying ISO contents for modification
Setting up guestfs handle for factory-build-084033e9-fc7b-45af-90b5-2bc8ecf936a1
Adding ISO image /var/tmp/koji/tasks/8375/19668375/oz_data/isos/Fedora22aarch64-url.iso
Launching guestfs
Mounting ISO
Checking if there is enough space on the filesystem
Extracting ISO contents
Cleaning up old ISO data
Releasing named lock (Fedora-22-aarch64-url)
Removed builder from BuildDispatcher on notification from image 084033e9-fc7b-45af-90b5-2bc8ecf936a1: FAILED
Getting metadata in class (<class 'imgfac.BaseImage.BaseImage'>) my metadata is (())
Executing metadata in class (<class 'imgfac.BaseImage.BaseImage'>) my metadata is (('identifier', 'data', 'template', 'icicle', 'status_detail', 'status', 'percent_complete', 'parameters', 'properties'))
Saved metadata for image (084033e9-fc7b-45af-90b5-2bc8ecf936a1): {'status': 'FAILED', 'identifier': '084033e9-fc7b-45af-90b5-2bc8ecf936a1', 'parameters': {'install_script': '#version=DEVEL\n# Keyboard layouts\nkeyboard \'us\'\n# Root password\nrootpw --iscrypted --lock locked\n# Use network installation\nurl --url="http://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170521.n.0/compose/Server/aarch64/os"\n# System language\nlang en_US.UTF-8\n# Firewall configuration\nfirewall --enabled --service=mdns\nrepo --name="koji-override-0" --baseurl=http://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170521.n.0/compose/Server/aarch64/os\n# Shutdown after installation\nshutdown\n# Network information\nnetwork  --bootproto=dhcp --device=link --activate\n# System timezone\ntimezone US/Eastern\n# Use text mode install\ntext\n# System authorization information\nauth --useshadow --passalgo=sha512\n# Run the Setup Agent on first boot\nfirstboot --reconfig\n# SELinux configuration\nselinux --enforcing\n\n# System services\nservices --enabled="sshd,NetworkManager,chronyd,initial-setup"\n# System bootloader configuration\nbootloader --append="no_timer_check console=tty1 console=ttyS0,115200n8" --location=mbr --timeout=1\nautopart\n# Clear the Master Boot Record\nzerombr\n# Partition clearing information\nclearpart --all --initlabel --disklabel=msdos\n\n%post\n\n# Setup Raspberry Pi firmware\ncp -Pr /usr/share/bcm283x-firmware/* /boot/efi/\nmv -f /boot/efi/config-64.txt /boot/efi/config.txt\ncp -P /usr/share/uboot/rpi_3/u-boot.bin /boot/efi/rpi3-u-boot.bin\n\nreleasever=$(rpm -q --qf \'%{version}\\n\' fedora-release)\nrpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-primary\necho "Packages within this disk image"\nrpm -qa\n# Note that running rpm recreates the rpm db files which aren\'t needed or wanted\nrm -f /var/lib/rpm/__db*\n\n# remove random seed, the newly installed instance should make it\'s own\nrm -f /var/lib/systemd/random-seed\n\ndnf -y remove dracut-config-generic\n\n# Disable network service here, as doing it in the services line\n# fails due to RHBZ #1369794\n/sbin/chkconfig network off\n\n# Remove machine-id on pre generated images\nrm -f /etc/machine-id\ntouch /etc/machine-id\n\n%end\n\n%post\n\n# setup systemd to boot to the right runlevel\necho -n "Setting default runlevel to multiuser text mode"\nrm -f /etc/systemd/system/default.target\nln -s /lib/systemd/system/multi-user.target /etc/systemd/system/default.target\necho .\n\n%end\n\n%packages\n@arm-tools\n@container-management\n@core\n@domain-client\n@hardware-support\n@headless-management\n@server-product\n@standard\nbcm283x-firmware\nchrony\ndracut-config-generic\nglibc-all-langpacks\ninitial-setup\nkernel\nlibcrypt-nss\nrng-tools\nuboot-images-armv8\n-dracut-config-rescue\n-generic-release*\n-initial-setup-gui\n-ipw*\n-iwl*\n-libcrypt\n-trousers-lib\n-uboot-images-armv7\n\n%end\n', 'offline_icicle': True}, 'icicle': None, 'data': '/var/tmp/koji/tasks/8375/19668375/output_image/084033e9-fc7b-45af-90b5-2bc8ecf936a1.body', 'status_detail': {'error': 'Fedora/Red Hat installs can only be done using a boot.iso (netinst) or DVD image (LiveCDs are not supported)', 'activity': 'Base Image build failed with exception.'}, 'template': "<template>\n    <name>Fedora-Server-26-20170521.n.0.aarch64</name>\n        <os>\n            <name>Fedora</name>\n            <version>22</version>\n            <arch>aarch64</arch>\n            <install type='url'>\n                <url>http://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170521.n.0/compose/Server/aarch64/os</url>\n            </install>\n        <icicle>\n                <extra_command>rpm -qa --qf '%{NAME},%{VERSION},%{RELEASE},%{ARCH},%{EPOCH},%{SIZE},%{SIGMD5},%{BUILDTIME}\n'</extra_command>\n            </icicle>\n        </os>\n    <description>Fedora-Server-26-20170521.n.0.aarch64 OS</description>\n    <disk>\n        <size>7G</size>\n    </disk>\n</template>\n", 'percent_complete': 0, 'type': 'BaseImage', 'properties': {}}
Exception encountered in _build_image_from_template thread
Fedora/Red Hat installs can only be done using a boot.iso (netinst) or DVD image (LiveCDs are not supported)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/imgfac/Builder.py", line 132, in _build_image_from_template
    self.os_plugin.create_base_image(self, template, parameters)
  File "/usr/lib/python2.7/site-packages/imagefactory_plugins/TinMan/TinMan.py", line 327, in create_base_image
    self.threadsafe_generate_install_media(self.guest)
  File "/usr/lib/python2.7/site-packages/imagefactory_plugins/TinMan/TinMan.py", line 431, in threadsafe_generate_install_media
    guest.generate_install_media(force_download=False)
  File "/usr/lib/python2.7/site-packages/oz/RedHat.py", line 684, in generate_install_media
    customize_or_icicle)
  File "/usr/lib/python2.7/site-packages/oz/Guest.py", line 1846, in _iso_generate_install_media
    self._check_iso_tree(customize_or_icicle)
  File "/usr/lib/python2.7/site-packages/oz/RedHat.py", line 102, in _check_iso_tree
    raise oz.OzException.OzException("Fedora/Red Hat installs can only be done using a boot.iso (netinst) or DVD image (LiveCDs are not supported)")
OzException: Fedora/Red Hat installs can only be done using a boot.iso (netinst) or DVD image (LiveCDs are not supported)
ABORT called in TinMan plugin
Domain not found: no domain with matching name 'factory-build-084033e9-fc7b-45af-90b5-2bc8ecf936a1'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/imagefactory_plugins/TinMan/TinMan.py", line 245, in abort
    guest_dom = self.guest.libvirt_conn.lookupByName(self.tdlobj.name)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 4227, in lookupByName
    if ret is None:raise libvirtError('virDomainLookupByName() failed', conn=self)
libvirtError: Domain not found: no domain with matching name 'factory-build-084033e9-fc7b-45af-90b5-2bc8ecf936a1'
No Oz VM found with name (factory-build-084033e9-fc7b-45af-90b5-2bc8ecf936a1) - nothing to do
This likely means the local VM has already been destroyed or never started
Resetting dropped connection: koji.fedoraproject.org
"POST /kojihub?session-id=29752028&session-key=3801-nu41rNcUviwcVkcusfw&callnum=8 HTTP/1.1" 200 114

[clalancette]

Hm, interesting. So x86_64 does have isolinux/vmlinuz, but aarch64 does not. Any thoughts on where aarch64 gets the compressed kernel image from by default? Is it ./images/pxeboot/vmlinuz , or is it somehow embedded somewhere else?

[dustymabe]

Is it ./images/pxeboot/vmlinuz

i believe so. at least that is what the treeinfo is telling us to use:

[images-aarch64]
boot.iso = images/boot.iso
efiboot.img = images/efiboot.img
initrd = images/pxeboot/initrd.img
kernel = images/pxeboot/vmlinuz

which means that's what we are grabbing for the direct boot case anyways