Proposal:

• 00* - 1100_prepare-build-machine
• 01* - Qubes only, nothing Whonix specific
• 02* - Qubes only, nothing Whonix specific
• (virtual) 03* - Qubes only, nothing Whonix specific
• 04* - take 02_install_groups_pre.sh and rename it to 04_install_qubes_post.sh (all inside chroot: creates packages, installs packages, run the chroot scripts)
• 09* - 09_cleanup_post.sh

Would that be better?

Still non-ideal. Ideally, packages would be build outside the chroot so the build dependencies do not get installed inside the image. Combining the two build systems in a non-hacky, non-complex way seems difficult. Would it be better if qubes-builder would build all the Whonix packages? I wonder if that could be done without a giant components.conf listing all the packages git repos. And without adding a Makefile.builder to each and every Whonix package.

Perhaps makefile-generic-packages (genmkfile) could be extended to implement all make targets, that qubes-builder requires.

Proposal:

• 00* - 1100_prepare-build-machine https://github.com/Whonix/Whonix/blob/master/build-steps.d/1100_prepare-build-machine
• 01* - Qubes only, nothing Whonix specific
• 02* - Qubes only, nothing Whonix specific
• (virtual) 03* - Qubes only, nothing Whonix specific
• 04* - take 02_install_groups_pre.sh https://github.com/adrelanos/qubes-template-whonix/blob/master/whonix-gateway/02_install_groups_pre.sh and rename it to 04_install_qubes_post.sh (all inside chroot: creates packages https://github.com/Whonix/Whonix/blob/master/build-steps.d/1200_create-debian-packages, installs packages https://github.com/Whonix/Whonix/blob/master/build-steps.d/1700_install-packages, run https://github.com/Whonix/Whonix/blob/master/build-steps.d/2300_run-chroot-scripts-post-d the chroot scripts https://www.whonix.org/wiki/Dev/Source_Code_Intro#Chroot_Scripts)
• 09* - 09_cleanup_post.sh https://github.com/marmarek/qubes-template-whonix/blob/master/whonix-gateway/09_cleanup_post.sh

Would that be better?

Yes, exactly.

Still non-ideal. Ideally, packages would be build outside the chroot
so the build dependencies do not get installed inside the image.
Combining the two build systems in a non-hacky, non-complex way seems
difficult. Would it be better if qubes-builder would build all the
Whonix packages? I wonder if that could be done without a giant
components.conf listing all the packages git repos. And without
adding a Makefile.builder to each and every Whonix package.

Whonix uses submodules for all the components, right? So it can be
considered a single component to qubes-builder, which produces a lot of
packages. Are some Whonix-specific steps required to build the
components, or generic dpkg-buildpackage is enough? In case of the
later, it would be enough to place single Makefile.builder in main
Whonix repository which lists all the components in DEBIAN_BUILD_DIRS
(list can be easily generated based on submodules list, or some script
output). Note that the order is important to satisfy build depends. If
something more than dpkg-buildpackage is required, it may be enough to
place it in SOURCE_COPY_IN step, or some more flexible mechanism
should be designed.

In T402#6662, @marmarek wrote:

Whonix uses submodules for all the components, right? So it can be
considered a single component to qubes-builder, which produces a lot of
packages. Are some Whonix-specific steps required to build the
components, or generic dpkg-buildpackage is enough? In case of the
later, it would be enough to place single Makefile.builder in main
Whonix repository which lists all the components in DEBIAN_BUILD_DIRS
(list can be easily generated based on submodules list, or some script
output). Note that the order is important to satisfy build depends. If
something more than dpkg-buildpackage is required, it may be enough to
place it in SOURCE_COPY_IN step, or some more flexible mechanism
should be designed.

This would mean that it isn't possible to build a single Whonix package, only all of them. If that's a big issue, we could either introduce some qubes-builder mechanism to build a single package of multi-package component (IMO preferred, useful not only here), or consider each Whonix package as a separate components (as you've noted, not ideal).

Whonix uses submodules for all the components, right?

Yes.

In case of the
later, it would be enough to place single Makefile.builder in main
Whonix repository which lists all the components in DEBIAN_BUILD_DIRS
(list can be easily generated based on submodules list, or some script
output). Note that the order is important to satisfy build depends. If
something more than dpkg-buildpackage is required, it may be enough to
place it in SOURCE_COPY_IN step, or some more flexible mechanism
should be designed.

Ok.

This would mean that it isn't possible to build a single Whonix package, only all of them. If that's a big issue,

I don't think this is an issue.

Looks like I'll be working on the 02 to 04 transition now. Required for other purposes. My experiments show, that this use of apt-get in 09_cleanup_post.sh does not work. (No proper chroot anymore at that stage?)

If qubes-builder-debian provides 04_install_qubes.sh, then qubes-template-whonix cannot provide an additionally executed 04_install_qubes.sh, right? At least, that seems so in my experiments. Is this a bug or feature?

That would also mean, that likely qubes-builder-debian and qubes-template-whonix's 09_cleanup_post.sh would also conflict, i.e. only qubes-builder-debian one would run.

- build Whonix in 04_ instead of 02_ as suggested by @marmarek - https://phabricator.whonix.org/T402
- removed many hacks

https://github.com/adrelanos/qubes-template-whonix/commit/4dc8648e5d3e3a1004565f0bb73d19f2ceec2215