Yes, it's probably difficult. I see two sensible ways;

1. Use builder-debian, which means setting DEBIAN_BUILD_DIRS in

Makefile.builder to a list of debian packages (since a Whonix is seen as
a single component - with submodules). And handle all Whonix-specific
steps (if any) in SOURCE_COPY_IN. If building Whonix packages is just
like any other Debian packages - it should be just enough.

2. Ignore builder-debian and create separate builder plugin to just

build Whonix.

The later probably will be less work (can just call the current build
script, then retrieve deb packages), but not sure if that would be a
better solution.

Actually there is another option - do not do this at all!
And simply use apt repository with binary packages you've built. Then work on getting them build reproducible instead. Since Whonix is based on Debian, and Debian have a lot of progress in reproducible builds, it shouldn't be that hard.

This, besides solving original problem, will also greatly improve template build time.

[Whonix's build script currently already does set up an apt repository. A local one. First packages are build, then added to the local Whonix apt repository. Then installed from the local apt repository.]

marmarek (Marek Marczykowski-Górecki):

Actually there is another option - do not do this at all!
And simply use apt repository with binary packages you've built. Then work on getting them build reproducible instead.

Then the build system would rely on Whonix's official binary and remote apt repository?

Then, if the repository was down, building Whonix would be no longer possible. So I guess the would have to be configurable so someone can point at another, apt repository? Remote one or also local one? Local one by running a local web server?

Hm. Even if packages were reproducible, this looks kinda untransparent to me. Builders would have hard time to know which package versions they get. TOCTOU issue. Even if they check before the build, it could change by the time they started the build as the remote repository is not under their control. Trust issues. Until now, it has always been a goal for me, to build every code that was created under the Whonix umbrella, gets build from source code without touching any remote repositories.

Even if packages were reproducible, the build script or builder wouldn't check if they get the right packages. It would only verify the package using the usual apt package verification. To check they are actually using the package version they think they are using, they would have to actually create the packages themselves. [But then, back to root one.] Then have a script that does the comparison.

I was wondering about that. Even if packages were reproducible, it would also require a package verification daemon? A daemon that keeps locally building packages, fetching remote packages over Tor (to prevent sending backdoor infected packages to targeted builders), and comparing them.

We certainly have a double standard here. Being harsher with Whonix than Debian. Having a higher security standard for building Whonix packages. If you compare how Debian base images are build, we are trusting apt verification only. Building all Debian packages from source code is not even fully developed. And a hard issue. (Source: See this ticket/3k USD bounty. [1]) Also Debian upstream package versions do change. [2] But Whonix isn't a big distribution as Debian that is historically grown that way. If you look closer at Debian build security... Oh well. Who has access to the machines running the build daemons. Last time I checked, some were hosted at some Debian Developers home. However, making this argument is difficult for social, not technical reasons. Debian is a huge distribution and subject to more leniency. If Whonix did it like this, it could result in FUD. And someone could rightly make the argument, that Whonix can with acceptable effort prevent these issues, while that is not so simple for whole Debian.

[1] https://www.bountysource.com/issues/9115540-build-debian-packages-from-source-code
[2] unless using the frozen http://snapshot.debian.org repository. But then you're also just trusting they keep sending the frozen versions. It's not enforced by version pinning / verification.

Then the build system would rely on Whonix's official binary and remote apt repository?

By default - yes. But that shouldn't be a problem to have it
configurable.

Hm. Even if packages were reproducible, this looks kinda untransparent to me. Builders would have hard time to know which package versions they get.

You can specify versions implicitly if that's an issue.

Even if packages were reproducible, the build script or builder wouldn't check if they get the right packages. It would only verify the package using the usual apt package verification. To check they are actually using the package version they think they are using, they would have to actually create the packages themselves. [But then, back to root one.] Then have a script that does the comparison.

Yes, that would need some additional means to verify the package - not
only with apt metadata, but also with some external signature(s).

I was wondering about that. Even if packages were reproducible, it would also require a package verification daemon? A daemon that keeps locally building packages, fetching remote packages over Tor (to prevent sending backdoor infected packages to targeted builders), and comparing them.

Actually it may be simply enough to compare with others. So is all of
trusted builders get the same results, it should be fine. But if any of
them got different result, you should investigate that carefully - you
don't know which one is bad (even the majority may be wrong here).

Ok, lets settle now on building packages locally. But generally I think
the idea worth some further discussion (CCC?).

Marek, do you know my argument "just because lots of Debian packages are
reproducible, it won't mean, that we can get reproducible raw images
anytime soon"? [Because they keep care of deterministic package builds
for now. Not deterministic installed packages. During installation a lot
of packages create entropy.]

marmarek (Marek Marczykowski-Górecki):

But generally I think
the idea worth some further discussion (CCC?).

Sure thing. Just made a note.

Marek, do you know my argument "just because lots of Debian packages are
reproducible, it won't mean, that we can get reproducible raw images
anytime soon"? [Because they keep care of deterministic package builds
for now. Not deterministic installed packages. During installation a lot
of packages create entropy.]

Yes, we have the same problem with templates. But in case of Whonix
installation, you don't use any of this as an input.

Hmm, maybe we can somehow distinguish release builds from user builds?

1. Release build: made by maintainer(s), so basically you have full control over apt repository. And you are building and uploading (the same) packages there anyway.
2. User build: made by non-maintainer(s), a way to have some assurance (other than binaries signatures) that the product is really built from that source code.

In the first case, it's really frustrating that you build the packages twice, including preparation of the whole build environment. Waste of time, especially when building over Tor.

So maybe we could have two builder modes:

1. Use network (upstream) apt repository
2. Use local apt repository

In the first case, you may skip packages build phase. It should be as small change as possible in template-builder to make sure that the output template will be (functionally) the same. Something like "WHONIX_APT_REPOSITORY" settings with either network path or local path.
Supporting the second case still require integrating Whonix package build into qubes-builder.

BTW Hold on with implementing this. We're considering some rework of package build scripts (both Debian and Fedora) to make it as close to the upstream process as possible. To be able to reuse their code/fixes. This is probably mostly about preparing chroot environment, but needs some research.