converted chroot-script 40_add_torprojects_key to postinst script:
https://github.com/Whonix/anon-shared-build-apt-sources-tpo/commit/495fc4cf73392dc9b9fd7e780afb92eb16c32d80

log build version

So the chroot-script
https://github.com/Whonix/anon-shared-build-log-build-version/blob/master/usr/lib/anon-dist/chroot-scripts-post.d/70_log_build_version
can be deprecated.

https://github.com/Whonix/anon-base-files/commit/a9752eb30fa3347fd0b0cf555720b6218ffb0d32

removed anon-shared-build-log-build-version

Because that is now implemented in anon-base-files postinst.

https://github.com/Whonix/anon-meta-packages/commit/a8d9e666da27e04abf5e567cdcddd97eaee54339

Its a great leap towards upstreaming and overlaps with some work needed for a hypothetical Whonix-Host metapackage.

This could solve the porting nightmare to new architectures. All a user would need to do on an ARM machine is just download the vanilla Debian images, install apt-transport-tor, add the Whonix repo and BAM! They are converted it into the Whonix images.

removed anon-shared-build-upgrade-torsocks since no longer required
    
(version from Debian is recent enough)

thereby chroot-script deprecated
    
https://phabricator.whonix.org/T461

https://github.com/Whonix/Whonix/commit/cc79fa15580bcb43f01e14d8f306b38eb8e0073e

removed anon-shared-build-inst-tb because it has been merged into tb-updater

https://phabricator.whonix.org/T461

https://github.com/Whonix/Whonix/commit/5b9189fc6397d24a20cd4d06abc74d63fc9f306c

converted chroot scripts into postinst script

https://phabricator.whonix.org/T461

https://github.com/Whonix/anon-shared-build-apt-sources-tpo/commit/495fc4cf73392dc9b9fd7e780afb92eb16c32d80

Created a new package:
https://github.com/Whonix/apt-during-apt

More background:

• T472#8603
• T472#8606

download packages tor and tor-geoipdb from The The Project's apt repository

using apt-during-apt

thereby abolished previous chroot scripts for doing that

https://github.com/Whonix/anon-gw-anonymizer-config/commit/8db878bb634f01e6789c4e44ee10ce2d24f3d1d1

deprecated anon-gw-build-upgrade-tor package since now integrated as postinst

in package anon-gw-anonymizer-config

https://phabricator.whonix.org/T461

https://github.com/Whonix/Whonix/commit/b40ad931432498b5865d341d10c102fdd487cfda

cleanup, got rid of obsolete /usr/lib/anon-dist/chroot-scripts-pre.d/20_sanity_checks

https://phabricator.whonix.org/T461

https://github.com/Whonix/anon-shared-build-sanity-checks/commit/55d1c95b06d38245e6505191cd3f03f440dbbd8b

fixed logging version of the package as build version

in case anon_dist_build_version is not set

https://phabricator.whonix.org/T461

https://github.com/Whonix/anon-base-files/commit/a2bc954be3cadd8bbb710dfbdd1351267f6a35ad

removed anon-shared-build-log-build-version because it was merged into anon-base-files

https://phabricator.whonix.org/T461

https://github.com/Whonix/Whonix/commit/a82bbc43599767d40b3151246b74d159276b4979

There are only 4 chroot scripts left.

find . -type f -not -iwholename '*.git*' | grep chroot-script

• https://github.com/adrelanos/anon-shared-build-sanity-checks/blob/master/usr/lib/anon-dist/chroot-scripts-post.d/20_sanity_checks
• https://github.com/adrelanos/whonix-initializer/blob/master/usr/lib/anon-dist/chroot-scripts-post.d/80_cleanup
• https://github.com/adrelanos/anon-shared-build-ban-nonfree/blob/master/usr/lib/anon-dist/chroot-scripts-post.d/75_vrms
• https://github.com/adrelanos/anon-shared-build-remember-sources/blob/master/usr/lib/anon-dist/chroot-scripts-post.d/75_sources

I am wondering how/where to most elegantly re-implement them.

check for nonfree packages during

https://phabricator.whonix.org/T461

https://github.com/Whonix/whonixcheck/commit/41eb9e1eba1b1e45c39ff20b3730bf184d6432fe

removed anon-shared-build-ban-nonfree because it was merged into whonixcheck

https://phabricator.whonix.org/T461

https://github.com/Whonix/Whonix/commit/471b9aa5cbe0a28d6b5631dff3ba1b08420f3895

merged chroot scripts from anon-shared-build-sanity-checks and anon-shared-build-remember-sources

https://phabricator.whonix.org/T461

https://github.com/Whonix/whonix-initializer/commit/151eec132b99f64d50fea0e8d5b909c68d24c4ad

added dependencies debsums and damngpl for new chroot scripts that were merged

https://phabricator.whonix.org/T461

https://github.com/Whonix/whonix-initializer/commit/d087c4490a63928b922869aa3ede852cd2994a3d

removed anon-shared-build-remember-sources and anon-shared-build-sanity-checks

because they were merged into whonix-initializer

https://phabricator.whonix.org/T461

https://github.com/Whonix/Whonix/commit/e74de8c44c29c4888b05586546f069b14c99fb6a

• Chroot scripts were reduced to a minimum.
• They are no longer relevant for installation of Whonix from repository using apt-get install whonix-gatewayetc.
• Non-Qubes-Whonix: chroot scripts are still relevant for building Non-Qubes-Whonix images using Whonix's build script. To my knowledge, there is no sane way to run the cleanup chroot script from within a Debian package maintainer script.
• Qubes-Whonix: chroot-scripts are mostly irrelevant. Mostly. When the whonix-gateway or whonix-workstation package gets installed into a Qubes Debian template - with the purpose of morphing it into Whonix - perhaps while running on top of the Qubes installer DVD - then the cleanup of the template would be up to the script doing that. It could do so by running /usr/lib/anon-dist/chroot-scripts-post.d/80_cleanup or otherwise.

Great! This will allow major reduction of duplicated data on installation ISO (Debian base packages, Whonix gw/ws common packages). And also result in smaller Whonix templates (no build depends installed there).
As for the cleanup - that's fine - it can be done using salt management stack.

One minor thing: when installing Whonix from firstboot, it should be done without network access, because we can't assume user have set it at that stage. But that isn't a problem - we can bundle copy of appropriate deb packages in installation image. Then creating Whonix templates would look like this:

1. Clone debian-8 template to whonix-gw/whonix-ws
2. Start whonix-ws/whonix-gw
3. qvm-block --ro -A whonix-ws dom0:/usr/share/qubes-whonix/repository.img
4. qvm-run whonix-ws 'mkdir /mnt/whonix-repo && mount /dev/xvdi /mnt/whonix-repo
5. Add /mnt/whonix-repo to /etc/apt/sources.list.d/whonix-local.list (or sth like this)
6. Install appropriate packages (whonix-gateway/whonix-workstation)
7. Remove temporary repository, execute cleanup script, shutdown VM

That local repository could be easily created using apt-get --download-only install whonix-gateway whonix-workstation.
Instead of mounting repository as a block device, we could ship AppVM/ProxyVM with HTTP server configured, but IMO that would be overkill.
Any better idea? Do you see any drawbacks of such approach?

Question: do we need Debian minimal template for that? Or standard template also can be morphed into Whonix template?

I yet have to actually try apt-get install whonix-... and see how it goes. Will work on this next.

In T461#8640, @marmarek wrote:

One minor thing: when installing Whonix from firstboot, it should be done without network access, because we can't assume user have set it at that stage. But that isn't a problem - we can bundle copy of appropriate deb packages in installation image.

One complication. Tor Browser. If tb-updater no longer runs in a chrooted environment with network access, it cannot download Tor Browser from torproject.org. Any idea how to solve that?

We concluded earlier already we don't want to have a debian package hosting Tor Browser tarballs (tb-binary). (Mentioned in T417.) We might reconsider if not avoidable.

One way or another, Tor Browser tarballs would have to end up on Qubes installer DVD, then be copied into the to-be-morphed-to-Whonix Qubes Debian template. From there, tb-updater could take over with installation. (reusing that existing code)

That local repository could be easily created using apt-get --download-only install whonix-gateway whonix-workstation.

Sure, I don't think that would resolve and download the dependencies also?

Instead of mounting repository as a block device, we could ship AppVM/ProxyVM with HTTP server configured, but IMO that would be overkill.

Yes, that sounds overkill. At least if not needed for various other purposes also.

Any better idea? Do you see any drawbacks of such approach?

Your idea sounds alright. No objections, just the details mentioned in this comment.

Question: do we need Debian minimal template for that? Or standard template also can be morphed into Whonix template?

At the moment the packages expect nothing other than Debian, something as minimal as a debootstrapped system. [everything else could be seen as a package bug]

However, I don't think we need the Debian minimal template. For two reasons.

• It's less tested.
• It has had issues in past.

Maybe these are no valid reasons. (Issues in past were due to missing installed Qubes default packages. And after it was morphed into Whonix, it would be more widely tested.)

Debian standard template should work,

• as soon as ported to gnome to avoid duplicate applications (konsole plus gnome-terminal installed at the same time), i.e. Whonix 14.
• as long as it does not install any unwanted packages by default.

We concluded earlier already we don't want to have a debian package hosting Tor Browser tarballs (tb-binary). (Mentioned in T417.) We might reconsider if not avoidable.

Indeed that may be a complication. In worst case we can include downloaded tarball in the same image as debs and have some script which will hand it over to tb-updater (copy to /var/cache/tb-updater so tb-updater will use that?). But that soulds like a hack, not a clean solution.

Sure, I don't think that would resolve and download the dependencies also?

apt-get --download-only install should download everything what is needed. apt-get download downloads only a single package, not dependencies.

• It's less tested.
• It has had issues in past.

Hmm, maybe we should add it to the set of templates I'm running automated tests on? We don't have much tests for VM stuff, but at least something.

as long as it does not install any unwanted packages by default.

There is chrony + ntpdate. And there is also /usr/bin/wnpp-check, but it belongs to devscripts (that may be only my template, not installed by default).

I see two possible solutions:

• handled that using custom step in Whonix setup script (just before step 6)
• have some (optional?) package, which will "conflict" with all the unwanted packages ("conflict" = the type of dependency which removes the other package - is it "replaces"?)

current blockers:

• 1) Tor Browser tarball / installation (described above)

• 2) download of tor, tor-geoipdb, deb.torproject.org-keyring not possible (T472). If we have no networking during that stage... Which is on one hand a very good thing, so we do not depend on it.

If anyone has any ideas how to solve that without increasing maintenance effort...

2. download of tor, tor-geoipdb, deb.torproject.org-keyring not possible (T472). If we have no networking during that stage... Which is on one hand a very good thing, so we do not depend on it.

Just for installation it shouldn't be a problem - we can simply add deb.torproject.org repo for the time of apt-get --download-only install .... But it doesn't help in anyway how to keep track on tor package later.

Hmm, maybe for T472 do not download the package separately, but simply keep some package with strict version dependency like tor (= x.y.z)? That should work in both cases - collecting packages for offline installation and later tracking its version. Does deb.torproject.org keep old version of packages in repository?

In T461#8647, @marmarek wrote:

2. download of tor, tor-geoipdb, deb.torproject.org-keyring not possible (T472). If we have no networking during that stage... Which is on one hand a very good thing, so we do not depend on it.

Just for installation it shouldn't be a problem - we can simply add deb.torproject.org repo for the time of apt-get --download-only install .... But it doesn't help in anyway how to keep track on tor package later.

anon-shared-build-apt-sources-tpo could do that. It ships /etc/apt/souces.list.d/torproject.list. Then newer versions on installed systems would come from deb.torproject.org.

Hmm, maybe for T472 do not download the package separately, but simply keep some package with strict version dependency like tor (= x.y.z)? That should work in both cases - collecting packages for offline installation and later tracking its version.

Does deb.torproject.org keep old version of packages in repository?

No.

systemd-nspawn talked about as a drop in replacement for chroot in build environments:

http://0pointer.de/blog/projects/changing-roots.html
https://lindenberg.io/blog/post/debian-containers-with-systemd-nspawn/

Could it be an answer for the chroot problem in non-Qubes builds?

added install-from-local-repository script

https://phabricator.whonix.org/T461

https://github.com/Whonix/whonix-developer-meta-files/commit/976b7131002ddae909382236ccc907ae676345e5

more robust handling of /etc/hostname

https://phabricator.whonix.org/T461

https://github.com/Whonix/anon-base-files/commit/887094d1e21ec79ad03abc2364cb07fe0bc100cf