I prefer we not touch binary packages unless there are solid security guarantees - similar to verification of binaries hosted on a mirror.

How complicated is it for using apt to check a package against some signing key in the keychain?

HulaHoop (HulaHoop):

I prefer we not touch binary packages unless there are solid security guarantees - similar to verification of binaries hosted on a mirror.

I don't think there is such verification at the moment let alone anyone
on the internet keeping on auditing.

It won't change a lot. Rather than building a Whonix source package from
source and then comparing it with the binary one from the Whonix
repository, an auditor would download the binary tor package from
deb.torproject.org and then compare it with the one in the Whonix
repository which should have matching hashes.

How complicated is it for using apt to check a package against some signing key in the keychain?

Is this in scope of this ticket? If you are asking if someone who
downloaded the tor package from Whonix repository could verify it
against the deb.torproject.org signing key, no that is not supported by
apt-get. This is (also) because in apt repositories, the repository
metadata is signed. Packages are verified against that metadata. The
metadata is verified against all available apt keys. Individual packages
are not signed. Debian unfortunately does not have an end-to-end
authentication mechanism for binary packages signed by the package
maintainer up the user who downloaded them through apt-get. This is
(also) because Debian packages are usually build on Debian build
servers. [off-topic, FYI: Some of these build machines are just located
in 'some Debian Developer's home'. Which is not great for security.
That's why reproducible builds really matter.]

an auditor would download the binary tor package from

deb.torproject.org and then compare it with the one in the Whonix
repository which should have matching hashes.

Will this be done automatically by the build script? I don't want this to be a manual verification process that depends on the interest of someone doing it sometimes.

If you are asking if someone whodownloaded the tor package from Whonix repository could verify it against the deb.torproject.org signing key, no that is not supported by apt-get.

Yes that's what I was asking about. Very interesting as I wasn't familiar with details about apt.

There are some problems I foresee with critical update logistics. At the moment if the Tor package needs an emergency update, the people tracking Debian stable only will get it as its backported and those tracking deb.torproject.org get it as soon as the new "testing" release is uploaded. If we arbitrarily freeze some "testing" Tor release then how is it decided to upgrade? Will you need to read the changelog of every minor release to decide? Is this sustainable?

In T472#8166, @HulaHoop wrote:

an auditor would download the binary tor package from

deb.torproject.org and then compare it with the one in the Whonix
repository which should have matching hashes.

Will this be done automatically by the build script? I don't want this to be a manual verification process that depends on the interest of someone doing it sometimes.

There are no verification scripts for any Debians repository at the moment at all. More info:

https://www.whonix.org/wiki/Verifiable_Builds#Verifiable_Whonix_Debian_Packages

If you are asking if someone whodownloaded the tor package from Whonix repository could verify it against the deb.torproject.org signing key, no that is not supported by apt-get.

Yes that's what I was asking about. Very interesting as I wasn't familiar with details about apt.

There are some problems I foresee with critical update logistics. At the moment if the Tor package needs an emergency update, the people tracking Debian stable only will get it as its backported and those tracking deb.torproject.org get it as soon as the new "testing" release is uploaded.

If we arbitrarily freeze some "testing" Tor release then how is it decided to upgrade?

Upgrade every time. Unless the upgrade would break something that would require another Whonix package upgrade to make sure it does not break at the same time. Then upgrading would take longer.

Will you need to read the changelog of every minor release to decide?

Yes, I must do that anyhow.

Is this sustainable?

It has to be.

There are no verification scripts for any Debians repository at the moment at all. More info.

There is a semantics mix up. When I say verifiable I mean to check package signatures or hashes not build reproducibility.

Will the Whonix build script be able to download and compare hashes from the two different repos?

EDIT: I think Verifiable -> Deterministic makes more sense in the documentation.

It has to be.

OK and I'll help with keeping track too.

• The build using Whonix build script should not require using Whonix

binary repository. So there is nothing to compare.

• Automated downloading of the tor package from deb.torproject.org

certainly must use usual apt-get verification.

I could not think of yet how this gets together with builds from

source code. (You cannot run apt-get update from postinst scripts.)

Conflicts with goal T461 (getting rid of chroot scripts).

I'm confused. Source builds will have nothing to with these changes? This only applies for official builds?

I don't think placing such a critical package in a third party repo is a good idea if we don't have a reliable and automated way to trace the chain of trust back to the original binaries.

Conflicts with goal T461 (getting rid of chroot scripts).

Blocking this ticket is not really worth the gain. Why not switch to the older version in Debian's repos if testing is moving too fast?

In T472#8174, @HulaHoop wrote:

I'm confused. Source builds will have nothing to with these changes? This only applies for official builds?

• The chroot script https://github.com/Whonix/anon-shared-build-apt-sources-tpo should be abolished. (T461) Therefore it is related to builds from source code.
• deb.torproject.org should be removed (this ticket).
• Builds from source code should still download, verify and install tor from deb.torproject.org.
• You cannot run apt-get update from Debian packaging maintainer (postinst ...) scripts. (Because apt-get and dpkg is already running, therefore locked.)
  • (Running apt-get update would be required within the process of temporarily adding deb.torproject.org, downloading and verifying tor.)

Why not switch to the older version in Debian's repos if testing is moving too fast?

That would block lots of other stuff. Such as onionshare. (T448 etc.) And latest Tor security enhancements such as to the guard selection algorithm. Also in case they need to deploy a fix to cope up with another botnet attack on the Tor network, we need to react also.

• Builds from source code should still download, verify and install tor from deb.torproject.org.
• You cannot run apt-get update from Debian packaging maintainer (postinst ...) scripts. (Because apt-get and dpkg is already running, therefore locked.)
  • (Running apt-get update would be required within the process of temporarily adding deb.torproject.org, downloading and verifying tor.)

It should be quite easy to download and verify it manually, but there should
be also utilities to do that, without modifying /etc/apt/sources.list*.
Take a look at https://wiki.debian.org/AptMedium for example - basically
it calls apt-get with alternative configuration, pointing at different
directory to download packages without interacting with system database.

How to work around the dpkg lock?

I understand apt-medium uses a separate dpkg database.

So we could install another deb package using dpkg from the postinst
while dpkg is already running and installing a package?

That is the circle I am trying to square here which will likely not work.

No, we do not want to install it there, we want to download it only,
don't we? Then put it into local repository (same as other packages
build there), or upload to whonix apt repo.

Am I missing something?

It depends on how far Installation from Repository / proverbial "sudo apt-get install whonix" (T461) should go.

a)

1. start a (Qubes) Debian (template)
2. add Whonix binary apt repository [*]
3. sudo apt-get install whonix-gateway

b)

1 start a (Qubes) Debian (template)
2 have a script that
2a adds Whonix binary apt repository [*]
2b runs sudo apt-get install whonix-gateway
2c downloads and installs the tor package from deb.torproject.org

If it is supposed to become as simple as the following 3 steps in a)... Then we cannot download and install the tor package from deb.torproject.org. This is because while the package manager is running, we cannot run dpkg install package, since the package database would be already locked.

Which one do we require/want? a) or b)?

[*] Or create a local Whonix apt repository from source code.

If tor package is going to be uploaded to Whonix binary apt repository, it doesn't matter. Downloading tor package from deb.torproject.org is the part of build process, not installation process.

Update: I am trying to avoid uploading the tor package from deb.torproject.org to the Whonix binary apt repository. After thinking more about, it significantly adds up a maintenance burden. Instead I am considering to implement some usability enhancements.

• a script run from .bashrc thats shows the current Tor version / Tor Project repository once some file is created (touch ~/tester or so)
• more comments in /etc/apt/sources.list.d/torproject.list to more easily allow enabling tor-experimental-0.2.7.x-jessie etc.

These should simplify/remind to keep track of newer Tor versions and see how they work with Whonix before they flow into Tor Project's stable apt repository.

FYI, a hack (will probably not use it)...
install apt packages from deb postinst:

• http://javier.io/blog/en/2015/09/10/apt-packages-from-deb-postinst.html
• https://gist.github.com/chilicuil/74fddda2a31bba09b973

Here is my work in progress attempt for downloading the tor package during Debian maintainer postinst script. It uses a separate apt database. It works.

https://github.com/adrelanos/anon-gw-anonymizer-config/blob/5fdf627e7e4286cdab418e8e634df3c07e209133/debian/anon-gw-anonymizer-config.postinst#L24-L92

Installation from there is more tricky. The above hack of merging dpkg status file changes should probably be avoided. The least awful hack might be to have a blocking startup script that runs early and installs the package at next reboot.

That would move us closer to "sudo apt-get install whonix-gateway" etc. Getting rid of chroot scripts. (T461) Then one could create a Qubes Debain ProxyVM by using method a) to morph it into a Whonix-Gateway. That would greatly simplify Whonix installation process. Make it more robust, easier/quicker testing overall, easier porting. I hope I don't have to bend things too much to get there. (We are overwriting files such as /etc/resolv.conf, load firewall rules, etc. I hope none of this is a deal breaker during installation. I don't think there is any other Debian derivative yet that can be installed using apt-get alone.)

That doesn't give us a set of packages that could be installed from Qubes installer inside a Debian template to morph it into Whonix. (The purpose is to save space on the installer disc here.) But getting pure T461 done somewhat simplifies the development of that.

I succeeded to implement the download of

• tor
• tor-geoipdb
• deb.torproject.org-keyring

from deb.torproject.org using anon-gw-anonymizer-config postinst script and installation during next reboot. [*]

• https://github.com/adrelanos/anon-gw-anonymizer-config/blob/T461/debian/anon-gw-anonymizer-config.postinst
• https://github.com/adrelanos/anon-gw-anonymizer-config/blob/T461/lib/systemd/system/anon-gw-anonymizer-config.service
• https://github.com/adrelanos/anon-gw-anonymizer-config/blob/T461/usr/lib/anon-gw-anonymizer-config/startup
• https://github.com/adrelanos/anon-gw-anonymizer-config/blob/T461/usr/lib/anon-gw-anonymizer-config/startup_helper

anon-gw-anonymizer-config now depends on anon-shared-build-apt-sources-tpo.

Should the deb.torproject.org-keyring package be rather installed by anon-shared-build-apt-sources-tpo? That certainly would be cleaner. But then anon-shared-build-apt-sources-tpo would need a similar mechanism to download that package during postinst and to install it on next reboot. [*]

If yes, it would certainly be cleaner to a separate package for the download packages during postinst code being a standalone shell library and a generic install on next reboot mechanism [*] that gets used by both packages, anon-shared-build-apt-sources-tpo and anon-gw-anonymizer-config.

All doable but quite some work.

So I am wondering, do we still require an independent package add torproject apt repository (anon-shared-build-apt-sources-tpo)?

Do we still want to enable torproject's apt repository within Whonix-Workstation? We no longer install any packages from torproject's apt repository in Whonix-Workstation. We used to install torsocks from there, but since Debian jessie this is no longer necessary. We might need that in future though, if Tor Browser gets added to torproject's apt repository - if that ever happens.

[*] Or manually running sudo service anon-gw-anonymizer-config start after installation of the anon-gw-anonymizer-config package. (I wouldn't know how to automate this during apt-get install whonix-gateway etc. This is as long as dpkg is running, there is no (sane) way to dpkg install another package since the package database is locked. And there are no post apt-get hooks. So there is no sane way to not require manually running sudo service anon-gw-anonymizer-config start or reboot after apt-get install whonix-gateway.)

Ended up implementing exactly that:
T461#8622

The download during postinst as part of apt-get install whonix-... does not work. /etc/resolv.conf gets modified during distribution morphing, therefore breaks networking, so the download will fail. I wonder how this could be solved.

How exactly it gets modified? Maybe it is possible to make both old and new version working at the same time?

Small specification so we are on the same page. What I am not talking about here, is the installation without network access for morphing a Debian template into Whonix during Qubes firstboot. This is an issue with apt-get install whonix-... from a usual Debian system with network access. I.e. if subgraph was to ask "how to install Whonix" (for porting), it would be as simple as a). /etc/resolv.conf gets modified by package anon-gw-dns-conf. apt-get install whonix-gateway leads to installation for that package first. After that networking is broken so any other packages requiring networking won't work.

apt-get install whonix-... is different from the way packages are currently installed when using the build script. Then the script installs them in the right order, the script can also create a backup of such files and read-only mount them in chroot. Also currently: we do not start services during initial install (think: whonix-gw-firewall) vs new: start services during install. This is somewhat an attempt to dumb down build-steps.d/1700_install-packages to apt-get install whonix-....

It's great to do that. I just now need to solve issues I could not think of earlier how to solve.

Somehow I would have to express in Debian packaging terms to have the packages that require internet access (anon-gw-anonymizer-config) to be installed first. So other packages that do not require internet access and break internet access are installed after that.

After all the complication for software outside of the Debian repository and the workarounds being complex, I guessed that adding the TPO packages to the Whonix local and binary repository would be the most canonical and best understandable solution.

I've signed up for tor-announce which seems not to be super busy with stable release updates and seems to be on par with deb.torproject.org.

I've started working on scripts to automate the process of downloading (and verifying) packages from deb.torproject.org for reupload to whonix.org repository.

download binary and source packages from torproject repo and add to Whonix repo

The following packages:
tor tor-geoipdb deb.torproject.org-keyring

Are being added to Whonix local / remote repository.

https://phabricator.whonix.org/T472

https://github.com/Whonix/Whonix/commit/61397b057290f245c4ce0bebf16b64494b0d2e27

enabled deb-src so we can download source pages during build

https://phabricator.whonix.org/T472

https://github.com/Whonix/anon-shared-build-apt-sources-tpo/commit/822ca719f783b66877c4c82f9433d2df926ace81

added anon-info - shows Tor apt repository status and Tor package version

https://phabricator.whonix.org/T472

https://github.com/Whonix/anon-gw-anonymizer-config/commit/950b9c4edfc840a89fad630f3df14835c413789f

more out commented by default repositories to ease testing

https://phabricator.whonix.org/T472

https://github.com/Whonix/anon-shared-build-apt-sources-tpo/commit/f62153744865ccb2ce6478edf4ecef3fd0c8fe90

no longer download from deb.torproject.org because it gets uploaded to Whonix repo
    
https://phabricator.whonix.org/T472

https://github.com/Whonix/anon-gw-anonymizer-config/commit/93b31efe6df6f118f66098a47f6d6964e2249a05

have deb.torproject.org and its signing key removed during apt-get autoremove

for existing installations

https://phabricator.whonix.org/T472

https://github.com/Whonix/whonix-legacy/commit/e10915b86f3129a8c392ef2e53513e377757a816

simplify just refreshing deb.torproject.org packages in Whonix repository mirror

https://phabricator.whonix.org/T472

https://github.com/Whonix/Whonix/commit/8365ee40f6d3bc857b311404ce19f8a972273f5a

Adding anon-info to ~/.bashrc should be simple enough.

removed download of packages from deb.torproject.org

because using different implementation (uploading these packages to Whonix
local / remote repository.

Not installing apt-during-apt and not installing these packages also ensures a
smoother transition for existing Whonix versions where deb.torproject.org is to
be disabled.

https://phabricator.whonix.org/T472

https://github.com/Whonix/anon-shared-build-apt-sources-tpo/commit/b416da52403162172611f803d28631a9f42f2ecc

removed apt-during-apt because decided not using this approach

https://phabricator.whonix.org/T472

https://github.com/Whonix/Whonix/commit/c19b1c7a9b2d021c37a26078e60b1f3eead04498

removed anon-gw-build-upgrade-tor from whonix-gateway-shared-packages-shared-meta

because no longer required / existing

https://phabricator.whonix.org/T472

https://github.com/Whonix/anon-meta-packages/commit/5a440f5552d83a05adf4e87530890c9680a14bac

new builds: ok
upgraded builds: todo

Good users are not getting updates unfiltered from deb.torproject.org anymore since Whonix 13.

Users who did not upgrade and stayed on Whonix 12 did now run into an issue that was caused by a newer Tor version from deb.torproject.org that required anon-gw-anonymizer-config being upgraded beforehand:

https://forums.whonix.org/t/error-tor-bootstrap-result