Page MenuHomePhabricator

qubes-whonix build failure
Closed, ResolvedPublic

Description

Build of Whonix templates for upcoming Qubes 4.0 fails on downloading torbrowser. And interestingly, this applies also to Qubes 3.2 templates, which previously worked fine. So, probably it isn't related to 3.2/4.0 difference at all.

Full build logs:

https://travis-ci.org/marmarek/qubes-template-whonix/builds/258845862

Specific message:

https://dist.torproject.org/torbrowser could not be reached.

Possible reasons:
- https://dist.torproject.org/torbrowser is down
- download location changed

Please check: Start menu -> System -> Whonix Check
              or in Terminal: whonixcheck
              or in Terminal with debugging: whonixcheck -v

If whonixcheck reports no problems with internet activity and downloading Tor Browser keeps failing, please report a bug!

(Debugging information: curl_status_message: [6] - [Couldn't resolve host. The given remote host was not resolved.])

Details

Impact
High

Event Timeline

marmarek created this task.Jul 29 2017, 2:12 PM
Patrick changed Impact from Needs Triage to High.
Patrick triaged this task as High priority.

The build was done using jessie-proposed-updates, which is still at tbb_hardcoded_version="7.0.0", that download version was removed from torproject's download server. tbb_hardcoded_version wasn't updated to prevent more breakage due unresolved T671.

What does it mean in practice?
Also "Couldn't resolve host" doesn't look like file removed from torproject's download server...

What does it mean in practice?

Not having a deb package for Tor Browser while at the same time wanting preinstallation of Tor Browser and deterministic builds continues to create a mess. Builds will break as Torproject keeps deleting old versions and/or changing file locations.

Solution:

Also "Couldn't resolve host" doesn't look like file removed from torproject's download server...

Probably transient issue... Because...

From https://s3.amazonaws.com/archive.travis-ci.org/jobs/258845866/log.txt?X-Amz-Expires=29&X-Amz-Date=20170731T103501Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJRYRXRSVGNKPKO5A/20170731/us-east-1/s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=701eed06909cdcc0ebb19521abd90cce2491a0c55a1b03c518a498a778d2b8c0

This worked:

+++ /usr/lib/curl-scripts/curl-prgrs --fail --fail --tlsv1.2 --proto =https --max-time 180 --output /var/cache/tb-binary/.cache/tb/temp/tbb_remote_folder https://dist.torproject.org/torbrowser

This failed:

+++ /usr/lib/curl-scripts/curl-prgrs --fail --fail --tlsv1.2 --proto =https --max-time 180 --output /var/cache/tb-binary/.cache/tb/files/sha256sums.txt.asc https://dist.torproject.org/torbrowser/7.0/sha256sums.txt.asc
+++ curl_exit_code=22

(Because of T710#14252.)

SimonSelg added a subscriber: SimonSelg.EditedAug 6 2017, 2:06 AM

Hey @all,

I did further debugging. It appears that by the time the postinst task of tb-updater runs inside the whonix-gw chroot, /etc/resolv.conf looks like this:

## Torified DNS server for Gateway's own traffic.
nameserver 127.0.0.1

Since there is no torified dns server running on the gateway yet (it's not running after all, the build process just chroots into it!) tb-updater can't resolve hostnames and therefore fails, which explains the error message and the behavior. If one manually fixes this (pause the build process after tb-updater get's installed, edit /etc/resolv.conf in the mounted image (change dns to 8.8.8.8), resume, after tb-updater has run change it back) everything works.

anon-gw-dns-conf is responsible for setting the resolv.conf in whonix and gets installed before tb-updater, therefore no tb-updater has no internet connectivity.

Does someone with a better understanding of the whonix source know where to tackle this problem?

Cheers,
Simon

Edit: Why is tb-updater even installed on the gateway? It should only be installed on the workstation.

Edit 2: Overriding tbb_hardcoded_version using an envirement variable doesn't work - tb-updater will always use the version defined at /usr/share/tb-updater/tbb_hardcoded_version

Edit 3: I added a "fix" for this bug in my fork: https://github.com/SimonSelg/qubes-template-whonix/blob/SimonSelg-fix-tb-updater/whonix-gateway/04_install_qubes_post.sh#L65-L79

There are probably better ways to fix it (see comments) but it works, whonix-ws and whonix-gw are now building successfully for 4.0. Will try to run them soon.

True.

tb-updater should not be installed on Whonix-Gateway anyhow. That's
strange. That's the root cause to be fixed.

Thanks for looking into it! Development help very much needed at this
point. Please keep digging.

SimonSelg added a comment.EditedAug 9 2017, 1:24 AM

I tested the build whonix-gw and whonix-ws templates I build using my patch to qubes-template-whonix and my patch to qubes-builder and everything works just fine (on 4.0 RC1).

Which means whonix-gw and whonix-ws could be included in Qubes 4.0.

My patch doesn't solve the core issue (tb-updater gets installed while anon-gw-dns-conf is already in effect), it does the following two things:

  • Work around connectivity issue by installing tb-updater before installing the pack of packages containing anon-gw-dns-conf
  • It patches tbb_hardcoded_version in tb-updater to 7.0.3 since the package hasn't been pushed to the repo (T690)

I could create a PR to qubes-template-whonix if there is any interest.

I prefer the proper fix, which is a chain of three tickets in total: https://phabricator.whonix.org/T671#14310
Independently (not a blocker), it would be good to find out why tb-updater is installed in whonix-gw.

IMO the chain doesn't solve the issue of tb-updater not getting any internet connectivity during postinst, which is the root cause of this issue ("Couldn't resolve host. The given remote host was not resolved").

The chain would only fix the wrong tbb_hardcoded_version value.

Ah, you're right. So the second line in my comment _is_ a blocker too.

Nope, tb-updater getting installed in whonix-gw doesn't cause this issue.

It also gets installed in whonix-ws (which is definitly required) where it causes the same problem (because the dns config is already in effect there too).

Are you sure about that? According to build log, the issue with whonix-ws is missing 7.0.0 version on server. anon-gw-dns-conf is not installed in whonix-ws

SimonSelg added a comment.EditedAug 9 2017, 2:11 AM

In whonix-ws the package is called anon-ws-dns-conf . Yes I'm sure about that. The build log explicitly says "Couldn't resolve host".

Also, if you watch the build log you can see that in both whonix-gw and whonix-ws tb-updater and anon-{ws|gw}-dns-conf gets installed at the same time.

I verified that by stopping the build process, chrooting into the image and tested it there myself.

I'll start a new build and verify it again tho.

In above linked travis job, workstation build (17.6) fails with:

(Debugging information: curl_status_message: [22] - [HTTP page not retrieved. The requested url was not found or returned another error with the HTTP error code being 400 or above. This return code only appears if -f, --fail is used.])

Probably package installation order is non-deterministic here...

Also, it worked before (when tor browser 7.0 was still downloadable)... See builds history on travis (https://travis-ci.org/marmarek/qubes-template-whonix/builds).

K interesting. I'll check this out, thanks.

tb-updater must not be installed on Whonix-Gateway at all cost. It's a blocker, since that messes up a carefully selected and package selection.

From https://s3.amazonaws.com/archive.travis-ci.org/jobs/258845867/log.txt?X-Amz-Expires=29&X-Amz-Date=20170809T074523Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJRYRXRSVGNKPKO5A/20170809/us-east-1/s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=36184d2b752e87e36457a6ba48b1ca279482e37e1c2558cdd6345601ceda11bf .... This is might be the cause:

+ /usr/sbin/chroot mnt eatmydata apt-get -o Dpkg::Options::=--force-confnew --yes -o Acquire::Retries=3 -o Dpkg::Options::=--force-unsafe-io install qubes-whonix-gateway

Whonix used to be build using builder.conf:

TEMPLATE_ALIAS += whonix-gateway:jessie+whonix-gateway+minimal+no-recommends

This results in Qubes builder using apt-get with --no-install-recommends. Looks like this is what's missing in this build?

Indeed, TEMPLATE_OPTIONS variable wasn't properly propagated. Fixing this fixes whonix-gateway build:
https://travis-ci.org/marmarek/qubes-template-whonix/builds/263033866

tb-updater with updated hardcoded Tor Browser version is now available in Whonix jessie-proposed-updates repository. Could you try a build please? Quite likely it will go past that issue now.

Patrick closed this task as Resolved.Aug 30 2017, 10:40 PM
Patrick claimed this task.

The problem is back again, 7.0.4 is no longer available at https://dist.torproject.org/torbrowser/
What is the easiest/elegant way to choose different version, without modifying tb-updater package? Some env variable? Some config file? I don't consider https://github.com/SimonSelg/qubes-template-whonix/blob/SimonSelg-fix-tb-updater/whonix-gateway/04_install_qubes_post.sh#L65-L79 elegant...

marmarek (Marek Marczykowski-Górecki):

marmarek added a comment.

The problem is back again, 7.0.4 is no longer available at https://dist.torproject.org/torbrowser/
What is the easiest/elegant way to choose different version, without modifying tb-updater package? Some env variable? Some config file? I don't consider https://github.com/SimonSelg/qubes-template-whonix/blob/SimonSelg-fix-tb-updater/whonix-gateway/04_install_qubes_post.sh#L65-L79 elegant...

Agreed.

tb-updater (update-torbrowser script) as is should already support
environment variables for example tbb_hardcoded_version="7.0.6".

So

export tbb_hardcoded_version="7.0.6"

should do. Long time untested. Might have worked when Qubes-Whonix was
new if I remember right. Could you try that please?

The question is if that environment variable flows down from
qubes-template-whonix through sudo (?), through apt-get, through
tb-updater's debian/postinst to the actual update-torbrowser script?

Patrick reopened this task as Open.Oct 8 2017, 1:12 AM

https://github.com/Whonix/qubes-template-whonix/pull/1

Just setting tbb_version or tbb_hardcoded_version variable isn't enough, because it isn't propagated through all the layers to postinst of tb-updater. But creating temporarily a configuration file works (in /etc/torbrowser.d).
Use tbb_version there, because tbb_hardcoded_version is unconditionally overridden by /usr/share/tb-updater/tbb_hardcoded_version. But later is ignored if tbb_version is already set.

> Just setting `tbb_version` or `tbb_hardcoded_version` variable isn't enough, because it isn't propagated through all the layers to postinst of tb-updater.

That's unfortunate. I wonder why that is.

(Yes, tbb_version already set works and then tbb_hardcoded_version
is ignored.)

Patrick closed this task as Resolved.Oct 9 2017, 9:32 AM