Page MenuHomePhabricator

check Qubes build script for apt-get update network / gpg failure security issues
Closed, InvalidPublic

Description

There are various security issues when using apt-get update in scripts:

TODO:

Check if Qubes' build script for building Debian and/or Whonix templates catch those issues and fix them if necessary.

Details

Impact
High

Event Timeline

Patrick updated the task description. (Show Details)Feb 15 2015, 2:39 PM
Patrick added projects: Whonix, security, Qubes.
Patrick created this task.
Patrick raised the priority of this task from to Normal.

Currently the build process will fail and exit if any non-zero exit code is received.

No meaningful messages are reported; only the reason for exit as reported by apt-get. If VERBOSE mode is set, one can easily see why the failure happened,

A not-so-common problem is network failures. I do not have any retries or mirrors set up and do not foresee adding that feature in the current version since I may be re-factoring all the qubes template code again.

Currently the build process will fail and exit if any non-zero exit code is received.

Problem is, apt exit codes aren't reliable in scripts.

and do not foresee adding that feature in the current version since I may be re-factoring all the qubes template code again.

Probably best to do this for Whonix Qubes (or qubes-whonix ?) 11.

yeah, add it to qubes-whonix-next

nrgaway claimed this task.Feb 22 2015, 10:13 PM

Is this done?

nrgaway changed the task status from Open to Review.Jun 6 2015, 6:32 PM

The Qubes build script will fail the install of the template if any apt-get errors are returned.

the --force option has also been removed as well as adding retry.

There are no meaningful exit codes for failure but if verbose mode is turned on you can see what the error was by reviewing the output.

Patrick changed the task status from Review to Open.EditedJun 7 2015, 7:21 AM
In T170#5334, @nrgaway wrote:

the --force option has also been removed

Good.

The Qubes build script will fail the install of the template if any apt-get errors are returned.

I don't think this can be worked around at the moment without parsing the output of apt-get.

There are no meaningful exit codes for failure but if verbose mode is turned on you can see what the error was by reviewing the output.

(Just so we don't talk past each other: Nevermind meaningful exit codes for the build script. Not my point here.)

The issue is, that apt-get does not provide meaningful exit codes for gpg and network failures. "Not meaningful" here means, that apt-get update exits 0, even if it failed. This makes it hard to figure out in scripts if apt-get update actually succeeded loading repository metadata or failed.

The threat model here is using multiple repositories. Such as Debian's main repository as well as Debian's security repository. For consistent and secure builds, both have to be enabled. Now, an adversary could make fetching the Debian main repository succeed, but simulate a gpg or network failure for the Debian security repository. Then the build would go on only with the Debian main repository, without the Debian security repository.

In other words... Here is a practical example...

Consider the following sources.list which is fine.

deb http://security.debian.org wheezy/updates main contrib non-free
deb http://ftp.us.debian.org/debian wheezy main contrib non-free

Consider the following sources.list which is fine which contains an intentional typo for demonstration purposes.

deb http://not-security.debian.org wheezy/updates main contrib non-free
deb http://ftp.us.debian.org/debian wheezy main contrib non-free

Now, when you run...

sudo apt-get update ; echo $?

The output shows.

Err http://not-security.debian.org wheezy/updates Release.gpg                                                                                                                      
  Could not resolve 'not-security.debian.org'
Hit http://ftp.us.debian.org wheezy Release.gpg                                                                                                                                    
...
W: Failed to fetch http://not-security.debian.org/dists/wheezy/updates/Release.gpg  Could not resolve 'not-security.debian.org'

W: Some index files failed to download. They have been ignored, or old ones used instead.
0

Security repository was excluded. That's an attack an adversary can mount or it can happen for other reasons also. The bad thing here is, apt-get exited 0. Therefore hard to detect these situations in scripts. (Bugs are reported upstream - see ticket description - could take a very long time until fixed upstream.)

The way this was solved in Whonix 10 is by using a wrapper. /usr/lib/apt-get-wrapper, that parses apt-get update's output. Awful hack, but the real fix would require fixing apt-get upstream at Debian. I think the only way to make qubes-builder-debian equally secure would be using that or a similar script/way.

Patrick set Impact to Needs Triage.

I see your point.

I don't like really like the idea of using a wrapper if we don't need to since it would need to be installed before qubes packages are installed.

How about using something like this?

apt-get update 2>&1 | gawk 'BEGIN{} /^W:/{ret_code=125;}/^/ ; END{exit ret_code}' ; echo $?
Patrick added a comment.EditedJun 30 2015, 4:12 AM

If we don't have duplicate sources[1][2][3] while building... And that should be doable... Then this could work.

Few points:

  • should match for both, W: and E:
  • match W: and E: case-insensitive, i.e. also for w: and e: (You'll never know if apt-get changes the output at some point.)
  • there is no need to always exit 125 just because a W: or E: was found in the output, for many cases apt-get properly exits non-zero in case of W: or E:
  • applies to 'apt-get update' only
  • ideally, that new apt-get command should be a variable or function, so it does not have to be copied verbatim multiple times (but looks like you have this anyhow)
  • support for builders to inject options must not be broken (APT_GET_OPTIONS)
  • enable pipefail, set -o pipefail, otherwise apt-get's exit code that comes first in the pipe will not be forwarded to gawk
  • this stuff is fragile, some unit testing for these parsers would be good, but this can be left for future work

Footnotes:

[1] Because then the matching command gets too complex.
[2] Ignoring the duplicate sources issue, because it's not serious.
[3] https://github.com/Whonix/whonixcheck/blob/12b77823d3df94075f447212bd279e1c578ca65d/usr/lib/apt-get-wrapper#L55-57


Trivia:

  • Whonix copies the wrapper at build time before installing any packages so the wrapper can be used.
    • I don't like this solution too much.
Patrick updated the task description. (Show Details)Aug 6 2015, 4:04 PM
Patrick removed projects: qubes-whonix 12, Whonix 12.
Patrick changed Impact from Needs Triage to High.
Patrick removed nrgaway as the assignee of this task.
Patrick closed this task as Invalid.Aug 6 2015, 4:06 PM
Patrick claimed this task.

Moved to Qubes' tracker...
Debian Template: build script security - deal with 'apt-get update' unreliable exit codes:
https://github.com/QubesOS/qubes-issues/issues/1107