Page MenuHomePhabricator

Qubes Whonix 9.6.9 and 10.0.5 is ready
Closed, ResolvedPublic

Description

Hi Patrick,

I have completed both qubes-whonix 9.6.7 and 10.0.2

Currently both versions are identical except for the version numbers and changelog.

Please review and provide signed tags of 9.6.7 and 10.0.2. Once this is complete I will attempt another run of test builds using your repo. They are located in Whonix9 and Whonix10 branch in my repo.

I have manually tested upgrading from 9.6.2 to 9.6.7 using the local repo utility that is now included within qubes-whonix package (although it does not get included in build) but would be nice to have these packages included in either the testing or developer repos so we can test the upgrade via apt-get. The upgrade via apt-get will not be successful until the qubes changes I made are also available within the qubes repo as well.

@WhonixQubes:
I have also then upgraded to Whonix 10. Steps I took are (in the template vm):

  • Cloned original templatevm as a backup; just in case of failure
  • Make sure Whonix repo is disabled using old whonix_repository <-- Important
  • Manually enabled qubes-r3 repo in /etc/apt/sources.list.d/qubes-r3.list (repos are disabled by default)
  • apt-get update
  • Simulate an apt-get dist-upgrade ... used qubes-whonix 10.0.2 and unreleased qubes packages
  • Powered off template
  • Powered up template (to allow new qubes-whonix code to remove chattr +i)
  • Enable Whonix testers repo
  • apt-get update && apt-get dist-upgrade
  • poweroff templatevm
  • poweroff whonix-gateway proxyvm
  • poweron whonix-gateway proxyvm to test that upgrade was successful

Prompted to update the following file. Answered Y

/etc/whonix.d/30_whonixcheck_default

Will end with these two messages which is fine, since we are in a template...

Failed to read /qubes-netvm-gateway
Failed to read /qubes-netvm-gateway

whonixcheck report build as 9.6 still, but I definitely have 10.0 packages installed after update.

As far as I am concerned, it looks like Whonix 10 is a go... Great job!

Following is the changelog since 9.6.2 update:

qubes-whonix (0:9.6.7-1 / 0:10.0.2-1) wheezy; urgency=medium
  [ Jason Mehring ]
  * Update files to search and replace IP addresses Fix syntax typo for
    whonix workstation that prevented search and replace
  * start whonixcheck on startup for workstation
  * Use new whonix-setup-wizard directory for *.done files Use
    50_whonixcheck_user instead of 30_whonixcheck_default Enable new
    control-port-filter-python.service
  * Remove unneeded bind directories due to new localtion of whonix
    status files
  * - Remove references to old whonix status files; use new references -
    Start whonixcheck last - Add missing whonixcheck for workstation -
    Don't prompt to install repository in AppVM (Gateway or Workstation)
    - Prompt to install repository in templatevm
  * Add missing whonixcheck.conf file
  * Add systemd unit file for control-port-filter-python.service

qubes-whonix (0:10.0.1-1) wheezy; urgency=medium
  * version 10.0.1

qubes-whonix (0:9.6.6-1) wheezy; urgency=medium
[ Patrick Schleizer ]
  * added genmkfile to Build-Depends
  * updated makefile generic to version 1.5
  * updated readme
  * updated makefile generic to version 1.4

  [ Jason Mehring ]
  * Commented out watchdog as it was resetting tor every minute
  * More specific reference to be able to inject firewall code was
    needed for Whonix 10

qubes-whonix (0:9.6.5-1) wheezy; urgency=medium
  [ Jason Mehring ]
  * Remove chattr +i and replace with a protected files routine
  * Notes with issues not yet resolved due to changes in Qubes or qubes-
    whonix
  * Added wip whonixcheck systemd unit file
  * Added a tor systemd unit files along with a wip unit file to
    implement hardening
  * Added ability to upgrade and dist-upgrade from local test repo
  * Streamlined enable/disable services; remove immutable bits
  * Make sure qubes-network is started before qubes-firewall
  * Keep whonixcheck and sdwdate disabled and manually start them to
    prevent false positive errors that tor is not started
  * Send a 0 when enabling a service

qubes-whonix (0:9.6.4-1) wheezy; urgency=medium
  [ Jason Mehring ]
  * Bump version to 9.6.4
  * Fix a bug that gave error on upgrade when restarting service
  * Use debhelper package install to install files to prevent tests from being part of package
  * Fixed an issue with restarting services and added whonix-setup-wizard cache dir
  * Added more options to make sure unwanted dirs like rpm or deb do not make it into Debian package
  * Removed stale references from notes
  * Added a update test script that will install a local repo and perform an update of package
    The test suite is excluded from built package
  * Updated changelog for 9.6.3

qubes-whonix (0:9.6.3-1) wheezy; urgency=medium
  [ Jason Mehring ]
  * Added /var/cache/whonix-setup-wizard to list of dirs to bind on
    startup
  * Updated Makefile.builder to work with new qubes-builder api
  * Bumped version to 9.6.3

Details

Impact
High

Event Timeline

nrgaway created this task.Apr 24 2015, 1:17 PM
nrgaway raised the priority of this task from to Needs Triage.
nrgaway updated the task description. (Show Details)
nrgaway set Impact to Needs Triage.
nrgaway added subscribers: nrgaway, Patrick, WhonixQubes.
nrgaway renamed this task from Qubes Whonix 9.6.6 and 10.0.1 is ready to Qubes Whonix 9.6.7 and 10.0.2 is ready.Apr 24 2015, 1:18 PM

whonixcheck report build as 9.6 still, but I definitely have 10.0 packages installed after update.

This is expected and by design. But needs documentation. Short explanation and ticket: T276


Review:

  • 50_whonixcheck_user isn't a good name. _user indicates user. /etc/whonix.d/50_whonixcheck_qubes would be appropriate. Then you could also easily get ride of that file one day without risking to delete user changes.
  • Shipping systemd unit file control-port-filter-python.service will most likely prevent later [easy] upgrades, i.e. break the package manager, see https://www.whonix.org/forum/index.php/topic,1167.0.html
  • What about T193, we'll let's do this for Whonix 11?

Two tags / versions pointing to the same version looks a bit strange, no? Do we really need 9.6.7 with Whonix 10 package versions?

I guess we can release 10.0.0.5.5 as stable within the next 3 days or so. That would be elegant to reduce need for 9.6.7? What do you think?


There is some confusion on my side. You didn't merge my changes from master beforehand. Perhaps I should have created a separate notification ticket rather than mentioning it somewhere. Maybe my mistake, maybe my changes were too extensive. There was a small, easy to resolve merge conflict in debian/rules. Anyhow. Merged your latest master into master. The diff between adrelanos/qubes-whonix and nrgaway/qubes-whonix is minimal.

Tagged as 10.0.2-1, not 10.0.2. Using the usual make git-tag-sign way. I hope that's fine?

Also added to developers repository. Note: the one on sourceforge upgrades quite fast. The mirror.whonix.de has a lag of ~1 hour until all mirrors got the changes.

Just so you can test. I recommend another revision where at least the systemd / package manager / upgrade issue is fixed. But that's up to you.

Something that worries me. About https://github.com/Whonix/qubes-whonix/blob/master/tests/apt-update-from-local-repo

Key-Type: RSA
Key-Length: 1024

This is a weak key. Even if it's just a local key and perfectly secure... It's a source for FUD. Fear, uncertainty, doubt. And from a marketing perspective, FUD can kill a project. I don't want to distract discussion of real security issues with such easy-to-confuse-easy-to-fix false-positives.

Can you please use deb [trusted=yes] rather than local signing key for local apt repository? I.e.

deb file:$(readlink -m ${LOCAL_REPO}) ${DIST} main

-->

deb [trusted=yes] file:$(readlink -m ${LOCAL_REPO}) ${DIST} main

and removing the local key creation should do. Also less code and more elegant.

Minor: By the way there is an existing similar script https://github.com/Whonix/whonix-developer-meta-files/blob/master/debug-steps/locally-upgrade-whonix-debian-packages. It upgrades all Whonix packages from source code. But it seems yours has a different purpose?

In T275#3897, @Patrick wrote:

Something that worries me. About https://github.com/Whonix/qubes-whonix/blob/master/tests/apt-update-from-local-repo

Key-Type: RSA
Key-Length: 1024

This is a weak key. Even if it's just a local key and perfectly secure... It's a source for FUD. Fear, uncertainty, doubt. And from a marketing perspective, FUD can kill a project. I don't want to distract discussion of real security issues with such easy-to-confuse-easy-to-fix false-positives.
Can you please use deb [trusted=yes] rather than local signing key for local apt repository? I.e.

Sure, if I can get rid of the signing code, all the better. Be aware that any of the code that is in the test directory is excluded from the Debian packaging build so it is not available at all once the Debian package is created and solely lives in the git repo for developer testing purposes.

deb file:$(readlink -m ${LOCAL_REPO}) ${DIST} main

-->

deb [trusted=yes] file:$(readlink -m ${LOCAL_REPO}) ${DIST} main

and removing the local key creation should do. Also less code and more elegant.
Minor: By the way there is an existing similar script https://github.com/Whonix/whonix-developer-meta-files/blob/master/debug-steps/locally-upgrade-whonix-debian-packages. It upgrades all Whonix packages from source code. But it seems yours has a different purpose?

The purpose of that script is to simulate an update instead of having to use a live repo. I used it to update the qubes-whonix 9.6.2 to 10.0.2 to make sure it would not fail. I also added in the compiled Qubes Debian packages that will be available from their repo. For my purpose, I just clone a whonix-gateway templatevm, copy the test directory and wheezy repos that were built by qubes-builder for the current template build. Then the script runs apt-get dist-upgrade against the main repos and this local repo to test if the upgrade will be a success.

@nrgaway Great job! :)

Here is how I'd like to handle this release, if this is okay...

I'd like to create a new audit thread, announce and invite others in the Whonix and Qubes communities to join in on auditing/testing if they please, and give it an audit myself.

@Patrick can sign and put the new qubes-whonix package(s) into developers and testers Whonix APT repositories whenever, right now, etc.

We and the community complete a quick audit/testing phase, and then we can then ping Patrick and Marek to build and release final DEB and RPM packages.

With Qubes R3.0-rc1 and imminent Whonix 10 releases, I know we want to get this out there very soon, so we can make this a quick audit and assuming nothing major comes up, we can be all ready hopefully by say this Sunday/Monday or so to call it final and go to stable release.

Just wanting to give it a quick audit and be inclusive with the community.

I will initiate this now, assuming there is no problem with this.

Thank you!

@nrgaway What did you think of Patrick's proposal of going directly to Whonix 10 in the next few days?

Sure, if I can get rid of the signing code, all the better.

Great.

Be aware that any of the code that is in the test directory is excluded from the Debian packaging build so it is not available at all once the Debian package is created and solely lives in the git repo for developer testing purposes.

Yes, I was/am aware of that.

The purpose of that script is to simulate an update instead of having to use a live repo.

Alright. As a side note, that's also the purpose of whonix-developer-meta-files/blob/master/debug-steps/locally-upgrade-whonix-debian-packages. It doesn't touch any remote/binary repositories run Whonix. Uses source code by Whonix only. Binary packages are created and installed from local hdd. For trust reasons, for those who prefer to upgrade from source code. (Documented here: https://www.whonix.org/wiki/Dev/Build_Documentation/10_full)


@WhonixQubes: That's fine with me. It's in the developers repository currently. I can migrate it to the testers soon, if requested. I guess it's best if at least one developer tried upgrading from the remote developers apt repository at least once. Otherwise the risk is that the package manager breaks for all testers [of that package]. And manual fixes might be so complicated, that we can't help all users through it. Anyhow. I leave that decision to you. If someone tried a local upgrade from existing versions to the new one and it worked, the risk is a lot smaller.

In T275#3892, @Patrick wrote:

whonixcheck report build as 9.6 still, but I definitely have 10.0 packages installed after update.

This is expected and by design. But needs documentation. Short explanation and ticket: T276

Review:

  • 50_whonixcheck_user isn't a good name. _user indicates user. /etc/whonix.d/50_whonixcheck_qubes would be appropriate. Then you could also easily get ride of that file one day without risking to delete user changes.

Okay

Yes it would. I can rename packages and set some conditionals as described in the mentioned forum topic.

I need to have own tor for qubes-whonix for both 9.6 and 10.0, although as stated, I will rename it. And as I mentioned, both 9.6 and 10.0 currently share the exact same qubes-whonix code base; the systemd unit file for control-port-filter-python is included, but will not run due to a requirement not being met (namely, that the package does not exist hehe)

  • What about T193, we'll let's do this for Whonix 11? -----

Two tags / versions pointing to the same version looks a bit strange, no? Do we really need 9.6.7 with Whonix 10 package versions?

Since 9.6.7 may diverge. They Point to the changelog which is different and I have created a Whonix9 and Whonix10 branch which separates the versions by branch hierarchy.

9.6.7 is for those not wanting to update to 10.0 and contain the important whonix-setup-wizard fix as well as the chattr change.

I guess we can release 10.0.0.5.5 as stable within the next 3 days or so. That would be elegant to reduce need for 9.6.7? What do you think?

Does not matter to me so much, but there may be people not wanting to upgrade to version 10 yet?


There is some confusion on my side. You didn't merge my changes from master beforehand. Perhaps I should have created a separate notification ticket rather than mentioning it somewhere. Maybe my mistake, maybe my changes were too extensive. There was a small, easy to resolve merge conflict in debian/rules. Anyhow. Merged your latest master into master. The diff between adrelanos/qubes-whonix and nrgaway/qubes-whonix is minimal.

I did merge your changes in 0:9.6.6-1. I only noticed the 4 changes related to genmkfile. I typically do not merge wholesale as I prefer to cherry pick the commits and apologize if I missed any others; that is even the case with code from v10 branch to v9 branch.

Tagged as 10.0.2-1, not 10.0.2. Using the usual make git-tag-sign way. I hope that's fine?

Yes, anything fine so long as I can add it to qubes template code ;)

Also added to developers repository. Note: the one on sourceforge upgrades quite fast. The mirror.whonix.de has a lag of ~1 hour until all mirrors got the changes.

Thanks.

Just so you can test. I recommend another revision where at least the systemd / package manager / upgrade issue is fixed. But that's up to you.

Yes, I will need to address these issues as stated and submit updated revision asap

@nrgaway Great job! :)
Here is how I'd like to handle this release, if this is okay...
I'd like to create a new audit thread, announce and invite others in the Whonix and Qubes communities to join in on auditing/testing if they please, and give it an audit myself.
@Patrick can sign and put the new qubes-whonix package(s) into developers and testers Whonix APT repositories whenever, right now, etc.
We and the community complete a quick audit/testing phase, and then we can then ping Patrick and Marek to build and release final DEB and RPM packages.
With Qubes R3.0-rc1 and imminent Whonix 10 releases, I know we want to get this out there very soon, so we can make this a quick audit and assuming nothing major comes up, we can be all ready hopefully by say this Sunday/Monday or so to call it final and go to stable release.
Just wanting to give it a quick audit and be inclusive with the community.
I will initiate this now, assuming there is no problem with this.
Thank you!

You will not be able to test the package unless you build from source, and you would also need to use my qubes-agent-linux repo as there are changes to qubes that to allow the chattr to be removed did not make rc1. They would be available when Marek gets time merge that code, which I assume will be at the same time he works on this. You only need to build that one packaage though, and install it manually.

Until that point it is not ready for testers.

@nrgaway What did you think of Patrick's proposal of going directly to Whonix 10 in the next few days?

Ya, that's fine with me. We can always release the 9.6.7 if people complain, but I can't really see that.

nrgaway renamed this task from Qubes Whonix 9.6.7 and 10.0.2 is ready to Qubes Whonix 9.6.8 and 10.0.3 is ready.Apr 26 2015, 9:10 AM
nrgaway added a comment.EditedApr 26 2015, 9:27 AM

I have completed the changes previously discussed.

I will begin to tackle some of the outstanding tasks which mostly are for Whonix 11 once I can confirm this is stable since I do not want to be introducing too many changes at this point :)

I suppose the tag will be 10.0.3-1

@WhonixQubes
Marek has released a new version of qubes-core-agent that is compatible to Whonix to the Debian test repo. You would need to enable the test repo manually in /etc/apt/sources.d/qubes-r3.list but there is an issue with that package at the moment so I suggest you wait until it gets resolved. If you are impatient, then after you update, you will need to edit /etc/xen/scripts/vif-route-qubes and delete the -w in the iptables statement at about line 56, otherwise the gateway will crash when trying to connect to it.

Once Patrick is able to sign a new tag again, I will again attempt an apt-get update.

@nrgaway That all sounds good. I've updated the audit forum thread and
am reviewing the code myself today.

.d Issue Regression

  • You reintroduced issues, that you earlier fixed already.
  • For the same reason,
    • it's also a bug to edit /etc/cpfpy.d/30_controlportfilt_default. You can ship a higher numbered file and edit that.
    • Same for /etc/uwt.d/30_uwt_default.

etc/systemd/system/tor.service in the workstation issue

The qubes-whonix package also gets installed in the workstation. Likely the following clashes with https://github.com/Whonix/anon-ws-disable-stacked-tor, which gets installed by default.

ConditionPathExists = /etc/init.d/tor

Because anon-ws-disable-stacked-tor ships a /etc/init.d/tor. Your etc/systemd/system/tor.service that would also be installed on the workstation would then have the condition satisfied and try to start Tor, which is not installed. So that would fail. Probably not fatal, but I am sure users will report the failed systemd startup message.


control-port-filter-python.service

What's the following good for?

ConditionPathExists = /etc/init.d/control-port-filter-python

Users that delete that init script (because they think they want to do X with systemd) will have the issue that suddenly cpfpy does not start anymore.


Tagged as 10.0.3-1. Added to developers repository.

In T275#3915, @Patrick wrote:

.d Issue Regression

  • You reintroduced issues, that you earlier fixed already.
  • For the same reason,
    • it's also a bug to edit /etc/cpfpy.d/30_controlportfilt_default. You can ship a higher numbered file and edit that.
    • Same for /etc/uwt.d/30_uwt_default.

You are referring to the replace-ip list I imagine. I will change as indicated and make notes in the source.


etc/systemd/system/tor.service in the workstation issue
The qubes-whonix package also gets installed in the workstation. Likely the following clashes with https://github.com/Whonix/anon-ws-disable-stacked-tor, which gets installed by default.

ConditionPathExists = /etc/init.d/tor

Because anon-ws-disable-stacked-tor ships a /etc/init.d/tor. Your etc/systemd/system/tor.service that would also be installed on the workstation would then have the condition satisfied and try to start Tor, which is not installed. So that would fail. Probably not fatal, but I am sure users will report the failed systemd startup message.

I will check to see what is happening. I can ship with it disabled as well.


control-port-filter-python.service
What's the following good for?

ConditionPathExists = /etc/init.d/control-port-filter-python

That condition was put in so it will not conflict when you add systemd unit files to the main package. Therefore when you install the systemd unit file, and the sysv init script is removed, my control-port-filter-python unit script will not startup, since yours will.

Users that delete that init script (because they think they want to do X with systemd) will have the issue that suddenly cpfpy does not start anymore.

Tagged as 10.0.3-1. Added to developers repository.

It does not look like the systemd unit file is causing any issues in workstation. No errors are being reported...

root@host:/home/user# systemctl status tor
qubes-whonix-tor.service - Whonix Tor anonymizing overlay network for TCP
   Loaded: loaded (/etc/systemd/system/qubes-whonix-tor.service; enabled)
   Active: inactive (dead) since Sun 2015-04-26 13:37:39 UTC; 2min 40s ago
  Process: 1791 ExecStart=/usr/bin/tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc --runasdaemon 0 (code=exited, status=0/SUCCESS)
  Process: 1762 ExecStartPre=/usr/bin/tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc --verify-config (code=exited, status=0/SUCCESS)

Apr 26 13:37:39 host systemd[1]: Started Whonix Tor anonymizing overlay network for TCP.
root@host:/home/user# journalctl -u tor
-- Logs begin at Sun 2015-04-26 13:39:30 UTC, end at Sun 2015-04-26 13:40:14 UTC. --

root@host:/home/user# journalctl -u qubes-whonix-tor
-- Logs begin at Sun 2015-04-26 13:39:30 UTC, end at Sun 2015-04-26 13:40:14 UTC. --
Apr 26 13:37:39 host systemd[1]: Starting Whonix Tor anonymizing overlay network for TCP...
Apr 26 13:37:39 host systemd[1]: Started Whonix Tor anonymizing overlay network for TCP.

You are referring to the replace-ip list I imagine.

Yes, I was.

That condition was put in so it will not conflict when you add systemd unit files to the main package.

and the sysv init script is removed

When using proper systemd aliases, there should be no conflicts. Not sure about removal of old sysvinit scripts, see this comment:
https://phabricator.whonix.org/T106#3920

In T275#3921, @nrgaway wrote:

It does not look like the systemd unit file is causing any issues in workstation. No errors are being reported...

root@host:/home/user# systemctl status tor
qubes-whonix-tor.service - Whonix Tor anonymizing overlay network for TCP
   Loaded: loaded (/etc/systemd/system/qubes-whonix-tor.service; enabled)
   Active: inactive (dead) since Sun 2015-04-26 13:37:39 UTC; 2min 40s ago
  Process: 1791 ExecStart=/usr/bin/tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc --runasdaemon 0 (code=exited, status=0/SUCCESS)
  Process: 1762 ExecStartPre=/usr/bin/tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc --verify-config (code=exited, status=0/SUCCESS)
Apr 26 13:37:39 host systemd[1]: Started Whonix Tor anonymizing overlay network for TCP.
root@host:/home/user# journalctl -u tor
-- Logs begin at Sun 2015-04-26 13:39:30 UTC, end at Sun 2015-04-26 13:40:14 UTC. --
root@host:/home/user# journalctl -u qubes-whonix-tor
-- Logs begin at Sun 2015-04-26 13:39:30 UTC, end at Sun 2015-04-26 13:40:14 UTC. --
Apr 26 13:37:39 host systemd[1]: Starting Whonix Tor anonymizing overlay network for TCP...
Apr 26 13:37:39 host systemd[1]: Started Whonix Tor anonymizing overlay network for TCP.

Makes sense. The existing dummytor implementation (shipping substitute Tor binaries that are just minimal shell scripts in the workstation) prevents this.

Indeed unlikely to cause issues of that kind.

In T275#3922, @Patrick wrote:

You are referring to the replace-ip list I imagine.

Yes, I was.

That condition was put in so it will not conflict when you add systemd unit files to the main package.
and the sysv init script is removed

When using proper systemd aliases, there should be no conflicts. Not sure about removal of old sysvinit scripts, see this comment:
https://phabricator.whonix.org/T106#3920

I don't know what you mean my proper system aliases. I do know that if I have a control-port-filter-python systemd unit file and you add one as well, there is now a conflict. Both will start cpfpy.

I renamed mine to qubes-whonix-control-port-filter-python as not to cause a dpkg install error when installing yours, and I added the condition not to run mine when it notices the the sysv init script is no longer present. If you decide to keep the sysv init scripts as well, we are back to square one; guess we can deal with that then.

I have started the work on Whonix11, and will compile a list of hacks I needed to implement to get it to compile in another task. Still having a few issues and can test these systemd unit files better.

I meant 'proper systemd Alias='.

Yes.

Yes, for Whonix 11 I am looking forward to reduce the number of hacks to a minimum.

Okay, I've reviewed "qubes-whonix" 10.0.3 and give it a thumbs up for
release in its current form.

If any new functional changes are introduced, I'll review those and
provide a final check and update before release.

If no new functional changes are introduced, then I'm good with stable
release of the current package.

Awesome work, @nrgaway! :)

From https://github.com/Whonix/qubes-whonix/blob/master/tests/apt-update-from-local-repo

APT_GET_OPTIONS="-o Dpkg::Options::="--force-confnew" --force-yes --yes"

please remove the --force-yes. Reason: insecure, see also:
https://groups.google.com/forum/#!topic/qubes-devel/akv5B7TgRFQ

In T275#4079, @Patrick wrote:

From https://github.com/Whonix/qubes-whonix/blob/master/tests/apt-update-from-local-repo

APT_GET_OPTIONS="-o Dpkg::Options::="--force-confnew" --force-yes --yes"

please remove the --force-yes. Reason: insecure, see also:
https://groups.google.com/forum/#!topic/qubes-devel/akv5B7TgRFQ

Yes, its now gone. Will submit PR later today to Qubes

nrgaway renamed this task from Qubes Whonix 9.6.8 and 10.0.3 is ready to Qubes Whonix 9.6.8 and 10.0.4 is ready.Apr 28 2015, 3:47 PM
nrgaway claimed this task.

@Patrick

I have another version ready to tag, 10.0.4. Hopefully its the last since I have to gt back working on another task soon too :)

I removed the injected firewall code and dealt with the replace-ips issues as discussed earlier. I also added a work-around for when a user upgrades 9.6 to 10 do disable the old control-port-filter since it never got removed on update. Maybe the new control-port-filter should have conflicts or provides in the control file.

Tagged your Whonix 10 branch by the way as 10.0.4-1.

Just tagged. Not added to any repository. I can do that if you wish. I am not sure, if I should always do that or wait for explicit requests. I don't mind either way.

We really need a better structured release process that helps iron out
the issues and hiccups. I will plan to make a post on this so we can get
a more robust process in place for the future.

For now, at least auto uploading to developers repository probably
makes sense. Not sure if testers repository is also needed by @nrgaway
to test from right now, or if developers works out well for that too?
Not sure if always auto uploading to testers would also poorly affect
some users who may use it? Maybe also depends upon the typical state of
other Whonix packages kept in these repos?

The stable repository should obviously probably be at request though.

I am reviewing the 10.0.4 code now.

I've reviewed "qubes-whonix" 10.0.4 and give it a thumbs up for release
in its current form.

I was wondering about this also and going to post about that. It slows down the development/debugging process if you have to wait for me to tag releases just so you can do a developers-only test build. We're living in different time zones and perhaps I won't have time for a few days at some point.

qubes-whonix isn't build by git cloning https://github.com/Whonix/Whonix and using Whonix's original build script anyhow. Official redistributable template images are build by ITL. So we don't have any security/trust improvements by having "blessed" tags in https://github.com/Whonix/qubes-whonix.

I was wondering if the canonical location for the qubes-whonix package in the qubes-builder should not be rather https://github.com/nrgaway/qubes-whonix or so. We want also easy from-source-code-builds by anyone without having to rely on some official location to sign any tags.

Somehow it all needs to be more flexible and less centralized. The whonix_repository since Whonix 10 supports a --baseuri switch to easily allow to enable other locations. Perhaps that's of any help. I don't know what tools are missing to make this possible, but if there is anything we can do about this, I am eager to know.

In T275#4090, @nrgaway wrote:

since it never got removed on update. Maybe the new control-port-filter should have conflicts or provides in the control file.

That's already implemented.

Removal of the package works for me during upgrade. The first Whonix 10 testers-only version had this bug but it was fixed in the second Whonix 10 testers-only version that has now been blessed stable.

control-port-filter-python has in debian/control a Conflicts: control-port-filter.
control-port-filter (bash) is no longer available in the repository.
anon-gateway-packages-recommended (from anon-meta-packages repository) depends on control-port-filter-python.

@Patrick

Yes, I am for improving the process.

I do think there could be security and trust pitfalls with that specific
model though.

I don't want to hijack and veer off track the purpose of this ticket to
get 10.0.4 released though.

So it would probably be best for us to open another ticket or thread to
discuss details.

Okay, so to get this back on topic... What's next? I guess @nrgaway will inform us about his test results.

Right. I assume @nrgaway is testing his build, and if it checks out, he
can notify us here, submit a pull request to Marek, and we can have you
@Patrick update qubes-whonix 10.0.4 DEB package as stable.

Then Marek can build and release RPMs when he is ready to.

In T275#4092, @Patrick wrote:

Tagged your Whonix 10 branch by the way as 10.0.4-1.
Just tagged. Not added to any repository. I can do that if you wish. I am not sure, if I should always do that or wait for explicit requests. I don't mind either way.

hehe, I was wondering why it was not in repo. Yes, please add it as I want to do some upgrade testing.

In T275#4097, @Patrick wrote:

I was wondering about this also and going to post about that. It slows down the development/debugging process if you have to wait for me to tag releases just so you can do a developers-only test build. We're living in different time zones and perhaps I won't have time for a few days at some point.
qubes-whonix isn't build by git cloning https://github.com/Whonix/Whonix and using Whonix's original build script anyhow. Official redistributable template images are build by ITL. So we don't have any security/trust improvements by having "blessed" tags in https://github.com/Whonix/qubes-whonix.

qubes-whonix is built using the signed tag version from the Whonix repo. ITL uses both the Whonix and Debian template scripts I created to build. Currently qubes-whonix, Whonix and genmkfile are the related components that are cloned from your repo and verified from your signing key.

Now, stating that, we can move the qubes-whonix repo to ITL. That would make this process much simpler as I would just integrate it into the template-whonix module which I recently created to keep Whonix separate from Debian templates. If we go this route, we should do it right away for 10.0.

I would prefer combining qubes-whonix and template-whonix as they are directly related by version and would be much easier to maintain for everyone involved. This was not the case initially when Whonix was part of the Debian package repos. This will mean, we will need to remove any existing qubes-whonix from Whonix repo (and I would have to also make sure Marek is okay with this too :)

I was wondering if the canonical location for the qubes-whonix package in the qubes-builder should not be rather https://github.com/nrgaway/qubes-whonix or so. We want also easy from-source-code-builds by anyone without having to rely on some official location to sign any tags.
Somehow it all needs to be more flexible and less centralized. The whonix_repository since Whonix 10 supports a --baseuri switch to easily allow to enable other locations. Perhaps that's of any help. I don't know what tools are missing to make this possible, but if there is anything we can do about this, I am eager to know.

In T275#4128, @nrgaway wrote:

hehe, I was wondering why it was not in repo. Yes, please add it as I want to do some upgrade testing.

Done, added to developers repository.

In T275#4130, @Patrick wrote:
In T275#4128, @nrgaway wrote:

hehe, I was wondering why it was not in repo. Yes, please add it as I want to do some upgrade testing.

Done, added to developers repository.

Thanks!

I also sent Marek a note to see if it would be okay with him to merge template-whonix and qubes-whonix.

Alright.

How would users upgrade the qubes-whonix package then? Does the ITL repo also contain Debian packages?

How do we keep it sync? I mean, when there are package upgrades for Whonix and the qubes-whonix package needs to be upgraded as well, it would be difficult to make it happen at the very same time, no? (Just image, Whonix 10 packages are being released to the stable repository but you need a few more days for the qubes-whonix package. What I want to think through and prevent here is that some upgrade from either repository is breaking the system for everyone using the repository.)

In T275#4132, @Patrick wrote:

Alright.
How would users upgrade the qubes-whonix package then? Does the ITL repo also contain Debian packages?

Yes they also have a Debian repo for Wheezy and Jessie

How do we keep it sync? I mean, when there are package upgrades for Whonix and the qubes-whonix package needs to be upgraded as well, it would be difficult to make it happen at the very same time, no? (Just image, Whonix 10 packages are being released to the stable repository but you need a few more days for the qubes-whonix package. What I want to think through and prevent here is that some upgrade from either repository is breaking the system for everyone using the repository.)

Marek seems to agree that it would cause confusion to keep qubes-whonix on ITL repos. I received Marek's response. He is concerned that it would cause confusion to the end user if qubes-whonix was within the deb.qubes-os.org repo and the package would not make sense there without the other Whonix packages. The example he gave was a user will be confused if they install qubes-whonix in a debian-7 template [Currently the package would not even install due to a failed dependency whonix-setup-wizard if the Whonix repo was not added].

So it looks like maybe the best option may be to keep it as is. As I stated the official ITL build as well as end users builds both get the qubes-whonix package and use the signed tags to verify, from the Whonix repo.

It would not be desirable to have my repo as the main source since that would require the users to import and additional key. I am fine with how its working out so far so long as you are too. If you want to propose an alternate solution, I would think we should bring Marek into the conversation.

@WhonixQubes

Maybe you want to give an upgrade a try. The required qubes package is now also in the ITL test repo so you should be able to attempt an update. DO NOT FORGET TO BACKUP (CLONE) your existing whonix proxy :)

Following is the procedure I used to update a Release 3 Whonix 9.6 to 10.0.

I have not tested in in Release 2 as I do not have that installed. I also re-created a proxy-vm (sys-whonix) after the update (Qubes Release 3 uses sys-net, sys-firewall, so please use this term sys-whonix to identify Whonix proxyvm.). I think it would be best to re-create the Whonix proxy-vm since its quick and could avoid some missing path conflicts, but go ahead and test it if you want either way.

I have not tested updating workstation, since I did not have a 9.6 version installed, but I would assume the same procedure as listed below could be followed verbatim. I am building a 9.6 release to test with it.

In dom0

Backup (clone) existing whonix-gateway first (max 30 chars)
qvm-clone whonix-gateway-experimental whonix-gw-backup

In Whonix template-vm

Enable qubes TEST repo; uncomment test repo
sudo vi /etc/apt/sources.list.d/qubes-r3.list

Enable Whonix developers repo (for qubes-whonix)
sudo whonix_repository

Update index
sudo apt-get update

IMPORTANT Update qubes-whonix package first (Ignore any errors)

sudo apt-get install qubes-whonix qubes-core-agent

Update rest of system

Select Y to to install any maintainer version of files

XXX: @Patrick, do your scripts ignore backed up files in directories such as /etc/whonix_firewall.d.... If not, maybe consider using a standard extension of .conf and only include those?

Based on Patrick's answer to above, user may need to delete:
/etc/whonix_firewall.d/32_qubes.dist (may be named differently; depending on version upgrading from)

Start the Whonix 10.0 upgrade

sudo apt-get dist-upgrade

control-port-filter-python is not always being re-enabled properly via update; could depend on update method used (I have noticed this when updating all at once where I did not update qubes-whonix first). It will not hurt to do this step in the template-vm directly after upgrade to ensure service will start properly.

sudo systemctl is-enabled qubes-whonix-control-port-filter-python
sudo systemctl disable control-port-filter-python
sudo systemctl enable qubes-whonix-control-port-filter-python

Shutdown template-vm

sudo poweroff

Consider creating new proxyvm (sys-whonix)

in qubes-manager using the updated template as template-vm

NOTES

Even though control-port-filter was un-installed, it's sysv init configuration file remained, as well as the start directive in /etc/rc3.d/S17control-port-filter

In T275#4134, @nrgaway wrote:

XXX: @Patrick, do your scripts ignore backed up files in directories such as /etc/whonix_firewall.d....

The config folder sourceing code can be viewed here:
https://github.com/Whonix/whonix-gw-firewall/blob/master/usr/bin/whonix_firewall#L27
https://github.com/Whonix/whonix-gw-firewall/blob/5ebca43416269a5fd42c98a80af1b0b3d38adecf/usr/bin/whonix_firewall#L27

Files that end with ~ are ignored. Files that end with .dpkg-* are ignored. So you could rename these files for example to .dpkg-nouse.

If not, maybe consider using a standard extension of .conf and only include those?

Created T286 for it.

Based on Patrick's answer to above, user may need to delete:
/etc/whonix_firewall.d/32_qubes.dist (may be named differently; depending on version upgrading from)

Yes, delete it.

nrgaway renamed this task from Qubes Whonix 9.6.8 and 10.0.4 is ready to Qubes Whonix 9.6.8 and 10.0.5 is ready.Apr 29 2015, 2:08 PM

@Patrick, I created a new version that hopefully solves the control-port-filter-python not re-enabling properly... I had a typo in the maintainers postinst configuration file.

So when you get a chance, please tag and upload 10.0.5 to the developers repo

Thanks.

PS. Could you also tag 9.6.9 as well please. That version does not need to go into developers repo, but it is tied to the current qubes templates release and will be kept for 'history' to complete the qubes-template-whonix module. The master qubes-template-whonix branch will be updated with the reference to this tag, then its version will immediately be bumped and merge in 10.0. The purpose is to maintain a history, even though it was never released, since there were many changes since the last release of 9.6.2 and allows users to maintain 9.6 if they so choose.

Signed, pushed 9.6.9-1.

Signed, pushed 10.0.5-1 and added to developers repository.

Thanks, tests are successful for me.

I will start a testers task

nrgaway renamed this task from Qubes Whonix 9.6.8 and 10.0.5 is ready to Qubes Whonix 9.6.9 and 10.0.5 is ready.Apr 30 2015, 4:53 AM
Patrick triaged this task as Normal priority.May 3 2015, 6:00 AM
Patrick changed Impact from Needs Triage to High.
nrgaway closed this task as Resolved.May 16 2015, 3:00 PM