Page MenuHomePhabricator

Accelerating the Build Script
Closed, ResolvedPublic

Description

I've looked at some ways to accelerate the build script to make building Whonix less painful. This is a very long-term item because breaking the script is serious problem and because other priorities but its definitely something worth doing at some point.

Low hanging fruit:

  • Download one set of all packages and make a copy for the other image if gnw is selected to avoid downloading the same things twice. This alone would halve the time.

http://tinylittlelife.org/?p=283 (inspired by the post beneath it)
https://johntellsall.blogspot.nl/2009/04/fast-parallel-downloading-for-apt-get.html

  • Taking this further the image generation step can be done in parallel too though time savings will be less compared to download time reduction.

Disqualified but worth noting:

https://johntellsall.blogspot.nl/2009/04/fast-parallel-downloading-for-apt-get.html

I found the tool "apt-fast" which downloads one or two files quickly, by downloading with multiple streams per file. This is somewhat sketchy, as it requires installation of additional software, assumes the file gets spliced together correctly, and doesn't gracefully handle network problems.

  • apt-fast - an apt wrapper that downloads package chunks from multiple mirrors in a "bittorrent-esque" way. If you are interested in it I could request the author gives it a license.

http://xmodulo.com/speed-slow-apt-get-install-debian-ubuntu.html
https://github.com/ilikenwf/apt-fast

Details

Impact
Normal

Event Timeline

HulaHoop updated the task description. (Show Details)Dec 13 2016, 12:54 AM
HulaHoop updated the task description. (Show Details)Dec 13 2016, 12:58 AM

avoid downloading the same things twice

Did you try using an apt-cache yet as per build documentation, chapter apt cache?


It's not needed to always run the full build script. If you have an overview on which build step is doing what, there is no need to always rerun the whole build script, no need to always rerun all build steps.

Did you try using an apt-cache yet as per build documentation, chapter apt cache?

Hadn't noticed before. Is this affecting the host in any way? Can I do this in the VM only?

sudo -E ./build-steps.d/1100_prepare-build-machine --install-to-root --tor-gateway

Can you please post an example that applies to both ws and gw?

Offtopic:

What does "--install-to-root" do?

Looked it up: --install-to-root only relevant if building on the host

https://github.com/Whonix/Whonix/blob/master/build-steps.d/1100_prepare-build-machine

Is using "-E" only useful then?

export http_proxy=http://192.168.0.1:3142 -> http://10.152.152.11:3142

--install-to-root is deprecated. That's --target root for a few releases now. For physical isolation.

(So I don't know where you got --install-to-root from. Either outdated build documentation or you got outdated sources.)

(For building KVM images, stay miles away from --target root.)

(Also --tor-gateway is outdated so I guess your sources are outdated.)

To get an overview of all switches, use the following and scroll up a bit.

sudo ./whonix_build --gnw -- --help

Or look here: https://github.com/Whonix/Whonix/blob/master/help-steps/parse-cmd#L106

An apt-cache can be used anywhere. When building on the host and also when building inside the VM.

Using an apt-cache affects the host or VM in as far as using an apt-cache usually does while Whonix is not involved. So in case of apt-cacher-ng the daemon will be running and cache files somewhere in the root filesystem (configurable but usually it works fine out of the box without issues).

sudo -E is useful to preserve environment variables. When you do

  • 1) export http_proxy=...
  • 2) sudo ./whonix_build

then whonix_build will not know about http_proxy because sudo clears the environment.

sudo http_proxy=... ./whonix_build should also work. Environment variables wrt Whonix build script work the same way as they do on any Linux (at very least on any Debian based operating system).

It should also alternatively also be possible to set the http_proxy variable through a build configuration snippet.

export http_proxy=http://192.168.0.1:3142 -> http://10.152.152.11:3142

Probably neither. It's 127.0.0.1 unless you want to run apt-cacher-ng on a different system/VM over (virtual) LAN.


Some examples:

sudo ./build-steps.d/1100_prepare-build-machine --build --flavor whonix-gateway --target qcow2

sudo ./build-steps.d/1150_export-libvirt-xml --build --flavor whonix-gateway --target qcow2

sudo ./build-steps.d/1700_install-packages --build --flavor whonix-workstation --target qcow2

Thanks. The outdated section linked to confused me but I got it now.

Still acceleration required?

I am not sure it's great to add new features to Whonix build script. (parallel stuff, speed up stuff [not creating Debian base image and Whonix packages twice]...)

Whonix build script is already a victim of TLDR. Making it more complex leads to even fewer people seeing through and/or wanting to work on it. It's only used to build Non-Qubes-Whonix.

On the contary, I have been considering to throw out lots of non-essential Whonix build script features and/or to replace Whonix build script with some other VM image build system.

A list of features that Whonix build script currently features has been created just now. Grouped into essential, non-essential and undecided importance.

https://www.whonix.org/wiki/Dev/Source_Code_Intro#Build_Script_Features

Feel free to start a discussion or creating an alternative list (changing group priorities).

Finding a VM build system that can do the minimum we decide essential features is a time consuming task to research, implement and test.

Still acceleration required?

No. I consider this ticket complete. apt-cacher-ng is a pretty cool addon that satisfies this need without complicating the buildscript itself.

Whonix build script is already a victim of TLDR.

Don't say that. It was just a matter of RTFM. Its extensive features are a great thing actually.

Making it more complex leads to even fewer people seeing through and/or wanting to work on it. It's only used to build Non-Qubes-Whonix.

IMO its already feature complete

Finding a VM build system that can do the minimum we decide essential features is a time consuming task to research, implement and test.

True. Seems like waste of time to end up with something doing less than what we already have.

HulaHoop closed this task as Resolved.Dec 16 2016, 4:18 AM
HulaHoop claimed this task.