Whonix 15 has since come out. Has this been resolved? Not reproducible on Debian Buster either.
Jul 22 2019
This bug does not exist on Debian stable after I upgraded. I have it documented for Arch and a work around for it. Nothing more to be done on my end.
Apr 6 2019
Apr 4 2019
Please kindly consider jointing the related discussion improving compression of Whonix image downloads:
Mar 1 2019
I reopened this because KVM page (https://www.whonix.org/wiki/KVM#Arch_Linux) explicitly mentions Arch as a host OS.
If someone here is using Arch as well, maybe you guys can reproduce this after all. Also see my previous comment.
Feb 23 2019
I'm using Arch.
All the software versions (libvirt, QEMU, virt-viewer, kernel) are deemed stable upstream (by their respective developers, not by Debian folks).
Feb 21 2019
What distro are you using?
Feb 9 2019
Dec 9 2018
Dec 7 2018
Oct 15 2018
By running packages from your distro there is a higher chance that bugs are more visible for more people and more likely to be fixed.
Oct 13 2018
Sorry not reproducible on my end. May be related to the fact that you are running a non-standard setup with custom compiled binaries. By running packages from your distro there is a higher chance that bugs are more visible for more people and more likely to be fixed.
Oct 12 2018
made no difference.
It could be the VM is confused because apparently there are two types of mice attached. I assumed that by adding virtio-mouse it would override and replace the emulated one. Turns out its not this way and I went ahead and reverted this config which should be effective in the next release.
Oct 8 2018
Aug 9 2018
Jul 24 2018
Jun 30 2018
Apr 30 2018
virt-sparsify solution dropped because needs booting the image with qemu-system (not clean, to much unknown consequences, see attached ouptut).
Apr 26 2018
Mar 1 2018
NB for the record: with qemu-ga a guest can still shut itself off via crafted input to the agent. So besides removing timer access to the guest, there was no other advantage to removing ACPI.
Actually we don't have to suspend the guest. Execution of any command on the host after resume is enough to create a uniqu event in the qemu-ga's log file.
The proper and direct way to use virsh to communicate with guest agent:
The YAJL parser used in libvirt is tiny, modern (written in2007) and has no CVEs. It is an SAX type event-driven parser unlike the vulnerable, top-down recursive descent type that was used in QEMU.
Feb 28 2018
It turns out the QEMU guest agent warning was not relevant to those who use libvirt. With libvirt a safe parser is used. Breakouts can only happen if a process on the host is designed to parse guest input because there is no way to control that otherwise it should be safe for our uses. This potentially simplifies the design in many respects but a host package will still be needed. I will update the task list.
[libvirt-users] QEMU guest-agent safety in hostile VM?
Feb 23 2018
Feb 14 2018
Yes there are less moving parts especially when multiple WSs share a GW. Some way to exempt timesync traffic from the WS would be needed though.
Feb 12 2018
HulaHoop added a comment.
With qemu-ga code the whole clock drift detection code becomes redundant. If a
suspend event is triggered the GW should assume clocks are out of sync and
With qemu-ga code the hwclock drift detection code becomes redundant. If a suspend event is triggered the GW should assume clocks are out of sync and trigger lockdown.
Oops didn't realize ntpdate requires query of remote servers. ntpdate is obsolete anyhow but the newer clockdiff still talks to online servers instead of comparing local values. hwclock can give us that:
It's a very good rehash!
Feb 11 2018
@Patrick I wrote a rehash. If you think is too complicated, let me know. It was the simplest and most reliable way I could think of:
Feb 4 2018
Didn't rehash. What's next here? Looks like we learned a lot, but then things stalled. Could you please rehash, and then create a follow-up ticket with the way forward? @HulaHoop
Apr 13 2017
Mar 13 2017
Did you compare your --threads=30 archive with --threads=8 archive?
Mar 11 2017
If you have time could you check how long it takes with 5 or 6 threads? I think it will be near equal to 8, not for reproducibility reasons just for efficient use of system resources. There is probably no reason to use 16 cores on a machine that supports it which would be overkill
Mar 10 2017
Sorry for all this confusion, I think it is only a difference between whether the program "tries" to operate in a single-threaded mode or multi-threaded mode, when we use --threads=1 or don't specify it (default is 1) it compresses the whole file in a single block, however setting --threads 0 or bigger than 1 triggers the multi-threaded mode and the file is split into blocks depending on the compression level and then compressed resulting in a difference in the archive file. how many threads actually used is irrelevant. changing compression level or manually specifying the block sizes will change the outcome.
I will try this with xz utils binaries first in Windows host and then with half of the available cores in Windows VM
I may be wrong, the best way to test this is to maybe create the same archive with half of the available cores in a VM however I can't do this, I don't have debian stretch
I mean reproducibility "between" computers, not on the same one
Did you compare your --threads=30 archive with --threads=8 archive?
anonymous1 added a comment.
Could you please check how long it takes with 4 threads, using %50 of
cpu is expected, it does mean it will take twice as long
This quoted part indicates physical cpu threads:
Could you please check how long it takes with 4 threads, using %50 of cpu is expected, it does not necessarily mean it will take twice as long
I think in the worst case you could care less about a perfectly reproducible end archive (tar.xz) and instead focus on the extracted (tar) file being reproducible, for example linux kernel files are compressed either xz or gz but only the tar itself is signed
If you have 8 threads and if using more than 8 produces same checksum as 8, then what I said would be true
I think the default settings are optimal
anonymous1 added a comment.you could also try lowering or increasing the compression dictionary size to see how it affects the size and speed, however I don't know the commands
anonymous1 added a comment.But I have a feeling it would produce different archives with different number of threads, single core vs dual core vs quad core vs custom vm cores
But I have a feeling it would produce different archives with different number of threads, single core vs dual core vs quad core vs custom vm cores
set and export XZ_OPT="--threads=0" makes sense either way. Therefore added.
"--threads=0" results in 100% CPU usage, yay!
It seems you could use something like this with xz utils:
There is another tool called pxz (parallel xz): https://packages.debian.org/stretch/main/pxz
you could also try lowering or increasing the compression dictionary size to see how it affects the size and speed, however I don't know the commands
there is some related information here:
xz doesn't seem to store file names or timestamps so it should be reproducible, you could still use tar with reproducible options to tar the files and maybe combine with p7zip's xz compression
Other than lowering the compression level, maybe tar doesn't support multi threaded compression whereas on 7zip with xz or 7z compression I can utilize all of my cpu cores.
The xz command just uses about 10% of CPU and just about 90 MB RAM. iotop -a is below 1%.
Mar 7 2017
Jan 23 2017
Maybe because the xz version (5.1.1) in Jessie is single threaded? xz 5.2 gains this feature.
Jan 21 2017
is there any improvement? how long did it take before?
Still takes 70 minutes to compress both images (inside a VM, having an SSD already).
Jan 20 2017
I see. That's great news indeed. New Whonix builds for Whonix 14 must be
created on Debian stretch anyhow, so we have tar version 1.29 there.
Will try soonish.
Jan 19 2017
@Patrick good news
Perhaps Patrick haven't tried bsdtar yet or perhaps it didn't work at the time he asked this question because the required version of bsdtar or other requirements was not in stable repositories
Jan 18 2017
@Patrick try asking this on encode.ru. You may get the best answers there
Jan 15 2017
OK so its reproducibility > speed
I'm not experienced as to how to improve xz compression, assuming you don't want to experience with less known compressors.
What I do know is that freearc (with its many unique compression filters and technologies such as srep) and nanozip are the best compressors around in terms of both speed and compression ratio. I don't know the details of the ticket you mentioned but I think srep might be the best tool to speed it up again. But then it is not deterministic, I'm not sure if there is a way to make it so
Found this SE thread on the topic. The benchmarks put LZ4 as the fastest option for compression and decompression - by a lot.
Jan 14 2017
Dec 28 2016
Another LAN/Public wifi fingerprinting attack that Ethan's code can defeat:
Dec 14 2016
Agreed. This is good as dead. Feel free to close.
Dec 9 2016
Nov 30 2016
I somehow doubt rustybird is going to move this forward because rustybird is a Qubes host, not Debian host user. Without third party contributions to corridor, probably not going to happen. (Meanwhile other Qubes specific software was released, split-browser for Qubes.)
Nov 28 2016
Done. Added io limit commits to open pull requests. Each vm can only use a maximum of 25% of the host io resources.
Nov 27 2016
https://bugs.kde.org/show_bug.cgi?id=356864 has more answers.
Nov 20 2016
Should limits be enforced for GW too?