@Onion_Knight :

https://forums.whonix.org/t/whonix-desktop-installer-with-calamares-field-report/7350/109

By default, the VMs do not start because the virtual disks are not set to readonly. This is only needed when using the ISO though. Might stay this way as long as the user is correctly advised to change to set the disk to readonly mode.

I can invent a systemd unit file which detects if being run in live mode, and if so, set VM disks to readonly mode. Will add to whonix-libvirt package unless better suggestions.

For the record, this is the diff being generated.

diff --git a/qemu/Whonix-Workstation.xml b/qemu/Whonix-Workstation.xml
index edce9b1..749ae6e 100644
--- a/qemu/Whonix-Workstation.xml
+++ b/qemu/Whonix-Workstation.xml
@@ -52,7 +52,6 @@ or other application using the libvirt API.
       <driver name='qemu' type='qcow2'/>
       <source file='/var/lib/libvirt/images/Whonix-Workstation.qcow2'/>
       <target dev='vda' bus='virtio'/>
-      <readonly/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
     </disk>
     <controller type='virtio-serial' index='0'>

It seems that https://github.com/Whonix/whonix-libvirt/blob/master/usr/lib/whonix-libvirt/live-mode-to-read-only is not ran by root. Thus it cannot get the virsh list --all (returns void) nor change the VM xml configuration file.

We can already change

vm_names_list="$(virsh list --all | awk '{print $2}'| grep -v Name)"

to

vm_names_list="$(virsh -c qemu:///system list --all | awk '{print $2}'| grep -v Name)"

to get the VM list. But I don't know how to safely get root to run the command

virt-xml "$vm_name_item" --edit --disk readonly=on

?

Pretty sure it is run by root.

https://github.com/Whonix/whonix-libvirt/blob/master/lib/systemd/system/whonix-libvirt-set-live-to-readonly.service

That's systemd default.
(Unless User= is set which is not.)

I'll add a whoami for debugging purposes.

But maybe it shouldn't be run by root but as user user?

Also should https://github.com/Whonix/whonix-libvirt/blob/master/usr/lib/whonix-libvirt/install run as root or user user I guess as user user because https://www.whonix.org/wiki/KVM does "seldom use sudo".

Also created https://forums.whonix.org/t/help-welcome-kvm-development-staying-the-course/166/401 for it.

Yes, it should be run by root. Maybe it is run by root but somehow the changes don't take place as they should. More debugging could help.

I added whoami in the script and it confirmed it runs as root.

sudo systemctl status whonix-libvirt-set-live-to-readonly.service

is "succeeded". But it still doesn't work...

It works only if I run it manually.

EDIT: I think I know what the problem is:
whonix-libvirt-set-live-to-readonly.service fails to retrieve the domain names as it appears to be run BEFORE whonix-libvirt-install.service which sets the domain names...

Fixed by adding

After=whonix-libvirt-install.service

in https://github.com/Whonix/whonix-libvirt/blob/master/lib/systemd/system/whonix-libvirt-set-live-to-readonly.service
(under [Unit]).
Pull request: https://github.com/Whonix/whonix-libvirt/pull/109

Good catch! Merged.

I guess images will be set to kvm images read-only when booted in live iso mode (and probably live mode too). But once installed, images are still set to live mode. That would be probably kvm images read-only is set when run in iso live mode, cached in RAM and then installed to local disk?

How to solve this? Set to read-write when booting in non-live mode? That seems intrusive and might lead to things teh user does not want.

If that is the case, maybe at the end of calamares we need to run a script in chroot (same script with new paramater support?) that sets images in chroot (installed system on disk) to read-write?

I guess images will be set to kvm images read-only when booted in live iso mode (and probably live mode too). But once installed, images are still set to live mode. That would be probably kvm images read-only is set when run in iso live mode, cached in RAM and then installed to local disk?

Mmm I'm not sure, let me try it out to verify how it works.

If that is the case, maybe at the end of calamares we need to run a script in chroot (same script with new paramater support?) that sets images in chroot (installed system on disk) to read-write?

That seems a reasonable option, yes.

There are two read-only parameters:

• read-only permissions (0444) of Whonix VMs files set by https://github.com/Whonix/whonix-libvirt/blob/master/usr/lib/whonix-libvirt/install
• read-only KVM paramaters set by https://github.com/Whonix/whonix-libvirt/blob/master/lib/systemd/system/whonix-libvirt-set-live-to-readonly.service

When both are set Whonix VMs work (as of now with last commit) if in live-boot AND in live-mode (meaning live-mode has to be chosen for them too at grub menu when they are booted).

As expected, this behavior is currently copied over during the installation on a persistent drive (Calamares installer, I have just tried it).

As a result, once the user boots an installed version of Whonix-Host, he has to manually revert these ro paramaters to rw in order to have working Whonix VMs. Not practical/user-friendly.

I agree that a solution would probably to run some kind of script at the end of the Calamares installtion to revert ro to rw.

I agree that a solution would probably to run some kind of script at the end of the Calamares installtion to revert ro to rw.

Created:
https://github.com/Whonix/whonix-libvirt/blob/master/usr/lib/whonix-libvirt/persistent-mode-to-read-write

My idea was: keep the virtualizer specific scripts in the virtualizer specific package whonix-libvirt. The (calamares) hook script which calls that script can be added to package live-config-dist.

In T914#19538, @onion_knight2 wrote:

There are two read-only parameters:

Good catch. Would have forgotten about the next one and figured out later.

• read-only permissions (0444) of Whonix VMs files set by https://github.com/Whonix/whonix-libvirt/blob/master/usr/lib/whonix-libvirt/install

I think that is only here:
https://github.com/Whonix/Whonix/blob/master/build-steps.d/1800_copy_vms_into_raw#L35

In summary, related files, implementation so far:

• https://github.com/Whonix/Whonix/blob/master/build-steps.d/1800_copy_vms_into_raw#L35
• https://github.com/Whonix/whonix-libvirt/blob/master/usr/lib/whonix-libvirt/live-mode-to-read-only
• https://github.com/Whonix/whonix-libvirt/blob/master/usr/lib/whonix-libvirt/persistent-mode-to-read-write
• calamares hook script remains todo.

There might be an issue. There is a reason why write access (previously chmod 444) was removed during build in chroot, right? If I remember right, reason being setting chmod in iso live mode boot results the whole image to be modified in RAM?

Then what about hdd live mode boot? Does this issue apply there too?

I.e. we need to keep all boot modes in mind.

• iso live mode boot
• installed disk live mode boot
• installed disk persistent mode boot

This implementation path might not work for installed disk live mode boot?

I think that is only here:
https://github.com/Whonix/Whonix/blob/master/build-steps.d/1800_copy_vms_into_raw#L35

Yes, my mistake. Only here.

There might be an issue. There is a reason why write access (previously chmod 444) was removed during build in chroot, right? If I remember right, reason being setting chmod in iso live mode boot results the whole image to be modified in RAM?

Yes.

Then what about hdd live mode boot? Does this issue apply there too?
This implementation path might not work for installed disk live mode boot?

I don't know. Not implemented yet. Currently installed (persistent) Whonix-Host does not have live-boot option.

To sum up, we still need a calamares hook script to run https://github.com/Whonix/whonix-libvirt/blob/master/usr/lib/whonix-libvirt/persistent-mode-to-read-write at the end of installation, right?

In T914#19541, @onion_knight2 wrote:

I don't know. Not implemented yet. Currently installed (persistent) Whonix-Host does not have live-boot option.

Created for it:

• https://forums.whonix.org/t/whonix-host-operating-system/3931/189
• https://forums.whonix.org/t/running-live-host-via-grub-live/8912

To sum up, we still need a calamares hook script to run https://github.com/Whonix/whonix-libvirt/blob/master/usr/lib/whonix-libvirt/persistent-mode-to-read-write at the end of installation, right?

Yes.

(Though, Whonix-Host installed live mode might not have VMs with right read-only settings.)

Do you know how to run calamares hook scripts? I think I saw this before but I can't find it anymore. Or we have to invent our own mini calamares module similar to how package calamares-settings-debian invented new calamares modules?

Do you know how to run calamares hook scripts? I think I saw this before but I can't find it anymore. Or we have to invent our own mini calamares module similar to how package calamares-settings-debian invented new calamares modules?

I don't know yet. Will look into that.

As of 15.0.1.0.7, the following behavior is observed:

• Whonix-Host in ISO mode correctly sets the gw/ws VMs in read-only mode (kvm xml config files)
• Whonix-Host in ISO mode correctly sets the gw/ws image files permissions in read-only mode
• Whonix-Host in installed (persistent, post-Calamares) mode correctly unsets the gw/ws VMs read-only mode (kvm xml config files)
• Whonix-Host in installed (persistent, post-Calamares) mode fails to set correct permissions fpor gw/ws imafge files

[1] There is currently no trigger (systemd unit file) to execute /usr/lib/whonix-libvirt/persistent-mode-to-read-write.

You could try to manually run:

sudo /usr/lib/whonix-libvirt/persistent-mode-to-read-write

Just now added [1].

https://github.com/Whonix/whonix-libvirt/blob/master/lib/systemd/system/whonix-libvirt-set-persistent-mode-to-read-write.service

Untested. Not sure ConditionKernelCommandLine=!boot=live is going to work.

The exclamation mark ! hopefully means "do this only if boot=live kernel command line is not set".

Some progress made as of Whonix-Host 15.0.1.2.7:

• Whonix-Host in ISO mode correctly sets the gw/ws VMs in read-only mode (kvm xml config files)
• Whonix-Host in ISO mode correctly sets the gw/ws image files permissions in read-only mode
• Whonix-Host in installed (persistent, post-Calamares) mode correctly unsets the gw/ws VMs read-only mode (kvm xml config files)
• Whonix-Host in installed (persistent, post-Calamares) mode correctly sets rw permissions for gw/ws image files

However,

• Installed Whonix-Host in live-mode fails to set correct ro permissions for gw/ws image files (but it does correctly set the gw/ws VMs in read-only mode (kvm xml config files))

Reason: https://github.com/Whonix/whonix-libvirt/blob/master/lib/systemd/system/whonix-libvirt-set-live-to-readonly.service does not provide for setting gw/ws disk images to read-only since it is done during the build only.

On the installed host, this means that rw disk images are never set back to ro (440) permissions in live-mode once the installed host has been booted in persistent mode. Meaning, gw/ws VM are not usable in installed host booted in live-mode.

Solution: adding (ideally when meeting condition: "if rw permission, then...")

chmod --verbose --recursive ugo-w "/var/lib/libvirt/images/Whonix-Gateway.qcow2"
chmod --verbose --recursive ugo-w "/var/lib/libvirt/images/Whonix-Workstation.qcow2"

at the end of https://github.com/Whonix/whonix-libvirt/blob/master/lib/systemd/system/whonix-libvirt-set-live-to-readonly.service

Next problem: this does not work as in live-mode changing a file permission means copying its content into the live overlay... which is impossible due to the size of the disk images...

user@host:~$ sudo service whonix-libvirt-set-live-to-readonly status
● whonix-libvirt-set-live-to-readonly.service - sets all libvirt VM disks to read-only when Live Mode is detected
   Loaded: loaded (/lib/systemd/system/whonix-libvirt-set-live-to-readonly.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2020-04-21 00:19:03 UTC; 1min 2s ago
     Docs: https://github.com/Whonix/whonix-libvirt
  Process: 670 ExecStart=/usr/lib/whonix-libvirt/live-mode-to-read-only (code=exited, status=1/FAILURE)
 Main PID: 670 (code=exited, status=1/FAILURE)

Apr 21 00:18:54 host live-mode-to-read-only[670]: chmod: changing permissions of '/var/lib/libvirt/images/Whonix-Gateway.qcow2': No space left on device
Apr 21 00:18:54 host live-mode-to-read-only[670]: failed to change mode of '/var/lib/libvirt/images/Whonix-Gateway.qcow2' from 0660 (rw-rw----) to 0440 (r--r-----)
Apr 21 00:18:54 host live-mode-to-read-only[670]: ++ error_handler
Apr 21 00:18:54 host live-mode-to-read-only[670]: ++ exit_code=1
Apr 21 00:18:54 host live-mode-to-read-only[670]: + chmod --verbose --recursive ugo-w /var/lib/libvirt/images/Whonix-Workstation.qcow2
Apr 21 00:19:03 host live-mode-to-read-only[670]: mode of '/var/lib/libvirt/images/Whonix-Workstation.qcow2' changed from 0660 (rw-rw----) to 0440 (r--r-----)
Apr 21 00:19:03 host live-mode-to-read-only[670]: + exit 1
Apr 21 00:19:03 host systemd[1]: whonix-libvirt-set-live-to-readonly.service: Main process exited, code=exited, status=1/FAILURE
Apr 21 00:19:03 host systemd[1]: whonix-libvirt-set-live-to-readonly.service: Failed with result 'exit-code'.
Apr 21 00:19:03 host systemd[1]: Failed to start sets all libvirt VM disks to read-only when Live Mode is detected.

Workaround: we could create a systemd service that runs in persistent mode only right before shutdown/reboot and would set back the gw/ws disk images permissions to read-only after each session in persistent mode.

Awesome analysis and description!

In T914#19876, @onion_knight2 wrote:

Next problem: this does not work as in live-mode changing a file permission means copying its content into the live overlay... which is impossible due to the size of the disk images...

Maybe this can be fixed. I.e. change mode of file and at the same time don't copy it all to the live overlay.

https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt has a chapter Metadata only copy up:

When metadata only copy up feature is enabled, overlayfs will only copy up metadata (as opposed to whole file), when a metadata specific operation like chown/chmod is performed. Full file will be copied up later when file is opened for WRITE operation.

In other words, this is delayed data copy up operation and data is copied up when there is a need to actually modify data.

There are multiple ways to enable/disable this feature.
[...]

Could you have a look please?

Workaround: we could create a systemd service that runs in persistent mode only right before shutdown/reboot and would set back the gw/ws disk images permissions to read-only after each session in persistent mode.

That would be OK but this is not my preferred solution. Reason: an unclean shutdown in Whonix installed persistent mode would with a subsequent boot into live mode would result in a failed reboot into Whonix installed live mode.

That would be OK but this is not my preferred solution. Reason: an unclean shutdown in Whonix installed persistent mode would with a subsequent boot into live mode would result in a failed reboot into Whonix installed live mode.

Yes, that is correct.

Maybe this can be fixed. I.e. change mode of file and at the same time don't copy it all to the live overlay.

https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt has a chapter Metadata only copy up:

Very interesting. And guess what... It works :)

Just tried it on installed Whonix-Host 15.0.1.2.7.

In persistent mode, I created a module conf file, then updated the initramfs

echo "options overlay metacopy=on" > /etc/modprobe.d/overlay.conf 
update-initramfs -u

Rebooting in live-mode:

root@host:~# chmod -v 440 /var/lib/libvirt/images/Whonix-Workstation.qcow2
mode of '/var/lib/libvirt/images/Whonix-Workstation.qcow2' changed from 0660 (rw-rw----) to 0440 (r--r-----)

VM start normally!
And cat /sys/module/overlay/parameters/metacopy shows Y instead of N:

root@host:~# cat /sys/module/overlay/parameters/metacopy 
Y

Awesome!

Reminder: let's not forget to still do

Solution: adding (ideally when meeting condition: "if rw permission, then...")

chmod --verbose --recursive ugo-w "/var/lib/libvirt/images/Whonix-Gateway.qcow2"
chmod --verbose --recursive ugo-w "/var/lib/libvirt/images/Whonix-Workstation.qcow2"

at the end of https://github.com/Whonix/whonix-libvirt/blob/master/lib/systemd/system/whonix-libvirt-set-live-to-readonly.service

Also found this warning in the documentation you referred to, is it something we should be afraid of?

Do not use metacopy=on with untrusted upper/lower directories. Otherwise
it is possible that an attacker can create a handcrafted file with
appropriate REDIRECT and METACOPY xattrs, and gain access to file on lower
pointed by REDIRECT. This should not be possible on local system as setting
"trusted." xattrs will require CAP_SYS_ADMIN. But it should be possible
for untrusted layers like from a pen drive.

Great news! I am rebuilding the whole package Host+gw+ws now, excited to test it out! Will report asap.

Seems I have quite a flexible notion of "asap" :)...

Anyway, I have just rebuilt Host+gw+ws, 15.0.1.3.4-12.

Apparently everything went fine during the build, but the VMs don't boot into graphical target (stuck after GRUB bootloader, black screen with blinking white cursor). No output, even without the 'quiet' flag.

Probably a temporary error with this specific build(?).

Changing file permissions back and forth on the installed system still doesn't work. The installed system does not have the /etc/modprobe.d/30_live-config-dist.conf file that is however found on the ISO file.

The metacopy=on parameter is therefore not set on the installed system in live mode, which prevents to modify the VM file permissions (No space left on device):

root@host:~# cat /sys/module/overlay/parameters/metacopy 
N

In T914#20017, @onion_knight2 wrote:

Seems I have quite a flexible notion of "asap" :)...

:)

Apparently everything went fine during the build, but the VMs don't boot into graphical target (stuck after GRUB bootloader, black screen with blinking white cursor). No output, even without the 'quiet' flag.

That's probably because of T950. You'd need to remove both:

quiet loglevel=0

Added some enhancements to package #debug-misc just now. (T950#20021) Included in Whonix 15.0.1.3.5-developers-only.

It would be possible to install package debug-misc. Either,

• we can install it by default on Whonix-Host (and in VMs) for now. But then not forget to disable for stable releases. Cumbersome.
• Or better add install_package_list+=" debug-misc " to build configuration.

Example... If this is your usual build command:

sudo ~/Whonix/whonix_build [...]

Maybe easiest to change to:

sudo install_package_list+=" debug-misc " ~/Whonix/whonix_build [...]

(The extra variable needs to be set after sudo but before whonix_build script.)

Or by setting and environment variable and adding -E to sudo. I.e. sudo -E

export install_package_list+=" debug-misc "
sudo -E ~/Whonix/whonix_build [...]

Not yet tested but will test soon.

The latter I will test now. Let me know which way to set this build variable is the most usable one of if there would be a more sane way to do this.

Please add your build commands to Whonix wiki Dev/Whonix-Host, then I can add suggestion there how to improve these.

Probably a temporary error with this specific build(?).

I don't see why.

Changing file permissions back and forth on the installed system still doesn't work. The installed system does not have the /etc/modprobe.d/30_live-config-dist.conf file that is however found on the ISO file.

That's because /etc/modprobe.d/30_live-config-dist.conf part of package #live-config-dist which is removed during calamares. Maybe that is the bug?

Would it be sane if /etc/modprobe.d options overlay metacopy=on was also set during persistent boot? Could you try that please? (I guess can be tested with any Debian or even Whonix.) If so, I would remove that file from package #live-config-dist and add to package whonix-libvirt which is installed on both, Whonix-Host ISO and also Whonix-Host Installed.

That's probably because of T950. You'd need to remove both:

quiet loglevel=0

I see. But I won't lose time trying to debug this particular build, I will just try a new one and see if the problem persists. Had some problems with lack of space on the VM I am building with, maybe related. Not worth debugging if it's a one time thing. We'll see.

The latter I will test now. Let me know which way to set this build variable is the most usable one of if there would be a more sane way to do this.

I prefer this command, as it is a one-liner:

sudo install_package_list+=" debug-misc " ~/Whonix/whonix_build [...]

Please add your build commands to Whonix wiki Dev/Whonix-Host, then I can add suggestion there how to improve these.

Not sure what you mean here?

That's because /etc/modprobe.d/30_live-config-dist.conf part of package #live-config-dist which is removed during calamares. Maybe that is the bug?

Yes, most probably.

Would it be sane if /etc/modprobe.d options overlay metacopy=on was also set during persistent boot?

Yes, I don't see why it wouldn't. Just tested it on a Debian 10 custom VM which also has live-mode. Works fine. As expected module option is only loaded if the overlay module is loaded.
live-boot -> overlay module loaded -> overlay module option activated
persistent boot -> overlay module not loaded -> overlay module option not activated

Looks sane to me!

Please add your build commands to Whonix wiki Dev/Whonix-Host, then I can add suggestion there how to improve these.

Not sure what you mean here?

The full sudo install_package_list+=" debug-misc " ~/Whonix/whonix_build [...] command. But not important. Maybe I am overthinking this. Maybe not too complex to warrant documenting this on the wiki.

Would it be sane if /etc/modprobe.d options overlay metacopy=on was also set during persistent boot?

Yes, I don't see why it wouldn't. Just tested it on a Debian 10 custom VM which also has live-mode. Works fine. As expected module option is only loaded if the overlay module is loaded.
live-boot -> overlay module loaded -> overlay module option activated
persistent boot -> overlay module not loaded -> overlay module option not activated

Looks sane to me!

Makes sense. Implemented.

• https://github.com/Whonix/live-config-dist/commit/96b6b10c519b47eb29e21c92c01a22043077f018
• https://github.com/Whonix/whonix-libvirt/commit/4b827db5496553d7f71d0ebf003fdfe3ceb1df87

Included in 15.0.1.3.6-developers-only.

Great! Will try to build tomorrow and report back... asap :)

Just built 15.0.1.3.6-developers-only

But forgot to add sudo install_package_list+=" debug-misc "...

Good news: ro/rw mode seems fixed! Correctly set back to 0440 when booting installed Whonix-Host on live-mode. Great!

Bad news: booting virtual machines is very slow (on all boot modes and targets. Tested on KVM only, not real hardware).

They boot but are stuck for a few minutes at

EXT4-fs (vda1): mounted filesystem with ordered data mode.

before reaching the desktop environment.

Whonix-Workstation:

Startup finished in 43.014s (kernel) + 3 min 6.816s (userspace) ⁼ 3min 49.831s

I thought maybe a problem with the video settings, which are set to use "Virtio" (isn't it supposed to be QXL? don't remember). But changing the video model to "QXL" doesn't change anything.

Outside of Whonix-Host Whonix-Workstation 15.0.1.3.6 booted on KVM on my main machine works normally.

Startup finished in 4.069s (kernel) + 8.549s (userspace) = 12.619s 
graphical.target reached after 8.205s in userspace

Did we make any changes lately that could explain this behavior? Maybe some new kernel flag?

Anyway this is not related to this particular topic. Maybe worth a new ticket (or not, if it doesn't happen on real hardware. Not supposed to be working fast in nested virtualization anyway).

But forgot to add sudo install_package_list+=" debug-misc "...

Without that will be hard to gather debug output.

Perhaps virtual serial console?

Bad news: booting virtual machines is very slow (on all boot modes and targets. Tested on KVM only, not real hardware).

Previously wasn't the case? I guess nested virtualization will be unusable slow.

EXT4-fs (vda1): mounted filesystem with ordered data mode.

That message is just a useless message, usability issue, hard to hide by default. From now,when anything goes wrong that line will be cited. Very most likely not the cause. Until a way if found to silence that too.

I thought maybe a problem with the video settings, which are set to use "Virtio" (isn't it supposed to be QXL? don't remember). But changing the video model to "QXL" doesn't change anything.

I don't have an opinion but here are most recent discussions / relevant references that I could find:

• https://forums.whonix.org/t/make-screen-resolution-1920x1080-by-default-for-all-vms/9143/13
• https://www.whonix.org/wiki/Dev/KVM#Virgl3D

Did we make any changes lately that could explain this behavior? Maybe some new kernel flag?

Not that I know.

Anyway this is not related to this particular topic. Maybe worth a new ticket (or not, if it doesn't happen on real hardware. Not supposed to be working fast in nested virtualization anyway).

Yes, worth it. I guess lots of people are going to try Whonix-Host inside a virtual machine before considering installation on real hardware. That's why I even would like to have ability to run Whonix-Host inside VirtualBox.

Please post new tickets in forums as per:
https://forums.whonix.org/t/abolishing-whonix-phabricator-issue-tracker-moving-issue-tracking-to-forums-migrating-phabricator-whonix-org-to-forums-whonix-org/7112

Something left to do here?

Yes, worth it. I guess lots of people are going to try Whonix-Host inside a virtual machine before considering installation on real hardware. That's why I even would like to have ability to run Whonix-Host inside VirtualBox.

Please post new tickets in forums as per:
https://forums.whonix.org/t/abolishing-whonix-phabricator-issue-tracker-moving-issue-tracking-to-forums-migrating-phabricator-whonix-org-to-forums-whonix-org/7112

Ok, see:

https://forums.whonix.org/t/nested-virtualization-unbearably-slow-on-whonix-host-kvm/9550

Something left to do here?

No, I think we're good!