Wrt /bin/sync hangs forever in whonix-ws-dvm #1328 https://github.com/QubesOS/qubes-issues/issues/1328#issuecomment-147951214, where would be the proper place to implement this? The mount-home.sh https://github.com/QubesOS/qubes-core-agent-linux/blob/master/vm-systemd/mount-home.sh#L77-L89 script would become a mount-dirs.sh script?

Good idea.

mount-home.sh currently also processes /rw/usrlocal. Another reason to rename it to mount-dirs.sh.

Related to Bind mount /rw/usrlocal -> /usr/local instead of symlink:
https://github.com/QubesOS/qubes-issues/issues/1150

Where should this feature be configureable?

a) from TemplateVM only -> /etc/qubes-bind-dirs.d/, or
b) from TemplateBasedVM only -> /rw/config/qubes-bind-dirs.d/, or
c) both, a) and b)?

I think both.

Somehow offtopic: for per-VM configuration would you prefer /rw/config
(Qubes-specific path), or /usr/local/etc (somehow more standard)?
Maybe those two should be merged (using symlink)?

marmarek (Marek Marczykowski-Górecki):

I think both.

Ok.

Somehow offtopic: for per-VM configuration would you prefer /rw/config
(Qubes-specific path), or /usr/local/etc (somehow more standard)?
Maybe those two should be merged (using symlink)?

Hm. This is an important decision. I am always having in mind and
hoping, that some day, packages such as qubes-core-agent etc. will be
uploaded to the official Debian archive. [So distributions such as Tails
could simply install it.]

Neither. /rw/ clearly violates FHS. /usr/local/etc/ looks even more
strange. Debian policy forbids packages to write into local.

What about /home/config/? Hehe. No. Same.

What about something like /etc/qubes/rw/?

Compatibility symlink [or better, if possible, bind-mount] is a good idea.

Let's also ask at least another person. @bnvk

Also imho low priority, but okay to change.

Hm. This is an important decision. I am always having in mind and
hoping, that some day, packages such as qubes-core-agent etc. will be
uploaded to the official Debian archive. [So distributions such as Tails
could simply install it.]

I don't that would happen, at least not anytime soon. There are
multiple problems with that. For example need for different version
based on Qubes version, even on the same Debian release. But also a lot
of work to not break the system when installed in non-Qubes environment
(where services like qubesdb, qrexec-agent etc would fail to start).

Also imho low priority, but okay to change.

Yes.

In T414#7005, @Patrick wrote:

Where should this feature be configureable?

a) from TemplateVM only -> /etc/qubes-bind-dirs.d/, or

b) from TemplateBasedVM only -> /rw/config/qubes-bind-dirs.d/, or

c) both, a) and b)?

In T414#7006, @marmarek wrote:

I think both.

Wrong/non-existing entries should not make the script fail, right?

I guess, parse /etc/qubes-bind-dirs.d/ first (lower priority), parse /rw/config/qubes-bind-dirs.d/afterwards (higher priority)?

Should higher named/priority files (example: 50_user) in qubes-bind-dirs.d be capable to disable items by lower priority files (example: 30_qubes)?

I assume yes for now.

This is my far from prototype test script for now.

#!/bin/bash

#set -x
set -e

bind_directories+=('test file')
bind_directories+=('another one')
bind_directories+=('bar test')

delete() {
   bind_directories=( "${bind_directories[@]/'another one'}" )
}

#delete

array_length="${#bind_directories[@]}"
i=-1

while true; do
   i=$(( i + 1))

   if [ "$i" -ge "$array_length" ]; then
      break
   fi

   if [ "${bind_directories[$i]}" = "" ]; then
      ## Fix empty or removed ones.
      continue
   fi
   echo "${bind_directories[$i]}"

done

In the qubes-bind-dirs.d config folders, packages or users could drop bash snippets in the following form.

For adding:

bind_directories+=('test file')
bind_directories+=('another one')
bind_directories+=('bar test')

Or the more difficult case of removal:

bind_directories=( "${bind_directories[@]/'another one'}" )

Wrong/non-existing entries should not make the script fail, right?

Right. Maybe entries like /etc/something -> /rw/config/something
should try to create /rw/config/something? Or even initialize with
original /etc/something content? That would make per-VM configuration
of virtually any service really easy.

I guess, parse /etc/qubes-bind-dirs.d/ first (lower priority), parse /rw/config/qubes-bind-dirs.d/afterwards (higher priority)?

Yes. Or even follow systemd practice and have
/usr/lib/qubes-bind-dirs.d with lowest priority? Not sure about that.

Should higher named/priority files (example: 50_user) in qubes-bind-dirs.d be capable to disable items by lower priority files (example: 30_qubes)?

I assume yes for now.

I think 30_qubes in higher priority dir should override 30_qubes in
lower priority dir.

If we'd have /usr/lib/qubes-bind-dirs.d (or simply
/usr/lib/qubes-bind-dirs?), then user may override it with file in
either /etc (Template wide), or /rw/config (only one VM).

marmarek (Marek Marczykowski-Górecki):

marmarek added a comment.

> 1. Wrong/non-existing entries should not make the script fail, right?

Right. Maybe entries like `/etc/something` -> `/rw/config/something`
should try to create `/rw/config/something`?

Sorry, I am not getting it.

You mean it should be a simpler config file format such as
/etc/something1
/etc/something2
/etc/something3
?

Not using the variable+= syntax?

Not sure if I could implement that. Perhaps. Then the config parsing
code would be more complicated than a simple source.

Or even initialize with

original `/etc/something` content? That would make per-VM configuration
of virtually any service really easy.

I am not following either.

> 2. I guess, parse `/etc/qubes-bind-dirs.d/` first (lower priority), parse `/rw/config/qubes-bind-dirs.d/`afterwards (higher priority)?

Yes. Or even follow systemd practice and have
`/usr/lib/qubes-bind-dirs.d` with lowest priority? Not sure about that.

Can do. Only minimally more code. (One more entry in a for loop.)

> 3. Should higher named/priority files (example: `50_user`) in `qubes-bind-dirs.d` be capable to disable items by lower priority files (example: `30_qubes`)?
> 
>   I assume yes for now.

I think `30_qubes` in higher priority dir should override `30_qubes` in
lower priority dir.

Yes. Higher priority dirs always overwrite lower priority dirs
independently of lexical order. (That's seemingly happening mostly
nowadays.)

If we'd have `/usr/lib/qubes-bind-dirs.d` (or simply
`/usr/lib/qubes-bind-dirs`?),

With '.d' is better.

then user may override it with file in
either `/etc` (Template wide), or `/rw/config` (only one VM).

Sounds good.

I was thinking about simple config format, very similar to the current list:

/rw/config/something1:/etc/something1
/rw/config/something2:/etc/something2
/rw/config/something3:/var/lib/something3

Then load them with something like:

bind_dirs=""
for f in $(ls /usr/lib/qubes-bind-dirs.d /etc/qubes-bind-dirs.d /rw/config/qubes-bind-dirs.d | sort | uniq); do
  if [ -r "/rw/config/qubes-bind-dirs.d/$f" ]; then
    bind_dirs+=$(grep -v '^#' "/rw/config/qubes-bind-dirs.d/$f")
  elif [ -r "/etc/qubes-bind-dirs.d/$f" ]; then
    bind_dirs+=$(grep -v '^#' "/etc/qubes-bind-dirs.d/$f")
  elif [ -r "/usr/lib/qubes-bind-dirs.d/$f" ]; then
    bind_dirs+=$(grep -v '^#' "/usr/lib/qubes-bind-dirs.d/$f")
  else
    echo "WAT?!"
    exit 1
  fi
done

# Then process loaded entries:
for entry in ${bind_dirs}; do
 ...
done

Maybe better make ${bind_dirs} an array to better handle spaces. You're much better bash wizard than me, you'll do it right :)

This is what I have for now.

no longer requires rsync
now works also for files, not just folders
sync removed, should not be required
sorted out user access rights issues

A detailed review is not required yet. It's not fully tested yet. Only a glimpse would be nice.

However, I am wondering about the following...

1) What to do about symlinks? Copying them as symlink is possible and implemented. But mount fails. Example.

mount --bind /rw/bind-dirs/etc/hosts /etc/hosts

mount: special device /rw/bind-dirs/etc/hosts does not exist

So we could just skip mounting for symlinks. The symlinks that are copied to /rw/bind-dirs do link the the proper files in the system.

But then modifications to for example /etc/some-sym-link would not end up in /rw/bind-dirs.

Maybe bind-dirs should delete the old symlink /etc/some-sym-link. And create a new symlink for its place to /rw/bind-dirs/etc/some-sym-link?

The same in other words.

Originalls /etc/hosts is a symlink to /etc/hosts.anondist.
Now the user adds /etc/hosts to bind-dirs configuration.
bind-dirs will
"unlink /etc/hosts"
"ln -s /etc/hosts.anondist /rw/bind-dirs/etc/hosts"
"ln -s /rw/bind-dirs/etc/hosts /etc/hosts."

(Cannot mv symlinks - would not work for relative symlinks.)

A two layered symlink. But I guess that could cause some issues.

Another option would be follow the symlink. Copy the file it links to to /rw/bind-dirs. Then mount it as usual. However, I don't know if that could also cause issues. Perhaps if any scripts check if they got the symlink or otherwise. Probably less issues than above.

2) What about file system objects added to bind-dirs list, that do not exist? Fail closed, abort, exit error? Fail silent, open? Write to stderr, continue, fail open?

3) What about other file system objects such as block, character, fifo, special, device, ... (?)?

Posting the old notes by @nrgaway here.

'bind-directories' will automatically bind important Whonix configuration
directories within root image filesystem to the users /rw/ filesystem so the
changes will persist upon reboot eliminating the need to have Whonix be a
standalone VM.

The first time an AppVM is started the original content from the root image
are copied to the users /rw directory (one-time) to make sure any changes
that have already been made in template are reflected in AppVM.

Any further changes, like enabling 'Tor' are stored directly in the 'rw'
filesystem.  For example '/etc/tor' is bound to '/rw/Whonix/etc/tor'.

If a user then makes any changes in the TemplateVM, those changes will not
be reflected in the AppVM after the initial bind which may cause confusion
if a user is attempting to make configuration changes in the TemplateVM.

Possible solutions:
  - track original date / shasum of files when originally copying defaults to /rw
  - store that in /rw as .bind-dirs/filename ?
  - before binding; check for changes; update rw if there are changes; maybe
    just using file date would work?

In T414#7386, @Patrick wrote:

Posting the old notes by @nrgaway here.

'bind-directories' will automatically bind important Whonix configuration
directories within root image filesystem to the users /rw/ filesystem so the
changes will persist upon reboot eliminating the need to have Whonix be a
standalone VM.

The first time an AppVM is started the original content from the root image
are copied to the users /rw directory (one-time) to make sure any changes
that have already been made in template are reflected in AppVM.

Any further changes, like enabling 'Tor' are stored directly in the 'rw'
filesystem.  For example '/etc/tor' is bound to '/rw/Whonix/etc/tor'.

If a user then makes any changes in the TemplateVM, those changes will not
be reflected in the AppVM after the initial bind which may cause confusion
if a user is attempting to make configuration changes in the TemplateVM.

Right. I recognize the issue. We're adding another layer of complexity. Not great. Likely to cause confusion. However, I don't think there can be a solution to this issue. What we could consider would be something more drastic. [1]

Possible solutions:
  - track original date / shasum of files when originally copying defaults to /rw
  - store that in /rw as .bind-dirs/filename ?
  - before binding; check for changes; update rw if there are changes; maybe
    just using file date would work?

Okay, let's say you somehow found a modification. Was it done by the user or by the package manager? To my knowledge, there is no way to reliably distinguish this. But let's put that aside for now. Let's say the user made a modification. Now, we might have two modified versions. For example a modified /etc/tor/torrc in the Whonix-Gateway TemplateBasedProxyVM, and a different modified /etc/tor/torrc inside the whonix-gw TemplateVM. Which changes we keep, which ones we discard? Do we [silently] discard settings that were made in the TemplateBasedVM? Sounds confusing. Ask the user to merge them? Probably not. Do we try something as crazy as automatically merging them? Probably not. I don' think this is solvable, but I am happy to hear ideas.

What however might be doable, would be the following.

somehow track the file hashes of the files from non-persistent
track the file hashes of rw once we initially copy them over from non-persistent to rw
notice changes in the non-persistent folders (after TemplateVM upgrades)
now if the file in rw is still unmodified, we could copy over the modified version from non-persistent to rw to update the stale one on rw

But I think instead of making things easier, it would only add even more complexity and confusion.

[1] two different drastic ideas that do not rely on bind-directories

1) Making Whonix-Gateway ProxyVM a standalone VM by default. Not great for backups. Would waste space using Qubes usual backups. But while Whonix-Gateway backups are useful - keeping Tor entry guards; keeping Tor/Firewall config, perhaps keeping misc other config - is not super important. And also not well documented/encouraged at the moment.

2) Configure/hack Tor running as user user rather than system user debian-tor. Then Tor persistent data would be stored in the user's home folder. Other stuff such as Whonix Firewall settings are much less problematic and could also be stored in the user's home folder instead.

Maybe these are some interesting future directions. If this sounds interesting, we can move that discussion into a separate ticket.

(Even in case we decided to not use bind-directories by default it would be useful to finish this tool even if it will not be used by default.)

[1] two different drastic ideas that do not rely on bind-directories

- 1) Making Whonix-Gateway ProxyVM a standalone VM by default. Not great for backups. Would waste space using Qubes usual backups. But while Whonix-Gateway backups are useful - keeping Tor entry guards <https://www.torproject.org/docs/faq.html.en#EntryGuards>; keeping Tor/Firewall config, perhaps keeping misc other config - is not super important. And also not well documented/encouraged at the moment.

It isn't only about backups. It's also about disk space on actual
system.

2) Configure/hack Tor running as user user rather than system user debian-tor. Then Tor persistent data would be stored in the user's home folder. Other stuff such as Whonix Firewall settings are much less problematic and could also be stored in the user's home folder instead.

This doesn't solve anything. Still that config will initialized from
somewhere, then stored in home directory and not updated (from initial
source) ever. So basically you have the same "problem" as with bind
directory, but moved to home.

I think the one time synchronization (at the first VM start time) is the way
to go.

marmarek (Marek Marczykowski-Górecki):

This doesn't solve anything. Still that config will initialized from
somewhere, then stored in home directory and not updated (from initial
source) ever. So basically you have the same "problem" as with bind
directory, but moved to home.

Right. But such an implementation wouldn't require the bind-dirs script.
And from perspective of a user trying to grasp this, it's a bit simpler,
I suppose.

marmarek (Marek Marczykowski-Górecki):

It isn't only about backups. It's also about disk space on actual
system.

Yes. However, in case of the Whonix-Gateway ProxyVM, I don't think one
needs that many of them. Creating a TemplateBased one doens't give that
many advantages. It's just easier purging them and creating a new one.
In case of the workstation however, TemplateBased VMs are more useful.

Implemented a simple way to handle symlinks pointing to files.

see where it really links to
unlink the original symlink
copy the target where it really links to
then handle that file normally

https://github.com/adrelanos/qubes-core-agent-linux/commit/4b46a658dbe0e0d7b6e11a8625161525cf82e60f

Undone the above approach. Trying to implement the same for symlinks pointing to directories was a mess. (And writing to rw was strange anyhow.)

Using an even simpler mechanism now. Resolving where there symlink points to, and using for bind mount instead.

https://github.com/adrelanos/qubes-core-agent-linux/commit/eed39067adc24d825d124eb12d91bd65a61e65ee

https://github.com/marmarek/qubes-core-agent-linux/pull/58

Optional debug script that may be useful.

#!/bin/bash
set -x
set -e

umount /etc/testdir || true
umount /etc/testsymlink || true
rm -r /etc/testsymlink || true
rm -r /etc/testdir || true
mkdir /etc/testdir
echo test > /etc/testdir/a
ln -s /etc/testdir /etc/testsymlink
ls -la /etc/testsymlink
ls -la /etc/testsymlink/

sleep 1

sudo ./misc/bind-dirs 1
sudo rm -r /rw/bind-dirs
sudo ./misc/bind-dirs
ls -la /etc/hosts
ls -la /rw/bind-dirs/etc/hosts

upstream bind-directories functionality to Qubes is done. Follow up task: T501

upstream bind-directories functionality to QubesClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

upstream bind-directories functionality to Qubes
Closed, ResolvedPublic
Actions