Since you originally added this, do you think you could re-invent it? @HulaHoop

As in the seccomp stuff? I think I can but can you help me find the original topic so I can re-create the testing environment I used back then?

In T631#12242, @HulaHoop wrote:

As in the seccomp stuff?

Yes.

I think I can but can you help me find the original topic so I can re-create the testing environment I used back then?

https://forums.whonix.org/t/cpfpd-data-options-control-port-filter-python-hardening / T128?

As soon as the next dev release (with the working KDE menus) is out I'll build it and start working.

Yes, that would be great and there is still time until the final release.

@Patrick
What do we need for the next dev release for hula?

Are the kde menus things one of the open v14 tickets?

In T631#13769, @JasonJAyalaP wrote:

@Patrick
What do we need for the next dev release for hula?

I guess 14.0.0.2.6 should work.

Can you use https://forums.whonix.org/t/whonix-14-0-0-2-6-developers-only @HulaHoop?

Are the kde menus things one of the open v14 tickets?

The "no KDE start menu at all" bug was solved in 14.0.0.2.6.

Remaining KDE work T636 and T633 aren't blockers to implement this ticket.

I haven't tested it yet and unfortunately I'm very busy these days, so cpfp apparmor work is up for grabs.

@Patrick

Using the hardening broke Tails? What do you mean?

*EDIT*

Do you mean we ported it from Tails to Whonix?

My understanding:

1. Using SystemCallFilter enables a whitelist. The current whitelist isn't inclusive enough (that's the funny thing about whitelists!). And so all the lines with SystemCallFilter are commented out.
2. I'm not sure what the other commented out lines are for.
3. We need to find all the system calls that onion grater needs and add them with SystemCallFilter.

In T631#13827, @JasonJAyalaP wrote:

Do you mean we ported it from Tails to Whonix?

• control-port-filter-python (Whonix 13 version, will be obsoleted/not used(legacy in Whonix 14) was written by @troubadour.
• onion-grater was written by @anonym of Tails.
• Whonix 14 will be using onion-grater which works great.

My understanding:

1. Using SystemCallFilter enables a whitelist. The current whitelist isn't inclusive enough (that's the funny thing about whitelists!). And so all the lines with SystemCallFilter are commented out.

Correct. I did this as a quick fix so onion-grater integration of Whonix could be finished (done).

1. I'm not sure what the other commented out lines are for.

They are all related to hardening.

1. We need to find all the system calls that onion grater needs and add them with SystemCallFilter.

Yes.

Tails didn't feel the need to use system call filtering?

They happily take it if we contribute it.

Question: To install OG in whonix 14 dev, so I simply pull the repo, make deb-icup, stop the old tor control port filter proxy, and start onion grater?

Should be even easier since onion-grater debian/control contains
Replaces: control-port-filter-python. So just installing onion-grater
should do.

sudo service onion-grater status just tells me that it failed to load. Any clues about how to debug this?

See:

sudo journalctl -u onion-grater

Or watch it while it's happening.

sudo journalctl -f -u onion-grater

And in another tab.

sudo systemctl restart onion-grater

Does that show something useful?

Two network interfaces are available? eth0 and eh1?

Replicate what
https://github.com/Whonix/onion-grater/blob/master/lib/systemd/system/onion-grater.service
does.

sudo -u onion-grater /usr/lib/onion-grater

Does that show something useful?

Python is choking on the line:
server = FilteredControlPortProxy(address, FilteredControlPortProxyHandler)

Jul 07 18:32:17 host systemd[1]: Starting Tor control port filter proxy...
Jul 07 18:32:18 host onion-grater[8129]: Traceback (most recent call last):
Jul 07 18:32:18 host onion-grater[8129]:   File "/usr/lib/onion-grater", line 770, in <module>
Jul 07 18:32:18 host onion-grater[8129]:     main()
Jul 07 18:32:18 host onion-grater[8129]:   File "/usr/lib/onion-grater", line 759, in main
Jul 07 18:32:18 host onion-grater[8129]:     server = FilteredControlPortProxy(address, FilteredControlPortProxyHandler)
Jul 07 18:32:18 host onion-grater[8129]:   File "/usr/lib/python3.5/socketserver.py", line 440, in __init__
Jul 07 18:32:18 host onion-grater[8129]:     self.server_bind()
Jul 07 18:32:18 host onion-grater[8129]:   File "/usr/lib/python3.5/socketserver.py", line 454, in server_bind
Jul 07 18:32:18 host onion-grater[8129]:     self.socket.bind(self.server_address)
Jul 07 18:32:18 host onion-grater[8129]: socket.gaierror: [Errno -3] Temporary failure in name resolution
Jul 07 18:32:18 host systemd[1]: onion-grater.service: Main process exited, code=exited, status=1/FAILURE
Jul 07 18:32:18 host systemd[1]: Failed to start Tor control port filter proxy.
Jul 07 18:32:18 host systemd[1]: onion-grater.service: Unit entered failed state.
Jul 07 18:32:18 host systemd[1]: onion-grater.service: Failed with result 'exit-code'.

I noticed that the serive tor-control-port-filter is running (stopping it doesnt make a difference). This is normal?

Probably tor-controlport-filter systemd unit file (the old one) still
running and blocking the onion-grater systemd unit file.

JasonJAyalaP (Jason J. Ayala P.):

I noticed that the serive tor-control-port-filter is running
(stopping it doesnt make a difference). This is normal?

Not normal. That should be removed during the upgrade.

onino-grater's debian/control uses `Replaces:
control-port-filter-python`. That should do. That should remove the
control-port-filter-python package, stops the old tor-controlport-filter
systemd unit file. But I don't remember if I tested this.

sudo service tor-controlport-filter stop
sudo service onion-grater start
same failure
if i try
sudo apt-get remove control-port-filter-python
It wants to remove everything. I don't think 'Replaces' worked.

JasonJAyalaP (Jason J. Ayala P.):

JasonJAyalaP added a comment.

sudo apt-get remove control-port-filter-python
It wants to remove everything. I don't think 'Replaces' worked.

Migration issue.

Selective upgrading of individual packages without using apt-get, by
only using dpkg is hard.

control-port-filter-python cannot be removed because some
anon-meta-package still depends on it - you probably didn't update
anon-meta-packages.

It would probably work if you updated from rebuild local or remote
repository.

sudo service tor-controlport-filter stop
sudo service onion-grater start

Possibly also a migration issue. Not clear to my why that isn't working.
Possibly due to anon-gw-anonyminizer changes. So many things have
changed since.

Still something listening on that local port? Check sudo netstal -tulpen.

Maybe not a great idea to attempt selective package upgrades from this
state. All packages that changed in meanwhile should be rebuild and
reinstalled getting to a newer developers-only state. Either using the
local or a remote repository. A local one certainly goes faster during
debugging.

sudo netstal -tulpen

Port 9051 right? It's listening with python running, and nothing is listening when it's stopped

It would probably work if you updated from rebuild local or remote repository.

The whole system update thing you're writing about on the wiki? Maybe it's time for another Whonix Developers 14 version?

All yes.

https://github.com/Whonix/onion-grater/compare/master...madaidan:patch-1

https://raw.githubusercontent.com/madaidan/onion-grater/aecc6035490fb3b6cd5f1bb768e892344e12dff8/lib/systemd/system/onion-grater.service

Does not work yet. @madaidan

sudo systemctl daemon-reload

sudo systemctl restart onion-grater.service

Job for onion-grater.service failed because a fatal signal was delivered to the control process.
See "systemctl status onion-grater.service" and "journalctl -xe" for details.

sudo journalctl -f

Jun 23 08:19:16 host systemd[1]: Starting Tor control port filter proxy...
Jun 23 08:19:16 host audit[5071]: SECCOMP auid=4294967295 uid=107 gid=113 ses=4294967295 subj==/usr/lib/onion-grater (enforce) pid=5071 comm="onion-grater" exe="/usr/bin/python3.7" sig=31 arch=c000003e syscall=257 compat=0 ip=0x744b70eb64fd code=0x0
Jun 23 08:19:16 host audit[5071]: ANOM_ABEND auid=4294967295 uid=107 gid=113 ses=4294967295 subj==/usr/lib/onion-grater (enforce) pid=5071 comm="onion-grater" exe="/usr/bin/python3.7" sig=31 res=1
Jun 23 08:19:16 host systemd[1]: onion-grater.service: Main process exited, code=killed, status=31/SYS
Jun 23 08:19:16 host systemd[1]: onion-grater.service: Failed with result 'signal'.
Jun 23 08:19:16 host systemd[1]: Failed to start Tor control port filter proxy.

Does it work using this? It looks like it needs the openat syscall which it now allows.

Weirdly, I didn't get an error with it on a fresh install of Whonix.

Unfortunately not. On Qubes-Whonix. Could be Non-Qubes-Whonix vs
Qubes-Whonix?

Jun 23 17:51:58 host systemd[1]: Starting Tor control port filter proxy...
Jun 23 17:51:58 host systemd[9899]: onion-grater.service: Failed to set
up mount namespacing: No such file or directory
Jun 23 17:51:58 host systemd[9899]: onion-grater.service: Failed at step
NAMESPACE spawning /usr/lib/onion-grater-merger: No such file or directory
Jun 23 17:51:58 host systemd[1]: onion-grater.service: Control process
exited, code=exited, status=226/NAMESPACE
Jun 23 17:51:58 host systemd[1]: onion-grater.service: Failed with
result 'exit-code'.
Jun 23 17:51:58 host systemd[1]: Failed to start Tor control port filter
proxy.
Jun 23 17:51:58 host audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295
ses=4294967295 subj==unconfined msg='unit=onion-grater comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Jun 23 17:51:58 host sudo[9894]: pam_unix(sudo:session): session closed
for user root
Jun 23 17:51:58 host audit[9894]: USER_END pid=9894 uid=0
auid=4294967295 ses=4294967295 subj==unconfined
msg='op=PAM:session_close grantors=pam_permit,pam_unix acct="root"
exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
Jun 23 17:51:58 host audit[9894]: CRED_DISP pid=9894 uid=0
auid=4294967295 ses=4294967295 subj==unconfined msg='op=PAM:setcred
grantors=pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=?
terminal=/dev/pts/1 res=success'
Jun 23 17:51:59 host audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295
ses=4294967295 subj==unconfined msg='unit=onion-grater comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 23 17:51:59 host audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295
ses=4294967295 subj==unconfined msg='unit=onion-grater comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 23 17:51:59 host systemd[1]: onion-grater.service: Service
RestartSec=100ms expired, scheduling restart.
Jun 23 17:51:59 host systemd[1]: onion-grater.service: Scheduled restart
job, restart counter is at 1.
Jun 23 17:51:59 host systemd[1]: Stopped Tor control port filter proxy.

Does it work after you comment ProtectSystem=strict and ReadWriteDirectories=? I think on Qubes-Whonix it is trying to write to a directory in /var/run (probably /var/run/qubes-service). I can't test as I don't use Qubes.

If it does work, try adding /var/run/qubes-service to the end of the ReadWriteDirectories= string and restarting it with those options uncommented.

If that doesn't work, try commenting SystemCallFilter=.

Merged your changes.

https://github.com/Whonix/onion-grater/pull/3

https://github.com/Whonix/onion-grater/blob/master/lib/systemd/system/onion-grater.service

Had to add some syscalls.

Had to comment out ProtectSystem=strict and ReadWriteDirectories= indeed.

https://github.com/Whonix/onion-grater/commit/df49bbe15ef0a0589eb83a9a16592644a31b955f

Otherwise it would fail with the following:

Jul 01 07:59:59 host systemd[3378]: onion-grater.service: Failed to set up mount namespacing: No such file or directory
Jul 01 07:59:59 host systemd[3378]: onion-grater.service: Failed at step NAMESPACE spawning /usr/lib/onion-grater-merger: No such file or directory

In T631#18621, @madaidan wrote:

I think on Qubes-Whonix it is trying to write to a directory in /var/run (probably /var/run/qubes-service). I can't test as I don't use Qubes.

Nothing that onion-grater is doing writes to either.

It needs to read:

DEFAULT_COOKIE_PATH = '/run/tor/control.authcookie'
DEFAULT_CONTROL_SOCKET_PATH = '/run/tor/control'

That's weird. Onion-grater is trying to write to somewhere that's being mounted read-only by systemd.

Does ProtectSystem=full work? If so, we can use that to mount some areas of the filesystem read-only. It isn't as complete as ProtectSystem=strict would be though.

I just re-read the error message. Try adding

ProtectSystem=strict
ReadWriteDirectories=/usr/lib/onion-grater-merger

madaidan (madaidan):

ReadWriteDirectories=/usr/lib/onion-grater-merger

It's a file, not a folder. Nothing in the code of
/usr/lib/onion-grater-merger writes to /usr/lib/onion-grater-merger.

It's a file, not a folder.

Then use ReadWritePaths=/usr/lib/onion-grater-merger instead.

Nothing in the code of /usr/lib/onion-grater-merger writes to /usr/lib/onion-grater-merger.

It probably tries to create it which is counted as writing.

https://github.com/Whonix/onion-grater/blob/master/lib/systemd/system/onion-grater.service currently works without ReadWritePaths. So let's not add?

Also while /usr/lib/onion-grater-merger is executed, there is no source code in /usr/lib/onion-grater-merger that would write to /usr/lib/onion-grater-merger.

Even there is no source code to read /usr/lib/onion-grater-merger except if execution by systemd itself also required read permission.

https://github.com/Whonix/onion-grater/blob/master/lib/systemd/system/onion-grater.service currently works without ReadWritePaths. So let's not add?

It uses ProtectSystem=full which only mounts /usr, /boot and /etc as read-only. Using ProtectSystem=strict will mount the entire filesystem as read-only except API filesystems.

It would be better to use strict and find out which files exactly it needs and make only those read-write.

Error back after reboot.

Jul 07 21:32:22 host systemd[23920]: onion-grater.service: Failed to set up mount namespacing: No such file or directory
Jul 07 21:32:22 host systemd[23920]: onion-grater.service: Failed at step NAMESPACE spawning /usr/lib/onion-grater-merger: No such file or directory

The reason why I can reproduce this might be because Qubes starts with clean root image /run, /etc and so forth. Each start of onion-grater is like starting onion-grater for the first time. (Since onion-grater in AppVM which has only persistent /home [...] while onion-grater never starts in TemplateVM.

ReadWriteDirectories=/etc/onion-grater.d/ is bad anyhow.
Not onion-grater (the network daemon) is writing there.
Only the ExecStartPre=/usr/lib/onion-grater-merger (which generates from onion-grater-merger.d config folders the /etc/onion-grater.d/30_autogenerated.yml config file). onion-grater-merger is an offline tool.

Can we exclude ExecStartPre=/usr/lib/onion-grater-merger from systemd hardening?

Can we exclude ExecStartPre=/usr/lib/onion-grater-merger from systemd hardening?

Yes.

https://github.com/Whonix/whonix-gw-network-conf/commit/95198d2d8916b6e34675876320404bb970720cb1
https://github.com/Whonix/whonix-gw-network-conf/commit/dfdf6c776bb153f4f529277287d131449d3775dc

That helps.

It's really hard to debug if systemd does not say that actual folder which is missing. Someone reported a bug for this issue:

https://github.com/systemd/systemd/issues/12974

Yay, we have ProtectSystem=strict now.

https://github.com/Whonix/onion-grater/blob/master/lib/systemd/system/onion-grater.service

Could you please "reboot" SystemCallFilter? Empty it and repopulate from scratch? Now that onion-grater-merger is unconfined, onion-grater will need a lot less syscalls.

Yay, we have ProtectSystem=strict now.

Great!

Could you please "reboot" SystemCallFilter? Empty it and repopulate from scratch? Now that onion-grater-merger is unconfined, onion-grater will need a lot less syscalls.

The original list was gotten via strace -cf /usr/lib/onion-grater which didn't include syscalls for onion-grater-merger anyway. Removing the syscalls added with this commit and the openat syscall should be all we need. We probably won't need to remove anything anyway.

Removed a few. Would not start without openat, so kept.

https://github.com/Whonix/onion-grater/commit/630e6bd0a2423cdf6a430439b538d88cac9da155