I was trying obfs4proxy in whonix-gateway. I editted the torrc to UseBridges 1 and added the Client Transport line (note, torrc.examples says to add "managed" at the end; https://github.com/Yawning/obfs4 does not). I then added bridges from tpo (bridge obfs4 ip ... ).
Whonixcheck reports WARNING can't connect to bridge REASON=PT_MISSING
PT_Missing is an error from stem: "no pluggable transport was available"

What did I do wrong here?

Is the obfs4proxy package installed? Probably yes.

What did I do wrong here?

From a users pov, probably nothing. Then the ticket description might contain the right guess why this is happening. Tor probably thinks the pluggable transport is not available, because AppArmor prevents execution. TODO: somehow sort out the apparmor mess.

Whonixcheck reports

Or jump right to Tor:

https://www.whonix.org/wiki/Tor#Log_Analysis

You're right. /var/run/tor/log reports
"Could not launch managed proxy executable /usr/bin/obfs4proxy Operation not permitted"

Tor's own app armor profile breaks needed features (obs4). The ticket is 4 years old with no progress. Even they complained about needed to resolve or remove it (years ago).

That's why we need to sort it out in https://github.com/Whonix/apparmor-profile-anondist/blob/master/etc/apparmor.d/abstractions/base.anondist somehow.

Which app armor profile is blocking obfs4? Something from us or an apparmor profile that comes from tpo?

In T676#13825, @JasonJAyalaP wrote:

Which app armor profile is blocking obfs4?

/etc/apparmor.d/system_tor

To figure this out, keep watching logs:

https://forums.whonix.org/t/whonix-14-debian-stretch-apparmor-related-changes

Something from us or an apparmor profile that comes from tpo?

dpkg -S /etc/apparmor.d/system_tor

tor: /etc/apparmor.d/system_tor

/etc/apparmor.d/system_tor is unmodified, owned by Debian tor packabe. /etc/apparmor.d/system_tor Will #include <local/system_tor>.

/etc/apparmor.d/local/system_tor (/etc/apparmor.d/local/system_tor.anondist) has been taken over by Whonix using config-package-dev displace. Actually
/etc/apparmor.d/local/system_tor is where the fix belongs.

https://github.com/Whonix/anon-gw-anonymizer-config/blob/master/etc/apparmor.d/local/system_tor.anondist

(https://github.com/Whonix/anon-gw-anonymizer-config/blob/master/etc/apparmor.d/local/usr.bin.obfsproxy.anondist is unrelated, because it is obfsproxy, not obfs4proxy.)

To be clear:
Tor ships a broken apparmor profile (for the last 5 years? Suggested nuke of the profile 3 years ago), and we're trying to unbreak obfs4, correct?

Yes. Because the other solution "not use AppArmor for Tor" is not a great one. It worked in Whonix 13, just needs to be fixed for Whonix 14.

Please either fix it in Whonix and/or by submitting the appropriate patches to Debian.

AA doesn't report a denied message when tor tries to launch obfs4. However:

the line

/usr/bin/obfs4proxy PUx,

in
/etc/apparmor.d/abstractions/tor
when changed to

/usr/bin/obfs4proxy rix,

makes obfs4 work again. Maybe it should Ux or ix. I'm not sure. But what I really don't know is how system_tor interacts with abstractions/

But what I really don't know is how system_tor interacts with abstractions/

cat /etc/apparmor.d/system_tor
# vim:syntax=apparmor
#include <tunables/global>

profile system_tor flags=(attach_disconnected) {
  #include <abstractions/tor>

And by the way... Also does...

  # Site-specific additions and overrides. See local/README for details.
  #include <local/system_tor>
}

/etc/apparmor.d/system_tor after #include <abstractions/tor> and #include <local/system_tor> will be interpreted like the following, I think:

# vim:syntax=apparmor
#include <tunables/global>

profile system_tor flags=(attach_disconnected) {
  #include <abstractions/base>
  #include <abstractions/nameservice>

  network tcp,
  network udp,

  capability chown,
  capability dac_override,
  capability fowner,
  capability fsetid,
  capability setgid,
  capability setuid,

  /usr/bin/tor r,
  /usr/sbin/tor r,

  /proc/sys/kernel/random/uuid r,
  /sys/devices/system/cpu/ r,
  /sys/devices/system/cpu/** r,

  /etc/tor/* r,
  /usr/share/tor/** r,

  /usr/bin/obfsproxy PUx,
  /usr/bin/obfs4proxy PUx,

  owner /var/lib/tor/** rwk,
  owner /var/lib/tor/ r,
  owner /var/log/tor/* w,

  # During startup, tor (as root) tries to open various things such as
  # directories via check_private_dir().  Let it.
  /var/lib/tor/** r,

  /{,var/}run/tor/ r,
  /{,var/}run/tor/control w,
  /{,var/}run/tor/socks w,
  /{,var/}run/tor/tor.pid w,
  /{,var/}run/tor/control.authcookie w,
  /{,var/}run/tor/control.authcookie.tmp rw,
  /{,var/}run/systemd/notify w,

  ## Anonymity Distributions
  /etc/hosts.anondist r,
  /etc/resolv.conf.anondist r,
  /run/tor r,
  /run/tor/log rwk,

## Add permissions for obfsproxy and flashproxy.
## AUDIT:
## These profile rules may be too permissive. Needs audit and someone dedicated
## to work on AppArmor profiles. It's not a serious security issue in any case,
## because AppArmor isn't enabled by default in Debian stretch yet and little
## work is being done on it at time of writing. So it's a lax AppArmor profile
## versus no AppArmor at all.

  ## obfsproxy
  /usr/local/lib/python*/** r,
  /var/log/tor/log rw,
  /dev/urandom r,
  /dev/random r,
  /usr/** r,
  /etc/python*/sitecustomize.py r,
  ## https://forums.whonix.org/t/after-last-apt-get-upgrade-gateway-doesnt-connect-to-tor-anymore
  #/usr/bin/obfsproxy rix,

  ## flashproxy
  @{HOME}/.tb/tor-browser/App/flashproxy-client rix,
  @{HOME}/.tb/tor-browser/App/flashproxy-reg-appspot rix,
  @{HOME}/.tb/tor-browser/App/flashproxy-reg-url rix,
  @{HOME}/.tb/tor-browser/App/flashproxy-reg-http rix,
  @{HOME}/.tb/tor-browser/App/flashproxy-reg-email rix,
  @{HOME}/.tb/tor-browser/App/** rm,
  /usr/bin/python* rix,
  /usr/lib/python*/dist-packages/** m,
  /usr/lib/python*/lib-dynload/** m,
  /usr/lib/pyshared/python*/OpenSSL/** m,
  /usr/lib/python*/lib-dynload/** m,
  /usr/lib/pyshared/python*/numpy/** m,
  /proc/*/mounts r,
}

Ah. I didn't see the include. Makes sense.

PUx + those obfsproxy specific rules work fine.
Changing obfs4 to rix works fine.

Questions:

1. Is "include abstractions/tor" normal and desirable behavour? Is that something tor does or we do? Shouldn't all our changes be in .anondist?
2. I don't fully understand apparmors execution persions (http://wiki.apparmor.net/index.php/QuickProfileLanguage#Execute_permissions). But if obfsproxy is being executed as "unconfined" (PUx), are those obfsproxy specific permissions (under AUDIT) even doing anything? They were with rix, I believe, but not currently. Actually, the P means it will look for a obfsproxy specific profile. So we certainly don't need both.
3. I didn't fully understand the problem people were getting with obfsproxy rix. I wonder if it can be fixed (or if we should just run it unconfined.
4. I have *no clue* why a more restrictive exection permission (rix) would allow tor to run obfs4proxy, but a less restrictive one (PUx) would not.
5. That apparmor page isn't clear if capital PU is prefered for productive and not only development (I'm not sure what "scrub the environment" means)

In T676#13872, @JasonJAyalaP wrote:

Ah. I didn't see the include. Makes sense.

I mentioned it in T676#13830.

1. Is "include abstractions/tor" normal and desirable behavour?

Yes, many profiles do that.

Is that something tor does or we do?

Tor should do.

Shouldn't all our changes be in .anondist?

Ideally, yes in local/name.anondist. Seems most appropriate where distributions should add their overrides in absence of something better.

1. I don't fully understand apparmors execution persions (http://wiki.apparmor.net/index.php/QuickProfileLanguage#Execute_permissions). But if obfsproxy is being executed as "unconfined" (PUx), are those obfsproxy specific permissions (under AUDIT) even doing anything? They were with rix, I believe, but not currently. Actually, the P means it will look for a obfsproxy specific profile. So we certainly don't need both.
2. I didn't fully understand the problem people were getting with obfsproxy rix. I wonder if it can be fixed (or if we should just run it unconfined.

Running obs(4)proxy unconfined should be good enough for now until an obfs4proxy profiles is provided. We could contribute this as an enhancement later.

1. That apparmor page isn't clear if capital PU is prefered for productive and not only development (I'm not sure what "scrub the environment" means)

Environment variables. Should be preferred if possible, although environment should be clean anyhow since started by systemd.

In this case, a /local file can probably not do the trick.

If we run unconfined, the following is probably alright to remove.

## obfsproxy
/usr/local/lib/python*/** r,
/var/log/tor/log rw,
/dev/urandom r,
/dev/random r,
/usr/** r,
/etc/python*/sitecustomize.py r,
## https://forums.whonix.org/t/after-last-apt-get-upgrade-gateway-doesnt-connect-to-tor-anymore
#/usr/bin/obfsproxy rix,

Since /usr/bin/obfs4proxy PUx, is included by Debian tors /etc/apparmor.d/abstractions/tor, upstream has probably attempted to fix the apparmor profile thing. Since you don't get any apparmor warnings, the issue is not apparmor, but systemd seccomp whitelist.

Yes. Something in /lib/systemd/system/tor@default.service is causing this.

#NoNewPrivileges=yes
#PrivateTmp=yes
#PrivateDevices=yes
#ProtectHome=yes
#ProtectSystem=full
#ReadOnlyDirectories=/
#ReadWriteDirectories=-/proc
#ReadWriteDirectories=-/var/lib/tor
#ReadWriteDirectories=-/var/log/tor
#ReadWriteDirectories=-/var/run
#CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_NET_BIND_SERVICE CAP_DAC_OVERRIDE

Next steps. Please:

• figure out which one
• reproduce this on plain non-Whonix Debian
• report a bug against Debian's tor package (if there isn't already one)
• add a systemd drop-in .d file for tor's systemd unit as our Whonix specific override (to be included in package anon-gw-anonymizer-config)

To save you from somehow learning about systemd overrides the hard way...

If for example in line CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_NET_BIND_SERVICE CAP_DAC_OVERRIDE the CAP_DAC_OVERRIDE is the issue... Create a file...

/lib/systemd/system/tor@default.service.d/40_obfs4proxy-workaround.conf

[Service]
CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_NET_BIND_SERVICE

This *could* be the solution then. Just to show the general syntax of systemd override files.

It's

nonewprivileges=yes

Comment that and obfs4proxy can run as PUx (instead of needing ix)

from: https://www.freedesktop.org/software/systemd/man/systemd.exec.html#NoNewPrivileges=

NoNewPrivileges=

Takes a boolean argument. If true, ensures that the service process and all its children can never gain new privileges through execve() (e.g. via setuid or setgid bits, or filesystem capabilities). This is the simplest and most effective way to ensure that a process and its children can never elevate privileges again.

Is it the nature of obfs4 to require "gaining new privileges"? If so, we have to remove this line or change the execution permissions of obfs4 in apparmor. (btw, I haven't been able to debug tor without apparmor. If I aa-disable the tor profile, tor fails to load.)

I commented out the lines in local/system_tor about obfsproxy. This caused obfsproxy to fail. Changing obfsproxy to rix didn't work. But I'm confused at what I'm seeing, and so I'm still looking at it.

Pux (already Tor's default) is alright.

In T676#13940, @JasonJAyalaP wrote:

Is it the nature of obfs4 to require "gaining new privileges"?

I have no idea.

If so, we have to remove this line or change the execution permissions of obfs4 in apparmor.

(btw, I haven't been able to debug tor without apparmor. If I aa-disable the tor profile, tor fails to load.)

aa-disable does not apply to Tor's systemd service, because it's systemd unit file enforces apparmor independent from aa-disable / aa-enforce command line tools.

/lib/systemd/system/tor@default.service:

AppArmorProfile=system_tor

Two things work:

1. Changing obfs4 execution permission in system_tor apparmor profile (abstractions/tor) from PUx to ix.

2. Keeping PUx but removing "NoNewPrivileges" from tor@default systemd service (/lib/systemd/system)

I have no idea why PUx fails without an AA message with NNP enabled, but works with ix with NNP enabled. I don't know how they interact. I'm not even sure who to ask? Obfs4 for a patch? AA guys for an explanation about how PUx interacts with NNP? Systemd guys for debugging NNP? How much does it matter? Debian people about why they have NNP in their tor service?

What do you want me to do next?

JasonJAyalaP (Jason J. Ayala P.):

JasonJAyalaP added a comment.

Two things work:

1. Changing obfs4 execution permission in system_tor apparmor profile

(abstractions/tor) from PUx to ix.

2. Keeping PUx but removing "NoNewPrivileges" from tor@default

systemd service (/lib/systemd/system)

Please report this analysis at the Debian tracker for the tor package.
That's the right place and the right maintainer to figure out. Debian
bug reporting policy even recommends to report bugs against Debian,
because then maintainers will contact upstream of Debian.

(Sometimes we ignore this - when Debian maintainers don't do it, are
slow, or somehow it does not matter. For example, why should one request
a new add-on api for firefox from Debian if that can be requested at
Mozilla's tracker. But in this case, the apparmor profile and systemd
unit file is by Debian, not by Tor Project, I think.)

Meanwhile, until upstream sorts this out, please add a systemd .d
*drop-in* as mentioned above as temporary workaround disabling it.

[Service]
NoNewPrivileges=

Plus with ## a short comment and a link to this ticket to justify the
change (even to ourselves if we look into it again in future).

(If that does not work: Try NoNewPrivileges=No.)

I prefer this solution, because it's the most lightweight and least
intrusive workaround, most likely stable one as well. We can use a
drop-in and don't need to modify any files owned by upstream/third party
packages.

Once upstream has sorted that, we need to remove our workaround.

Ok I created the workaround as you described:
https://github.com/Whonix/anon-gw-anonymizer-config/blob/master/lib/systemd/system/tor@default.service.d/40_obfs4proxy-workaround.conf

Debian bug report:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867342

JasonJAyalaP (Jason J. Ayala P.):

JasonJAyalaP added a comment.

Ok I created the workaround as you described:

https://github.com/Whonix/anon-gw-anonymizer-config/commit/bfe28e340d03cc4d77e4f49e24bcc0a9da42da06

Please add also a link to this ticket so further discussion can be
directed to this ticket rather than having it all over the place.

Please keep the Whonix 14 tag. I guess this can be closed, resolved?

The idea is to keep tracking this as part of a list of tasks that were done for Whonix 14.

The thing is, without any version tag for any future Whonix version, the ticket gets kinda lost in the deepness of the task tracker and unlikely being worked again in future.

The only remaining TODO is to keep answering https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867342. But for that I am not sure we should leave this ticket open? Or rewrite the ticket? Or open a new one?

I usually closed it, because I'll get e-mail notifications for for replies from upstream tickets anyhow.

I changed it to
NoNewPrivileges=No
That's the only thing I can imagine that would be causing that parsing error. Testing now

torproject's stretch repository [1] does not contain tor_0.3.1.5 yet.

Once TPOs stretch repo contains the latest, this workaround will no longer be needed, correct?

Any ETA on the repo including the latest?

with =no, I'm no longer getting the parsing error

sudo journalctl | grep workaround

but /lib/systemd/system/tor@default.service is unaffected

# Hardening
AppArmorProfile=system_tor
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectHome=yes
...

In T676#14449, @JasonJAyalaP wrote:

with =no, I'm no longer getting the parsing error

sudo journalctl | grep workaround

but /lib/systemd/system/tor@default.service is unaffected

# Hardening
AppArmorProfile=system_tor
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectHome=yes
...

Expected so it is. systemctl cat tor@default (should be same as systemctl cat tor@default.service) should show it.

And does obfs4proxy actually work then?

JasonJAyalaP (Jason J. Ayala P.):

JasonJAyalaP added a comment.

I changed it to
NoNewPrivileges=No
That's the only thing I can imagine that would be causing that parsing error. Testing now

> torproject's stretch repository [1] does not contain tor_0.3.1.5 yet.

Once TPOs stretch repo contains the latest, this workaround will no longer be needed, correct?

That's what one could conclude from the bug report that you wrote
earlier in this ticket which is now resolved on the Debian bug tracker.

Any ETA on the repo including the latest?

For a different subject, iry has figured out the Tor life cycle here.

http://forums.whonix.org/t/graphical-gui-whonix-setup-wizard-anon-connection-wizard-technical-discussion/650/335

Ah I see.

user@host:~$ systemctl cat tor@default
# /lib/systemd/system/tor@default.service
[Unit]
Description=Anonymizing overlay network for TCP
After=network.target nss-lookup.target
PartOf=tor.service
ReloadPropagatedFrom=tor.service

[Service]
Type=notify
NotifyAccess=all
PIDFile=/var/run/tor/tor.pid
PermissionsStartOnly=yes
ExecStartPre=/usr/bin/install -Z -m 02755 -o debian-tor -g debian-tor -d /var/run/tor
ExecStartPre=/usr/bin/tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc -f /etc/tor/torrc --RunAsDaemon 0 --verify-config
ExecStart=/usr/bin/tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc -f /etc/tor/torrc --RunAsDaemon 0
ExecReload=/bin/kill -HUP ${MAINPID}
KillSignal=SIGINT
TimeoutStartSec=300
TimeoutStopSec=60
Restart=on-failure
LimitNOFILE=65536

# Hardening
AppArmorProfile=system_tor
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectHome=yes
ProtectSystem=full
ReadOnlyDirectories=/
ReadWriteDirectories=-/proc
ReadWriteDirectories=-/var/lib/tor
ReadWriteDirectories=-/var/log/tor
ReadWriteDirectories=-/var/run
CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_NET_BIND_SERVICE CAP_DAC_READ_SEARCH

# /lib/systemd/system/tor@default.service.d/40_obfs4proxy-workaround.conf
## The hardening option NoNewPrvileges=Yes in the Debian tor@default.service file breaks loading of obfs4proxy in combination with Debian's apparmor profile which uses PUx. (ix ex
## https://phabricator.whonix.org/T676
[Service]
NoNewPrivileges=no

Tested with bridges obfs4 from TPO. Connects.