Patrick,

#1 -
Can you provide me few real yml files to be placed under
/etc/tor-controlport-filter-merger.d
and
/usr/local/etc/tor-controlport-filter-merger.d
which I can use to write the /usr/lib/tor-controlport-filter-merger?

#2 - Also let me know how do we prioritise the
a) Directories - /etc/tor-controlport-filter.d/*.yml has higher priority than /usr/local/etc/tor-controlport-filter.d/
b) Files - lexical order?

Thanks

In T617#11897, @joysn1980 wrote:

#1 -
Can you provide me few real yml files to be placed under
/etc/tor-controlport-filter-merger.d
and
/usr/local/etc/tor-controlport-filter-merger.d
which I can use to write the /usr/lib/tor-controlport-filter-merger?

https://github.com/Whonix/control-port-filter-python/blob/master/etc/tor-controlport-filter.d/30_whonix.yml will go there.

With an optional (by user) combination of https://github.com/Whonix/control-port-filter-python/tree/master/usr/share/tor-controlport-filter/examples.

#2 - Also let me know how do we prioritise the
a) Directories - /etc/tor-controlport-filter.d/*.yml has higher priority than /usr/local/etc/tor-controlport-filter.d/
b) Files - lexical order?

Parse /etc/tor-controlport-filter.d/first. Parse /usr/local/etc/tor-controlport-filter.d/ second. Therefore the latter has higher priority.

take (and review) https://git-tails.immerda.ch/tails/plain/config/chroot_local-includes/usr/local/lib/tor-controlport-filter?h=feature/12173-end-whonix-controlport-filter-fork

Tails changes have

1. -u option
2. safe_load
3. -interface

the change about merging is not present

replace https://github.com/Whonix/control-port-filter-python/blob/master/usr/lib/tor-controlport-filter with above

Here it is -
https://github.com/joysn/control-port-filter-python/commit/3da7a49d7b4f43827578f00fc50489a05de4369f#diff-7414879ce81f5586d790820540d0ca05

invent /usr/lib/tor-controlport-filter-merger

Here it is
https://github.com/joysn/control-port-filter-python/blob/master/usr/lib/tor-controlport-filter-merger

1. This will create /etc/tor-controlport-filter.d/30_autogenerated.yml
2. This will add the following three lines at the top

## This file is part of Whonix.
## Manual edits to this file will be lost! 
## This file has been auto generated by...

Let me know if this is ok, or something more/modification is required.

3. There are couple of merges [typo in the path where this file to be created was fixed in subsequent merge]

(Manually overwrote this with tor-controlport-filter-merger.d in my tests.)

In /lib/systemd/system/tor-controlport-filter.service.d/30_cpfpy.conf please add above ExecStart=:

ExecStartPre=/usr/lib/tor-controlport-filter-merger

This needs some debugging. Fails for a unknown reason.

sudo service  tor-controlport-filter restart

Job for tor-controlport-filter.service failed because the control process exited with error code.
See "systemctl status tor-controlport-filter.service" and "journalctl -xe" for details.

sudo service tor-controlport-filter status

● tor-controlport-filter.service - Tor control port filter proxy

 Loaded: loaded (/lib/systemd/system/tor-controlport-filter.service; enabled; vendor preset: enabled)
Drop-In: /lib/systemd/system/tor-controlport-filter.service.d
         └─30_cpfpy.conf, 30_whonix_cpfpy.conf
 Active: failed (Result: exit-code) since Tue 2017-01-31 21:10:56 UTC; 944ms ago
   Docs: https://tails.boum.org/contribute/design/
Process: 10297 ExecStartPre=/usr/lib/tor-controlport-filter-merger (code=exited, status=1/FAILURE)

Main PID: 9346 (code=killed, signal=TERM)

Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Control process exited, code=exited status=1
Jan 31 21:10:56 host systemd[1]: Failed to start Tor control port filter proxy.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Unit entered failed state.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Failed with result 'exit-code'.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Service hold-off time over, scheduling restart.
Jan 31 21:10:56 host systemd[1]: Stopped Tor control port filter proxy.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Start request repeated too quickly.
Jan 31 21:10:56 host systemd[1]: Failed to start Tor control port filter proxy.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Unit entered failed state.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Failed with result 'exit-code'.

Did run the merger and filter manually... Might have found a bug. Added 40_ricochet.yml and 30_whonix.yml. The resulting 30_autogenerated.yml ...

cat /etc/tor-controlport-filter.d/30_autogenerated.yml

## This file is part of Whonix.
## Manual edits to this file will be lost! 
## This file has been auto generated by...


---
- commands:
    ADD_ONION:
    - pattern: NEW:(\S+) Port=9878,\S+:(\S+)
      replacement: NEW:{} Port=9878,{client-address}:{}
    - pattern: (\S+):(\S+) Port=9878,\S+:(\S+)
      replacement: '{}:{} Port=9878,{client-address}:{}'
    DEL_ONION:
    - .+
    GETCONF:                                                                                                                                                  
    - DisableNetwork                                                                                                                                          
    GETINFO:                                                                                                                                                  
    - pattern: status/circuit-established status/bootstrap-phase net/listeners/socks                                                                          
      response:                                                                                                                                               
      - pattern: 250-status/bootstrap-phase=*                                                                                                                 
        replacement: 250-status/bootstrap-phase=NOTICE BOOTSTRAP PROGRESS=100 TAG=done
          SUMMARY="Done"
      - pattern: 250-net/listeners/socks=".*"
        replacement: 250-net/listeners/socks="127.0.0.1:9150"
    - status/circuit-established
    - version
    - pattern: net/listeners/socks
      response:
      - pattern: 250-net/listeners/socks=".*"
        replacement: 250-net/listeners/socks="127.0.0.1:9150"
    SIGNAL:
    - NEWNYM
  confs:
    __owningcontrollerprocess: null
  events:
    CONF_CHANGED:
      suppress: true
    SIGNAL:
      suppress: true
    STATUS_CLIENT:
      suppress: true
  match-exe-paths: '*'
  match-hosts:
  - '*'
  match-users: '*'
  name: 'merged_filter_files:  40_ricochet.yml 30_whonix.yml'

The filter does not work with that.

sudo -u tor-controlport-filter /usr/lib/tor-controlport-filter --debug --listen-interface eth1
IP address for interface eth1 : 10.137.11.1
Tor control port filter started, listening on 10.137.11.1:9051





10.137.11.80:42806 (filter: None) connected: loaded filter: None
Final rules:
commands: {}
events: {}
restrict-stream-events: false

I think this needs to be tor-controlport-filter-merger.d?

https://github.com/joysn/control-port-filter-python/commit/70162e5508e7bca9002beeacf19a8edd9763013d#diff-3bd76e3ee329692a8dd8311964a60264

The filter does not work with that.

This is what I tried
/etc/tor-controlport-filter.d/
has only one file - 30_autogenerated.yml [manually put it there]
and then

sudo service tor-controlport-filter restart
sudo journalctl -f -u tor-controlport-filter

Everything worked fine. Though I did not put the new tails version of tor-controlport-filter. Did you replace that and then you saw the error/bug?

What I am trying to find out is -
Is the issue with new tor-controlport-filter?
OR
Is the issue is with the merged config file - 30_autogenerated.yml

OK, I got this one.

The issue is with the TAILs version of the tor-controlport-filter which we have originally merged some time back.

/etc/tor-controlport-filter.d/30_autogenerated.yml
and
with tor-controlport-filter
as in
https://github.com/Whonix/control-port-filter-python/blob/e504cabcdaf159f821f38edd4267112c71a190d7/usr/lib/tor-controlport-filter

[this is the last version before taking up the original tails version].
This works fine with this new auto merged config 30_automerged.yml file[generated from tor-controlport-filter-merger]

I fixed the issue with tails version
https://github.com/joysn/control-port-filter-python/commit/6f488c14980e8b5c58a42374649c4d5725c8296e#diff-7414879ce81f5586d790820540d0ca05

The matchers were spelled incorrectly

-                ('exe-paths', client_exe_path),
 -                ('users',     client_user),
-                ('hosts', client_host),

Should be

+                ('match-exe-paths', client_exe_path),
+                ('match-users',     client_user),
+                ('match-hosts', client_host),

So with this two fix, all should work fine.

Forwarded your patch https://github.com/joysn/control-port-filter-python/commit/6f488c14980e8b5c58a42374649c4d5725c8296e#diff-7414879ce81f5586d790820540d0ca05 to tails-dev mailing list for inclusion at upstream.

Merged.

Added a few minor commits on top.

• https://github.com/Whonix/control-port-filter-python/commit/a615611fbbbb3ebf3b6641a187f109d22da3fd51
• https://github.com/Whonix/control-port-filter-python/commit/5bf372be2708b1b86eff20e861607104597fbb80
• https://github.com/Whonix/control-port-filter-python/commit/6c78583df9350f47f4ee84531311ecdb5111072d
• https://github.com/Whonix/control-port-filter-python/commit/d05d1015c460812ae0faa1a699f95f248817d4c7

• https://github.com/Whonix/whonix-gw-network-conf/commit/ac459469b609262fa75df23a0aec85efb8649295

https://github.com/Whonix/control-port-filter-python/commit/6fe2d272462ff4bdf7f33fea0d2db551cb232bfe

Perfect Patrick. Thanks.

In T617#11999, @Patrick wrote:

Do you think you could git revert cf0c7afa93de78362236c5dc6539d74ccd1dec56 (so we share the same code with Tails) as well as remove match- prefixes in the merger code and configs?

My mistake. I meant git revert 6f488c14980e8b5c58a42374649c4d5725c8296e

Fixed in
https://github.com/joysn/control-port-filter-python/commit/f430cf3478d430851db3443b2984bc61f69a6605#diff-3bd76e3ee329692a8dd8311964a60264

Reverting the changes to tor-controlport-filter
https://github.com/joysn/control-port-filter-python/commit/d99eb8c077dd6fc2ec0f479b51e2be043e3f6ea0#diff-700d916dcb6f72233745db056c63f43c

Merged. Wondering about one thing...

# Replace matched-exe-paths, matched-hosts and matched-users
for i in range(len(merged_filter)):
   if 'match-exe-paths' in merged_filter[i].keys():
      merged_filter[i]['exe-paths'] = merged_filter[i].pop('match-exe-paths')
   if 'match-hosts' in merged_filter[i].keys():
      merged_filter[i]['hosts'] = merged_filter[i].pop('match-hosts')
   if 'match-users' in merged_filter[i].keys():
      merged_filter[i]['users'] = merged_filter[i].pop('match-users')

Why is that needed? Couldn't we avoid that code block? If we just s/match-//g in all our configs, could we avoid that code block?

This is possible when we are reading the file as "string" of characters [normally how we read a file]
But in this case we are reading our file as yaml which is a "dictionary" and each of the "match" we have to modified are the keys of the dictionary.

This is my understanding.

Again, we do not even need this code, if we modify our config file itself and replace these "keys" to the new expected "keys"

joysn1980 (Joy):

Again, we do not even need this code, if we modify our config file
itself and replace these "keys" to the new expected "keys"

Yes. Why not modify our config file and our merger example profiles?
Seems much better to me to use the same config file keywords as upstream
(Tails).

Basically s/match-//g for
etc/tor-controlport-filter-merger.d/30_whonix.yml and for
usr/share/tor-controlport-filter-merger/examples/.

Done that.

I have created a new git branch:
https://github.com/Whonix/control-port-filter-python/tree/keywords-as-tails

Just one commit:
https://github.com/Whonix/control-port-filter-python/commit/7b28225fbe6300631c89047a645a649fbed19208

Seems to work. I might not be fully understanding this.

Does that look good? If so, I am going to merge it into master.

Sure Patrick. Everything looks good (as per my understanding)