Page MenuHomePhabricator

implement /usr/lib/tor-controlport-filter-merger
Closed, ResolvedPublic

Description

As per:
https://mailman.boum.org/pipermail/tails-dev/2017-January/011172.html

TODO:

Bonus:

  • if possible and easy code wise, add some comments on top of 30_autogenerated.yml such as "Manual edits will be lost! This file has been auto generated by..."
  • each merged key gets a comment from which source file it is coming from

Details

Impact
Normal

Event Timeline

Patrick created this task.Jan 25 2017, 10:04 PM

Patrick,

#1 -
Can you provide me few real yml files to be placed under
/etc/tor-controlport-filter-merger.d
and
/usr/local/etc/tor-controlport-filter-merger.d
which I can use to write the /usr/lib/tor-controlport-filter-merger?

#2 - Also let me know how do we prioritise the
a) Directories - /etc/tor-controlport-filter.d/*.yml has higher priority than /usr/local/etc/tor-controlport-filter.d/
b) Files - lexical order?

Thanks

#1 -
Can you provide me few real yml files to be placed under
/etc/tor-controlport-filter-merger.d
and
/usr/local/etc/tor-controlport-filter-merger.d
which I can use to write the /usr/lib/tor-controlport-filter-merger?

https://github.com/Whonix/control-port-filter-python/blob/master/etc/tor-controlport-filter.d/30_whonix.yml will go there.

With an optional (by user) combination of https://github.com/Whonix/control-port-filter-python/tree/master/usr/share/tor-controlport-filter/examples.

#2 - Also let me know how do we prioritise the
a) Directories - /etc/tor-controlport-filter.d/*.yml has higher priority than /usr/local/etc/tor-controlport-filter.d/
b) Files - lexical order?

Parse /etc/tor-controlport-filter.d/first. Parse /usr/local/etc/tor-controlport-filter.d/ second. Therefore the latter has higher priority.

joysn1980 added a comment.EditedJan 31 2017, 11:05 AM

take (and review) https://git-tails.immerda.ch/tails/plain/config/chroot_local-includes/usr/local/lib/tor-controlport-filter?h=feature/12173-end-whonix-controlport-filter-fork

Tails changes have

  1. -u option
  2. safe_load
  3. -interface

the change about merging is not present

replace https://github.com/Whonix/control-port-filter-python/blob/master/usr/lib/tor-controlport-filter with above

Here it is -
https://github.com/joysn/control-port-filter-python/commit/3da7a49d7b4f43827578f00fc50489a05de4369f#diff-7414879ce81f5586d790820540d0ca05

invent /usr/lib/tor-controlport-filter-merger

Here it is
https://github.com/joysn/control-port-filter-python/blob/master/usr/lib/tor-controlport-filter-merger

  1. This will create /etc/tor-controlport-filter.d/30_autogenerated.yml
  2. This will add the following three lines at the top
## This file is part of Whonix.
## Manual edits to this file will be lost! 
## This file has been auto generated by...

Let me know if this is ok, or something more/modification is required.

  1. There are couple of merges [typo in the path where this file to be created was fixed in subsequent merge]
joysn1980 changed the task status from Open to Review.Jan 31 2017, 4:27 PM
Patrick changed the task status from Review to Open.Jan 31 2017, 7:35 PM

So far so good.

filter_files_set1 = glob.glob('/etc/tor-controlport-filter.d/*.yml')

filter_files_set2 = glob.glob('/usr/local/etc/tor-controlport-filter.d/*.yml')

I think this needs to be tor-controlport-filter-merger.d?

(Manually overwrote this with tor-controlport-filter-merger.d in my tests.)

In /lib/systemd/system/tor-controlport-filter.service.d/30_cpfpy.conf please add above ExecStart=:

ExecStartPre=/usr/lib/tor-controlport-filter-merger

This needs some debugging. Fails for a unknown reason.

sudo service  tor-controlport-filter restart

Job for tor-controlport-filter.service failed because the control process exited with error code.
See "systemctl status tor-controlport-filter.service" and "journalctl -xe" for details.

sudo service tor-controlport-filter status

● tor-controlport-filter.service - Tor control port filter proxy

 Loaded: loaded (/lib/systemd/system/tor-controlport-filter.service; enabled; vendor preset: enabled)
Drop-In: /lib/systemd/system/tor-controlport-filter.service.d
         └─30_cpfpy.conf, 30_whonix_cpfpy.conf
 Active: failed (Result: exit-code) since Tue 2017-01-31 21:10:56 UTC; 944ms ago
   Docs: https://tails.boum.org/contribute/design/
Process: 10297 ExecStartPre=/usr/lib/tor-controlport-filter-merger (code=exited, status=1/FAILURE)

Main PID: 9346 (code=killed, signal=TERM)
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Control process exited, code=exited status=1
Jan 31 21:10:56 host systemd[1]: Failed to start Tor control port filter proxy.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Unit entered failed state.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Failed with result 'exit-code'.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Service hold-off time over, scheduling restart.
Jan 31 21:10:56 host systemd[1]: Stopped Tor control port filter proxy.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Start request repeated too quickly.
Jan 31 21:10:56 host systemd[1]: Failed to start Tor control port filter proxy.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Unit entered failed state.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Failed with result 'exit-code'.


Did run the merger and filter manually... Might have found a bug. Added 40_ricochet.yml and 30_whonix.yml. The resulting 30_autogenerated.yml ...

cat /etc/tor-controlport-filter.d/30_autogenerated.yml
## This file is part of Whonix.
## Manual edits to this file will be lost! 
## This file has been auto generated by...


---
- commands:
    ADD_ONION:
    - pattern: NEW:(\S+) Port=9878,\S+:(\S+)
      replacement: NEW:{} Port=9878,{client-address}:{}
    - pattern: (\S+):(\S+) Port=9878,\S+:(\S+)
      replacement: '{}:{} Port=9878,{client-address}:{}'
    DEL_ONION:
    - .+
    GETCONF:                                                                                                                                                  
    - DisableNetwork                                                                                                                                          
    GETINFO:                                                                                                                                                  
    - pattern: status/circuit-established status/bootstrap-phase net/listeners/socks                                                                          
      response:                                                                                                                                               
      - pattern: 250-status/bootstrap-phase=*                                                                                                                 
        replacement: 250-status/bootstrap-phase=NOTICE BOOTSTRAP PROGRESS=100 TAG=done
          SUMMARY="Done"
      - pattern: 250-net/listeners/socks=".*"
        replacement: 250-net/listeners/socks="127.0.0.1:9150"
    - status/circuit-established
    - version
    - pattern: net/listeners/socks
      response:
      - pattern: 250-net/listeners/socks=".*"
        replacement: 250-net/listeners/socks="127.0.0.1:9150"
    SIGNAL:
    - NEWNYM
  confs:
    __owningcontrollerprocess: null
  events:
    CONF_CHANGED:
      suppress: true
    SIGNAL:
      suppress: true
    STATUS_CLIENT:
      suppress: true
  match-exe-paths: '*'
  match-hosts:
  - '*'
  match-users: '*'
  name: 'merged_filter_files:  40_ricochet.yml 30_whonix.yml'

The filter does not work with that.

sudo -u tor-controlport-filter /usr/lib/tor-controlport-filter --debug --listen-interface eth1
IP address for interface eth1 : 10.137.11.1
Tor control port filter started, listening on 10.137.11.1:9051





10.137.11.80:42806 (filter: None) connected: loaded filter: None
Final rules:
commands: {}
events: {}
restrict-stream-events: false
joysn1980 added a comment.EditedFeb 1 2017, 5:21 AM

The filter does not work with that.

This is what I tried
/etc/tor-controlport-filter.d/
has only one file - 30_autogenerated.yml [manually put it there]
and then

sudo service tor-controlport-filter restart
sudo journalctl -f -u tor-controlport-filter

Everything worked fine. Though I did not put the new tails version of tor-controlport-filter. Did you replace that and then you saw the error/bug?

What I am trying to find out is -
Is the issue with new tor-controlport-filter?
OR
Is the issue is with the merged config file - 30_autogenerated.yml

OK, I got this one.

The issue is with the TAILs version of the tor-controlport-filter which we have originally merged some time back.

/etc/tor-controlport-filter.d/30_autogenerated.yml
and
with tor-controlport-filter
as in
https://github.com/Whonix/control-port-filter-python/blob/e504cabcdaf159f821f38edd4267112c71a190d7/usr/lib/tor-controlport-filter

[this is the last version before taking up the original tails version].
This works fine with this new auto merged config 30_automerged.yml file[generated from tor-controlport-filter-merger]

I fixed the issue with tails version
https://github.com/joysn/control-port-filter-python/commit/6f488c14980e8b5c58a42374649c4d5725c8296e#diff-7414879ce81f5586d790820540d0ca05

The matchers were spelled incorrectly

-                ('exe-paths', client_exe_path),
 -                ('users',     client_user),
-                ('hosts', client_host),

Should be

+                ('match-exe-paths', client_exe_path),
+                ('match-users',     client_user),
+                ('match-hosts', client_host),

So with this two fix, all should work fine.

Patrick changed the task status from Open to Review.Feb 2 2017, 11:30 PM

https://github.com/Whonix/control-port-filter-python/commit/cf0c7afa93de78362236c5dc6539d74ccd1dec56

(That diff on github does not look great, however that diff looks quite sane in a better diff viewer.)

Also fully tested. Works great!

Do my changes look sane to you? If so, this ticket can be considered resolved.

Perfect Patrick. Thanks.

Patrick changed the task status from Review to Open.EditedFeb 3 2017, 1:07 PM

One more thing to do here.
s/match-//g
anonym replied.

Do you think you could git revert 6f488c14980e8b5c58a42374649c4d5725c8296e (so we share the same code with Tails) as well as remove match- prefixes in the merger code and configs?

In T617#11999, @Patrick wrote:

Do you think you could git revert cf0c7afa93de78362236c5dc6539d74ccd1dec56 (so we share the same code with Tails) as well as remove match- prefixes in the merger code and configs?

My mistake. I meant git revert 6f488c14980e8b5c58a42374649c4d5725c8296e

Merged. Wondering about one thing...

# Replace matched-exe-paths, matched-hosts and matched-users
for i in range(len(merged_filter)):
   if 'match-exe-paths' in merged_filter[i].keys():
      merged_filter[i]['exe-paths'] = merged_filter[i].pop('match-exe-paths')
   if 'match-hosts' in merged_filter[i].keys():
      merged_filter[i]['hosts'] = merged_filter[i].pop('match-hosts')
   if 'match-users' in merged_filter[i].keys():
      merged_filter[i]['users'] = merged_filter[i].pop('match-users')

Why is that needed? Couldn't we avoid that code block? If we just s/match-//g in all our configs, could we avoid that code block?

joysn1980 added a comment.EditedFeb 4 2017, 4:00 AM

This is possible when we are reading the file as "string" of characters [normally how we read a file]
But in this case we are reading our file as yaml which is a "dictionary" and each of the "match" we have to modified are the keys of the dictionary.

This is my understanding.

Again, we do not even need this code, if we modify our config file itself and replace these "keys" to the new expected "keys"

joysn1980 (Joy):

Again, we do not even need this code, if we modify our config file
itself and replace these "keys" to the new expected "keys"

Yes. Why not modify our config file and our merger example profiles?
Seems much better to me to use the same config file keywords as upstream
(Tails).

Basically s/match-//g for
etc/tor-controlport-filter-merger.d/30_whonix.yml and for
usr/share/tor-controlport-filter-merger/examples/.

Done that.

I have created a new git branch:
https://github.com/Whonix/control-port-filter-python/tree/keywords-as-tails

Just one commit:
https://github.com/Whonix/control-port-filter-python/commit/7b28225fbe6300631c89047a645a649fbed19208

Seems to work. I might not be fully understanding this.

Does that look good? If so, I am going to merge it into master.

Sure Patrick. Everything looks good (as per my understanding)

Patrick closed this task as Resolved.Feb 4 2017, 6:00 PM

Merged.