subjects
convention on listen port local or all network interfaces etc.
draft:
https://github.com/Whonix/proposals
TODO:
improve draft(done)send to debian-devel mailing list(done)- discuss on debian-devel
Description
Description
Details
Details
- Impact
- Normal
Related Objects
Related Objects
- Mentioned In
- T659: Publish "Listen Port Convention"
T660: Implement "Listen Port Convention" on Whonix
T661: Implement a config parser
T634: write draft for stackable wrappers on debian-devel
T561: find way to have Tor ephermal hidden service using applications in Whonix-Workstation bind on all interfaces
Event Timeline
Comment Actions
I agree, this looks great! The only thing I noticed is that the dpkg conflict resolution reference is probably missing a :. Maybe that's why it looks like a checked check box? ([x] instead of [x]:?)
Comment Actions
Right. Fixed. It was a checkbox because the phabricator interpreted it as markup. I am not putting the whole thing into a code box since then long lines would not be auto wrapped in your browser, which would make this a lot harder to read. Anyhow. [x]:indeed looks better in phabricator and will look better on the mailing list.
Comment Actions
One thing is still too vague.
Does
# /etc/server-config.d/30_default.conf listen_ip=0.0.0.0 # /etc/server-config.d/50_user.conf listen_ip=127.0.0.1
mean
- a) listen on 0.0.0.0 as well as on 127.0.0.1?
- b) Or does the latter listen_ip=127.0.0.1 deactivate listen_ip=0.0.0.0?
a) is how systemd parses systemd unit files. In that case one would have to use listen_ip= to disable listeners by lower priority configuration files.
# /etc/server-config.d/30_default.conf listen_ip=0.0.0.0 # /etc/server-config.d/50_user.conf listen_ip= listen_ip=127.0.0.1
b) is easier understand and easy to parse with bash (just use source as per the pseudo code in the draft). bash however is not so important here. A python parser could easily translate this for bash and most daemons will not be written in bash and not be using bash startup wrappers.
In other words... How to listen on multiple instances?
Like this...
listen_ip=127.0.0.1,10.152.152.10,UNIX-LISTEN:/path/to/<application-name>.sock
...?
Please take into consideration the difficulty of adhering this at the python implementation level. I mean, nothing is won if a superb convention is formulated that is then ignored by applications developers like you @dau because it looks too difficult to implement. We might want to bend the convention to suit python implementation difficulty.
Yet another style would be "lower priority configuration files are ignored, only highest priority configuration file is adhered".
And if we don't know the answer, we'll just integrate these questions into the draft.
Comment Actions
That's interesting. I did think that all files would be read and interpreted, but in cases where multiple listen_ip= would be found, the last one would override the previous. So indeed, it seems they would have to disabled.
In other words... How to listen on multiple instances?
Like this...
listen_ip=127.0.0.1,10.152.152.10,UNIX-LISTEN:/path/to/<application-name>.sock...?
I like it and seems easy to parse.
Please take into consideration the difficulty of adhering this at the python implementation level. I mean, nothing is won if a superb convention is formulated that is then ignored by applications developers like you @dau because it looks too difficult to implement. We might want to bend the convention to suit python implementation difficulty.
As a python developer it would be great to easily get the ip that should be used. However, I do not agree with you on this point because a *nix standard is being proposed. I think that it should be consistent with how systems interpret these configurations and then it is up to each app to access them. That might actually be decisive if the proposal is accepted or not.
I am sure that if it is really hard to read the configs, let's say in python, some package could be developed to parse the files and offer them in a very convenient way. I would just import it, tell which file to read or which specific config to get and do whatever I have to do with this object/dict/string returned. If the settings became environment variables I assume it would be pretty easy to use in any language, but if the file is too long then the environment might get polluted (?).
And if we don't know the answer, we'll just integrate these questions into the draft.
I agree. I think we could give them all the options we have and all questions that came up. The more information, the better.
Thanks!
Comment Actions
Thank you for your feedback!
I don't dare to call this a standard, because then this could take ages and face lots of resistance. A volunteer convention seems nicer.
There is no standard on configuration file parsing. Many applications have many very different configuration languages. I personally like the way systemd goes about it. It allowed Whonix specific overrides to be implemented with a very few lines and cleanly.
Comment Actions
Two more edits. It's now all included based on your feedback which I hopefully interpreted correctly (btw being not a native English speaker).
Please have a look if it's ready to go.
Comment Actions
No problem. As I said, I do not have much experience/knowledge about this, so I am just giving opinions/ideas (not sure which ones make sense or are relevant).
I don't dare to call this a standard, because then this could take ages and face lots of resistance. A volunteer convention seems nicer.
Hmm you are right, that's true! At this point I think that Whonix will implement it regardless if it is adopted or not, right?
There is no standard on configuration file parsing. Many applications have many very different configuration languages. I personally like the way systemd goes about it. It allowed Whonix specific overrides to be implemented with a very few lines and cleanly.
Sounds good. If you are familiar with that and things work, then I'd say it's probably the way to go.
Cool! I cannot help you with that because I am not a native speaker as well :P
In my opinion it looks great, I think it is ready.
Comment Actions
Great! Sent.
Not yet on the mailing list archive. I'll share the link once available here.
https://lists.debian.org/debian-devel/2017/02/threads.html
Sure. That's all I am looking for. Much appreciated! :)
I don't dare to call this a standard, because then this could take ages and face lots of resistance. A volunteer convention seems nicer.
Hmm you are right, that's true! At this point I think that Whonix will implement it regardless if it is adopted or not, right?
It is my hope to get some good feedback on debian-devel. One never knows what could have been overlooked. Lots of people reading there. The more developers following this convention, the better.
But yes. Even if only you representing unMessage like this convention, by all means, Whonix should go for it. I am however very positive that also onionshare, ricochet and ZeroNet would be open to it. Even if just the Tor ephemeral hidden services software packages adhere this convention that would be a huge win.
Comment Actions
As we discussed in [0], I am updating the draft.
Based on Tomas Pospisek's posts [1] and [2], I made the following changes (it's attached). Following your reply [3] to Marco d'Itri's post [4], I agree and think he did not fully understand our intentions, so I did not take that into consideration.
From my understanding, Tomas Pospisek would like to have additional configuration includes explicit and manually added to the main file so that it is clear what is supposed to happen instead of thinking "apparently placing customizations in /home/.config/server-config.d/ just work". I do not fully agree as I am usually in favor of "convention over configuration" and either just adding configurations to some of the server-config.d dirs or customizing main.config still require you "to have a priory knowledge" (you need to know what you are doing and what files to change), but I do like the solution he presented by using include, which makes the file a lot "cleaner".
From your reply [5] to his post:
Would you suggest just a single file /etc/server/main.config?
Or should there be also:
- /usr/lib/server/main.config
- /etc/server/main.config
- /home/.config/server/main.config
I think that by naming that file main, he intended to have just a single file so that not much knowledge is required (otherwise you would have know about not only all the .config files, but also the .conf ones). Also, from his comments in [2] it would be a good idea to use multiple .config files and multiple .d subdirs for complex configs, but I do think it is the case here. I agree with that but one problem that we might face is having that single .config file "restored" from the template when using Qubes. What do you think? Maybe having a directory in main.config that is not restored solves that? (i.e., all customizations should be in that dir)
Should main.config file[s] only contain include statements and
comments and nothing else?
I am pasting /etc/logrotate.conf below, which contains more than include statements. I think that there is no need for that as this convention is very clear about what the related .conf files should be and their contents are consistent to each other. I do like that the other content in /etc/logrotate.conf is consistent to the content of the other files in /etc/logrotate.d/.
# see "man logrotate" for details
# rotate log files weekly
weekly
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# uncomment this if you want your log files compressed
#compress
# packages drop log rotation information into this directory
include /etc/logrotate.d
# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
missingok
monthly
create 0664 root utmp
rotate 1
}
/var/log/btmp {
missingok
monthly
create 0660 root utmp
rotate 1
}
# system-specific logs may be configured hereThis new approach would not change the way the .conf files work when dealing with configurations from the previous files, right? If it is needed, they are able to to ignore the previous statements with, for example, listen_ip=. I am saying that because it looks like now main.config is allowed to change and just having include statements suggests that prefixing them with # is enough to ignore. That could work for non-Qubes systems, but as I mentioned above having main.config restored can be a problem.
Let me know what you think. Thanks!
[0]: https://github.com/AnemoneLabs/unmessage/pull/24
[1]: https://lists.debian.org/debian-devel/2017/02/msg00442.html
[2]: https://lists.debian.org/debian-devel/2017/02/msg00581.html
[3]: https://lists.debian.org/debian-devel/2017/02/msg00448.html
[4]: https://lists.debian.org/debian-devel/2017/02/msg00562.html
[5]: https://lists.debian.org/debian-devel/2017/02/msg00561.html
Comment Actions
If you are using git anyhow to generate the diff, could you please fork https://github.com/adrelanos/listen-port-convention and commit? That would make the diff a lot easier to apply and read for me.
From your diff...
I think I agree for the most part.
+include /home/.config/server-config.d/*.conf
Looks like a user name is missing. There is no user .config.
Do you mean it would automatically interpreted as from
- .. /home/.config/server-config.d/*.conf
- to /home/user1/.config/server-config.d/*.conf
- to /home/user2/.config/server-config.d/*.conf
- etc.? (Depending on the user under which the application is running?)
include vs home folder is not obvious yet in the draft? Perhaps the config should contain a variable` $HOME or so? And only be adhered by applications running as user such as unMessage? And ignored by applications run by the init system (such a web servers)?
Thanks!
Based on Tomas Pospisek's posts [1] and [2], I made the following changes (it's attached).
Following your reply [3] to Marco d'Itri's post [4], I agree and think he did not fully understand our intentions, so I did not take that into consideration.
Ok.
From my understanding, Tomas Pospisek would like to have additional configuration includes explicit and manually added to the main file so that it is clear what is supposed to happen instead of thinking "apparently placing customizations in /home/.config/server-config.d/ just work". I do not fully agree as I am usually in favor of "convention over configuration" and either just adding configurations to some of the server-config.d dirs or customizing main.config still require you "to have a priory knowledge" (you need to know what you are doing and what files to change), but I do like the solution he presented by using include, which makes the file a lot "cleaner".
Ok.
From your reply [5] to his post:
Would you suggest just a single file /etc/server/main.config?
Or should there be also:
- /usr/lib/server/main.config
- /etc/server/main.config
- /home/.config/server/main.config
From my current self, my question was a bit pointless. Of course only /etc/server/main.config and not /usr/lib/server/main.config etc. No need to make it overly complex.
I think that by naming that file main, he intended to have just a single file so that not much knowledge is required (otherwise you would have know about not only all the .config files, but also the .conf ones).
Yes.
Also, from his comments in [2] it would be a good idea to use multiple .config files and multiple .d subdirs for complex configs, but I do think it is the case here.
Yes.
I agree with that but one problem that we might face is having that single .config file "restored" from the template when using Qubes. What do you think? Maybe having a directory in main.config that is not restored solves that? (i.e., all customizations should be in that dir)
Not sure I understand. I guess you are referring to Qubes template implementation (and Qubes bind-dirs and such)? Yes, indeed! How could I forget about this.
So let's sneak into the proposal....?
include /usr/local/etc/server-config.d/*.conf
/usr/local is persistent in Qubes TemplateBasedVMs. Then users would not have to modify their template, but could for example also persistently change settings in their TemplateBased AppVMs.
Should main.config file[s] only contain include statements and
comments and nothing else?I am pasting /etc/logrotate.conf below, which contains more than include statements. I think that there is no need for that as this convention is very clear about what the related .conf files should be and their contents are consistent to each other. I do like that the other content in /etc/logrotate.conf is consistent to the content of the other files in /etc/logrotate.d/.
# see "man logrotate" for details # rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed #compress # packages drop log rotation information into this directory include /etc/logrotate.d # no packages own wtmp, or btmp -- we'll rotate them here /var/log/wtmp { missingok monthly create 0664 root utmp rotate 1 } /var/log/btmp { missingok monthly create 0660 root utmp rotate 1 } # system-specific logs may be configured here
I like that.
This new approach would not change the way the .conf files work when dealing with configurations from the previous files, right? If it is needed, they are able to to ignore the previous statements with, for example, listen_ip=.
Yes.
I am saying that because it looks like now main.config is allowed to change and just having include statements suggests that prefixing them with # is enough to ignore.
Yes.
That could work for non-Qubes systems, but as I mentioned above having main.config restored can be a problem.
Yes. Let's see if you like my suggestion to cover this using /usr/local/etc/server-config.d/*.conf as mentioned above.
Comment Actions
Great, there is a repo! I'll send a PR then.
Looks like a user name is missing. There is no user .config.
Indeed /home/.config/ is wrong. I believe I did not copy that properly from the draft. Sorry!
include vs home folder is not obvious yet in the draft? Perhaps the config should contain a variable` $HOME or so?
That would be a good idea. Is that how other conventions reference ~?
And only be adhered by applications running as user such as unMessage? And ignored by applications run by the init system (such a web servers)?
That's also a good idea. Configs in ~ are probably the most specific we can get. I am not sure though if in the end that will be useful/used. For example, listen_ip is customized likely due to some network/system scenario (in our case, Whonix' workstation/gateway setup), but do you think that would change for different users of the same network and system? Maybe it's useful when you do not have privileges to change the other files?
Not sure I understand. I guess you are referring to Qubes template implementation (and Qubes bind-dirs and such)? Yes, indeed! How could I forget about this.
So let's sneak into the proposal....?
include /usr/local/etc/server-config.d/*.conf/usr/local is persistent in Qubes TemplateBasedVMs. Then users would not have to modify their template, but could for example also persistently change settings in their TemplateBased AppVMs.
That certainly solves the problem!
Yes. Let's see if you like my suggestion to cover this using /usr/local/etc/server-config.d/*.conf as mentioned above.
Completely agree with that.
I can send the changes. The only thing I wanted to clarify before that is whether we are going to have ~/.config/server-config.d/ by default or not, and how we are going to reference ~ in main.config if so.
Thanks!
Comment Actions
include vs home folder is not obvious yet in the draft? Perhaps the config should contain a variable` $HOME or so?
That would be a good idea. Is that how other conventions reference ~?
I am not aware of any other conventions using $HOME. ~ might be shell specific. $HOME is being used in /etc a few times. I think $HOME is a good choice, because it's a very common environment variable in user's environments. (Not sure stuff even breaks without it.)
And only be adhered by applications running as user such as unMessage? And ignored by applications run by the init system (such a web servers)?
That's also a good idea. Configs in ~ are probably the most specific we can get. I am not sure though if in the end that will be useful/used. For example, listen_ip is customized likely due to some network/system scenario (in our case, Whonix' workstation/gateway setup), but do you think that would change for different users of the same network and system? Maybe it's useful when you do not have privileges to change the other files?
With parsing config files in $HOME we may over complicate the proposal with a rule like this... When using include $HOME/server-config.d/*.conf,
- if $HOME is set, actually do include $HOME/server-config.d/*.conf (this is when applications runs under a user account, not run by the init system)
- if $HOME is unset, include $HOME/server-config.d/*.conf has no effect (happens most likely when applications are run by the init system)
I am also getting uncertain of the usefulness of parsing config files in $HOME.
Perhaps there are cases where they don't have privileges to change /etc but install as user? But why at the same time they want to run a server software on a system they don't have write access to /etc? Strange case.
The Whonix use case which started with "let's make applications usually listening on 127.0.0.1 such as onionshare, ricochet, unMessage and ZeroNet actually listen on eth0 without having all these applications having to provide individual config files and parse these". For Whonix use case, parsing of config files in $HOME is not required.
- we'd pre-configure that for the user by shipping a config file with Whonix
- other changes could be done by the user (perhaps VPN users should change it)
If we don't find a convincing argument for $HOME config file parsing for this proposal, I guess we better leave it out. We can listen for feedback and if a compelling use case is provided at a later time, we could still upgrade to a higher version of the proposal at a later point.
Not sure I understand. I guess you are referring to Qubes template implementation (and Qubes bind-dirs and such)? Yes, indeed! How could I forget about this.
So let's sneak into the proposal....?
include /usr/local/etc/server-config.d/*.conf/usr/local is persistent in Qubes TemplateBasedVMs. Then users would not have to modify their template, but could for example also persistently change settings in their TemplateBased AppVMs.
That certainly solves the problem!
Great!
Yes. Let's see if you like my suggestion to cover this using /usr/local/etc/server-config.d/*.conf as mentioned above.
Completely agree with that.
I can send the changes. The only thing I wanted to clarify before that is whether we are going to have ~/.config/server-config.d/ by default or not, and how we are going to reference ~ in main.config if so.
Sure thing. Please go ahead. :-)
Comment Actions
Alright, I just made the PR with the changes we discussed. Also made minor changes to make it a bit more friendlier.
Completely agree. Removed the ~ entry but added a note about using home for the configurations.
One thing I noticed: Tomas' suggestions used /etc/server/main.config for the main file and server/config.d/ for the directories. I think that makes sense and makes things more consistent. I'm just not sure if server is the best name because that might be too generic. What do you think?
By the way, I think it is a good idea to use VCS for this. Would it be possible to create a proposals repository? Similar to the one Tor uses.
Comment Actions
dau (Felipe Dau):
dau added a comment.
Alright, I just made the PR with the changes we discussed. Also made minor changes to make it a bit more friendlier.
Thanks, merged!
In https://phabricator.whonix.org/T635#12791, @Patrick wrote: > With parsing config files in `$HOME` we may over complicate the proposal with a rule like this... When using `include $HOME/server-config.d/*.conf`, > > - if `$HOME` is set, actually do `include $HOME/server-config.d/*.conf` (this is when applications runs under a user account, not run by the init system) > - if `$HOME` is unset, `include $HOME/server-config.d/*.conf` has no effect (happens most likely when applications are run by the init system) > > I am also getting uncertain of the usefulness of parsing config files in `$HOME`. > > Perhaps there are cases where they don't have privileges to change `/etc` but install as user? But why at the same time they want to run a server software on a system they don't have write access to `/etc`? Strange case. > > The Whonix use case which started with "let's make applications usually listening on 127.0.0.1 such as onionshare, ricochet, unMessage and ZeroNet actually listen on eth0 without having all these applications having to provide individual config files and parse these". For Whonix use case, parsing of config files in `$HOME` is not required. > - we'd pre-configure that for the user by shipping a config file with Whonix > - other changes could be done by the user (perhaps VPN users should change it) > > If we don't find a convincing argument for `$HOME` config file parsing for this proposal, I guess we better leave it out. We can listen for feedback and if a compelling use case is provided at a later time, we could still upgrade to a higher version of the proposal at a later point. Completely agree. Removed the `~` entry but added a note about using home for the configurations.
Great!
One thing I noticed: Tomas' suggestions used `/etc/server/main.config` for the main file and `server/config.d/` for the directories. I think that makes sense and makes things more consistent. I'm just not sure if `server` is the best name because that might be too generic. What do you think?
Neither /etc/server-config.d nor /etc/server look like ideal choices
for folder names to me. I was hoping for better suggestions. =)
/etc/server/main.config is not so pretty. A folder with a single
config file. Why a folder then.
(I also had in mind /etc/listen/main.conf as well as
/etc/listen/conf.d/*.conf. That would be similar to /etc/dpkg. But
/etc/listen/conf.d/ would be less consistent with /usr/lib/listen.d/.)
What about as name, listen? Is this specific enough only on the topic
of listen or will it likely expand to also covering "server"?
Let's go with the following?
- /etc/listen.conf
- /etc/listen.d/*.conf
- /usr/local/etc/listen.d/*.conf
- /usr/lib/listen.d/*.conf
By the way, I think it is a good idea to use VCS for this. Would it be possible to create a `proposals` repository? Similar to the one Tor uses.
Good idea!
Comment Actions
Renamed repository from https://github.com/adrelanos/listen-port-convention to https://github.com/adrelanos/proposals.
Comment Actions
Maybe a bit late, but perhaps now that the proposal is available in git, we could even mention something like this...?
This proposal is available in git [xx], pull requests are welcome.
[xx] https://github.com/Whonix/proposals/blob/master/listen-port-convention.txt
I think sometimes a shorter comment plus pull request would be less ambiguous, quicker to implement fixes, and if someone is up for that, more power to them.
What do you think? If that is good as is or modified, could you add it please in your next pull request?
Comment Actions
Sure, sounds good. I am really bad at names (specially in English), so I think it looks good and if this thing starts to grow with more general server content and people start to support it, they will either change to a better name or just get used to it.
Good idea!
Btw, can I use numbers to index the references instead of x, xx, etc?
I think sometimes a shorter comment plus pull request would be less ambiguous, quicker to implement fixes, and if someone is up for that, more power to them.
I agree. I just really hope the discussions do not get split between github and phabricator. Hopefully one day github adds support to other trackers?
What do you think? If that is good as is or modified, could you add it please in your next pull request?
Sure, I will. But before that, can I make another suggestion?
I think that once the proposals start to be created we might get confused to reference all of them, so what if we prefixed them just like Tor does, but using the ticket numbers instead? That way we not only have an ID for each one of them, but which is related to a ticket and can be easily found/referenced/etc.
That means the current proposals would become:
- 634-stackable-wrappers.txt
- 635-listen-port-convention.txt
What do you think?
Comment Actions
> Maybe a bit late, but perhaps now that the proposal is available in git, we could even mention something like this...? > > > This proposal is available in git [xx], pull requests are welcome. > > > > > [xx] https://github.com/Whonix/proposals/blob/master/listen-port-convention.txt Good idea! Btw, can I use numbers to index the references instead of `x`, `xx`, etc?
Sure.
> I think sometimes a shorter comment plus pull request would be less ambiguous, quicker to implement fixes, and if someone is up for that, more power to them. I agree. I just really hope the discussions do not get split between github and phabricator.
Indeed an issue.
Hopefully one day github adds support to other trackers?
I'd hope so. Sadly I have little hope for that. Too closed / commercial.
More likely that some open source github clone takes off.
> What do you think? If that is good as is or modified, could you add it please in your next pull request? Sure, I will. But before that, can I make another suggestion?
Sure! :)
I think that once the proposals start to be created we might get confused to reference all of them, so what if we prefixed them just like Tor does, but using the ticket numbers instead? That way we not only have an ID for each one of them, but which is related to a ticket and can be easily found/referenced/etc. That means the current proposals would become: - `634-stackable-wrappers.txt` - `635-listen-port-convention.txt` What do you think?
Good idea. Please add. And if you find it useful, perhaps the subject
and link to the ticket would be useful for each proposal / draft as well.
Comment Actions
Let me know what you think of the new PR.
Should we add "(Proposal X)" (or something like that) to the title of the documents as well?
After making the changes, I am not sure if with the above you actually meant to set a new order to the parsing of these. Was that the case? Because I changed the order to that.
Otherwise I can move back to how they were such as :
- /usr/lib/listen.d/*.conf
- /usr/local/etc/listen.d/*.conf
- /etc/listen.d/*.conf
Thanks!
Comment Actions
Can do.
Perhaps like this. A dividing marker... (To keep things simple for copy and paste, not to mess that up when posting it to the final posting destination.)
subject here ################################################## text here
After making the changes, I am not sure if with the above you actually meant to set a new order to the parsing of these. Was that the case? Because I changed the order to that.
Otherwise I can move back to how they were such as :
- /usr/lib/listen.d/*.conf
- /usr/local/etc/listen.d/*.conf
- /etc/listen.d/*.conf
Thanks!
The latter is better. (Similar on how systemd does it.)
(I messed that up. Wanted to show how /etc/listen.conf looks nice vs /etc/listen.d/*.conf.)
Comment Actions
# Description is clearly visible as well. No ################################################## required.
Comment Actions
Feedback on the ticket and pull requests on the repository are welcome.
Right. Could you please add that text the Description part? I mean, the Description part is the part that gets actually copied over to the Debian mailing list in this case.
I'd suggest to add the link to the ticket and repo rather as a bonus. To be added somewhere at the end. We are mainly addressing the people on the mailing list. Beginning the text with two external links, I mean with...
might deter readers? Or am I over thinking this? I don't want to start painting the proverbial bikeshed.
Comment Actions
Right, that's much better. That first section is important just for the ones familiar with Whonix' proposals and that feedback note would not make sense there. I moved to the very end of the documents.
Just made another update. Let me know what you think!
Thanks!
Comment Actions
https://lists.debian.org/debian-devel/2017/03/msg00352.html
Hi Patrick, On Sun, Mar 26, 2017 at 11:22:00AM +0000, Patrick Schleizer wrote: > A convention on listen port local or all network interfaces etc. would > be desirable. I agree, though I'm not sure we can get consensus on this… However, I also think that you should restart this discussion once stretch has been released. Now it's not the best time to have such discussions… > Any questions? Any suggestions? What do you think? > > Feedback on the ticket [0] and pull requests on the repository [1] are > welcome. (I haven't looked at neither…) but I'd suggest you file a bug against the "general" pseudo-package in the Debian BTS as that's a more proper place to discuss changes in Debian. Thanks! -- cheers, Holger
Now that I am thinking about it, there is something that is not clear to me: is this convention a change to Debian or a "good practice" that any *nix system could/should do?
I am asking that because requesting a change to Debian seems to take a really long time, but proposing it like a "suggestion" which some (Debian) people agree to might not. Does that make sense?
Comment Actions
Holger Levsen:
[...]
However, I also think that you should restart this discussion once stretch has been released.
[...]
suggest you file a bug against the "general" pseudo-package in the Debian BTS
[...]
[zzz] Well, why not. For the fun and learning experience by it, let's wait until Debian version 9 codename Stretch gets released and then open a bug against general on Debian BTS.
Thank you for monitoring for replies!
Now that I am thinking about it, there is something that is not clear to me: is this convention a change to Debian or a "good practice" that any *nix system could/should do?
I did not dare to suggest a change to Debian. Did not have this in mind originally.
I am asking that because requesting a change to Debian seems to take a really long time, but proposing it like a "suggestion" which some (Debian) people agree to might not. Does that make sense?
Right. That's what I had in mind.
I think [zzz] is a bonus. No need to wait for it or to let it block our progress here. We got some great suggestions for our initial draft which we then implemented. I think in this case it's fair to interpret the absence of criticism and praise as an okay signal to move forward. This is probably also how things work. Some people create some convention, then just do it. If it then proves to be useful and working, it might become an official Debian standard. Neither Debian nor freedesktop want to be the gatekeepers of innovation.
( This is similar to https://bugs.linuxfoundation.org/show_bug.cgi?id=1367. )
( If you like to go further down into the rabbit hole on that one... See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=675008. )
Meaning, if you'd like to develop a python lib for parsing this convention and/or adhered this convention in unMessage, that'd be awesome! I would also be suggesting other application developers to implement this convention.
What do you think?
Comment Actions
Cool!
( This is similar to https://bugs.linuxfoundation.org/show_bug.cgi?id=1367. )
( If you like to go further down into the rabbit hole on that one... See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=675008. )
Meaning, if you'd like to develop a python lib for parsing this convention and/or adhered this convention in unMessage, that'd be awesome! I would also be suggesting other application developers to implement this convention.
What do you think?
Alright, I should be able to take a look into that earlier next week.
It would be great to have a generic tool that parses these directories. For example parse('listen') would go through all those directories, merging the configs.
Could we make some kind of Python binding for an existing solution?
Comment Actions
I have no idea if something like this exists. Might not as I don't know python applications doing this.
Comment Actions
Well, what I meant was using something written in another language (for example, systemd) that would then be called by this Python package we would create. Users would use it like any other Python package, but in fact it is calling something else underneath. I honestly do not know if there is any kind of API or some way to read the final config though. Do you think that's possible?
Comment Actions
Well, there is run-parts. Example:
run-parts --list /etc/grub.d/
But I doubt it would be of big help here.
Comment Actions
This class is reasonably flexible so that people are able to customize how files are found and parsed. I pretty much just wrote the API, but here is what I have in mind:
import argparse
class Parser:
@classmethod
def parse_global(cls, filepath):
"""Read `filepath`, parse it and return all directory paths found.
The directories in the list are in the order they are intended to be
read.
"""
@classmethod
def check_filename(cls, filename):
"""Return `True` if `filename` is a config file. `False` otherwise."""
@classmethod
def find_files(cls, dirpath):
"""Walk through the directory and return all file paths found.
Depending on `check_filename`, each file is prepended by the
directory's path and added to the list. The files in the list are in
the order they are intended to be read.
"""
@classmethod
def parse_file(cls, filepath, configs=None):
"""Read `filepath`, parse it and return a `config: value` dict.
If no configs are provided, a new dict is created. As the file is read,
values are added to the configs with
`add_config(configs, config, value)`."""
@classmethod
def add_config(cls, configs, config, value):
"""Add `config: value` to the configs dict and return it.
In case the config did not exist, it is added to the dict. Otherwise,
it replaces or merges to the existing one.
"""
@classmethod
def parse(cls, name):
"""Read the name of the config and return a `config: value` dict.
The name defaults to a `/etc/<name>.conf` file path but can optionally
specify a different path and extension. Its format should be something
like `[/custom/path/]name[.custom]`.
"""
cfgs = dict()
for dirpath in cls.parse_global(name):
for filepath in cls.find_files(dirpath):
cfgs = cls.parse_file(filepath, cfgs)
return cfgs
@classmethod
def serialize(cls, configs):
"""Serialize the configs dict into a convenient format."""
@classmethod
def parse_to_serialized(cls, name):
"""Parse configs and return them serialized."""
return cls.serialize(cls.parse(name))
def call_parse():
"""Read a config name and print the serialized configs.
This function should be accessed through an entry point. The name is passed
via a command-line call and returns the configs in a format that
applications are able to read in the language they use.
"""
argparser = argparse.ArgumentParser()
argparser.add_argument('name')
args = argparser.parse_args()
print Parser.parse_to_serialized(args.name)It could be used with:
import Parser
cfgs = Parser.parse('listen')
listen_ip = cfgs['listen_ip']
# bind on `listen_ip`Based on man systemd.unit "The syntax is inspired by XDG Desktop Entry Specification[1].desktop files, which are in turn inspired by Microsoft Windows .ini files.", which also inspired Python's ConfigParser. I am going to take a look into systemd's code, but I think using Python's own config parser might be a better idea.
Once this is implemented, any non-Python app can make a command-line call to get a config parsed and serialized (maybe to json?).
If this becomes used by other people, they may implement their own file parser for their own syntax and override its methods. If other parsers become popular, we can implement them as well so that users can specify which parser they would like to use. Depending on all the customizations we allow and provide in this package, it might be easier to move from class methods to instance methods.
I chose to update the configs dict for every config parsing, for every file parsing. The processing increases but memory usage decreases. Then you do not have to merge multiple (possibly big) config dictionaries in the end as they are merged during the process.
We need to pick another name other than Parser or ConfigParser.
What do you think?
P.S. I noticed the link in the description needs to be updated. Would you change to https://github.com/Whonix/proposals/blob/master/635-listen-port-convention.txt?
Comment Actions
Perhaps
check_filename -> is_config_file
find_files -> find_config_files
?
On the other hand... I personally prefer the very at the end and to have some consistency, i.e. config_read, config_check and so forth. Do you know how other popular libraries do it? Perhaps there is even a convention on that?
json is not trivially parsed in shell scripts. For shell scripts, something that can be simply evaluated would be nice. (Just echo listen_ip=xxx, so listen_ip=xxx gets set.)
I may not be the best person to comment on that. Could you ask your mentor or otherwise please for more input on this one?
Comment Actions
I like your suggestions. Especially is_config_file - It makes much more sense. Thanks!
Do you know how other popular libraries do it? Perhaps there is even a convention on that?
I do not know and I am not aware of such convention, but it's worth to check.
json is not trivially parsed in shell scripts. For shell scripts, something that can be simply evaluated would be nice. (Just echo listen_ip=xxx, so listen_ip=xxx gets set.)
Hmm, maybe I was not very clear on the explanation. I did not suggest changing the configs format (the .conf files). They should still look like something evaluable.
What I suggested was the use of json to return the *parsed* configs to applications. As a Python user (and as this would be a Python package) I would expect to get a dict with the configs to use on my application. For non-Python applications, that dict would be serialized to json because I believe that is one of the most popular formats these days with support from many languages. Also, we can offer multiple "serializers" if needed.
Was that your concern?
I may not be the best person to comment on that. Could you ask your mentor or otherwise please for more input on this one?
Sure, I will talk to him. And can you clarify what exactly you are unsure about?
Can I send PRs to phabricator? The error is in the description of this ticket (at the top of the page):
[...]
draft:
https://github.com/adrelanos/proposals
TODO:
[...]
Thanks!
Comment Actions
json is not trivially parsed in shell scripts. For shell scripts, something that can be simply evaluated would be nice. (Just echo listen_ip=xxx, so listen_ip=xxx gets set.)
Hmm, maybe I was not very clear on the explanation. I did not suggest changing the configs format (the .conf files). They should still look like something evaluable.
I did not interpret your suggestion as a change in the conf files.
What I suggested was the use of json to return the *parsed* configs to applications.
Yes, got that. That's why I wanted to point out, that json is hard to parse in shell scripts.
As a Python user (and as this would be a Python package) I would expect to get a dict with the configs to use on my application. For non-Python applications, that dict would be serialized to json because I believe that is one of the most popular formats these days with support from many languages. Also, we can offer multiple "serializers" if needed.
Yes, something that works for shell scripts would be nice.
Was that your concern?
Yes.
I may not be the best person to comment on that. Could you ask your mentor or otherwise please for more input on this one?
Sure, I will talk to him. And can you clarify what exactly you are unsure about?
Just generally.
Can I send PRs to phabricator?
No.
The error is in the description of this ticket (at the top of the page):
Ah, now fixed. :)
Comment Actions
Ohh, now I got it! Sure, it should be possible to offer multiple serializers and also allow users to implement their own.
I may not be the best person to comment on that. Could you ask your mentor or otherwise please for more input on this one?
Sure, I will talk to him. And can you clarify what exactly you are unsure about?
Just generally.
Alright!
Can I send PRs to phabricator?
No.
The error is in the description of this ticket (at the top of the page):
Ah, now fixed. :)
Thanks!
Comment Actions
The original write draft for local listener standard on debian-devel seems done.
Could you please kindly close this ticket / create follow up ticket(s)?
Comment Actions
Sure! What do you think these should be?
- Implement the convention on Whonix (one just needs to create files following the structure described by it and find a way to parse it, right?)
- Implement the config parser
P.S. I don't think I have permission to close/edit this ticket.
Comment Actions
Also:
- Find a place to "officially" publish the convention (has it moved from "proposal" to "convention"?)
Comment Actions
Hi @Patrick,
Sorry for taking a long time to open these:
- Publish "Listen Port Convention"
- Implement "Listen Port Convention" on Whonix
- Implement a config parser
Let me know what you think.
Thanks!