Page MenuHomePhabricator

msgdispatcher_dispatch_x lacks new lines for output from console applications
Closed, ResolvedPublic

Description

Sometimes it is useful for whonixcheck to add stuff like this to the output.

sudo -u debian-tor tor --verify-config

Naturally, lines aren't prepended or terminated with <br></br>. Therefore new lines are ignored. They end all up in one huge line.

https://github.com/Whonix/msgcollector/blob/master/usr/lib/msgcollector/msgdispatcher_dispatch_x

Event Timeline

Patrick created this task.Jan 17 2015, 1:14 AM
Patrick raised the priority of this task from to Normal.
Patrick updated the task description. (Show Details)
Patrick added projects: msgcollector, Whonix 10.
Patrick added subscribers: Patrick, troubadour.

Example. Result...

Jan 17 00:15:03.916 [notice] Tor v0.2.5.10 (git-43a5f3d91e726291) running on Linux with Libevent 2.0.19-stable, OpenSSL 1.0.1e and Zlib 1.2.7. Jan 17 00:15:03.916 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning Jan 17 00:15:03.916 [notice] Read configuration file "/etc/tor/torrc". Jan 17 00:15:03.918 [warn] /var/run/tor is not owned by this user (root, 0) but by debian-tor (106). Perhaps you are running Tor as the wrong user? Jan 17 00:15:03.918 [warn] Failed to parse/validate config: Couldn't access/create private data directory "/var/run/tor" Jan 17 00:15:03.918 [err] Reading config failed--see warnings above.

Desired:

Jan 17 00:15:03.916 [notice] Tor v0.2.5.10 (git-43a5f3d91e726291) running on Linux with Libevent 2.0.19-stable, OpenSSL 1.0.1e and Zlib 1.2.7.
Jan 17 00:15:03.916 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Jan 17 00:15:03.916 [notice] Read configuration file "/etc/tor/torrc".
Jan 17 00:15:03.918 [warn] /var/run/tor is not owned by this user (root, 0) but by debian-tor (106). Perhaps you are running Tor as the wrong user?
Jan 17 00:15:03.918 [warn] Failed to parse/validate config: Couldn't access/create private data directory "/var/run/tor"
Jan 17 00:15:03.918 [err] Reading config failed--see warnings above.

@troubadour, do you think there is any good way to do this on the python level? (Without requiring us to do lots of formating changes...)

Otherwise I guess I could also inject the <br></br> at the end of each line using a bash function.

There is no easy way to do this in msgdispatcher_dispatch_x. It just shows what it receives. The formatting of the messages is all done in whonixcheck and /usr/lib/msgcollector/msgcollector (pretty_type_x), and I don't think it makes sense to even try to do it at python level.

We indeed shouldn't mix this into msgdispatcher_dispatch_x.


Are you any good at string manipulation stuff?

Maybe you could write a small python tool that does the trick?

For testing purposes, suppose any input as this.

x="$(telnet -h 2>&1)"

Variable $x will now contain the following content.

telnet: invalid option -- 'h'
Usage: telnet [-4] [-6] [-8] [-E] [-L] [-a] [-d] [-e char] [-l user]
        [-n tracefile] [ -b addr ] [-r] [host-name [port]]

Let's say I was running something like this...

x="$(telnet -h 2>&1)" ; ./some-python-script "$x"

Could you come up with an output that looks like this?

telnet: invalid option -- 'h'<br></br>
Usage: telnet [-4] [-6] [-8] [-E] [-L] [-a] [-d] [-e char] [-l user]<br></br>
        [-n tracefile] [ -b addr ] [-r] [host-name [port]]

In other words, can you append <br></br> at the end of every line but the last line?

Don't put too much effort into this.

It's not that much a question of string manipulation than of finding a common mark in each line to place the extra html. Do we want a universal script?

In this:

Jan 17 00:15:03.916 [notice] Tor v0.2.5.10 (git-43a5f3d91e726291) running on Linux with Libevent 2.0.19-stable, OpenSSL 1.0.1e and Zlib 1.2.7. Jan 17 00:15:03.916 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning Jan 17 00:15:03.916 [notice] Read configuration file "/etc/tor/torrc". Jan 17 00:15:03.918 [warn] /var/run/tor is not owned by this user (root, 0) but by debian-tor (106). Perhaps you are running Tor as the wrong user? Jan 17 00:15:03.918 [warn] Failed to parse/validate config: Couldn't access/create private data directory "/var/run/tor" Jan 17 00:15:03.918 [err] Reading config failed--see warnings above.

if there is no newlines, we could take the colon, providing the preceding characters(mmm dd hh) have always the same format/length.

In the telnet output example, in python, we would first have to know that we want to grab stderr, and replace \n with ,br></br>

>>> repr(err)
'"telnet: invalid option -- \'y\'\\nUsage: telnet [-4] [-6] [-8] [-E] [-L] [-a] [-d] [-e char] [-l user]\\n\\t[-n tracefile] [ -b addr ] [-r] [host-name [port]]\\n"'

That's a lot of conditions.

Was carired a little. ./some-python-script "$x" is possible if we have new lines

Do we want a universal script?

Yes, universal. To terminate any kind of terminal output

common mark in each line

The common mark is \n, the newline character. Before (or after?) that, we need to inject <br></br>. Except for the last line.

Forgot that one. Yeah, the solution is there. Only, do we expect a \n at the end of the last line?

Good question. Dunno. Is it important?

Damn. Looks like I messed up that question highlighting "except last line".

Dunno either. The question is, if it is there, do we want to strip it?

We probably don't want to strip any \n at all?

I believe I made it complicated from the beginning. The following should do the job.

#!/usr/bin/python
import sys
in_ = sys.argv[1]
out = '<br />'.join(in_.split('\n'))
print out

Note the '<br />'. Just found it.

Example:

x="$(ls -l)"
python add_html.py "$x"

Looks good. Works fine! :)

Perhaps change that line to.

out = '<br />\n'.join(in_.split('\n'))

Then it looks a bit better when running scripts using xtrace for debugging.

Could you commit that to usr/lib/msgcollector/br_add or so? Not sure. Perhaps you have a better name in mind.

Ah. Overlooked add_html. Fine either way.

In T97#2559, @Patrick wrote:

Looks good. Works fine! :)
Perhaps change that line to.

out = '<br />\n'.join(in_.split('\n'))

Then it looks a bit better when running scripts using xtrace for debugging.

The problem is that the \n will be displayed literally in the html, not interpreted as new line.

I don't think so. It works for me. Perfectly. Example:

(Stored the modified script as "a" for testing.)

x="$(cat --help)"

br_added="$(./a "$x")"

/usr/lib/msgcollector/msgdispatcher_dispatch_x "info" "test title" "$br_added" "" ""
echo "$br_added"
Usage: cat [OPTION]... [FILE]...<br />
Concatenate FILE(s), or standard input, to standard output.<br />
<br />
  -A, --show-all           equivalent to -vET<br />
  -b, --number-nonblank    number nonempty output lines, overrides -n<br />
  -e                       equivalent to -vE<br />
  -E, --show-ends          display $ at end of each line<br />
  -n, --number             number all output lines<br />
  -s, --squeeze-blank      suppress repeated empty output lines<br />
  -t                       equivalent to -vT<br />
  -T, --show-tabs          display TAB characters as ^I<br />
  -u                       (ignored)<br />
  -v, --show-nonprinting   use ^ and M- notation, except for LFD and TAB<br />
      --help     display this help and exit<br />
      --version  output version information and exit<br />
<br />
With no FILE, or when FILE is -, read standard input.<br />
<br />
Examples:<br />
  cat f - g  Output f's contents, then standard input, then g's contents.<br />
  cat        Copy standard input to standard output.<br />
<br />
Report cat bugs to bug-coreutils@gnu.org<br />
GNU coreutils home page: <http://www.gnu.org/software/coreutils/><br />
General help using GNU software: <http://www.gnu.org/gethelp/><br />
For complete documentation, run: info coreutils 'cat invocation'

The \n is not inside $br_added, so it won't be visible in the html either.

Nice demonstration. Was testing by adding \n randomly in the text.

Added file br_add to msgcollector.
https://github.com/troubadoour/msgcollector/commit/c68878afb05d9d9e5560fa819f3027b57ffd1918

Merged.

Can you please add a license header on top and a newline at the end?

Thanks! Merged. And added a minor fix on top (right order for license header and shebang).
https://github.com/Whonix/msgcollector/commit/ef03b5dc44873e441207667df9b813a991c9fdd9

Patrick changed the task status from Open to Review.Mar 5 2015, 4:57 PM

Created a follow up task: T216

Anything left to do here or closable, @troubadour?

We can close it.

Patrick closed this task as Resolved.Mar 5 2015, 8:40 PM
Patrick assigned this task to troubadour.