The standard-practice way to do this is that the gateway sends an http put request to an API (http://whonix.org/api/v1/gateway-check-in), and all the logic (is this the same machine? have they reported in today already?) is on the server. With basic logging, it's a trivial project for any backend programmer (that is, even I could do it :P)

The logic inside whonixcheck:

• not download the file more often than once in 24 hours
• run at bootup if we need to fetch the file
• don't fetch again if the user runs whonixcheck manually
• run at randomized times (already implemented in whonixcheck)
  • partially implemented already https://github.com/Whonix/whonixcheck/blob/master/usr/lib/whonixcheck/autostart.bsh
• maybe whonixcheck is not the right place to run this (since autorun vs not autorun discussion https://forums.whonix.org/t/whonixcheck-whonix-14-ideas)

We need something lightweight to code and to run on the server.

Can we count how many times a file was downloaded for a specific file on the server? @fortasse (Given our setup with nginx, varnish and apache)

We can definitely do this, especially if that's the way the client works.

We can just grep it from the logs every 24 hours, and do whatever with the line count. I don't know how fancy we want to get with it, but we can just email it or something?

Great! We don't need publication yet. Viewing it manually on the server
would suffice.

run at bootup if we need to fetch the file

Can you clarify that? By "fetch the file" you mean "report in", correct? What is the logic at bootup? Do you mean: "Check at bootup if we've reported in today. If not, do it now." ?

Would the simplest code be something like...

Inside whonixcheck autostart:

read a text file with a time stamp (/etc/whonixcheck/lastreport.cfg or wutever) , handle errors.
If time stamp is greater than 24 hours  from now then 
wget --method=PUT https://usercount.whonix.org/api/1/countme (or curl -X PUT)
echo $current_time > /etc/whonixcheck/lastreport.cfg

JasonJAyalaP (Jason J. Ayala P.):

JasonJAyalaP added a comment.

> run at bootup if we need to fetch the file

Can you clarify that? By "fetch the file" you mean "report in", correct?

fetch the file = download the file

download the file = counter as "report in" on the server

What is the logic at bootup? Do you mean: "Check at bootup if we've

reported in today. If not, do it now." ?

Yes.

JasonJAyalaP (Jason J. Ayala P.):

JasonJAyalaP added a comment.

Would the simplest code be something like...

Something like this, yes.

Inside whonixcheck autostart:

  read a text file with a time stamp (/etc/whonixcheck/lastreport.cfg or wutever) , handle errors.

/var/cache/whonixcheck/user_counter.txt

If time stamp is greater than 24 hours then 
wget --method=PUT https://usercount.whonix.org/api/1/countme

Use curl. (See whonixcheck for other invocations of curl and see which
command line options make sense.)

Download from Whonix onion.

We place the file on a download.whonix.org sub folder (so we don't have
to maintain a new sub domain).

https://download.whonix.org/whonixdevelopermetafiles/internal/gateway_counter
or so? Would that work? @fortasse

echo $current_time > /etc/whonixcheck/lastreport.cfg

Access rights need to be sorted out. (Check if we are already using
/var/cache/whonixcheck/ and if it's owned by user whonixcheck).

That should be a perfectly fine file path. Do we need to separate out .onion downloads vs .org downloads?

fortasse (fortasse):

Do we need to separate out .onion downloads vs .org downloads?

If that is doable without too much effort... Then this would spare us a
few curious manual downloads and thereby needlessly increasing the counter.

Perhaps the file location should be even more obscure to prevent that?

Download from Whonix onion.

No need to download a blank file. Just "pinging" server with a put request is all that's needed to appear on the log. Correct, @fortasse?

/var/cache/whonixcheck/user_counter.txt
echo $current_time > /etc/whonixcheck/lastreport.cfg

@Patrick Do you want the last check to be /var/cache/whonixcheck or /etc/whonixcheck ? I think /var/lib/whonixcheck is the correct FHS location (http://www.pathname.com/fhs/2.2/fhs-5.8.html)

JasonJAyalaP (Jason J. Ayala P.):

JasonJAyalaP added a comment.

> /var/cache/whonixcheck/user_counter.txt
>  echo $current_time > /etc/whonixcheck/lastreport.cfg

@Patrick Do you want the last check to be /var/cache/whonixcheck or /etc/whonixcheck ? I think /var/lib/whonixcheck is the correct FHS location (http://www.pathname.com/fhs/2.2/fhs-5.8.html)

Yes, /var/lib/whonixcheck folder seems better suited.

JasonJAyalaP (Jason J. Ayala P.):

JasonJAyalaP added a comment.

> Use curl. (See whonixcheck for other invocations of curl and see which

command line options make sense.)

> Download from Whonix onion.

No need to download a blank file. Just "pinging" server with a put request is all that's needed to appear on the log. Correct, @fortasse?

Yes, if downloading a blank file could be avoided that would be cool.
That would prevent messing up the log by curious users downloading that
empty file when scrutinizing download.whonix.org.

(Might also be easier to prevent varnish from caching the request?)

(Still use the onion.)

OK so I asked upstream if there is a way they can collect the type ofstats we want but unfortunately Tor clients do not and likely never will be able to collect this type of data safely.

https://lists.torproject.org/pipermail/tor-dev/2017-June/012327.html

The important thing I'm getting is that to do this in a privacy preserving way these criteria need to be covered:

Any client protocol would need to have the following properties:

1. avoid client tracking, and
2. secure aggregation of counts for each type of client, so counts could not be linked to individual guards.

I think you have this covered since no individual clients have identifying IDs or anything similar.

Here's some ruby:

file = "checkin.log"
if File.owned? file then # Deleting the log file or changing the owner will disable the check
    now = Time.now
    if File.mtime(file) + 86400 < now then # Always wait at least a day after the last modification of the log file
        require 'net/http'
        require 'uri'
        begin
            uri = URI('http://whonix.org/api/1/checkin') # API address
            params = {system: "gateway"} # uri?system=gateway
            response = Net::HTTP.post_form(uri, params)
        rescue SocketError
            STDERR.puts "Connection error while checking in"
            exit 1
        rescue StandardError # Handles all forms of errors, inelegantly.
            STDERR.puts "Error while checking in"
            exit 1
        end
        File.open(file, 'a+') do |f| # Appends to file
            f.puts(now.to_s + " -> " + response.to_s)
        end
    end
end

Design decisions:

1. There's no need to read the file to see last check time. Just read the modification time.
2. Delete the file is a hacky (but easy) way to disable the check.
3. The file can be used as a log. Each line is "Time : Response". Right now, response is simply "moved" since there isn't a proper response set up. I can remove the response or just leave.
4. file = checkin.log will be file = /var/lib/whonixcheck/checking.log

Problems or possible changes:

1. /api/1/ is just me dreaming. It can just be whonix.org. With the system=gateway params, the log can be searched for that. It's up to fortasse.
2. !!! Ruby is being blocked from making the tcp connection to resolve the address on gateway
3. Should this check just clearnet, onion, or both?
4. This could be quickly translated to bash (with curl -X POST -d system=gateway https://whonix.org/), Patrick, if that's easier than piping out to ruby.
5. If the file was deleted (presumably on purpose to disable the check), checkin.rb will still $? as 0. This can be changed

In T689#13771, @JasonJAyalaP wrote:

1. Delete the file is a hacky (but easy) way to disable the check.

If we implement it into whonixcheck, use whonixcheck_skip_functions mechanism.

whonixcheck_skip_functions+=" function_name "

If we implement it into whonixcheck:

• we must keep autorun whonixcheck at boot on Whonix-Gateway
• we must keep autorun whonixcheck (at least in user counting mode) every 24 hours
  • https://github.com/Whonix/whonixcheck/blob/master/lib/systemd/system/whonixcheck.service
  • https://github.com/Whonix/whonixcheck/blob/master/usr/lib/whonixcheckdaemon
  • https://github.com/Whonix/whonixcheck/blob/master/usr/lib/whonixcheck/autostart.bsh

Since that mechanism is kinda complex, and since we wanted (long term) to get rid of whonixcheck autostart anyhow... Might be better to have a separate systemd unit file / daemon for this purpose. Disabling could be done by using sudo systemctl mask whonix-user-counting or so.

On the other hand, whonixcheck already is covered with apparmor and systemd hardening.

1. /api/1/ is just me dreaming. It can just be whonix.org. With the system=gateway params, the log can be searched for that. It's up to fortasse.

Please use the onion. Perhaps api.onion/...?

1. !!! Ruby is being blocked from making the tcp connection to resolve the address on gateway

Because Whonix-Gateway has intentionally no system default networking.

( https://www.whonix.org/wiki/Whonix-Gateways_Own_Traffic_Transparent_Proxy )

Therefore probably better to use curl, as per below.

1. Should this check just clearnet, onion, or both?

Just onion.

1. This could be quickly translated to bash (with curl -X POST -d system=gateway https://whonix.org/), Patrick, if that's easier than piping out to ruby.

Yes, because then stream isolation gets easier. Also then because invocation of curl can remain standardized. Example usage of curl in whonixcheck.

curl.anondist-orig --fail --proxy socks5h://user:password@127.0.0.1:9110 --tlsv1 --proto =https --max-time 180 --output /tmp/tmp.Bg7xq574AW/check_tpo_tor_socksport.html https://check.torproject.org

• using curl.anondist-orig so we're not using the user's default stream isolated instance
• --fail so curl will exit non-zero in case of issues for better error handling
• using --proxy socks5h://user:password@127.0.0.1:9110 to make connectivity even work and use use a separate Tor SocksPort (we can use 9110 because connections to onions are stream isolated anyhow and the gateway otherwise won't connect to Whonix onion on that socks port)
• --tlsv1 --proto =https should be omitted, since we will be using the onion
• --max-time 180 so it won't hang forever
• --output may not be required since we won't download anything, however, we might want to keep it anyhow to store server responses for debugging

As for $CURL invocation, please compare with how it's done here:

• usr/lib/whonixcheck/check_news.bsh
• usr/lib/whonixcheck/check_tor_socks_port_reachability.bsh

I mean, please also use timeout. Alternatively, not sure if curl's --connect-timeout and --max-time alone would be reliable enough to never hang. Would require testing against a prepared server. Replicating timeout method is probably easier.

1. If the file was deleted (presumably on purpose to disable the check), checkin.rb will still $? as 0. This can be changed

Deleting a file to disable a check does not sound right. That sounds rather surprising.

1. Keep most of it as ruby
2. Change the native ruby http commands instead to pipe out to curl
3. return 1 if curl returns 1
4. make the ruby file a daemon/service
5. create log file if doesn't exist, return 1 if not writeable.

I've never done the daemon thing before. Any tips? I might need you to create the basic bash setup. Dunno. I just need some way for checkin.rb to fire every X hours. Or maybe on bootup and day change?

I don't see why we need ruby for this. There is another issue. Running this after boot will likely fail, because we also need to wait until Tor is enabled, Tor is fully bootstrapped, clock is sane, and in absence of other system issues. Thereby reinventing whonixcheck. Also no need to reinvent the daemon mechanism. Your lock file mechanism makes sense. Let me implement this one in bash.

@fortasse could you please use the following test command.

curl --fail --proxy socks5h://user:password@127.0.0.1:9050 --location --max-time 180 --output ./whonix_census.txt --request PUT --data gateway-sign-in http://api.kkkkkkkkkk63ava6.onion/whonix_census

Current server reply is:

curl: (22) The requested URL returned error: 405 Method Not Allowed

Any idea how to make the server reply something sensible such as ok?

Mostly done.

https://github.com/Whonix/whonixcheck/commit/3da614e8a02e3606de93618bea3f32075aa354c4

Needs more work. Let's give @fortasse time to do the server part.

In T689#13857, @Patrick wrote:

Mostly done.

https://github.com/Whonix/whonixcheck/commit/3da614e8a02e3606de93618bea3f32075aa354c4

Needs more work. Let's give @fortasse time to do the server part.

Up to me since 06/27/2017 getting this done.

documented here:
https://www.whonix.org/wiki/Whonixcheck_Hardening#Prevent_Downloading_Whonix_.E2.84.A2_News_and_Whonix_.E2.84.A2_User_Census_Counting