Unless the VPN operator goes out of their way to do a sybil attack, Tor already protects users from the choke-point situations you describe:

"Tor tries not to choose relays which might belong to the same operator.

• Relay is not in the same /16 subnet. So Tor only uses one IP address in a range from 10.12.0.0 to 10.12.255.255 (10 and 12 can be any number).
• Operators who run more than one relay should declare those relays their 'family' (There is a special option in the configuration). Tor doesn't choose more then one relay from a family.
• A client can also set such a family option. This will also be respected."

https://tor.stackexchange.com/a/231

The VPN used as first or last proxy is not a relay that Tor picked. It's
something the user has set up. So we cannot be sure it is covered by that.

The question can be re-phrased. "I am running a Tor daemon in relay mode
on my computer. And I am running a separate Tor daemon in client mode.
Will the Tor in client mode avoid using my own Tor relay?"

scenario 1)

a) User uses VPN IP A on the host, thereby using it as it's first relay.
b) User's Tor client happens to pick a Tor exit relay running on VPN IP A.
Conditions a and b match at the same time. The user is now using the same IP as first and last proxy.

Simplified: VPN IP A - could be considered as any other ISP running surveilling a user. And the Tor Exit running on a VPN - a malicious exit. Some Exits are malicious but that does not break Tor. The Exit lasts for 10 minutes. The user is safe if the Tor Guard is OK. That is the most common situation anyway.

scenario 2)

a) User sets up a VPN inside Whonix-Workstation. Thereby that results in user -> Tor -> VPN -> internet. Using VPN IP A.
b) A Tor entry guard is being hosted on VPN IP A.
Conditions a and b match at the same time. The user is now using the same IP as first and last proxy.

Simplified: Malicious guard is pretty bad but on its own is not enough to deanonymize. As long as the Tor Exit is not also cooperating with the same party as the one hosting the Guard - Tor should not fail.

I am running a Tor daemon in relay mode
on my computer. And I am running a separate Tor daemon in client mode.
Will the Tor in client mode avoid using my own Tor relay?

This is a third (and different) scenario.

[fr33d0m4all] You can explicitly configure Tor client to exclude your own node by using ExcludeNodes and specifying the ID of your relay
[fr33d0m4all] user: ExcludeNodes node,node,…
[fr33d0m4all] A list of identity fingerprints, country codes, and address patterns of nodes to avoid when building a circuit. Country codes must be wrapped in braces; fingerprints may be preceded by a dollar sign. (Example: ExcludeNodes ABCD1234CDEF5678ABCD1234CDEF5678ABCD1234, {cc}, 255.254.0.0/8)

1) a) Is Tor clever enough to detect and avoid such situations?

Tor cannot detect something outside its network but a newer research version takes AS path selection and network adversaries into account when creating circuits. Based on this research:

http://freehaven.net/anonbib/cache/DBLP:conf/ccs/EdmanS09.pdf
http://freehaven.net/anonbib/cache/oakland2012-lastor.pdf

EDIT:

Its called ASToria

http://arxiv.org/pdf/1505.05173.pdf
https://github.com/sbunrg/Astoria/tree/master/astoria-tor-client
https://www.dailydot.com/politics/tor-astoria-timing-attack-client/

In T492#8634, @HulaHoop wrote:

scenario 1)

a) User uses VPN IP A on the host, thereby using it as it's first relay.
b) User's Tor client happens to pick a Tor exit relay running on VPN IP A.
Conditions a and b match at the same time. The user is now using the same IP as first and last proxy.

Simplified: VPN IP A - could be considered as any other ISP running surveilling a user. And the Tor Exit running on a VPN - a malicious exit. Some Exits are malicious but that does not break Tor. The Exit lasts for 10 minutes. The user is safe if the Tor Guard is OK. That is the most common situation anyway.

Considering it an ISP is a category mistake. You cannot go up to that threat model for the sake of the argument. Because Tor entry guards in the first place defeat only a weaker threat model, where an adversary hosts malicious Tor entry guards. The original case for Tor entry guards was to that your chances are better if you stick to the Tor entry guards. From the point of view, the ISP can be considered a permanent entry.

Afaik, Tor is most vulnerable if the adversary controls the first and the last relay and runs an end-to-end correlation attack. That's one reason why it's 3 and not 4 hops in Tor. Because those in the middle are less important. This is why Tor tries to avoid using a relay by the same host more than once in its chain.

scenario 2)

a) User sets up a VPN inside Whonix-Workstation. Thereby that results in user -> Tor -> VPN -> internet. Using VPN IP A.
b) A Tor entry guard is being hosted on VPN IP A.
Conditions a and b match at the same time. The user is now using the same IP as first and last proxy.

Simplified: Malicious guard is pretty bad but on its own is not enough to deanonymize. As long as the Tor Exit is not also cooperating with the same party as the one hosting the Guard - Tor should not fail.

After the Tor exit, it connects to the VPN before connecting to the destination. For practical purposes, is the VPN to be considered an extension of the Tor chain? Fact remains, that you are connecting to a server hosted by the same provider twice in your chain. That should make certain attacks easier. Perhaps some passive attacks. More certainly it allows to run active attacks. By tampering with the traffic at the earliest link in the chain, i.e. by adding a artificial delay, these effects could be measured at latest link in the chain. A more rough attack would be to selectively not serve most but one user [or delay] until there is a correlation.

It may not be the end of the world, but something useful to avoid.

I am running a Tor daemon in relay mode
on my computer. And I am running a separate Tor daemon in client mode.
Will the Tor in client mode avoid using my own Tor relay?

This is a third (and different) scenario.

[fr33d0m4all] You can explicitly configure Tor client to exclude your own node by using ExcludeNodes and specifying the ID of your relay
[fr33d0m4all] user: ExcludeNodes node,node,…
[fr33d0m4all] A list of identity fingerprints, country codes, and address patterns of nodes to avoid when building a circuit. Country codes must be wrapped in braces; fingerprints may be preceded by a dollar sign. (Example: ExcludeNodes ABCD1234CDEF5678ABCD1234CDEF5678ABCD1234, {cc}, 255.254.0.0/8)

You can certainly exclude it manually. But that was not the question. Good to have this information so it can be incorporated when asking the question. Because I am not looking for a workaround here that may or may not be needed. An authoritative answer would be useful since we are trying to establish a negative here. ("No, Tor does not figure out it's external IP and then tries to avoid using other relays hosted within the same /16 subnet.")

Tor routing algorithm questions:
https://lists.torproject.org/pipermail/tor-talk/2016-July/041753.html

answer by Roger:
https://lists.torproject.org/pipermail/tor-talk/2016-July/041756.html

tor-talk discussion:
using a VPN, proxy or ssh can make you actually less anonymous:
https://lists.torproject.org/pipermail/tor-talk/2016-July/041757.html

Maybe add a note that there is Tor research on picking routes through Autonomous Systems and IXPs less likely to be surveilled.

Somewhat related:

If this is as dangerous as you think then people using meek-google are completely hosed? Worth asking IMO.

Note that for correlation attacks to work they must be running the nodes themselves not merely looking at them from the outside. Though avoiding the latter is still better.