North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: verizon.net and other email grief

  • From: Rich Kulawiec
  • Date: Fri Dec 10 12:29:31 2004

On Fri, Dec 10, 2004 at 02:43:21PM +0000, Simon Waters wrote:
> The most obvious is none of the three UK ISPs I have ready access to can 
> connect to port 25 on relay.verizon.net. (MX for all the verizon.net email 
> addresses). We can ping it (I'm sure it isn't singular?), but we have no more 
> luck delivering email than contacting verizon technical staff, logs suggests 
> we are in day 3 of this. I'm now listening to hold music at International 
> rates - ouch.

I think I can shine a little bit of light on what might be your
Verizon problem.

Summary:

Verizon has put in place an exceedingly stupid "anti-spam" system which
does not work, which facilitates DoS attacks, and which provides active
assistance to spammers.  Verizon has been told all of this, and it's
been discussed on Spam-L.  If there's been a response from Verizon,
I haven't seen it: and AFAIK the practice continues.  Anyone trying to
deliver mail there might want to at least skim this to get an idea of
the issues they may bump into.  Please note that in places this is
sketchy because it seems impossible to get Verizon to provide the
information necessary to make it otherwise (or correct any errors).

Details:

When an incoming SMTP connection is made to one of Verizon's MX's, they
allow it to proceed until the putative sender is specified, i.e. they
wait for this part of the SMTP transaction:

	MAIL From:<blah@example.com>

Then they pause the incoming connection.  And then they start up an
outbound SMTP connection from somewhere else on Verizon's network, back
to one of the MX's for example.com.  They then attempt to verify that
"blah" is a valid, deliverable address there.  Since most people have
long since disabled SMTP VRFY, they actually construct a fake message
and attempt delivery with RCPT.  If delivery looks like it's going to
succeed, they hang up this connection (which is rude), and un-pause
the incoming one, and allow it to proceed.  If delivery looks like
it's going to fail, then they also hang up their outbound  connection
(still rude), un-pause the incoming one, and reject the traffic.

This also means that if the MX they try to connect to is (a) busy
(b) down (c) unaware of all the deliverable addresses (d) something else,
that they'll refuse the incoming message.

It also means that if the address that's trying to send mail to Verizon
is something like "support@thuleracks.com", which is the address that
the people at Thule Racks emit support traffic from, but which doesn't
accept traffic, that Verizon will deny the message.  (Yeah, this isn't
very bright on Thule's part, either.)

Whoops.

This is bad for a whole bunch of reasons: two of the more obvious ones
are (a) it's a pathetic "anti-spam" measure because ANY forged address
ANYWHERE will do, and (b) it doesn't scale.  Add to that (c) it abuses
RCPT because apparently Verizon is unwilling to use VRFY and to accept
the decision of many mail server operators to disable it.  Oh, and (d) the
behavior of their probe systems is nearly indistinguishable from that of
spam-spewing zombies, which don't obey the SMTP protocol either.

	[ (b) is also how it lends itself to DoS attacks.  Sure, Verizon
	could rate-limit the rate at which they make outbound connections,
	but then attacker X could impose significant delay on mail
	from domain Y just by forging a boatload of messages purporting
	to be from addresses in Y to Verizon.  If Verizon rate-limits
	their outbound connections, then any real messages from Y will be
	stuck in the verification queue along with a kazillion forgeries.

	And beyond that: other people are foolishly adopting this
	"callback" nonsense as well.  Slashdot carried a note the other
	day about a program _designed_ to do this.  This allows attacker
	X to forge messages from domain Y to idiots I1, I2... In, for 
	a very large "n", and then stand back as all of them simultaneously
	try to connect to the MX's for domain Y.

	General principle: any "anti-spam" measure that generates more
	junk SMTP traffic at a time when we're drowning in it is probably
	a bad idea. ]

One thing that's not clear is whether or not Verizon caches any of
this information.  Doing so might help cut down on DoS attack methods
that involve them, but of course it doesn't do anything about those
which leverage everyone else who's doing "callbacks".

And this is unfortunately, not the end of it.

A lot of people, including me, are blocking particularly problematic
spammer-controlled networks at (a) our border routers (b) our firewalls
or (c) our mail servers.  In other words, we not only won't accept mail
from them, we won't even allow them to connect: we're blocking all IP
traffic from them.  This prevents them from spamming (at least directly
from their own network space); it also prevents them from using their
resources to build lists of deliverable addresses to sell to other
spammers by poking at our mail servers.

Since Verizon is doing this probing *from their network*, spammers can
easily get around all of our blocking by getting Verizon to do it for
them.  They can thus use Verizon to build/check their lists...and there's
no way for us to find out who's on the other side of these probes.

Which means that Verizon is running a free, anonymizing, spam support
service.

Note that refusing Verizon's probes, either accidentally or deliberately,
will of course cause Verizon to deny incoming mail.  Which I suppose might
be okay for some folks, it isn't workable here.

So for now all we can do is explain to our users that it's causing problems
and deal with it.  But if you look in your MTA logs and find things like:

Jul 15 07:24:51 <XXXXXXXXXXXXXXXXX@gsp.org>... User unknown
Jul 15 07:24:51 lost input channel from sc014pub.verizon.net [206.46.170.58] to MTA after rcpt
Jul 15 07:24:51 from=<antispam332078@west.verizon.net>, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc014pub.verizon.net [206.46.170.58]

then you can see what they're doing.

---Rsk