DNS and How it WorksThe Domain Name System (DNS) is a relatively simple concept. Every computer and network on the internet is connected via an Internet Protocol (IP) address. An IP address uniquely identifies a machine or network gateway - and remembering the more than 4 billion addresses is a task for computers, not mere mortals. The domain name system associates a text name with one or more numerical IP addresses which provides many benefits beyond simple ease-of-recall. Redundancy can be enhanced by having a single name like 'www.google.com' point to many IP addresses. If one of those IP addresses becomes unreachable there are still other IPs your computer can try. Google is acknowledged as one of the most robust services on the internet and is always available. The very first public-facing method they use to maintain that robustness is by having 'www.google.com' point to (as of this writing) four IP addresses. DNS also provides portability. IP addresses sometimes have to be changed - but the name(s) stay the same which allows one to move a server from one network to another with little or no downtime.
Your home computer does not know how to turn text names into their associated IP addresses and offloads that task to DNS servers operated by your network or Internet Service Provider.
A useful analogy which will be used throughout this article is to envision a visit to the world's largest shopping mall - a massive imaginary edifice miles long and hundreds of stories high. As this is our first visit to the Mall, you stop at the local Information booth and ask for directions to the widget store. The operator of the Information booth can give you three possible answers:
- I'm not sure if there is a widget store, let me search our directory for it.
- I learned where the widget store is several days ago - it's right over there.
- There is no widget store in the mall.
If the booth operator doesn't know where the store is (and in a Mall this size that's common) then the booth operator has to look up the answer in the Store Directory. The Store Directory is truly massive - like a thousand volume encyclopedia set - but like a set of encyclopedias it has several Index volumes that allows the booth operator to rapidly find the right book to look up the answer for you.
In the DNS world the Master Index is maintained by the Root Name Servers - a set of geographically dispersed master DNS servers that by themselves cannot resolve names, but instead point queries to the right book in the encyclopedia set. There is an additional Index for each Top Level Domain (TLD), so when an operator needs to answer a query for 'www.example.com' they first consult the Root Name Server index and look up .COM. The root name servers tell the operator that all of the .COM domains can be looked up within Index N. The operator then consults Index N which contains all of the .COM listings and looks up 'example.com'. That index lists volumes 2,350 and 2,351 of the encyclopedia set as having 'example.com' in it. The operator then pulls volume 2,350 from the shelf and looks up 'www.example.com'. If that volume is not available, the operator can look up the same information in volume 2,351 instead.
So now through a series of 3 quick lookups the operator has the location of the widget store in the mall. When the operator confidently answers your question, you have no choice but to trust that they've provided you accurate information. Only... what if the operator decides to deliberately lie to you? Or the operator is truthful, but the directory they consulted to answer your question was deliberately loaded with misleading information?
The First Lie: "Site Finder"In 2003 Network Solutions (NetSol, aka Verisign) was the first to begin violating the trust placed in them with their "Site Finder" service. (The name chosen is at odds with its function since the entire concept depended upon not finding a site - hence the Lie.) To continue our analogy, Verisign is the keeper of the .COM and .NET Indexes that our information booth operator consults whenever they need to find a .COM or .NET domain name. That operator was now lying to everyone that asked for a store that didn't exist in the mall - pointing all of those shoppers to Verisign's store.
Verisign changed the way their indexes work. Normally if a domain name does not exist the DNS server responds with 'NXDOMAIN' (to indicate a Non-eXistent Domain). With Site Finder, NXDOMAIN itself ceased to exist and was replaced with a "wildcard" entry pointing all requests for non-existent domains to a special Verisign owned web site filled with advertising. This effectively "typo-squatted" the entire .COM and .NET namespace and was designed purely to make money for Verisign. What they failed to consider is that web browser traffic is not the only traffic that depends upon DNS. Email is especially sensitive to DNS issues because as most everyone who has received spam has discovered the "From" address of spam is rarely where that spam actually came from. There is never any need for a mail server to accept email if the "From" domain does not exist and this check is extremely effective against many forms of spam - but with Verisign's wildcard in place that check no longer worked. All .COM and .NET domains now "existed" thanks to Verisign's lie. Even more frustrating - Verisign didn't bother to localize the web site they directed all of that traffic to, so non-English speakers all over the world who typed in a non-existent domain name were confronted with a web site they didn't request in a language they might not read - instead of being presented with an error message in their native language that would have informed them of their typo.
ICANN, the governing body tasked with overseeing the IP address space and name management of the internet eventually was able to force Verisign to disable the "Site Finder" non-finder "service", but the damage had already been done. Verisign had shown every unscrupulous network and internet service provider how to make money by violating the trust placed in them. DNS Squatting had been born and it was only a matter of time before others began to mimic them.
Phorm and British TelecomDNS Squatting has never really gone away since 2003. Many implementations and variants have been discovered since then as providers began putting their bottom line before their scruples. In late 2007 and early 2008 DNS Squatting exploded to epidemic proportions when British Telecom, Virgin Media and Carphone Warehouse (three of the largest ISPs in the UK) began using their own even more invasive version of a Site Finder type program provided by the marketing firm Phorm. Not content with simple DNS Squatting, Phorm's "service" also acted as a proxy, a device standing between the internet and the end-user.
To continue the shopping mall analogy, imagine visiting this Mega Mall only to discover that you were never allowed to go anywhere inside it without an escort. The escort follows you from store to store, recording literally everything you look at and purchase. As you progress through the Mall your "escort" begins to get more and more invasive as it discovers your likes and dislikes, eventually reaching the point where that escort starts suggesting stores and products to you as you go. This type of detailed examination of everything you do is called "user profiling". The proxy is recording everything you do. Every site you visit, every search you conduct, everything - and it keeps that data about you forever. You have no privacy on the internet with Phorm and there is no end-user defense against this type of invasive analysis of your internet habits.
While what Phorm has been doing and the reaction to it is well beyond the scope of this article, the media storm caused by the revelation of what British Telecom was up to put a blazing spotlight upon this issue - which brings us to The Phormettes.
The PhormettesThe term was coined by the reporters at The Register to describe other marketing firms and ISPs jumping on the Phorm/Verisign style invasive marketing bandwagon. The Phormettes include:
- Paxfire - www.paxfire.com
- NebuAd - www.nebuad.com
- Front Porch - www.frontporch.com
- Adzilla - www.adzilla.com
- Project Rialto - www.projectrialto.com*
Paxfire provides a DNS appliance that essentially mimics Verisign's "Site Finder". If you type a domain name into your web browser and that domain name doesn't exist the Paxfire appliance sends you to a page loaded with advertising.
Front Porch is perhaps the least distasteful of this lot - they keep track of your browsing habits (as all of these can and will do) however where Front Porch kicks in is when you're visiting web sites that already contain advertising served up by ad networks. Front Porch (apparently) tips off the ad network about your habits to better target the ad to you personally. In other words, the ad network is going to throw something in your face on that web site anyway - Front Porch just lets the ad network know what ads they should try on you.
NebuAd has taken the Phorm route of a proxy server that includes the added "bonus" of Deep Packet Inspection.
Adzilla and Project Rialto are rumored to be in this category of Phormettes but currently not much is known about either of them as they are currently "in development". Translation of this type of spin is "we could tell you what we do, but then you'd freak out and we would be out of business".
The Big Lie - Improving Your Internet Experience?When called to the carpet every single one of these invasive marketing firms and the ISPs that utilize them attempts to spin their activities as "improving the internet experience". Nothing could be further from the truth. Their reason for existence and the reason ISPs sign on with them has one overarching concern: The Bottom Line. The internet advertising market is a $20 billion-with-a-b market and the ISPs, DNS Squatters and Invasive Marketers all want a chunk of that. I have no qualms about web sites and providers earning money from advertising - I earn money through advertising (or hope to at any rate). That said, the advertising model everyone is most familiar with is certainly a fair one: A web site or service places content on the web that is of relevance and interest to visitors and along with that content they include advertising. The visitor gets "free" content, and the web site operator might make a few pennies from the advertising included with that content. Everyone gets what they want.
With invasive marketing the advertising is shoved down your throat - whether you want it or not. Worse, the marketers take more from you than you might be willing to give. They track your habits and history and that information is of immense value. What did you get in return for that information they effectively stole from you? Oh that's right... nothing but grief.
Still, they claim they "improve the internet experience"? Pure marketing spin as I have proof that they in fact make it worse.
How Paxfire Stole GoogleI wrote this article over the course of a few weeks (I'm a slow writer) and my entire impetus for doing so is this section right here.
Paxfire stole Google.com.
Seriously. Stolen... or at least, that’s certainly what it looks like if you’re unfortunate enough to be stuck using a Paxfire DNS appliance like the two I found a few weeks ago.
The PHBs at Paxfire are at it again. This time instead of just reading about the issue in news reports or on Bugtraq I got to encounter it all first hand.
I’m an admin for an ISP (which shall remain nameless) and recently some of our dial-up customers complained that www.google.com stopped working for them. Other Google sites were fine - just not the main web URL. This lasted for the better part of a day and, as you might imagine, generated quite a few emails and telephone calls to our support minions.
As any good network admin can tell you - all else being equal if Google’s web site is unreachable then the only possible cause is Armageddon.
Upon receiving the first ticket I glanced out the window, saw the gloomy skies of Seattle and that the Earth was not currently clicking hot from an asteroid strike, and chalked the problem up to some misbehaving CPE and sent it back to the support people to figure out - perhaps that abomination known as Zone Alarm or something similar preventing the user from getting to Google’s web site.
After the second ticket from a different customer I started actually doing my job...Google was, of course, fine and I could not localize a problem. Then I got a third ticket, but this time it was from one of our many Linux using customers. (A DSL user on the road and using our dial-up service while he was out of town.) Most users with a problem like this pop off an email or call us with an “It’s broke” being about the extent of their analysis. This user instead did a quick nslookup and traceroute - and immediately spotted the problem. It appeared that the GlobalPOPs DNS servers had suffered from some cache poisoning because the IP they were returning for ‘www.google.com’ was definitely not Google’s. We couldn’t see the problem from our side because, surprisingly, we don’t attempt to manage several hundred servers in three datacenters by using dial-up. Here's what dial-up users were seeing:
www.google.com. 60 IN A 188.8.131.52 www.google.com. 60 IN A 184.108.40.206 www.google.com. 65535 IN NS WSC2.JOMAX.NET. www.google.com. 65535 IN NS WSC1.JOMAX.NET.
Like many ISPs our dial-up access is provided through a modem
aggregation service. What this means is that we don't actually own
the modems our customers dial in on. By outsourcing the modems to
a third-party an ISP can more easily offer nation-wide dial-up access
without the massive expense of setting up points-of-presence (POPs) all
over the country. The provider of our modem service is a company
OK so what’s going on here? After making several calls to the minions at GlobalPOPs and trying to get them to clean up their apparent cache poisoning someone there finally came clean: GlobalPOPs DNS wasn't poisoned: they're using Paxfire DNS appliances.
Paxfire boxes however are supposed to trap and ad-serve on bad domain names - not valid domains. Clearly that isn’t the case here. The IPs shown for ‘www.google.com’ are SWIPed to:
Internet Search Services INAP-DEN-INTERNETSEARCH-16579 (NET-63-251-179-0-1) 220.127.116.11 - 18.104.22.168 Co-Location.com Inc. LVLT-COLOC-1-8-15-7-96 (NET-8-15-7-96-1) 22.214.171.124 - 126.96.36.199
The lookup on the CoLo service is a dead end, but "Internet Search Services" gives us what we need:
NETfirstname.lastname@example.org: CustName: Internet Search Services Address: 462 Herndon Parkway Address: suite 201 City: Herndon StateProv: VA PostalCode: 20170 Country: USThat address looks familiar. A quick WHOIS on the paxfire.com domain and…
462 Herndon Parkway
Herndon, Virginia 20132
So as it turns out Paxfire is less capable than Google at keeping their service available to visitors. Who knew? Proving the lie about their service "improving the internet experience" Either the Paxfire network dropped or a server died and had stopped responding - which generated the tickets and started this whole mess.
Here’s a quick shot of how this presents itself to the unsuspecting dial-up user: (Click for full size image)
like Google, doesn’t it? I must admit if it weren’t for the tcpdump
it’d be impossible to tell the two apart. Note: The tcpdump in the
screenshot is edited to show the same packets both resolved as the host
name and unresolved as the IP address the data was coming from.
Paxfire is selectively lying to users of their devices. Queries for other search engines like Yahoo do not return results for Paxfire IPs. (From what I can tell) This Paxfire appliance only targets the 'www.google.com' name.
If all of this isn’t disturbing enough, iGoogle sign-ins use the ‘www.google.com’ hostname and while Paxfire would be unable to just grab the username and password used (thanks to SSL, man-in-the-middle attacks being what it was designed to foil) the fact remains that this remains a risk. Paxfire could easily stand in as Google by presenting their own SSL server certificate to the client and even stand is as the CA to verify its authenticity. *(Paxfire has the client machine’s DNS at its mercy after all) With that accomplished they “own” the supposedly encrypted traffic. On the other side of Paxfire’s server they could simply proxy their own connection, mirroring requests between Google and the client.
Scary thought ‘eh? Still, the good news is that Paxfire hasn’t taken their depravity that far. The SSL certificate presented by the connection through Paxfire is the same certificate being presented on a “clean” connection to Google. (Fingerprints and serials match) The fact remains they could pull it off if they wanted to.I and my fellow admins were Not Amused to discover what GlobalPOPs had done, and I put an override into our radius server to give our dial-up customers our DNS servers, instead of GlobalPOPs'.
I also notified Google Legal about this, but have not heard back from them. I honestly do not know if there's anything Google can do to compel Paxfire to stop pretending to be Google, but the fact that Paxfire is selectively choosing to proxy Google and no-one else is disturbing and, hopefully, actionable by Google.
How To Protect YourselfFor those caught in this mess, hard code the DNS of your computer or connection to that of a trusted DNS server. Your ISP or Network Provider should be able to walk you through doing so quite easily so give them a call.
If your ISP can’t provide you with clean DNS then visit OpenDNS. They do kinda-sorta the same thing as Paxfire so recommending them might seem odd, but OpenDNS does it without the invasiveness and their reputation depends on them not getting caught doing nefarious things like Paxfire does. OpenDNS is completely free and subscribers can use the service to block access to adult sites etc. (I have no affiliation with OpenDNS)
Personally? I run my own Linux DNS server for my home network. So long as the Root servers are OK, my DNS is clean.
Running your own DNS server may be beyond the abilities of the typical home user or small office network. If you want to run your own DNS without having to tangle with Linux or pay Redmond for their Bloatware license, or hire a professional then I would suggest SmartDNS Plus. SmartDNS is a Windows DNS server suitable for both end users or small offices. *Full Disclosure: I do have an association with SmartDNS. That's an affiliate link. If you click it and purchase their product I'll make money. See, this is how advertising should be done: Right up front so everyone knows what's going on.
Hopefully the information here will help people understand what DNS is, how it works, and how important it is that it be left alone. If a domain does not exist, DNS should return NXDOMAIN and that's that. Comments and critique welcome - if you think there's something here that needs expanding let me know.
IPv4 address space contains 255^4 (4,228,250,625) addresses. IPv4's
successor IPv6 contains 2^128 addresses - a number too big to print and
enough to supply every observable star in the known universe with a
IPv4 and IPv6 Fact Sheet (pdf)
published concerns regarding Verisign's "Site Finder", including a
detailed synopsis of everything Site Finder ended up breaking.
Phorm Files - a continuously updated compilation of all of The
Register's reporting regarding Phorm and British Telecom. An excellent
and detailed brief on what Phorm is up to and the fight against them.
Packet Inspection allows the network (or appliance in this case) to see
where you're going on the internet, but also to see what you're seeing
on the internet. This is the ultimate level of invasiveness.