Minor Error—Big Headache

"I rebuilt the DNS server twice according to our documentation, with no results. Finally, I called the hosting company, who informed me the parent company had changed all DNS and MX records to their servers."
—Scott Russell, Network Administrator

When network administrator Scott Russell answered his phone at 8:00 on a Thursday night and heard the IT administrator for his previous employer on the line, Scott knew it wasn't a social call. The administrator sounded desperate as he described the problem. About 2 days earlier, the company's Internet service had gone down. Employees couldn't access the Internet, send or receive external email via the company's Exchange server, or even log on to the company's Windows 2000 Server network. The two-person IT staff had tried everything they could think of to restore the network services, with no success. Scott generously volunteered his time and expertise to troubleshoot the problem—which, 6 hours later, he determined to be a DNS configuration error. Windows IT Pro senior editor Anne Grubb spoke with Scott, a 10-year IT veteran who's currently a network administrator for ABC Window Company in Ontario, California, about how he traced the problem's cause and restored the company's Internet and email services.

Your previous employer called you on a weeknight to solve a network problem that had been plaguing the company for more than 2 days. What was going on?

Basically, they couldn't log on, use the Internet, or send external email. While I was the network administrator there, I had set up two domains: an external company.com domain, for the Web and SMTP, and an internal company.net domain. We never registered the .net domain because it was internal, and an ISP provided the company's Web service and hosted the external DNS and MX records.

When I arrived on site, the first thing I did was check and recheck the DNS settings several times to be sure everything was correct. The company had three domain controllers (DCs) and two DNS servers, which we'd set up while I was there, by using the Windows DNS service on the clients and integrated Active Directory (AD). I actually rebuilt the DNS server twice, using the documentation I'd created before I left. It was a big process.

We wanted to figure out why DNS wasn't resolving the IP address to the DC in the local .net domain. So we ran the Traceroute utility on the IP address that we thought we were using, and we saw that it went out to a different external IP address. We had no idea what system the address was going to, although the address was in the same range of addresses as ours. Our first thought was that we'd been hijacked.

That's pretty scary. How did you figure out what server the IP address led to?

Well, first I went to Microsoft Help and Support (http://www.support.microsoft.com) and spent about 2 1/2 hours there. As I was looking through articles on the site, it occurred to me that someone else had registered the domain name. So at that point, I decided to call the ISP. I waited 3 hours for the ISP to call me back, and when they did, they told me that the unknown address was pointing to a server that belonged to ISP's parent company. That's when we started to figure things out—that the parent company had registered our .net domain to another IP address.

Next, we called the ISP's parent company, and they confirmed what we suspected. The parent company had registered the .net domain and changed all the DNS records, including the MX records, to their servers—without telling their customers.

So you figured out that the ISP's parent had taken over your former employer's internal domain name. Changing the domain name probably wasn't an option. How did you solve the problem?

When I called the parent company, I asked them to send me the new IP address and zone information for their DNS server, which they did. Next, we needed to set up our DNS record as a secondary DNS server to their DNS server, so that we could actually log on and correct everything. I added a secondary zone to our DNS record and listed their DNS zone as our secondary DNS zone, so that our DNS server could resolve the external company.com address to the correct address.

At that point, we could actually log on to the Internet again. The next step was to replicate the secondary DNS zone to AD. After we did this, everything worked correctly, and the company's users could log on to the server and use their external email and Internet service again.

Your story contains some valuable lessons for Windows network administrators.What advice would you give your fellow IT pros, as a result of this experience?

First, if you're using an ISP and you're having an Internet problem, be sure to communicate with the ISP and its parent company. Before you start changing your DNS setup, ask the ISP's technical staff whether they've changed anything in their configuration. Today with all the mergers, that's a big issue. I think after our experience, the ISP's parent company learned that they needed to communicate better to their customers when making such changes.

Also, I learned my lesson about using the .net domain. Now I use .local for all internal DNS settings. You can't get into trouble with that one because you can't register it!

Anne Grubb (agrubb@windowsitpro.com) is a senior editor for Windows IT Pro. She has more than 20 years of experience as a writer and editor of articles, books, and other materials in the computer, business, and legal fields.

End of Article