linux - Correct way to set up DNS primary/secondary/... for redundancy and latency reduction?

I thought DNS primary/secondary for redundancy purposes
was straightforward. My understanding is that you should have a primary and at least one
secondary, and that you should set up your secondary in a geographically different
location, but also behind a different router (see for example href="https://serverfault.com/questions/48087/why-are-there-several-nameservers-for-my-domain"
title="this
question">https://serverfault.com/questions/48087/why-are-there-several-nameservers-for-my-domain)

Currently,
we have two name servers both in our main data center. Recently, we've suffered some
outages for various reasons that took out both name servers, and left us and our
customers without working DNS for a few hours. I've asked my sysadmin team to finish
setting up a DNS server in another data center and configure it as the secondary name
server.

However, our sysadmins claim that this
doesn't help much if the other data center is not at least as dependable as the primary
data center. They claim that most clients will still fail to look up properly, or time
out too long, when the primary data center is
down.

Personally, I'm convinced we're not the
only company with this kind of problem and that it most likely is already a solved
problem. I can't imagine all those internet companies being affected by our kind of
problem. However, I can't find good online docs that explain what happens in failure
cases (for example, client timeouts) and how to work around
them.

What arguments can I use to poke holes in
our sysadmins' reasoning ? Any online resources I can consult to better understand the
problems they claim exist ?

Some
additional notes after reading the
replies:

• we're on
  Linux


• we have additional complicated DNS needs; our DNS
  entries are managed by some custom software, with BIND currently slaving from a Twisted
  DNS implementation, and some views in the mix as well. However we're completely capable
  of setting up our own DNS servers at another data
  center.


• I'm talking about authoritative DNS for outsiders
  to find our servers, not recursive DNS servers for our local clients.