Re: Repository services and budget

[Vince Wolodkin <wolodkin@digitalink.com>]

Perry E. Metzger wrote:
> 
> Vince Wolodkin writes:
> > > Furthermore, many organizations contracting to do the work will likely
> > > have existing 24x7 operations staffs who can deal with the
> > > problem. Remember that these machines will involve essentially no work
> > > other than backups on an ordinary basis. You care about the marginal
> > > work imposed on such a staff -- you don't need the 24 hour staff just
> > > for this. They'd end up spending all day and all night playing cards
> > > or watching videos. There would be nothing for them to do.
> >
> > This is what operations staff do when there aren't emergencies and all
> > their daily work is done.  Some of them study etc...  It is the nature
> > of the job.
> 
> Thats just plain false. I've been intimately involved with several
> operations staffs. Usually the policy is to keep them busy. If they
> have nothing to do, you are wasting personel. You staff to the level
> of work you need done. These aren't firemen -- they are
> operators. Usually a well run place will have the night staff
> performing activities that cannot be done during the day like systems
> maintanance, hardware upgrades, installs, etc, -- tasks that can be
> put off in case an emergency strikes.
> 

This is true if you have more than one present with nothing to do.  If
you want your center staffed 24x7 just to have a body in the room, then
it is quite possible he/she will have times with nothing to do.

> > Again, in my opinion, 24x7 means having someone on site at
> > all times.  If you don't you risk serious downtimes on the foibles of
> > the paging system.
> 
> Having a machine at a site where someone is at all times and having
> someone dedicated to the task at the site at all times are
> different. You can pay to colocate at a staffed facility, or the
> contract might be taken up by a company that, as I noted, already has
> a staffed facility and for which this would not be more than a
> marginal load.
> 
> Either way, the multimillion dollar figures are way exagerated.
> 

Hosting a machine in a staffed facility is not cheap.  I asked earlier
for prices, I haven't seen any forthcoming.  I know what it costs us to
co-locate.

> > > The price numbers I've seen, which keep rocketing up, seem way
> > > overinflated. This is not a multimillion dollar operation.
> >
> > It is what it is.  And by definition a large, shared multinational
> > database with 24x7 operations staff and either good private
> > connectivity, a hosting arrangement or a co-location arrangement costs
> > money.  You cannot expect to do this on a shoestring and expect it to
> > succeed.
> 
> I have a wee bit of experience here. I've been involved with running
> networks that spanned multiple continents, where we had 24x7
> operations staffs, and where if the network went down so did a large
> financial institution. Not only didn't those staffs sit idle all
> night, but we often managed most of the locations lights out thanks to
> clever management. Redundant hardware and remote management equipment
> beats extra humans. At one firm, we were fairly proud of the fact
> that all our equipment was redundant and we could perform any task --
> including reinstalling the operating system on a machine -- from a
> different city, without having to make anyone leave their desk.
> 

Great, sounds like a multi-million dollar operation.

> One of the interesting things I've learned over the years is this:
> that being cheap is stupid, but being extravagant is also
> stupid. Bigger boxes with more money spent on them aren't always
> better. Sure, you can make your web server a cray, but will anyone
> notice when a high end PC already saturates your T1?
> 

Couldn't agree with you more.

> Similarly, this problem could indeed be done on giant expensive
> hardware, but it isn't a big problem.
> 
> What you need here, chiefly, is a fairly ordinary database, like
> Oracle. The problem is by database management standards a fairly low
> level of load -- we aren't talking about millions of transactions per
> day. The database is also fairly small by modern standards.
> 

How many registrations are we expecting per day?  In the beginning I
would expect quite a lot as people rush off to get new domains.  Of
course not immediately, it would take severl months for everyone to
catch on.  Of course an active database with lots of additions each day
needs regular maintenance to keep it functioning normally.  Whenever you
design a facility and those who staff and manage it you make choices at
every step.  You weigh cost versus ability and you make your choices. 
If you demand 24x7, it doesn't leave much room for choices.

> This is not some bleeding edge operation, nor is it a challenging
> problem. You don't need three programmers at $80k a year, because it
> isn't going to take a year of time for three programmers to build the
> system. You don't need vast amounts of money for 24x7 staff because
> even if you need 24x7 staff you can piggyback on someone's existing
> staff. You also don't need huge computers, although redundant
> computers for availability and survivability would be a big plus.
> 

You forget that there will be a CORE, and that this CORE will need to
update the repository.  What happens when they have problems?  Well, the
smart thing would be to have the repository contractor  have the
experience and manpower to assist CORE members with technical problems. 
And as for piggybacking on someone else's staff, I believe this costs
more than you think it does.  And generally they will at best monitor
and page and backup.  You can probably get them to reboot.  Beyond that
I think you will be paying even more.

> A pair of modestly priced computers sitting in colocation facilities
> across the country from each other, one acting as primary and the
> other replicating the database and taking over in case of failure, is
> probably more than adequate. A couple of guys in charge of the project
> for the contracting organization can probably swing the entire
> problem, in cooperation with a shared operations staff that they will
> use very little time from.
> 

Please name somewhere suitable to do a co-location.  I will call them
and get a quote.  We are talking about someone who regularly offers
co-location as a business product aren't we?  Or do you have some
buddies at a NAP who will let you bring a machine in the back?

> Perry
> Speaking for myself, and not for the IAHC

I just think it is important for people to realize the scope of this
undertaking.  I am not suggesting mainframes and a ton of staff.  But I
am not agreeing with the skeleton operation you suggest either.

This might not even cost this much.  The central database manager may
already have their own equipment or CORE may decide it is better to
lease equipment.  We also cannot design this over a few e-mails and our
imaginations.  A solid specification of requirements needs banged out
before we can even guess reliably at staffing and hardware.

I would, however, still like to know about those cheap co-location
agreements.  I am always looking for cheap bandwidth for hosting servers
at:-)

Vince Wolodkin