tail -f carlo.log

Mar 31 2004

Mar 31st 2004, 17:47 GMT

Quicksilver must be the slickest application / OS enhancement I’ve seen in my whole life. I’ve installed it on our iBook, and I can’t do without it anymore. It’s like crack. And free. Like free crack. Yeah, that pretty much sums it up.

And it’s for OSX only. Which is great if you’re an OSX user, but it sucks if you have a Windows machine as your primary work horse. Like me.

Mar 31 2004

More info on the recent outage

I just got another mail from my provider, consoling my beliefs that they are indeed the best damn provider a guy like me can have. ;) And since I suspect you want to know what went wrong at what point, I’ll repost the mail here.

[Quote] Hello,

The severe network outage that occurred throughout the day on Monday, March 29,
2004 was due to a failure of several parts of our system as a whole, both
technical and procedural. We have already begun taking steps to improve our
service and strengthen our network to better handle any future outages.

We now know the network outage was the result of a malicious Denial of Service
attack aimed at a website hosted on our servers. According to our network
graphs, the first wave of the attack started at around 2:30am PST (GMT -8). It
then resumed at around 5am PST, and continued in earnest from there with only a
few hours of network availability until it was resolved completely at around 5pm
PST. Due to the nature of the problem itself and the complexity of our
redundant multiple router setup, it was not immediately apparent that we were
under attack. Ordinarilly you can immediately tell because the router itself is
overloaded. In this case however, it was so overwhelmed that it completely
malfunctioned and the ordinary diagnostic tools did not paint a full picture of
the situation. We were also unable to access our own network graphs due to the
scale of the attack so it was assumed a hardware failure of the router was
responsible. As a result, many misdirected solutions were attempted and failed
while the mental pressure on our network engineers continued to mount.

After several solutions failed, the scope was broadened and we finally realized
the true nature of the situation. To resolve the Denial of Service attack, we
worked with our multiple redundant upstream network providers to divert and
eliminate the malicious traffic before it reached our network. Some network
providers are more responsive than others so this didn’t happen on all of them
simultaneously, but the largest chunk of the attack was diverted very quickly.

Once network access was restored to our system, several of our servers crashed
under the sudden load or other problems stemming from the previous lack of
network access. We quickly dealt with those server problems and got all sites
back up within an hour or so. That explains why some of your sites were still
down while our own sites were up and available. There were also a few
outstanding network issues that weren’t fully dealt with until later in the
evening and some of you were probably affected by that.

Several announcements were sent out during the course of the day, but they were
not immediately delivered due to the network problem. We are dealing with that
Catch-22 situation in a way that I will detail below as I outline our steps to
prevent this problem in the future.

We were slower to handle the problem than we should have been for a couple of
reasons. Our weekend night staff was not properly prepared to handle the
situation. We are now in the course of preparing a procedural guide for this
type of network attack and will be thoroughly training all of our night staff.
Also, our own network monitoring system was not properly configured to notify us
of this particular type of situation and our response time was slower as a
result. We already had one type of monitor in development and it did catch the
problem early on, but since it is still in development it was not configured to
send out proper notifications. Needless to say, development on that monitor
will be accelerated!

We are also accelerating our plans to deploy new, more powerful routers. Our
existing routers were already nearing the end of their service to us and we will
now speed them off to an early retirement. The new routers will be able to
handle several times more traffic than our existing equipment and will not
buckle under the load so easily, if at all. We will not stand by and allow
malicious attackers to take advantage of us so easily again. We expect the new
routers to be installed and running in the next couple of weeks. We obviously
must avoid as much downtime as possible during the transition so it will be
handled delicately.

Many of you mentioned that the lack of information from us compounded the
problem severely and we have also already begun taking steps to alleviate that
issue in the event that a similar situation should occur in the future. We are
setting up an off-network status and updates system. We will publish more
detailed information about this in the future. It will consist of a server
hosted on a secondary network that will be accessible even when our primary
network is not for some reason. The status system will be updated with the
latest news during any emergency situation. We will also set it up to allow us
to notify all of our customers at secondary email accounts so we can keep you in
the loop with all available information. We hope to never have to use this
emergency system of course, but we will perform periodic tests of it to ensure
it is always functioning properly.

That covers the network outage from yesterday, but several of you also had
questions about our previous central database problem. Our central database is
the core of our entire DreamHost automation system and is what provides you with
the extensive self-service control over your accounts you have no doubt come to
appreciate. Our central database is vital to our service and it is set up in a
fully data redundant manner. Unfortunately, it was not able to handle software
failures and we experienced some downtime as a result. We have already taken
some initial steps to reduce the potential for problems in the short-term and
are in the process of developing our next generation fully-clustered central
database. Once that’s installed and running, overall performance and stability
of our central database will be much improved. Note that our central database
is separate from our customer database machines. Those machines were fully up
and running throughout the central database problems.

As before, we will continue to keep close watch on all of our hosting servers
and work quickly to eliminate individual problems as they arise. Note that the
vast majority of these problems are caused by individual users so anything you
can do to reduce the server impact of your own web applications would benefit
all of your fellow DreamHost users.

Please let us know if you have any more questions about this situation. We are
currently a little behind on our support queue, but we are working extra hard to
clear it out as quickly as possible. We will get to your questions eventually
so bear with us. Thank you for taking the time to read this message and have a
good day!

Happy DreamHost Major Overhaul Team [Quote]

There were problems, they are admitting it and furthermore explaining the situation to their customers. I didn’t expect anything else from them.

Mar 31 2004

Mar 31st 2004, 10:15 GMT

gods art: Beautiful site. [via ISM]

UPDATE: I accidently linked a nice picture in the depths of the site instead the front page. Apologies… link updated.

Edited on Mar 31st 2004, 19:23 by Gossip

Mar 31 2004

Mar 31st 2004, 09:50 GMT

— Posted under , , , , ,

World of Warcraft Interface Customization Guide:

The interface of World of Warcraft is built from XML files which describe the look and layout, and lua files which contain scripting functionality. This document is a short introduction into modifying these files to customize your interface. Customizing the interface is a very technical endeavor, and you should not attempt it unless you have a good working knowledge of XML and Lua.

[via Morn] I paraphrase: They are officially allowing customization of the WoW user interface. Bold move, smart move, and I love it.

At this point, I want to have Blizzard’s puppies.

Mar 30 2004

Mar 30th 2004, 08:04 GMT

Bloglines New Feature: Ignore Updated Items:

[Quote] For blogs that are constantly updating items, you can now have Bloglines ignore those updates, and only show new items. From the My Blogs page, click on the subscription you wish to change. Clicking the Edit link on the blog header in the right pane will let you edit the attributes of the subscription, including whether updated items are to be ignored. [Quote]

Bueno!

← Older Entries
photo of Carlo Zottmann Carlo Zottmann carlo@zottmann.org
München Germany
AIM YIM Jabber