Stuff Network Admins Don't Like to Talk About
Right now I am writing to you from the lion's den. Today I am working in my most serious mode, as the company network administrator. As you well may know, there is a most serious e-mail virus running rampant across the world wide web. (No, it's not my newsletter!) It is the "I love you" virus, a particularly nasty one at that. One of our employees opened it, and it nailed us. Our anti-virus program, (only one day old) was worthless. In fact, no anti-virus software made before this very morning could detect it. It is simply too new-- it has an entirely different "signature" (the computer virus equivilent to DNA code) than anything seen before in virus-detection. This is what happens when a destructive computer virus infects a computer network, and this is what I, the administrator, have to do about it.
9:45AM. I get an urgent message from our office in England, warning us about the "I Love You" virus. It has been ravaging through Europe and the UK for several hours. I check out the story on the Web, discover it is true (not urban legend), and I start to compose a message to all the users, warning them about it. Before I can send it, my phone rings. The office personnel cannot run major programs, such as our manufacturing program. I verify this on my own desktop workstation, and several others.
10:05 AM. I look at the file server. The network has stalled. All software applications are "hung"- meaning that no one can access any file server based programs from any workstation. A short investigation indicates the worst possible diagnosis-- we've been hit by the virus. It's going to be
a VERY long day for me. I plot a strategy. My first job is to stop the virus from spreading further through the system, so I instruct all employees to shut down their workstations and disconnect them from the network. I shut down our Internet and E-mail connections next, thus isolating us from the World Wide Web-- and giving us a chance to recover our network before the virus can sneak in again.
10:20 AM. Via telephone, I contact several system integrators- these are the people gbbwho created much of our file system and operating software. They send me plenty of information about the virus, and they send it by fax-- because it's the only safe way they can get it to me. I learn as much as is currently known about the 'love bug' and how to vanquish it from my network. It is going to be a brutal job. This "I love you" malevolence has three distinct parts in its "love gift." First, it deletes all computer files that end with the file extension ".JPG". These are usually graphic files or photographic images-- like the pictures of our product we were going to use in our new catalog and on our web site. Then the virus goes to work on all ".MP3" files -- these are music and sound files-- thankfully we don't use any of these in my work, but if I were working at a music publisher--- OUCH! It destroy lots of other files too, and copies itself into their places. While these things are going on, the virus goes to the e-mail program and sends itself out to EYERYONE on the global e-mail address list. Finally, it makes some changes to the booting sequence, and commands the web browser to access a special web site-- and send all your security passwords there. This 'I love you' isn't really just a computer virus---- it is electronic pestilence.
10:30 AM. I call a meeting with the engineering department and the company president. I give a report on the network status, and outline a plan to fight the virus. First we need to inspect and troubleshoot the desktop workstations. There are about 20 of them in the building, so we break off into teams. Within forty-five minutes we have isolated the workstation that opened the virus, deleted its virus files and checked all the other desktops in the building.
11:40AM. We are now reasonably sure that the virus files are gone and the workstations are "clean." I now turn to the network file server. The file server is the "nerve center" of the network. The operating system and the major business data is stored here. It is most vulnerable link in the networking chain, and it is here that damage can be the most severe.
12:00PM. I go into the communications room and inspect the file server.
The virus has propagated itself throughout the file data. It looks very bad, but not fatal. So I sit here trying to sift through the damage, wondering if
there is enough data left for me to salvage in a few hours, or will I need to stay all night working a total reconstruction job. The decision is
mine, really. I have to make a good choice, for any mistake now will have dangerous consequences tomorrow. This is a point in time when I need to
draw upon my experience, and the experience of those whom I trust.
12:25PM The server is stable at the moment. I put in a few calls and pages to our support team. I try to eat my lunch and wait for them to call back. Virus reports are broadcast nearly constantly on the TV and radio news. I logon to the Internet from my laptop computer, and download the newest anti-viral software-- only an hour old. I will do it twice more during the day as newer versions are made available.
1:00PM No one has called me back. I decide to wait an hour. It seems like a year. I go ahead and delete the infected files and programs, as indicated by all the information I amassed from the anti-viral software websites.
2:00PM I get one call back from a consultant who is telling me that all the network guys are scrambling all over the U.S. trying to "get a handle on it." He advises me to "wait until tomorrow" when somebody can "come out and look it over." I told him that it was my computer network that was down, not my garage door opener. He said, "Sorry!" and hung up. My sense of humor had evaporated an hour ago. At this point, I was on my own.
2:30PM I had to come up with something. We have an informal meeting between the company president, the chief engineer and myself to plan the best course of action. My first choice is to do a total restoration, by erasing all the old information on the file server, thus removing any possibility of having
any residual parts of the virus left into the system. This seems to be wise at first, even though the process will take nearly thirteen hours to complete, including the necessesary system reconfigurations.
But in reality this is a foolish choice and the others chide me for it. It is too dangerous, as there are many possibilities for error. We believe that by deleting the infected files we have extinguished the virus in our system, and that we need only repair the damage it caused. This notion leads us to the plan of restoring only the lost and damaged computer files, using the archived backup files we have on hand. Our company policy is to make a total system backup onto a tape every night. This option has the least number of variables or "question marks" as we like to say. We do a little more research with the laptop and decide to go with this plan.
3:30PM I start examining the tape backups we made over the last weeks. I discover that they are not necessarily complete by any fashion. My predesessor was not verifying the integrity of the tape backups very well. If I had attempted to do a total deletion and restore from the tapes, we would have crashed beyond reasonable repair. A bad crash without a backup is like tumbling off the Earth into deep space - it doesn't matter what you do now-- you're done. I have one foot off of the planet now, and I could have
fallen any time.
4:00PM -8:30PM I restore the deleted files into the file server. Actually, the company president, the chief engineer and I work as a team, rebuilding databases, confirming overwrites and restorations. It is grinding, tedious and painful work. We try to keep a sense of humor about us, but the seriousness of the situation forms a strong undercurrent of tension that lurks beneath the veneer of our joviality. We are stressed. We kid each other about the jobs we are doing, but we never walk away from the file server for more than a few minutes at a time, and somebody always stays at the console. When the file server makes a noise, any noise, all conversation stops, and we look towards the machine with trepidation.
9:00PM The file restoration is complete. I reboot the machine and we watch for any anomaly that could signal disaster. The machine comes back to life, and we dash off to our workstations to test the software. It worked. Next I installed the latest antivirus software and updated all the workstations so the 'I love you' virus and its demonic variations would not get through our security system. Finally, we connected back to the Internet. Over the next twelve hours, the system would detect 165 attempts of the
virus to come in with the e-mail. (All of them were blocked)
10:15PM I called home to let Wendy know that the job was finished, and that I was coming home. She knew that it was a monumental task, and that I was physically and emotionally drained. She said that she was proud of me and ended the conversation my saying, "Remember I love you!" I responded, "Augh!
Don't say those words to me now!"
Anyway, I just wanted to let you folks know what a computer virus can do, and what is done to remove them. About the perpetrator of this 'I love you' thing, I will tell you now, that if I ever meet him, rather than 'love', I will be thinking 'justice.'
Gotta go........ I need to look for somebody.