Tag Archives: work

Minnesota Terminal Shutdown

Greetings from fabulous Edgerton, Wisconsin!  Thanks to the powers of snow and ice, this is where I am staying tonight.  There’s been anywhere from 4-8″ of snow along I-94 and I-90 today, and with night coming, its starting to freeze.

If you’re too lazy to click the Wiki link up above, it basically says that Edgerton is slightly larger than Fremont, and has just as lame of a claim to fame.  For claiming to be “Tobacco City USA”, I’m noticing a distinct lack of discount tobacco shops.  Admittedly, this is in the middle of the night and I’m not gonna go looking for them.  But you’d think they’d be all over the place with a name like that.

I finished shutting down the Minnesota terminal today, and am in the process of hauling all the equipment and paperwork back to GR.  Unlike the Ft. Wayne closing last week, this is going to be a single trip, and only needs a 16′ truck.  About half the staff was in the office today while I was closing it.  While they did try and put a cheerful face on, they did seem understandably bummed.  They seemed like nice people, and I hope they’ve got other jobs lined up.

‘enable’ Requires 200mg Caffeine

Here’s a quick refresher on why it’s a good idea to stay away from enable mode on routers before you’ve had a couple cups of coffee:

drvr-wifi#show flash
System flash directory:
File  Length   Name
  1   4707392  cxxxx-bnt-3mz.122-5f.bin
[4707456 bytes used, 3681152 available, 8388608 total]
8192K bytes of processor board System flash (Read/Write)
drvr-wifi#delete cxxxx-bnt-3mz.122-5f.bin
Delete filename [cxxxx-bnt-3mz.122-5f.bin]?
Delete flash:cxxxx-bnt-3mz.122-5f.bin? [confirm]
drvr-wifi#copy tftp://10.254.1.5/cxxxx-bnt-3mz.122-5f.bin flash:cxxxx-bnt-3mz.122-5f.bin
Destination filename [cxxxx-bnt-3mz.122-5f.bin]?
Loading cxxxx-bnt-3mz.122-5f.bin from 10.254.1.5 (via FastEthernet0): !
%Error copying tftp://10.254.1.5/cxxxx-bnt-3mz.122-5f.bin (Not enough space on device)
drvr-wifi#reload
Proceed with reload? [confirm]
System Bootstrap, Version 12.0(3)T, RELEASE SOFTWARE (fc1)
Copyright (c) 1999 by cisco Systems, Inc.
C1700 platform with 32768 Kbytes of main memory
boot: cannot determine first file name on device "flash:"

For those of you not familiar with Cisco routers, what I inadvertently did was delete the firmware on the router, try to copy a new firmware to it, and missed the ‘Not enough space’ error that popped up. Upon reboot, the router couldn’t boot anymore and I was dropped to a rommon prompt.

For anyone else who might run into this, there is a handy guide on reflashing IOS from rommon and TFTP available at the Cisco Blog.

The Power of Numbers

Yesterday, I learned a valuable lesson on why you should never assume that a situtation will ‘never happen’ when it comes to a server or network.  In this case, it revolves around a POP3 mailserver and email attachments.

One remote site accesses all of its email via POP3, unlike all the other sites which are on Exchange.  This site is connected to the core office by a T1 link.  Normally, problems are few, as most traffic on the link is telnet.  Then, one user sent an email message.  A very large message.  Around 23MB large.  To all 89 employees at that site.

In going over the logs for that timeframe, the source message took about 10 minutes to send.  That didn’t cause any problems.  Its what happened once it hit the server that brought everything to a crawl.  All the users are setup on a Linux VM.  When sendmail received that 23MB attachment, for all 89 users, it made 89 distinct copies and gave one to each user (this here is why I love Single Instance Storage in Exchange).  The copies immediately chewed up a little over 2GB of space. 

Within minutes, the T1 link was suddenly brought to a standstill by the other 89 users’ Outlook doing a send/receive operation automatically.   The phones began ringing, and the problem was quickly tracked down.  However, the network link for the server did have to be disconnected for a few minutes to prevent users from getting a lock on their mailboxes, so we could clean them up.

Had the message been allowed to sit in place, it would have taken slightly over 3 hours and 5 minutes, at full saturation of the T1, for everyone to get their mail (and do nothing else during that time).  The results of this little fiasco?  Attachments now have a file size limit to match our Exchange limits, and POP3 traffic is rate-limited on the link to 768kbps.

I will be so glad once this location has been switched to Exchange.

Driver Confessed to 6 Killings

MendenhallFrom The Tennessean

“Metro police say Mendenhall, 56, a husband and father of two from Albion, Ill., made statements implicating himself in six killings in four states, including two in Middle Tennessee. … Postiglione said there is a “pretty good possibility” that Mendenhall could be linked to deaths besides these six.”

Here is a picture of police searching Mendenhall’s truck .

Curse you Backdoor.SDBot!

Well, Im finally getting a widespread infection of SDBot under control here at work. Spreading via various DCOM and RPC exploits, SDBot caught us completely off guard, as we were used to email viruses instead (we still get ILOVEYOU.vbs opened every now and then).

Partly, it was our fault for not keeping our Win2K/XP machines up to date. We figured they are all behind the firewall, so nothing on port 445, 5000, or whatever it uses can get in. This overlooked the fact that someone might (against company policy) bring in a home PC and plug it into the network (which is probably how this got in).

To clean each system takes about 2 hours, which consists of installing SP2 (SP4 + an RPC patch on Win2K), resetting all Internet Explorer settings, and running SpyBot and AdAware on the system (because this variant drops about 6 different spyware programs). It is a royal pain in the ass. The person who wrote such a beast must be subjected to horrendous pain and torture by my hand.

Mr. Operator

Yes, genius. Call me before you call the person you want just so I can tell you if they are there or not. Actual phone transcript follows:

“Hey Jim, its ___ in Minnesota. Is ___ there?”
“Yes, shes here. Is she not answering her phone?”
“Oh no, I havent tried yet. I was just calling you to find out if shes there. Can you transfer me?”