After work Cari convinced me to goto Dan’s and hang out. They were playing Scrabble. I was invited earlier by Dan and Jenn but I passed. Cari talked me into it since I had a terrible busy day. I ended up playing a round of Scrabble. Now first I have to mention if you don’t know me that I cannot spell. I mean I really CANNOT spell. I am not stupid (or I don’t think I am), but ever since I was little I had issues with spelling. it is partly due to a learning disability, but that is off topic. The point is for someone who considers themselves a terrible speller, how did I end up in a room with 8 people playing scrabble? And how did I actually have a decent score for half the game? Granted I ended up in last place out of the 2 others I played, but for a while I was actually giving them a run for their money. Come on people, I suck at spelling how can I do ok at Scrabble?
Day: 22 October 2004
A Single Minded Day
Some days I cannot remember what it was that I did at work. Today I only did one thing almost the entire day. I cannot forget what I did if I tried. I worked on fixing our Call Manager all day. Call Manager is the Voice Over IP version of the PBX. It is basically the computers that control our phone system. Last night Kai and Jayson upgraded our Cisco Call Manager’s to version 3.3.3 from 3.3.2. When you read the version number change you (I) think it is a minor fix. Well the latest version (it is the latest version we are using, but there is a 3.3.4, and a 4.0 now) does some things differently. Kai and Jay got the upgrade done, and tested all the phones in our call center. Everything seemed to work. What they couldn’t test was our remote phones. The phones in our NYC office didn’t work. We didn’t know this until hours after they went off to sleep. They left word that we may need to upgrade the firmware on the phones, but that was it. Turns out the firmware upgrades automatically when you reboot the phone and their is a newer version of the firmware.
The problem was the phones could make outbound calls, but could not receive calls. when you called the phones in a remote office the phone would ring but if you picked it up the Call Manager didn’t recognize that the call started. this is not good if we want to our reps to take calls. So off to call Cisco I went. This all happened before or as I got to the office. What a way to start the day that I knew I was going to be short handed anyway. Kai and Jay were both off to sleep because of their overnight. So after a long and drawn out troubleshooting session with Cisco we figured out our problem. I actually spoke to 2 guys over there. the first guy gave me the right answer but could not explain why we had to do what he asked. Since it required me to change firewall rules and we upgraded a phone system not the firewall I was skeptical. So the second Cisco guy came into the picture. He walked me through the same troubleshooting process (a bit quicker than the first guy). he then made some phone calls and got back to me. Turns out the first cisco support guy was right we needed to change some rules on our Cisco Pix’s. Why? Well in the new version of our CCM (Cisco Call Manager) they changed how some protocol’s operate. So what worked in older versions of CCM didn’t work in the newer version. We had to remove to fix-up protocol lines on all of our Pix’s that are involved in the VPN that makes up our WAN. Sure enough Kai was right. Kai as in Kai the cisco rep I spoke to, not Kai the guy I work with. I made the two firewall rules and the phones started working. Elapsed time on the whole saga, 8 hours. I got the phones working exactly at 5PM.
During this adventure I had other fun things to think about. Sean and danny dealt with problems with an index on a database somewhere that was causing one of our websites to be slow. Word of advise to people I work with. When I am fiddling with one phone, on another phone and talking to someone on a nextel and you know a system is down, don’t come and tell me about another problem that you need me to work on. I can only do 4-5 things at once. Thankfully Sean was able to get a handle on the index problem and fix it with little to no help from me.
Danny was helpful in my network trouble shooting saga today. Everyone else was surprisingly not bothersome. usually when I have a major problem people come out of the woodwork to bug me about minor issues, or that is how it feels.
I was crazed today. Am I glad it is over? Of course I am, but there is something to be said about days like this. First, it goes by so quick. Second, it is the type of day that you earn your salary. You get a few of the each year, and when you live it you hate it, but after you live through those days you are a better person for surviving them. On a personal note I am glad I got through the issue mostly by myself. Danny did give me great assistance in the network trapping, but allot of it was second opinion from what we got from Cisco. It is good having someone else around who you can sound off ideas to. I think Danny and I work great in that respect. On a whole I had to tackle most of this issue by myself. In the past that is not a big deal. Recently I have been delegating allot of the day to day technical responsibilities. It is something that I have to do, but I feel like I get rusty by not doing hands on work all the time. Days like today keep me sharp and lets me prove to myself that I am still in the game and can get dirty with the best of them, or so I think.