Work

A Single Minded Day

by

Some days I cannot remember what it was that I did at work. Today I only did one thing almost the entire day. I cannot forget what I did if I tried. I worked on fixing our Call Manager all day. Call Manager is the Voice Over IP version of the PBX. It is basically the computers that control our phone system. Last night Kai and Jayson upgraded our Cisco Call Manager’s to version 3.3.3 from 3.3.2. When you read the version number change you (I) think it is a minor fix. Well the latest version (it is the latest version we are using, but there is a 3.3.4, and a 4.0 now) does some things differently. Kai and Jay got the upgrade done, and tested all the phones in our call center. Everything seemed to work. What they couldn’t test was our remote phones. The phones in our NYC office didn’t work. We didn’t know this until hours after they went off to sleep. They left word that we may need to upgrade the firmware on the phones, but that was it. Turns out the firmware upgrades automatically when you reboot the phone and their is a newer version of the firmware.

The problem was the phones could make outbound calls, but could not receive calls. when you called the phones in a remote office the phone would ring but if you picked it up the Call Manager didn’t recognize that the call started. this is not good if we want to our reps to take calls. So off to call Cisco I went. This all happened before or as I got to the office. What a way to start the day that I knew I was going to be short handed anyway. Kai and Jay were both off to sleep because of their overnight. So after a long and drawn out troubleshooting session with Cisco we figured out our problem. I actually spoke to 2 guys over there. the first guy gave me the right answer but could not explain why we had to do what he asked. Since it required me to change firewall rules and we upgraded a phone system not the firewall I was skeptical. So the second Cisco guy came into the picture. He walked me through the same troubleshooting process (a bit quicker than the first guy). he then made some phone calls and got back to me. Turns out the first cisco support guy was right we needed to change some rules on our Cisco Pix’s. Why? Well in the new version of our CCM (Cisco Call Manager) they changed how some protocol’s operate. So what worked in older versions of CCM didn’t work in the newer version. We had to remove to fix-up protocol lines on all of our Pix’s that are involved in the VPN that makes up our WAN. Sure enough Kai was right. Kai as in Kai the cisco rep I spoke to, not Kai the guy I work with. I made the two firewall rules and the phones started working. Elapsed time on the whole saga, 8 hours. I got the phones working exactly at 5PM.

During this adventure I had other fun things to think about. Sean and danny dealt with problems with an index on a database somewhere that was causing one of our websites to be slow. Word of advise to people I work with. When I am fiddling with one phone, on another phone and talking to someone on a nextel and you know a system is down, don’t come and tell me about another problem that you need me to work on. I can only do 4-5 things at once. Thankfully Sean was able to get a handle on the index problem and fix it with little to no help from me.

Danny was helpful in my network trouble shooting saga today. Everyone else was surprisingly not bothersome. usually when I have a major problem people come out of the woodwork to bug me about minor issues, or that is how it feels.

I was crazed today. Am I glad it is over? Of course I am, but there is something to be said about days like this. First, it goes by so quick. Second, it is the type of day that you earn your salary. You get a few of the each year, and when you live it you hate it, but after you live through those days you are a better person for surviving them. On a personal note I am glad I got through the issue mostly by myself. Danny did give me great assistance in the network trapping, but allot of it was second opinion from what we got from Cisco. It is good having someone else around who you can sound off ideas to. I think Danny and I work great in that respect. On a whole I had to tackle most of this issue by myself. In the past that is not a big deal. Recently I have been delegating allot of the day to day technical responsibilities. It is something that I have to do, but I feel like I get rusty by not doing hands on work all the time. Days like today keep me sharp and lets me prove to myself that I am still in the game and can get dirty with the best of them, or so I think.

More VOIP Updates

by

Several things finally came together on my VOIP odyssey this weekend. For one I solved a problem I have been having for some time. I also got off my ass and plugged in my new Linksys VPN Router. I started to setup static VPN tunnels to work. For one thing I proved I can setup a static IPSEC tunnel to a Pix with a cheap ($150 or cheaper) router. At first I still could not connect to one of our offices. I had the same problem with the Pix 501 I had. Or at first I thought it was the same problem. With the 501 I couldn’t connect to that office either. I was beginning to think it was not me but the Pix at that office. Turns out when I had my Pix 501 I had 1 issue. With the new Linksys VPN router the problem was a new one. Turns out the fail-over Pix we have at the office uses the IP address of 10.1.1.1. That is all nice and good, but that was what I used for my router at home. That didn’t work well. I had to give my router a new address and recreate the VPN tunnel. Everything started working then.

Now I have 2 tunnels (if and when I want them) to work. I don’t need VPN software. This is a good test since now I do not need VPN software on a computer in order to establish a VPN connection. What that means is I can plug in a VOIP phone and connect to my office’s call manager. I did just that. Now I have a 7940 phone connecting into our CCM (cisco call manager) system sitting on my desk at home. This is different to what I was trying to do with the 7940 and my BroadVoice connection. Now I can log into the phone at home and get my extension from work. I can prove that we can do this with remote agents if we choose to do so. We probably won’t but it is a nice technological feat to say we can. We want to go with the cheaper soft-phone option for remote agents, but having the ability to put a phone in someone’s house is nice to know you can. Have I mentioned that I have a cool job, or that VOIP is awesome???

In addition to the work phone I now have setup. I finally got my VOIP ATA adapter from the lovely folks at Broadvoice on Monday. I plugged it in after calling them and changing my account back to use that adapter. In minutes the adapter registered and I was off and running. I plugged in my new cordless phone and I had phone service. Did I mention it is only $10 a month? Now I was an early adopter of not having a home phone. I still use my cell as my main phone, but it is nice to not have to worry about the battery dying on a long call. Truth be told I want the VOIP service for the simultaneous ring option that I have. I am able to have calls ring on my home phone, my cell, blackberry, and work phone at the same time. I have a few kinks I am working out but otherwise that has been very cool.

Some problems I have to work out. I am back to using my old 802.11b wireless point. My 802.11g point was built into my old router. Now that I have a VPN router I can’t use that router. I need to buy a stand alone “g” point. I also need to break out a 5 port switch and daisy chain it off of my router. With the Wifi point, both my personal and work phones, plus my desktop and laptop, plus a tivo I have ran out of network ports on my 4 port router. Thankfully I think I have enough components to build 2 computers and extra network gear in my closet. He I don’t throw out very much stuff.

Ok, time for bed…

System Monitoring

by

I spent most of the day yesterday trying to fix a problem we had with Whatsup Gold monitoring system. One developer group said they were not getting pages. We saw tons of pages for them, but we looked into it. Well turns out the system was working, but had trouble with reboots to the server we had it on. After a reboot the monitoring system didn’t start up properly, even after we initialized it. While I was looking at the system it completely stopped sending pages. I got fed up and moved it to a different box. now it is all happy and working. Not sure why it wasn’t working properly. Maybe it was Backup Exec running on the same box, or Windows 2003 (even though we have used it on 2k3 before). I don’t know. I just know Gus wanted it working, and now it is.

While I was investigating my problems I checked (for like the 10th time) to see if Microsoft posted Microsoft Operations Manager 2005 (MOM) to MSDN. It finally had, and I downloaded it. I really want to see if it will solve my other issues I am having with Whatsup Gold. We have the guys over at BMC also giving us a demo of their software, but I haven’t had time to run a test server of their BMC Patrol. I will look over MOM in a bit more detail next week.

The Backlog

by

Since I am feeling fine today, and I am not in DC (like I planned on) I decided to make a dent in the backlog of work I have piling up. Right now I am taking a break (while my printer is working for me) from going over resume’s. I am still looking for a helpdesk person in NYC. It is a grueling process to go over hundreds of resume’s. I just printed out like 40 for someone else to look at before we call some of these people up.

I am also trying to do some updates to my companies domains. We own way too many of them. The amount of changes I have gotten are piling up. I need someone to do some data entry and make all the changes. It will take days. For now I am doing the most urgent stuff when I have time (like now).

I just finished cleaning out my inbox. I had a few hundred emails to go through and sort. I also went back and cleared out a ton of automated messages I get every day. I am a slave to mail…

Server Crash

by

Yesterday Jay and I had to deal with a domain controller crash. it is never good when a box crash’s, but it is even worse when it doesn’t come back after a crash. A rather important domain controller (they all are important I guess) totally died yesterday. We ended up spending 3 hours at our lovely (and very cold) data center bringing online a replacement. The good news is we sort of know what we are doing, so we had alternate DC’s online, but we still acted quick to get a replacement server up. I never like running with less DC’s (Windows Domain Controller) than normally. If you loose all of them, no one can log into the servers.

Due to that yesterday was kind of stressful. To users, nothing really was down, but we were busy. Along with the DC being gone we have to re-setup 2 applications that will take a while to get exactly like we had setup before. I just love having more work.

Today was taken up by meetings. I maybe had 1 1/2 hours of no meetings today. Have I mentioned I don’t like meetings? These meetings were actually productive, but I still don’t like meetings.

I should have gone home right away after I heard the Subway flooded. It rained allot last night. I waited while at least 3 subway trains went by totally full. I was dripping with sweat from the heat in the station. By the 4th or so train I just pushed into one, like everyone else. I got luck and room opened up further into the train and I was not too bad off. that is why people do push into trains people. Because it is crowded by the door and opens up a bit further inside the train. Lesson is move into the middle of the train. I ended up even getting a seat halfway through my trip. The down side was it took me almost 3 times as long to get to work as it normally does. That should have been the sign to go home and get back into bed! Speaking of bed. It is time for me to turn in. I got little sleep last night, and I have a poker game tomorrow night that may go late. Sleep is a premium recently!

Taking The Day Off

by

I am taking the day off tomorrow. I have no plans, other than taking the day off. I may just lie around. I may go and do something productive. What I won’t be doing is going to work, or answering my cell phone for work.

I wonder how long this plan will stay in effect?

The Picnic

by

I had a good time at the company picnic on Sunday. We had it in upstate NY. The company rented a bus and brought the NYC people up there. We had a good time. The place was very nice. They had a lake, a pool and lots of open land.

Our NYC softball team played the Kingston people in a game. It was fun. I got really sore from running. I didn’t even run that much, but I am out of shape. I didn’t realize how out of shape until I ran that little and could barely walk afterward.

It was a nice day. it was not too hot out. I got to jump in the bouncy gym by myself. I didn’t get egg on me from the egg toss. And I ate allot of BBQ hamburgers and hot dogs.

I got a lift home with Glenn instead of taking the bus back to the office. Got dropped off a block from my place. Not bad.

The Fish

by

A few weeks ago Jayson showed up to work with a small fish tank. He proceeded to buy a yellow clown fish and put it in the tank. he said it was to liven the office up. Fast forward 2-3 weeks and we now have a 10 gallon tank sitting on the dresser behind my desk. We have a shrimp we call Ralph. We have Krusty the clown fish. our never named green fish didn’t make it, but we don’t speak of him.

We are trying to get a lobster, but Petco keeps being out of stock on them. Jay is calling Gretchen’s cousin who can ship us a purple lobster. We are still learning the perils of a salt water tank. It is the talk of the office. People keep coming over to look at the fish. it is really funny. Gus wants to get a tray of butter and put it next to the tank to show the shrimp his fate if he doesn’t behave.

AC Issues

by

Air conditioning issues seem to follow me. My AC in my bedroom has been flaking for several weeks. It works, but barely. Then it makes loud noises, and just run’s loud. That was until last week. I called for the second time to have it serviced. Now it seems to keep the room nice and cold. I am happy.

Now all I need to do is get my living room unit to get the place a bit cooler. That may be asking too much.

At least work is cool, and comfy now. It hasn’t been really hot recently but the AC is at least working. 3 years of hardship may actually be over, but I am not an optimist most of the time!!!

Went To Kingston Again Today

by

Gus, Dan, and I took a road trip to kingston today to see a demo of some call monitoring software. it was really cool software. It can record calls our reps are on (we can do that with this package already), view screen shots of what the rep does on the call, and QA how they do and give them a rating. Nifty stuff.

I learned that Gus drives crazier than Dan. I also learned that when the NYC AC is working it is more comfortable in NYC than in Kingston. I survived the drive so I am happy. I am back home now. I was on the way out the door to meet up with people from work for a drink but I got a call stating that our dam reports website was not updating. I don’t know if anyone will be around by the time I get downtown.