Cisco

A Single Minded Day

by

Some days I cannot remember what it was that I did at work. Today I only did one thing almost the entire day. I cannot forget what I did if I tried. I worked on fixing our Call Manager all day. Call Manager is the Voice Over IP version of the PBX. It is basically the computers that control our phone system. Last night Kai and Jayson upgraded our Cisco Call Manager’s to version 3.3.3 from 3.3.2. When you read the version number change you (I) think it is a minor fix. Well the latest version (it is the latest version we are using, but there is a 3.3.4, and a 4.0 now) does some things differently. Kai and Jay got the upgrade done, and tested all the phones in our call center. Everything seemed to work. What they couldn’t test was our remote phones. The phones in our NYC office didn’t work. We didn’t know this until hours after they went off to sleep. They left word that we may need to upgrade the firmware on the phones, but that was it. Turns out the firmware upgrades automatically when you reboot the phone and their is a newer version of the firmware.

The problem was the phones could make outbound calls, but could not receive calls. when you called the phones in a remote office the phone would ring but if you picked it up the Call Manager didn’t recognize that the call started. this is not good if we want to our reps to take calls. So off to call Cisco I went. This all happened before or as I got to the office. What a way to start the day that I knew I was going to be short handed anyway. Kai and Jay were both off to sleep because of their overnight. So after a long and drawn out troubleshooting session with Cisco we figured out our problem. I actually spoke to 2 guys over there. the first guy gave me the right answer but could not explain why we had to do what he asked. Since it required me to change firewall rules and we upgraded a phone system not the firewall I was skeptical. So the second Cisco guy came into the picture. He walked me through the same troubleshooting process (a bit quicker than the first guy). he then made some phone calls and got back to me. Turns out the first cisco support guy was right we needed to change some rules on our Cisco Pix’s. Why? Well in the new version of our CCM (Cisco Call Manager) they changed how some protocol’s operate. So what worked in older versions of CCM didn’t work in the newer version. We had to remove to fix-up protocol lines on all of our Pix’s that are involved in the VPN that makes up our WAN. Sure enough Kai was right. Kai as in Kai the cisco rep I spoke to, not Kai the guy I work with. I made the two firewall rules and the phones started working. Elapsed time on the whole saga, 8 hours. I got the phones working exactly at 5PM.

During this adventure I had other fun things to think about. Sean and danny dealt with problems with an index on a database somewhere that was causing one of our websites to be slow. Word of advise to people I work with. When I am fiddling with one phone, on another phone and talking to someone on a nextel and you know a system is down, don’t come and tell me about another problem that you need me to work on. I can only do 4-5 things at once. Thankfully Sean was able to get a handle on the index problem and fix it with little to no help from me.

Danny was helpful in my network trouble shooting saga today. Everyone else was surprisingly not bothersome. usually when I have a major problem people come out of the woodwork to bug me about minor issues, or that is how it feels.

I was crazed today. Am I glad it is over? Of course I am, but there is something to be said about days like this. First, it goes by so quick. Second, it is the type of day that you earn your salary. You get a few of the each year, and when you live it you hate it, but after you live through those days you are a better person for surviving them. On a personal note I am glad I got through the issue mostly by myself. Danny did give me great assistance in the network trapping, but allot of it was second opinion from what we got from Cisco. It is good having someone else around who you can sound off ideas to. I think Danny and I work great in that respect. On a whole I had to tackle most of this issue by myself. In the past that is not a big deal. Recently I have been delegating allot of the day to day technical responsibilities. It is something that I have to do, but I feel like I get rusty by not doing hands on work all the time. Days like today keep me sharp and lets me prove to myself that I am still in the game and can get dirty with the best of them, or so I think.

More VOIP Updates

by

Several things finally came together on my VOIP odyssey this weekend. For one I solved a problem I have been having for some time. I also got off my ass and plugged in my new Linksys VPN Router. I started to setup static VPN tunnels to work. For one thing I proved I can setup a static IPSEC tunnel to a Pix with a cheap ($150 or cheaper) router. At first I still could not connect to one of our offices. I had the same problem with the Pix 501 I had. Or at first I thought it was the same problem. With the 501 I couldn’t connect to that office either. I was beginning to think it was not me but the Pix at that office. Turns out when I had my Pix 501 I had 1 issue. With the new Linksys VPN router the problem was a new one. Turns out the fail-over Pix we have at the office uses the IP address of 10.1.1.1. That is all nice and good, but that was what I used for my router at home. That didn’t work well. I had to give my router a new address and recreate the VPN tunnel. Everything started working then.

Now I have 2 tunnels (if and when I want them) to work. I don’t need VPN software. This is a good test since now I do not need VPN software on a computer in order to establish a VPN connection. What that means is I can plug in a VOIP phone and connect to my office’s call manager. I did just that. Now I have a 7940 phone connecting into our CCM (cisco call manager) system sitting on my desk at home. This is different to what I was trying to do with the 7940 and my BroadVoice connection. Now I can log into the phone at home and get my extension from work. I can prove that we can do this with remote agents if we choose to do so. We probably won’t but it is a nice technological feat to say we can. We want to go with the cheaper soft-phone option for remote agents, but having the ability to put a phone in someone’s house is nice to know you can. Have I mentioned that I have a cool job, or that VOIP is awesome???

In addition to the work phone I now have setup. I finally got my VOIP ATA adapter from the lovely folks at Broadvoice on Monday. I plugged it in after calling them and changing my account back to use that adapter. In minutes the adapter registered and I was off and running. I plugged in my new cordless phone and I had phone service. Did I mention it is only $10 a month? Now I was an early adopter of not having a home phone. I still use my cell as my main phone, but it is nice to not have to worry about the battery dying on a long call. Truth be told I want the VOIP service for the simultaneous ring option that I have. I am able to have calls ring on my home phone, my cell, blackberry, and work phone at the same time. I have a few kinks I am working out but otherwise that has been very cool.

Some problems I have to work out. I am back to using my old 802.11b wireless point. My 802.11g point was built into my old router. Now that I have a VPN router I can’t use that router. I need to buy a stand alone “g” point. I also need to break out a 5 port switch and daisy chain it off of my router. With the Wifi point, both my personal and work phones, plus my desktop and laptop, plus a tivo I have ran out of network ports on my 4 port router. Thankfully I think I have enough components to build 2 computers and extra network gear in my closet. He I don’t throw out very much stuff.

Ok, time for bed…

VOIP Update

by

I have given up with the Cisco 7940 phone. I went out and bought a Motorola 5.8GHZ cordless phone. I will plug it into the ATA adapter (the device that actually turns a VOIP call into something an analog phone can understand) that Broad Voice sends me on Monday.

On another VOIP note I still need to configure the linksys VPN router I got so I can see if I can establish a static tunnel to the office. If I can do that I can get a Cisco phone to work with my office extension.

Also got the approval to purchase the network gear for our ongoing build out of our call center. I have to get the paper work together and get the integrator to order the switch. Looks like we will go with a Cisco Catalyst 4000 series switch. I am also getting some more product details on some other projects we are looking at from our Cisco engineer on Monday.

My New VOIP (Home) Phone

by

I finally broke down and purchased another VOIP phone for my house. If you are someone who actually reads this regularly you will know last summer I got Vonage for about 2 weeks. it was too expensive for me to justify keeping it so I got rid of it. I am not home enough to warrant a home phone. I have been fine with my mobile.

Fast forward 14 months and I found a company that offers relatively cheap VOIP and offers all the cool features I wanted. Jayson found Broad Voice. They offer a $9.99 plan that gives you unlimited in state calls. That is all I needed so I got it. They also offer a function that AT&T Callvantage offers that allows me to have my phone right simultaneously on several phones. Now I can have my home phone ring at work, on my cell, etc at the same time. I have wanted one number to route to multiple lines for years. I used to use AT&T’s 500 number service back in the early 90’s but that was expensive and you paid per inbound calls. This is better. Whatever phone I pick up first is the phone that the call gets routed to.

What is also cool is that this company unlike allot of other VOIP providers allow you the option of bringing your own VOIP device or ATA adapter. What that means for me is if want I can configure a Cisco 7940 phone I have at work and use that with this service. I actually have been trying to do that today, but I am running into issues since our Cisco phones are configured for Skinny (to work with a Cisco Call Manager) and for this service I need SIP. No matter what Cisco says it is not simple to change the configuration. For now I am just routing all my calls to my mobile. I get 30 days to try it out, and it is only like $15 a month after tax’s and extra features. Not bad for unlimited local calls. I can use my mobile for long distance!

More reviews as I get it. Jayson and Gus are both waiting to see how I like this company, but they are both interested in the service.

Pix Configuration Problems

by

I am having a problem with the new Pix. I think it is a minor issue, but I just cant figure it out. I am trying to make a second static VPN tunnel, and the pix keeps telling me the crypto map is not complete. I am doing nothing different from the first tunnel I created but it still wont work. I am going to call Cisco and try to figure out what the deal is.

Mini Pix

by

After some consideration I went out and bought a Cisco Pix 501e. You may be wondering why in the world a person needs one of those. Need may not be the word, but I do have uses for it. In the past 24 hours I have learned more about IKE tunnels than I had in the past 12 months. Allot of what I want it for is work related, but I doubt they will pay for it so I got one.

I want to prove (i know it can be done, but me doing it and having someone say it can be done are two different things) that I can setup a static VPN tunnel securely and allow a VOIP phone to connect to our office. I want to be able to lock down the connection so only the voice traffic we want can go over the VPN. It will work since to my office pix, it is just connecting to another office with a pix. The 501 is exactly the same as the larger 515 but you can only do less. The config and the PDM is all the same.

It is very cool. The only problem I am having is that I cannot initiate a VPN client connection through my pix to another pix somewhere else. We have this problem at work, and I have not been able to find a fix for it. I need to call Cisco about it. It is a pain, but not a show stopper.

I showed jayson what we can do with the personal Pix and he wants one, but he just got a big screen tv so he is not spending anything for a while.

Allot To Write About

by

I have allot to write about, but for some reason don’t feel that up to writing today. So here will be a quick recap of the weekend.

I continued to throw stuff out of my apartment in anticipation of my move next month.

I am also getting ready to ebay a bunch of tech stuff I have. I realized that allot of my crap is not worth that much anymore used. So it is easier to just keep it and or give it away.

Had dinner with Carrie, and my grandmother tonight. Morgan also! My grandmother was in to visit my cousin who is sick. She took us to dinner. it was actually pleasant. That is not always the case with my grandmother.

Got a hair cut on Saturday. My regular barber wasn’t there. Not sure what his deal was. His father did my hair. Didn’t do a good job. it is ok, but he didn’t listen to me when I gave him instructions. That may be the last time I go to that place since I am moving. I need to find a new place in the city for next time.

We had an issue with one of our Call Manager Servers Saturday that caused it to fail. The second machine in the cluster took over fine, but we had about 15 seconds of down time while that happened. Kai is working on getting a patch for it. I spent allot of Saturday working on the phone system because of that, and other issues.

The plus side is I did some more work on our IVR. I even setup our own for Support, to prove the concept. An IVR for support is overkill but we had extra 800#’s and it shows that they system can work.

This blog entry was written while listening to Machine Head by Bush

Blackberry’s & Nextel

by

I finally approved getting the first of several new blackberry’s for our on call teams. I was holding off since the cost of the device was allot ($350). The current blackberries we have were only between $100 and $200 to purchase. Of course they are the smaller pager stye ones. Unfortunately no one seems to be selling them anymore. Everyone is trending towards the all in one pager, cell phone device. For us this isn’t the greatest. Almost all of us already have cell phones, so the pager/phone combo just adds weight to what we want to carry around.

The reason we are buying the more expensive units is simple. The current system we use with the pager style blackberry’s are charging a flat rate for up to 1mb of email. Then they charge you for every kb over that. We send alert pages to these devices and some months our overages can be $100 – $150 per device. Knowing that, by switching to Nextel’s blackberry server with a flat rate all you can receive plan we will over time save more money. I figure if we pay for $50 overages per pager per month, the $350 pager pays for itself after 7 months. We have had some of our older pagers for over 2 years and some of them are in great shape. That means in the long run we save cash.

Another added feature that made the decision easier was the fact that these new blackberry’s have Nextel direct connect. We get unlimited use of direct connect for free with the plan we have. It becomes useful when we deploy that to everyone who is on call. The goal is to improve our paging and on call system as best as possible while saving money. Man I am really starting to sound like a manager:(

We get our new blackberry today I hope. That would give me the weekend to play with it. Jayson already has one. We can test the direct connect. He loves the email functions on his. I have been using my Treo for a while, and I like it. But the blackberry seems to deliver mail faster, and it is easier to type on. I think my blackberry use may be making a comeback. Not sure yet!

This blog entry was written while listening to Sad Songs from the album “Beneath These Fireworks” by Matt Nathanson

Jayson At Work

by

Jayson will be in New York Monday night. He is going to do some contract work for my company for a few weeks while we search for a helpdesk person. I have a bunch of projects he is going to help me on. If all goes well we will improve our data center allot!!!

I am a bit nervous about him coming up. Not sure why. He is very professional and works fast. I am a bit rattled by Keith leaving, and Jay is another person I know coming to work. I vouched for him, so I am a bit nervous. I don’t think he will let me down, but I worry anyway. It will be interesting to work with him again. The last time we did was back in 1999 when we were in Baton Rouge.

It will be weird having one of my oldest friends working for me, even temporarily. On the plus side is we get along so well, and he is no nonsense like me. We should get allot done.

On the flip side I have a guy starting monday that will be our Release Manager. He will also work with our production systems providing support. He seems pretty sharp, so I am hoping for the best. Lots of change recently. Change is good, but it can be scary.

This blog entry was written while listening to Wherever You Will Go by The Calling

Extension Mobility

by

With Cisco Call Manager we can do something called extension mobility. it allows you to goto any phone in the office and log in with your phone extension. Our call center software Zeacom Smartconnect has always had problems with extension mobility. Once we use it the extension would not work with the software until the system was rebooted. We put a patch on the system last Tuesday that was supposed to fix the problem. Turns out that it didn’t. Some extensions work, and others do not. it is driving Kai and I crazy.

It turns out that Cisco uses the MAC address of the device you are logging into to track the profiles for extension mobility. That doesn’t work with how Zeacom does things. They like netbios. We have had issues with Smartconnect and netbios before when we wanted to put devices on networks other than the network where the smartconnect server resides. Zeacom is working on yet another patch to get the two system to gel and work together. I am still eagerly awaiting extension mobility. I have been waiting since September 2003. Granted EM will work with the cisco phones, but I just cannot use the call center software once I use it. it becomes a real pain in the ass.