Networking

Wiring Day

by

Today Jayson and I spent most of the day rewiring a cabinet in the office. Try as we might these cabinets get super messy with cables everywhere after a while. We rewired this same cabinet over a year ago but it looked horrible again. The issue is that we keep putting in and taking out different kinds of gear. The cables get really messy.

To keep things cleaner we moved the PDU’s, rewired all the electrical, moved some servers to different racks, and took out the thick analog KVM cables and replaced them with IP KVM’s that run over regular CAD5. The result is a much cleaner rack. That and everything is neat and tied down helps. We next need to get the electrical guys to come in and add some more circuits.

We still have to clean up the mess of cables and old servers we pulled from the cabinet, but we can do that during the week.

The work was messy, and we came in on a Saturday but the results where worth it!

The Evil Empire

by

I have so much I want to say, I don’t even know where to start or how much I will get out. Some people have their own impressions of “evil” corporate empires exist out there. I have friends that bash some retail store chains as being evil, or others who can’t stand some or all media empires. Most of those opinions are based on ideals, and some passing interaction with said companies. I on the other think the true “evil empire” out there is one that is partially transparent to people. I am talking about a company I will call the LEC that will not be named. If you don’t know a LEC is better know as “Local Exchange Carriers”. They are the SBC’s, Bell South’s, & Verizon’s of the world. You most likely deal with them with your home phone. The thing is everyday people don’t realize is that if you use an alternate provider for anything like phone service, or even business class T-1’s you still must use a LEC. The LEC owns and operates the physical lines in a given area. They own the copper and or fiber into buildings and houses. in some rare cases you have providers also having their own fiber or copper into an array, but that is uncommon.

So here is the situation most people don’t realize can happen. You have big name internet provider X as your data T-1 or voice T-1 service. They are a huge company but they do not own the local lines in the area you do business. That means between your office and provider X’s POP (Point of presence) you must utilize a LEC. If you don’t know better provider X won’t ever really bring that up, but I know this is the case. it is not that they hide that fact from you, but it is not something that will be highlighted on page one of a contract.

So this is the situation. You take time and effort to design a highly resilient internet backbone for an office that needs high availability on their voice and data setup. You spend the time and money to get multiple POP’s from provider X. You also get multiple routers setup with HSRP & BGP. All bases are covered, right? Wrong. The LEC who will not be named has a problem at their central office. All those nicely diversified circuits all go through the same LEC. Remember provider X will give you diversity, but they don’t own or control the lines into your office. If your area is serviced by 1 LEC, all your lines go through the same conduit out to the same CO (central office). Well if that CO has a problem with lets say some hardware, all your network diversified circuits are down. The LEC is the pinch point in most situations. Now if you have provider X, they will try to fix the issue. If they can’t figure it out, they will escalate the issue with the LEC who will not be named. Here is one problem. YOU or I are not a customer of the LEC who will not be named. Provider X is. You are considered a wholesale customer of the LEC who will not be named. To them you are the least important person. Now, they may say otherwise, but if you ever negotiated a T-1 between provider X and the LEC who will not be named, the LEC all but says you are not important if you use provider X. Is it true? I think so, but it may be a negotiation tactic. In any case when you have a problem, the LEC who will not be named seems to not care. Even if they ARE quick to respond they have the attitude of “we are big LEC who you have to use. We will get to you whenever we want”. I have had that feeling several times over the years. That is why I have come to the conclusion that the LEC who will not be named is the true evil empire.

So in a vague (or not so vague depending on if you know the true story) I have vented about my technical woes today. I know shit happens. When people ask why something is down (when telecom circuits and major network gear is down, not the minor stuff) I tell them I honestly don’t know how the stuff works the 99.999% of the time it does. Really. People think I am kidding, but if you ever had to spend 12+ hours trying to get the LEC who will not be named on the phone when you have an outage and when you finally do get them to do something it is fixed in 20 minutes you will begin to think like I am. I mean come on at least look like are trying to care about my problem!

And that reminds me of what Howard said to me when he stopped by my office today in the middle of the disaster that was my day. He was like, at least you are not having capacity problems like Sixaprt and their Typepad service was having. I had to laugh. He didn’t realize I use them. I think he thought I was using Gus’ server still. The funny thing was I gave them prop’s for how they were handling the situation. I sent an email to their CEO commenting on their outages and how they are communicating the issues to customers and I got a response in like 2 hours. How then can billion dollar LEC who won’t be named take 3 hours just to get one of multiple tickets into their system when calling the emergency support line? I am not talking about calling the “hi, my home phone isn’t working” number. I am talking about the “I spend allot of money and many circuits are down” number. It is really scary how a small company can be so responsive and a large one just plain old suck. Of course I should know that by now, since my company is not that big and we are SO much more responsive to issues than people we deal with. Yes small plug for my own tech group, but it is true.

Ok, by this point in my writing tonight I think I am just rambling. It has been a long day, and by now I am sobering up, but yes I did have a few drinks before I got home and started writing tonight. Hey it is Halloween and I went out for a few with friends from the office before coming home. I can’t remember if I have been over my opinion on the evil empire? If not I think I made myself clear tonight. Am I asking for too much? All I want is to get a person on the phone when I have a problem, and have them seem interested in solving my issue and get me back up and running. That person should also speak clear understandable english, and must understand when others speak clear and concise english. Forgot to mention that issue. Not sure what was worse, the FULLY automated ticketing system of one company today, or the get an offshore support person who cannot fully understand what you are saying number? Then there was the automated update system that called every 30 minutes with a message telling us nothing has changed. that would have been ok, if it wasn’t for the fact that we had 4 issues open at the same time, so Jayson had calls every few minutes. And in their attempt to be good about contacting people on alternate numbers if you didn’t answer your primary one, they would call Jay’s cell phone if he didn’t pick up his work one. The issue was he didn’t want to get more calls from auto response guy, but they kept calling.

Really I have come to the conclusion that I should not write about things that really bother me right after they happen. When I do, I write allot of stuff that is true, but when I am calmer I might not have written. For my own safety names of companies and details of issues have been deliberately modified in this post. The general issue is true, and yes I had a bad day today. I need to take a vacation day soon! On that note, I am going to stop writing. If I have more things to say about the LEC who shall not be named I will write later. On a semi positive note, provider X was not as bad as the LEC who shall not be named. They sucked allot, but at least their sales guy who I deal with allot was able to get some escalations in for me. That is saying something, in a day full of problems. Did it help? I don’t know, but it made me feel a bit better.

The Security Myth

by

Security. I am a fan of it. Security is like a nice warm um well security blanket! No really. It is good, and most people take it for granted. The problem is allot of time security is this myth that people believe in that may not really exist. Take Wifi for example. I just used macstumbler while I am sitting at my desk at home. Do you know what I found? 8 wireless networks. One of them was mine. Of the other 7, I saw 4 open networks. Of those 2 had the default network names, and one was just named my network. That means that 50% of the networks around me where not just open for anyone to go into. That is crazy. I bet the people using those open networks don’t know they have a huge security hole on their network, or they don’t care. The network device manufacturer’s have a big problem. Make the setup of the devices too hard and people won’t buy them. Make them too easy (as they are most of the time now) and you have tons of unsecured networks. Having the majority of the people using this gear not know the mechanics of how the gear works does not help the situation. It is like having everyday people work on their cars instead of taking them to mechanic’s.

I don’t think most computer people will argue with the assessment I have made above. Or they can if they want. Wifi security has been discussed to death. Even with proper WEP or WPA encryption the system is still not safe. I know that. I have WPA setup on my wifi point. I know I can also add MAC address filtering, etc. I know better, but I still think I have secured the system enough. Have I really? I think for the most part yes. I think of WPA as the club. you can still steal the car (aka break into my network) but why would you waste time with my network or car when you can steal the guy down the streets car who left the door unlocked or just doesn’t have a club? I have a myth of security.

Another example of gaping security wholes is another growing wireless standard, Bluetooth. I have been a fan of it since I first read about it almost a year before the first mobile phone with bluetooth came out. And when it did, I bought one. A Ericsson (they were just Ericsson back then) r520. So for the record I am a fan of Bluetooth. I am a fan of wifi for that matter. I remember when I was at my first tech job back in 96 I got to play with a demo of a 1mbit (i think) wireless card and point from Raytheon. The problem is bluetooth has the same security myth. It also has the problem of the media blowing the issues into this huge security crisis. The simple fact is that most phones and other bluetooth devices were configured to be as easy to configure as the manufacturer could make them. That means allot of devices are setup to be discoverable by default. That means that if the bluetooth radio on a phone is on, someone else looking for bluetooth devices can see your phone if you are in range. To prove that, last week on Amtrak home from my trip I was able to view up to 4 other bluetooth devices from my seat. To protect yourself all you usually have to do is make a change in the default configuration of your device to not be “discoverable”. Do most people do this? Nope. But if you turn discovery off by default you have people complain that setting up partnerships are too hard. See the problem?

You have people then go around thinking all is ok, until they have a problem or someone tells them their phone is at risk of being broken into. First of all that may or may not be true given that you have to set passkeys, etc. For argument sake lets say it is an accurate assessment. These people then freak out and get mad at hardware vendors for delivering unsecured devices. How do you win?

Most of the time people live in the dream world that their stuff is safe. The crazy thing is that maybe 99% (or the vast majority) of the time people’s fantasy worlds are not broken. That perpetuates the myth that all is safe. Even if someone has been using their unsecured wireless internet connect for free for months.

The more I think about it, the more security myths I think about. And I am only thinking in terms of personal computer security. Don’t get me started on other society security concerns.

A perfect example is a few years ago my mom called me after she saw an Oprah on TV. She was calling to warn me that email I send wasn’t secure and that anyone can intercept and read it. She was shocked, but Oprah set her straight. I was like, yeah mom of course email is not secure. Old news. She was surprised that I knew that. It is scary that the general population assumes something like email is secure, and it isn’t. On the flip side can email be intercepted? Of course if it is not encrypted. Is most mail not encrypted? Yes. Will my mom have to worry about her neighbor reading her email or some stranger intercepting it? Probably not. It is very possible to do, but come on who really is going to try and sniff out her mail? its a real threat, but I don’t think most people won’t ever have to worry about it. Doesn’t mean I don’t think we should all get certificates and secure our mail. I would love to do that, but it is impractical in today’s world. So you see even I let the myth of my stuff is secure live on some level. We all do it, and if you don’t think you do, you are kidding yourself.

Roller Coaster Day

by

When I didn’t think the day could get any crazier it did. I was very busy all day putting out fires at work (figuratively not really). Just as I recovered from a week delay on our AD project, I get major complications from our IPCC deployment. Dan and I had a opinionated discussion with our integrators. We also are very close to purchasing our first SAN. We are pretty sure we got the price to a number we are happy with.

Finalized plans to make changes to our voice circuits to allow for more resiliency if we have a failure. Stupid issues arose regarding mislabeling of circuits. The little details that we are not following up on are killing me, but when asked to get tons done in a certain amount of time, you have to give somewhere. I just don’t like doing it.

I have other issues going on with VPN tunnels to partners of ours. It has been very stressful. I thrive on stress, but sometimes it is allot. Hopefully tomorrow will be calmer.

Remote Problems

by

Tuesday I had to deal with problems while I was 2000+ miles and 3 time zones away. First we are having lingering issues with a mail server. No one can log into it locally but the email services are running. so for now we are in a wait and see state with it. we are making sure we have good backups of everything on the box before we mess with it more. Also Kai and I being away isn’t helping.

A reporting function of Zeacom failed again. The only solution is to reboot the system. This is what the brain trust at Zeacom support tells us. Rebooting that system is not without its own risks. Kai and I are the only ones who have rebooted it before. Brian will do it tonight, but it will be his first time. He has seen Kai do it so hopefully we have no issues.

We also had performance issues with Jabber and some other minor crap. I had to juggle this, Dan complaining of site performance issues and 4 lectures. it was a busy day. The problem with the site performance issues is we see nothing wrong on our end. Dan is just frustrated. Not sure what else we can do (from my end) right now. it is not like the local director is acting up again (thank goodness), that I can make a change and magically it is all better. that is what he is hoping for.

MailHop Forward

by

Now that I am once again messing with my own mail server I need some sort of backup in case my server at home goes down. I used to use DynDNS’s MailHop Backup MX service. It was great. If I lost my server, they would act as my backup mail server and redirect my mail to me when my box came back online.

DynDNS now has a service that will allow me to create alias’ and direct mail from my domain to any mailbox. So my mail mail domain can redirect to any other domain and mailbox I want, or to my gmail account, etc. This is great if I have/want to change my mail setup quickly, or direct mail to different systems. The concept is probably overkill for what I need, but I wanted to give it a try. I needed some sort of mail backup anyway so it was worth the price I paid.

I have been a happy user of DynDNS’s Custom DNS service for almost 2 years. I am a huge fan, they have a good solid track record for me!

Cisco Network Conference

by

I got my reservations for the Cisco Network conference in Las Vegas this year. I made my reservations and called everyone I know that is going. I will hopefully meet up with Joe out there. He is taking Karen, so hopefully we can get together for dinner one night. I hope to also meet up with one of the Cisco engineers I know. I still have to set my schedule for the conference but with tons (200+) things to do I will have a hard time picking just a few stuff to do. I am really excited to go. I get out of the office that much and this really sounds like an interesting conference. That it is in Vegas has no bearing on my happiness on going!

I also spoke to Sean about the trip. I am hoping to hook up with him on the sunday I fly in. It has been a while since we got together. I hope he and Kelly have some free time with the baby and all. More news as it develops.

Static IP Bliss

by

I finally took the plunge. I finally forked over the extra money to my cable company for a static IP address. Granted I have had the same ip since I moved in, but my provider (RCN) blocks web and SMTP ports. That means I couldn’t host a website or a mail server. With my static ip (and RCN having $20 a month from me) I can (according to tech support) host stuff on my server now. I am kind of pissed that I have to pay since I was able to do this with Time Warner for free in the past. Granted I am not sure if TW actually allowed it but it was technically possible to do. I could have gotten a port redirector service to allow me to host my site with port 80 being blocked but RCN did offer me this pay for an IP option. I cut back on my cell phone since I haven’t been using it so the cost to me is a wash.

I have just configured my router for the static VPN tunnels that I have. Stupid changes were needed on the Pix’s on the other end to get it to work, but after about 10 minutes on each tunnel I was up and running again. With 2 configuration changes on my router I was able to get a demo page up on apache. Seems to be working now. All I have to do is take the virtual machine I am running my site on, back it up and move it home. Hopefully everything comes up ok. Once I move the site to my home machine I can begin to work on a linux only version of the site.

Once the site is moved I can really test and see if RCN’s claims are accurate about their ip service. More news on that later.

Switch Move

by

Jayson and I are off to Kingston Thursday. We are bringing up some gear and covering for chris and kai who will be doing a move of our Core switch to its new home on the first floor. We christen the 1st floor telco room Thursday night with moving the core downstairs. Besides acting as the core for the entire office it will act as the switch for all users on the first floor.

Hopefully nothing goes wrong since it is (dare I say just) a hardware move. Of course when you are dealing with large, complex, & expensive hardware nothing is simple.

Jay and I will stay overnight and cover the Friday day shift to also ensure that no problems ocure after the move. Lets hope all is well…

Faster Internet, Much Faster

by

We completed our upgrade to a brand new 4.5 meg internet link tonight in our office. We dropped our redundant T-1 connections in favor of the less reliable (arguably) single link with triple the amount of bandwidth. We (i) did it for like $400 cheaper than we were paying before using the same provider. Lets hope this alleviates some of the bandwidth issues we have seen in the office.

Some minor issues slowed us down, but in the end we were up and running in like 2 hours. Next up is the analysis on our Kingston office to see if they warrant an upgrade to 3 or 4.5 meg links. They require a more redundant connection but they have been expanding and their bandwidth may start becoming a problem. I think we can find the point in time where we believe the bandwidth will trend higher than we can support and get an upgrade done.