Month: October 2005

The Evil Empire

by

I have so much I want to say, I don’t even know where to start or how much I will get out. Some people have their own impressions of “evil” corporate empires exist out there. I have friends that bash some retail store chains as being evil, or others who can’t stand some or all media empires. Most of those opinions are based on ideals, and some passing interaction with said companies. I on the other think the true “evil empire” out there is one that is partially transparent to people. I am talking about a company I will call the LEC that will not be named. If you don’t know a LEC is better know as “Local Exchange Carriers”. They are the SBC’s, Bell South’s, & Verizon’s of the world. You most likely deal with them with your home phone. The thing is everyday people don’t realize is that if you use an alternate provider for anything like phone service, or even business class T-1’s you still must use a LEC. The LEC owns and operates the physical lines in a given area. They own the copper and or fiber into buildings and houses. in some rare cases you have providers also having their own fiber or copper into an array, but that is uncommon.

So here is the situation most people don’t realize can happen. You have big name internet provider X as your data T-1 or voice T-1 service. They are a huge company but they do not own the local lines in the area you do business. That means between your office and provider X’s POP (Point of presence) you must utilize a LEC. If you don’t know better provider X won’t ever really bring that up, but I know this is the case. it is not that they hide that fact from you, but it is not something that will be highlighted on page one of a contract.

So this is the situation. You take time and effort to design a highly resilient internet backbone for an office that needs high availability on their voice and data setup. You spend the time and money to get multiple POP’s from provider X. You also get multiple routers setup with HSRP & BGP. All bases are covered, right? Wrong. The LEC who will not be named has a problem at their central office. All those nicely diversified circuits all go through the same LEC. Remember provider X will give you diversity, but they don’t own or control the lines into your office. If your area is serviced by 1 LEC, all your lines go through the same conduit out to the same CO (central office). Well if that CO has a problem with lets say some hardware, all your network diversified circuits are down. The LEC is the pinch point in most situations. Now if you have provider X, they will try to fix the issue. If they can’t figure it out, they will escalate the issue with the LEC who will not be named. Here is one problem. YOU or I are not a customer of the LEC who will not be named. Provider X is. You are considered a wholesale customer of the LEC who will not be named. To them you are the least important person. Now, they may say otherwise, but if you ever negotiated a T-1 between provider X and the LEC who will not be named, the LEC all but says you are not important if you use provider X. Is it true? I think so, but it may be a negotiation tactic. In any case when you have a problem, the LEC who will not be named seems to not care. Even if they ARE quick to respond they have the attitude of “we are big LEC who you have to use. We will get to you whenever we want”. I have had that feeling several times over the years. That is why I have come to the conclusion that the LEC who will not be named is the true evil empire.

So in a vague (or not so vague depending on if you know the true story) I have vented about my technical woes today. I know shit happens. When people ask why something is down (when telecom circuits and major network gear is down, not the minor stuff) I tell them I honestly don’t know how the stuff works the 99.999% of the time it does. Really. People think I am kidding, but if you ever had to spend 12+ hours trying to get the LEC who will not be named on the phone when you have an outage and when you finally do get them to do something it is fixed in 20 minutes you will begin to think like I am. I mean come on at least look like are trying to care about my problem!

And that reminds me of what Howard said to me when he stopped by my office today in the middle of the disaster that was my day. He was like, at least you are not having capacity problems like Sixaprt and their Typepad service was having. I had to laugh. He didn’t realize I use them. I think he thought I was using Gus’ server still. The funny thing was I gave them prop’s for how they were handling the situation. I sent an email to their CEO commenting on their outages and how they are communicating the issues to customers and I got a response in like 2 hours. How then can billion dollar LEC who won’t be named take 3 hours just to get one of multiple tickets into their system when calling the emergency support line? I am not talking about calling the “hi, my home phone isn’t working” number. I am talking about the “I spend allot of money and many circuits are down” number. It is really scary how a small company can be so responsive and a large one just plain old suck. Of course I should know that by now, since my company is not that big and we are SO much more responsive to issues than people we deal with. Yes small plug for my own tech group, but it is true.

Ok, by this point in my writing tonight I think I am just rambling. It has been a long day, and by now I am sobering up, but yes I did have a few drinks before I got home and started writing tonight. Hey it is Halloween and I went out for a few with friends from the office before coming home. I can’t remember if I have been over my opinion on the evil empire? If not I think I made myself clear tonight. Am I asking for too much? All I want is to get a person on the phone when I have a problem, and have them seem interested in solving my issue and get me back up and running. That person should also speak clear understandable english, and must understand when others speak clear and concise english. Forgot to mention that issue. Not sure what was worse, the FULLY automated ticketing system of one company today, or the get an offshore support person who cannot fully understand what you are saying number? Then there was the automated update system that called every 30 minutes with a message telling us nothing has changed. that would have been ok, if it wasn’t for the fact that we had 4 issues open at the same time, so Jayson had calls every few minutes. And in their attempt to be good about contacting people on alternate numbers if you didn’t answer your primary one, they would call Jay’s cell phone if he didn’t pick up his work one. The issue was he didn’t want to get more calls from auto response guy, but they kept calling.

Really I have come to the conclusion that I should not write about things that really bother me right after they happen. When I do, I write allot of stuff that is true, but when I am calmer I might not have written. For my own safety names of companies and details of issues have been deliberately modified in this post. The general issue is true, and yes I had a bad day today. I need to take a vacation day soon! On that note, I am going to stop writing. If I have more things to say about the LEC who shall not be named I will write later. On a semi positive note, provider X was not as bad as the LEC who shall not be named. They sucked allot, but at least their sales guy who I deal with allot was able to get some escalations in for me. That is saying something, in a day full of problems. Did it help? I don’t know, but it made me feel a bit better.

Backup of Files

by

I am trying to weekly rsync my files from the firewire drive on my Powerbook to the mini. I am using my Mac Mini and its Firewire drive as a backup set of my laptop. I have had trouble with the rsync choking on large initial copies. Not sure what is up with that. Some of the replications needed to be ran 3-5 times before they would complete. I have had better luck with the weekly updates. I guess with less file coping I get less errors. It kind of works for now, so I wont go crazy trying to debug it more.

I have a few more items to replicate but most of the important stuff is done.

Virtual vs Physical

by

At work I am deploying a growing number of servers for both production use and development & testing. One of the things we are doing is paying more attention to replicating our production environment at every step of the development and deployment environments. That means allot more servers. How much is allot? We added 25% more gear enterprise wide in the month of August. We have been busy, and we are not yet done. I have filled one computer room at a facility and we are bringing online another at that location sooner than originally planned. We also moved into a larger cage at our data center earlier this summer. Then we added cabinets to the new cage to accommodate the larger growth. In yet another office we added a cabinet, and are working on beefing up the power in the room to accommodate yet more gear. Then we need to look at more HVAC. I never had to deal with ancillary issues such as not having enough power to run gear. 5 years ago I would never have thought I would be in a situation like this. It is very interesting to me to look at how many rack U’s a server takes up when quoting them out and determining what to buy based on the cost of a 2 vs 4U server and how much it would cost to just add another cabinet if you got the bigger gear. As a plain old engineer I would just recommend buy this server because it did the job. In my current position I need to look at the entire picture.

The next few months will bring another burst of server sprawl. One of the things we are looking at is the cost of buying hardware for every server role we need to fill, or the cost to do the same amount of computing power in virtual machines. I must sound like a broken record talking about one of my favorite software companies, VMWare. Our GSX server has served us well, and is a great proof of concept to show how we can expand the use of VM’s. I don’t think we will deploy VM’s en-mass at our data center to do production work, but we have plenty of other uses for the technology elsewhere that makes looking into GSX or ESX server a viable alternative to buying more gear.

To me it boils down to 2 factors. 1 is of course cost. How much does it cost us to buy and deploy a dozen servers, power them, get KVM’s, and rack space for them, vs purchasing hardware for a VM server (or 2) that can handle the same amount of work load.

The 2nd factor is ease of use. How quickly can we get build a physical server for use? Restore it from a preset level of configuration for use in dev and qa? How fast can we buy and deploy hardware when a need comes up for a new server? The same questions apply for virtual machines.

I am a bit biased. I want to virtual machines. The flexibility they give you is amazing. I also know that I have SLA’s to keep, and costs to consider. So if the per server (or server instance) costs are too high we can’t do it. For now I spoke with my boss late this week to identify what applications need homes in what environments. The next step is to crunch the numbers to get all of our options. The VMware user groups have been helpful in figuring out realistically how many VM’s you can get per GSX and ESX server. More news as the project unfolds.

Lost Contact

by

I have been bad this summer. I have not kept in contact with some friends and family as much as I would like to. Don’t feel bad, my parents are mad at me for not seeing them enough either. Half of me says that contact is a two way street. if I have to call or email you first all the time, then you are not a good friend. The other half of me says, if I want to keep in touch with someone I should make an effort no matter what they do.

That being said, I have been very bad. I have had allot going on at work. It really does sap the time away. I am not proud of that fact, but it is the truth.

So what am I going to do about it? First I need to get out to Queens for John’s next Halo party. Then I need to see my folks. Goto my cousin Arielle’s birthday party later this month. Call my cousin synde and wayne, and then stay in touch for once in recent history. Email Jennifer and get her and steve to come visit again. There is more, but I know that will take me a while to do in the first place. I don’t want to make blanket statements and then never follow through with them.

The Security Myth

by

Security. I am a fan of it. Security is like a nice warm um well security blanket! No really. It is good, and most people take it for granted. The problem is allot of time security is this myth that people believe in that may not really exist. Take Wifi for example. I just used macstumbler while I am sitting at my desk at home. Do you know what I found? 8 wireless networks. One of them was mine. Of the other 7, I saw 4 open networks. Of those 2 had the default network names, and one was just named my network. That means that 50% of the networks around me where not just open for anyone to go into. That is crazy. I bet the people using those open networks don’t know they have a huge security hole on their network, or they don’t care. The network device manufacturer’s have a big problem. Make the setup of the devices too hard and people won’t buy them. Make them too easy (as they are most of the time now) and you have tons of unsecured networks. Having the majority of the people using this gear not know the mechanics of how the gear works does not help the situation. It is like having everyday people work on their cars instead of taking them to mechanic’s.

I don’t think most computer people will argue with the assessment I have made above. Or they can if they want. Wifi security has been discussed to death. Even with proper WEP or WPA encryption the system is still not safe. I know that. I have WPA setup on my wifi point. I know I can also add MAC address filtering, etc. I know better, but I still think I have secured the system enough. Have I really? I think for the most part yes. I think of WPA as the club. you can still steal the car (aka break into my network) but why would you waste time with my network or car when you can steal the guy down the streets car who left the door unlocked or just doesn’t have a club? I have a myth of security.

Another example of gaping security wholes is another growing wireless standard, Bluetooth. I have been a fan of it since I first read about it almost a year before the first mobile phone with bluetooth came out. And when it did, I bought one. A Ericsson (they were just Ericsson back then) r520. So for the record I am a fan of Bluetooth. I am a fan of wifi for that matter. I remember when I was at my first tech job back in 96 I got to play with a demo of a 1mbit (i think) wireless card and point from Raytheon. The problem is bluetooth has the same security myth. It also has the problem of the media blowing the issues into this huge security crisis. The simple fact is that most phones and other bluetooth devices were configured to be as easy to configure as the manufacturer could make them. That means allot of devices are setup to be discoverable by default. That means that if the bluetooth radio on a phone is on, someone else looking for bluetooth devices can see your phone if you are in range. To prove that, last week on Amtrak home from my trip I was able to view up to 4 other bluetooth devices from my seat. To protect yourself all you usually have to do is make a change in the default configuration of your device to not be “discoverable”. Do most people do this? Nope. But if you turn discovery off by default you have people complain that setting up partnerships are too hard. See the problem?

You have people then go around thinking all is ok, until they have a problem or someone tells them their phone is at risk of being broken into. First of all that may or may not be true given that you have to set passkeys, etc. For argument sake lets say it is an accurate assessment. These people then freak out and get mad at hardware vendors for delivering unsecured devices. How do you win?

Most of the time people live in the dream world that their stuff is safe. The crazy thing is that maybe 99% (or the vast majority) of the time people’s fantasy worlds are not broken. That perpetuates the myth that all is safe. Even if someone has been using their unsecured wireless internet connect for free for months.

The more I think about it, the more security myths I think about. And I am only thinking in terms of personal computer security. Don’t get me started on other society security concerns.

A perfect example is a few years ago my mom called me after she saw an Oprah on TV. She was calling to warn me that email I send wasn’t secure and that anyone can intercept and read it. She was shocked, but Oprah set her straight. I was like, yeah mom of course email is not secure. Old news. She was surprised that I knew that. It is scary that the general population assumes something like email is secure, and it isn’t. On the flip side can email be intercepted? Of course if it is not encrypted. Is most mail not encrypted? Yes. Will my mom have to worry about her neighbor reading her email or some stranger intercepting it? Probably not. It is very possible to do, but come on who really is going to try and sniff out her mail? its a real threat, but I don’t think most people won’t ever have to worry about it. Doesn’t mean I don’t think we should all get certificates and secure our mail. I would love to do that, but it is impractical in today’s world. So you see even I let the myth of my stuff is secure live on some level. We all do it, and if you don’t think you do, you are kidding yourself.

More VMWare Work

by

This week I actually am working on a technical project, not just managing them. I built a test environment to put up a windows 2003 terminal server. I used VMware’s GSX server to do it. I was testing what ports I would need to open up if I wanted to access a terminal server via the TSWeb client Microsoft gives you. Turns out even if the web server and the terminal server are the same machine you have to open up the RDP port on your firewall anyway. That wasn’t the answer I wanted to hear, so I am looking into Citrix to see if I can use their product and only open up port 80? Waiting to hear back from them.

Technorati Tags: , , ,

Budget Surplus?

by

OK, I am a bit confused and pissed off. I read now that the MTA has a budget surplus of 1 billion? Thats US dollars people. Aren’t these the same people that said they were going to have a 400 million plus deficit and they needed to raise fares? This was like a year ago. What is going on? Do they truly have no f–king clue?

I have two opinions on this. 1, if they have such a surplus they could have tried to predict this and well maybe not raised the rates. or not raised them so much? Opinion 2, I understand raising rates (to a point) to not be in debt. If you then later on find yourself having a surplus you should (maybe) use the money wisely, instead of cutting prices for a month or so. How about more trains? Speed up renovation of existing stuff, or expand the system? Did I mention more trains? Or how about putting the money away and save it for the next deficit? I would bet you get some nice interest on a billion dollars.

I am so f–king confused. These are the same idiots that say they had to cut back night and weekend service and they now have a ton of cash. I know I am repeating myself. I am just baffled…

And for some of the riders who commented on the article I link to. Are you FREAKIN crazy. You want a quick $1 back on your ride instead of ensuring the dam system runs in 1-10 years? So short sited. And who was the moron who said “there’s already tunnels to brooklyn so they don’t need more”. Call me crazy, but If it takes you an hour to get through the tunnel durring rush hour, maybe a new one would be a good idea?

There are some real rocket scientists living in this city.

More VMWare Work

by

This week I actually am working on a technical project, not just managing them. I built a test environment to put up a windows 2003 terminal server. I used VMware’s GSX server to do it. I was testing what ports i would need to open up if i wanted to access a terminal server via the TSWeb client Microsoft gives you. Turns out even if the web server and the terminal server are the same machine you have to open up the RDP port on your firewall anyway. That wasn’t the answer i wanted to hear, so i am looking into Citrix to see if i can use their product and only open up port 80? Waiting to hear back from them.

, , ,

Another Monday

by

I am running out of catchy titles for posts on some days. Who am I kidding. I ran out of good post names for most posts back in 2003! I had as typical a day as I could. I had a few meetings. Went over issues with Kai, and had to deal with some last minute requests. Nothing major to report, or I should rephrase that. nothing major that I can report here! Read into that what you will! And yes I like being cryptic sometimes. It’s more than I have to be, I actually enjoy it.

After work I went to Best Buy with Jayson to fix the shipping issues with my TV. After about 40 minutes of working with customer service they think they fixed the problem. It took forever to fix. it also took like 3 people to figure out the issue, but they did figure it out. they were also very very nice and attentive about the whole issue. It may have taken a long time, but it could have been MUCH worse. Kudos to the ladies that assisted me.

Treo Refresh

by

Last night I ended up wiping my Treo 650 and reinstalling all my software. I have been having issues with it rebooting every time I tried to sync my mail using Snappermail. That program is half the reason I use my Treo over the Blackberry, so I was not too happy when it started acting up earlier this week. I had to go on my trip Thursday without being able to use the Treo for email. I got by on the Razr and the Blackberry but I like the Treo better. After the reinstall Snappermail seems to be working. But I have had 2 random reboots since the hard reset. Once after a call to my sister right after I got out of the subway this morning. The other when I was syncing with my Powerbook. The sync reboot also happened often before I wiped the Treo, but I am not sure if it is because of the config on my Powerbook or the Treo.

I will try using for a while longer, but if the random reboots continue I will have to seriously consider an alternate PDA phone. As much as I like this device, I cannot have it randomly rebooting on me almost daily!