Technology

Best Way To Send A Large File?

by

Do you know the best way to send a large file? Lets say a gig or bigger for argument sake? The answer may surprise you. Someone once told me they heard a CEO of a big bandwidth company ask that question. The answer wasn’t some whizzing technology. It was sneakernet. Don’t get the reference? Take the file and walk it wherever you need it to go. Part of our operations tasks is couriering around drives with updated data between offices. It is cheaper to do that then send over the internet a 100 gigs of data. I sometimes find it funny we do it, but it makes sense. With many servers of the same hardware configuration we can also build servers and send the drives to their intended location and plug them into a different shell. Old idea, but it never ceases to amaze me how simple and effective it can be.

Crazy the things you think about when your mind isn’t preoccupied with something!

ESX Server?

by

I have praised VMWare’s products for some time on this blog. As things look now, I will continue to do so. I have used their workstation product for almost 2 years. I have used GSX server for about a year. Now comes ESX server. It looks awesome. I have a project that requires allot of new hardware to be built. ESX server looks like an good alternative to server sprawl. Instead of building dozens of servers, the same solution done virtually can be done with a few (not many) ESX servers. The cost benefit is real. I ran some figures over the weekend and it can save us a nice amount of money by going virtual. Then there are the saving space, and energy issues. And the flexibility we get by having a virtual setup. I am really excited about what we can do.

I have been trying out ESX on a test box we setup (reminds me I have a VM building right now that probably needs some click next help). The install was quick. I now have to read the dozen or so white papers spread across my desk (my computer desktop and physical desk). This software solves problems I didn’t even think about until I got to reading about the solutions. Multi vlan support is nice, as well as NIC teaming. I have allot of reading and testing to do in the short amount of time I have until we have to deploy the solution. I don’t think have been this excited about a technology since Call Manager. More info as I play (i mean test).

Technorati Tags: , , ,

The Evil Empire

by

I have so much I want to say, I don’t even know where to start or how much I will get out. Some people have their own impressions of “evil” corporate empires exist out there. I have friends that bash some retail store chains as being evil, or others who can’t stand some or all media empires. Most of those opinions are based on ideals, and some passing interaction with said companies. I on the other think the true “evil empire” out there is one that is partially transparent to people. I am talking about a company I will call the LEC that will not be named. If you don’t know a LEC is better know as “Local Exchange Carriers”. They are the SBC’s, Bell South’s, & Verizon’s of the world. You most likely deal with them with your home phone. The thing is everyday people don’t realize is that if you use an alternate provider for anything like phone service, or even business class T-1’s you still must use a LEC. The LEC owns and operates the physical lines in a given area. They own the copper and or fiber into buildings and houses. in some rare cases you have providers also having their own fiber or copper into an array, but that is uncommon.

So here is the situation most people don’t realize can happen. You have big name internet provider X as your data T-1 or voice T-1 service. They are a huge company but they do not own the local lines in the area you do business. That means between your office and provider X’s POP (Point of presence) you must utilize a LEC. If you don’t know better provider X won’t ever really bring that up, but I know this is the case. it is not that they hide that fact from you, but it is not something that will be highlighted on page one of a contract.

So this is the situation. You take time and effort to design a highly resilient internet backbone for an office that needs high availability on their voice and data setup. You spend the time and money to get multiple POP’s from provider X. You also get multiple routers setup with HSRP & BGP. All bases are covered, right? Wrong. The LEC who will not be named has a problem at their central office. All those nicely diversified circuits all go through the same LEC. Remember provider X will give you diversity, but they don’t own or control the lines into your office. If your area is serviced by 1 LEC, all your lines go through the same conduit out to the same CO (central office). Well if that CO has a problem with lets say some hardware, all your network diversified circuits are down. The LEC is the pinch point in most situations. Now if you have provider X, they will try to fix the issue. If they can’t figure it out, they will escalate the issue with the LEC who will not be named. Here is one problem. YOU or I are not a customer of the LEC who will not be named. Provider X is. You are considered a wholesale customer of the LEC who will not be named. To them you are the least important person. Now, they may say otherwise, but if you ever negotiated a T-1 between provider X and the LEC who will not be named, the LEC all but says you are not important if you use provider X. Is it true? I think so, but it may be a negotiation tactic. In any case when you have a problem, the LEC who will not be named seems to not care. Even if they ARE quick to respond they have the attitude of “we are big LEC who you have to use. We will get to you whenever we want”. I have had that feeling several times over the years. That is why I have come to the conclusion that the LEC who will not be named is the true evil empire.

So in a vague (or not so vague depending on if you know the true story) I have vented about my technical woes today. I know shit happens. When people ask why something is down (when telecom circuits and major network gear is down, not the minor stuff) I tell them I honestly don’t know how the stuff works the 99.999% of the time it does. Really. People think I am kidding, but if you ever had to spend 12+ hours trying to get the LEC who will not be named on the phone when you have an outage and when you finally do get them to do something it is fixed in 20 minutes you will begin to think like I am. I mean come on at least look like are trying to care about my problem!

And that reminds me of what Howard said to me when he stopped by my office today in the middle of the disaster that was my day. He was like, at least you are not having capacity problems like Sixaprt and their Typepad service was having. I had to laugh. He didn’t realize I use them. I think he thought I was using Gus’ server still. The funny thing was I gave them prop’s for how they were handling the situation. I sent an email to their CEO commenting on their outages and how they are communicating the issues to customers and I got a response in like 2 hours. How then can billion dollar LEC who won’t be named take 3 hours just to get one of multiple tickets into their system when calling the emergency support line? I am not talking about calling the “hi, my home phone isn’t working” number. I am talking about the “I spend allot of money and many circuits are down” number. It is really scary how a small company can be so responsive and a large one just plain old suck. Of course I should know that by now, since my company is not that big and we are SO much more responsive to issues than people we deal with. Yes small plug for my own tech group, but it is true.

Ok, by this point in my writing tonight I think I am just rambling. It has been a long day, and by now I am sobering up, but yes I did have a few drinks before I got home and started writing tonight. Hey it is Halloween and I went out for a few with friends from the office before coming home. I can’t remember if I have been over my opinion on the evil empire? If not I think I made myself clear tonight. Am I asking for too much? All I want is to get a person on the phone when I have a problem, and have them seem interested in solving my issue and get me back up and running. That person should also speak clear understandable english, and must understand when others speak clear and concise english. Forgot to mention that issue. Not sure what was worse, the FULLY automated ticketing system of one company today, or the get an offshore support person who cannot fully understand what you are saying number? Then there was the automated update system that called every 30 minutes with a message telling us nothing has changed. that would have been ok, if it wasn’t for the fact that we had 4 issues open at the same time, so Jayson had calls every few minutes. And in their attempt to be good about contacting people on alternate numbers if you didn’t answer your primary one, they would call Jay’s cell phone if he didn’t pick up his work one. The issue was he didn’t want to get more calls from auto response guy, but they kept calling.

Really I have come to the conclusion that I should not write about things that really bother me right after they happen. When I do, I write allot of stuff that is true, but when I am calmer I might not have written. For my own safety names of companies and details of issues have been deliberately modified in this post. The general issue is true, and yes I had a bad day today. I need to take a vacation day soon! On that note, I am going to stop writing. If I have more things to say about the LEC who shall not be named I will write later. On a semi positive note, provider X was not as bad as the LEC who shall not be named. They sucked allot, but at least their sales guy who I deal with allot was able to get some escalations in for me. That is saying something, in a day full of problems. Did it help? I don’t know, but it made me feel a bit better.

Backup of Files

by

I am trying to weekly rsync my files from the firewire drive on my Powerbook to the mini. I am using my Mac Mini and its Firewire drive as a backup set of my laptop. I have had trouble with the rsync choking on large initial copies. Not sure what is up with that. Some of the replications needed to be ran 3-5 times before they would complete. I have had better luck with the weekly updates. I guess with less file coping I get less errors. It kind of works for now, so I wont go crazy trying to debug it more.

I have a few more items to replicate but most of the important stuff is done.

Virtual vs Physical

by

At work I am deploying a growing number of servers for both production use and development & testing. One of the things we are doing is paying more attention to replicating our production environment at every step of the development and deployment environments. That means allot more servers. How much is allot? We added 25% more gear enterprise wide in the month of August. We have been busy, and we are not yet done. I have filled one computer room at a facility and we are bringing online another at that location sooner than originally planned. We also moved into a larger cage at our data center earlier this summer. Then we added cabinets to the new cage to accommodate the larger growth. In yet another office we added a cabinet, and are working on beefing up the power in the room to accommodate yet more gear. Then we need to look at more HVAC. I never had to deal with ancillary issues such as not having enough power to run gear. 5 years ago I would never have thought I would be in a situation like this. It is very interesting to me to look at how many rack U’s a server takes up when quoting them out and determining what to buy based on the cost of a 2 vs 4U server and how much it would cost to just add another cabinet if you got the bigger gear. As a plain old engineer I would just recommend buy this server because it did the job. In my current position I need to look at the entire picture.

The next few months will bring another burst of server sprawl. One of the things we are looking at is the cost of buying hardware for every server role we need to fill, or the cost to do the same amount of computing power in virtual machines. I must sound like a broken record talking about one of my favorite software companies, VMWare. Our GSX server has served us well, and is a great proof of concept to show how we can expand the use of VM’s. I don’t think we will deploy VM’s en-mass at our data center to do production work, but we have plenty of other uses for the technology elsewhere that makes looking into GSX or ESX server a viable alternative to buying more gear.

To me it boils down to 2 factors. 1 is of course cost. How much does it cost us to buy and deploy a dozen servers, power them, get KVM’s, and rack space for them, vs purchasing hardware for a VM server (or 2) that can handle the same amount of work load.

The 2nd factor is ease of use. How quickly can we get build a physical server for use? Restore it from a preset level of configuration for use in dev and qa? How fast can we buy and deploy hardware when a need comes up for a new server? The same questions apply for virtual machines.

I am a bit biased. I want to virtual machines. The flexibility they give you is amazing. I also know that I have SLA’s to keep, and costs to consider. So if the per server (or server instance) costs are too high we can’t do it. For now I spoke with my boss late this week to identify what applications need homes in what environments. The next step is to crunch the numbers to get all of our options. The VMware user groups have been helpful in figuring out realistically how many VM’s you can get per GSX and ESX server. More news as the project unfolds.

The Security Myth

by

Security. I am a fan of it. Security is like a nice warm um well security blanket! No really. It is good, and most people take it for granted. The problem is allot of time security is this myth that people believe in that may not really exist. Take Wifi for example. I just used macstumbler while I am sitting at my desk at home. Do you know what I found? 8 wireless networks. One of them was mine. Of the other 7, I saw 4 open networks. Of those 2 had the default network names, and one was just named my network. That means that 50% of the networks around me where not just open for anyone to go into. That is crazy. I bet the people using those open networks don’t know they have a huge security hole on their network, or they don’t care. The network device manufacturer’s have a big problem. Make the setup of the devices too hard and people won’t buy them. Make them too easy (as they are most of the time now) and you have tons of unsecured networks. Having the majority of the people using this gear not know the mechanics of how the gear works does not help the situation. It is like having everyday people work on their cars instead of taking them to mechanic’s.

I don’t think most computer people will argue with the assessment I have made above. Or they can if they want. Wifi security has been discussed to death. Even with proper WEP or WPA encryption the system is still not safe. I know that. I have WPA setup on my wifi point. I know I can also add MAC address filtering, etc. I know better, but I still think I have secured the system enough. Have I really? I think for the most part yes. I think of WPA as the club. you can still steal the car (aka break into my network) but why would you waste time with my network or car when you can steal the guy down the streets car who left the door unlocked or just doesn’t have a club? I have a myth of security.

Another example of gaping security wholes is another growing wireless standard, Bluetooth. I have been a fan of it since I first read about it almost a year before the first mobile phone with bluetooth came out. And when it did, I bought one. A Ericsson (they were just Ericsson back then) r520. So for the record I am a fan of Bluetooth. I am a fan of wifi for that matter. I remember when I was at my first tech job back in 96 I got to play with a demo of a 1mbit (i think) wireless card and point from Raytheon. The problem is bluetooth has the same security myth. It also has the problem of the media blowing the issues into this huge security crisis. The simple fact is that most phones and other bluetooth devices were configured to be as easy to configure as the manufacturer could make them. That means allot of devices are setup to be discoverable by default. That means that if the bluetooth radio on a phone is on, someone else looking for bluetooth devices can see your phone if you are in range. To prove that, last week on Amtrak home from my trip I was able to view up to 4 other bluetooth devices from my seat. To protect yourself all you usually have to do is make a change in the default configuration of your device to not be “discoverable”. Do most people do this? Nope. But if you turn discovery off by default you have people complain that setting up partnerships are too hard. See the problem?

You have people then go around thinking all is ok, until they have a problem or someone tells them their phone is at risk of being broken into. First of all that may or may not be true given that you have to set passkeys, etc. For argument sake lets say it is an accurate assessment. These people then freak out and get mad at hardware vendors for delivering unsecured devices. How do you win?

Most of the time people live in the dream world that their stuff is safe. The crazy thing is that maybe 99% (or the vast majority) of the time people’s fantasy worlds are not broken. That perpetuates the myth that all is safe. Even if someone has been using their unsecured wireless internet connect for free for months.

The more I think about it, the more security myths I think about. And I am only thinking in terms of personal computer security. Don’t get me started on other society security concerns.

A perfect example is a few years ago my mom called me after she saw an Oprah on TV. She was calling to warn me that email I send wasn’t secure and that anyone can intercept and read it. She was shocked, but Oprah set her straight. I was like, yeah mom of course email is not secure. Old news. She was surprised that I knew that. It is scary that the general population assumes something like email is secure, and it isn’t. On the flip side can email be intercepted? Of course if it is not encrypted. Is most mail not encrypted? Yes. Will my mom have to worry about her neighbor reading her email or some stranger intercepting it? Probably not. It is very possible to do, but come on who really is going to try and sniff out her mail? its a real threat, but I don’t think most people won’t ever have to worry about it. Doesn’t mean I don’t think we should all get certificates and secure our mail. I would love to do that, but it is impractical in today’s world. So you see even I let the myth of my stuff is secure live on some level. We all do it, and if you don’t think you do, you are kidding yourself.

More VMWare Work

by

This week I actually am working on a technical project, not just managing them. I built a test environment to put up a windows 2003 terminal server. I used VMware’s GSX server to do it. I was testing what ports I would need to open up if I wanted to access a terminal server via the TSWeb client Microsoft gives you. Turns out even if the web server and the terminal server are the same machine you have to open up the RDP port on your firewall anyway. That wasn’t the answer I wanted to hear, so I am looking into Citrix to see if I can use their product and only open up port 80? Waiting to hear back from them.

Technorati Tags: , , ,

Treo Refresh

by

Last night I ended up wiping my Treo 650 and reinstalling all my software. I have been having issues with it rebooting every time I tried to sync my mail using Snappermail. That program is half the reason I use my Treo over the Blackberry, so I was not too happy when it started acting up earlier this week. I had to go on my trip Thursday without being able to use the Treo for email. I got by on the Razr and the Blackberry but I like the Treo better. After the reinstall Snappermail seems to be working. But I have had 2 random reboots since the hard reset. Once after a call to my sister right after I got out of the subway this morning. The other when I was syncing with my Powerbook. The sync reboot also happened often before I wiped the Treo, but I am not sure if it is because of the config on my Powerbook or the Treo.

I will try using for a while longer, but if the random reboots continue I will have to seriously consider an alternate PDA phone. As much as I like this device, I cannot have it randomly rebooting on me almost daily!

HL DL 320 SATA Servers

by

I have been working with a few HP Prolient DL 320 SATA servers. The price was right on them and they have decent specs. The issue I had with the last round of Supermicro SATA box’s weren’t the supermicro box’s them selves, but the SATA RAID cards that went into them. They would fail much more than their SCSI counterparts. Also the array controllers would not rebuild without crashing computers. We tried several brands. We are using 3ware for new deployments of older chassis SATA servers. They seem the best out of all I have seen. They were a crap shoot. These HP box’s seem to rebuild fine in our tests. Time will tell if they drives hold up, but so far I think HP finally got it right with a low end non SCSI RAID system.

More Data Cleanup

by

I moved all my files off of one of my firewire drives this week. This was so I can format it to work on my Mini. Dam NTFS not working on the mac. It took 36 hours to robocopy the data off of the firewire drive onto another USB drive. Even-though it was a USB 2.0 drive, I think it was running at 1.1 speeds. 36 hours was an awfully long time to copy 90 gigs of stuff. Even if it was mostly small files.

I am in the process of backing up old backup files to DVD’s. I am also just deleting tons of old crap that I don’t need anymore.