davidlaporte.org

It's a cloud in a box!

2010-11-04T12:30:00.002-04:00

To better understand cloud computing, I wanted to build my own cloud. Ubuntu Enterprise Cloud, an install option for Ubuntu promised to make this a pretty straight-forward exercise and was the obvious choice. The current version, 10.10, is based on Eucalyptus 2.0. A Eucalyptus deployment has the following components:

A cloud controller that is exposed to the world through a basic web interface and an Amazon EC2-compatible API
A "walrus" service that implements an S3-compatible service
One or more cluster controllers that manages the running instances and provides networking (public IPs, etc)
One or more storage controllers that provide EBS-type block storage
One or more node controllers that actually run the virtual instances (using a KVM or Xen hypervisor)

The following is stolen from Wikipedia and shows how the Eucalyptus components interoperate:

As I mentioned in my previous post, I have a single box on which everything will run virtualized. All cloud components can run on a single physical box (or VM, in my case), but that would be too easy! After quickly setting up a single node cloud, I decided to try building something larger. This wasn't exactly straight-forward - I'll describe the problems I ran into while building everything. First thing... CLC and Walrus want to be together, don't separate them! It took some trial and error before determining that the "At least 3 Physical Systems" topology located here was the path of least resistance. I ended using that exact topology, with the addition of one more node controller. In retrospect, the IP masquerading issue I mention below might have caused my CLC/Walrus problems, but I still recommend keeping them together.

The instructions on the Ubuntu site (here) are actually pretty good. Avoid using UEC web interface to configure the components - it just tripped me up. Everything you need to edit is in /etc/eucalyptus/eucalyptus.conf, /etc/eucalyptus/eucalyptus-ipaddr.conf, and /etc/eucalyptus/eucalyptus.local.conf (only on the CC). Only specific sections apply to each component, but I found it easier to just keep them synced.

For networking, I used one network to connect the components together and another to connect the CC/SC and NCs. To make things simple, I created two bridges on the KVM host:

auto dummy0
iface dummy0 inet manual

auto br1
iface br1 inet static
address 10.168.12.1
netmask 255.255.255.0
bridge_ports dummy0
bridge_stp off
bridge_fd 0
bridge_maxwait 0

auto dummy1
iface dummy1 inet manual

auto br2
iface br2 inet static
address 172.19.1.254
netmask 255.255.0.0
bridge_ports dummy1
bridge_stp off
bridge_fd 0
bridge_maxwait 0

The kernel only wanted to let me create one dummy interface by default, so it was necessary to add dummy numdummies=2 to /etc/modules. On the CLC/Walrus box VM, I created one NIC bridged to br1. The CC/SC and NCs each had two interfaces, bound to br1 and br2, respectively. On each, I created a br0 (bridging to eth0) and a br1 (bridging to eth1).

The 10.168.12.0/24 network (br1=>eth0=>br0) connected all Eucalyptus components and is the "public" network. The 172.19.0.0/16 network (br2=>eth1=>br1) was used by the instances, has DHCP provided by the CC, and is the "private" network. Set:

VNET_PUBINTERFACE="br0"
VNET_PRIVINTERFACE="br1"

On the CC/SC and NCs and:

VNET_BRIDGE="br1"

on the NCs (or just set them on all components like I did). For what it's worth, this is what /etc/eucalyptus/eucalyptus.local.conf looks like on my CC/SC:

VNET_MODE="MANAGED-NOVLAN"
VNET_SUBNET="172.19.0.0"
VNET_NETMASK="255.255.0.0"
VNET_DNS=""
VNET_ADDRSPERNET="32"
VNET_PUBLICIPS="10.168.12.200-10.168.12.254"
VNET_CLOUDIP=""

After updating any addressing-related parameters, be sure to pass CLEAN=1 to each component for a clean restart! (eg. /etc/init.d/eucalyptus-cloud restart CLOUD=1)

With networking configured, I now needed to get the components to work together. This was a non-trivial exercise, most of which was my own fault. First, watch the /var/log/eucalyptus/cloud-debug.log, /var/log/eucalyptus/cc.log, /var/log/eucalyptus/nc.log log files closely. If you see any exceptions, you have a problem and things probably won't work. In my case, I had configured IP masquerading for outbound traffic originating from the IP addresses of the cloud components. Somehow this mangled the avahi advertisements that get sent amongst them. Disabling the IP masquerading quickly solved the problem, but it took some time to find the root cause. Another issue is that something keeps changing ownership of /var/log/eucalyptus directory to root on the node controllers. Each time I bounced them, I needed to manually chown them to eucalyptus. Not sure what caused that...

Once the logs look clean and euca-describe-availability-zones verbose shows available capacity, you can try download/install an image and run an instance. The easiest way to get an image is through the "Store" tab on UEC web interface. If you want to download something more recent (like this), you can install it from the CLC using the uec-publish-tarball command.

Hybridfox is pretty nice Firefox extension to manage your instances.

It works well - the credentials you'll need are all available through the UEC front-end. One problem I had was that my node controllers had a pretty respectable 2 CPUs and 3GB of RAM, yet Eucalyptus/Hybridfox kept telling me that only two instances would run on each. The solution was to modify the MAX_MEM and MAX_CORES settings in /etc/eucalyptus/eucalyptus.conf on the node controllers. Tweaking those parameters will let you oversubscribe CPU (but not memory). The other gotcha is that you need to explicitly allow traffic into the instances. Run the following on the CLC to allow ICMP and SSH traffic:

euca-authorize default -P icmp -s 0.0.0.0/0
euca-authorize default -P tcp -p 22 -s 0.0.0.0/0

Conclusion

This was a bit of a brain dump, apologies for the lack of organization. This was certainly a valuable learning experience and helped to thin the heavy vapor surrounding cloud computing for me. If you have any questions, need working configurations, or have explanations for some of the quirks I saw, please let me know!

Turns out, the rabbit hole goes pretty deep.

2010-11-04T09:16:00.000-04:00

A few months ago I built a test system at home to experiment with various hypervisors. As my "base" hypervisor, I chose KVM. KVM is widely supported across all major Linux distributions and, with RedHat and Ubuntu dropping support dropping support for Xen, seems to the de facto open-source virtualization platform. I chose Ubuntu as the host OS since I wanted to avoid compiling the KVM components and Ubuntu stays more up-to-date with KVM than CentOS or Fedora. For hardware, I choose an AMD Phenom II X6 1055T and, using an arbitrary rule of thumb of 2GB/core, went with 12GB of RAM. I chose AMD for 2 reasons:

At the same price point, I could get two more cores from AMD. On a desktop, I would've gone Intel, but in a virtual environment I prefer more, slower cores to fewer, faster ones.

KVM can virtualize AMD's SVM (secure virtual machine) instructions, allowing for nested virtualization. Not any AMD CPU will do - it needs to be a Barcelona or Phenom.

A few months later, and here is the rat's nest I've built:

I'm using libvirt to manage the Openfiler, Xen, Proxmox (KVM and OpenVZ containers), and UEC (KVM/Xen) VMs that are nested inside KVM. Although KVM supports nested virtualization, libvirt currently does not (although a patch seems to be on the way). To enable it, I needed to create a /usr/bin/kvm-nested shim that simply calls /usr/bin/qemu-system-x86_64 with the right options:

#!/bin/bash
exec /usr/bin/qemu-system-x86_64 -enable-nesting -cpu host "$@"

I need to manually modify the XML file for each guest to point to the shim instead of /usr/bin/kvm directly. Kludgy, but it works! The keymap also doesn't appear to be set properly through virt-manager, so I need to add keymap='en-us'/ to the graphics stanza of the XML.

For storage, I have Openfiler using a raw partition on the host (passed through virtio) to maximize performance. That pool of storage is sliced up and exported via iSCSI to the nested hypervisors, which spin up guests on it.

Clearly, this isn't something recommend for any production environment, but does let you play with multiple hypervisors, clustering, failover, DRBD, cloud computing, and all sort of other technologies that would otherwise require an entire lab environment. The glaring omissions are Hyper-V and ESXi/vSphere. I've been told Hyper-V is working in development code and that ESXi can be massaged into running. If/when I'm successful in getting those working, I'll post a follow-up.

Please Support Team Peen 2010!

2010-05-06T21:28:00.003-04:00

On Sunday, June 13th, Team Peen will be participating in NSTAR's Walk for Children's Hospital. It's our fifth year and we need your support!

Our son Aaron was born with Transposition of the Great Arteries and saved when he was just three days old. Almost five years have past, but time hasn't diminished the deep gratitude we feel towards the heroes at Children's - the most compassionate, competent, and professional group of people we have ever met. Please help us support the good work they do at Children's by sponsoring us with a donation or joining Team Peen for the walk.

Thank You!
Lyssa, Dave, Aaron, and Adrian LaPorte

Asus WL-520gU, you're my hero!

2008-11-07T21:35:00.004-05:00

Since I apparently don't have anything better to do, I wrote a tutorial on how to quickly get an Asus WL-520gU running as a Linux filer. If you've got nothing better to, check it out here.

Yearbook Family Photos

2008-09-30T19:47:00.005-04:00

Now this is scary...

Life Poster

2008-09-02T00:05:00.013-04:00

Back in early 2005, Mike Matas blogged on how to create a life poster, a 20x30 collage of ~100 images. His instructions were specific to iPhoto (which I don't use), so I was forced to come up with my own. I've been using this method to create posters of my son since 2005. Each year I have to remember how I did it the year before, so this is more for my benefit than yours!

The first thing you'll need to do is find 98 landscape photos to use. You'll need to crop every image to a 4:3 aspect ratio. This needs to be exact...it seems that the Photoshop Elements crop tool prefers to work in inches and the rounding leaves things slightly less than precise. I worked around this by cropping each image 4in x 3in and then bulk resizing the images (File->Process Multiple Files) to 2048x1536.

The next step will use the image filenames to order them in the collage, so its worth randomizing them to spice things up. The following Perl script will do just that:

#!/usr/bin/perl

opendir(DIR, ".");
@files = grep { /\.jpg$/i } readdir(DIR);
closedir(DIR);

foreach my $file (@files) {
 $new = int(rand(10000000)) . ".jpg";
 print "renaming $file to $new\n";
 rename($file, $new) || die "Unable to rename $file to $new: $!\n";
}

Once you have the images aspect normalized and names randomized, you can assemble the collage. Select File->Contact Sheet II and choose the folder containing your images. Select "Pixels" as the units, specify "8000" as the width, and "11955" as the height (the height should be 12000, but that left extra whitespace in the collage...YMMV). Uncheck "Use Auto-Spacing" and set vertical and horizontal spacing to 0. Specify 7 columns and 14 rows. It'll take a few minutes to build, so click OK and walk away.

When you return, the collage should be built. You should probably zoom in and check all the borders to make sure there's no extra whitespace. If there is, you'll need to tweak the height setting in the contact sheet settings and re-run. If you don't like the ordering of the images, just re-run the Perl script until you find a better one. If you've needed to tweak the height setting, be sure to resize to 8000x12000 pixels when you're done.

Once you're satisfied, you can upload your poster to your preferred photo processor. I like Shutterfly, and they offer 20x30 prints for $23. It's worth mentioning that Posterino, a utility built specifically for creating these type of posters, is available for only $25. I'll probably be going that route next year - it's well worth the time saved.

Fun with HDR

2008-08-03T19:03:00.005-04:00

I've been taking pictures in a RAW file format for the past few months. The images are much larger than an equivalent JPEG and they require some post-processing, but I think it's worth it. With the growth in flash sizes (16GB for ~$50?!?!) and software that automatically handles the post-processing, you might as well grab as much data as your camera will give you.

I've also been reading up on high dynamic range imaging (HDR) and picked up a nice little program called Bracketeer that can combine multiple exposures of an image to create a more evenly exposed image. Typically this is done with bracketed exposures by the photographer, but a tripod is needed to allow for a clean merge.

With a RAW image, though, you can play with the exposure details in software after the fact. It's debatable whether the final result is as impressive as a true bracketed exposure (or so I'm told), but the convenience can't be beat! Below is an HDR image created by merging three exposures (-2EV, 0EV, and +2EV) and a little bit of Photoshop action. The difference is subtle (mouse-over for the HDR), and I probably could've obtained the same result by tweaking the RAW, but I think it has a punch that the original lacks.

exchange2ics released

2008-05-10T10:11:00.004-04:00

Inspired by Ryan Hadley's Goodbye Outlook post, I decided to build on what he had done. exchange2ics, a perl re-write with some additional functionality, is a multi-user gateway to allow calendaring clients read-only access to Exchange calendars via ics subscription. It implements caching to reduce server load and does not store any user credentials.

Another Geek Toy

2008-04-19T10:57:00.004-04:00

A friend recent pointed me at Phun, a physics engine. It's in the same vein as the Bridge Construction Set (which I'm mentioned previously), but it does so much more. There's a great YouTube video posted that shows what it can do. Check it out!

I love FiOS, part 2

2008-03-18T00:01:00.011-04:00

Verizon now offers a symmetric 20Mbps service in my area. In my testing so far the uplink doesn't quite reach the quoted speed, but it could be my router. It's a beautiful thing!

Aaron's Adventures

2008-02-28T20:50:00.016-05:00

We recently painted Aaron's room a lime green (thanks Steve!) that works great as a greenscreen in Apple's Photo Booth software. Click the thumbnail to see some of the crazy places Aarons's been.

Introducing Adrian Richard LaPorte

2008-02-09T01:20:00.001-05:00

Adrian Richard LaPorte was born at 10:36pm on February 8, 2008, weighing 6lbs, 11oz. He was a bit early, but mother and son are both doing well!

VM Encryption on the Cheap

2008-02-06T15:52:00.001-05:00

I've long run virtualized servers using VMware's workstation, and more recently server, product. Something that's always concerned me - not so much in my particular circumstance, but in the general sense, is that the there's no encryption of the VM disk images. Sure, you could encrypt the actual files, but that may not allow on-the-fly decryption and probably won't give you the necessary granular administrative access and scalability. The solution, in my opinion, is to have the virtualization software handle encryption natively. Since none of the available packages does that, the next best solution is for the guest to handle it.

Anyways, I've found a poor man's solution to the problem. Using the free VMware Server and in conjunction with the recently released TrueCrypt 5.0, it's possible to do full disk encryption (FDE) for free inside a VM. I've tested it myself (for all of an hour) and it seems to work great with only a minor performance impact.

I'm running VMware Server 2.0 beta and I'm a little wary of recommending it. There are some bugs regarding interface bridging that took forever to figure out. Basically, you'll need to manually edit

C:\Documents and Settings\All Users\Application Data\VMware\
VMware Server\netmap.conf

if you need to anything beyond basic host-only, NAT, or bridged networking. The web interface is absolutely horrible and they've removed the windows client application. Fortunately, the Virtual Infrastructure 2.5 client works great (better than the old client, actually) and comes with the Linux version of VMware Server. Download it here and look for

vmware-server-distrib/lib/hostd/docroot/client/VMware-viclient.exe

Fun FiOS Facts

2008-02-05T16:38:00.000-05:00

A friend recently mentioned that FiOS now has HD content on-demand. I was wondering how it compared with SD bandwidth, so I ran a test. Turns out, an SD stream is 3870Kbps while an HD stream is 15480Kbps, exactly 4 times the bandwidth. Another thing I found interesting is that the stream is not variable - the stream was constant regardless of the on-screen action.

Verizon claims that it automatically increases your cap (or uses a dedicated PVC, I'm not sure which is true) when watching on-demand content. I started an HD stream and, sure enough, I locally saw 35Mbps while a remote bandwidth testing site showed the full 20Mbps.

Have I mentioned... I love FiOS?

He really looks like Aaron!

2007-12-28T20:58:00.000-05:00

Lyssa and I went for a 3D ultrasound today. It has no medical value, but it was definitely interesting to see the baby in utero. As part of the package, we got a bunch of pictures, a DVD, and a nice video. It's amazing how similar the baby looks to Aaron!

Tell telemarkers where to go

2007-12-05T11:40:00.000-05:00

I've updated my OSX callerID scripts to include integration with whocalled.us, a service that allows users to submit phone numbers of known telemarketers. You'll need to register with the site and it slows the notification pop-up for unknown callerIDs slightly, but it works great!

It's (going to be) a boy!

2007-10-04T21:33:00.000-04:00

Aaron's going to have a brother! We had an ultrasound today and it was confirmed for us. Now, if only Lyssa would let me name him "Otto"...

Konichiwa!

2007-10-03T22:57:00.000-04:00

I will be presenting, along with Eric Kollmann of Boise State University, at Black Hat Japan 2007. We will be talking about Passive OS Fingerprinting Using DHCP. If you're interested in the topic, Eric has written an excellent white paper.

Lots of pictures to be posted when I return, no doubt!

Wi-Foo in Mac Land

2007-09-04T22:20:00.001-04:00

A few weeks ago we were preparing for a wireless presentation on wireless insecurity. Being a Mac user, I thought it'd be great if I could use all my (Linux-based) wireless tools inside a virtual machine instead of using a dedicated system or dual-booting. Turns out, you can! Booting BackTrack 2 in a VMware Fusion VM works perfectly and, providing you have the right wireless adapter, you have direct access to the hardware. I've posted a tutorial with the details as well as an updated BackTrack 2 VM image to get you started.

Family Portrait, Simpsons Style

2007-09-03T16:56:00.000-04:00

The Simpsons movie site has a fun little toy that lets you build a Simpsons version of yourself. Here's what what we'd look like if we lived in Springfield.

Little LaPorte #2

2007-08-24T21:35:00.000-04:00

On or around March 12th 2008 , we'll be adding another member to the LaPorte clan! Lyssa has her first ultrasound yesterday, and the little peanut (literally, it looks like a peanut) is doing well. I'm amazed, but Aaron seems to really understand that there's a baby in Mama's belly. He'll be a great big brother.

Gone in 14 seconds

2007-08-16T20:59:00.000-04:00

Once upon a time, WEP was considered "screen door" security. Sure it was broken soon after it was released, but it kept honest people (or at least those without a few spare hours/days to spare) honest. Since then, the time required to crack a WEP key has steadily decreased. The latest PTW attack - as implemented by aircrack-ng - reduces the time dramatically. Below is a crack of a 40-bit WEP key on a nearly silent WLAN. Using nearly 20 thousand weak IVs generated by about 4 minutes of ARP injection, it took 14 seconds to crack the key. WEP gives a false impression of security, making it even more dangerous than no security at all.


                                                      Aircrack-ng 0.9.1


                                      [00:00:14] Tested 72/140000 keys (got 18773 IVs)

KB    depth   byte(vote)
 0    2/  4   37(  95) 1B(  92) 91(  91) 70(  90) 8B(  90) A2(  90) C4(  90) E6(  90) 4A(  89) 67(  89) 7E(  89) 81(  88)
 1    0/  3   9D( 103) A1(  99) B7(  97) 97(  92) 47(  91) 5C(  91) E6(  91) 5D(  90) 65(  90) 67(  90) 84(  90) 5E(  89)
 2    1/  3   DA( 101) 87(  95) 1E(  93) 88(  93) 66(  91) E3(  91) 48(  90) 11(  89) 46(  89) 4B(  89) AA(  89) 68(  88)
 3    1/  2   48( 103) 26(  93) BD(  93) 86(  92) D0(  92) BB(  91) C0(  91) 8F(  90) 47(  89) DD(  89) 51(  88) E9(  88)
 4    0/  1   3D( 111) A2( 100) B4(  97) 31(  95) 41(  94) 21(  93) 36(  91) 2E(  90) C7(  90) C8(  90) D0(  90) 4A(  89)

                      KEY FOUND! [ 1B:9D:DA:48:3D ]
     Decrypted correctly: 100%

I love FiOS!

2007-06-19T10:19:00.000-04:00

A friend mentioned that Verizon FiOS is now offering a 20Mbps/4Mbps service for $5/month more than the existing 15Mbps/2Mpbs. I called them up this morning and, sure enough:

London Bridge is falling down

2007-06-17T21:40:00.000-04:00

I discovered the Bridge Construction Set after reading this blog entry a few weeks back. It's an extremely cool "game" that let's you build bridges of different types (suspension, drawbridge, truss, etc) across spans. Building is fun, but the best part is simulating how your bridge holds up . It's not terribly scientific - and the interface could use improvement - but it's definitely worth a look if you (or your child) is into physics or watching trains fall to their doom.

Thanks for the support

2007-06-10T21:25:00.000-04:00

It was a perfect day for a walk and we had 13 people walking with us. Thanks to everyone who walked and those who donated. With your support, we've passed our goal and have raised over $3500.00 for Children's Hospital Boston!