Posts from *scottstuff*...
Posted by Scott Laird 54 days ago
So, one of the features that’s been in the WebKit tree for a while but finally showed up in Safari today is downloadable font support. You can use CSS to apply a specific font to an element and Safari will download the font from a URL provided.
That’s really cool, but the odds of an exploitable buffer overflow somewhere in the font rendering pipeline has to be almost 100%. I mean, almost every graphics format has had multiple exploitable bugs on every platform, and I can’t see how a complete OpenType renderer can be any less complex than JPEG. Even worse, this is a new attack vector, against a part of the system that wasn’t part of the security perimeter before. Is there a way to turn this off?
Posted by Scott Laird 54 days ago
The Verizon FiOS installers have been circling the neighborhood lately, tearing up sidewalks, digging holes in lawns, and (finally) pulling fiber down the telephone poles in the direction of the nearest CO. According to their call center, they’ll probably be able to upgrade me from 3/768 DSL to 15/15 fiber sometime around the middle of next month. I’ve been waiting for years for them to finally make it to my place, so I’ll probably have to have a party or something :-).
Just for the fun of it, I was reading Verizon’s business FiOS system requirements page. Here’s a snippet:
| Speed | Recommended CPU speed | Recommended FSB speed | Recommended free disk space |
|---|
| 5/2 Mbps | 600 MHz | 100 MHz | 128 MB |
| 5/5 Mbps | 733 MHz | 133 MHz | 200 MB |
| 50/10 Mbps | 2 GHz | 330 MHz | 500 MB |
So, not only do I need a faster CPU to enjoy a faster connection, I also need additional free disk space? Huh. Who knew?
Posted by Scott Laird 54 days ago
I really thing I should get some credit for this: I managed to wait until March 2008 to buy myself an iPhone. I didn’t rush out and wait in line on the day they shipped. I didn’t buy myself one when the price fell. I even bought my wife one first, in December for our anniversary. Admittedly, I was in a car on the way to the nearest Apple store when Steve first announced the iPhone last January, before he bothered to mention that it wouldn’t ship for 6 months, but it’d be totally unfair to count that against me.
More seriously, it took me quite a while to convince myself that it was time to retire my trusty Nokia E61. The E61 served me well for almost 2 years, but it was time to swap. In theory the two phones are fairly similar–fairly large screens, WiFi, EDGE (the E61 does 3G in Europe, which doesn’t help me much here), and Safari-ish browsers. In reality, they’re a wonderful demonstration of why feature checklists are worthless. Here are the things that I care about most:
- The iPhone is hands-down better for reading and writing email via Gmail. The native IMAP client is good enough, and the iPhone version of the Gmail web interface is vastly better then the version that we feed to the E61, even though they’re running very similar WebKit-based browsers. The E61’s keyboard is better, but the HTML edit box in Gmail’s mobile web interface is so bad that it cancels out the keyboard advantage.
- The browser is better. It’s faster, it doesn’t crash on every third Amazon page that I try to load, and the touchscreen scrolling is better than the joystick on the E61.
- It’s actually usable as a music and video player. In theory, the E61 can play movies and music, but (1) there’s no easy way to copy content onto it (unlike the N-series phones, it doesn’t come with an iTunes plugin), (2) out-of-the-box it only supports wacko video codecs, and (3) the UI’s bad.
- It’s easier to charge. My E61 has never charged right; swapping batteries and chargers never made a big difference. The iPhone, on the other hand, uses a semi-standard connector and charges via USB. It’s easy to find iPod cables and USB jacks, but finding a spare Nokia charging cable is tough, at least around here.
- I’m never, ever going to have to see Nokia’s stupid “which network connection do you want to use?” dialog box again. For some reason, Nokia decided that asking the user before letting apps use the network every single time was a good move. It’s smart enough to know which networks are available, and which ones I’ve configured it to use, but it’ll still show me a list with one or two choices every time. Bleh.
- The on-screen phone keypad includes letters. It’s a stupid thing, but the E61 doesn’t give you an easy way to dial vanity phone numbers, because there’s no way to tell which numbers map to which letters. I mean, can you tell me off the top of your head which numbers you need to press to dial ‘1-800-884-SOIL’?
The E61 wins a few points, though:
- It comes with a SIP client that’s actually be useful at home for me.
- It’s open, and you can install useful software.
- The Nokia podcast client does a great job of copying new episodes of Escape Pod for me on the fly.
- It’s louder. That makes it harder to miss calls.
SMS is kind of a push between the two; the E61’s ringer is louder and it has a better keyboard, but it takes way too many button presses to do anything.
So, for now I’m using the iPhone. Yeah, I could have waited for the 3G iPhone or Android, whenever they appear, and I may swap for one (or both?) of them when they’re available. From everything that I’ve seen, Android’s programming model will be vastly better than the iPhone SDK, at least for the weird types of things that I care about, but it’s not shipping yet.
Posted by Scott Laird 99 days ago
Soo Valley, near Whistler, B.C.
I spent Thursday and Friday in Whistler along with a few hundred co-workers, enjoying the Seattle version of Google’s annual ski trip. I took the opportunity to go snowmobiling for the first time, and took my camera along.
I need to find another excuse to go play in the snow with my camera; that was too much fun.
Posted by Scott Laird 110 days ago
One of my favorite places to take pictures is on Fir Island, in Skagit County, Washington. It’s a farming community in the Skagit River delta, and it’s home to around 1 million migratory birds every winter, including Snow Geese and Swans. Ever see a cloud of geese turn the sky white?
It’d been weeks since I’ve been able to spend time hiking around with my camera, so I drove up Monday morning before dawn to see what I could find.
Dawn on Fir Island
There were geese and swans flying by all morning, but I never got a really great shot of any of them from up close. There were thousands of them visible in the distance, though.
Morning flight
It’s been really cold lately, and there was ice everywhere, including the sea shore. There was a weird layer of office over everything; I assume that it was left behind by the falling tide. The plants looked like they’d been wrapped in cellophane.
Plants in Ice
Finally, on the way out, I spotted this Heron hiding in a drainage ditch a few feet from the road. He was my third or fourth heron of the day.
Cold Heron
I can’t help thinking that a bit of fill-flash would have helped there, but he was close enough that it probably would have startled him.
Posted by Scott Laird 113 days ago
As any experienced digital photographer can tell you, the trick to getting repeatable color out of a lab is to get a good ICC color profile for the lab’s printer and using it for every print that you make. I’ve been a big fan of Dry Creek Photo’s printer profiling service for years; they work with labs to build quality profiles and then publish them for free on their website, along with some documentation on how to use them.
Unfortunately, actually using the profiles is a pain for most users. The process looks roughly like this:
- Using a color-profiled monitor, get things looking the way that you want on the screen. Save.
- Resize the image to the correct resolution for the print size and resolution that are required for your lab. Different labs want 300, 320, or 400 DPI.
- Convert the image to the color profile for that lab that you’re using. Possibly re-adjust colors slightly.
- Pad the image out to a specific number of pixels to keep the lab’s printer from trying to re-scale your image. The exact settings depend on the printer and paper size. There’s a big chart on Dry Creek Photo’s website.
That’s not a big deal if you’re printing one or two pictures for framing, but it’s a huge pain when you have dozens or hundreds of pictures to process. Photoshop actions can help, but it seems like something is always going wrong.
When Adobe released their Lightroom SDK, I had high hopes that it’d make it easier to automate most of this. Unfortunately, Adobe didn’t expose their profile conversion engine to the SDK in the first SDK release, so it’s not really possible to build pre-profiled images directly out of Lightroom without using some external tool to do the profile conversion. A few weeks ago, Timothy Armes released LR/Mogrify, which uses ImageMagick’s mogrify command-line tool for profile conversion. It’s a cool tool, but it replaces a 5-6 step process in Photoshop with a dozen text boxes that you need to fill in to get good results. It shows promise, but it’s not quite the tool that I’m looking for.
What I really want is a Lightroom export plugin that asks you three simple questions:
- Where are you going to print this?
- Which paper are you going to use?
- What size do you want?
Then it does all of the hard work on its own. It’d fetch the correct profiles from Dry Creek, install them, figure out which resolution to use, handle image rotation, do some amount of pre-print sharpening, and then spit out a JPEG for you. Yeah, you could do a bit better with Photoshop and spending some time dealing with soft-proofing and sharpening, but I’m not willing to do that for 50 4x6 prints.
I’ve spent a bit of time this week building the first part of the tool–a machine-readable profile database. I’ve extracted a list of 765 current ICC profiles from Dry Creek Photo’s website, discarded the profiles that haven’t been updated in years, and produced an XML file that looks sort of like this:
<lab>
<name>Costco #747name>
<address>24008 Snohomish-Woodinville Rd. SE, Woodinville, WA 98072address>
<phone>425-806-7708phone>
<printer>Noritsu 3411printer>
<paper>Fuji Crystal Archivepaper>
<resolution>320resolution>
<notes>Note: This lab has multiple printers. Request your profiled prints be run on Noritsu 34?Pro-B.notes>
<profile>
<name>Glossy paper profilename>
<date>January 17, 2008date>
<url>http://www.drycreekphoto.com/icc/Profiles/IccFiles/Washington/Costco-WA-Woodinville-Gls.iccurl>
profile>
<profile>
<name>Lustre paper profilename>
<date>January 17, 2008date>
<url>http://www.drycreekphoto.com/icc/Profiles/IccFiles/Washington/Costco-WA-Woodinville-Lus.iccurl>
profile>
<size>4x6insize>
<size>5x7insize>
<size>8x10insize>
<size>8x12insize>
<size>12x12insize>
<size>12x18insize>
lab>
<lab>
...
The
Right now, the XML file lists 380 labs. Entertainingly, that’s 379 Costcos plus Adorama. It looks like Dry Creek has dropped almost everyone else. If there’s a second source of non-Costco profiles, I’d love to know about it.
So, anyway, I have this XML available at http://profiles.sigkill.org/profiles.xml. It’s mostly automatically generated, and I can rebuild it in under 5 minutes. I’ll keep it up to date if people are interested in the data, if not it’ll die off eventually. If you have a tool that wants to use it, then send me mail and let me know that it’s useful. If you have any changes that you’d like to see, or anything that I should add, let me know and I’ll see what I can do.
Posted by Scott Laird 161 days ago
I’m really not looking forward to going home from my vacation.
Sunset from Waikiki
Posted by Scott Laird 162 days ago
I bought my wife an Olympus 790SW point-and-shoot camera before we left for Hawaii on vacation, and I’m growing increasingly fond of the little thing. It doesn’t really compare to my Canon 5D’s image quality, but it’s so small and handy that it’s easier to carry. Even better, it seems to be indestructible–it’s submersible and can be dropped up to 4.5 feet without breaking anything.
In other words, it’s the perfect beach camera for families with small kids. Plus, you can take it snorkeling, just in case one of these pops up:
Sea turtle
or some of these:
Reef fish
There are more pictures on Flickr if you’re interested.
It also takes semi-decent video. I wouldn’t confuse it with a HD camcorder, but I wouldn’t take the camcorder in the water, either. Here’s my son’s first time snorkeling:
It’s around $260 on Amazon, if you’re interested.
Posted by Scott Laird 171 days ago
So, I’m in Hawaii this week for vacation with the family. It’s been a while since I’ve had a real vacation that didn’t involve cross-country drives or tight deadlines. I’ll write more eventually, but for now, have a few pictures:
Lighthouse at Dawn
Obligatory palm tree
Sunset Beach surfing
Posted by Scott Laird 177 days ago
It looks like Lightroom 1.3 is out. More excitingly, the Lightroom Export SDK is now available. I guess it’s time to learn Lua.
I might wait until after next week’s vacation to pick up Lua, though. I have a beach calling my name.
Posted by Scott Laird 202 days ago
Interesting trivia: staggered drive spin-up is an optional part of the SATA II spec. Unlike most SCSI drives, though, it’s not controlled through a jumper. Instead, it uses one pin on the SATA power connector. If pin 11 is floating, then the drive is supposed to wait to start spinning.
Apparently the 650W power supply in my new server only provides enough current to spin up 10 drives at once, because adding an 11th drive makes it turn off on its own immediately. Sigh. I wonder if anyone makes delayed-spin SATA power dongles?
Posted by Scott Laird 202 days ago
I mentioned that I bought a Gigabyte GC-RAMDISK (a.k.a. i-RAM) to go in my new home file server, largely to see if using it as a solid-state log device would improve ZFS performance.
Unfortunately, I’ve been completely and totally unable to get the card to do anything at all. I’m not sure if I have a defective card or if Gigabyte’s SATA implementation is just really buggy. When I plugged it into the motherboard’s ICH9R SATA ports, the BIOS didn’t even show it on the boot-up scan and Solaris reported it as failing to initialize correctly. When I plugged it into the Supermicro AOC-SAT2-MV8 8-port SATA card, the BIOS could see it but Solaris gave a similar error. Connecting it to the motherboard’s Marvell eSATA ports made the Marvell hang at bootup and made Solaris really unhappy, spewing drive failure messages all over the console.
I can’t find a single review that suggests that anyone has got this to work with a recent motherboard. Digging through the Linux kernel mailing list suggests that it has a really spotty SATA implementation. Apparently they developed it using a couple Windows drivers as a comparison, instead of actually paying attention to the SATA spec.
So, it’s going back to Amazon today. It was a nice idea, but it just doesn’t appear to work. I don’t know if it’s broken or just poorly designed, but either way it’s not useful to me.
Posted by Scott Laird 204 days ago
I now have Solaris up and running and reasonably stable-looking, after only 12 hours of work. A number of things turned out to be bigger issues than I’d anticipated, largely because it’s been years since I last used Solaris and, frankly, Solaris’s disk partitioning and formatting tools suck.
- My first problem is still unresolved: my BIOS refuses to boot from the IDE DVD drive that I installed. Once the system boots, it works just fine, so I’m not sure what’s up. Maybe a BIOS bug. Fortunately, the system’s perfectly happy booting off a USB DVD drive, and (amazingly) Solaris is happy installing from it.
- The GC-RAMDISK card that I was looking forward to testing is a complete failure so far. I don’t know if I have a bad card or if it’s simply incompatible with both SATA chips in the system, but the BIOS completely ignores it if its plugged into the motherboard, and Solaris fails to talk to it on either bus. If it’s plugged into the MB, then I get a
device failed initialization error; if it’s plugged into the PCI-X SATA card then I get device on port 5 still busy after reset. I’ve swapped cables and RAM. I’d really like to get it to work, so I’m going to try it with an older system before RMAing it.
- Actually getting a working Solaris install took me 3 tries. The first time I installed it to the wrong drive (the first disk on the PCI-X card, not the first disk on the motherboard), and it was unable to mount the root partition after rebooting. Next, I managed to install it onto an EFI partition, and that wouldn’t boot either. Finally, I installed it onto the second drive on the right bus, and that worked.
- Since Solaris’s installer doesn’t support ZFS yet, I had to manually copy the root filesystem onto a newly created ZFS filesystem mirrored across a pair of drives. The directions are helpful, but I kept screwing things up. First, I accidentally created the ZFS pool using the entire disk, which made ZFS re-label the drives with EFI, which makes them unbootable. Then, I missed the line in the directions that says to run
format -e instead of using format; that left me with a pair of nicely partitioned drives that still used EFI. The third try worked, and the system is now booting off of ZFS via GRUB without problems. Er…
- Well, one problem–I can’t change the GRUB
menu.lst file for some reason. I don’t know where GRUB is looking for it, but it’s not in /boot/grub/menu.lst on my boot array. My changes are being completely ignored. I can live with this for the weekend.
- OpenSolaris doesn’t ship with drivers for the ASUS P5K WS’s onboard Ethernet chips. I had to grab them from Marvell, but it was easy enough to install.
- Creating an 8-drive ZFS filesystem is trivial. One command takes care of RAID, logical volume management, creates the filesystem, and mounts it:
zpool create -f space raidz2 c0t1d0 c0t2d0 c0t3d0 c0t4d0 c3t2d0 c3t3d0 c3t4d0 c3t5d0.
- ZFS performance is decent. Here’s a bonnie with and without ZFS compression, using 10 GB of data on a box with 2 GB of RAM:
-------Sequential Output-------- ---Sequential Input-- --Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine GB M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU /sec %CPU
zfs 10 105.0 55.3 163.3 27.3 121.0 30.4 119.2 88.4 287.1 36.2 169 1.8
zfs+c 10 112.9 59.7 181.5 30.3 127.8 29.1 118.1 86.0 424.9 52.2 198 2.1
- 163 MB/sec writing and 287 MB/sec reading is good enough for me. I was expecting slightly higher numbers, but there’s nothing here to complain about. Adding compression improves writing a bit and makes a big difference reading. It’s quite a bit faster then GigE, which was my goal.
Posted by Scott Laird 205 days ago
A few days ago, I mentioned that my home NAS box had failed, and that I was considering replacing it with a PC server running OpenSolaris and ZFS. I’ve read a pile of ZFS docs, and it looks like the best option available to me today, so I decided to order some suitable hardware.
At that point, pretty much everything broke down. I have a hard enough time keeping track of which hardware works with Linux this week, and OpenSolaris is completely new to me. Sun’s list of officially-supported hardware is pretty sparse, and digging through their mailing list archives gets frustrating quickly. From what I can tell, it boils down to:
- Current Intel and AMD CPUs are all fine.
- Most of Intel’s chipsets are fine.
- Most of nVidia’s AMD chipsets are fine.
- nVidia and Intel video chips are good.
- Most common Ethernet chipsets are either supported natively or have drivers available.
- The only SATA controllers that work are Intel’s ICH southbridges, Silicon Image’s PCI and PCI-E chips, Marvell’s PCI chips, and nVidia’s southbridges. It’s not clear that Marvell’s PCI-E chips work. Most motherboards with additional, non-southbridge SATA ports probably won’t work.
- Venturing too far outside of this list will probably result in problems.
I was looking for a motherboard with 8 SATA ports, and was hoping that the Intel D975XBX2 (“Bad Axe 2”) would work, but 4 of its 8 SATA ports belong to a Marvell PCI-E SATA chip that doesn’t appear to be supported. I went through every single 8-port motherboard in Newegg’s (the ‘WS’ is important–the P5K is a different board). It only has 6 on-board SATA ports, but it includes a PCI-X slot. That’ll let me use the Supermicro AOC-SAT2-MV8, which is far and away the cheapest 8-port SATA card on the market. That’ll give me a total of 14 SATA ports, which should be enough for a whatever I want to throw at it. The Marvell PCI-X chip at the heart of the Supermicro card is the same one used in Sun’s Sun Fire x4500 48-drive server, so it’s safe to assume that Sun has put a lot of effort into the driver.
Most of the test of the system is fairly generic–a cheap nVidia 7200GS video card (the cheapest PCI-E card that NewEgg carries), a nice case and power supply, RAM, and a boatload of drives.
The one odd component that I’ve added is a Gigabyte GC-RAMDISK with 1 GB of RAM. The GC-RAMDISK is a battery-backed SATA ramdisk; it looks like a hard drive to the system and can survive up to 18 hours without power. I’ve had my eye on this thing for years, and it looks like it’ll be a perfect external log device for GFS. I had to ask to see how ZFS will behave if the device fails, and it looks like manual intervention may be required after an 18+ hour power outage, but it should be pretty minimal. I’m planning on posting some benchmarks here once I’ve had a chance to try it out.
Assuming that I’m able to get this whole mess to work at all, I should have lots to write about here over the next week or so. I’m going to start by explaining why I want to use Solaris instead of Linux or *BSD, and why I’m building something instead of buying a pre-build NAS box.
Posted by Scott Laird 205 days ago
So, as part of my new home server series, I want to explain why I’m using OpenSolaris instead of Linux.
I’ve used Linux since 0.97.1, in August of 1992. I’ve had at least one Linux box at home continuously since 1993 or so. I’ve had a few small chunks of my code added to the kernel over the years. I’ve built several install disks and one embedded appliance distro from scratch, starting with a kernel and busybox and going on up from there. I’ve written X drivers, camera drivers, and drivers for embedded devices on the motherboard. I’ve managed Great Heaping Big Gobs of Hardware at various jobs. Basically, I know Linux well, and I’ve used it for almost half of my life.
That in itself might mean that it’s time for a change–professionally, I’ve been very tightly focused on Linux, and diversity is a good thing. But that’s not why I’m using Solaris this week. I’m using it because I’m fed up with losing data to weird RAID issues with Linux, and I believe that OpenSolaris with ZFS will be substantially more reliable long-term. Things I’m specifically fed up with:
- md (the Linux RAID driver)’s response to any sort of drive error, even a transient timeout, is to kick the drive from the array, no matter what. Most of the IDE drives that I’ve had over the years have been prone to random timeouts every few months, at least once you bundle more then 2 or 3 of them in a single box and then try snaking massive ribbon cable through the case. My SATA experiences haven’t been substantially better. Linux will happily bump an otherwise working 4-drive RAID 5 array to a 3-drive degraded RAID 5 array on the first failure, and then on to a 2-drive failed array on the second failure. Even when a simple retry would have cleared both errors. This has cost me data repeatedly, because I’ve been forced to manually intervene and re-add “failed” disks to RAID arrays. If I was too slow, then a second drive failure risked total data loss. Even worse, these random transient failures blind you to real drive failures, like the one that ate my NAS box last weekend.
- Actual drive failures can hang the kernel. I’ve had at least 3 cases at home where broken drives either caused system lockups or completely kept the system from booting. That sucks. Odds are some drivers are good while others are broken; apparently I’ve just had bad luck.
- None of Linux’s filesystems are particularly resilient in the face of on-disk data corruption. Compare with ZFS, which checksums everything that it reads or writes.
In short: everything works great when things are perfect, but building a reliable multi-drive storage system requires careful component and kernel compatibility work, and then you have to stay right on top of things if you want everything to keep working. When things stop working, they usually fail badly. That’s almost the complete antithesis of what I want for home: plug it in, and it just keeps working. I don’t want small failures to cascade through the system. Little failures should isolated, identified, and automatically repaired whenever possible. OpenSolaris and ZFS seems to provide that, while Linux with md and ext3 does not.
That’s why I’m planning on using ZFS. My logic for building a server vs. buying another little NAS box is simple: none of the little NAS boxes on the market use ZFS right now, and none of the cheap ones have room for more then 5 drives. I’m planning on using a double-parity system (RAID 6 or ZFS’s raidz2, where the system can cope with a 2-drive failure) plus a spare drive, and that’d only leave me with 2 data disks. The only way that I can get enough data with only 2 disks would be to use 1TB drives, and they’re too pricy right now.
So, I’m willing to spend the time to build a somewhat complex server because I believe (hope?) that it’ll save me time in the future, and it’ll let me avoid ever having to do the reconstruct-from-the-source dance again. I don’t think I lost anything critical last weekend, and I’m reasonably confident that I’ll be able to get things limping along well enough to recover data anyway, but I’ve now done this 3 times in the past 4 years, and I’ve had it.
Coming up soon: backups, OpenSolaris hardware compatibility, and GC-RAMDISK performance benchamarks. Stay tuned :-).
Posted by Scott Laird 209 days ago
So, the comments on yesterday’s post about my nasty RAID failure encouraged me to spend some time looking at ZFS on OpenSolaris, and I really like what I see. I’ve ordered some new hardware, so I should have lots to write about by next weekend.
Reading the ZFS docs reminded me of my Holy Grail of Storage: a storage system that could actually do reasonably smart things with 3–5 drives. Imagine a system where you could start with 3 drives and simply plug new drives in as you need more space, without worrying about RAID or data layout. When you run out of slots, then just unplug the oldest, smallest drive and plug in a new, larger one, and the data will resync, giving you more disk space without needing any special work on your part. For bonus points, you’d be able to designate specific bits of your data as more or less important, so Bittorrent files might not be replicated at all, while your Word documents might be replicated onto every available drive.
I’ve wanted that for years, but I’ve largely dismissed it as a pipe dream, because it doesn’t fit cleanly into the drive/RAID/LVM/filesystem model that everything uses. The only thing that I’ve seen that even comes close is Drobo, and it’s supposedly fairly slow and really just too “magic” for me to trust.
I realized this morning that it’d be easy to build a storage system like this using ZFS. Just create a zpool with 3 drives to start, and then create zfs filesystems with copies=2 on top of it. When you add new drives, just add them to the pool. Blindly removing a single old drive will only leave you with a single copy of some of your files, but that shouldn’t be fatal, and ZFS can copy everything off of it if you give it a chance. There are some corner cases that will give you less redundancy–if you manage to fill the system 98% full before adding a new drive, then all of the replicas of new data will probably end up on the same disk. There are a couple obvious workarounds, and Sun will probably add replication rebalancing at some point, if it isn’t there already.
Posted by Scott Laird 210 days ago
What’s worse then the sound of one hard drive going “click, click?” Why, two drives going “click, click, click” in the same RAID 5 array, of course.
I’m not very happy with my little Infrant NAS box right now. I think I’ve had it with RAID 5–if I’m going to pile my life onto a disk array, then I really want something that can survive a 2-drive failure without croaking, and that’s basically impossible in a 4-drive enclosure.
I’m seriously considering replacing the Infrant with an OpenSolaris box running ZFS over RAID-Z2 with 6–10 drives; that should live through 2-drive failures, right? Anyone feel the need to talk me out of it?
Posted by Scott Laird 222 days ago
Today’s the big release day for Charlie Stross’s new book, Halting State. I had the good fortune to pick up a review copy through work last week, since we’re one of the official stops on his tour schedule (w00t!).
Short review: go buy it. You can thank me later.
Slightly longer review:
Stross is my favorite science fiction (-ish) author right now, and has been for a few years. He’s written a number of amazing short stories, but his novels have been a bit hit-or-miss. The Atrocity Archives is deeply awesome, as is Accelerando (which is basically a collection of 9 of his short stories), but Singularity Sky didn’t really work for me.
Halting State, on the other hand, is the best science fiction novel that I’ve read in years; perhaps since The Atrocity Archives. Like most of his work, Halting State is less scifi then “geekfi”–it’s about people and technology extrapolated slightly into the future.
In this case, it’s set in 2014-ish Scotland, newly independent from England. It all started with a police call. The caller, panicked, reported a theft. Something about a bank robbery, which made the police operator sit up and take notice, until the caller started blathering about orcs and a dragon. They didn’t send anyone out to investigate until the second call, which still had orcs and the dragon, but sounded more insistent. The detective follows the address provided and ends up in the middle of nowhere, at a former nuclear command bunker surrounded by expensive cars, to discover that someone did rob a bank. Not a brick-and-mortar bank, though–this bank’s is inside of a MMORPG. The total take? About 120 million Euros, once you count the ebay-able value of the missing goods and the hit to the newly-IPOed “in-game economic stabilization” company. The dragon blew the doors off of the bank and the orcs grabbed the loot. They escaped into a portal to another MMORPG where the bank’s managers don’t have root access. The cop’s stumped–how do you investigate this? You can’t dust for dragon prints. So she logs back into CopSpace to see what she can find…
By the time Stross is done, he’s wrapped up gaming, gamers, virtual reality, cryptographic security, in-game economics, the nanny state run amok, mobile phones, and something much more sinister then any of the characters expected to find.
Just go read it, and thank me afterwards. Or thank Stross, that’s probably more appropriate.
Posted by Scott Laird 267 days ago
It looks like Canon’s finally about to announce the long-rumored 40d and 1Ds mk III. They’ve been rumored forever, but they now have product pages at Amazon, so they’re probably legitimate this time:
The 40d looks fantastic–they increased the resolution slightly to 10 MP, which isn’t all that exciting, but they also raised the frame rate to 6.5 fps, bumped the buffer to 75 JPEG/17 RAW images, added their anti-dust shaker, sRAW, 14-bit DACs, weather shielding, and live view. They also completely re-did the AF unit–it still uses the same 9-point format as the 30d, but all nine sensors are now cross-type sensors up to f/5.6; this should make a huge difference in AF accuracy. They don’t appear to have improved the camera’s ISO range; it’s still 100-1600 + 3200, but other then that they seem to have picked up almost every other improvement that the 1D mk III added. I’m impressed; in the past Canon left features out of their mid-range series to avoid competing with higher-end models, but that doesn’t seem to have happened this time.
There’s also a new wireless module, the WFT-E3A, but Amazon doesn’t have any details yet. Presumably it’s a very slight change from the 1D mk III’s new wireless transmitter, hopefully at a lower price point. It doesn’t make a lot of sense to pay $999 for a wireless transmitter for a $1,299 camera. I could actually see myself picking up a 40d in addition to my 5d–it looks like a great camera for taking sports pictures, and the 1.6x multiplier would make my 100-400 lens a lot more useful for bird pics.
The 1Ds mk III is a bit out of my price range at $7,999. It’s up to 21.1 MP now, at 5 fps. It looks like they added everything from the 1D mk III, plus a new UDMA CF module, doubling write speeds with Sandisk Extreme IV and the equivalent Lexar cards. Amazon doesn’t mention ISO sensitivity at all, so it’s not clear if they now support ISO 6400 or not.
Finally, they updated their 14mm lens for better compatibility with the 1Ds mk III. Yawn.
All in all, this looks pretty good. Way to go Canon.
Posted by Scott Laird 270 days ago
I mentioned last week that I’ve been working on building sort of a lightweight home security system for my house so my 4-year-old daughter won’t be able to sneak out of the house again.
I spent about a day researching possible solutions, to see if there was something simple that I could buy that would make me happy, and I couldn’t find anything on the market that was cheap and would tell me which of my 6 exterior doors had been opened loudly enough to hear from across the house. There’s no point in putting a door buzzer on the basement door if you can’t hear it from the master bedroom. The only solutions that I found were from professional alarm companies, and they probably would have charged me a couple thousand for installation plus $30-$50 per month for monitoring. I’m just not willing to pay that much, and it’s not really what I was looking for–it’s gross overkill for my problem.
So, I decided to build it myself. After a few hours’ searching, I decided to use Perceptive Automation’s Indigo home-automation software for the Mac. It’s commercial software, but I like its user interface and capabilities better then any of the open-source solutions that I’ve seen. It’s under active development, supports just about everything that I need, it’s client-server so I can access it from any Mac I own, and it comes with a nice web interface that I can get to via my phone. I figured that it’d be cheaper to pay the money for Indigo then to spend most of a week hacking away at one of the open-source packages to get them to do what I want. Plus, er, pretty much every open source home automation program that I could find was written in Perl, and I’ve been successfully avoiding Perl for almost 5 years now. The last thing I really want to do is spend a week modifying someone else’s Perl. That kind of thing gives me nightmares.
So I fired Indigo up on an old Powerbook that I had laying around and bought a bunch of hardware from MacHomeStore and SmartHome.com. The important bits are a W800RF32A wireless reciever and a whole bunch of $12 DS10a door and window sensors. The sensors broadcast their state over the air, and the receiver feeds them into my Mac.
That took care of the input side of the equation, but I still needed some relatively cheap way to play audio around my house. I poked around for a while looking at random X10 hardware and radio solutions before I realized that most VoIP phones support audio paging. Since my house is full of VoIP phones and I run my own Asterisk server, all I’d really need to do was write a couple dozen lines of Asterisk dialplan code, and everything should just work. I ended up ordering two new VoIP phones (Grandstream GXP-2020s) because two of the phones that I have are too old to be usable for paging. I probably could have used $45 Budgetones, but I have future plans for the GXP-2020’s big displays, so I decided to spend a bit more for them.
It took me about 30 minutes to unpack everything, install Indigo, and have it receiving data from a test sensor. Each sensor assigns itself a random 8-bit ID when it’s powered up, so the first problem was mapping semi-random sensor IDs onto logical names. Indigo comes with a blob of sample AppleScript for doing this, and it only took me 5 minutes to modify it so that one of Indigo’s internal variables changed state to reflect the state of the door sensor–true when the door is closed and false when opened. Five more minutes and I had a nifty web page with a green blob that turned red when the door opened. A half-hour after that, I had a PNG floorplan of my house that I could use as a backdrop for a bunch of little red/green blobs:
After this, it’s all just a matter of plumbing. First, I modified Indigo’s sensor-handling AppleScript example into something that knows how to talk to Asterisk, using one of the Asterisk/AppleScript integration examples on the voip-info.org Wiki. Here’s I ended up with:
(*
Door sensor tracking code for Indigo and Asterisk.
By Scott Laird
http://scottstuff.net
This code handles incoming security events, maps each numeric
device ID onto a logical name, and then tells Asterisk to page all
VoIP phones with a device-specific message.
Net effect: opening the front door causes "Front door opened" to echo
throughout the house.
I'm calling doDialOut directly rather then using an Indigo trigger for 4
reasons:
1. I have the extension name handy here, while I'd have to parse it back out of the variable name if I used triggers.
2. Creating an identical trigger for each of 10+ sensors is a pain in the neck.
3. We need to do *something* with unknown sensor events, but creating variables for them is pretty clearly wrong.
4. My AppleScript is lousy, and I can't get triggers to call doDialOut correctly.
*)
using terms from application "IndigoServer"
on receive security event of eventType with code devID
set extension to "unknown"
-- Map sensor IDs onto logical names. In a real language,
-- I'd use some sort of hash and skip the if ... else if ... code,
-- but I don't see anything suitable in AppleScript. Sigh. I'm
-- stuck writing code in blub.
if devID is 95 then
set extension to "frontdoor"
else if devID is 175 then
set extension to "deckdoor"
else if devID is 99 then
set extension to "stairs"
else if devID is 201 then
set extension to "backyard"
else if devID is 55 then
set extension to "upgarage"
else if devID is 155 then
set extension to "downgarage"
else if devID is 253 then
set extension to "slider"
end if
-- If we get a request for an unknown devID, then send it on to Asterisk
-- so we get an audible indication that *something* happened. Since batteries
-- falling out of DS10 modules can cause the ID to change, I'd rather not ignore
-- these. YMMV, however.
if extension is "unknown" then
my doDialOut(devID)
else
-- Indigo variable names are derived from Asterisk extension names.
-- In retrospect, doorClosed_ is a great name for doors, but not so hot
-- for doorbells or motion sensors. Feel free to change this.
set var to ("doorClosed_" & extension)
set seen_var to ("lastSeen_" & extension)
set change_var to ("lastChanged_" & extension)
set timestamp to (current date) as string
-- Figure out if it opened or closed. There are actually 4 different
-- results that the sensors can return (normal/active X min/max),
-- but we only care about normal/active.
if eventType is sec_SensorNormal_min then
set val to "true"
else if eventType is sec_SensorNormal_max then
set val to "true"
else
set val to "false"
end if
-- Create the Indigo variables if they don't already exist.
if not (variable var exists) then
make new variable with properties {name:var, value:val}
end if
if not (variable seen_var exists) then
make new variable with properties {name:seen_var, value:timestamp}
end if
if not (variable change_var exists) then
make new variable with properties {name:change_var, value:timestamp}
end if
-- Did the value of this variable just change?
if not (value of variable var is val) then
set value of variable change_var to timestamp
-- If it just changed and it's now false, then send an alert.
-- The DS10a sensors send a signal once per hour, even if nothing's changed.
-- So we don't want to alert *unless* something's changed.
if val is "false" then
my doDialOut(extension)
end if
end if
-- We need to keep track of the time that each DS10 was last seen.
-- This way we can spot bad batteries.
set value of variable seen_var to timestamp
set value of variable var to val
end if
end receive security event
-- doDialOut tells Asterisk to dial a specific extension using
-- Asterisk's management interface. This doesn't require that Asterisk
-- runs on the same machine as Indigo.
--
-- To get this to work in your environment, you'll need to change the IP
-- address, username, and password, and possibly the contexts and/or
-- extension names used at the bottom.
on doDialOut(extension)
log "Paging extension " & extension using type "Dialer"
set expectscript to "set timeout 20;
spawn telnet 10.0.0.1 5038;
expect \"Asterisk Call Manager/1.0\";
send \"Action: login
username: my_username
secret: secretsecret
\";
expect \"Message: Authentication accepted\";
send \"Action: originate
Exten: " & extension & "
Context: security-paging
Channel: Local/all@paging
Priority: 1
Callerid: Security: " & extension & " <0>
\";
sleep 1;
send \"Action: logoff
\";
"
set results to do shell script "/usr/bin/expect -c '" & expectscript & "'"
end doDialOut
end using terms from
In addition to simply monitoring the current state of the sensors, this also tracks their last change as well as the last time each sensor sent out a “nothing’s changed” report. Eventually I’ll add monitoring for this so I can spot failing batteries immediately, and not two months later when I discover that two doors aren’t monitored anymore. Indigo tracks each variable internally, and gives you a handy window for viewing and modifying them:
Once all of that was in place, it was time to add paging logic to Asterisk. I created two new contexts, security-paging to contain the messages that need to be played back and paging to handle the phones. The AppleScript above basically glues the two ends together, sending ‘deckdoor@security-paging’ to ‘all@paging’. Here’s the relevant bit of my Asterisk config:
[security-paging]
exten => frontdoor,1,Playback(security/frontdoor)
exten => deckdoor,1,Playback(security/deckdoor)
exten => backyard,1,Playback(security/backyard)
exten => stairs,1,Playback(security/stairs)
exten => upgarage,1,Playback(security/upgarage)
exten => downgarage,1,Playback(security/downgarage)
exten => slider,1,Playback(security/slider)
exten => _.,1,SayNumber(${EXTEN})
[paging]
exten => all,1,Page(Local/203@paging&Local/204@paging&Local/205@paging&Local/206@paging)
; Cisco 7940; no Call-Info support, but you can create a new line and turn on auto-answer on the phone.
exten => 203,1,Dial(SIP/203aa)
; Sipura 841. It needs a semicolon before answer-after.
exten => 204,1,SipAddHeader(Call-Info: \;answer-after=0)
exten => 204,n,Dial(SIP/204)
; Grandstream GXP-2020s, although the same config will work for most modern phones.
exten => 205,1,SipAddHeader(Call-Info: answer-after=0)
exten => 205,n,Dial(SIP/205)
exten => 206,1,SipAddHeader(Call-Info: answer-after=0)
exten => 206,n,Dial(SIP/206)
Then I just had to record some audio files to use for annoucements. The easiest way to do this is just to call yourself up and leave voicemail, and then copy the VM files over into Asterisk’s sound file directory, usually /var/lib/asterisk/sounds. You could get better quality recordings with a good microphone and audio-processing app, but I’m not sure that there’s really a point, given the quality of most speakerphone speakers.
All told, it took me about 8 hours of research to put this all together, and maybe 10 hours to implement it all, including learning a bit of AppleScript. Now I have a programmable system for monitoring my house, and all of the pieces in place for adding X10 or Insteon components as they make sense. It’s all under my control; if I can code it, then I can make it happen.
So, does it all work? Yep–Monday morning it caught my daughter trying to sneak into the garage. Mission accomplished.
