Steve: Developing on the Edge

Y!'s Hadoop job best practises

Sat, 21 Aug 2010 14:11:00 GMT

Arun Murthy has put up Yahoo!'s recommended Hadoop best practises.

These are good as they show what things are bad -generally anything that bothers the namenode too much, or any work where the input or intermediate files aren't that big. Small jobs are the enemy. Presumably Arun's team are monitoring stat's and identifying the troublemakers. What you could do there is just recognise these "inefficient" jobs and schedule them differently; allowing them, but penalising the caller.

How to win friends and influence people: oracle sues google

Fri, 13 Aug 2010 09:58:00 GMT

The first thing I see on the Planet Apache feed is the news that Oracle are suing Google over Dalvik and, implictly, Apache Harmony

This is really going to cause fun in the open source world. I can see a fork coming on. Can you trust Oracle not to add features to Java just because they have patents on it?

Google Android must be wonderful and terrible for the Java team. Wonderful, because it finally supported the Java language on mobile phones on a platform that was actually good, finally built an application ecosystem. This was the reason why when Sun open sourced Java, they made sure the mobile version was GPL, so anyone embedding it in a phone would be motivated to pay $ for the real thing. Even if it was $0.15 a phone, at scale, that becomes profit. Which is why Sun wouldn't give Apache the test kit for the Harmony Runtime unless the ASF said "not to be used in mobile/embedded devices" (as if they could), and it's hence why Java 7 development has stalled (trouble in the Java oversight group). What development does go on uses the term "JDK 7" to get around the stalemate. Did all that posturing help Sun get Java into phones? Nope.

Android ships with Dalvik, which is not quite Java, according to Stefano, and I believe him.

However, there are patent issues, and had Sun given the ASF access to the TCK -as I believe they were obliged to do- those would have gone away, as compliant implementations get rights to those patents when they pass the tests. But by keeping the TCK, Harmony couldn't pass the tests, could it?

The other funny is that if you look at the complaint, you see at the top the names "BOIES, SCHILLER & FLEXNER". If I am not mistaken, they are the lawyers behind SCO's copyright lawsuit against IBM and Novell about Linux being based on Unix. Well, that was successful wasn't it? I consider, as I type this on my home laptop running Ubuntu 10.4 as every physical computer I use at work also does, from the desktop to the datacentre. It was such a good idea it created a whole web site Groklaw.

Looking forward to this. It could be entertaining.

Patch week

Wed, 11 Aug 2010 09:13:00 GMT

SANS is reporting that alongside this Month's MS patches, Adobe has patches for Flash and other things out. Not acroread, fortunately, but for anyone who doesn't use flashblock to turn flash off, their browser only needs to go to one subverted site for their machine to belong to someone else

Sigh

This is one of the big problems with Virtualisation. Any VM image you don't keep up to date is a security risk, cost of keeping up to date is therefore proportional to the #of VM images you have. And any backed-up VM from months earlier is danger.

SANS also have a good article on Keeping SSH keys secure, including some find commands to locate keys in tar files and temp dirs. Better not forget those VM disks too though...

Ubuntu 10.04 upgrades

Sat, 07 Aug 2010 17:02:00 GMT

ubuntu 10.04 upgrades

I am currently upgrading the last machine in the portfolio to Ubuntu 10.04 from Ubuntu 9.04, that being the last version I liked. The only ways to do this upgrade is a complete reinstall of the root disk (probably the cleanest) or an upgrade to ubuntu 9.10 then to version 10.04. If the latter is done (as I do), copy ~/.mozilla somewhere first to stop intermediate firefox upgrades making a mess of things.

Also: expect all thunderbird filters to get lost. This is inconvenient if you use thunderbird as an IMAP client to a large server which keeps many years of ASF email lists and other mail to hand.

Recommendations

Make a note of the filter settings
Turn off the indexing if you have a big server full of data
The junk mail data gets lost too, so keep an eye on the junk folder
If you have >1 account, only one seems to be retained in the upgrade. Consider noting down all the details of all accounts

Other than that, the upgrades have gone fairly well. Sometimes during the install process it stops asking you if some config file you've never heard of should be overwritten, and you have to look at the changes in a file you've never seen or whose meaning you understand. This is not a good end-user experience. And the work desktop sound system stops working on hibernates, so it's a good thing alsa force-restart exists to fix that.

The one thing I don't trust, yet, is the ext4 filesystem. Only one machine -a home laptop with a new HDD- got built up with ext4 on the clean install, and after less than two months that laptop no longer boots after the machine powered off unexpectedly. Filesystems should not do that. I have no further data/experience on ext4 reliability, but it does worry me. I'm curious if anyone is using ext4 at scale in their datacentres, and if so, how reliable its been. Of course, if your datacentre has a power system which never fails, these problems may not show up. At least, not at first.

Gelato

Sat, 07 Aug 2010 14:51:00 GMT

First, the meal

Then, time to walk

It's night, but the buildings are lit up

Everyone else is out and about too

And what is available: gelato. Ice cream.

Lovely

Organizational Announcement

Sat, 07 Aug 2010 14:34:00 GMT

My 7:30am doze through the clock radio was interrupted by the announcement that our CEO is no more; he has resigned. Oddly enough, this year's Standards of Business Conduct video/questionnaire did include some q's on the topic of sexual harassment.

Not one media outlet has asked me for a quote, which is probably because they suspect I have little of interest to say on the matter. I stick to things like Ubuntu versions, ext4 filesystem stability and similar stuff.

Italy

Sat, 07 Aug 2010 14:29:00 GMT

I've uploaded our Italy photos.

We stayed in Puglia, the bottom S.E corner of Italy; south of Naples, East of Sicily, but without such "reputations".

The house was an old Trulli thing with these traditional roofs. Apparently this is very quaint, but when you are staying there the main thing is the inside is very cool when you are having your mid afternoon siesta.

During the morning/day we'd normally be lazy, the sprog would play in the pool and demand one more parent in there to entertain him, which was about the hardest chore of the day.

Evenings were the time to come out, head into town, enjoy Italian towns by night.

Where we were staying that meant a drive in, but it was so provincial it wasn't that bad, provided you bear in mind that the who-gives-way rules in cities are fairly ambiguous.

At night, the towns were busy with everyone walking around, enjoying themselves, being seen. The sprog learned an important new italian word.

Gelato

back from holiday

Thu, 29 Jul 2010 17:23:00 GMT

I've been offline, away in Italy. Good fun, though being utterly offline is odd. Catching up on emails. As my colleagues have pointed out: never go away leaving the build broken, as it generates a lot of mail in your inbox.