Self-Signed SSL Certificates

So you'd think there are enough blog posts about this already. Well, think again, since this particular summary is what actually ended up working for me. Obviously your mileage may vary.

1. openssl genrsa 1024 > host.key
2. openssl req -new -key host.key -out host.csr
3. openssl x509 -req -days 730 -in host.csr -signkey host.key -out host.crt

Of course that's what everybody has, so why write about this? Three reasons:

• Make sure you chmod 400 host.key since you don't want anybody to see that.
• Using lighttpd? Do a cat host.key host.crt > host.pem and chmod 400 that as well.
• The "Common Name" you have to enter in step 2. If you have various subdomains like www.example.com and mail.example.com and so on, you don't want to enter "example.com" here. Instead you'd enter something globtastic like "*.example.com". But wait, that doesn't match just plain example.com anymore! Better use "*example.com" and wow, that actually works.

An Internet. Wow! It's so pretty... Who would've thunk? :-D

Program like it's 1975?

Looking through various proxies and caches (don't ask why) I ran across Varnish and was struck by a short piece one of their developers wrote in 2006. Let me quote a line or two (emphasis mine):

Take Squid for instance, a 1975 program if I ever saw one: You tell it how much RAM it can use and how much disk it can use. It will then spend inordinate amounts of time keeping track of what HTTP objects are in RAM and which are on disk and it will move them forth and back depending on traffic patterns. Well, today computers really only have one kind of storage, and it is usually some sort of disk, the operating system and the virtual memory management hardware has converted the RAM to a cache for the disk storage.

You should read the whole thing, it's a nice summary of technological changes that still haven't made it into everyone's head. I feel guilty myself: I recently taught our OS course, but I don't think I ever made this sufficiently clear when we talked about virtual memory. I'll try to add a relevant assignment to the Unix course next semester... :-D

Pocco: Another Take on Documentation

I just discovered a cute tool called pocco on github. Apparently pocco is the Python version of what is also known as docco, rocco, and (gasp) shocco out there. Quite the zoo really! :-D So what does it do? It parses Python source and separates comments from code. It then dumps everything into HTML with a little CSS, making the comments appear on the left lined up with the corresponding code on the right. Neato.

Or it is? What could this possibly be useful for? Comments in Python are not usually used for documentation: we have docstrings after all! But docstrings are not perfect, for example you can only (by "natural" means anyway) attach a docstring to a module or class or function: What about those globals over there? Do I really have to document them in the module docstring?

But I realized that something like pocco is actually useful in Python. After all, the documentation you put into a docstring should be concise and clean: it should say exactly what it needs to say for someone else to use your stuff, nothing else. Boring! What about all those humorous anecdotes you accumulate while you hack? What about false starts, things that others should be told about before they head down the same wrong way. And so on, and so forth. With pocco, you can use comments for commentary and docstrings for documentation yet extract and (nicely?) format both. How cool is that? :-D Let's just hope that pocco will leave docstrings alone for good...

The other use I could come up with is code handouts in programming courses. Often the comments I write in this context have more of a "let me take you by the hand and work this through with you" feeling. Reading them in a nicely formatted and syntax-highlighted way may be nicer for everyone involved. True, since all comments are extracted it's hard to "comment on comments" that way... But that's okay, at least for me. Now all I need for this second application is Java, C, and C++ support! :-D

Know one editor well...

For years I've been telling people new to Unix that it doesn't matter what editor they pick. "Just pick one and learn it well!" I still believe that's true, but I've recently come to realize that I may not have followed my own advice in this regard.

I've been using vi (well vim) for years, yet apparently I have not even come close to exploiting it like I should have. Here are a few links to get you started. Take a look even if you think you know vim well, you may be surprised what it has to offer these days.

Efficient Editing with vim
Automatic Word Completion
vimsh
Indenting Python with vim
Python and vim: Make your own IDE
Vim as Development Environment
Code Navigation and Completion
Vim as Python IDE

Obviously I am focused on Python here because that's what I hack the most for my personal projects, but much of the information is applicable regardless of what language (programming or otherwise) you have to work with every day. I still can't believe that I missed out on many of these options for years. Don't make the same mistake! :-D

su, sudo, and /dev/pts/*

I have a few separate accounts even on my own local machine just to organize things slightly differently for different projects I am working on. Let's call my standard account "s" and one of the other accounts "q". When my machine starts up, I log in as "s", then when I need to work as "q" I use su - q to switch. Seemed like a good idea at the time.

The problem is that /dev/pts/* is owned by "s" since apparently it gets created when "s" first logs in. Now as "q" I can't use things like /dev/stdin or run screen because the permissions on /dev/pts/* are not "permissive" enough. I could probably find a way to open up the permissions, but that doesn't seem quite right. What I really want is to have "q" create it's own /dev/pts/* owned by "q".

For the longest time I couldn't find a way to get this done. Then yesterday it occurred to me to try sudo login instead of su - q and that actually works! You can't do login by itself, it'll complain, but sudo login seems fine. So my problem is apparently fixed, although now I do wonder if there are any security problems with my new approach to switching users. Anyone?

POSSE Worcester, Day 5

With a small delay, here the summary of Day 5. First we moved to another building since the Science and Technology Center was undergoing some kind of loud demolition. :-( We hacked for a little while and then Walter talked about what to do over the next few days and how to push when we're done and do the merge request. Since I had not done much on the Measure activity but on Physics instead, I wasn't very affected by this.

Then we had our huge teaching chat, which started with Garry demonstrating what he's doing at WPI with his Software Engineering course. After that "semi-formal" part it was just open discussion. Mel wanted to make things more concrete, so she started a transcript on Pirate Pad that we all edited furiously for about 40 minutes or so. Then we went over the transcript with more discussion. And then we were done. :-D

Some people left at this point, but some of us gathered around and went to Karl's favorite Hot Dog place which was quite the treat. I had three Cony Island Dogs (trust me, they are kinda small :-D), a Chilli Cheese burger, and two "dry" Orange sodas. The only drawback was that they didn't have a booth for five, so I was a bit lonely. :-( But on one of my breaks I found a sticker outside that had the picture of a guy on it and said "Pickels, 5' 10", 160 lbs. has a posse" and that made us smile. :-D

I went back to my hotel and made the mistake (or was it one?) of youtubing around. Ended up watching "The Last Lecture" again, wow. So I showed up for the final dinner with Mel and Peter a little teary-eyed, but I don't think they noticed. We went to the Brazilian Steak House I wanted to check out the first night, and guess what? It was absolutely delicious (and none of us "smelled like meat" afterwards, Mihaela :-D)!

In summary, I had a great week at POSSE in Worcester, and I'd like to thank everybody who was there for making it such a very, very cool event. Oh, and Mihaela posted some photos, thank you so much for that too!

POSSE Worcester, Day 4

Day 4 started badly because I couldn't wake up on time. While I got on IRC to tell people not to wait for me, Mel is just too nice and waited anyway. So I held everybody up for about an hour. :-(

We did some more specialized discussions in smaller groups before lunch. Mihaela and I were walked through the details of the translation process as used in the Sugar project by Walter. I had used the web interface to this before to do some translations, but it was good to hear it all in context. (Future students beware: Walter convinced me that I can actually add this to a C course as well, given that gettext is completely pervasive on GNU systems. :-D)

I didn't have too much success with code, so I just did some more translation work before lunch. Lunch was at the WSC cafeteria again, I am still amazed at how good the food is there (for a university outfit). I just wish JHU would get their stuff together in this regard.

After lunch we were supposed to have a few more "split into groups" things but we actually got "distracted" (in a good sense) by Heidi, a guest from another local college. At first we just wanted to do a round of introductions, but this quickly diverged into a long discussion of how to put FOSS to use in the classroom (which is also the topic for the last day I think).

Mel then went into an overview/review of the various pieces of infrastructure for FOSS projects. I mentioned doodle.com but I held back on recommending github.com (I still think they have the best "basic" setup right now, but I might be wrong). I think I'll start using IRC more in the future, Mel pointed out that "being overheard" in a public chat is a Good Thing (tm) and I believe that's true.

Since it was "dinner night" many of us stayed on campus late. I went back to the Physics activity, desperately looking for something to fix in my favorite piece of Sugar. And I found something: The "grab and move" feature only worked with the simulation running, but not when the simulation was paused. Walter agreed that it should work, so I started hacking on that but didn't get far by myself. Luckily Mihaela stopped by and we started working on it together, and after a few hours we actually figured some of the pieces out. I started reading some of the library source code which promptly helped unravel the last mystery, and we got basic grabbing working before dinner.

The restaurant was somewhat hard to find, but it was certainly worth it as it seems to be somewhat of an "institution" in Worcester. The food was great, and we met "Michael" who works at Akamai and hacks on Sugar every now and then, our "special guest" for the night. Good food, good conversations, good beer, it was a fun evening.

After dinner I finished the patch for Physics and sent it off to the maintainer. I wasn't quite happy with it, but at least I had something to contribute. Gary emailed back pretty quickly that they were looking for that feature for a while, which provided additional motivation to make it better. The problem with my solution thus far was that you didn't get to move an object directly. Instead you clicked on it and then dragged a line to the future position you wanted; when you released the mouse button, the object would "pop" into its new place. It was actually quite straightforward (well, except for one synchronization issue :-D) to get a real "direct manipulation" drag to work, so I sent off another patch to Gary an hour later. I went to bed quite happy with myself for a change. :-D

POSSE Worcester, Day 3

It's frustration day. I should have known that things were going too well when I got up this morning. :-D The day was dedicated to hacking on the Measure activity. I was still using Debian on my netbook and had a hard time following Walter's advice of using the special sugar-jhbuild system to setup a working version of the Sugar emulator. So I checked the Debian packages, and sure enough they offered Sugar! I installed it and all it's dependencies only to find my system completely hosed on the next reboot. So that's awkward dear Debian maintainers. :-(

Seeing as I was in a room full of Fedora machines and Redhat people (well, maybe not so many Redhat people :-D), I begged Mel to make me a Fedora 13 install USB stick. I spent all my time before lunch reinstalling from scratch (and losing my ssh-keys in the process since the encrypted file system I was using under Debian couldn't be recovered properly). And guess what Debian? Under Fedora 13 my sound works just fine, even the microphone; I wonder what I'll find when I try out wireless networking later...

Over lunch Walter mentioned looking into making the display for Measure scrolling. So once we got back from eating I looked at the code for this. I actually got scroll bars wrapped around the display just fine, but then it occurred to me that data was coming in way to fast for the scrolling to be very useful without further additions. So I scrapped that and decided to simply look for Python code to refactor so I would achieve something (anything!) at all. That's still pending, but I did a first pass through most of the code and cleaned up indentation problems as well as unused imports and stuff. But it certainly wasn't a very satisfying day.

Until maddog showed up that is. :-D Jon gave a wonderful pitch for FOSS to our audience of teachers. And even though some of his jokes backfired a little, overall I was very happy to be there and hear him talk. I think we'll have beers in a few, so I look forward to that just like every evening, but especially today since I want to ask Jon a few more things, for example about the "proper" ownership model for basic infrastructure like phone lines. :-D

POSSE Worcester, Day 2

Today was all about hacking. First Walter went over the basic commands for git, then we started working on his Abacus activity. Walter suggested adding a new kind of Abacus, but I saw a piece of code that was repeated like 10 times in a row and decided to replace that with a loop instead. Yeah, I don't follow instructions well... :-D But hey, now I left my (very small) stamp on a piece of free software that others actually work on as well, not too shabby!

The lucky thing about the Abacus was that it runs fine outside of Sugar. I like Sugar in principle, but I don't want to hack inside its unusual confines: I want my actual Unix system! The activity we picked for the homework tonight, Measure, sadly doesn't run outside of Sugar. I played with Physics a bit too, but couldn't get that to work well outside of Sugar either. Also there are a few Physics forks, some of which work better in Gnome than others. Sadly mainline doesn't work well at all. :-( So I am not too sure what to work on tonight, but maybe I'll just keep working on Abacus instead. :-D

I had a great experience with IRC today: I actually got help from a complete stranger! We have a "side task" not related to code, and I picked translating some leftover strings from the Measure activity to German. But the registration for translate.sugarlabs.org was borked for a while this afternoon (has been fixed since). Luckily one of the admins was around and approved my account without the confirmation email that never got to my gmail.com inbox. Of course now that they fixed stuff, I finally got the email. :-D

I am looking forward to having dinner again tonight with the other three people who are stranded in Worcester hotels: Kristina, Mihaela, and Peter (yes, another one!). Last night we had some delicious Mexican food, and tonight I believe we're all going to "smell like meat" (to quote Mihaela) when we roll around in Brazilian BBQ. :-D

Update: Actually, we went to a place called Brew City and they had Bavarian beer: Ayinger! I don't usually drink that, but I had to have two bottles even at $8.50 a pop (wow). Very good stuff, and the food was decent too!

POSSE Worcester, Day 1

Just a quick note that we got started with POSSE today. Yay! We all introduced each other and why we're here, then Mel gave an overview of FOSS development and Walter introduced Sugar, the project we'll be working on for the rest of the week.

The exercises came next, so I now have even more accounts to keep track off: bugs.sugarlabs.org, git.sugarlabs.org, wiki.sugarlabs.org, admin.fedoraproject.org, and bugzilla.redhat.com. I even have a user page on the Sugarlabs wiki, courtesy of Karl. :-D

Lunch at Worcester State College was amazing, possibly the best Chilli I've had on any campus anywhere. At least outside of Texas anyway!

On a more serious note, Walter got me thinking about the Python course I am designing for next Fall. Seems that Sugar may actually be a very good environment to start people out in. It seems to offer a path that gradually introduces more Python and de-emphasizes Sugar, at least if I understand it correctly. Not sure if that should be called "Lighter" then? :-D I'll play with Sugar much more this week, so I'll certainly find out whether it's a good fit by Friday.

Python ORMs

There is really no point to this post except to remind me of all the Python ORMs I promised myself I'd look at over the summer. I want to convert my web application from raw SQL(ite) to some ORM, but there are way too many of them. I could roll dice, but that doesn't seem appropriate somehow... :-D

Autumn
Axiom
DejaVu
Elixir
Membrane
Storm
SQLAlchemy
SQLObject
XRecord

And then there are these Gadfly and buzhug and SnakeSQL things, not ORMs of course, but interesting anyway...

Update 2010/06/24: Alright, I've looked at a few of these in more detail now. Seems that Autumn has not been updated in a while, so it's off the list. Also Elixir being a declarative layer over SQLAlchemy seems a little strange now that SQLAlchemy got it's own declarative layer, so it's off the list.

I implemented a basic model layer for my web application using both SQLObject and Storm, so those two I actually sort of grok now. The first major difference is that Storm requires writing some raw SQL to create tables and related schema stuff, whereas SQLObject tries to hide SQL even for those tasks. The second major difference is that Storm separates object creation from persistence whereas SQLObject combines the two to some extent; this can be good or bad depending on what your application needs to do. Also, coming from a "raw SQL" background, both Storm and SQLObject have some "issues" when it comes to formulating complex queries. Nothing much to be done about that I guess, but I still don't like it all that much.

If I had to pick an ORM right now, I'd probably pick SQLObject. But I have a few more to evaluate so stay tuned. :-D

QEMU: The Machine Park Replacement?

Alright, so teaching 600.318/418: Operating Systems last semester forced me to finally take a serious look at QEMU. And guess what? I liked it! :-D (Special thanks to Venkatesh Srinivas for helping me getting used to QEMU!)

Since QEMU supports some "exotic" platforms like PowerPC and SPARC, platforms I love to use for 600.328/428: Compilers and Interpreters, I started a little project to see if I could replace my aging "machine park" with QEMU instances.

The bad news is that installing a standard Linux distro for anything but x86 is somewhat complicated on QEMU, at least I had a very hard time with it. The good news is that I found some very nice Debian images tailored specifically for QEMU. And most of those even work! :-D

A few minutes ago I was finally able to SSH into my first MIPS QEMU instance! Now I'll work on finishing the same setup for ARM and PowerPC. So far it looks like I can replace at least my Cobalt Qube (MIPS) and my iMac (PowerPC) with QEMU instances. SPARC is a bit of a problem child right now, so I'll keep working on my Ultra 60 (I made some good progress there BTW, a later post will have the details).

But why replace my "machine park" in the first place? Granted, it's great fun to keep those old machines running and to hack compiler backends on them. However, it's also a big drain on my time. And what's worse, it's obvious that eventually each of these machines is going to fail beyond repair. In addition it's going to be easier to backup QEMU images every now and then, and having them all hosted on the gaming lab server with its nice RAID-6 makes things a tad more reliable and predictable as well. So while I am not planning on actually getting rid of my old machines just yet, overall QEMU seems to be a much better tradeoff for instructional purposes.

Getting Debian ARM to work with screen: I like to run my QEMU instances in screen, so I want the console output to go to, well, the console. Here is how:

qemu-system-arm -M versatilepb -m 256 -kernel vmlinuz-2.6.26-2-versatile -initrd initrd.img-2.6.26-2-versatile -hda arm.img -append "root=/dev/sda1 console=ttyAMA0" -nographic

The important part is the ttyAMA0 thing, that convinces the Debian ARM kernel to use stdin/stdout for everything. So now I am happy with my virtual ARM box. :-D

Getting Debian PowerPC to work with screen: At first I couldn't get the PowerPC QEMU instance to work with screen at all, so I had to "fake" a display as follows:

qemu-system-ppc -m 256 -hda powerpc.img -no-reboot -vnc :0

This worked fine, but it just wasn't very satisfying. So I poked around a bit more and found a getty process listening on ttyPZ0. Given my experience with the other QEMU instances, I figured this must be where the serial console is. However, since I didn't provide a kernel to QEMU directly, I couldn't append anything to the kernel command line either. So I copied the kernel from the PowerPC Debian image out and tried booting that way, but it wouldn't mount the root partition. So I copied the Debian initrd image out as well, and voila:

qemu-system-ppc -m 256 -hda powerpc.img -no-reboot -nographic -initrd initrd.img-2.6.26-1-powerpc -kernel vmlinux-2.6.26-1-powerpc -append "root=/dev/hda3 console=ttyPZ0"

Now I have a PowerPC QEMU instance that works with screen. Not much more I can ask for at this point, I now have three platforms for the next compilers course and that means my students will finally have to write native ELF backends next time around. Yay! :-D

A Modern Baldur's Gate Install?

I've been playing Baldur's Gate since it first came out in 1998, and I still think it's one of the best computer role-playing games ever made. It may be a little surprising, but in 2010 there's still a very active community around Baldur's Gate and other Infinity Engine games. How many other mainstream computer games can boast of a 12 year development cycle? I can only think of Quake 3 as coming even close...

Getting Baldur's Gate and friends to run on my Linux box wasn't very complicated, mostly because Wine provides pretty excellent support for it out of the box. However, I have not yet tapped into the vast array of modifications and customizations available for these games, something I want to rectify now. The problem? There are too many modifications! Some of those even overlap, so installation order determines the resulting gameplay experience in somewhat non-obvious ways.

The only "basic" decision I've made so far is that I will use the Baldur's Gate Trilogy framework which integrates all of the existing Baldur's Gate titles (Baldur's Gate, Tales of the Sword Coast, Shadows of Amn, Throne of Bhaal) into a single game. There are other ways of getting the more advanced Baldur's Gate 2 engine to play original Baldur's Gate content, but Baldur's Gate Trilogy seems better maintained than BG1Tutu as far as I can tell.

What to install on top of that? That's what I hope to detail in updates to this post. I'll experiment with various modifications and their installation and I'll try to document the effects of those here for fellow Baldur's Gate fanatics. I'll first try to work my way up to the install suggested in Dan Simpson's FAQ, which will probably take a few weeks given my schedule. Stay tuned! :-D

Notes on Wine: I had to update Wine to 1.1.42 to get Shadows of Amn installed. I am not sure which version broke it since an older Wine installed SoA fine before. Also, after being unable to install Throne of Bhaal into a new path due to a previously installed ToB, I decided to start with a fresh Wine directory for all of this. Maybe you want to do the same.

Baldur's Gate and Tales of the Sword Coast: I have the 4 in 1 boxed set, so Baldur's Gate was already patched to 1.1.4315 on install. However, Tales of the Sword Coast was at 1.3.5508 by default, so I patched it to 1.3.5512 before continuing. I also applied the DirectX 8 patch, which may or may not be a good idea, we'll see.

Shadows of Amn and Throne of Bhaal: I installed Shadows of Amn and patched it to 23037, then installed Throne of Bhaal and patched it to 26498 before continuing. I also applied the 26499 beta patch, which once again may or may not be a good idea, we'll see.

Checkpoint: I made a backup copy of the .wine folder at this point, mainly so I won't ever have to sit through the demo movies again (the BG and TotSC installers won't let you break out of those horrible movies). The .tar.gz for this was 4.2 GB. Pretty darn big! :-D

Baldur's Gate Trilogy: The first thing I tried was to install BGT without anything else. That fails bigtime since the installation scripts rely on various Windowsisms (yeah, that's a word!) that are not present in Wine. Luckily someone already worked out the kinks but it makes the installation process a little more involved; I found this article helpful too. So we start by installing dos2unix and mmv (I used emerge in Gentoo). Then grab mospack and compile it using "make -f makefile.unix" in the source directory, then add the source directory to your PATH. Now grab Baldur's Gate Trilogy and bgt_linux.rar and extract them into your Shadows of Amn directory. Now grab the Linux version of WeiDU, extract it, and add the resulting directory to your PATH. Now execute the "tolower" program from WeiDU in both the Baldur's Gate and the Shadows of Amn folders to convert everything to lower case. Note that for the scripts to work you need to remove the spaces and capital letter from your path names to the installation directories as well.

Update 2010/05/22: Bad news. I had to give up on this project because the install scripts simply assume way too much about Windoze to be processed nicely with Wine. So I bit the bullet and installed an ancient Windows 2000 CD I had lying around in VirtualBox instead. Now I can play all my favorite games with all the fun extensions I want. And over the last few weeks, the "guilt" I felt whenever I started VirtualBox also diminished. Of course it's still frustrating that I had to give up. :-(