Andre Lewis on Tech

Highgroove's Scout is live! Scout is our server monitoring and reporting application. If you didn't have the opportunity to test-drive it during the beta period, you can now sign up for any of the plans. Why? Three reasons, based on my own Scout usage over the last few months:

• Scout makes you look smart: if something goes wrong with a client's app, I generally know about it before they call me. So I get to be very proactive. In one case I was able to fix a crashed server before the client even realized anything was wrong.

• Scout helps you build better apps: I had some memory-sucking CSV handling in one of my applications ... but it only happened under certain circumstances in production. With Scout, I was able to correlate the mongrel memory spikes with user activity and zero in on the problem. The result: mongrel instances that don't spike to 200MB, and a better experience for users.

• Scout helps you sleep better: uncertainty is a stressor, and the more apps you have out there the more things you have to be uncertain about. Just knowing that I'll get an email notification if something goes astray helps my peace of mind a lot. Which means my mind is clear to ... design and build more apps!

Ready to learn more? Here's Derek's official launch post. Or cut to the chase and hit the Scout signup page!

Hotspotr -- my community-driven WiFi hotspot site -- announced a content-providing partnership with LightPole today. You can read the press release over at http://lightpole.net/press/index.html.

Hotspotr lists over 8,000 user-contributed WiFi hotspots. LightPole provides a mobile client you can use to browse Hotspotr from your mobile phone using an interactive, maps-based interface:

You can download the LightPole mobile client for free from the Hotspotr homepage.

Separately, Hotspotr is also taking advantage of the Google Maps streetview functionality. For locations where streetview is available, you will have the option to browse the 3-d streetview imagery:

See an example at the Dolores Park Cafe in San Francisco.

Following up on my previous post on low-cost staging environments, I chose a host and set up my staging environment.

Note: these are the steps I took to make it work. They are probably not the optimal steps. Actually, I'll go one step further: I guarantee you these are not the optimal steps. But, I did get my staging environment up and running in less then three hours.

1) signed up for SilverRack.

Why? Dave from SilverRack offers $10 off if you're involved in a Ruby user group, which makes the total cost of a 256MB vps $10/mo. Hard to beat that from a pure cost perspective.

• signup took two minutes including paying my first month on paypal
• the VPS was provisioned about two minutes after that. I had originally chosen centos, but realized I needed Ubutu 7.1 for deprec 2 -- SilverRack has a simple web-based reinstall you can do yourself, and it took a minute. Note: the Rails stack on CENTOS (before I changed to Ubuntu) looked like it was Rails 1.2 based.

2) install new cap

(I hadn't upgraded to 2 yet)

sudo gem install capistrano -v2.1.0

3) install deprec 2 preview

... I had a hard time finding the gem; grab it from the wiki page like so: curl http://www.deprec.org/attachment/wiki/WikiStart/deprec-1.99.13.gem?format=raw > deprec-preview.gem

sudo gem install deprec-preview.gem

4) decided to create myself a little deprec play area, but I'm not sure it's necessary:

cd ~/projects mkdir deprec cd deprec mkdir config depify .

4.5) set up ssh keys

. . . I know deprec provides some tasks for this, but I just did it manually (copied my local ~/.ssh/id_rsa.pub) into my new vps's ~/.ssh/

5) let's setup a rails stack!

Used an IP because my dns hasn't resolved yet. If your dns has resovled ... good for you!

export HOSTS=xxx.xxx.xxx.xxx 

# here comes the big one . . .
cap deprec:rails:install_rails_stack

ok, it almost made it through ... ran for a few minutes and failed on sqlite3. Specifically, it failed here:

executing `deprec:rails:install'
executing `deprec:rails:install_deps'
executing "sudo -p 'sudo password: ' \n      
sh -c \"DEBCONF_TERSE='yes' DEBIAN_PRIORITY='critical' 
DEBIAN_FRONTEND=noninteractive apt-get -qyu --force-yes  
install libmysqlclient15-dev sqlite3 libsqlite3-ruby libsqlite3-dev \""

.. with the error:

Package sqlite3 is not available, but is referred to by another package. This may mean that the package is missing, has been obsoleted, or is only available from another source. E: Package sqlite3 has no installation candidate

After ssh'ing on the server, fooling around with apt-get myself, googling sqlite3/ubuntu, etc, still no love. So I went ghetto on this one:

sudo vi /opt/local/lib/ruby/gems/1.8/gems/deprec-1.99.13/lib/deprec/recipes/rails.rb

.. and removed the two sqlite related libs from the recipe (sqlite3 libsqlite3-ruby), at around line 77. Bottom line is that all my apps use mysql, so I knew I wouldn't miss sqlite

6) NOW let's setup a rails stack!

Since deprec's installrailsstack had made it almost all the way though, I checked out the recipe to see what's left. Turned out it's just one task:

cap deprec:rails:install

That's it. At this point my server has Rails2, a bunch of gems including mongrel, nginx, mysql, and probably a whole bunch of other stuff I don't know about yet.

7) so . . . how do you deploy to staging?

I didn't want to figure out the "right" way to configure Capistano for a staging environment right now. Plus, my project already has a (non-depified) deploy.rb, which I don't want to mess with right now.

In the interest of expediency, I just checked out a separate copy of my app and:

rm config/deploy.rb
depify .
vi config/deploy.rb

... add set :user, 'root', updated :application, :repository, :domain, and the three roles (:app, :web, :db)

cap setup 

cap deprec:deploy

... and had to work through some problems with access to my SVN repo, which is probably reflective of the sorry state of said repository. As a takeaway, however, note that capistrano2 uses scmusername and scmpassword instead of svnusername/svnpassword.

8) create the database

First you have to create your database user. Again, I know deprec helps with this, but it wasn't immediately obvious to me how to do it. SSHing onto the server and issuing a few commands inside mysql was easy enough though:

mysql> create user general@localhost identified by 'yourpassword';
mysql> GRANT ALL ON *.* TO 'general'@'localhost';

And back on my local machine:

cap deprec:db:create

from here you could do cap deprec:db:migrate, but I wanted a fresh snapshot from production via mysqldump.

9) fire it up

deprec:nginx:restart
deprec:mongrel:restart

It works! Total time elapsed: about three hrs.

This is a good stopping point, since my staging environment is up and running. The next step is to replace Mongrel with Thin -- but that will have to wait for the next post.

Notes on what's where

• Mongrel Cluster config is in `/etc/mongrelcluster/[appname].yml
• ninx config is in /usr/local/nginx/vhosts/[app_name].conf

I did some research today on low-cost VPS setups for Rails deployments. My immediate need is to set up some staging environments, but I'm also interested in being able to quickly spawn new production-grade Rails environments as easily (and cheaply) as possible. Note: there's a good discussion on VPS's here in the Rails Business group.

Here are my findings so far:

Rails stack setup / Application management

For the providers that don't have a Rails stack, there are some good options for setting up your own stack and/or deploying/managing your apps after you get your server running:

• RailsMachine gem -- Stack setup + App management
• RubyWorks -- Stack setup -- "RubyWorks implements a load-balanced cluster of Mongrel servers, which in most situations is the best way to deploy a Rails application today. It is packaged as a Yum repository (for RedHat/CentOS) or APT repository (for Debian/Ubuntu)" Works on a range of distros, including Centos 4,5. Uses HAProxy for balancing, doesn't include a web server.
• Deprec -- Stack setup + App management -- "Deprec is a collection of automated recipes, written in Ruby, for setting up production ready Ruby on Rails servers. This includes everything from creating admin accounts and setting up your ssh keys to compiling and installing the packages required to get a Rails application running on a freshly installed Ubuntu server." Works on Ubuntu 7.10
• Slingshot gem -- Stack setup + App management -- "a set of recipes, tasks, extensions and plugins that work with Capistrano to setup and configure a server for a Rails Application, quickly, and easily."
• FiveRun's production stack -- Stack setup -- "provides a fast, easy way to get everything you need to develop and deploy Ruby on Rails applications. RubyStack includes Ruby, Subversion, MySQL, SQLite, ImageMagick and several Ruby Gems, and will optionally install Apache 2.2 with rewrite and proxy support"

Additions / recommendations / corrections? Leave a comment.

For a recent Rails project, I had to use PostgreSQL instead of the standard MySQL for the database. Setting up Postgres on Mac OS 10.5 has some quirks, which I will share with you here.

• There are three primary ways to get PostGRES onto Leopard: a pre-built binary, MacPorts, and source. My choice was to build with source. There are several good posts on building from source:
  • stiefels.net
  • developer.apple.com
  • brilliantcorners.org
  • forums.realsoftwrae.com

• I do not recommend creating a separate user to run postgres. Doing so clutters your startup screen with another user, and requires you to su to that user to start the database. Instead:

  1) install Postgres as root

  2) after install, from the command line: chown -R andre /usr/local/pgsql/data. The -r flag changes the ownership recursively. Obviously, substitute your own username for "andre" in the example.

• Since I don't use PostGres all the time, I don't want it to be started up automatically. Instead, I set up two aliases in by .bashrc:

  alias startpostgres='/usr/local/pgsql/bin/pgctl -D /usr/local/pgsql/data -l /usr/local/pgsql/data/logfile start' alias stoppostgres='/usr/local/pgsql/bin/pgctl -D /usr/local/pgsql/data -l /usr/local/pgsql/data/logfile stop'

• when you start postgres, you'll probably get the dreaded "shared memory error," which looks like this:

  FATAL: could not create shared memory segment: Cannot allocate memory DETAIL: Failed system call was shmget(key=1, size=1081344, 03600). HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory or swap space. To reduce the request size (currently 1081344 bytes), reduce PostgreSQL's sharedbuffers parameter (currently 50) and/or its maxconnections parameter (currently 10).

• some resources will tell you to edit the /etc/rc file, which is incorrect for Leopard. Instead, create /etc/sysctl.conf (if it does not already exist -- mine didn't), and add the following lines:

  kern.sysv.shmmax=167772160 kern.sysv.shmmin=1 kern.sysv.shmmni=32 kern.sysv.shmseg=8 kern.sysv.shmall=65536

You'll need to su to root to create /etc/rc

Have fun with Postgres!

I have a geocoding recipe coming out in Pragmatic's Advanced Rails Recipes. The recipe is titled "Finding Things by Location," and it offers how-tos and best practices on using my GeoKit plugin to, well, Find Things by Location. The most recent PDF of the beta book includes my recipe along with over 30 other recipes by community leaders like Chris Wanstrath, Dan Benjamin, and Geoffrey Grosenbach, to name just a few.

Much of what Rails provides to get your apps up and running isn't optimized for performance. It's crafted to be more efficient for developers, not more efficent at runtime. before_filter callbacks on your RESTful controllers to get the current object? That's an extra database call. All those nifty plugins you are using to kickstart your app? They probably generate far more SQL (and slower SQL) than if you coded the same functionality ad-hoc. ActiveRecord itself is slow compared to raw SQL and object instantiation.

If your project grows huge -- 10's of millions of PV/day huge -- you're gonna have to revisit some of that stuff. Some of it you can compensate for with smart caching techniques and more hardware. And some of it you will have to throw away and rewrite. If you get really huge, you're going to have pay back some of the technical debt you incurred by choosing a tool like Rails.

Yes, using a tool like Rails incurs some debts. But just like the rest of the world, there are good debts and bad debts. If you're smart about the kind of debt you take on, you can build far more, and build it faster.

Many of your projects will never reach the level where you need to "pay back" for all that developer productivity you enjoyed on the front end. That means you can try more ideas, and (hopefully) fail fast at the more speculative ones. If some of your projects do need to scale radically (beyond the basics of better caching, more hardware, etc), you have an incredible amount of upfront productivity you can leverage against that work of -- say -- optimizing some key queries by hand. As long as you go in with eyes wide open and realistic expectations, then I say that's a smart kind of technical debt.

I just did the last round of updates on PlaceShout using Netbeans 6RC2's Ruby support. Based on my brief exposure, here are some things it offers which made it worth by time in a concrete way.

1. really excellent svn integration
   • it shows you lines you've added/deleted/changed in your regular editing view, as you work -- it's a great visual cue on the extent of your changes without leaving your normal coding flow.
   • the source compare is outstanding, it highlights not just changed lines but parts of lines. Very smart and intuitive, and better than anything I've used before.
2. control-click on any method to go to its definition. This is especially useful in .erb templates to go to a helper definition. If there are multiple definitions, Netbeans gives you a list.
3. as-you-type "jump to class" in addition to "jump to file". Command-o brings it up, and it works on project classes, native ruby, and gem and plugin code. It's snappy enough to be useful, but could get a little faster.
4. method completion on AR attributes/finders: netbeans gives you a nice as-you-type completion list on the dynamic, data-driven getters/setters Rails infers from the database columns. Nice! There's a lot of other method completion it tries to do, but this is the one that stood out to me.

Netbeans has got a ton of functionality and a rich plugin ecosystems, and I'm sure there are more gems in there. The points I've indicated here are just the ones that were useful, accessible, and intuitive enough to engage me right away.

Tor Norbye is publishing a regular "Ruby Screenshot of the Week", give it a read to see what else is possible.

My take is that if 1) method/paramater completion is refined a bit, 2) debug mode works seamlessly and effortlessly (couldn't get it to work on Leopard), and 3) performance improves a bit -- Netbeans will be a home run for Rails development.

Huge bonus points if the collaboration plugin delivers on its promises (and supports Ruby!)

PlaceShout has hit the streets! What is it? It's a “cheatsheet”-style summary of places around town, created by you and arbitrated by the community. See my full announcement on The Napkin for more details.

Have you noticed how much easier it is to remember directions in your own city? Think about the last time your printer ran out of ink and you had to jot directions to a restaurant on a post-it. Compare that to writing out directions from a hotel to a restaurant in an unfamiliar city.

It's a lot easier on familiar turf. Why? Near home, the directions are anchored by points of familiarity in your mind. You already know how to get to someplace nearby, so you can use that as a ready point of reference. Closer to home, you get to use all kinds of reference points: your work, familiar street names, the park you go to, etc. Because that's the way your brain works. In your brain, everything is a relative reference.

This is equally true if you are communicating directions to someone, rather than just writing them down for yourself. The better you know the person, the easier it is: "go like you're headed to work, but turn left just before that Thai restaurant you like." Not only are the directions concise, they are simple enough that you probably don't need to write them down. Communicating is a lot easier when there are shared experiences, reference points, and a sense of "knowing what the other person knows."

There are lessons in here somewhere for those of us who create software. After all, we spend a lot of time with our computer trying to find things, either on the computer (photos, spreadsheets) or with the computer (a book on Amazon, that page you Delicious'd -- or did you mark it in Google Reader?). I often think of this when I'm using Delicious. I know I bookmarked something, but wading through the tag cloud to find it again takes too long. I usually end up finding it with a few Google searches. When the first search fails, there's usually something in those first results which triggers my memory on the keywords that will retrieve it.

Can software infer what is familiar terrain for us, and provide navigation relative to that? Can Google or Delicious know what my mental "anchors" are, and help me find stuff from there?

An OS-level example: I have a dozen or so Ruby on Rails projects in /Users/andre/projects/rails/, and I spend a lot of time in the immediate subdirectories. If there were a heat map of the places I spend time in, this directory would be hot. Would that be useful as a navigational device? Possibly. If a place got hot enough, the OS could ask me to label it in a way that's meaningful to me. I might use that as a jumping-off point as long as spend a lot of time on Rails projects.

Obviously there are pitfalls when trying to get a computer to guess what you're trying to do, or what's important to you (R.I.P Clippy). Still, there are bound to be payoffs for trying to get the computer to present navigation the way you think, rather than how it computes. A good start is to think about how our brains tend to remember things clustered relative to familiar points of reference.

So I set up a tumblelog over at Andre Lewis' tumblelog. Original name huh?

I like tumblr because it is very convenient to post to. Adding the tumblelog may be too many publishing venues for me though. Here's the breakdown:

• blogging: longer, more analytical or technical posts
• tumblelog: short posts, bookmarks, pictures from my phone
• facebook status updates: things that are short and/or ephemeral and/or only of interest to my friends
• delicious: bookmarks . . . despite the fact that I never look back at them, because Google always leads me to it quicker (this is another post however)

I don't use twitter, except to pipe my Facebook status updates. If I did, it would be for things even more ephemeral than Facebook status updates.

At some point, all this is going to have to come together and be much easier than it is today.

This is an odd list of things that tripped me up at one point or another during Rails app deployment. They've been living on the whiteboard next to my desk for a while; now it's time to set them free.

• App URL in Apache vhosts config is missing the .com. The requests to this url go to the next available application configured in vhosts. Symptom: accessing your newly launched app actually gives you some other app you have hosted on the same machine.

• Apache listener is configured on the same port as another Rails app. Symptom: don't remember exactly, but I think Apache fails to start.

• gem names case-sensitive in deployment environment (Linux), not in development environment (Windows -- this is from the dark days when I was developing on Windows). Symptoms: 1) can't install the gems you need; 2) free-floating mac envy.

• reverse slash in 'require' works in development but causes error in production -- another Windows/Linux gotcha.

• Rake gotcha: one missing include breaks all other tasks and everything fails silently.

• Failure to clear out existing sessions after major rework and deployment. Symptoms: things failing with no reasonable explanation; pointed pangs of stupidity when you realize that you refactored stuff you store in session. rake db:sessions:clear.

• rake not defaulting to production environment on production systems. This was a while ago, but it tripped me up a few times.

• gems you installed in dev, but forgot to install in production.

There, now I can reclaim that space on my whiteboard . . .

Hotspotr is now available as an interactive map on your cell phone. You can browse for WiFi hotspots on the go -- which is exactly when you need to find hotspots right? See the link on the home page, or jump right to the mobile download.

The mobile goodness is provided through a partnership with LightPole. I tried several similar mobile services, and LightPole is doing a fantastic job. I've been using the application for some time myself during development.

Note that you have to be on Sprint, AT&T, or T-Mobile, and that the app is free. give it a try.

Ruby and Rails seems to be taking a beating on blogs the last few days. Lots of assertions, rants, and general discontent. And, a few valid points made on both sides of the spectrum.

At the end of the day, nothing I've read will change what I'm doing tomorrow: having fun creating great products with a language and framework that is a joy to use. Working in a space that is dominated by innovative thinking. Engaging a vibrant community of smart, creative technologists.

Everyone knows that Ruby and Rails have limitations. They are not perfect for every project. But, they are perfect for my projects right now!

• catch up on some of those neglected RSS feeds. Google Reader makes it easier to read a few posts at a time without committing to a long session.
• check in on those Google groups you belong to but rarely read.
• go back to NYTimes, since Select is free again.
• browse upcoming and squidlist for interesting things happening in the real world.
• read DZone. There are a lot of good links there.

John Resig and team have officially launched their long-awaited jQuery UI library early this morning. jQuery UI provides "higher-order" effects like magnification and drag-and-drop. It also provides themeable widgets (tabs, accordions, sortable tables, calendars, etc). Here are the key links:

• John's blog post annoucement
• the UI site
• the UI docs

My impression so far is that this initial release is a little rough. Like any ambitious cross-browser effort, it will go an iteration or two, gaining polish and stability.

A final note: if you're used to check in on the jQuery group for news, note that there's a separate group for UI: http://groups.google.com/group/jquery-ui.

I chose jQuery as my JavaScript library, and I opted not to use iUI. The iUI choice was driven primarily by the specifics of the application, and the kind of user interaction that made sense for it.

Here is the most basic JS/CSS to get the iPhone orientation detection going:

<head>
<meta name="viewport" content="width=320; initial-scale=1.0; maximum-scale=1.0; user-scalable=0;">
<%=javascript_include_tag 'jquery' %>
<script language="JavaScript">
  window.onload=function(){
    setTimeout(ip.updateLayout, 0);
    setInterval(ip.updateLayout, 100);
    setTimeout(ip.hideURLbar,100);
  };

  ip={
    pageWidth:null,
    updateLayout:function(){
      var width=window.innerWidth;
      if (width != ip.pageWidth) {
        ip.pageWidth = width;
        var orientation = width == 320 ? "profile" : "landscape";
        // fire transition events
        orientation == 'profile' ? ip.transitionToProfile() : ip.transitionToLandscape();
        $(document.body).attr('class', orientation);
        setTimeout(function(){
            window.scrollTo(0, 1);
        }, 100);
      }
    },
    transitionToProfile: function(){
    },
    transitionToLandscape: function(){
    },  
    hideURLbar:function(){
      window.scrollTo(0, 1);
    }
  };   
</script>
<style>
  body {
    margin: 0;
    padding: 0;
    font-family: Helvetica;
    color: #FFFFFF;
    -webkit-user-select: none;
    cursor: default;
    -webkit-text-size-adjust: none;
  }

  body.profile { 
    height:416px;
    background-color:red;
  }

  body.landscape { 
    height:270px;
    background-color:green;
  }
</style>
</head>