In the course of building the frontend application for Earbits, I’ve been constantly annoyed with the responsiveness of my local development server. As a pretty complex AJAX application, there are lots of little (and some bigger) calls to the Rails backend APIs to do lots of potentially slow things. This means that there are lots of calls that are originating in the browser and being served by one mongrel, so the server ends up handling requests too slowly. Which leads to me being frustrated. Which leads to me finding a nice solution to the problem.

Warning: This May Not Apply To You

If you have slow running requests that have to run in series, or if you’re not doing AJAX-y things, this isn’t going to help you much. This setup is only useful if your app sends a bunch of parallel requests to the backend. For instance, if after your main page loads, you need to do AJAX requests to a) load the user’s friends, b) get their comment history, c) calculate some suggestions, and d) check to see if you should show a popup promo to the user, then this might help you out. But if you need the results of request (a) to get (b) and the results of (b) to get (c), then you’re out of luck and this will be a waste of your time.

Ok, ready to do this?

Step 1 – Install Nginx

This is much simpler if you have macports installed. So go do that first if you haven’t already. If the first step below doesn’t do anything, you don’t have macports. Once you have macports, it’s a simple two-command process to install nginx.

$ sudo port -d selfupdate
$ sudo port install nginx +ssl +debug

Now, while nginx is installing you can open a new terminal window and keep going. The install will take a few minutes.

Step 2 – Configure Foreman

Foreman is a great tool released by David Dollar to make running web processes and workers crazy easy. It uses a Procfile to define the different things it controls and you should refer to his blog or a Google search to learn about the finer points of Foreman. We use a pretty simplistic file here as an example.

To get started just run:
$ gem install foreman

Then create a file called Procfile in the root of your project using the text editor of your choice. My Procfile has just one line that define a web process using the standard Rails server:

web:    bundle exec rails server -p $PORT

Step 3 – Configure Nginx

Now that nginx is probably finished installing, we can get it configured to act as a local Load Balancer for our Rails server. I’m going to set it up using port 80 locally. If you are running an HTTP server locally already on port 80, then you should adjust your configuration accordingly. But if you’re not using port 80, Rails redirects might cause you some headaches. So I suggest you stick with the standard port.

First, you’ll need to find your nginx.conf configuration file. If you used macports to install it, it’s probably in /opt/local/etc/nginx/nginx.conf but if you can’t find it, run nginx -V to see where it’s hiding.

If you’re not familiar with nginx at all, there are two main directives that you’ll be using, server and upstream. The server directive defines the front-end part of nginx (i.e. where it listens for incoming requests). The upstream directive defines the back-end (i.e. where to send requests so that Rails can handle them). Both of these snippets should exist within the http section of your config file. I’ll provide a complete nginx.conf at the end of this section so that you can just replace the default file and then tweak it if necessary.

Upstream section

This is the configuration that I’m using, I will explain it in a second:

upstream foreman4000 {
  server localhost:4000;
  server localhost:4001;
  server localhost:4002;
  server localhost:4003;
  server localhost:4004;
}

Here on line 1 we’re defining the upstream which we will reference later. Note that “foreman4000″ is just an arbitrary name used to reference it within the config. Then since we plan on running 5 web processes starting at port 4000, we list out each of the ports where our Foreman will be listening for requests.

Server section

This will define a server so that nginx will listen to requests on port 80 and direct them to the upstream we created above (line 6). We also pass the real IP of the request and the hostname along to the server (lines 7-9, in case you need either of these bits of data.

 server {
    listen       80;
    server_name  localhost;
    access_log  /opt/local/var/log/nginx/foreman4000.access.log  main;
    location / {
      proxy_pass http://foreman4000;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $remote_addr;
    }
 }

Test the config

After you’re done making the changes above (or if you download my nginx.conf example file), you should run $ sudo nginx -t to test the config and make sure it’s all good. If it fails, double check everything. If it’s good, you’re ready to start it up.

Step 4 – Start your Engines

Now all we have to do is start everything up. In your Rails project directory:
$ sudo nginx
$ foreman start -p 4000 -c web=5

Note: I use the non-standard port 4000 as the starting point so that other random servers can run on 3000 (rails default) and 5000 (foreman default) without conflicting with this setup.

If it’s working you should be able to open up http://localhost in your browser and your Rails site should load. In the terminal running Foreman, you should see it handing requests to each of the 5 web processes (web.1 – web.5) like this:

Now, enjoy your new, super-responsive AJAX server! I know I do.

Notes and Extras

There are a few reasons my local development environment is slow. Firstly, I’m using a hosted mongodb that is shared among a few of our developers so that we can more easily collaborate on work. It makes it possible for me to add a new feature and seed the database with some good example data, then hand it off to our UI guy to style it up with zero configuration on his end other than a git pull. Secondly, we make heavy use of the Facebook Graph APIs which can at times be sluggish over my mere-mortal internet connection. Getting someone’s friends might take a full second, posting something to a wall might take a couple seconds. You can imagine how with just one process running, it would get a bit slow.

The third reason our site is slow in development is the simple fact that it’s designed to be chatty. Instead of large monolithic views getting rendered for the user, we do a lot of client-side template rendering using underscore.js based on json data requested by the client app (which leverages backbone.js for much of the interaction logic). In production, this works great because we can fire up a bunch of web processes and everything scales on out. But in development, between the slow database access and the numerous calls, things can take seemingly forever to fully load. But not anymore!

BTW – if you haven’t tried Earbits out for your online radio needs, you absolutely should. Great way to discover new music, share with your friends and really just have a great time. And you should stop listening to so much Genesis anyway. It’s time for something fresher.

This method and code is inspired by two blog posts, which got me started but neither of which really fulfilled my complete requirement. The first used two records per friendship and the second was a twitter-style friend/follower without the approval process.

So, here I break down my Friendship model which hopefully you will find useful and/or insightful as a complete solution or a starting point for your own implementation.

First step is to generate a new model that we’ll call Friendship.  A Friendship will connect two users, tracking who made the request, who the friend is, and whether the request has been approved. So we generate it like so:

$ script/generate model Friendship user_id:integer friend_id:integer approved:boolean

This will create the db migration script and the app/models/friendship.rb file which we’re going to edit to include relationships to the User model. When first created, the Friendship model will be mostly empty so we need to add two lines.

class Friendship < ActiveRecord::Base
  belongs_to :user
  belongs_to :friend, :class_name => "User", :foreign_key => "friend_id"
end

The first line is obviously creating a relationship to the User model, but the second line is a little more complex. It also needs to create a relationship to the User model (there isn’t a Friend model) but we want to refer to this as the friend. So we have to specify both the :class_name and the :foreign_key so that Rails knows how to hook up this :friend field to actual models.

Now that we have a Friendship, we need to configure the User model to allow for these self-referential relationship through the Friendship model. Before we hit the code, let’s walk through the concepts here that define the state of friendships and friends.

A friendship is a two-way relationship which is initiated by one user and approved by the other user. So, we have two states of a friendship (not-approved-yet or approved) and we have two directions of friendship relative to the current user: friendships requested by the current user (we’ll call these “direct” friendships), and friendships requested to the current user (we’ll call these “inverse” friendships).

So using this terminology, we have 4 possible states of Friendship:
1) direct approved
2) indirect approved
3) direct not-approved
4) indirect not-approved

States #1 and #2 are simply what we will be calling “friends” – they are approved relationships, no matter the direction. Because we don’t care who requested the friendship once it’s approved, we will group these together.

State #3 is what we call “pending friends” – other users whom the current user has requested to be friends with and which are awaiting the other user’s approval. These are out of the control of the current user, and just waiting to be approved or rejected.

State #4 is what we’ll call “requested friends” – other users who have requested that the current user be their friend, and are awaiting the current user’s approval. These are the actionable items for the current user to approve. Ignoring a friend request simply deletes the non-approved Friendship, which is the Facebook method (doesn’t tell the other person they were rejected, but allows them to send another request if they want.)

Now that we have that groundwork, here is the User model:

class User < ActiveRecord::Base
  [...]
  has_many :friendships
  has_many :inverse_friendships, :class_name => "Friendship", :foreign_key => "friend_id"
  has_many :direct_friends, :through => :friendships, :conditions => "approved = true", :source => :friend
  has_many :inverse_friends, :through => :inverse_friendships, :conditions => "approved = true", :source => :user

  has_many :pending_friends, :through => :friendships, :conditions => "approved = false", :foreign_key => "user_id", :source => :user
  has_many :requested_friendships, :class_name => "Friendship", :foreign_key => "friend_id", :conditions => "approved = false"

  def friends
    direct_friends | inverse_friends
  end
  [...]
end

The first two lines are pretty self-explanatory as the first references our “direct” Friendships and the second our “inverse” Friendships which we’ll be using as the basis of all of our friend lists. Lines 5 and 6 simply create our direct and inverse friend lists, only looking at friendships which have been approved. These are states #1 and #2 from our discussion above and are complete, approved bi-directional friends lists.

Line 8 here creates the list of pending friends (state #3 above) by looking for direct friendships which have not been approved. Then finally in line 9 we handle state #4 which are the “requested friendship” waiting for us to approve them, by taking all the non-approved Friendships where the current user is the target of the request (which is why we specify the foreign_key of “friend_id”).

And finally, because it would be a hassle to deal with two lists of approved friends, we create a method to simply combine them under the name of “friends”, in lines 11-13.

That’s it! You should be able to create and manage facebook-style friend requests now.

I’m not going to cover creation of the Friendship controller here in this post, but I may in the future if people would find it helpful. In short, the controller needs to handle just five actions: index, create, approve, ignore, delete. In my implementation, all actions except for index are used only for ajax calls through link_to_remote, so they’re pretty lightweight.

Updated: the direct_friends line in the User model above was incorrectly using “:source => :user” when it should be “:source => :friend” so that the returned models are the Friends you requested, not yourself the requestor.

Yesterday, I found Sphinx Search and decided to try implementing it in place of the (horrid) MySql Fulltext searching for my Photocore project. After downloading the binaries and indexing a couple million rows of metadata, I was amazed at how well it performs. It indexed all my data in less than a minute (compared to the 30 minutes required by MySql Fulltext) and I haven’t come up with a search that takes longer than 0.005 seconds. I was hooked immediately. So I needed a .NET API because I didn’t want to patch my database server to use the Sphinx plugin.

Source download after the jump.

Download v0.2 source | binaries

Download has no warranties, blah, blah, blah. It’s based on the Java API and is released under the GNU GPL license.

Important Note
It does not have all the features implemented. Most notably, it does not have support for Exceprts. If you modify code to make it work, send it back and I’ll post the updated version here. Or, if I get enough requests to implement it, I might. But my use of Sphinx does not include excerpts, so I didn’t bother at this point.

Other Notes
This is a pretty quick-and-dirty project, I’ll proabably clean it up over time and repost new versions, but for now make sure you test it thoroughly before putting it into production on your own sites. That’s your responsibility…

It is tested against the Sphinx searchd server version 0.9.8-dev running on both linux (ubuntu) and windows (vista) with simple queries only. I didn’t do anything fancy in my queries.

It does not work against 0.9.7 or earlier because those did not support multiple queries. It should work against future versions of Sphinx server. No promises though.

Usage
Super-simple to use after you have a searchd server running with an index. Three lines will do a search for “rose bowl gooley” matching any index (“*”) and will get the second page of results (items 41-80) with a max result count of 10,000 documents.

// set hostname and port defaults
SphinxClient cli = new SphinxClient(hostname, 3312);

// set the pagination params
// (this is page 2, 40 per page, 10000 max)
cli.SetLimits(40, 40, 10000);

// define the query and index
// ("*" means any index)
cli.AddQuery("rose bowl gooley", "*");

// run the query and get the results
SphinxResult[] results = cli.RunQueries();

The SphinxResult object contains an array of SphinxMatches which will provide you with the DocumentId numbers for all matches. Simple as that.

Happy searching!

I guess I was pretty naive to think that it would be that easy. The distribution doesn’t include some pretty important parts. A .csproj file for one. The entire Resources class for another.

So I had to work my way through the process of getting a compile-able version of the library ready to replace the official System.Web.Extensions binary in my project. The server code Reference License prevents me from simply providing the project to you, but here are the high points if you need to do the same.

Be aware:
This is neither easy nor straightforward and I probably forgot a few steps when I was writing this. You’re gonna have to be diligent to get it to work, even with this howto.

Getting Started
1) Download and install the source
2) Create a new C# project in Visual Studio. You can call it whatever you want but I would suggest you avoid using the same namespace because the System.Web namespace is strongly named and exists in several assemblies. I called mine Dave.Web.Ajax
3) Copy all the files and folders from the source installed directory (C:Program Files (x86)Microsoft ASP.NETASP.NET 2.0 AJAX Extensionsv1.0.61025SourceSystem.Web.Extensions on my computer) into your new project directory and include them in the project.
4) Make sure your Default Namespace on the project is something good – again Dave.Web.Ajax in my case.

Get the Missing Files
1) Start the .Net Reflector (if you’re a serious .Net developer, you should have downloaded this to your computer long ago, btw)
2) Open the official System.Web.Extensions.dll file
3) Save all the Resources into a new Resources folder in your project. When you save them, drop the “System.Web.Resources” part of the filename if it’s there.
4) You should now have a file called AtlasWeb.resources in your folder
5) Open a VS Command Prompt and in your resources folder run
resgen AtlasWeb.resources AtlasWeb.resx. This will generate the resx file that you need in visual studio to dynamically generate the class.

Setup your Embedded Resources
1) In VS, include all the files in the Resources in your project (except the *.resources files) if you haven’t already.
2) Select them all and set the Build Action to Embedded Resource
3) Click on the AtlasWeb.resx file and set the Custom Tool property to “ResXFileCodeGenerator”. This will generate the AtlasWeb.Designer.cs file which gives you a real class with properties for each item in the resource file.
NOTE) You might need to repeat this process (and the resgen.exe process) for the other *.resources files, but I haven’t done it and havn’t had any issues yet.

Rename the Namespace and cleanup
If you are going to stick with using the System.Web.* namespace, then you can skip this step.
1) Basically, I handled this in an iterative fashion. I did a project Find-and-Replace for “System.Web.” and replaced with “Dave.Web.Ajax.” Of course this process will break stuff because now objects that are really in the System.Web namespace (and not in the System.Web.Extensions assembly) will dissappear.
2) Get your iteration on: compile, add “using System.Web.something;” to a file with the error, compile, repeat.

Fix the Embedded Resource references
At this point, your project should compile and if you reference it in a web project in place of the System.Web.Extensions.dll file (or the GAC reference) the web project should compile too. But as soon as you load up a page with AJAX controls on it, you will get a runtime error saying the Resource File could not be found.

Update the reference names of your embedded .js files to be fully qualified. Your new fully qualified resource name will be “DefaultNamespace.Resources.Filename.js” Mine is “Dave.Web.Ajax.Resources.MicrosoftAjax.js” for example.

There are references in these files
UIScriptManager.cs:931,935
UIWebResourceUtil.cs:64+
UITimer.cs:147

Update those references to use the fully qualified names. After that, in WebResourceUtil.cs there is an optimization in the SystemWebExtensionsContainsWebResource(string) function that you have to modify. It does a switch statement on the length of the input string before comparing to the resource name string. Update each case conditional to be the length of the string in that case statement. If you forget this step you will be very confused why it says it can’t find a resource that you know is there…

Now, all embedded resources need an assembly reference so that they can be detected using reflection (which is how the AJAX library looks for a valid embedded resource). I just created a dummy class called Scripts.cs inside the Resources file so I could include the assembly attributes:

[assembly: System.Web.UI.WebResource("Dave.Web.Ajax.Resources.MicrosoftAjax.debug.js", "text/javascript")]
[assembly: System.Web.UI.WebResource("Dave.Web.Ajax.Resources.MicrosoftAjax.js", "text/javascript")]
[assembly: System.Web.UI.WebResource("Dave.Web.Ajax.Resources.MicrosoftAjaxTimer.debug.js", "text/javascript")]
[assembly: System.Web.UI.WebResource("Dave.Web.Ajax.Resources.MicrosoftAjaxTimer.js", "text/javascript")]
[assembly: System.Web.UI.WebResource("Dave.Web.Ajax.Resources.MicrosoftAjaxWebForms.debug.js", "text/javascript")]
[assembly: System.Web.UI.WebResource("Dave.Web.Ajax.Resources.MicrosoftAjaxWebForms.js", "text/javascript")]
namespace Dave.Web.Ajax.Resources
{
class Scripts
{
}
}

Just make sure you have a WebResource attribute for each of the javascript files in the project.

That’s it
Assuming you’ve gotten this far, you must be dedicated or desperate. Now you can go in and make your mods to any of the AJAX files and you’re good to go. Hopefully this was helpful and I wish I could just post the project, but rules is rules.

Its been about 6 hours since I finished my conversion, so its pretty fresh in my head, but I may have forgotten a step somewhere. I’ll try to answer questions in the comments but I’m pretty busy these days and I just blew half my day because the release of AJAX doesn’t play well inside iframes (which is the whole reason I had to do this mod).

Good luck to you.

• All-Day events now are imported onto the correct day (instead of a day early)
• Non @gmail.com usernames are accepted. If you use “thegooley” it will assume “thegooley@gmail.com”, but you can also use “thegooley@hotmail.com” by providing your entire email address
• The FAILED error message has been modified to display the actual error text, so please report what it says when you post about a problem
• Error log file changed from ApplicationDataGCalSyncGCalSyncErrors.xml to GCalSyncErrors.txt (it was never actually an XML file, but was named that due to an oversight by me)

Download after the jump

Download installer cab | source

Source zip file now includes the Google API library with my modification described here.

Download installer cab (updated 12/25/06)

What happened?
In the gauthrequest.cs file, provided by Google, lines 408-9 look like this

// check the content type, it must be text
if (!response.ContentType.Equals(HttpFormPost.ContentType))

and this was working until last night. In this case HttpFormPost.ContentType = “text/plain”. Problem appears that now, the response.ContentType of the authrequest is being returned as “text/plain; charset=utf-8″. So that .Equals conditional no longer does the right thing, thus the authentication breaks.

To fix, I change line 409 to

...ContentType.StartsWith(HttpFormPost...

Now all is well again. Re-install and give it a go. I guess this is going to be standard for interfacing with a “beta” Google API. And we’ll just have to get used to it…

Download installer cab | source

If you just want to use it on your mobile, hit that binary cab link from pocket IE and it will install for you. No, I didn’t sign my assemblies (i don’t have a real certificate), so be prepared to click through some warnings. You’ll need the .NET 2.0 Compact Framework installed for this app and you can get it from Microsoft here.

I have been using it for a day now with no issues but all the usual disclaimers apply. Use it at your own risk. I’m releasing this under the Creative Commons Attribution-ShareAlike 2.5 License. Share it and Remix it, but let me know what you’re doing with it so we can enjoy it too.

More detailed release notes after the jump…

Important Note
Make sure your device timezone is set correctly.
When the app launches, the status bar will show your timezone, if it’s not correct go fix it before you sync. Google stores in GMT, and the app has to convert to local time for you. If your timezone isn’t right you’ll get wierd times on all your appointments, then you’ll have to delete them all off your device and re-sync. Not fun. Trust me.

Basic Usage

1. After you put in your account information and click Pick Calendars, it will connect and attempt to list your available calendars for you
2. Check off one or more calendars.
3. Make sure you go to Options->Choose Default. This will allow events created on the mobile device to be exported back to the google calendar.
4. Click Done and then Yes to do the initial import.
5. Import will take a little while, depending on how many events you have in the calendar. (All events are imported, this will be more flexible in the future)

Notes
Persistent Storage
All user settings are stored in plaintext in an xml file in the application directory. If you click the “Save Password” checkbox, then your password will also be stored in plaintext. I am minimally concerned about this myself, if that bothers you consider not saving your password, though you will have diminished functionality if you do that. There are a few places where the app uses the stored password instead of the text box value (an oversight on my part, this will be changed in the next release).

Sync Functions
New events created either on the Mobile or Online are synced. Changes made Online to dates and titles will be reflected on the mobile device. Other changes will not be merged. Date/Title changes on the mobile device to events already existing online will be overwritten by the online data. I need to write a better sync/merge function and that will come in the next release.

Authentication Issues
It appears that on occasion there is a random authentication failure to the API. I am guessing this happens if you try to sync too often, but couldn’t find any supporting documentation from google. If anyone wants to track that down, I’d be grateful.

Error Reporting
Errors which occur during runtime are recorded in a file somewhere near \applicationData\GCalSync\GCalSyncErrors.xml – this will help developers, and if you want to report bugs to me, include the stack trace from the error when you do.

Outlook
If you sync using ActiveSync or the Vista Sync Center, events added to your mobile calendar by this application will be merged back into your outlook calendar on the desktop. I don’t use Outlook, so I don’t think it’s a big deal, but it might affect you so you should know about it.

Future Plans
Merge Sync
As mentioned above, this feature is in progress.

Import Date Range Selection
To allow user to specify which events to import. i.e. everything from one week ago through two months in the future. or something.

Deletion Handling
Right now I have no way to detect and mirror event deletions, if you delete something from your mobile calendar, it will be re-imported at the next scheduled time. If you delete something online, it won’t be deleted off the mobile but will be orphaned instead. To handle this will require a more persistent state mechanism so that I know when an online event that used to be there doesn’t show up in the event list anymore.

Public Calendar URL Support
Right now you can import public calendars by just adding them to your Google Calender collection of calendars. Public calendars are also visible via a direct URL and I want to add a function so you can anonymously connect to calendars (even if you don’t have a Google Calendar account at all, you should still be able to import events from the Georgia Tech Basketball 06-07 public calendar for instance)

That’s it for now
If you have suggestions for improvements and features, leave me a comment and I’ll see what I can do. If you take the source and do something cool with it, drop a note so that everyone can check out your take on it.

I’m not quite ready to release it this second, so here are some screenshots of the process of using the app. Like all Windows Mobile apps, after you close it it can still run in the background, so that allows it to run scheduled imports. Basically every minute it hits a timer interval and checks to see if it should import again (default time between imports is 60 minutes). Right now it is more of an Import than a “Sync” tool. But in the next release I will should have two-way syncro going so that you can make changes right on your phone and have those changes sent back up to Google…

And now, the screenshots (after the jump)

Step 1: Enter your username and password

Step 2: Click “Pick Calendars” – at this point, we connect to the Google API to list all the calendars you own and those you subscribe to

Step 3: Check off the calendars you wish to import

Step 4: Click “Done”, then it asks if you want to do an immediate import. Say “Yes”

Step 5: Import in progress

Step 6: Done, # of events is reported. You can now close the app and it will continue to import/update every 60 minutes in the background.

Binaries and source coming tonight.