Explore Posterous

Android Application Launched: App Engine News

First (?) Production Site on Open BlueDragon for Google App Engine

Yesterday Peter Farrell pointed me to a brochure site he did for a friend. Looks great, it's running Mach-II 1.8 beta, using SES URLs, all great stuff.

Then he told me it was running on Open BlueDragon for Google App Engine (GAE). VERY cool. You can read a bit more about it on Peter's blog. I could be wrong, but I think this may be the first production site running on OpenBD on GAE. (If there are others out there, I'd love some links!)

The interest in running CFML apps on GAE has really been picking up lately, and with good reason. It's a dead simple way to deploy CFML applications to Google's cloud, and unless the site is going to get a huge amount of traffic it's completely free.

Free CFML engine, free hosting, easy deployment right from Eclipse ... there's a lot to love here. No more hunting around for cheap shared hosting accounts that are so restrictive they're barely usable, no more spending money on a VPS if you don't need one (though I highly recommend them!), you just build your app and deploy right to GAE.

Since a lot of people I've been talking with recently aren't all that familiar with GAE, I'd like to point out that it's a bit of a different paradigm than many other cloud computing services. Unlike Amazon EC2, where you're dealing with things at the server level, GAE is application-oriented. So you don't have a server with an operating system on which you install a servlet container and OpenBD, instead you're simply deploying individual applications to Google's Java infrastructure. It's a really nice way to do things since the server-level stuff is all handled for you.

Be aware that there are some restrictions on what you can do on GAE vs. deploying on your own server, but really these are just differences rather than any huge impediments:

• Unless you use the Virtual File System (VFS) that Vince Bonfanti has been building, you can't write to the file system. Note that you can use the VFS on all Java projects on GAE; it isn't specific to OpenBD.
• The limit on the length of a single request is 30 seconds. (Not a problem unless you're generating reports or something.)
• You can't ...
• open a socket or direct connection to another host. You can use GAE's URL Fetch Service to call other hosts on ports 80 and 443.
• spawn new threads
• make system calls
• There are some limitations on request/response sizes, maximum file sizes, etc. but all are quite reasonable.
  

The big difference is that you can't use a traditional RDBMS like MySQL. Instead you use Google's Datastore, which is a "schemaless object datastore." You can read more about Datastore here. So what does this mean for CFML developers? Well, OpenBD for GAE implements persistence for CFCs to Google Datastore, so you can save your CFCs (works for structs as well) to the Datastore by using the GoogleWrite() function, and there are also functions to run queries and read single objects from the Datastore. You can read more about OpenBD's Datastore integration on the OpenBD wiki.

There has also been some work done on making a file-based RDBMS like H2 work with the VFS, and although it works apparently the performance is pretty slow at the moment. When the issues around that get resolved that will be another nice option, since a database could be deployed to GAE right along with your application code.

Personally I'm really excited about the opportunities this offers CFML developers. It could not be simpler to build a site and deploy it on Google App Engine, and the benefits of being on the Google infrastructure are pretty amazing. If your application doesn't get a huge amount of traffic it's completely free, and if your application starts to grow, the underlying infrastructure will scale up automatically as needed. That's one of the many benefits of cloud computing.

If you're interested in getting started with OpenBD on GAE, make sure and join the OpenBD Google Group, check out the GAE section of the wiki, and there are some great blog posts by members of the OpenBD community as well:

• CFML on GAE with OpenBD (Paul Kukiel)
  
• OpenBD GAE and CFCs (Paul Kukiel)
• CFML on GAE with OpenBD (Aaron Lynch)
• Google App Engine's Virtual File System with OpenBD (Aaron Lynch)
• A Better Way to Serve Files from the GAE VFS (Aaron Lynch)
• SES URLs on OpenBD GAE (Peter Farrell)
  

As I said, there's lots to love here, so I encourage you to give it a try.

Slow Custom Domain on GAE

I recently deployed an application to Google App Engine (GAE). Accessing the application via the appspot.com subdomain was fast however the custom domain consistently was slower (sometimes 5 seconds versus 200ms). I believe the issue is that custom domain was set up on the same day and that the DNS has not propagated through the internet completely. This leads to some crazy routing to Google's network.

I'll wait a while longer because I ask the Google App Engine group (not much else other than this explanation is mentioned in the searches I've done). I'm going to reboot my local wireless/wired routers.

SES URLs on OpenBD GAE

I've been working on a brochure style site for a friend this week.  The budget is nothing and I'm doing this as a favor for my friend so it's my time. Instead of doing the same old, I decided that I should try something new and learn since I'm donating my time.  I hope to be posting more about using Google App Engine (GAE) as free hosting for CFML applications.  GAE is cloud computing and it does turn the usual concept of building applications on its head.  Since GAE's daily http request limit for the free account is 1.3M requests, I don't foresee any issues with overing over our quotas.

The application I'm building is using Mach-II Simplicity (1.8) and is running on Open BlueDragon (GAE Edition).  My friend wanted to use SES URLs so I investigated to see if the SES URL filter that is usually bundled with Open BlueDragon was commented out in the web.xml file.  It was not even present in the file (whereas in the normal OpenBD edition it is commented out) so I was a little nervious this wasn't going to be possible.  Instead of wasting time researching, I used GAE's nifty one click (ok, it's two clicks and your Google account password) deployment option to just try by testing it.  Here is the snippet I added in my web.xml right at the top after the <web-app ...> node:

<filter>
    <filter-name>SearchEngineFriendlyURLFilter</filter-name>
    <display-name>SearchEngineFriendlyURLFilter</display-name>
    <description>SearchEngineFriendlyURLFilter</description>
    <filter-class>com.newatlanta.filters.SearchEngineFriendlyURLFilter</filter-class>
    <init-param>
        <param-name>extensions</param-name>
        <param-value>cfm,cfml</param-value>
    </init-param>
</filter>
<filter-mapping>
    <filter-name>SearchEngineFriendlyURLFilter</filter-name>
    <url-pattern>/*</url-pattern>
</filter-mapping>

Bam! I deployed it to the cloud and it worked.  SES URLs on OpenBD GAE is a go!  Thought I would share it with the world.

CFML on the Google App Engine with Open BlueDragon | Progressive Overload

Send 10,000 emails for $1. Zero-cost services

Moore’s Law has prevailed in the online services space  - the cost of online infrastructure is now at a point close enough to nothing to cost nothing.  Thanks to Google, only pay for what you use, on demand and accessed anywhere because it’s stored in ‘the cloud’ for $0.10 a gigabyte of bandwidth, or $0.0001 per email.

Check out Google App Engine, a platform that let’s developers host and run applications. Up to a generous quota for free, and then by the gigbyte.

http://code.google.com/appengine/docs/whatisgoogleappengine.html

One of the cool ‘useful’ apps using this ‘zero cost’ platform is behavioural targeting and segmentation app BTBuckets. 

A behavioural targeting and segmentation tool running off the Google App Engine, along with Amazon Web Services AWS platform, it’s delivered free for up to 5 million impressions a month. The interface is 2.0 with easy to use, obvious functions that so often get missed in more expensive options. The app segments and triggers different kinds of events for different users. It’s useful, and equivalent to traditional software options worth $10,000 a month or more. But, there’s a catch, this app also aggregates all of the user information in its system to provide demographic and behavioural data into ad-serving systems. Subsidised by the new marketers quest for more granular targeting of their audience. Be prepared to have to change your privacy policy.

http://www.btbuckets.com/

http://aws.amazon.com/publicdatasets/

Setting up the Dev Environment

In this post I'll cover the setup I use for developing my Compojure/GAE app.

Emacs
Of course you can use any editor you like to edit your source code, but Emacs has some great features for editing Lisp code that make developing in Clojure a pure joy. When properly set up, you can add new or modify existing code and inspect all your data while the application is running.

I followed the instructions at http://technomancy.us/126 to set up Slime/Swank for Clojure and it worked flawlessly without any modifications. I can also recommend the Emacs Starter Kit that is mentioned in the above post.

Clojure
There are several options for getting the clojure.jar that you'll need:

• Download it from http://code.google.com/p/clojure/downloads/list
• Clone the git repository at git://github.com/richhickey/clojure.git and build from source using ant
• If you're using Emacs and followed the instructions above, you can automatically download the source and build it using the clojure-install command.
• Compojure provides a zip file of its dependendies (see below), this also includes clojure

Compojure
You can download a tarball of the Compojure sources from http://github.com/weavejester/compojure/downloads, but I'd recommend cloning the git repository at git://github.com/weavejester/compojure.git. Either way you'll build the compojure.jar from source using ant.

Compojure has a few dependencies. You can download a zip file containing all needed jars (including clojure.jar and clojure-contrib.jar) from the download link above, or you can just run ant dep, which will download the zip for you.

Google App Engine SDK
Download the latest GAE SDK from http://code.google.com/intl/de-DE/appengine/downloads.html. Choose the SDK for Java and unzip it somewhere.

In the next post I'll create a simplistic Hello World Compojure app.

Compojure on GAE: What's this about?

I'm interested in the Clojure programming language and the Google App Engine platform right now, and I'm playing around with the combination of both. Since there isn't a whole lot of information on this on the net, I thought I'd share the results of my experiments in this blog. Here's the rundown on what this will (and won't) be about:

Clojure
Clojure is a functional programming language of the Lisp family, designed with concurrency in mind to make it easy to write multi-threaded programs. It runs on the Java Virtual Machine and can call (and be called from) Java code. I'm not going to write a tutorial on programming Clojure (many others can do this better than me) nor am I going to try to convince you that Clojure is the greatest language in the world. I just think it's very interesting and I assume you do, too - otherwise you probably wouldn't have stumbled across this site. If you want to learn more about Clojure, the official website provides very good documentation. I highly recommend watching the videos of Rich Hickey's talks, they got me hooked in the first place.

Compojure
Compojure is one of the frameworks for developing web applications in Clojure that are being developed right now. I don't know if it's the best, but it's the one I'm playing around with. The documentation isn't very extensive, but you can find some at http://preview.compojure.org and there's an active Google group. You can grab the code at GitHub. For getting started I recommend this short tutorial by Compojure's creator, James Reeves.

Google App Engine
Google App Engine (GAE) is a service provided by Google that let's you develop web applications that can be deployed to Google's server infrastructure. Google provides several APIs that applications can use, e.g. for user authentication and storing data in a distributed data store. The idea is that applications can easily scale to handle everything from very small to very large loads. When App Engine was initially released it only provided a Python runtime environment. In April 2009 introduced the Java runtime for App Engine. This opened the door not only for Java programs but for a number of languages that run on the JVM, including Clojure.

The example app: A blog! (yawn)
I chose to implement a little blogging application to experiment with Clojure/Compojure on GAE. This isn't very exciting at all and I don't intend to actually use it. But it is simple and straightforward and displays much of the functionality of a typical web app: displaying pages, handling form input, storing and retrieving persistent data and authenticating users. (Edit: After reading this question on stackoverflow I feel a little bad about my choice of example app. But at least I'm in good company...)

So, that's the plan. In the next post I'll talk about setting up the development environment.

My big secret project with Fuzzy Rainbow

I wish I could post screenshots of the app I'm working on with Fuzzy Rainbow.  But I can't talk too much about it yet.

It's a Python/AppEngine + OpenSocial + JavaScript + jQuery UI app.  All the interaction & graphics are done with pngs and crazy JavaScript tricks because @shazow introduced me to the magic of jQuery & jQuery UI and convinced me that JavaScript could do anything.  Working on Fluffy Puffy Cloud Pets also taught me this to some extent, although I did nothing but draw clouds (and integrate my teammates' code) for that project, and I was skeptical of JS back then.

No canvas, no Flash.  If this turns out to be the wrong approach, I'm not afraid to throw away 2 months of code and rewrite it. 

The app is pretty odd.  Things do funny things.  It's either a game or a tool, depending on the user's perspective.

More pieces are coming down the line.  There will likely be a Pinax-based standalone site centered around the app.  There will also likely be a mobile client (maybe PhoneGap).  And more that I absolutely can't mention.  It's pretty ambitious, but it's just a matter of doing it piece-by-piece and cutting out pieces if needed. 

All the seemingly-scattered hacks & experiments that I've done over the summer are now coming together.  It is much more planned and well-thought-out than it might have appeared.  I am excited that I can say this much & can't wait to reveal more.

What Google App Engine needs to improve, and why I'm [not] moving on [anymore]

Update 9/15:  One of the folks on the GAE team saw this!  First he gave me more apps. Then he sent me another email saying he "just pushed a change live which should fix deployments so that you actually have 250 (instead of the mysterious 100). Please give this a try and let me know!"

This totally changes things.  Wow!

****

I like GAE a lot.  But in the end, it's hard to stick with it for major, time-sensitive projects where you don't have a lot of time to waste on workarounds.  As a developer, you just don't have enough control over your work. 

My biggest troubles:

• You can't pack up and leave appspot.com.  There is the dev_appserver, which you can use to test locally.  But there's no easy, reliable way to run a GAE server clone on your own box.  Appdrop, a project that tried to address this, was last updated in 2008 and seems to have fallen behind the new GAE SDK releases (I could understand - what a pain to keep updating).  Want to run a GAE clone on EC2?  You can't without a lot of work.
• Lots of arbitrary limits.  Sure, they were made with good intentions, to improve performance.  But you're bound to run into one trap or another with no way out.  Max # of files in an app, short timeout for processes, max # of IN queries, etc.
• The local dev server is inadequate for testing.  You need to be able to test your app in an environment identical to your production setup.  I thought I could get away with using one of my 10 appspot apps as a test app, but I was hitting the max 100 deployments/day limit by 4pm and having to wait until midnight for more quota (and the max 250 deployments that they said is untrue, it's really 100).  I had issues with cross-domain requests being treated differently on appspot than on my local server.  When you're working with OpenSocial, you find that different social networks are finnicky about cross-domain requests in different, often surprising ways. 
• (on Python GAE) If an unsupported library isn't pure Python, you can't use it.  Workarounds for this involve using another server, which is irritating.  This is often inevitable - when you're starting a project, you often don't know what libraries you're going to need.
• And finally, you can only have 10 apps.  So it's inadequate for even little hobby or test projects.  As someone who experiments a lot, I usually create (and finish) at least 1-2 mini-projects per week.  I've been reusing my appspot apps, but there are ones that I'd like to leave up and running for others to see and use.

I do want to mention that GAE is a really amazing product that just has a few kinks that need to be worked out.  The GAE team has made great outreach efforts in the developer community and has always been helpful.  And GAE so close to being what I've always wanted.

I'll still use GAE every now and then, and I'm likely to move my OpenSocial app back to GAE in the future.  But GAE has too many roadblocks for rapid development.  I was spending more time on workarounds than on my actual app.

If App Engine were open-source, I could run my own clone of the production server somewhere.  I'd be able to get around all the roadblocks and develop my app using their framework and datastore quickly, in a setup basically identical to production, and then I'd be able to come back and optimize it to meet the appspot limitations if I wanted to have their automatic scaling/load balancing.  I'd also love to be able to peek under the covers for my own knowledge.

Alternately, if App Engine would just let me pay to get around all these issues, I'd get my company's credit card out right now.