Explore Posterous

Feeling nostalgic: Amazon Webservices back in 2006

Tomorrow I'm invited to the Amazon Webservices Customer Advisory board in London. And I'll attend with wico from the #vz team at the http://aws.amazon.com/enterprise_london/.

At my former company - Lovely Systems - we were using Google App Engine and Amazon Webservices all the time.
After beeing abstinent for almost 10month - and working hard to get the Data Processing Agreements compliant to German law - I fell a little bit nostalgic.

On December 12th 2006" I made it to the AWS frontpage - "Amazon Elastic Compute Cloud powers Lovely Systems" *proud* :)

Send 10,000 emails for $1. Zero-cost services

Moore’s Law has prevailed in the online services space  - the cost of online infrastructure is now at a point close enough to nothing to cost nothing.  Thanks to Google, only pay for what you use, on demand and accessed anywhere because it’s stored in ‘the cloud’ for $0.10 a gigabyte of bandwidth, or $0.0001 per email.

Check out Google App Engine, a platform that let’s developers host and run applications. Up to a generous quota for free, and then by the gigbyte.

http://code.google.com/appengine/docs/whatisgoogleappengine.html

One of the cool ‘useful’ apps using this ‘zero cost’ platform is behavioural targeting and segmentation app BTBuckets. 

A behavioural targeting and segmentation tool running off the Google App Engine, along with Amazon Web Services AWS platform, it’s delivered free for up to 5 million impressions a month. The interface is 2.0 with easy to use, obvious functions that so often get missed in more expensive options. The app segments and triggers different kinds of events for different users. It’s useful, and equivalent to traditional software options worth $10,000 a month or more. But, there’s a catch, this app also aggregates all of the user information in its system to provide demographic and behavioural data into ad-serving systems. Subsidised by the new marketers quest for more granular targeting of their audience. Be prepared to have to change your privacy policy.

http://www.btbuckets.com/

http://aws.amazon.com/publicdatasets/

Sending email in the cloud

Having run a couple of startup companies in the cloud on Amazon Web Services, the issue of sending email has cropped up a number of times.

The problems

Amazon doesn't allow you to set reverse DNS lookup entries for your mail server and a large number of mail servers will not accept mail from a server that doesn't have this.  This is a spam prevention feature to try and cut down on mail being relayed through insecure mail servers.

The other blocking issue you will come across is blacklisting.  Because of the nature of cloud systems it is likely that someone else has had your IP address in the past.  If the person wasn't entirely honest then they will probably have been blacklisted at some point.  Here is an example error message you will receive if you try and mail from a blacklisted host:

553 Mail from xx.xxx.xxx.xxx not allowed - 5.7.1 [BL21] Connections not accepted from IP addresses on Spamhaus PBL; see http://postmaster.yahoo.com/550-bl21.html [550]

or

550 DY-001 Mail rejected by Windows Live Hotmail for policy reasons. We generally do not accept email from dynamic IP's as they are not typically used to deliver unauthenticated SMTP e-mail to an Internet mail server...

The solutions

So, what can you do about it?  The simple answer is do not send mail from the cloud, instead relay your mail through a server hosted somewhere else.

The first option is to send email through GMail.  This works fine for small volumes of email but as your web application becomes more succesful you will quickly start hitting the volume limits imposed by Google and will start seeing entries like this in your mail log:

550 5.4.5 Daily sending quota exceeded.

The best long-term solution is to run your own mail server hosted outside of the cloud.  I recommend slicehost as a provider, you can have a mail server up and running from as little as $20/month and this will allow you to send masses of email.  This might sound a bit ominous if you haven't set one up before but they have a great tutorial on setting up a mail server.

Dodging the spam folder

Once you have your mail server up and running you should set an SPF record for your domain.  SPF records are TXT DNS records that you create to announce that your new mail server is allowed to send mail for your domain.  Lets say you have just finished setting up your mail host as mail.example.com then you need to create a TXT entry for example.com with the following contents (including the quotes!):

"v=spf1 include:mail.example.com -all"

You should now be sending mail happily from the cloud and with a bit of luck your recipients will receive the mail in their inboxes.

Amazon's Dynamo - All Things Distributed

pythonでAWS. SimpleDBを触ってみる(1). - lolloo-htnの日記

AWS: django-storages : install

(dev)hdknr@domU-12-31-39-00-D9-A1:~/.ve/dev/src$ hg clone
https://hdknr@bitbucket.org/david/django-storages/
destination directory: django-storages
requesting all changes
adding changesets
adding manifests
adding file changes
added 43 changesets with 105 changes to 51 files
updating working directory
32 files updated, 0 files merged, 0 files removed, 0 files unresolved
(dev)hdknr@domU-12-31-39-00-D9-A1:~/.ve/dev/src$ cd django-storages/
(dev)hdknr@domU-12-31-39-00-D9-A1:~/.ve/dev/src/django-storages$ ls -al
total 68
drwxr-xr-x 6 hdknr users 4096 2009-10-11 05:20 .
drwxr-xr-x 3 hdknr users 4096 2009-10-11 05:20 ..
-rw-r--r-- 1 hdknr users 562 2009-10-11 05:20 AUTHORS
drwxr-xr-x 2 hdknr users 4096 2009-10-11 05:20 backends
drwxr-xr-x 2 hdknr users 4096 2009-10-11 05:20 docs
drwxr-xr-x 4 hdknr users 4096 2009-10-11 05:20 examples
drwxr-xr-x 3 hdknr users 4096 2009-10-11 05:20 .hg
-rw-r--r-- 1 hdknr users 103 2009-10-11 05:20 .hgignore
-rw-r--r-- 1 hdknr users 1539 2009-10-11 05:20 LICENSE
-rw-r--r-- 1 hdknr users 315 2009-10-11 05:20 README
-rw-r--r-- 1 hdknr users 21229 2009-10-11 05:20 S3.py
-rw-r--r-- 1 hdknr users 988 2009-10-11 05:20 setup.py

(dev)hdknr@domU-12-31-39-00-D9-A1:~/.ve/dev/src/django-storages$ python
setup.py install
zip_safe flag not set; analyzing archive contents...

Installed
/home/hdknr/.ve/dev/src/django-storages/setuptools_hg-0.2-py2.5.egg
running install
running bdist_egg
running egg_info
creating django_storages.egg-info
writing django_storages.egg-info/PKG-INFO
writing top-level names to django_storages.egg-info/top_level.txt
writing dependency_links to django_storages.egg-info/dependency_links.txt
writing manifest file 'django_storages.egg-info/SOURCES.txt'
writing manifest file 'django_storages.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-i686/egg
running install_lib
running build_py
creating build
creating build/lib
copying S3.py -> build/lib
creating build/lib/backends
copying backends/__init__.py -> build/lib/backends
copying backends/s3boto.py -> build/lib/backends
copying backends/couchdb.py -> build/lib/backends
copying backends/symlinkorcopy.py -> build/lib/backends
copying backends/s3.py -> build/lib/backends
copying backends/database.py -> build/lib/backends
copying backends/mosso.py -> build/lib/backends
copying backends/overwrite.py -> build/lib/backends
copying backends/ftp.py -> build/lib/backends
copying backends/mogile.py -> build/lib/backends
copying backends/image.py -> build/lib/backends
creating build/bdist.linux-i686
creating build/bdist.linux-i686/egg
creating build/bdist.linux-i686/egg/backends
copying build/lib/backends/__init__.py ->
build/bdist.linux-i686/egg/backends
copying build/lib/backends/s3boto.py -> build/bdist.linux-i686/egg/backends
copying build/lib/backends/couchdb.py -> build/bdist.linux-i686/egg/backends
copying build/lib/backends/symlinkorcopy.py ->
build/bdist.linux-i686/egg/backends
copying build/lib/backends/s3.py -> build/bdist.linux-i686/egg/backends
copying build/lib/backends/database.py ->
build/bdist.linux-i686/egg/backends
copying build/lib/backends/mosso.py -> build/bdist.linux-i686/egg/backends
copying build/lib/backends/overwrite.py ->
build/bdist.linux-i686/egg/backends
copying build/lib/backends/ftp.py -> build/bdist.linux-i686/egg/backends
copying build/lib/backends/mogile.py -> build/bdist.linux-i686/egg/backends
copying build/lib/backends/image.py -> build/bdist.linux-i686/egg/backends
copying build/lib/S3.py -> build/bdist.linux-i686/egg
byte-compiling build/bdist.linux-i686/egg/backends/__init__.py to
__init__.pyc
byte-compiling build/bdist.linux-i686/egg/backends/s3boto.py to s3boto.pyc
byte-compiling build/bdist.linux-i686/egg/backends/couchdb.py to couchdb.pyc
byte-compiling build/bdist.linux-i686/egg/backends/symlinkorcopy.py to
symlinkorcopy.pyc
byte-compiling build/bdist.linux-i686/egg/backends/s3.py to s3.pyc
byte-compiling build/bdist.linux-i686/egg/backends/database.py to
database.pyc
byte-compiling build/bdist.linux-i686/egg/backends/mosso.py to mosso.pyc
byte-compiling build/bdist.linux-i686/egg/backends/overwrite.py to
overwrite.pyc
byte-compiling build/bdist.linux-i686/egg/backends/ftp.py to ftp.pyc
byte-compiling build/bdist.linux-i686/egg/backends/mogile.py to mogile.pyc
byte-compiling build/bdist.linux-i686/egg/backends/image.py to image.pyc
byte-compiling build/bdist.linux-i686/egg/S3.py to S3.pyc
creating build/bdist.linux-i686/egg/EGG-INFO
copying django_storages.egg-info/PKG-INFO ->
build/bdist.linux-i686/egg/EGG-INFO
copying django_storages.egg-info/SOURCES.txt ->
build/bdist.linux-i686/egg/EGG-INFO
copying django_storages.egg-info/dependency_links.txt ->
build/bdist.linux-i686/egg/EGG-INFO
copying django_storages.egg-info/not-zip-safe ->
build/bdist.linux-i686/egg/EGG-INFO
copying django_storages.egg-info/top_level.txt ->
build/bdist.linux-i686/egg/EGG-INFO
creating dist
creating 'dist/django_storages-1.0-py2.5.egg' and adding
'build/bdist.linux-i686/egg' to it
removing 'build/bdist.linux-i686/egg' (and everything under it)
Processing django_storages-1.0-py2.5.egg
creating
/home/hdknr/.ve/dev/lib/python2.5/site-packages/django_storages-1.0-py2.5.egg
Extracting django_storages-1.0-py2.5.egg to
/home/hdknr/.ve/dev/lib/python2.5/site-packages
Adding django-storages 1.0 to easy-install.pth file

Installed
/home/hdknr/.ve/dev/lib/python2.5/site-packages/django_storages-1.0-py2.5.egg
Processing dependencies for django-storages==1.0
Finished processing dependencies for django-storages==1.0

Amazon SimpleDB

SimpleDB:Python SimplDB/dev のテスト

(jail)hdknr@mailjail:~/.ve/jail/src$ svn checkout http://simpledb-dev.googlecode.com/svn/trunk/ simpledb-dev
A    simpledb-dev/simpledb-dev
A    simpledb-dev/simpledb-dev/src
A    simpledb-dev/simpledb-dev/src/simpledb_dev.py
A    simpledb-dev/simpledb-dev/src/portalocker.py
A    simpledb-dev/simpledb-dev/src/templates
A    simpledb-dev/simpledb-dev/src/templates/Query.xml
A    simpledb-dev/simpledb-dev/src/templates/GetAttributes.xml
A    simpledb-dev/simpledb-dev/src/templates/ListDomains.xml
A    simpledb-dev/simpledb-dev/src/templates/QueryWithAttributes.xml
A    simpledb-dev/simpledb-dev/src/templates/DeleteAttributes.xml
A    simpledb-dev/simpledb-dev/src/templates/error.xml
A    simpledb-dev/simpledb-dev/src/templates/DeleteDomain.xml
A    simpledb-dev/simpledb-dev/src/templates/CreateDomain.xml
A    simpledb-dev/simpledb-dev/src/templates/PutAttributes.xml

(jail)hdknr@mailjail:~/.ve/jail/src$ pip install web.pyRequirement already satisfied: web.py in /usr/lib/pymodules/python2.5
Installing collected packages: web.py
Successfully installed web.py
(jail)hdknr@mailjail:~/.ve/jail/src$ dpkg -l | grep webpy
ii  python-webpy                      1:0.32+dak1-1              Web framework for Python applications

まぁ、いいか。

(jail)hdknr@mailjail:~/.ve/jail/src/simpledb-dev/simpledb-dev/src$ pwd
/home/hdknr/.ve/jail/src/simpledb-dev/simpledb-dev/src

(jail)hdknr@mailjail:~/.ve/jail/src/simpledb-dev/simpledb-dev/src$ python simpledb_dev.pyhttp://0.0.0.0:8080/

hdknr@mailjail:~/.ve/jail/src$ curl http://localhost:8080/<?xml version="1.0"?>
<Response xmlns="http://sdb.amazonaws.com/doc/2007-11-07/">;
        <Errors>
                <Error>
                        <Code>NoSuchVersion</Code>
                        <Message>SimpleDB/dev only supports version 2007-11-07 currently</Message>
                        <BoxUsage>0.0000219907</BoxUsage>
                </Error>
        </Errors>
        <RequestID>5ba318a0-001f-4df0-9542-886cbf6cd705</RequestID>
</Response>

(jail)hdknr@mailjail:~/.ve/jail/src/simpledb-dev/simpledb-dev/src$ python simpledb_dev.py test > /tmp/simpledb_dev.log

simpledb_dev.log の確認。

Running tests and printing out sample XML output...

Sample GetAttributes:

?AWSAccessKeyId=Test&DomainName=TestDomain&Timestamp=XXX&Version=2007-11-07&Signature=XXX&Action=GetAttributes&ItemName=0385333498

<?xml version="1.0"?>
<GetAttributesResponse xmlns="http://sdb.amazonaws.com/doc/2007-11-07/">;
        <GetAttributesResult>
                        <Attribute><Name>Rating</Name><Value>5 stars</Value></Attribute>
                        <Attribute><Name>Rating</Name><Value>*****</Value></Attribute>
                        <Attribute><Name>Rating</Name><Value>Excellent</Value></Attribute>
                        <Attribute><Name>Keyword</Name><Value>Book</Value></Attribute>
                        <Attribute><Name>Keyword</Name><Value>Paperback</Value></Attribute>
                        <Attribute><Name>Title</Name><Value>The Sirens of Titan</Value></Attribute>
                        <Attribute><Name>Author</Name><Value>Kurt Vonnegut</Value></Attribute>
                        <Attribute><Name>Year</Name><Value>1959</Value></Attribute>
                        <Attribute><Name>Pages</Name><Value>00336</Value></Attribute>
        </GetAttributesResult>
        <ResponseMetadata>
                <RequestId>3175e02f-a69f-4e88-ad98-a22ceb6d8a9f</RequestId>
                <BoxUsage>0.0000219907</BoxUsage>
        </ResponseMetadata>
</GetAttributesResponse>

Sample Query:

?AWSAccessKeyId=Test&DomainName=TestDomain&Timestamp=XXX&QueryExpression=%5B%27Year%27+%3D+%272007%27%5D+intersection+%5B%27Author%27+starts-with+%27%27%5D+sort+%27Author%27+desc&Version=2007-11-07&Signature=XXX&Action=Query

<?xml version="1.0"?>
<QueryResponse xmlns="http://sdb.amazonaws.com/doc/2007-11-07/">;
<QueryResult>
    <ItemName>B00005JPLW</ItemName>
    <ItemName>B000T9886K</ItemName>
</QueryResult>
<ResponseMetadata>
        <RequestId>4f4bcb4e-56cb-43a5-a9aa-7a5da26ca46e</RequestId>
        <BoxUsage>0.0000219907</BoxUsage>
</ResponseMetadata>
</QueryResponse>

Sample QueryWithAttributes:

?AWSAccessKeyId=Test&DomainName=TestDomain&Timestamp=XXX&QueryExpression=%5B%27Title%27+%3D+%27The+Right+Stuff%27%5D&Version=2007-11-07&Signature=XXX&Action=QueryWithAttributes

<?xml version="1.0"?>
<QueryWithAttributesResponse xmlns="http://sdb.amazonaws.com/doc/2007-11-07/">;
<QueryWithAttributesResult>
    <Item>
            <Name>1579124585</Name>
            <Attribute><Name>Rating</Name><Value>4 stars</Value></Attribute>
            <Attribute><Name>Rating</Name><Value>****</Value></Attribute>
            <Attribute><Name>Keyword</Name><Value>Hardcover</Value></Attribute>
            <Attribute><Name>Keyword</Name><Value>Book</Value></Attribute>
            <Attribute><Name>Keyword</Name><Value>American</Value></Attribute>
            <Attribute><Name>Title</Name><Value>The Right Stuff</Value></Attribute>
            <Attribute><Name>Author</Name><Value>Tom Wolfe</Value></Attribute>
            <Attribute><Name>Year</Name><Value>1979</Value></Attribute>
            <Attribute><Name>Pages</Name><Value>00304</Value></Attribute>
    </Item>
</QueryWithAttributesResult>
<ResponseMetadata>
        <RequestId>fc29f8ef-6298-4712-8f61-16ccc9e48c73</RequestId>
        <BoxUsage>0.0000219907</BoxUsage>
</ResponseMetadata>
</QueryWithAttributesResponse>

Sample PutAttributes:

?AWSAccessKeyId=Test&DomainName=TestDomain&Timestamp=XXX&Attribute.0.Name=Rating&Version=2007-11-07&Signature=XXX&Action=PutAttributes&Attribute.0.Value=%2A%2A%2A%2A%2A&Attribute.0.Replace=true&ItemName=B00005JPLW

<?xml version="1.0"?>
<PutAttributesResponse xmlns="http://sdb.amazonaws.com/doc/2007-11-07/">;
        <ResponseMetadata>
                <RequestId>2131099b-8f38-4b14-a803-f9bd09c26fce</RequestId>
                <BoxUsage>0.0000219907</BoxUsage>
        </ResponseMetadata>
</PutAttributesResponse>

Sample Query:

?AWSAccessKeyId=Test&DomainName=TestDomain&Timestamp=XXX&QueryExpression=%5B%27Pages%27+%3C+%2700320%27%5D&Version=2007-11-07&Signature=XXX&Action=Query

<?xml version="1.0"?>
<QueryResponse xmlns="http://sdb.amazonaws.com/doc/2007-11-07/">;
<QueryResult>
    <ItemName>0802131786</ItemName>
</QueryResult>
<ResponseMetadata>
        <RequestId>829b9b6e-c91e-47e8-8e27-58c59084136c</RequestId>
        <BoxUsage>0.0000219907</BoxUsage>
</ResponseMetadata>
</QueryResponse>

Sample CreateDomain:

?AWSAccessKeyId=Test&DomainName=TestDomainXXX&Timestamp=XXX&Version=2007-11-07&Signature=XXX&Action=CreateDomain

<?xml version="1.0"?>
<CreateDomainResponse xmlns="http://sdb.amazonaws.com/doc/2007-11-07/">;
        <ResponseMetadata>
                <RequestId>2e4e3435-a629-4f5b-9fb1-d752680567f7</RequestId>
                <BoxUsage>0.0000219907</BoxUsage>
        </ResponseMetadata>
</CreateDomainResponse>

Sample ListDomains:

?AWSAccessKeyId=Test&DomainName=TestDomain&Timestamp=XXX&Version=2007-11-07&Signature=XXX&Action=ListDomains

<?xml version="1.0"?>
<ListDomainsResponse xmlns="http://sdb.amazonaws.com/doc/2007-11-07/">;
<ListDomainsResult>
    <DomainName>TestDomain</DomainName>
    <DomainName>TestDomainXXX</DomainName>
 </ListDomainsResult>
<ResponseMetadata>
        <RequestId>8d429dd8-adb5-4224-a632-eae03e40b20b</RequestId>
        <BoxUsage>0.0000219907</BoxUsage>
</ResponseMetadata>
</ListDomainsResponse>

Sample DeleteDomain:

?AWSAccessKeyId=Test&DomainName=TestDomainXXX&Timestamp=XXX&Version=2007-11-07&Signature=XXX&Action=DeleteDomain

<?xml version="1.0"?>
<DeleteDomainResponse xmlns="http://sdb.amazonaws.com/doc/2007-11-07/">;
        <ResponseMetadata>
                <RequestId>9ebcae0b-4a2e-4441-a3a1-d856d8b2f774</RequestId>
                <BoxUsage>0.0000219907</BoxUsage>
        </ResponseMetadata>
</DeleteDomainResponse>

OK

MOONGIFT: » Python製開発用SimpleDB「SimpleDB/dev」:オープンソースを毎日紹介

Public Data Sets on Amazon Web Services (AWS)