1031 Investment Services project

I've been working with my friend Scott Sheehan on his website 1031investmentservices.com. It's a Joomla website which is new for me but luckily I'm not a programmer on this project (php? yeeks!) but I'm always interested in opportunities to try something new. So what is the purpose of this blog entry, on a blog I rarely write to any more? I think the biggest part of this project is going to be developing the inbound links. One down, hundreds more (hopefully) to go.

Two Scoops of Django Review

I've just finished reading Two Scoops of Django: Best Practices for Django 1.5 by Daniel Greenfield and Audrey Roy, currently for sale as an e-book for $12 and easily the least expensive (except free) form of professional investment a Django developer can make.

It's not a "I couldn't put it down!" book; rather you will want to put it down about every four pages so you can implement an idea before moving on to the next topic.

I've had one big Django project under my belt, that was my full time gig for over a year (the now defunct 10LocalCoupons.com). I had a team of developers and, for any of us, it was our first Django project and our first python project -- not for a lack of trying to find experienced djangonauts. I recall finishing the django tutorial and then looking for a resource to learn best practices and not finding much. This book fills that need well.

I'm currently working on a solo project in Django, and am still in the early stages, so I easily revamped my project to include many of the best practices espoused here, the first being project layout. Having a project root for your apps and within that a configuration root for settings, urls and wsgi makes sense to me.

The next idea was a sane settings management that avoids many of the pitfalls we encountered in the past: tests that fail in different environments due to different settings, local settings that don't get tracked in source control, and forcing developers to track changes to local-example.py settings.

Other topics include best practices for class based views, templates, how to structure apps, and plenty more.

So it took a week to get through the book because I kept stopping to put the ideas to work.

Plently of the big ideas espoused I already adhere to: using South for database migrations, pip and virtualenv for environment building, and others. Some of the more subtle ones, like using environment variables for your secret key and using a separate requirements file for each environment don't apply to my early stage project yet but in the future I'll be sure to adopt.

Some of the content in the book should be fleshed out a bit more. For example, the book recommends using logging and gives reasons why, but doesn't show logging settings or an example of how to use it. 

For those interested, here is how I use logging in my local development environment where I want to see all things my code is doing; most of this is bolierplate django settings but I've left them in so you can see how it interacts with my customizations. I'm not suggesting that my method is to be considered a best practice, just something to get you started. The console handler and the catchall logger '' are the key parts.

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse'
        }
    },
    'formatters': {
        'verbose': {
            'format':
                '%(asctime)s %(name)s %(levelname)s %(module)s %(message)s',
            'datefmt': '%Y-%m-%d %H:%M:%S'
        },
        'simple': {
            'format': '%(asctime)s %(name)s %(levelname)s %(message)s',
            'datefmt': '%Y-%m-%d %H:%M:%S'
        },
    },
    'handlers': {
        'null': {
            'level': 'DEBUG',
            'class': 'django.utils.log.NullHandler',
            },
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler'
            },
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'simple',
            },
        'file': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/var/log/my_project/my_project.log',
            'maxBytes': 20000000,
            'backupCount': 5,
            'formatter': 'verbose',
            }
    },
    'loggers': {
        'django': {
            'handlers': ['null'],
            'propagate': False,
            'level': 'INFO',
            },
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
        'south': {
            'handlers': ['null'],
            'propagate': False,
            'level': 'INFO',
            },
        '':{
            'handlers': ['console', 'file'],
            'level': 'DEBUG',
            'propagate': True,
            'disabled': False
        }
    }
}

And here is a drop-dead simple use of logging:

import logging

LOG = logging.getLogger(__name__)
LOG.setLevel(logging.DEBUG)

def my_function(my_arg):
    my_var = my_arg + 1
    LOG.debug('my_var: %s' % my_var)
    return my_var

Another place that could have done with some easy examples is reStructuredText; I know its the fancy way to document and could clicked a link learn all about it, but this book could give a little bit more to whet my whistle than "study the documentation for reStructuredText and learn at least the basics." How do others use it for django apps?

This is a good book and well-worth the money. It'll make you more productive, and so is easily worth the money. I look forward to reading further revisions.

Two Scoops of Django: Best Practices for Django 1.5

10LocalCoupons.com

I realize I haven't blogged since I un-retired. So here's the story...

We sold RegionalHelpWanted.com in February 2008. After a year and a half off, I founded a new company with my previous partners to see if we could duplicate that success. (Anyone can get lucky once, but if we could do it again, maybe it it was more than luck?) We are using the same exact business model: working with local media partners using unsold inventory to advertise a local website on a revenue share basis. Instead of help wanted ads, or personal ads which is at the root of what Cupid.com was/is, this time around we are targeting coupon advertising. So in Nashville you'll hear ads for 10NashvilleCoupons.com on radio and see them on TV, and in Portland OR the website is 10PortlandCoupons.com, but it is all one website serving local content to you based on how you get there. Hopefully, you'll find your local pizza guy on there soon, or a discount on an oil change nearby.

Not only is the product different, but the software stack is a complete switch. RegionalHelpWanted.com we did in ColdFusion, Cupid.com in .net, both on IIS against SQL Server. All on Windows. 10LocalCoupons.com is done in django, an awesome framework for Python, on Apache behind nginx against postgreSQL, all on Ubuntu. I'm really enjoying the new way things are done. Much of the tediousness of writing control panel type stuff -- record insert, updates, deletes -- for customer service and accounting needs is a gimme with django's admin package, allowing my development team to hit the ground running. The django community has been a great resource to us.

So the software landscape is very different, but the hardware difference between old and new is even more dramatic. In our previous endeavors, we were paying about $1500 a month per web server for managed services. Now, using open source software on Amazon's EC2, we pay less than one tenth of that. It's a running joke every month when I announce our EC2 cost. My guess is about half of that comes from dropping Microsoft licensing fees, the rest is from virtualization efficiencies and dropping the human support.

Not all things are different however. This new project has given us the opportunity to hire back several of the awesome people we've worked with in the past. That has made it easy for me to go back to work.

I'll be blogging more soon about what we are up to, and pointing out things I've learned along the way, but for now know that I am having tons of fun.

Bulk COPY a CSV into PostgreSQL, skipping first row

Looked for a solution to this yesterday and couldn't find it. Asked my Linux guru Jeremy today and he had a easy solution, so this might be helpful to others.

The scenario is you have a big CSV file, and you want to bulk copy it into PostgreSQL, but the first row of the file isn't data, it's got the column names in it. In my case, the text file is 65 Megs so it's not like you can just edit it in a text editor to delete the offending line. (The data happens to be the combined US and Canada zip/postal code database from ZipInfo.com, fyi.)

SQL Server has a bulk insert GUI that lets you specify a start row. Needed that functionality here.

Solution:

Use wc to find out how many rows are in your file:

$ wc ZCUG.TXT
872135 1871133 69105493 ZCUG.TXT

That first number returned, in my case 872135, is the number of rows in the file. Subtract one and and tail that number, outputting to a new file:

tail -872134 ZCUG.TXT > ZCUG-trimmed.txt

Boom! A new file without the row of column names.

owasp-esapi-python configuration

I tried to send this issue to the esapi-python mailing list (after subscribing) but it doesn't look like that is a functioning list. So any help with the following would be greatly appreciated.

Hi!

Thanks for your work on owasp-esapi-python! I am trying to integrate it into a project and will certainly spread the word to help drum up support for this as I make headway.

I've run into an issue during configuration:

When I do this at the python 2.6 interactive shell, it returns a single line of output...

>>> from esapi.core import ESAPI
>>> ESAPI.encryptor().gen_keys()
Creating new keys in /esapi/keyring/

The documentation leads me to believe that it will also output an Encryptor_MasterSalt but, if it's supposed to do that here, it isn't for me. Let me know any info I can provide. This is on Ubuntu 9.10.

Thanks in advance,
- Steve

Is http://downforeveryoneorjustme.com Down?!? lol

Ok so I don't think it's down, because the homepage does serve. But when I'm trying to use it I'm getting an error dump:


Traceback (most recent call last):
File "/base/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 507, in __call__
handler.get(*groups)
File "/base/data/home/apps/downforeveryoneorjustme/1.337010419190071564/main.py", line 137, in get
self.render_down(u)
File "/base/data/home/apps/downforeveryoneorjustme/1.337010419190071564/main.py", line 103, in render_down
db.put(downer)
File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1212, in put
keys = datastore.Put(entities)
File "/base/python_lib/versions/1/google/appengine/api/datastore.py", line 179, in Put
apiproxy_stub_map.MakeSyncCall('datastore_v3', 'Put', req, resp)
File "/base/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 72, in MakeSyncCall
apiproxy.MakeSyncCall(service, call, request, response)
File "/base/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 266, in MakeSyncCall
rpc.CheckSuccess()
File "/base/python_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 111, in CheckSuccess
raise self.exception
OverQuotaError: The API call datastore_v3.Put() required more quota than is available.


I love the irony. It's working for sites that are up, like google.com. But I'm working on my sister's web site dev.huppahs.com which is being hosted by my friend, and his site is down at the moment.

Hopefully Intuit Won't Shelve Mint.com

Intuit, the makers of Quicken, announced they are buying web 2.0 darling Mint.com. I wrote about my concerns for Mint.com last year, and apparently Mint.com did a good job answering those. Congrats to Mint!

My concern now is that Intuit bought them just to shelve the Mint technology and eliminate a promising competitor. I use Quicken for all my finances, and love it for what it does: give me immediate insight into total net worth, track spending by category, and manage my rental properties. Nothing else out there can do that.

But there are some things about Quicken that just plain suck. They are infamous for regularly launching buggy software. For example, I'm running the latest release of Quicken Rental Property Manager on 64 bit Vista Home Premium, and the software never remembers my preferences. And it says I have 3 reminders, with a "View Reminders" button that, when clicked, brings me to a calendar view with no reminders on it.

I just tried to use Intuit's live chat for tech support for assistance with the above, and after forcing me to pick a support category even though my problem didn't fit any of them, it launched an IE window (even though Firefox is my default browser) that did not connect to anything. Support for Quicken is typically done through its user forums, which can be hit or miss.

So although it is the best thing out there for my specific needs, it is still an unsatisfying user experience, and I'd love for something better to come along. Apparently that won't be Mint.com. Hopefully Intuit will incorporate the best of Mint's quality technology into their own, instead of smugly shelving it.

How to Survive Authorize.Net Outages

I was inspired to write this by the recent heavily reported Authorize.net outage. Credit card processing is one aspect of web site development that you have no choice but to out-source (unless you happen to be a bank). There are several out there, and Authorize.net is one of the biggest. No matter how robust the system is, there are bound to be periodic problems. Over nine years of business, and multiple processors, we saw many of them. Some outages were over in minutes, some lasted hours. Your customers, ready to make a purchase, don't want to hear that your credit card processor is having problems. It is possible to code your website to stay in business during an outage.

At RegionalHelpWanted.com (RHW), we were doing tens of thousands of dollars a day in credit card sales of help wanted advertising, none of which we wanted to lose during an outage by the processor. In the US, our credit card processor was Authorize.net. In Canada, it was Verisign. (We used separate services because we ran the businesses as separate companies.)

On Cupid.com, the problem was similar: we were selling online dating subscriptions; I think our processor was iPay.

In both cases, we were selling electronic services, but the method we used to survive a payment gateway outage with minimum business interruption is applicable to online businesses that are shipping tangible goods.

On your website, your customer should not receive any kind of error message saying that there is a problem with the payment system. The customer does not care about your problems, even if you are not to blame.

When Authorize.net is down, typically your web application will time out when trying to connect to them to make a sale. In a minority of cases, the connection will not time out but will give an invalid response, one that does not fit the normal specification. I don't remember any cases where a processor outage occurred and the processor sent back a valid response with a valid error code like "We're down! Try agin later!" In all of these cases, your web application should behave the same way:

- Capture the order information, and store it for later processing.
- Give the customer a success message
- Use an asynchronous process to retry these transaction until they are completed.

Capture the order information
On RHW, we let users opt-in to storing their credit card information to speed future orders, so we had already done the work necessary to do this safely and securely. On Cupid.com, the whole business was built on reoccurring billing, so same deal there. Your Terms of Use must allow you to always retain order information for enough time to process that order, even if the user does not opt in to longer information retention. While your at it, log any response that you did get from the payment gateway, scrubbed of any security sensitive info like credit card number, as it might help with forensics later.

Give the customer a success message
Error messages are the opposite of user friendly. Tell the customer that you have the order information, that it typically only takes a moment to process and that they will receive an email its done. And then let them be on their way.

On Cupid.com, we'd let the user start using the site with all the privileges of a paying member. If they sent any messages to other members, those messages would be queued until their order was successful.

On RHW, we would post the help wanted advertising. If it later turned out that the credit card was rejected, the ad would be removed. But this also could have been setup so that the ad was not posted until the credit card was accepted.

Process the transactions asynchronously
Queuing is a great way to handle any work that the user should not have to wait around for. You could use it just when there is a connection timeout or for every single transaction.

Cupid.com was a .net application, so it made sense to use Microsoft Message Queqing (MSMQ) for this functionality. The web application writes a message containing the order ID. A queue runner reads the queue, using the order ID to select the order and payment info from the database. It attempts to process the order. If Auth.net times out or returns an unexpected response, the message is sent to back of the queue. It is trivial to build a time delay into the queue runner, otherwise you may find yourself making hundreds of attempts per minute for the same transaction. And it's not nice to kick your credit card processor THAT often when they are down.

RHW was a ColdFusion application, and now it is very easy to do queuing in ColdFusuion too. Our need pre-existed this functionality, so we used MSMQ. We wrote the queue runner in VisualBasic, and the connection to Authorize.net was done as a CF web service.

Once the payment is either accepted or rejected, do make sure you follow up with the customer. On RHW, if a card was declined after waiting in the queue because of a gateway outage, we would call the customer and try to receive alternate payment before we removed their posting.

The queue runner needs to also delete the payment info unless the customer authorized retaining it.

That's it. Once we had that coding in place, Authorize.net outages (or Verisign or iPay) were non-events to the development team. There was some alerting built in so that we'd know if the queue was building up, otherwise we might not even know that Auth.net was down. That is, until the accounting department came over and complained that they couldn't log into the payment gateway.

Audible.com Web Site is Lousy (To Be Polite)

My wife was surprised this week when I told her Audible.com has been charging her $14.95 every month for almost a year. She checked into it and realized she had signed up for a monthly subscription without even know it. That's a problem right there. Before closing the account, she wanted to redeem six book credits her account had accumulated. We decided to divide them evenly.

When it was my turn to select books, I was expecting an experience similar to Amazon.com. First of all, the Audible.com web site is very slow. Not snappy like Amazon.com, or any other website I use by choice for that matter. Seconds lagged each time I clicked a link.

Secondly, the category navigation is not very intuitive.

Thirdly, it looks like the site is quite different depending on if you are signed in or not. The site is more appealing when they are trying to lure you into registration, and frankly it's faster too. Tolerably fast. Audible.com offers fewer categories, perhaps to remove clutter and reduce load times.

Once you log in, its the slow site. Clicking on "Science Fiction & Fantasy" now and you can feel the delay before the page loads. You get to a page that displays, among other sub-categories for fantasy etc:

# Sci-Fi: Classic (217)
# Sci-Fi: Contemporary (555)

That is not a great number of titles. It gets more frustrating after clicking around twenty pages or so, the amount it took me to figure out "Contemporary" means anything since 1980. There is no way to browse with multiple filters, for example all "contemporary" sci-fi that was also on the New York Times best sellers list. The closest thing Audible.com offers is all 555 titles, sorted by how often they are sold on Audible.com.

I clicked on that link, got frustrated at how long the page was taking to load and was able to type "I'm composing this" before the page loaded.

Suffice it to say, Audible.com: now that Amazon.com has entered the audio book market place, you web site needs a huge overhaul to be competitive:

1) Speed. The competition is a click away.
2) Breadth of content: I understand sci-fi may be your weak area, but if you don't have what I want for all my audio book needs, I'll go elsewhere.
3) Navigation. Browsing a book store is easy and pleasurable. You're website needs to be even easier. Let me go to sci-fi, and then narrow down to New York Times best sellers. Then let me sort by newest first. By letting me narrow in, and showing me what path I am on, you would let me feel like I am getting closer to the perfect purchase. Otherwise I'm lost in the woods, and leave.

No ESAPI in Python yet?

I'm considering Python for my next big project. I've been doing ColdFusion for the last 10 years but am liking the karma from open source. So I've been digging into the Python documentation, finished reviewing the Python Standard Library and the Django tutorials.

Next, I wanted to look at some reference implementations in Python as a way of further familiarizing myself with coding best practices. I had stumbled across an implementation of OAuth in Python last week but wasn't ready yet. Specifically, I wanted to look an a reference implementation of ESAPI, the Enterprise Security API from the brilliant folks at OWASP.

Surprisingly, all OWASP offered was the referece implementation in Java, plus some "under construction" pages for .Net and ColdFusion. Elsewhere on the web I found an implementation in pHp, but nothing in Python.

So I know the whole idea of open source depends on the community effort, but I'm not thinking ESAPI can be my first Python app. I'll work on gaining l33t Python skills, but in the mean time it'd be great to see ESAPI in Python. Django community, I nominate you guys!

Google, why do you hate SEO so much?

See the post on GetClicky blog re:Google's new Ajax-powered search results breaks search keyword tracking for everyone.

In a nutshell, the referrers you get from Google SERPs may no longer have the keyword phrase on the query string. So good luck trying to figure out how your customers are finding you!

Web Developers: Don't Reinvent the Non-secured Wheel

I'm thinking about beginning another web project. Before one gets coding the fun part of any web application, though, there is tons of core code that needs to we written: login, user management, session management, user registration, logout etc. But writing secure applications can be tricky, and any attempt to roll your own is likely to have security flaws. Open source can solve both of these problems: the code is already written, letting you get onto the fun stuff; and if it's been vetted by a large developer community already you get the security benefit of past mistakes fixed.

One might think that reference implementations would be readily available for these in all web languages, and that we would all be using them by now.

Kudos to OWASP for developing it themselves, in their Enterprise Security API (ESAPI) Project. It details all the functions that a secure application needs. Much more, though, they also offer reference implementations in Java.

I'd love to see the web development community support this project by developing reference implementations in pHp, ColdFusion, and .net. Implementations in each of the popular frameworks would go a long way toward making the web a safer place, and would make the development of every new web application that much easier.

Journalspace.com dies instantly, for lack of a smart CTO

Hearts out to anyone who blogged on Journalspace.com. The service is dead now, in a heartbeat, because they did not back up their data. Ever, apparently.

Hard to believe a website can remain popular for six years, whilst its IT team merrily whistle through their work day without once stopping to think about data backup.

Maybe I'm myopic, but I've seen this happen with companies started by business and marketing people without a technical stakeholder, albeit no implosion has been so instantaneous. Not everyone can be technically minded, but if you aren't, and you are starting a dot com, better hire someone who is, give them a stake in the company, and listen to them about things like contingency planning.

So, what would you do if your data was lost? This question applies to home users and business people alike. As a CTO, this question should keep you up at night, in many different manifestations:

- What if a HD in the database server goes?
- What if the whole database server blows up?
- What if your web server blows up?
- What if your data center goes off line?
- What if the CEO looses his laptop?
- What if someone hacks into the development environment?

There are hundreds of variations on this theme. Good sleep is for the naive, and the retired, and those that have worked very hard for high availability, disaster recovery, and security.

When does 9 mins = 0 mins? When it's Google Apps SLA

TechCrunch covered this. In a worst-case-scenario nutshell, Google Apps can be down 90% of the time and be considered 100% up, if it is "up" for at least one minute (or to be pedantic, instant) for every nine minutes of "down."

Commentors at TechCrunch surmise that is not a real-world scenario where a web site can be up for one minute then down for nine for a persistent amount of time.

I wish that were true. I recalled working for iWon.com during the original dot com bubble. Version 1.0 of that site was written against Vignette CMS. The cost for that was rumored to be around $1M per processor (that was the talk around the office) but we had piles of money to burn (and give away).

The application was so unstable that, by the time I left, we had about 24 web servers in the cluster, and each was rebooted every 7 minutes. The fine folks at Vignette gave us that "work around" with a straight face. My brother was still with iWon and helped them move to open-source Tcl.

Since then, I've made sure I don't even hold any mutual funds that have Vignette stock.

Gizmodo site down

wtf? Gizmodo site is down? How am I to goof off now?

LinkedIn valuation leads to site downtime?

LinkedIn is really lagging this morning. I've never experienced lag on LinkedIn before, so maybe its the news of their 1 billion valuation. Maybe LinkedIn will now be hip and trendy, and downish, just like Twitter!

Downtime killed Friendster when it's valuation was through the roof.

So LinkedIn, take some of that new money and go shopping.

URL spoofing

I love it when web sites allow you to enter anything you want on the url, because silly people like me are easily amused. Here's fark.com's example.

Too high brow? Here's my first mod.

Odysseus in Firefox

I've been blogging here for 361 days. :D My first blog was about using Odysseus as a proxy server with IE for testing web pages. I shared that blog with some colleagues recently, and the balked because, you know, IE sucks. But you can use this great tool with Firefox or any HTTP tool, also.

First, download Odysseus, and run it so you can see the little Roman helmet in your system tray (Windows users). Its got a red ... feathery-decoration thingy on top. To turn that green, and thereby turn on this proxy server, right click on it and select "Interceptor."

To configure Firefox to temporarily use Odyssesus as a proxy server so that you can see, and edit, outgoing GET and POST requests, and to view the REPLY from the web server with all the headers, in Firefox choose Tools > Options... Advanced, Network, "Settings" button, select "Manual proxy configuration:" and enter "localhost" as the server and "50000" as the port. Confirm the dialog boxes.

Now, when you load a web page in Firefox, an Odysseus window will pop up letting you see (and edit) the HTTP data. Test away!

When you are all done, go back to your browser and turn the connection option back to normal.

Wikia Search dead?

search.wikia.com was released January 7, 2008 as an open source search engine enhanced by user submitted content. But looking at http://search.wikia.com/wiki/Search_Wikia, the wiki about the project, there has been no news in over a month. And the forums look dead too. This could be a really interesting project, but seems like the open source community hasn't embraced it yet. Maybe we're tired of Jimmy? Maybe he's tainted?

If anyone is interested in working on the concept of open search, I recommend Lucene which is the engine Wikia Search is based on, or solr which is an enhacement of lucene. I've done a solr proof of concept in the past several months, and I love what solr can do. I think I'll be spending time their. Google can't be the final solution to search.

Compete Aquired

Mashable reports Compete Acquired by TNS for $75 Million. Compete offers free public web metrics, like Quantcast. Unlike Alexa, in that it doesn't suck.

I predict that in five years either Compete and Quantcast will be purchased by Google, and the new company will *replace* Nielsen/Net Ratings and comScore. Information wants to be free, and the industry desperately needs a trusted third party to step in so we can all start comparing traffic using the same measuring stick. Mergers and acquisitions rely on this data, and beyond the top 10 sites, Nielsen and comScore methodologies can't see through the data storm.

The survivor will combine data that works by getting anonymized data from a significant portion of ISPs, like HitWise does, and data gathered from javascript that can be copied and pasted onto your website -- does this part sound like Google Analytics to anyone? Google also has the peering relationship with ISPs to get the first part done. And they have the "preventing fraud" chops to get it done right.

Nielsen will not be trusted any more then it currently is (not a lot) and it's Golden Age has past. comScore hasn't shown it can do a better job. Long live {insert successor here}!