post

Revamped isbn.net.in

New & revamped isbn.net.in

New & revamped isbn.net.in

 

Because of constant badgering from loyal isbn.net.in users (Navdeep, Chinmay, Hari, Sandip, Vidyaraj, Kartik, Vivek, Arjun, Leo, Ravi and others), I finally had to dedicate a weekend to fixing up isbn.net.in. However, instead of just fixing up the old code base, I rewrote it to use Compojure instead of the deprecated Noir library, and along the way, I re-did some of the code design to make it more flexible for editing and debugging.

I can’t believe people still use the site 3 years after I wrote the first version and put it up, especially with so many comparison shopping sites for India announced in these 3 years which are more functional and cover more categories. But, hey, can’t argue with those users :)

Caveats: It’s still a work-in-progress, the JSON API, etc. are still not present in this version and more ecommerce stores have to be added, will work on those going forward. And I can use the help if anybody has time, it’s open-sourced at https://github.com/swaroopch/isbnnetinclj2.

Advertisements
post

Learning Clojure

I once happened to attend a RubyConfIndia talk by C42’s Steven Deobald who said:

data > functions > macros > compilers

That kind of stuck in my head even though I didn’t know what it meant at that time. I understood it only after learning Clojure and “The Clojure / Lisp way”. I realized it when I was writing Python code for work, and I suddenly noticed I was writing code differently and I had one of those good aha moments that is supposedly the start of a person’s Lisp journey.

I’m now amused at how often I break down my Python or Java code into lots of little functions instead of the 100-liner functions that I used to write before and am still surprised that I never realized I was writing them! The good thing about the “lots of little functions” is the modularity and the ease with which I can write, read, understand and importantly test the code without having to build an object hierarchy first.

For example, my code has now suddenly started looking like this, where data structure is explicitly written down and the processing code is separate from it – this makes the code really reusable. It is a contrast to my earlier programming style where I would’ve probably had the data structure implicit in the parsing code (which makes it less maintainable) or worse, had classes and objects to do the same and it would certainly have not been so reusable! Think of a typical Java programming workflow where I would have had to create a class to represent the data input and passed that to a processor class instance and so on.

# http://www.lexicon.net/sjmachin/xlrd.html
import xlrd

DATA_SHEET_NUMBER = 0
START_ROW = 3 # skip headings

# Explicit structure of the data
COLUMN_MAPPING = {
    'name' : 0,
    'class' : 1,
    'maths' : 2,
    'geography' : 3,
    'english' : 4,
}

def row_to_dict(sheet, row_number):
    assert isinstance(sheet, xlrd.sheet.Sheet)
    assert isinstance(row_number, int) and row_number > 0 and row_number < sheet.nrows
    # Code that will work with changing structure
    return dict([(key, sheet.cell_value(rowx=row_number, colx=COLUMN_MAPPING[key])) for key in COLUMN_MAPPING.keys()])

def import_excel(content):
    book = xlrd.open_workbook(file_contents=content)
    sheet = book.sheet_by_index(DATA_SHEET_NUMBER)
    # Code that will work with different spreadsheet formats
    sheet_data = [row_to_dict(sheet, row_number) for row_number in range(START_ROW, sheet.nrows)]
    sheet_data = [data for data in sheet_data if len(data['name']) > 0] # Ignore empty rows
    return sheet_data

if __name__ == '__main__':
    from pprint import pprint
    pprint( import_excel(open('test.xls', 'rb').read()) )

To be clear, Python was a good first step, what changed was the mindset after attempting to learn a Lisp language. As Peter Norvig once said:

Basically, Python can be seen as a dialect of Lisp with “traditional” syntax (what Lisp people call “infix” or “m-lisp” syntax). One message on comp.lang.python said “I never understood why LISP was a good idea until I started playing with python.” Python supports all of Lisp’s essential features except macros, and you don’t miss macros all that much because it does have eval, and operator overloading, and regular expression parsing, so some–but not all–of the use cases for macros are covered.

A good friend of mine once said that Python is more popular because it is more approachable by traditional programmers and hence a more “social” programming language, whereas Lisp is a powerful language but not for everyone. That is explained in detail in the Lisp Curse essay.

So first good thing about Clojure is that it is a Lisp. Second is that it runs on the JVM which has solid performance, sometimes 20x better if you use it right. Third is solid Java interoperability. This was important to me because as a consultant, Java is unavoidable and I’ve written more Java code this year than I ever have. And using a good dynamic language on top of JVM with good Java interoperability is a path to making my work go faster. At least, that was how I got started. After all, your code will end up reflecting your company.

The downside I felt when I was grokking Clojure is that syntax is not simple even though that is the claim of traditional Lisps, for example #”” is regex, #{} is a set, #_() elides the form (compiler checks the code but acts as if it was commented out), #() is an anonymous function, #’ derefs to vars, and so on.

Here is a quick idea about Clojure’s philosophies that I was pointed to:

clojure three circles

Another interesting point is that functional programming languages are growing and it is probably because the future is DSLs again.

If you’re still not convinced, you should watch The Curious Clojureist. And you should definitely watch all the Rich Hickey talks.

How to learn Clojure

The O’Reilly Clojure book is best book that I’ve come across yet.

However, equally important, my strong recommendation is that Clojure is good only when combined with Emacs and ghoseb’s emacs setup. After learning Clojure in that environment, writing Python again makes me miss so many goodies (To get up to the same productivity in a few ways, I’m using PyCharm these days and am enjoying that).

To make my learning solid, I rewrote isbn.net.in for the third time in Clojure. The source code is at https://github.com/swaroopch/isbnnetinclj – be prepared to read some amateurish Clojure code.

I got a lot done in ~280 lines of Clojure code compared to 480+ lines of code in Ruby/Rails and a ton more boilerplate code. This difference in number of lines of code repeats often.

One interesting point is that because of the Clojure way of thinking, I ended up using a simple combination of future and core.cache to do the fetching of prices from book stores in parallel rather than bringing a full-fledged background jobs processor (delayed_jobs) to do that which vastly simplified the system. You can read that code in stores.clj.

Ending Thoughts

I got started with this journey because of frustrations with Java and at the same time I was trying to be not be narrow-minded with experience in just Python/Ruby/Perl languages (they are so similar). I kept reminding myself of what Douglas Crockford said:

WHAT WERE THE TRAITS OF THE WEAK PROGRAMMERS YOU’VE SEEN OVER YOUR CAREER?

That’s an easy one—lack of curiosity. They were so satisfied with the work that they were doing was good enough (without an understanding of what ‘good’ was) that they didn’t push themselves.

I’m much more impressed with people that are always learning. The brilliant programmers I’ve been around are always learning.

You see so many people get into one language and spend their entire career in that language, and as a result aren’t that great as programmers.

Programming languages becoming popular is almost never about the merits of the language itself and rather just a virtuous cycle of availability of programmers or platform requirements – Javascript and Objective-C are popular because you have no other choice, not only because of the merits of the language. Similarly, Clojure is leveraging the JVM and whatever native platform it runs on and hence is getting that initial lift needed to make the language appealing since people don’t want to learn and start on yet another ecosystem.

This is best explained by Alan Kay himself:

Q: What should Java have had in it to be a first-quality language, not just a commercial success?

Alan Kay: Like I said, it’s a pop culture. A commercial hit record for teenagers doesn’t have to have any particular musical merits. I think a lot of the success of various programming languages is expeditious gap-filling. Perl is another example of filling a tiny, short-term need, and then being a real problem in the longer term. Basically, a lot of the problems that computing has had in the last 25 years comes from systems where the designers were trying to fix some short-term thing and didn’t think about whether the idea would scale if it were adopted. There should be a half-life on software so old software just melts away over 10 or 15 years.

It was a different culture in the ’60s and ’70s; the ARPA (Advanced Research Projects Agency) and PARC culture was basically a mathematical/scientific kind of culture and was interested in scaling, and of course, the Internet was an exercise in scaling. There are just two different worlds, and I don’t think it’s even that helpful for people from one world to complain about the other world—like people from a literary culture complaining about the majority of the world that doesn’t read for ideas. It’s futile.

Did you know that Lisp and Smalltalk are not so much in vogue because they were killed by bad hardware!?:

Alan Kay: Yes, actually both Lisp and Smalltalk were done in by the eight-bit microprocessor—it’s not because they’re eight-bit micros, it’s because the processor architectures were bad, and they just killed the dynamic languages. Today these languages run reasonably because even though the architectures are still bad, the level 2 caches are so large that some fraction of the things that need to work, work reasonably well inside the caches; so both Lisp and Smalltalk can do their things and are viable today. But both of them are quite obsolete, of course.

Lastly, I wanted to mention that my Clojure journey would not have sustained if it wasn’t for Baishampayan Ghose (a.k.a. @ghoseb, a.k.a BG) whose untiring answers to my dumb questions was instrumental in me finally gaining some understanding of Clojure and Lisp in general. Thanks BG!

P.S. Watch this 2011 talk by Alan Kay. As @ghoseb would say, Be prepared to blow your mind.

post

isbn.net.in open-sourced

I have been getting many emails regarding isbn.net.in since I put it out 9 months ago. For example:

Hi Swaroop,
I am following your blog for last year.
I have visited and used ur isbn.net.in service. It is a good idea.

To easy of mobile phone buyers, I wanna create a application like isbn.net.in for the mobile phone availability and comparison.

Can i do this as a Final year project? Is it take that much work?
To create that what should I know!

I have knowledge in Open source softwares, LAMP and basic python (From A Byte of Python).

and this:

Hi Swaroop,

Love your site isbn.net.in!

I am planning to start an online bookstore to sell used books. I need help to acquire book info like title, author etc. based on ISBN. Will you be able to help? Do you have an API that I can use? I see that you use Amazon. I was not sure if Amazon would have the details for books published in India. Please let me know your thoughts. Appreciate your help.

I actually wrote isbn.net.in as a means of learning Ruby, so I had written some hacked-up code using Sinatra. As it was my first Ruby project, the code was very amateurish and was not something I wanted to share.

Since so many people have been asking how it works, I decided it would be easier to just open source it. I took this opportunity to rewrite it using Rails 3 and try out Heroku as well. So, voila, Rails 3, Heroku and using MongoDB for tracking background jobs (to do the screen scraping) means a brand new isbn.net.in that is now live.

And the source code is now at https://github.com/swaroopch/isbnnetin (Note that this software requires Memcached and MongoDB to run).

The code is under an Apache license, so use it as you please.

Now that the source code is out there, there are some contributions that can make the site better:

  • Detection of ‘out of stock’ / delivery time
  • Detection of extra shipping charges / other hidden charges
  • Mobile version
  • Any improvements to the web design at all (I suck at UI design)

I would love any help on the above. If you’re interested, please fork away and when you’re ready, send me a pull request on GitHub :-)

Still Spreading

I am surprised to still see the response to isbn.net.in, mainly because I do zero marketing, and yet I still get 1-2 emails per day about it. That is amazing considering that the site has been there for nearly nine months and I didn’t do much other than write a blog post about it. I don’t tweet it or facebook it and all that jazz. I guess it’s a case of “Build it and they will come”?

For example, it was featured in startup articles, it was even mentioned in interviews of new online ecommerce stores that are coming up.

My favorite part is that people are using it as a bookmark URL (which was a use case that I had in mind), see @xrivatsan, @PurpleFoodie and @Shalin10

And I get emails and tweets when the site used to go down for even a few hours!

Oh, and thanks for all the feedback:

Shashank ND says:

“isbn.net.in saves a ton of time when trying to buy books online. Indian e commerce needs more of them. Thank you @swaroopch.””isbn.net.in saves a ton of time when trying to buy books online. Indian e commerce needs more of them. Thank you @swaroopch.”

Swaroop Hegde says:

“Been searching for Kochan’s Objective-C book for a day now but hitting out of stock everywhere. And oh, Kudos to @SwaroopCH’s isbn.net.in”

Nandan Dubey says:

http://isbn.net.in/ best site to compare price of the books visit it once before buying any online books”

Vinay says:

isbn.net.in is one simple and great site which is give you the best prices online. Great service for book lovers

The greatest flattery, though, is when it has inspired many other book price comparison websites.

For example, take the story behind mySmartPrice:

I chanced upon a book price search engine by Swaroop through a post on Flipkart. I will always remain thankful for that since I have gained a lot of insight by going through his blog. If you intend to ever start your startup then do subscribe to his blog.

A lot of users of Swaroop’s project liked the product. However many of them were asking for a way to start with the title of the book instead of the isbn number. I guess Swaroop has not made the changes since it was not his idea to start with and he might be busy with his new job at Infibeam. Coincidentally I am on the exit mode from my job to get into the startup phase.

One of the biggest inspirations for working on this project has been my own desire to prove to myself that I continue to be a geek even 5 years after I stopped coding to go to IIM Bangalore. There are many improvements that are required even now and I will talk about them in future posts. However I am already proud of the product that I have.

I will close this post by wishing myself and MySmartPrice luck and by thanking Tapas for introducing me to the wonderful blog of Swaroop and Swaroop for providing me such a nice homework to bring out the coder in me

There have been other sites such as MyDiscountBay that also seem to have taken inspiration from isbn.net.in. Update: confirmed via twitter.

And we seem to be spawning many more.

The irony is that I myself was inspired by isbn.nu. We all stand on the shoulders of giants :-)

Oh, and isbn.net.in is certainly just a side project, definitely not meant as a competition to these startups.

The single biggest crib that I hear is that one needs to know the ISBN number to use isbn.net.in – it seems that most people forget that I have a bookmarklet which solves that problem, sigh. I really don’t see the need to make a completely new search engine when all these ecommerce sites are already putting in effort to make a great search engine. But I guess people want everything online and don’t want to even use a browser feature, such is the Net effect! (If you do have an idea in mind how to solve this, as I mentioned before, fork away ;-))

This was my list of updates about isbn.net.in. Feedback welcome!

Update: Someone has written a Flipkart to isbn.net.in link greasemonkey script. Nice to see people building on top of isbn.net.in! On the same note, if you ever want to get the prices from isbn.net.in, just append .json to the URLs, example http://isbn.net.in/0142000280.json for the JSON data of the prices.

post

isbn.net.in updates

A while back, I released a side-project called http://isbn.net.in – a simple tool for comparing book prices in India. I received lots of feedback, suggestions and praise. I have updated it with fixes for the bugs reported and implemented most of the suggestions.

It was interesting to see people writing blog posts and linking to the corresponding book page on isbn.net.in as a “canonical page” about the book. I hadn’t thought of that.

Feedback

Lots of bug reports, suggestions and praise came via email, such as from Onkar:

“Nice idea with simple implementation. I am sure this will make my father happy. Thanks for your work. :-)”

And as expected, Twitterers were most vocal about it:

@saurabh says: isbn.net.in is awesome #recommended #ftw #awesomeness

@kranium256 says: isbn.net.in is actually quite bloody awesome!

@kr0y says: For all those who love to order books online, this site can really help you get a good deal http://isbn.net.in/

@abhinittiwari says: Awesome book price comparision engine! http://isbn.net.in/

@vineetmundhra says: A wonderful tool for comparing book prices in India http://isbn.net.in

@l0nwlf says: http://isbn.net.in -> a pretty neat site to compare prices of book

@yarooruvann says: http://isbn.net.in/ very good tool to compare book prices in India

@jasdeep says: isbn.net.in is awesome, thank you @swaroopch

@tan1337 says: Awesome!

And some of the blog comments were heartening to note as well, especially this one:

Chandan V says: I was searching for a book from past 1 week and was unable to find it. Thanks to you, finally I was able get my book at flipkart. It was like, I thought I’ll not get that book any where in Bangalore and I open my google reader to see your link. Bingo, I have placed an order and eagerly looking forward for the delivery. Thanks a ton. You do not know how much it meant for me to have that book.

Note that last sentence. That is the stuff that creators love! :)

Search by title

The biggest feedback was: “Getting ISBN numbers is a little difficult for everyone. Consider taking a book title as your input and searching prices based on that directly.”

I understand the motivation behind this. But unfortunately, this was what I was exactly trying to avoid! I do not want to build a search engine! That is a non-trivial task, as I’m sure you can imagine.

My idea was to piggyback on top of people who are already doing that well. For example, Flipkart and Infibeam are supposed to have the most titles for the Indian market. So my idea was this: Why not use those search engines which are being constantly updated and tweaked by those companies to search for the books, and then use the bookmarklet + isbn.net.in to compare the actual prices. I actually don’t want you to use isbn.net.in as the starting point.

If you still want to search by book title, then head on over to the new Google Product Search for India. The reasons why you would use isbn.net.in over Google Product Search, is that isbn.net.in is comprehensive, accurate, has latest prices (as much as possible), and helps you decide whether to buy the book using the full description and Amazon rating.

Fixes and Updates

Regarding the fixes and updates based on your suggestions, here is the list:

  1. Fixed error on multiple pages such as http://isbn.net.in/8190453025 (via @sudhiru) and http://isbn.net.in/0074637762 (via email from Abhinav Sood)
  2. Fixing fetching of prices from a1books, thanks to bug report from Amit Sharma
  3. Added link to Google Product Search for India, because of many queries to allow search by title.
  4. Added CoralHub.com to the list of online book stores that is searched.
  5. Linked to iglooo.in and bookase.com in the about page under the list of similar projects.
  6. Added a “generic grep” to make the bookmarklet try a little harder for sites that is not known in its default list – IIRC, this was a suggestion by @talonx
  7. Bookmarklet now works with Amazon pages, but for this, you will need to take the bookmarklet again from http://isbn.net.in frontpage
  8. Added Kindle prices.

Favorite New Feature

My favorite new feature is Kindle ebook prices because, sometimes, buying the Kindle edition is cheaper than getting the paper book. That’s what I did with Seth Godin’s new book.

Further suggestions and feedback are welcome.


post

isbn.net.in – One Place to find the best online price for a book in India

I had an itch – I wished there was a simple way of deciding whether to buy a book and where to buy a book. So I created http://isbn.net.in

isbn.net.in

The initial idea I had was to make a bookmarklet that will do everything – it will figure out the unique book number (the ISBN) from the current book page (whether a publisher’s site or any ecommerce site), and then search on all the potential Indian online book stores. I then realized that you can’t fetch from other domains because of the same-domain policy of AJAX (I could’ve used YQL or something like that, but I felt it was a slippery slope).

So I had to create a web backend that will do the searching on behalf of the bookmarklet and changed the idea to simply show a jQueryUI dialog showing the sorted list of prices.

Then I chanced upon http://isbn.nu and immediately said to myself: “I want that with Indian prices”. Since I was half-way there already, it took a few additional steps of buying a good domain name and configuring to use the simple URL format they used.

There was one major problem with the bookmarklet – on sites which already have jQuery, it used to conflict, and although jQuery itself can live with multiple versions side-by-side, I could never figure out if jQueryUI was loaded properly or not. I tried various things but had to give up in vain.

Finally, I decided the pop-up overlay thing was not important, and the bookmarklet can just simply take you to the correct isbn.net.in page directly.

So the “where” part of the question was answered.

I still had to answer the “whether” part of the question – that’s when a friend told me about Amazon ECS using which I was able to get the very useful Amazon ratings. Then I was able to get the image of the cover of the book and other details.

Then I came across bookseer.com which makes great book recommendations, so I included an automatic link to that on the book page.

So, after much ado, I present http://isbn.net.in to you. All the instructions are on the homepage.

If you have any feedback, please read the disclaimers on the homepage and the About page, and then send me feedback.

Implementation was a lot of fun – I used Ruby, Sinatra, HAML, Mechanize, amazon-ecs, jQuery, Blueprint CSS. It was the first time I had really used any of these.

Disclaimer: I created isbn.net.in because I needed a tool like this. This has nothing to do whatsoever with my employer. It is just a personal side-project.

Updates on Praise, Feedback and New Feature: See isbn.net.in updates article.