New & revamped
New & revamped


Because of constant badgering from loyal users (Navdeep, Chinmay, Hari, Sandip, Vidyaraj, Kartik, Vivek, Arjun, Leo, Ravi and others), I finally had to dedicate a weekend to fixing up However, instead of just fixing up the old code base, I rewrote it to use Compojure instead of the deprecated Noir library, and along the way, I re-did some of the code design to make it more flexible for editing and debugging.

I can’t believe people still use the site 3 years after I wrote the first version and put it up, especially with so many comparison shopping sites for India announced in these 3 years which are more functional and cover more categories. But, hey, can’t argue with those users :)

Caveats: It’s still a work-in-progress, the JSON API, etc. are still not present in this version and more ecommerce stores have to be added, will work on those going forward. And I can use the help if anybody has time, it’s open-sourced at

[Tech] Why is “Database Layer as a REST API” not common?

We have “database APIs” such as abstraction layers over multiple SQL databases and ORMs. But why not take it to the next step and make it a REST API like any other network call that we can make?

Database as a REST API
Database as a REST API

Advantages would be:

Did we just sort-of reinvent Datomic?

Of course, this is not a new idea at all, take restSQL as an example – my question is why is this not talked about more often?

Do most frameworks support this? If not, why not? If so, why don’t most frameworks don’t talk about such a use case in their documentation? If I use Django, I’ll start writing the models and use South to create migrations, and that’s that. If I have to reuse those model, from say, Java, then you’re on your own. The point is that, by default, Django (or Rails) doesn’t encourage you to do such a thing. If you go for a lighter framework such as Flask, then this becomes easier because the ORM is anyway not part of the framework.

Is this concept felt needed only in a polyglot case (multiple database systems, multiple programming languages)?

P.S. Also read Stevey’s Google Platforms rant.

Update on [2013-04-28 Sun]: Also see the very useful tech talk Designing a Beautiful REST+JSON API.

Micro targeting

Recently the Obama re-election campaign employed similar population-wide behavioral analytics to micro-target voters to ensure his re-election. There is no reason why we in India must not look to technology to devise ingenious methods for near real time data collection and population-wide analytics of social performance. This will not only help micro-target and localise welfare […]

Back to Org Mode

A HBR article titled Smartphones, Silly Users perfectly describes why I have moved my personal information management system away from apps that sync across desktop and mobile:

  1. “We don’t remember anything anymore.”
    • “We’re increasingly outsourcing our personal memory banks to Google and other search engines, effectively wiping our own brains of easily accessible information.” a.k.a. the Google effect
  2. “We waste time preserving optionality.”
    • “We’re refusing to finalize our plans until critical moments. The ability to make reservations, check opening hours, look up driving directions, and review ratings on our mobile devices means that we’re increasingly iterating our schedules and keeping our options open until the very last moment before that meeting, lunch, or coffee catchup is set to begin.”
  3. “We get stuck in the infinite notification loop.”
    • “As we endlessly loop between Facebook, Twitter, LinkedIn, and other app notifications, our attention fragments, and it becomes difficult to focus on larger, more important tasks.”

Till this month, I was obsessed with syncing everything across my desktop and mobile. The problem was that I became obsessed with the mobile phone unnecessarily and once you’re using the phone, Point no. 3 kicks in – the infinite notification loop swallows a lot of time and attention.

Once I shifted my system to laptop-only, I don’t have all my tasks and calendar at hand, I’m forced to remember things (see point 1 above), and strangely, I’m more likely to remember things to pick up from the grocery store now than I was likely to remember to check my mobile phone app for things to buy when I was near a grocery store!

The most important thing is that notes and todos are in the same place, for example, if I’m on a call, I can take notes and then I can keep referring back to those notes while creating todos and working on tasks. The tasks come out of notes, they’re not separate! It really helps to have one system that can handle and encourage the normal flow instead of being forced to use separate notes and tasks apps.

Today, I’m all OrgMode. Again.

Read More »

Review: Big Data Book by Nathan Marz

Recently, I finished reading the latest “early access” version of the Big Data Book by Nathan Marz.

What is Big Data

Let’s look up Wikipedia:

In information technology, big data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, analysis, and visualization.

So, Big Data is relevant for any technical and business person whose company deals with lots of information and wants to make use of it. For example, Gmail search, etc.

Why this book is awesome

The book has been a fascinating and engaging learning for me because of two reasons:

First, it has a strong and simple “first principles” approach to an architecture and scalability problem, as opposed to the confusing (to me) and mushrooming complexity and treating Hadoop as a panacea in the Big Data world.

Second, Nathan Marz was one of the only 3 engineers who made the BackType search engine (the company was acq-hired by Twitter):

BackType captures online conversations, everything from tweets to blog comments to checkins and Facebook interactions. Its business is aimed at helping marketers and others understand those conversations by measuring them in a lot of ways, which means processing a massive amount of data.

To give you an idea of the scale of its task, it has about 25 terabytes of compressed binary data on its servers, holding over 100 billion individual records. Its API serves 400 requests per second on average, and it has 60 EC2 servers around at all times, scaling up to 150 for peak loads.

It has pulled this off with only seed funding and just three employees: Christopher Golda, Michael Montano and Nathan Marz. They’re all engineers, so there’s not even any sysadmins to take some of the load.

Note: BackType’s (now open sourced) real time data processing engine Storm powers Twitter’s analytics product and real-time trends among other things.

Read More »

Created an app for live preview of Pandoc

When my wife was editing my books, she used for live preview of the text so that she knows what the output is going to be like. The caveat was that does plain Markdown and not Pandoc format which would mean the preview would be screwed up whenever there was a code block, etc., so, today morning, I hacked up an app called “Kalam” which does exactly that – live preview for Pandoc text.

The app is based on top of node-webkit (which I came across when I was wondering what Light Table is built upon), created by Roger Wang and others at Intel China Open Source centre, they’ve basically integrated node.js into webkit and disabled all the security restrictions, which makes it the almost-perfect cross-platform desktop toolkit – write HTML, CSS, JavaScript and use any node.js module!

Update: There’s also AppJS which is the same concept as node-webkit but looks more polished (via @aravindavk)

Why is Whatsapp so pervasive?

I’m astounded on how popular Whatsapp (messaging app for phones) really is:

  • My wife’s friend who runs a boutique went to an old market to buy cloth material for her shop – the salesman asked her to send the specific color she wants with a picture via Whatsapp. Think of an old dusty market and think of this again.
  • My wife’s friend who is a recent mom talks to her paediatrician via whatsapp for advice and general questions, and the doctor replies back (regardless of location).
  • An uncle and aunt in US go for shopping in the big malls and send photos to each other of whether they should pick up that item or not.
  • My uncle and aunt were in town and we went shopping – again, we sent photos of the T-shirts to their son in another town to ask whether he likes the shirt enough to buy it – decision done, shirt bought, no risk of a T-shirt going unused.
  • Recently, there was an incident in Bangalore because of which SMSes were restricted, which is like a heart attack for teenagers, including my sister – they all downloaded Whatsapp and shifted to it in an instant. “SMS costs – be gone!” Luckily, Whatsapp was free for a day or so in the iTunes app store around the same time and my sister, who is using my old iPhone (which is still working after 4 years) and does not have a credit card, grabbed it with eager fingers and is loving it.

I could go on and on, the point remains that the pervasiveness of it still surprises me. And it surprises many of my peers who grew up with email, Yahoo! Messenger, Google Talk, etc.

So I was imagining what could be the reasons that Whatsapp is so popular, and here are some wild guesses:

  • 2G on mobile is finally affordable? Now that 3G is more common and has been around for a couple of years, the slower predecessor has finally become cheap enough.
  • WiFi is more common now?
  • BlackBerry “BB-PIN” popularized the concept of instant messaging to a new phone-using generation, but people needed something cross-platform and Whatsapp was in the right place at the right time?
  • Whatsapp is available on most mobile operating systems including many older generation platforms such as Symbian, so people are not left out of the conversation.
  • Why didn’t people simply use GTalk? I’m guessing it’s because of the “create a Google account” barrier as well as GTalk not being as feature-full?
  • Talking about features – groups, photo sharing, video sharing is a natural extension that was just meant to happen, Whatsapp makes it free (as opposed to SMS/MMS)
  • The details in Whatsapp are great – for example, every message has two ticks – one that says it has gone from your phone to the server, the second tick shows that it has gone from the server to the other person’s phone – an in-built message delivery status as opposed to guessing whether the SMS has reached the other person
  • Did I mention how useful the groups feature is? I’m keeping in touch with friends all over the world through the same – in particular, one group has people in USA, Singapore, India all in one group and having a conversation at the same time.
  • So why didn’t email do the same? Because people have a work email/personal email distinction whereas a phone is undoubtedly personal? Because people don’t like to differentiate between a subject line and a body line (don’t laugh, I did too until I realized this is actually a barrier), they just want to “chat” because people are already familiar with SMS?

These are just my rough line of thought about this whole thing and I just wanted to write it down because many of my friends have asked the same question.

What are your thoughts?