60/40 rule

From Lessons on Leading by Jessica Steel : Leaders recognize that we should feel 60% mastery and 40% stretch in our jobs at any given time. We draw our confidence from the 60% we know we’re nailing, and we grow from the challenge of the 40% that is new and uncomfortable for us. Leaders focus […]

Micro targeting

Recently the Obama re-election campaign employed similar population-wide behavioral analytics to micro-target voters to ensure his re-election. There is no reason why we in India must not look to technology to devise ingenious methods for near real time data collection and population-wide analytics of social performance. This will not only help micro-target and localise welfare […]

Back to Org Mode

A HBR article titled Smartphones, Silly Users perfectly describes why I have moved my personal information management system away from apps that sync across desktop and mobile:

  1. “We don’t remember anything anymore.”
    • “We’re increasingly outsourcing our personal memory banks to Google and other search engines, effectively wiping our own brains of easily accessible information.” a.k.a. the Google effect
  2. “We waste time preserving optionality.”
    • “We’re refusing to finalize our plans until critical moments. The ability to make reservations, check opening hours, look up driving directions, and review ratings on our mobile devices means that we’re increasingly iterating our schedules and keeping our options open until the very last moment before that meeting, lunch, or coffee catchup is set to begin.”
  3. “We get stuck in the infinite notification loop.”
    • “As we endlessly loop between Facebook, Twitter, LinkedIn, and other app notifications, our attention fragments, and it becomes difficult to focus on larger, more important tasks.”

Till this month, I was obsessed with syncing everything across my desktop and mobile. The problem was that I became obsessed with the mobile phone unnecessarily and once you’re using the phone, Point no. 3 kicks in – the infinite notification loop swallows a lot of time and attention.

Once I shifted my system to laptop-only, I don’t have all my tasks and calendar at hand, I’m forced to remember things (see point 1 above), and strangely, I’m more likely to remember things to pick up from the grocery store now than I was likely to remember to check my mobile phone app for things to buy when I was near a grocery store!

The most important thing is that notes and todos are in the same place, for example, if I’m on a call, I can take notes and then I can keep referring back to those notes while creating todos and working on tasks. The tasks come out of notes, they’re not separate! It really helps to have one system that can handle and encourage the normal flow instead of being forced to use separate notes and tasks apps.

Today, I’m all OrgMode. Again.

Read More »

Review: Big Data Book by Nathan Marz

Recently, I finished reading the latest “early access” version of the Big Data Book by Nathan Marz.

What is Big Data

Let’s look up Wikipedia:

In information technology, big data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, analysis, and visualization.

So, Big Data is relevant for any technical and business person whose company deals with lots of information and wants to make use of it. For example, Gmail search, etc.

Why this book is awesome

The book has been a fascinating and engaging learning for me because of two reasons:

First, it has a strong and simple “first principles” approach to an architecture and scalability problem, as opposed to the confusing (to me) and mushrooming complexity and treating Hadoop as a panacea in the Big Data world.

Second, Nathan Marz was one of the only 3 engineers who made the BackType search engine (the company was acq-hired by Twitter):

BackType captures online conversations, everything from tweets to blog comments to checkins and Facebook interactions. Its business is aimed at helping marketers and others understand those conversations by measuring them in a lot of ways, which means processing a massive amount of data.

To give you an idea of the scale of its task, it has about 25 terabytes of compressed binary data on its servers, holding over 100 billion individual records. Its API serves 400 requests per second on average, and it has 60 EC2 servers around at all times, scaling up to 150 for peak loads.

It has pulled this off with only seed funding and just three employees: Christopher Golda, Michael Montano and Nathan Marz. They’re all engineers, so there’s not even any sysadmins to take some of the load.

Note: BackType’s (now open sourced) real time data processing engine Storm powers Twitter’s analytics product and real-time trends among other things.

Read More »

Wrote an EDN format reader and writer in Python

I was reading about the EDN format over the weekend. EDN (pronounced like in “eden garden”) is a data format in the same league as JSON but is supposed to have some nifty features such as sets, keywords, date-time type, custom types, and also being a proper subset of Clojure.

Having a date-time type as well as custom types seems useful to me, so I was taking a look at the current Python implementations of the EDN format and I didn’t find them satisfactory, for example, one of the listed ones had all custom parsing code which was difficult to read, one was not even a real implementation, just boilerplate code, etc.

So I thought why not create a better implementation and I did – it is up on GitHub at https://github.com/swaroopch/edn_format.

It has been a long time since I did lex and yacc, so it was a fun weekend project :)

Created an app for live preview of Pandoc

When my wife was editing my books, she used Mou.app for live preview of the text so that she knows what the output is going to be like. The caveat was that Mou.app does plain Markdown and not Pandoc format which would mean the preview would be screwed up whenever there was a code block, etc., so, today morning, I hacked up an app called “Kalam” which does exactly that – live preview for Pandoc text.

The app is based on top of node-webkit (which I came across when I was wondering what Light Table is built upon), created by Roger Wang and others at Intel China Open Source centre, they’ve basically integrated node.js into webkit and disabled all the security restrictions, which makes it the almost-perfect cross-platform desktop toolkit – write HTML, CSS, JavaScript and use any node.js module!

Update: There’s also AppJS which is the same concept as node-webkit but looks more polished (via @aravindavk)

Delhi By Cycle

I had an interesting morning today cycling around Old Delhi guided by Arkash of Delhi by Cycle tour.

Seeing a city by cycle was a great incentive by itself and was fun. But I ended up with a case of the Paris Shock Syndrome because of the state of filth of the city, even the grand Red Fort is nestled in a pile of garbage and we saw the city’s poo and pee being dumped into the “holy Yamuna river.”

The fun part for me was cycling through parantheywali gully which is impossible to imagine in the evenings.