We were discussing cricket with Prof Montealegre at a small get together that we had last Thursday. The conversation went something like this:

Prof Montealegre : So, I hear it takes a whole day to play a game of cricket.. that's quite fantastic!
ISBian Cricket Buff #1:Sir, there is another format that lasts five days.
Prof Montealegre : What! Oh.. that's amazing! So, which are the good teams?
ISBian Cricket Buff #1: Australia is the best. South Africa's good . India is good too. Pakistan are alright, but they play their best against India.
Prof Montealegre : Really?
ISBian Cricket Buff #2: Yeah, Indo-Pak games are very sought after. The roads are empty when there's an India-Pakistan match. Everyone takes off, nobody works.. and there are firecrackers if we win.
Prof Montealegre : Hmm.. very interesting. So, when is India's first game?
ISBian Cricket Buff #1: Day after tomorrow. We're playing Bangladesh.
Prof Montealegre : Okay, are they a good team?
ISBian Cricket Buff #2: Well, they're new, they have a lot to learn. They'll learn from us.
Prof Montealegre : Hmm.. nice, nice.

I hope the Professor has followed up on the latest.

Tale of two networks

I think I'm getting slightly obsessed with this topic of web-based social networks, so I solemnly swear this'll be my last post on this subject. For a while, in any case :)

Some people believe that it takes a maximum of six people to link up any two people in the world. This is a familiar theory called the six degrees of separation. Mathematically, this makes sense. Let's say every person on earth knows 500 people. I don't think 500 is too high a number - I have 472 people on my orkut list, which include seven people I haven't ever met (they could be fake for all I know), but I think I know more people who don't have an orkut account. The 500 number means that through six other people, I could theoretically form links with 500^5 = 31250000000000, which is definitely greater than the population of the earth. Yes, there might be repeats, but there's a good chance I could cover everyone. At least that's what the theory says. But here's where I disagree -- there might be some clusters of people which are so remotely connected from the rest of the world, that they might not have a link. This is especially true with tribes that have been untouched by other civilised humans.. so it's difficult for me to believe in the Six Degrees theory outright. Nevertheless, if you keep increasing the degrees - from six to eight to maybe 15 - you are bound to connect most humans on the planet. For me, the more interesting fact about the "Six Degree of Separation" theory isn't in the number six but that most of us are connected to each other, most of us know each other. At least indirectly.

Let's draw a parallel to this in the Internet space - quite arbit, but please bear with me. Let's say that email addresses are the equivalent of humans. Well, they are born, they live and then they die. And while they're living, they contact other email addresses. How many degrees separate any two email addresses in the world? Maybe six, maybe more, maybe infinite (you do have closed loop intranets where all email addresses are internal to the network - pretty much like the tribes that have been left in isolation). If I assume the equivalent of knowing another person is an email sent from on email address to another, how many degrees would separate any two email addresses on the Internet?

Here's what I'm getting at. There are two networks - one of humans and one of email addresses. These two networks are separate, they are distinct, and they should not be confused for each other. If I am serious about Web 2.0 in any form, I have to draw a distinction between Internet users and the means they adopt to communicate on the Internet.

What orkut & yahoo! 360 and Linked in have done is to differentiate between the two networks. On Linked in, you don't invite an email address to hook up with you, you invite a person to join your circle. Even on orkut, if I ignore the malicious bots created by crackers (crackers, not hackers), it is people that I am connecting with. When the programmers behind this system forgets that he's trying to connect people and not email addresses, you end up getting a system like tagged.com. The website starts trying to find more and more email addresses to send spam invitations to in the vain hope that viral marketing will work. Well, sir, it doesn't. If gazzag.com were to get hold of my addressbook and sent invitation to everyone on it, they'd send invites to email addresses like that of the placement office at ISB. They won't be inviting the people working in the placement office, but the placement office itself - now what kind of a social network is that?

Lesson for running a web 2.0 site that bases its growth on viral marketing - let the users decide whom to invite and whom not to. That would be social; cracking through users' addressbook is not.

First Gazzag, now Tagged & Facebox

The site www.tagged.com is a new social networking site. I haven't been tracking on its popularity, but the site claims to be the fastest growing teen social community on the Internet. From the corporate information available on the site,

Tagged.com is the premier social networking destination for the Millennial Generation and an ideal place for advertisers who are trying to reach the teen market.

OK, so that tells me (and my friends) to stay away from the site.. we've not been teen-agers for quite some time now. But there are bigger problems with this site than that.

A few months back, I had written the articles de-gazzaging my inbox and more on gazzag about a popular web nuisance that called itself Gazzag. I think tagged.com is just another example of such Internet annoyances. Let's walk through tagged.com's registration process to see why:

Step 1
OK, here's the website. It does look a whole lot uglier on Internet Explorer with some nasty banner advertisements. Anyway, let's create a new account on this.

Step 2
Yep, filled in my details. But take a look at something that I've done - I specified my date of birth as the 31st of February, 1978. Not only did the website allow the non-existent date, it didn't reject me because even though I am not a teenager. I'm wondering what happened to their tagline of being a teen social community.

Step 3

Whoa! Give me a minute.. why does Tagged.com want my hotmail password?!? Yep, one reason only - to get my addressbook, so that it can spam all of my friends with invitations to Tagged.com. Another example of viral marketing in which the existing users don't get a say in who gets contacted. What's more, there doesn't seem to be a straightforward way to get around this screen. I guess the naïve user would not look to bypass this screen and simply provide their hotmail password. While I didn't check if the tagged.com guys have devised mechanism to import contacts from yahoo and gmail, I won't be surprised if they have.

The same problem exists on the website facebox.com, another social networking site. When you register on this, there does not appear to be any way to create your account without importing contacts from your e-mail address books.


But if you click on the "by email address" option, you'd get the option to bypass this step:

Nasty, isn't it?

Let me re-iterate what I had written earlier about gazzag and other sites that ask for e-mail passwords:

... The tool that allows one to import contacts from yahoomail & orkut requires one to enter the username and password of those sites. And a lot of my friends, including a few tech savvy ones, have done exactly that. My question is why??

Would you trust me with your yahoomail password? I think not, I hope not. Then why would one trust gazzag or any other site asking for your yahoomail password? ...

Tagged & facebox are just a few more site that are trapping users into sharing their addressbooks. But the point is that it could get worse.. what if someone in Tagged.com's office is reading your e-mail? Their privacy policy will probably mention that they won't - but how do you know they're going to stand by that? Anyways, the next time a site asks you for your hotmail/gmail password, please think about if you'd like to share your password with me.

If you've sent me a tagged or a facebox or a gazzag invitation recently, I hope you'll understand why I've not responded.

Work-ex for an MBA at ISB

As an ISBian, I keep getting queries from potential applicants whether their profile is good enough to be selected at ISB. One of the more worrying aspects for many aspirants is the amount of work experience they need before they can apply. ISB has a policy that reads

Preferably two years of full-time post qualification work experience.

We'll come back to my interpretation of the "preferably" in this statement in a short while, let me look at the aspirants first. Some of these people are really young; it's not rare to come across people in their final year of graduation and they go - "How should I spend my next two years before I apply to ISB?". Whew! talk about focus! Do you know why you want to do an MBA, I ask. And the reply usually starts with "ummm..."

Back to the subject - the number of years of work-ex is just one method of quantifying how much you have worked. The other method to measure the work-ex, perhaps not quantitatively, is to look at one's résumé. When one writes one's résumé, it takes a single line to mention how many years one has worked. What about the other lines? What is the story that they tell? Do they speak of a leadership that raised the value of your organisation? Does it speak of a benchmark you set for yourself, or for others? Does it speak of knowledge? Does it say that you chased your professional desire? Well, what does it speak of?

At the end of the day, when ISB's admission committee receives a huge bunch of profiles, it's the résumés that are going to get compared (yes, there are some other things that get compared too - a GMAT score, recommendations, essays, yadda yadda yadda - but let's stick to CVs for now). Naturally, a person who has worked for long would have a greater chance of having a richer résumé than one who's worked for say, just a year.

I also came this article earlier today: Opposing the Youth Movement. It justifies business schools seeking only those who have lots of work-ex. Essentially,

The whole concept of an MBA program is based around peer-to-peer learning—people coming with diverse experiences into a structured framework, where they're learning potentially how to run a business. So I don't feel that people who have experience are going to gain as much from [a program that admits undergrads]

This is probably a good description of the thought process running at ISB.

Right, so what can an ISB aspirant draw from all that's written above? Here would be my quick list:

1. Forget an MBA - focus on your career. When the time to think about an MBA comes, you will know.
2. Pick a career that gives you exposure to multiple aspects of business. B-schools value diversity all the way - in fact, a person who has spent two years in a programming job, followed by two years in sales might have a better looking résumé than one who's done four years purely in sales. Not trying to make a blanket statement here - but do think about it.
3. Try to stay with an organisation long enough to have seen different aspects & levels of the company's hierarchy. Unless you've spent enough time to experience hierarchy first-hand, it might get difficult to learn from it, or comment on it.

If you prefer to comment or ask questions by e-mail about this post, my address is pqtsocss@trashmail.net.

0417 hrs - 2nd platoon Bravo team being overrun by the Viet Cong
the heroes - Charlie Sheen, William Dafoe, and a few others
the movie - Platoon
the sound - rapid fire, going rat-tat-tat...

1043 hrs - the Professor's voice overrun by the typing of students
the heroes - my classmates
the lecture - Web 2.0
the sound - frantic typing to complete an assignment, going rat-tat-tat...

Nike & holi

Managed to come across some very good advertisements recently, thought I'll write about them:

Nike



Not only has the airing of this advertisment been timed very well with the world cup season, it also captures the Indian love for the game very accurately. As soon as the advertisement starts, the traffic stops, and the cricketers take over. It doesn't matter who's bowling or who's batting.. it's just the game that has to be played.. the game has to go on till the fat lady sings, or in this case, the traffic starts moving.

When the first batsman hits the ball too hard and the bowler goes "kya kar raha hai, yaar?", it's almost like the kids playing in a neighbourhood street with rules like "if you hit the ball beyond that wall, you're out". There's more that makes the ad very Indian - the chicken that pops its head out of the basket, the kerosene lanterns, the elephant that adds to the madness, the confused thulla.. it's hard to make out it's a Nike ad except for the less than half-a-second frame where the familiar swoosh pops by.

---

A donation
A friend shared this link with me on the day of Holi. Unless you've seen this before, make sure you watch this clip till the end.



I'm surprised on not having seen this on television. I would expect this to be a lot more effective than the Aishwarya Rai advertisements that used to be aired on the same subject.

Placements at IIMI, the Chronicle version

A friend of mine forwarded me this article from Central Chronicle about placements at IIM Indore: http://www.centralchronicle.com/20070303/0303102.htm

Here is Chronicle's version of the companies that came visiting IIMI:

This year several renowned companies visited here to offer placements to the students; some of them are Macancy, Assenger business consultant, ... Deloty consultancy, ... Lehmen brothers, Goldman Saches, Duche bank, ... City group, ....

Some IT companies include .. Cognijent, ... Kotek Mahindra and Kevonsis etc.

:-)

3,2,1: My placement story

shortlists - 3
offers - 2
accepted - 1

Hyderabad = home for quite some time in the near future :-)

BTW, does anyone know a place that I can rent?

Using awk to parse HTML

Sorry for disappearing away like that… we had lots of stuff happening on campus (read our placements). And between tracking job posts and interview preps and and the interviews themselves, my blog was the last item on my priority list. Anyways, now that placements are done (and successfully for me), let me get back to the second post about working with data from the web.

The objective
One thing I did not make clear in the last post was the objective of our exercise. What we were trying to build was a model that could suggest the price for a user if (s)he decides to put up a car for sale. The model would be based on how people are currently pricing their used cars.

So, in order to build a model, we needed to work up some data from www.autoindia.com, as we did the last time. The next step is to convert all of that data in the 17,000 html files into a single spreadsheet. Each listed car on the website should have its details on a separate record in the spreadsheet.

In this post, let's explore how to do this using awk.

A simple sample HTML
Let’s take a look at the listing http://autoindia.com/UsedVehicle/buy-used-car130625.html. The webpage looks like this


In order to build our spreadsheet, we need all the data such as the year of manufacturing, the make of the car, the model and the style. Let’s take a look at the html source code.


For most of the variables that we’re interested in, there are specific identifiers that have been written in the html, like lblYear, lblMake and so on.

Given all such identifiers, it becomes a simple programming task to list out all the lines that match these identifiers, get rid of the tags, and bingo! we’ll have all the information that we need.

Let’s try out a simple awk script to see how we can do this.

The first awk script
I’m not going to delve into the details of how awk works. You can find out more detail on the GNU awk guide at http://www.gnu.org/software/gawk/manual/gawk.html Yep, GNU’s implementation of awk is known as gawk. Though I have used gawk for all of the examples, I’ve interchangeably used the names awk and gawk. The name’s inconsequential, the result is the same.

Back to our job in hand, we’re interested in specifying a pattern and printing out the details of the lines that these patterns match.

/lblYear/ ||
/lblMake/ ||
/lblModel/ ||
/lblStyle/ {
print $0;
}

In the above snippet, we’re telling awk to print the entire line (represented by the $0) if it matches any of the regular expressions listed between the two forward slashes.

If we save the above snippet into a file named “split.awk” and run the following command

C:\> gawk –f split.awk buy-used-car130625.html

We get the following output:

2002
Hyundai
Santro
1.1 GS Zip Drive

Eliminating the span tags
Let’s look at the above information closely – if we can eliminate the stuff between and including the angle brackets, we’d be left with exactly the information that we need. Let’s modify the awk script we wrote earlier.

The two gsub commands are global search and replace commands. They will look for the regular expression (specified as the first argument) and replace it with the substitution string ( the second argument -- in this case, a blank) in the text that is specified as the third argument). Now that we’ve wrapped that up in a neat function, let’s look at the output that this gives us

C:\> gawk –f split.awk buy-used-car130625.html
2002
Hyundai
Santro
1.1 GS Zip Drive

In a spreadsheet, please!!
The last teeny-weeny bit of stuff that we need to do is to plug the above into a single line, preferably in a comma-separated or tab-delimited format, so that it can open up in a spreadsheet program like MS Excel.

We can use a simple trick. Each time the pattern is found, instead of printing the output after stripping off the tags, we can concatenate that output to a string, that can be printed off later using the END code block of awk.

Here we go…


And the corresponding output

C:\> gawk –f split.awk buy-used-car130625.html
2002 Hyundai Santro 1.1 GS Zip Drive

If I decide to redirect this into a file using the > operator of DOS,

C:\> gawk –f split.awk buy-used-car130625.html > file.txt

I can open file.txt using MS Excel (by using the tab-delimited option) to get something like this:


Expanding the above
The last bit that remains is to run the above technique across all the HTML files that we’ve generated. This can be done by writing a single batch script in DOS with the awk command that we’ve used for all the 17000 HTML files that we’ve downloaded

Of course, since we needed more information than the simple example that we’ve seen above, our split.awk script ran into many more lines. There were some fields that we handled using associative arrays

Our final code looked like this:

And our output in a spreadsheet:


Downloading awk
If you’re going to use awk for the first time, I’d recommend you try it on a linux or Unix machine. There are a whole lot of other tools that gel well with awk, using the pipe mechanism that Unix scripts provide. Plus, I’ve still not been able to write simple awk scripts that can be written on the windows command line. Even simple stuff like

C:\> awk ‘BEGIN{ print “hello world”; }’

fails miserably. Probably the quotes or something – haven’t been able to figure it out.

Nevertheless, if you do need to use awk on Windows, then awk comes as part of an entire Unix toolkit available at http://www.delorie.com/gnu/ Try the zip picker – it should give you the entire stuff – awk, sed, the whole nine yards.

Alternately, you could try using cygwin. Runs a bit slower as it uses something called a Unix emulation layer. But it's the closest replication of a Unix environment on Windows that I've seen, and is packed with all the unix tools that one can imagine.

Additional reading

1. The GNU Awk User’s Guide - http://www.gnu.org/software/gawk/manual/gawk.html
2. Unix classic tools - Awk Programming - http://www.softpanorama.org/Tools/awk.shtml
3. Awk man pages for Linux - http://www.die.net/doc/linux/man/man1/awk.1.html
   

Other Comments

In response to my previous post, Punit had asked me why I didn’t use RSS and an XML parser - here's why:

1. It didn't look like autoindia.com has been refreshing it's RSS feed (I wasn't able to subscribe to it) and the XML file that's on the server contains some very old data from 2005, and about 15 records from 2007 - it was insufficient for us.
2. RSS is typically used to only publish the latest n items that the site has listed. n could be 15, 100, or 1000, depending on the webmaster. If you want all the data that a site has, like we wanted, you'd be extremely lucky to get that off RSS. In our case, even if the RSS feed has been refreshed regularly, we would have needed to monitor it over a long period of time (probably a year) to build a sufficiently large data set.
3. Depending on how the webmaster has configured the XML, the description may not contain all the fields that one is looking for. In our case, we wanted everything - from the price of the car to the location to whether it has power steering. The site's webmaster didn't publish all of that. So, RSS was clearly out.

awk was useful for us in this exercise because of the way html is structured. Had the html been a little more complicated, I’d probably have had to use perl and an HTML parser module off CPAN. In the end, using awk has to be a conscious judgement decision – it can give you results quickly, but you need to be careful – it can be quite painful to debug. And if you’re the kind of person who doesn’t believe in documenting code, you’ll end up having loads of fun :-)

Our project
What I've described above was just the initial part of our project, where we gathered the data from the web. We went on to build a price-prediction model that turned out to be relatively successful. We had an error in the range of 10,000 rupees for the top 37% data, I guess that was fairly acceptable. Anyways, the report has now been submitted.. let’s wait for the grades.

Crawling web data using "wget"

This is the first of a two-part post in which I discuss tools that we used for our project in our Business Intelligence using Data Mining course. During this project, we were to work on some real-world data to come up with some intelligent analysis using the tools we learned during the course.

Let's take a look at the website www.autoindia.com. If you try to search for all cars that are for sale, you'd reach http://autoindia.com/BuyUsedVehicle/BikeCarListings.aspx. If you click on any vehicle that is listed there, the site lists each individual vehicle in a separate webpage that looks like

Let’s take a look at the URL - http://autoindia.com/UsedVehicle/buy-used-car17262.html If you change the trailing number (17262) to some other number, the site would take you to a different vehicle that they’re listing. For our project, we wanted to download the html files for all the cars listed on the site – we’re talking about 17,000 files here. If you find yourself in a similar situation, chances are you would not want to save each page manually – even if you can save 15 pages a minute, it’d take you 18 hours to save 17000 web pages. Whew!

wget provides a smarter solution. From wget’s website,

GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.

In a nutshell, it means you can open a command window, and type something like

C:\> wget http://autoindia.com/UsedVehicle/buy-used-car17262.html

and hit enter. And wget will download that html file for you.

Now our job has become a little simpler. We can take any spreadsheet software, have it generate the numbers 1 through 17262. Use a simple search and replace to add the leading string “wget http://autoindia.com/UsedVehicle/buy-used-car” and the trailing “.html” to the generated number. Save that into a text file. Copy the entire file into a command window, and then go and have a cup of black tea. Do make sure you add a dash of lemon. Depending on your network connection speed, you’d have downloaded all the 17000 files by the time you finish your tea.

Now that we have 17000 files with us, these are still in html. We now need to convert the data in html to something in a spreadsheet, so that the data for a given car lies on a single row. I’ll discuss how to do exactly that using awk in my next post.

Misguiding content

The website of the National Institute of Smart Governance (NISG) loops through five images on the opening page -- two of these images are pictures from ISB. Even in the photogallery page, there are seven images shot at ISB.

I'm surprised why an organisation would put up photographs of facilities that are not their own. I mean, let them put up pics of the IIIT campus where they reside. Or at least, let them mention that these pics were shot at ISB.

BTW - does this count as image plagiarism?

Advertising for Morphy Richards

I spent some time this weekend working on an advertising assignment that was due today. We were asked to prepare a single-page print advertisement for Morphy Richards, for any/all of their products.

So here's how I went thinking about this assignment:

1. Identify the attributes of the current advertising Morphy Richards uses:
1. Clean advertising & simplicity - no garish images
2. Their products don't have any major differentiator, so current advertising doesn't use the "buy this because we're better" tag line. Instead, their advertisements try to instil a positive association of the product.
2. Most of their products are for the homemaker (read the woman in the household) - the ad has to appeal to women, it would influence buying decisions on their entire product range.
   
3. Use creativity to generate emotional appeal - reinforce the feeling of homeliness with Morphy Richards products.
4. Identify which product(s) I want to create the advertisement for. I ended up deciding between sandwich makers and hair dryers.

Once these decisions were done, I went about using a combination of google's image search, my creativity, and my skills at gimp & MS paint to come up with these.


The first one


This one turned out to be really clean with a "can't hurt the eye" white background. But the image was bad - it was a sandwich that wasn't from a sandwich maker! Surely the prof (or anyone else) would see through that. And see through that, my quaddie did. Back to the drawing board...


Another sandwich maker ad



So this one featured an improved tag line. Had gotten rid of the "A sandwich" trailer. But there were three problems with this one:

1. the colours are not uniform throughout - half the ad is in a shade of brown and the other half in a bluish grey
2. the sandwich is ugly - it is a very greasy sandwich that no health-concious homemaker would ever giver her child.
3. In hindsight, the tag line doesn't make any sense. How can a sandwich be the next best thing to sliced bread - a sandwich should be better than sliced bread, shouldn't it? I'd worked on the tag line from the phrase "the greatest invention since sliced bread", but I'd lost the meaning somewhere during the transition.

Time to move over to another product - let's check out the hair dryers.

The first hair dryer ad
So, who would buy a hair dryer? A woman who has a bad hair day. Hmm, more googling and more work on gimp made me come up with this:


All right, now I'm violating the cleanliness principle. Yeah, the colours probably look good, and the format is fairly simple - visual description of the problem, textual description of the problem, visual description of the solution. But the woman is not right - she looks agitated. And people never look agitated in a Morphy Richards ad.

So the challenge is now - come up with a visual description of a bad hair day that would appeal to women, without indicating any agitation.

Well, more googling, and this is what we come up with:

The final one



And did the ad work? I don't know - I submitted this to the prof only today. But a few "oh cho cute" comments from the target segment did make me think it's come out well :-)

Thanks V, I couldn't have done this assignment without your feedback.

Words Worth

ISB is conducting a competition to test your thinking.
The prize, for the 15 best entries, is a trip to ISB.

Check out the competition at http://www.isb.edu/wordsworth/

an afterthought

A few weeks back, we had our annual alumni reunion, Solstice (just realised I'd been misspelling it all along as Solistice). When the alumni introduced themselves, they always tagged their year of graduation to their name. They'd go "I'm R, '04" and "I'm S, '06" and so on.

Next year, I'll able to introduce myself as, "I'm Chiranth, double o' seven".

----------------

Among other things, I managed to write three exams last week, and a 4918 word essay on Outsourcing. Was about to overshoot the limit of 5 grand, but managed to bring it down. Weekend was a good break - watched loads of movies. Did some productive work too, like changing my desktop's wallpaper, contemplating to clear my desk, etc.

Classes start again tomorrow... er, in a few hours. I'd better get some sleep.

HNY!!

Wish you all a very happy new year!!