Crawling web data using "wget"

This is the first of a two-part post in which I discuss tools that we used for our project in our Business Intelligence using Data Mining course. During this project, we were to work on some real-world data to come up with some intelligent analysis using the tools we learned during the course.

Let's take a look at the website www.autoindia.com. If you try to search for all cars that are for sale, you'd reach http://autoindia.com/BuyUsedVehicle/BikeCarListings.aspx. If you click on any vehicle that is listed there, the site lists each individual vehicle in a separate webpage that looks like

Let’s take a look at the URL - http://autoindia.com/UsedVehicle/buy-used-car17262.html If you change the trailing number (17262) to some other number, the site would take you to a different vehicle that they’re listing. For our project, we wanted to download the html files for all the cars listed on the site – we’re talking about 17,000 files here. If you find yourself in a similar situation, chances are you would not want to save each page manually – even if you can save 15 pages a minute, it’d take you 18 hours to save 17000 web pages. Whew!

wget provides a smarter solution. From wget’s website,

GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.

In a nutshell, it means you can open a command window, and type something like

C:\> wget http://autoindia.com/UsedVehicle/buy-used-car17262.html

and hit enter. And wget will download that html file for you.

Now our job has become a little simpler. We can take any spreadsheet software, have it generate the numbers 1 through 17262. Use a simple search and replace to add the leading string “wget http://autoindia.com/UsedVehicle/buy-used-car” and the trailing “.html” to the generated number. Save that into a text file. Copy the entire file into a command window, and then go and have a cup of black tea. Do make sure you add a dash of lemon. Depending on your network connection speed, you’d have downloaded all the 17000 files by the time you finish your tea.

Now that we have 17000 files with us, these are still in html. We now need to convert the data in html to something in a spreadsheet, so that the data for a given car lies on a single row. I’ll discuss how to do exactly that using awk in my next post.

Misguiding content

The website of the National Institute of Smart Governance (NISG) loops through five images on the opening page -- two of these images are pictures from ISB. Even in the photogallery page, there are seven images shot at ISB.

I'm surprised why an organisation would put up photographs of facilities that are not their own. I mean, let them put up pics of the IIIT campus where they reside. Or at least, let them mention that these pics were shot at ISB.

BTW - does this count as image plagiarism?

Advertising for Morphy Richards

I spent some time this weekend working on an advertising assignment that was due today. We were asked to prepare a single-page print advertisement for Morphy Richards, for any/all of their products.

So here's how I went thinking about this assignment:

1. Identify the attributes of the current advertising Morphy Richards uses:
1. Clean advertising & simplicity - no garish images
2. Their products don't have any major differentiator, so current advertising doesn't use the "buy this because we're better" tag line. Instead, their advertisements try to instil a positive association of the product.
2. Most of their products are for the homemaker (read the woman in the household) - the ad has to appeal to women, it would influence buying decisions on their entire product range.
   
3. Use creativity to generate emotional appeal - reinforce the feeling of homeliness with Morphy Richards products.
4. Identify which product(s) I want to create the advertisement for. I ended up deciding between sandwich makers and hair dryers.

Once these decisions were done, I went about using a combination of google's image search, my creativity, and my skills at gimp & MS paint to come up with these.


The first one


This one turned out to be really clean with a "can't hurt the eye" white background. But the image was bad - it was a sandwich that wasn't from a sandwich maker! Surely the prof (or anyone else) would see through that. And see through that, my quaddie did. Back to the drawing board...


Another sandwich maker ad



So this one featured an improved tag line. Had gotten rid of the "A sandwich" trailer. But there were three problems with this one:

1. the colours are not uniform throughout - half the ad is in a shade of brown and the other half in a bluish grey
2. the sandwich is ugly - it is a very greasy sandwich that no health-concious homemaker would ever giver her child.
3. In hindsight, the tag line doesn't make any sense. How can a sandwich be the next best thing to sliced bread - a sandwich should be better than sliced bread, shouldn't it? I'd worked on the tag line from the phrase "the greatest invention since sliced bread", but I'd lost the meaning somewhere during the transition.

Time to move over to another product - let's check out the hair dryers.

The first hair dryer ad
So, who would buy a hair dryer? A woman who has a bad hair day. Hmm, more googling and more work on gimp made me come up with this:


All right, now I'm violating the cleanliness principle. Yeah, the colours probably look good, and the format is fairly simple - visual description of the problem, textual description of the problem, visual description of the solution. But the woman is not right - she looks agitated. And people never look agitated in a Morphy Richards ad.

So the challenge is now - come up with a visual description of a bad hair day that would appeal to women, without indicating any agitation.

Well, more googling, and this is what we come up with:

The final one



And did the ad work? I don't know - I submitted this to the prof only today. But a few "oh cho cute" comments from the target segment did make me think it's come out well :-)

Thanks V, I couldn't have done this assignment without your feedback.

Words Worth

ISB is conducting a competition to test your thinking.
The prize, for the 15 best entries, is a trip to ISB.

Check out the competition at http://www.isb.edu/wordsworth/

an afterthought

A few weeks back, we had our annual alumni reunion, Solstice (just realised I'd been misspelling it all along as Solistice). When the alumni introduced themselves, they always tagged their year of graduation to their name. They'd go "I'm R, '04" and "I'm S, '06" and so on.

Next year, I'll able to introduce myself as, "I'm Chiranth, double o' seven".

----------------

Among other things, I managed to write three exams last week, and a 4918 word essay on Outsourcing. Was about to overshoot the limit of 5 grand, but managed to bring it down. Weekend was a good break - watched loads of movies. Did some productive work too, like changing my desktop's wallpaper, contemplating to clear my desk, etc.

Classes start again tomorrow... er, in a few hours. I'd better get some sleep.

HNY!!

Wish you all a very happy new year!!

Very Funny Ads

Check this site out: http://veryfunnyads.com/

gmail search tip

Gobbledygook version
gmail indexes whole words separated by whitespaces, not partial subsets of text.

In plain english
When searching for different word forms, such as publish and publication in gmail, it's necessary to search for both words separately. Searching for publi is not going to fetch any results :-(

What MGTO missed out

Watch Kramer vs Kramer or Jerry Maguire? The scenes where the two chaps get fired are pretty neat, eh? Here's a thought. Would an MBA grad be able to fire the two characters any differently?

Is there a fundamental disconnect from being an MBA grad and being a manager? Sure, an MBA (maybe) teaches us the quant skills and the hard facts to look out for when we are running a show. But then, doesn't it take a bit more to be a manager? Let's go back to the firing aspect - how would I, as an MBA grad, be able to tell a subordinate that he is going to lose his job, because of no fault of his? Should I take solace in the fact that it is a decision based on profitability?

A lot of companies go through crunch situations, and lay people off at one time or the other. Strangely, some people are willing to go back and work for the very same companies when times improve. I think that probably happens out of goodwill, more than anything else, of the managers. And I doubt one can learn how to create that goodwill at a b-school. Yeah, you could have courses which focus on how to improve one's personality, and how one should train oneself to react to situations; you could also have other influences during an MBA program, such as ideas of social responsibility. That, however, is one end of the spectrum. On the other end, an MBA might teach you how ensuring goodwill with fired employees reduces recruitment cost later. Somewhere between these two extremes is where MBA programs are creating the manager of today.

MGTO = Management of Organisations, a course we had in the fourth term, one of the few people-related courses we've had.

Another ad mismatch

Paresh Rawal danced to a "ta-ra-ra-ra-ra-ri, pizza aaye free" for Domino's. That tune doesn't seem to go very well in the Arshad Warsi ad, does it?

PM on the steel industry

Excerpt from the PM's speech at the IISCO steel plant, Burnpur, West Bengal. Here's a link to the story on NDTV.

Why does Lakshmi Mittal or Ratan Tata have to buy foreign companies to establish a global presence and expand steel capacities when our per capita consumption of steel is so low and when there is so much opportunity here?

Let me see if I understand the PM's statement correctly. Is he suggesting that:
1. Mittal & Tata should not be establishing a global presence? OR
2. Mittal & Tata buy only Indian companies to establish a global presence? OR
3. Mittal & Tata should match their companies' production capacity only to match domestic demand, which currently happens to be low, but could rise in the future?

Maybe we should just stop attempting to make sense of ambigous statements made by politicians and assume they're playing to the crowd.

The network comes home

They come in all sorts of shapes and sizes. And they come with different ideas of life. Each one of them is brimming with gyaan about the college, about resumes, about life, about careers, about everything. And boy, are they getting an audience. Here's what's happening. Our annual alumni reunion - Solistice - is on this weekend, and the alumni have descended from all over to our campus. I guess it's a good way to end a week that was jam-packed with assignments for almost everyone.

I ran into alienwan shortly after he landed up on campus. Almost expected him to be reviewing someone's resume, and he didn't disappoint. Also met ford prefect. There are loads of other alumni I met, some for the first time. Interesting to hear what they had to say about post-ISB-life.

They've started to test the speakers on the lawns outside our block.. looks like it'll be a very loud party tonight. I'd better finish that SOMG work in the next 120 minutes :-(

INPE + CFPD + MHTP = a big smile

Advertisements, rebranding & bias

• KitKat's latest advertisement shows people wearing pants twice the size they usually wear. So, is KitKat now a health food?
• Sony Ericson advertises a music player.. it takes a while for the viewer to realise it's a mobile phone. Got to admit, I love this ad; should try to locate it on youtube sometime.
• ThumsUp rebrands itself as a men's drink, something that women won't understand.

Seriously, what's up with these brands?

--------------------------------------------

When on of our professors fired a "name three ads that you've seen today", we were stumped for a few minutes. He went on to say, "See, that's advertising for you. There's hardly any recall." I agreed with him then. But now I don't. If you ask people to name three political news headlines they've seen today, chances are good that they'll be stumped on that question too. That, dear readers, is selection bias.

--------------------------------------------

BTW, as the first three lines show, I seem to fare better on ad recall than the news recall :-( I hope advertising turns out to be a good elective next term.

Using tab-stops in MS Word

This post ought to interest people who are busy updating their resumes.

Please open up your resume in MS Word, and hit the button that looks like this symbol - ¶. That button is available on the standard toolbar – you might have to add buttons to view it.

Now, let’s take a look at two snapshots of a resume.

Figure 1

Figure 2

Make note of the poor alignment on the right side in Figure 1. The two lines don’t end at exactly the same length. What’s more, there’s an arbitrary number of tabs and space bars one has had to use to get the alignment as close as possible to ideal. But hey, what happens when you change the margins of the document? You’d have to go tabbing and spacing all over again to fix the alignment. Figure 2, on the other hand, seems a lot crisper. There’s only one tab that has been used, and that has taken care of everything. If figure 1 is what your resume looks like, and figure 2 is how you would like it to look, please read on.

The following steps will help you use a feature called “tab-stops” which is present in most word-processing software (including MS Word), but can be quite a pain to understand and use.

So, here’s what you need to do:

1. Create a new style
1. Menus --> Format --> Styles & Formatting
2. Click on New Style. Specify the name, font, font-size, etc. Ensure that the Style Type is set to Paragraph.
3. Click on “Format”
4. Click on “Tabs”. A new options box should pop-up. The tab stop position should be set by default to 1’’
5. Click on “Clear All”
6. In the tab stop position, write in the length of your horizontal ruler. This would typically be the width of your page minus the margins that you’ve specified. For a page setup using “Letter”, and margin 1.25”, this would be 8.25” – 1.25” -1.25” = 6”
7. Set the alignment as "Right".
   
8. Click on “Set”, then “Ok”
9. Save the style with a meaningful name
2. Write out your line saying “Indian School of Business Hyderabad India”. Ensure that you provide only one tab between the words “Business” and “Hyderabad”
3. Apply your newly created style to the line.

Hopefully, your alignment problem would be solved. Now, if you need to change the margins of your page, you’d need to reset the tab stop position to something that suits the new margins.