Clearly they must have seen the brilliant work Steve and I are doing and decided to jump on board. Check out the link to see it at work, pretty simple but effective visualization they’ve got going on.

[ Engadget ]

Final Project Proposal
Stephen Varga & Kunal D Patel

1. Overview
A short summary of what your project will be. Give me your best elevator pitch here.

Our project focuses on the polarity of comments on the New York Times website as a means of tracking public opinion about articles and topics. Through a robust time-based interface, we are interested in seeing the potential ebbs and flows of commenting, if we can spot “influencers” that shift polarity, and explore the topics generating the most positive and negative discussion.

2. Data
The data sources you will be using

• The Community API – we’re making a call 1x a day to pull all of the previous day’s comments and store them in a SQL database
• The Article Search API – after grabbing all the comments for the previous day, we store the articles they correspond to into a separate table in the same database
• Amplify API – we’ve now been granted access to the client services for Amplify, a sentiment engine that we are using to parse through the comments

3. Design Questions
A set of questions that you intend to answer or explore. At least one question should be about the data itself (i.e., what is the story you’re hoping to tell?), but these questions may also address design methods or technical approaches.

• Does the polarity of an article (how positive or negative it is) substantially influence the polarity of its comments?
• Do comments for an article tend to fall within a narrow range, or is there often variation?
• Through a time-based visualization, will be able to identify “influential” comments that (qualitatively) appear to shift comment polarity?
• Will Amplify be able to conduct meaningful and consistent topic extraction and sentiment analysis?
• Do we have enough time to develop a robust enough interface components to support comment AND topic searches?

4. Prior Art / Precedents
Discuss at least two existing works that are similar in some respect to your proposed project. How do you see your project in relation to this ecosystem of other works? Will it contribute something unique? Will it address problems that you see in other works?

10×10 – Jonathan Harris’ hourly scraping of several international news feeds, visualized as a sorted list of the 100 most “important” words in the news connected to a 10×10 grid of corresponding images. These words are generated by his own algorithms from the articles themselves. Our focus is on comments to see what the NYT community feel are the most important and polarizing topics for discussion.

Twittermood – Twittermood conducts basic sentiment analysis on tweets using the ANEW dataset (which assigns emotional values to words) and maps the results in real-time. Besides the difference in audience, our planned interface is far more robust, allowing for navigation by week, day, author, and article in order to view patterns.

5. Collaboration
A brief explanation of how you plan to collaborate with your partner.

Steve and I worked together to write the PHP scripts to parse through the NYT API’s, pull the relevant data, and store it into a SQL database running on Steve’s web server. I’ve been focusing on working with the Amplify API and developing interface wireframes while Steve’s attention has been devoted to developing a working prototype in openFrameworks. For the rest of the semester, we will retain our individual foci in order to efficiently divide tasks, but our work will converge much more as the interface takes shape.

1. Overview
A short summary of what your project will be. Give me your best elevator pitch here.

Our project will be a real-time, split-screen image comparison of trending topic popularity on The New York Times and Twitter. Through analysis of comment activity and article keywords, we hope to generate a list of ‘trending topics’ to compare to Twitter’s well-established system. Both sets of keywords will be mined for relevant images using Bing’s recently announced API, with the images displayed in a side-by-side feed to serve as an immediate visual comparison of active topics amongst both environments. This visualization will compare and contrast popular subjects within a controlled environment in which users are provided content to discuss (NYT articles) versus an uncontrolled, user-generated environment (Twitter).

2. Data
The data sources you will be using

• The Community API – to determine articles with highest current comment rates
• The Article Search API – once ‘trending’ articles have been determined, mine them for keywords
• Twitter API – mine trending topics
• Bing API – search NYT keywords and Twitter trending topics for relevant imagery

3. Design Questions
A set of questions that you intend to answer or explore. At least one question should be about the data itself (i.e., what is the story you’re hoping to tell?), but these questions may also address design methods or technical approaches.

• Are there discernible differences in content between what active participants in formal and informal news networks are interested in?
• Can an analysis of trending topics generated by each user group offer insights into the demographics of the groups?
• How do we ensure that our trending topics are viable metrics for judging user groups?
• How can we regulate image searches for these topics to return relevant images?

4. Prior Art / Precedents
Discuss at least two existing works that are similar in some respect to your proposed project. How do you see your project in relation to this ecosystem of other works? Will it contribute something unique? Will it address problems that you see in other works?

10×10 – Jonathan Harris’ hourly scraping of several international news feeds, visualized as a sorted list of the 100 most “important” words in the news connected to a 10×10 grid of corresponding images.

Pingwire – Allan Grinshtein’s real-time visualization of images from 3 popular Twitter image-hosting services (twitpic, yfrog, and twitgoo).

In both of these works, the role of the user for each network is not readily addressed, which is where our focus lies. Harris uses his own algorithms for determining the 100 most important words in the news, and Pingwire is simply a real-time aggregated feed of hosted images. By approaching data mining and collection from the perspective of the users, we believe we are contributing a unique comparison of the cultures of The Times and Twitter.

5. Collaboration
A brief explanation of how you plan to collaborate with your partner.

For the purposes of data mining, the most effective solution will likely be to split the work between data sources. We both have experience with PHP and mySQL for mining and storing data, and the Bing API offers protocols we are both familiar with (JSON, XML). While we have yet to finalize a programming language for visualization, we will likely use openFrameworks for it’s speed and our familiarity with it. We will plan the visualization, data mining, context and interaction design processes together and split tasks as needed.

Here’s a screenshot and link to my SWF that returns back The Times section with the highest frequency of articles within the last 20:

[ SWF ]

Here’s a screenshot link to a PDF I made of the mockup for a proposed project that would include this:

[ Mockup (PDF) ]

[ The Preservation of Favoured Traces ]