The page for MS&E 237, Spring 2010 will be up in March 2010.
Looking for the notes for this course? All on the course wiki!
Want to listen to the lectures? Download the mp3 recordings and transcripts.
Data Mining and Electronic Business: The Social Data Revolution
Extract Insights from Twitter
In the last year, your location data and personal medical information have become the latest streams in the river of data, joining email, clicks, searches, social networking, and buying patterns. This course will dramatically change how you think about your data.
How can these data sources make our lives easier, more effective, more interesting? How can we get better recommendations, based on our behavior and the behavior of our friends? How can reputation systems help with decisions about who to trust?
Gathering, sharing, and storing data has become trivial. But what shall we collect, and what applications can we build that users really want?
Moving beyond graph and guess, push and pray, launch and learn, and so on, this course gives you tools and strategies for successful applications. How can you optimize virality and engagement, and spot weaknesses early? How can you entice users to interact with the app, and recommend it to their friends?
Each class is structured according to PHAME: define relevant Problems, invent Hypotheses, create Actions, design Metrics, and conduct Experiments. We also introduce a key driver to encourage users to provide critical data: Return on Personal Engagement (ROPE). Users who gain a benefit (tangible or psychological) from participating are far more likely to do so, and we discuss how to design incentives to encourage participation.
Course time is enriched by notable speakers, from notable companies. In addition to discussing applications that succeeded, we also discuss applications that failed, and try to distill out the reasons for success and failure. This course also includes highlights from two courses I developed and taught at Berkeley last year, including the popular"Marketing 2.x" at the Haas School of Business.
Data mining is no longer the process of digging through data morgues to uncover scraps of still-viable information. Success in the online marketplace now hinges on people and the data they create. E-business is no longer about selling books.
Assignments and Readings
Hands-on assignments include leveraging web analytics, applying geolocation, creating a recommender system for Twitter, and building a Facebook app.
Students are expected to actively engage in class discussions, to have their assumptions challenged, and to bring their diverse backgrounds to bear. After each class, a detailed write-up is created by the students as the course wiki (see 2009, 2008, 2007).
The reading material is very recent, originating from several academic disciplines. Besides statistics and computer science, it discusses modern marketing techniques, behavioral economics, social network analysis ideas and other concepts. For background reading, the following books might be useful:
In the Bay Area. The Social Data Revolution course meets Tue 3:30-6:30. Coming up: next week the head of Data Science of Yelp will come to class with 5 product managers for a design thinking exercise, then and a conversation between a former spy (and also ex CTO or Network Solutions) and a former student (also CTO of a fascinating startup) sharing his experiences working as data scientist for a special unit in Afghanistan on predicting explosions and money laundering. We will have the CEO of a blockchain company discussing how recording everything changes everything, and end the semester with one or the people who change the world by bringing social data to travel (Expedia), real estate (Zillow) and jobs (Glassdoor). Videos of all the classes are posted on youtube. And to learn more about "Data for the People" (published in January), follow @ourdata on Twitter for bite-sized pieces. My mobile: +1 (650) 906-5906. Email:
Click on an airport to see when I'll be there : AUS : EUR : IAH : LUX : PAO : PVG : SAN : SFO : SJC : UCB : USA : YVR : ZRH : all