WSDM 2014 Trip Report

Last week I attended the Seventh ACM International Conference on Web Search and Data Mining (WSDM 2014) in New York City. The conference hotel was located right in Times Square, and I enjoyed visiting Cornell Tech, eating delicious food, and catching up with college friends. I also enjoyed attending sessions. WSDM is single track, which means the research being presented isn’t always directly relevant to everyone, but conference attendees have a shared experience and get exposed to research they might not otherwise.

Much of the content presented at WSDM focused on real-world existing challenges associated with web-scale search and data mining. The PC Chairs presented an analysis of the top strengths and weaknesses highlighted by reviewers, and found that “novelty” and “tackling real world problems” were highly valued. In contrast, the main weaknesses identified during review included “technical issues” and “simplicity.”

Recommendation systems and the use of entities seemed to be the big themes in the WSDM program this year. For example, Xiao Yu et al. presented a paper titled “On Building Entity Recommender Systems Using User Click Log and Freebase Knowledge” that explored how to provide personalized recommendations during search using query logs and an entity graph.

There was relatively less focus on social data mining at this WSDM than I recall from previous conferences. And consistent across all WSDMs, users were primarily represented within the community as behavioral traces. I would love to see more mixed methods used to capture a richer picture of the people who create web data and use web systems. The exception to this was Leysia Palen’s keynote.

Leysia Palen is the tornado chaser of computer science, in that whenever there is a disaster she is there virtually, collecting and analyzing the data that people produce. She founded the field of crisis informatics, a research area that takes an integrated perspective on the technical, social, and informational aspects of large-scale emergency response. In her keynote, Leysia noted that disasters aren’t like disaster movies – instead of people fleeing the scene, disasters tend to be sites of mass convergence with participants behaving thoughtfully. As such, disasters present a rich opportunity for us to use the web to understand and support real world events. The data that people create during an emergency provide situational awareness, facilitate self-organization, and support new ways of working.

Given my interest in large-scale log analysis, I particularly enjoyed the papers presented during the Log Analysis session. A few of these include:

• Discovering Common Motifs in Cursor Movement Data for Improving Web Search Ranking (Dmitry Lagun, Mikhail Ageev, Qi Guo and Eugene Agichtein): Currently most log analysis is done over discrete events like clicks and queries, in part because continuous data like eye and cursor movement is so rich that it is hard to identify salient features. This paper presents an approach to identifying common subsequences in mouse cursor movement data, and shows that these can be used as features to improve relevance assessment. The paper won the Best Student Paper award.

• Modeling Dwell Time to Predict Click-level Satisfaction (Youngho Kim, Ahmed Hassan, Ryen White and Imed Zitouni): The amount of time a person dwells on a search result is often used as a measure of satisfaction, with dwell times longer than 30 seconds typically being considered “satisfactory.” However, dwell time can vary greatly by task and user, and the current 30 second approach performs only a little better than random at measuring satisfaction. This paper presents a better approach that uses other search performance predictors as feature.

• The Last Click: Why Users Give Up Information Network Navigation (Aju Thalappillil Scaria, Rose Marie Philip, Robert West and Jure Leskovec): The goal of this work was to understand why people might stop directed browsing, and I really appreciated the unique approach that the authors used to gather data to study this. They created an online game that challenges people to navigate from one Wikipedia article to another just by clicking links. For example, you might be asked to get to my Wikipedia entry from the entry on hyperdata (one path: hyperdata > Web Science Trust > Jaime Teevan). The found that people start by navigating to a high degree hub. Successful users then gravitate towards the target, with failing users orbiting around it. Failing users tend to backtrack even when they are on the right track, suggesting it is hard for them to tell that they are moving in the right direction.

• Lessons from the Journey: A Query Log Analysis of Within-Session Learning (Carsten Eickhoff, Jaime Teevan, Ryen White and Susan Dumais): Carsten presented work we did during his summer internship at Microsoft Research looking at what people learn as they search. We found that people develop expertise over time, but that some sessions result in learning while others don’t. Knowledge acquisition search sessions tend to be long, topically diverse, and exploratory. Over time, people’s queries on a topic get more complex, and this learning is sustained over time. Visiting a search result page appears to be particularly important for learning. Carsten concluded by challenging the audience to think about “ranking to learn” (a plan on “learning to rank”): Can we rank results to help people maximize what they learn?

New this year at WSDM was a series of invited talks by industrial researchers. The goal was to highlight real-world web research that is breaking new ground but does not end up in publication. I found most of these talks interesting. For example, at the start of the session on Recommender Systems Guy Lebanon from Amazon highlighted a number of challenges with doing academic research on recommender systems that make transferring research difficult. These included:

• The fact that there are few public datasets available, so research tends to focus on existing datasets.
• The end goal for a company is user engagement and revenue, but our frameworks support predicting ratings.
• Conducting A/B tests is currently impossible for academics. Additionally, we don't have good approaches for searching a large space using A/B tests.

Vaclav Petricek from eHarmony presented some fun tidbits about how people use the dating site, such as the fact that couples are most willing to connect when the guy is 4 to 8 inches taller than the girl, and when the pair share the same level of attractiveness. He also discussed which profile pictures people found most attractive (hint: don't use a picture with your ex cut out). This reminded me of some work I did with Merrie Morris and Katrina Panovich, looking at the relationship between people's profile pictures and the answers they received to questions posed to their social networks. We found that people with close, social pictures were particularly successful at getting answers.

Many of the industry talks highlighted the value of building simple approaches using large-scale data for real world problems. It is interesting that “simplicity” was highlighted as a killer weakness during the WSDM review process, but seen as a strength in the industry talks. The talks were interesting, but at an hour each they took up valuable time in a tightly packed, single-track schedule.

Another experiment the organizers explored was to ask presenters to compare their work with the other papers in their session to provide context and create continuity. However, the paper presentations were very short (at times only 10 minutes including questions!), and this left authors little time to draw meaningful connections. I wonder if the idea might be stronger if combined with the idea of invited presentations. Rather than having sessions start with related industry presentations, they could start (or end) with a brief overview of the papers in the session that places them in the larger research context. This would be an interesting play on the HCIC discussant model, and provide significant value to audience members, particularly for sessions outside of one's area of expertise.

Thanks for a fun conference. Hope to see everyone in Shanghai next year for WSDM 2015!