Monday, November 10, 2014

A Formula for Academic Papers: Related Work

The Related Work section of an academic paper is often the section that graduate students like writing the least. But it is also one of the most important sections to nail as the paper heads out for review. The Related Work section serves many purposes, several of which relate directly to reviewing:
  • The person handling the submission will use the referenced papers to identify good reviewers,
  • Reviewers will look at the references to confirm that the submission cites the appropriate work,
  • Everyone will use the section to understand the paper's contributions given the state of existing research, and
  • Future researchers will look to the Related Work section to identify other papers they should read.


When placing your research in the context of the existing literature, there is no need to show that prior work is all wrong to show that your paper makes a contribution. In fact, doing so is likely place reviewers and readers who have written relevant content in a combative frame of mind as they read. But these are the people you most want to listen to what you have to say! So rather than focusing on why your work is better than what has already been done, show how it builds on existing knowledge to provide additional insight. Write your Related Work section as if you were telling the cited authors why they should care about the work that you present. After all, they are the people who will probably be reviewing your paper.

Papers do not need to exist in a vacuum to be interesting, and the typical contribution that can be made in ten pages is relatively limited. While you should aim to change the world with your body of research, you are unlikely to do so within a single paper. Avoid over stating your contributions. Be particularly wary of declaring that your paper is the "first paper that we are aware of" in an area. While I know it is tempting to do so (and I am even guilty of doing it myself), statements like this usually trigger an intense related work search when I see them as I review. It should concern you if you believe nobody else has looked at an analogous problem to yours. Chances are very high that there is something similar out there that your work can draw from.

Conversely, do not freak out if you happen to come across related work during your literature search that seems to address the exact same problem that you are studying. In an ideal world you would identify all relevant work prior to starting your own research so that it can inform your approach, but in the real world that doesn't always happen. You may have holes in your initial lit review, or related work may be published after you begin a study. Surprisingly, however, this is generally a good thing. It means that other people are excited and interested in your topic, and it provides you with a fresh way of looking at things. There is a large space to study surrounding any problem, and your work probably makes a contribution. The challenge is just to figure out what that is and how to communicate it clearly to your readers. 

Because reviewers will be drawn in part from the papers you cite, cite papers written by people you would like reviewing your work. Reviewers are likely to look at your citation list to ensure completeness -- and, sometimes, to ensure that papers they have written are cited. To avoid bruised egos, do not leave significant holes and try to include papers by a variety of different authors.

Include citations to your own papers when relevant, even if the paper you are writing is being submitted anonymously. There is no need to anonymize these citations. Instead, cite your papers the same way you would cite any other paper, in the third person. For example, in a paper on cross-session search I write, "Teevan et al. [32] showed, via query log analysis, that nearly 40% of queries were attempts to re-find previously encountered results." But while it is fine to cite your own work, be wary of over-citing yourself. Too many papers by an unexpected person typically signals that that person is an author, and generally looks bad.


A typical Related Work section follows a basic structure:
  • It starts with few sentence overview of the general space, and
  • A preview of areas that are particularly relevant and will be discussed in detail.
  • The body consists of several paragraphs, each discussing a different relevant thread of research.
  • The section ends with a paragraph summary of the paper's contributions over existing research.
The first paragraph of the Related Work section should help a knowledgeable reader in the field place the work into a general bucket. Be aware of what is common knowledge in the community where the paper will appear to avoid giving too sweeping of an overview. There is, for example, no need to explain that there has been a lot of research into web search at the start of an information retrieval Related Work section. Instead, highlight the specific sub-area and tell the reader what aspects of that sub-area are particularly relevant.

In the body of the Related Work section, do not just list paragraphs that each summarize a single related paper. Summaries can be a useful way for you to build a picture for yourself of existing related work. But in the Related Work section, you should help your reader get the lay of the land by grouping and organizing the existing research. Start each paragraph with a sentence describing why the papers discussed in that paragraph are related, citing all of the papers to which the criteria applies. Then write a sentence or two about several of the most relevant papers from the group, highlighting the approach used and relevant findings. End the paragraph with a sentence explaining how the work in your paper contributes something new in light of these papers.

An example of this basic structure for a paragraph in the body of a Related Work section can be found in A Crowd-Powered Socially Embedded Search Engine:
ACM papers use numbers to cite related work, which provides limited context compared to other citation formats. Do not treat these numbers as part of the paper's text. You should not, for example, say, "In [23] the authors explore ..." Instead, write the text as if the numbers weren't there, saying, "Teevan et al. [23] explore ..."

End the Related Work section with a paragraph that summarizes what is know given existing literature, and highlight why the work to be presented in your paper offers a valuable contribution beyond this. An example can be found in Understanding How the Projection of Availability State Impacts the Reception of Incoming Communication:
In summary, the work presented in this paper builds on previous research to explore how availability information relates to people’s communication decisions. While earlier work focused on how availability information impacts the people initiating communication, we focus on its impact on the decisions of the recipient. Further, we are able to study this behavior at a much larger scale than previously possible by looking at the users of a popular enterprise communication system that infers its users’ availability.


Even though the Related Work section is very important to the overall paper, it should not be too long. It a ten page paper, a good rule of thumb is that you should be done with the paper's set-up (including the Introduction and Related Work sections) and on to the meat of the paper by the start of the third page. A good target length for a Related Work section in standard ACM format is one to two columns. To keep the section short, avoid subsections unless really necessary.

The actual list of references at the end of the paper should also be compact. I generally aim to fit the Conclusion and reference list entirely within a single final page. One way to do this is to target a reasonable number of references. I've averaged 31 references across all 10 page papers I've written since 2010. While you should not shrink the font size of your reference list, there are tricks you can play to minimize the amount of space each individual reference takes. For example, you can refer to common, well known outlets by their acronym. Do not, however, be sloppy with how you refer to papers. This can offend the paper authors and make it hard for readers to find the associated papers. Include correct, consistent details for each. Be sure to list references in alphabetical order to make it easy for the reader to scan to see if a paper is cited, since this is a task your reviewers are likely to do several times while reviewing.

Some communities place the Related Work section at the beginning of a paper, while others place it at the end. When writing a paper you should follow the norms of the community where you are publishing. However, if there is a choice about placement I recommend putting the section at the beginning. When I read papers that do not cover related work until after the main content, I find I spend much of my time reading wondering how what is being presented fits into the bigger picture.

Hopefully this post has convinced you to be thoughtful with how you place the research papers you write in context. The Related Work section is an extremely important part of the paper, and the resulting citation graph helps define the structure of the field. As a bonus, here are links to a few papers we have written that take advantage of the citation graph to make it easier to explore related work:


This post is part of a series of posts about the formula for academic papers. The components being discussed are:

Wednesday, October 22, 2014

Data Banks

Each of us individually create a huge amount of data online. Some of this data we create explicitly, such as when we make webpages or public facing profiles, write emails, or author documents. But we also create a lot of data implicitly as a byproduct of our interactions with digital information. These implicit data includes the search queries we issue, the webpages we visit, and our online social networks.

The data we create is valuable. We can use it to understand more about ourselves, and services can use it to personalize our experiences and understand people’s information behavior in general. But despite the fact that we are the ones who create the data, much of it is not actually in our possession. Instead, it resides with companies that provide us with online services in exchange for it. A handful of powerful companies have a monopoly on our data.
Definition of monopoly: the exclusive possession or control of the supply or trade in a commodity or service
Definition of data monopoly: the exclusive possession or control of the supply or trade in an individual’s personal data

Wednesday, October 8, 2014

Help! I'm Sexist!

The research studies I posted last Friday about the role gender plays in the STEM workplace paint a consistent picture: women face significant discrimination. Women are paid (and hired, and tenured) less than men with the same qualifications, and these gender differences are particularly large for parents. While women are often encouraged to address the existing disparities by advocating for themselves (e.g., by being assertive, negotiating, or encouraging diversity), research shows this type of behavior typically incurs a further penalty.

Instead, gender disparities in the STEM workplace are a problem that the entire community must address. Hiring managers need to hire more women. Managers need to promote more women. And peers need to accept diverse communication styles without the lens of gender.

Importantly, however, this does not just mean that MEN need to hire (and promote, and accept) more. Because the other consistent picture that arose from the studies I posted on Friday is that both men AND WOMEN discriminate against women. We all have deep seated biases that contribute to the problem.

Friday, October 3, 2014

Research about Gender in the STEM Workplace

Science Faculty’s Subtle Gender Biases Favor Male Students by Corinne A. Moss-Racusina et al.
In a study with 127 science faculty at research-intensive universities, candidates with identical resumes were more likely to be offered a job and paid more if their name was "John" instead of "Jennifer." The gender of the faculty participating did not impact the outcome.

How Stereotypes Impair Women’s Careers in Science by Ernesto Reuben et al.
Men are much more likely than women to be hired for a math task, even when equally qualified. This happens regardless of the gender of the hiring manager.

Measuring the Glass Ceiling Effect: An Assessment of Discrimination in Academia by Katherine Weisshaar
In computer science, men are significantly more likely to earn tenure than women with the same research productivity. [From a summary]

Wednesday, August 13, 2014

Evidence from Behavior


Doug Oard at the Information School at the University of Maryland is teaching an open online course on information retrieval this fall (INST 734). Above is the brief cameo lecture I recorded using Office Mix for the segment on Evidence from Behavior.

Tuesday, July 29, 2014

The #GreatWalk Recap

Cale and I completed our 100 mile #GreatWalk from Bellevue, WA to Great Wolf Lodge. We live-blogged on Twitter as we walked, and I have recorded our tweets on this blog in chronological order to make them easy to read. Thanks for sharing our journey with us!
  • Day 1: We depart!
  • Day 2: A long walk to the airport
  • Day 3: Getting tired and frustrated
  • Day 4: A candy discovery
  • Day 5: Skirting the military base
  • Day 6: A wet and rainy day
  • Day 7: Into the wilderness
  • Day 8: We arrive at Great Wolf!
  • Day 9: A day of rest
  • Day 10: The trip home
Some interesting external links about the adventure:

#GreatWalk: Day 10

[This post includes my tweets (@jteevan) from the tenth day (July 27, 2014) of Cale and my 100 mile walk to Great Wolf!]

#GreatWalk: Day 9

[This post includes my tweets (@jteevan) from the ninth day (July 26, 2014) of Cale and my 100 mile walk to Great Wolf!]

#GreatWalk: Day 8

[This post includes my tweets (@jteevan) from the eighth day (July 25, 2014) of Cale and my 100 mile walk to Great Wolf!]

#GreatWalk: Day 7

[This post includes my tweets (@jteevan) from the seventh day (July 24, 2014) of Cale and my 100 mile walk to Great Wolf!]