This semester at Columbia, I am taking two research focused classes on Causal Inference and AI Agents. Reading academic papers is a main component of both classes. While in the past I have read research papers, I did not have a set system to efficiently read, understand, and annotate the papers. The purpose of this post is to detail my current method for my own reference, to source feedback from others (see about page), and (maybe!) help others read papers more efficiently.
To start, I came across Dr. Keshav’s How to Read a Paper, which was a good starting point. In summary, he proposes taking 3 passes when reading a paper. I would recommend reading it since it is short and to the point. I found this method to be helpful in not becoming overwhelmed by proofs or more complex aspects of a paper (especially when its your first time reading a paper for a new field) during the first read. Another good learning from this paper is that there may be papers you only do one pass at, and leave it at that - you don’t need to finish a paper once you started it.
While How to Read a Paper gives you a good strategy on how to read and breakdown a paper, it did not provide a good strategy on what one should look for while reading a paper. For that, I consulted a few friends and professors, and consolidated their recommendations into the following list:
1. Who are the authors & what are the references?
Dr. Keshav also makes note of this in the first pass.
2. What is the problem?
This is the gensis of the entire paper - There is a problem, and we think we found out a way to address it.
3. What did they do to address the problem?
In particular, I focus on what methods the paper used. If you are also focused on doing research, the methods and how they are implemented can be a source of inspiration on future research. What do they believe is the cause of the problem. Do they unique insights on how to address the cause? Do you agree with that cause?
4. What were the results?
What did the paper achieve. Were they successful in solving the problem? Do their results seem credible?
5. What are remaining problems?
This is also part of #4. Usually in the conclusion the authors will cite areas of future work. However, I have also noticed that authors will mention other problems not directly related to the original problem. These are good to note for future reference as well!
For now, those are the 5 things I look for when reading a paper, highlighting them as I find them as I do my pass of the paper. I am a big fan of highlighting papers and use the following color coding scheme
- Red: Problems & Limitations
- Blue: Methods to solve the problem
- Green: Results
- Orange: Things I do not understand / want to understand better
- Yellow: Ongoing / unaddressed problems.
I found the color coding especially helpful for reviewing the papers. Below I attached a sample of how I annotated the first page of a paper I recently read, SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. Don’t mind the messy handwritting.
Finally, when I am done with the number of passes I will try to summarize points 2-5 quickly and think about remaining questions / critiques I have. If I have the time, I’ll also try to see what papers reference the current paper I’m reading. I’ve found this helpful to see what people have said about the paper more recently (are the methods outdated or is it still relevant?) as well as potentially finding solutions to remaing questions I had about the paper.