Wednesday, August 26, 2015

How to Create a Graphics Research Paper

This article shares some strategy for creating computational graphics research publications, such as those found at SIGGRAPH, I3D, HPG, EG, and EGSR. It is inspired by the coming I3D'16 submission season as I reflect on process and offer advice to my junior collaborators.

I focus on process over specific writing tips, because there are already a lot of good resources for strong technical writing, such as:
Those primarily apply at the prose phase and don't in themselves help you to reach the prose phase, which is what I'm seeking to explain. The following advice also holds for class final projects, independent studies, and theses at varying time scales.

Publishing is a Skill

Implementing computer science methods is one skill, inventing new methods is another skill, and publishing new methods is yet another skill. When you're fortunate enough to give a public talk on your work--yes, that's a fourth skill.

Unfortunately, your experience with one of these skills won't necessarily translate to the others. You have to explicitly train for each. For junior researchers in academia or industry, learning to publish and present work is critical. Inventing new ideas in research is useless unless the knowledge is shared. (In contrast, developers can benefit from deploying their inventions internally.)

The authors you see producing multiple papers per year and winning paper awards are not doing so only on the strength of their innovation. They are also extremely skilled at explaining their work in writing, and have developed efficient processes for minimizing risk and driving towards an acceptable paper.

In most cases, prolific researchers are constantly greenlighting and killing projects. They're also pipelining multiple projects. It is common for a research lab to have many projects targeting a publication deadline and then cut about half of them two months before submision. This allows doubling down on resources for the strongest work by bringing in new coauthors during the crunch period.

It is also common to decide to slip a paper to the next conference deadline, and to resubmit previously rejected work to a new conference. Due to the noise on the peer review process, even the best papers might be rejected two or three times before eventual acceptance. Since there is some graphics deadline roughly every other month and each paper might take a year for completion, a big lab must have many projects in flight at any time.

The Paper is the Contribution

We're trying to advance the state of the art in our field. The ideas that you invent and code that you write have no value towards this in themselves. Many of your coauthors won't even see or run your code, nor will your readers. Code exists primarily to implement the experiments that you will report on in the paper.

So, regardless of the state of your code, your project is highly at risk until the paper is drafted. Polishing the code adds little value. It should only be done to ensure correctness, accelerate experimentation, or as support for a code release after publication.

Note that this means telling your coauthors (e.g., advisor) that you've written 500 lines of code this week is unlikely to impress them. In contrast, showing them a complete set of result figure drafts will earn you deserved praised.

Minimize Risk and Revision

Complete drafts of your work in bottom-up order, according to this pyramid:


This is a pyramid because each step builds on the lower ones. "Write talk" is in parentheses because often we don't actually write the presentation before the paper is accepted for publication. That's unfortunate, though. Usually, after creating a presentation I know how I should have written the paper in the first place. When time permits (e.g., on a thesis project), I recommend drafting the talk during the writing process.

The above order is designed to meet two goals. First, it ensures that if a project-killing problem will arise that it is encountered as soon as possible--if you fail, fail fast! Second, it drives you to work out the objective and core elements of your paper early so that you don't have to revise notation and the pitch multiple times. The later parts of the pyramid contain elements that are more subjective and are easier to scale to the remaining time budget.

Some "fail fast" examples: You should be familiar with the previous work on the topic before you begin your own. The literature contains the hard won experience of others who have looked at the same problem. You want to get that information the easy way by reading it, rather than rediscovering it. Plus, if there's an obscure paper that invented the same algorithm as yours a decade ago, you'd rather find that out at the beginning of the project while researching the bibliography than at the end when it will invalidate all of your work.

Some dependency examples: Lucid diagrams and equations take a long time to create. You need to find a minimal but clear notation and a way of structuring the steps that guides the reader intuitively. The best design for these usually illuminates the best way of re-structuring your code and typically drives the exposition prose, which exists to explain the compressed notation. In a good paper, one of the contributions is a new way of looking at the problem, which then leads to a new solution. Writing the explanation of the algorithm often can lead you to this insight, so, the introduction to the problem rests on the explanation of the solution. The abstract of the whole work can't possibly be written until you have drafted that work, and the title is a single-phrase abstract of everything, so must necessarily be last.

Drive to Minimum Viable Product 

I was careful to say that the preceding section gave the order in which to draft content. You should not complete each section before moving on to the next one, however.

Industry developers speak about "minimum viable product" and "vertical integration." These concepts are relevant to any project, including creating a research paper.

Once every section has been drafted, make multiple vertical passes over every step to iteratively drive towards completion. Use the system on the left of sweeping over the entire process, and not the one on the right of working strictly from the bottom up:
This is important for several reasons:
  1. Allows coauthors to work in parallel once the draft structure is complete
  2. Minimizes the time to a complete first draft--and the project is tremendously at risk until a draft exists!
  3. Allows submission at any point after the first draft, thus scaling to fit shrinking time resources
  4. Gives everyone a shared vision of the work and allows for high-level course corrections
It is important to recognize that each pass over the whole project is just that--a revision of a draft. Acknowledging this removes the pressure to get it right the first time, which creates a kind of writer's block.

With the knowledge that you're going to revise everything, you should observe that it is not worth polishing anything until you've worked up the pyramid and received feedback from collaborators or colleagues. I usually draw the first version of my diagrams on a whiteboard and then photograph them. My first text drafts are bulleted outlines. Below is an example from the Phenomenological Transparency paper of the evolution of some key figures (there were many more iterations that are not shown here), from our first draft on the left to the final published results on the right:



Poorly investing time by prematurely polishing also runs into a danger: you'll be reluctant to improve those parts because they're "done." But, they aren't done: they are over-polished early drafts that probably have to be completely reworked.

Since you'll be revising your results several times, I recommend automating the process of creating them. For complex figures and diagrams, we usually make scripts that re-run all of our experiments without manual intervention and directly output Latex or CSV data that we can paste into the paper. Of course, it is important to not go overboard with automation. If your results are a only handful of figures from completely different programs, then maybe it is worthwhile recreating them by hand when you make a minor change.

Save the "source" versions of figures: Excel, PPT, PSD, iMovie, etc. files. Check them in to your revision control system, too. You'll often need to go back to a complex multi-image figure and adjust one layer in Photoshop to insert updated results, and should not waste time re-creating the unmodified rest of it by hand.

What Results to Show

Anything that you draw on a whiteboard while explaining your work to a colleague or visualize in a program for debugging is a wonderful result to include in a paper. For example, if you think about your work through oct-tree diagrams, quiver plots, or program traces, then the final readers (not to mention the peer reviewers) will benefit from those visualizations as well.

Include results that show your algorithm failing as well as succeeding. This is good science, because it fully discloses the limitations so that others can later address those. But it is also essential for understanding that the algorithm is correct. If you showed 20 perfect pictures of ray traced reflections, then how does the reader know that they aren't photographs, or that you didn't accidentally leak the ground truth data into your experiments? Showing the cases where it exhibits exactly the kinds of errors predicted when assumptions are violated confirms a correct implementation and gives insight.

Likewise, think about your paper from the perspective of a skeptic. If someone claimed to have reproduced your work, what results would you ask them to show you to prove that it worked? These are exactly what readers (and before them, the reviewers) will need to see to trust your implementation and algorithms.

After Publication

Here are some nice resources for how to create a good graphics presentation once your work is accepted:



Morgan McGuire (@morgan3d) is a professor at Williams College, a researcher at NVIDIA, and a professional game developer. His most recent games are Project Rocket Golfing for iOS and Skylanders: Superchargers for consoles. He is the author of the Graphics Codex, an essential reference for computer graphics now available in iOS and Web Editions.
at