This living guide is a collection of best practices surrounding paper preparation, which grows every so often as I realize that I’ve given the same advice multiple times. All my students are required to read through this guide before distributing drafts, even internally. Addressing any issues incrementally avoids a last-minute scramble to fix them later.

Note: This guide is a work in progress.

Typography and Typography-Related LaTeX Tips

Prefer to use small caps for acronyms. Full capital acronyms, like MDP or POMDP, tend to stand out in the text and draw the eye. Instead, using small caps—like mdp or pomdp—look nicer when used frequently in the text. This can be achieved in LaTeX via \textsc{mdp} for mdp, noting that text inside \textsc is itself lowercase to achieve this effect. If you use the glossary package for acronym management, this becomes easier; see details below. Using small caps isn’t always the best choice, especially when you inherit acronyms or, worse, mixed-case terms like LaTeX or ProcTHOR aren’t easily dealt with. I have done odd things before like \textsc{l}​a\textsc{t}​e\textsc{x}, which results in the unconventional latex or the not-as-bad procthor, but really it’s non-ideal. Use your judgment.

Using small caps isn’t always the best choice, especially when you inherit acronyms or, worse, mixed-case terms like LaTeX or ProcTHOR aren’t easily dealt with. I have done odd things before like \textsc{l}​a\textsc{t}​e\textsc{x}, which results in the unconventional latex or the not-as-bad procthor, but really it’s non-ideal. Use your judgment.

Know the difference between hyphens, en dashes, and em dashes.

• A hyphen is for connecting words with a combined meaning, e.g., learning-augmented. It is a single - in LaTeX.
• The en dash is for ranges of values, e.g., pp 3–4. It is a double -- in LaTeX.
• The em dash is for surrounding phrases or clarifications at the end of a sentence—just like this. I use these on occasion when I might have otherwise thought to surround a phrase in parentheses. They are achieved via a triple --- in LaTeX.Most style guides recommend not puting spaces around an em dash. The dash should touch the text on both sides, as I have done here.

Most style guides recommend not puting spaces around an em dash. The dash should touch the text on both sides, as I have done here.

Never use double-quotes " in LaTeX; use single quotes and backticks. LaTeX is not designed to identify double quotes, so surrounding a word or phrase by quotes requires the following syntax: ``something''.

Punctuation goes before footnotes and closing quotes. Proper use looks like `something.' or bar.\footnote{A footnote.}.

Internal links should be neither colored nor surrounded by a colored box. By default the hyperref package, upon which many other packages depend, tends to color internal links or puts colored boxes around links, which means that every use of \cite or \ref results in some ugly markup. These should be disabled. This fix can be achieved either during import \usepackage​[hidelinks]​{hyperref} or later with \hypersetup​{hidelinks}.

Some conferences or publishers will require that the colored reference boxes remain. RA-L is one of these.

Some conferences or publishers will require that the colored reference boxes remain. RA-L is one of these.

Understand proper use of e.g. and i.e. Use e.g. for examples and i.e. for clarification. Both of these should be followed by a single comma and proceeded by either a comma, parenthesis, or an em dash—e.g., or surrounded as in this example.

References and Bibliography Management

Enforce proper capitalization in BibTeX. Many BibTeX styles automatically update convert titles to sentence case, lower-casing many of the words. This behavior is by design, but does not play nice with acronyms. For example, a BibTeX entry with title = {DESPOT: Online POMDP Planning with Regularization} will produce “Despot: Online pomdp planning with regularization.” Instead, you can enforce capitalization with an extra set of curly braces around words you would like to preserve: title = {{DESPOT}: Online {POMDP} Planning with Regularization} will produce the correct “DESPOT: Online POMDP planning with regularization.”

Surrounding the entire title with additional curly braces will preserve capitalization for the entire title and is not recommended, since it will then over-enforce capitalization and thus deviate from the BibTeX-recommended style.

Surrounding the entire title with additional curly braces will preserve capitalization for the entire title and is not recommended, since it will then over-enforce capitalization and thus deviate from the BibTeX-recommended style.

Use a non-breaking space before cite commands and other cross-references. The reference number should not begin a line, as it is both unpleasant asthetically and hurts readability. LaTeX uses the tilde character ~ to signify a non-breaking space, which allows a space between what it connects, yet will not break the line.

• Bad: “Stein \cite{stein}”
• Bad: “Stein ~\cite{stein}” Note: This adds two spaces
• Good: “Stein~\cite{stein}”
• Good: See Sec.~\ref{sec:method}.
• Good: See Fig.~\ref{fig:overview}.
• Good: See Eq.~\eqref{eq:bellman}.

You must manually maintain and clean BibTeX entries; Google Scholar and automated systems are imperfect. Simply downloading the BibTeX entries and using them will result in poor quality citations. Similarly, bibliography managers are not perfect. It is your responsibility to clean those entries.

Reference document elements in a way that matches the template, e.g., Fig. versus Figure. Many LaTeX templates are opinionated about how figures, tables, and algorithms are referenced. If the template displays “Fig. 1” when placing a figure, all mentions of that figure in the text should also read “Fig. 1” and not “Figure 1.”

Every section, figure, table, and equation must have a label. Add \label{...} at creation time for every sectioning command and every float/equation environment. This makes cross-referencing fast, avoids fragile “search for the number” workflows, and reduces errors during late-stage edits.

Do not refer to “subsections” in prose. When referencing a subsection, the template will format it appropriately. For example, prefer Sec.~\ref{sec:related} (or “Sec. II-A” depending on the style) rather than “subsection A of section II.”

In the final document, references must appear homogeneous. If you collected BibTeX entries from multiple locations, it is possible that you will have a mix of different ways to reference conferences or journals, e.g., “icra” or “2024 ieee International Conference on Robotics and Automation” or “International Conference on Robotics and Automation (icra).” It is your responsibility to homogenize the references and comply with the norms for the particular conference or journal.

Avoid using references as nouns. When referring to a paper, refer to the authors or to the algorithm rather than using the reference number. Instead of “[5] uses a novel approach.”, say “Stein and Roy [5] use a novel approach.” as this is both nicer to look at and also easier to follow.

When referencing a paper by its authors, the author names should not reappear in the citation. For example, “Work by Khanal and Stein (2024)…” is preferred to “Work by Khanal and Stein (Khanal and Stein, 2018).” Some citation packages have the \citet textual citation command, which will add the authors and year automatically. If that command is unavailable, manually typing the authors and using \citeyear is preferred. If the citation style is numeric and so does not use author names, this change is not needed.

Peer-reviewed venues should be cited over ArXiv versions. It is you responsibility to make sure your citations are up-to-date. Citing papers uploaded to the ArXiv is okay, but only if the paper has not be published elsewhere. Before a final submission, do a quick search for all ArXiV papers in your references to see if they have been accepted to another venue.

Math and Equations

Equation subsubscripts must use \text for words or abbreviations. When using subscripts that are words, enclose them with \text{} (or sometimes \textit{}) to ensure proper formatting and letter spacing. For example: V_{\text{known}} ($V_{\text{known}}$) is correct, while V_{known} ($V_{known}$) is not. Even when full words are not used, this may be an appropriate choice. Additionally, use text size commands—e.g., \tiny—to control the size of the text.

All equations should have a number and be followed by appropriate punctuation. This rule is fairly standard practice. All equations should have an equation number. They should also include punctuation as if they were part of the inline text. Thus, if the equation is the end of the sentence, it should be followed by a period, and a comma if the sentence continues.

Use Eq.~\eqref to refer to equations. Equations are often referenced in the text as “Eq. (1).” Using \eqref{eq:name} ensures that the equation number will be in parentheses. Using Eq.~\ref{eq:name} will result in “Eq. 1,” which is incorrect.

Max, min, argmax, and similar should be in body-text font and defined as operators. The \max and \min operators are built in and are in the body text font. For more complex and custom operators, the amsmath package provides support to define operators as follows: \DeclareMathOperator*​{\argmax}​{arg\,max}. Defining this as an operator means that the subscript will be placed below the middle of the text, which is expected behavior.

Working with LaTeX: Best Practices and Useful Packages

Prefer smaller .tex files, each added to a main file via \input. It is often easier to navigate and edit smaller files than having a single massive file with all of the content in it. This guideline is particularly important when collaboratively editing via Overleaf, where jointly editing a single large file can lead to rendering issues in the editor.

Tables and figures should each get their own .tex file and added via \input. It is almost always necessary to move around figures within the .tex document. Putting the code for figures and tables in their own files means they can be added via the \input macro and moved around with ease. I usually have a separate top-level folder called “inputs” that includes such files.

If allowed, use booktabs for professional tables. The booktabs package makes for some very nice looking tables. If you are using booktabs, use \toprule, \midrule, and \bottomrule, and do not use \hline. In general, avoid vertical rules in tables unless the venue explicitly requires them.

The glossary package should be used for managing acronyms and abbreviations. The glossary package is incredibly useful for managing acronyms. For example, \newacronym{TAMP}{\textsc{tamp}}{task and motion planning} at the top of the file will expand \gls{TAMP} as “task and motion planning (tamp)” on the first use and then simply tamp on subsequent calls. It is also possible to define custom abbreviation styles for abbreviations that shouldn’t be expanded at the outset. Here is an example configuration one might use:

\usepackage[acronym]{glossaries-extra}
\glsdisablehyper % Avoid hyperref
\setabbreviationstyle[acronym]{long-short}
\setabbreviationstyle[short]{short-nolong}
\newacronym{POMDP}{\textsc{pomdp}}{partially observable Markov decision process}
\newacronym[category={short}]{PDDL}{\textsc{pddl}}{The Planning Domain Definition Language}

The glossary has helpful macros for other automatic formatting as well: \glspl will pluralize the acronym and \Gls will ensure the first letter is capitalized, useful at the beginning of the sentence.

The glossary has helpful macros for other automatic formatting as well: \glspl will pluralize the acronym and \Gls will ensure the first letter is capitalized, useful at the beginning of the sentence.

• Other commonly-used terms should be defined via custom functions. TODO with an example
• Make a custom todo or notes function for draft text, which clearly highlights/colors comments and other items and makes it easy to edit them out. I like to use the todonotes package for this. TODO

Figures

Figure Design

Try to make the most of the space you have and avoid large white space. TODO

Prefer labeling figure elements to describing them in the caption. Try to avoid captions like “the frontier to the left of the robot has high success probability.” Instead, place a short label and/or arrow directly on the figure indicating the frontier (or region, failure case, threshold, etc.). The reader should not be forced to glance back and forth between the figure and the caption to understand the figure’s meaning. Having labels for various items or subfigures appear near to the entities being discussed helps readability quite a lot. The tradeoff is that the figure becomes more cluttered, so use your judgment to maximize clarity.

Match figure typography to the paper typography whenever possible. Use the same (or visually compatible) font family as the paper, and choose sizes so the figure text does not “pop out” as mismatched relative to the surrounding body text.

Figure captions should be as self-contained as possible. A reader should be able to understand what the figure shows (and why it matters) by looking at the figure and reading the caption, without needing to hunt through the main text for basic context.

Figure Preparation

TODO: Always use software with a grid; powerpoint is not allowed for figure preparation.

Create figures at the size they will appear in the paper. Start with a canvas sized to the intended final footprint (e.g., one-column width vs. two-column width). Doing so makes it much easier to choose correct font sizes and line weights.

Figures take iteration. You rarely get a figure “right” by thinking about it. Expect multiple passes: adjust layout, typography, and annotation until it reads clearly at paper scale.

Preparing Drafts

Best Practices

When writing a draft, always start by using the template for the submission venue. TODO: important since it lets you format as you go and gives you an impression of the rough length of the document so far.

• (use the template provided by the conference or journal you plan to use).

Construct results figures and captions while writing the results section. What figures you include are often defined by what you say in the writing and what you write will be defined by what results you decide to show. Relatedly, write figure captions as you write the body text, as crafting captions is similarly informative.

Floats should not break columns of prose. Figures, tables, and code listings should typically appear at the top of a column (or page) unless there is a clear reason not to. With limited exceptions, avoid placements that split paragraphs or interrupt the reading flow in awkward places.

When asking for feedback, specify what kind of feedback you want. If you share a draft section or figure and ask “what do you think,” you will receive feedback on everything–including formatting. To avoid wasted cycles, include 1-2 sentences stating what you are ready for: e.g., “Please focus on the experimental story and caption clarity; ignore typography,” or “Please review the color scheme and readability at one-column scale.”

Do not outsource interpretation of your results. You cannot simply show plots/statistics and say “the results aren’t good” or ask “what do you think.” Come with a claim you are trying to support, your best interpretation of what the data says, and (if applicable) a short list of plausible causes/follow-ups.

Avoid the question “Do I have enough results?” Instead, ask a concrete question tied to the paper’s claims: “Do these results support Claim X?” “What ablation is missing to rule out Y?” “Is the comparison set adequate for the story we’re telling?”

Use visually distinct draft-note macros and keep them easy to remove. When drafting, use a dedicated macro or package (e.g., todonotes) so comments, open questions, and placeholders are obvious and can be cleanly disabled/removed for submission.

Language Use

This guide’s section on Language Use is far from comprehensive. I strongly recommend purchasing Strunk & White’s famous Elements of Style.

This guide’s section on Language Use is far from comprehensive. I strongly recommend purchasing Strunk & White’s famous Elements of Style.

Ensure proper use of which vs. that. When used to describe or clarify something, which follows a comma while that never does.

Avoid an unqualified this in your writing. The word “this” is a deictic reference: its meaning depends on context. Rather than the potentially ambiguous “This can be overcome by […],” prefer to ground “this” to avoid the ambiguity: “This pervasive challenge can be […].”

Avoid using etc., particularly at the end of a sentence. I have been trying to avoid the use of “etc.” in general, since its use tends to be lazy writing. Prefer instead to clarify the point or say something a bit more descriptive, such as other items.

Writing suggestions and guidance

Technical Details

For work involving planning or reinforcement learning: know the difference between $Q$ and $V$. $Q$ takes two arguments; $V$ takes one. If you have more inputs, they should be grouped into a single object, e.g., $V(\{x, y, z\})$

Consider this example: the robot is asked to serve water at a desk. If it behaves myopically—focusing only on the current task—it might pick up an empty cup from the desk, fill it with water at the sink, and return it to the desk. This solution is reasonable, as it efficiently completes the immediate task. If the user drinks the water and then asks for more, the robot must then repeat the same sequence of operations to pick up the cup, go all the way back to the sink, fill it up, and then return. Over time, this short-sighted approach of only caring about the current task and ignoring the future tasks can lead to higher cumulative effort or cost.

Instead, if the robot were to know that the user would ask for more water on a few occasions, the robot should choose a different strategy: instead of using a cup, the robot could bring a larger water-filled jar to the desk. While using a jar may initially seem more costly from the perspective of completing a single task, yet becomes more efficient when considering the entire task sequence, since the jar can serve multiple future refillings without requiring repeated trips to the sink. Clearly, being able to anticipate how its actions may impact future tasks. The goal of our anticipatory planning is just this: to estimate the impact immediate actions are expected to have on subsequent tasks the robot may later be assigned

Myopic Approach: Completes the first task with a lower immediate cost but ignores the possible higher expected cost for the next task. Myopic Approach: Completes the first task with a lower immediate cost but ignores the possible higher expected cost for the next task.

Our Approach: Completes the first task with a slightly higher cost now but reduces the cost of the future tasks thus reduces the overall cost of completing all the tasks. Our Approach: Completes the first task with a slightly higher cost now but reduces the cost of the future tasks thus reduces the overall cost of completing all the tasks.

Anticipatory planning [1] is formalized as planning to reduce the joint cost of (i) completing the current task and (ii) the expected cost of completing future tasks. We use learning to esimate the expected future cost, so as to mitigate the computational burden of computing expected future cost directly. Scaling up anticipatory planning to home-scale environments is difficult. Such spaces often have many rooms, dozens or even hundreds of objects, and a wide variety of possible tasks the robot may be assigned. As the environment grows in size and complexity, it becomes harder for the robot to both estimate and plan effectively at scale.

In our most recent work, we present a learning-augmented model-based anticipatory task planning framework tailored for home-scale environments. In the post below, we discuss our recent ICRA 2025 paper Anticipatory Planning for Performant Long-Lived Robot in Large-Scale Home-Like Environments which seeks to provide answers to these questions. This post is geared towards a somewhat technically minded audience. However, we have tried to provide enough context that we expect it will also be of interest to non-roboticists.

There is also a 3 minute video presentation that has significant overlap with this post that you may enjoy instead:

Real-World Challenge: Planning in Home-like Environments

While planning in a home like environment the environment is often going to be large and have many objects present. As the number of objects and possible tasks increases, it becomes harder both to estimate future costs accurately and to find efficient plans in a reasonable amount of time. But both issues need to be addressed if we are to use anticipatory planning in large environments.

Challenge: Planning at scale

In everyday household settings where there are many objects around, a task can often be done in several different ways. But checking every possible option of plan while also considering all the other objects takes too much time and computing power to be practical.

Given a task, there are often many plans that solve it. As the environments grows the number of plans that solve the task grows exponentially with more assets, making exhaustive search intractable. Given a task, there are often many plans that solve it. As the environments grows the number of plans that solve the task grows exponentially with more assets, making exhaustive search intractable.

Challenge: Learning at scale

Anticipatory planning [1] on robotics domain relies on learning to mitigate the computational burden of computing expected future costs, providing cost estimates that guide planning toward behaviors that avoid side effects and improve long-lived performance. However, to compute the expected future cost for large environment learning needs to scale such that anticipatory planning gains the flexibility to operate on any kind of home layout with any large number of objects present. At the same time the learning needs to be good enough to guide the search effectively. How to represent the environment so that learning can effectively scale for anticipatory planning remains a challenge.

Approach: Anticipatory planning in large scale environment

We present an approach for anticipatory planning at scale that seeks to minimize the immediate and future planning costs. Our approach involves (1) sampling plans that complete the given task and (2) iterating over the sampled plans to get the plan that minimizes the sum of immediate plan cost and future expected cost.

We use focused sampling to reduce the planning search space to allow anticipatory planning at scale

When the environment is large and diverse set of objects are present, a given task can be completed in multiple different ways. However, the number of such solution plan grows exponentially with more assets that limits exhaustive search. To help the robot come up with better ways to complete a task, we give it slightly modified versions of the original task. These versions include an additional task related to objects the robot might pass by while doing its job. By focusing only on objects in the vicinity of the base solution we limit the number of extra tasks it needs to consider. This helps the robot explore useful options within its computational budget.

To search over the space of solutions, we perturb the task specification by proposing alternate states for objects in the vicinity of the base plan. To search over the space of solutions, we perturb the task specification by proposing alternate states for objects in the vicinity of the base plan.

The robot then searches over these modified tasks to see which plan leads to the best outcome. For each version, it checks how much effort it takes to finish the task now (immediate cost) and how much it might help with tasks in the near future (expected future cost). It uses this information to pick the plan that saves the most overall time and effort. By narrowing down the choices and only keeping the helpful ones, our robot avoids wasting time on plans that do not lead to long-term benefits.

We use a gnn on the 3D scene graph representation to estimate expected future cost at scale

We represent the environment using a 3D scene graph—a structured, layered map that shows the layout of objects and how they relate to each other. This kind of graph helps us to represent environments with any number of assets presents and then use powerful learning tools such as Graph Neural Networks (gnns), which can learn patterns from the structure of the environment. In the graph, each object is a node with useful details like what it is, where it is, and what condition it’s in (for example, whether a cup is full or empty). The connections between objects (edges) show how they are arranged or interact. By using this graph structure, the gnn learns to estimate how much effort future tasks might take depending on how the environment currently looks while maintaining scalability for large environment.

The map is converted to a 3D Scene Graph to define the GNN. The map is converted to a 3D Scene Graph to define the GNN.

Preparation: Anticipatory Planning in advance of receiving a task

Preparation means the robot takes some time in advance—before it is given any task or it is idle—to get the environment ready in a way that makes future tasks easier. To do this, the robot tries out small changes to where things are or how they are arranged, one step at a time. If a change looks helpful or has a chance of being helpful, the robot keeps it and tries more changes from there. After repeating this process many times, it settles on a state of the environmental that is expected to make its upcoming tasks faster and less costly.

Results: anticipatory planning in home-scale environment

We test our approach in large simulated environments using ProcTHOR and a custom-designed restaurant setting. The tasks for ProcTHOR come from the alfred dataset and mainly involve moving objects from one place to another. To better evaluate our method in more complex situations, we also create a restaurant environment with tasks that involve serving, cleaning, and interacting with objects in more diverse ways. To highlight the scale and complexity of the environment, ProcTHOR includes a wide range of home layouts—from small 1-room apartments to large 14-room houses. Across these homes, there are over 1,600 household objects grouped into more than 100 categories. Each environment can have 50 to 200 different tasks based on the alfred dataset.

As our robot minimizes both the immediate plan cost and the future expected cost so while completing the current task of serving water at Table 1, it anticipates the possible future need to serve another cup of water at a nearby location. Thus, it decides to fill a jar with water that can hold multiple cups of water and keep it on Table 1, so that when it is asked to serve water again, it can do so at a lower cost.

Myopic Baseline: cost 2982. Not knowing what the future task would be, myopic planning serves water in cup going back and forth and incuring more cost for the total sequence. Myopic Baseline: cost 2982. Not knowing what the future task would be, myopic planning serves water in cup going back and forth and incuring more cost for the total sequence.

Our Anticipatory Planner: cost 1655. With anticiaptory planning our approach predicts possible future tasks and serves water in jar that saves another trip to water refill resulting in lower cost for the total sequence. Our Anticipatory Planner: cost 1655. With anticiaptory planning our approach predicts possible future tasks and serves water in jar that saves another trip to water refill resulting in lower cost for the total sequence.

Additionally, our robot can prepare the environment in advance. It does this by finding a setup where future tasks are likely to be easier and quicker to complete. The robot then adjusts the environment step by step, trying different changes. It repeats this process as many times as allowed before a new task is given. Preparing environment like this further improves performance by minimizing both current task future expected costs and thus reducing the total planning cost.

Example Showing Preparation Our robot can prepare the enviornment given enough time. Once the environment is prepared, the expected cost of future tasks is reduced. Example Showing Preparation Our robot can prepare the enviornment given enough time. Once the environment is prepared, the expected cost of future tasks is reduced.

Conclusion & References

We introduce a planning approach that helps a robot perform better in large, home-like environments. This approach helps the robot plan ahead by learning from experience. To make planning manageable at a large scale, we use a sampling strategy that narrows down the best options. We also combine a Graph Neural Network (gnn) with a 3D scene graph to help the robot learn from its surroundings and make learning possible at scale. Our results show that the robot can use this learning to prepare the environment in advance and make better decisions during tasks. As a result, it performs more efficiently across long sequences of tasks. We also want to highlight the fact that we care about using robots efficiently for task that human usually finds mundane (like folding laundry, cleaning up after meals) and allow humans instead to spend more time on creative things (like designing, acting, singing, drawing etc).

References

• R. Dhakal, M. R. Hossain Talukder and G. J. Stein, “Anticipatory Planning: Improving Long-Lived Planning by Estimating Expected Cost of Future Tasks,” 2023 IEEE International Conference on Robotics and Automation (ICRA), paper.

This document gives an overview of how I, as a faculty mentor, think about when and how often my PhD students need to meet with me and what the substance of those meetings should be. While every student and their needs are different, my hope with this guide is to outline a set of principles rather than a set of hard-coded meeting policies, so that the resulting set of meeting practices is flexible to changes in circumstance.

Why have meetings? What characterizes a “worthwhile” meeting?

My PhD students are both students and researchers: working to advance the forefront of human knowledge while also learning the skills necessary to do so. Meetings exist to help them along that path, to bring me up to speed about how their projects are going, and also make sure they and I come to a shared understanding of what we think are the best next steps. When I meet with a student, I am trying to determine if a student needs course correction: feedback from me to best guide them towards more productive learning or research progress.

With this in mind, I consider a meeting most worthwhile if: I have found that many students seem to believe that being stuck is a reason not to have a meeting, but in reality it’s quite the opposite. A student unable to make progress is often the best reason to have a meeting, so that we can work together towards a solution.

I have found that many students seem to believe that being stuck is a reason not to have a meeting, but in reality it’s quite the opposite. A student unable to make progress is often the best reason to have a meeting, so that we can work together towards a solution.

• I give a student direction on how to become unstuck: to make independent progress where they were previously unable. This can include also giving them references to papers or learning materials to better understand something they were not aware of before.
• The student and I collectively come to a new understanding about a problem. I don’t always (usually?) have the answers, and so discussions surrounding shared problems can be incredibly productive. As I’ll mention later, meetings are only typically productive in this way if the student has already put in considerable time or thought into trying to understand these problems on their own in advance of the meeting. Coming to my office with a simple I don’t know what to do next is rarely particularly valuable.
• I point out something the student has missed. This is a much more “organic” or “soft” consequence of having meetings and the most difficult to predict. When a student thinks they are making progress but may have overlooked or misunderstood something, my pointing it out can save them days of time or more. This third element—the “unknown unknowns” for meetings with one’s advisor—is the hardest for students to anticipate, and is a core reason to have meetings even when students don’t believe a meeting is needed.

Even being able to clearly articulate a new problem is productive! Understanding that a problem exists and making an effort to write it down are important steps towards being able to solve it.

Even being able to clearly articulate a new problem is productive! Understanding that a problem exists and making an effort to write it down are important steps towards being able to solve it.

Notably, a consequence of this rather utilitarian list is that “good news” isn’t valued, since it doesn’t really help me to help the student. Of course, I’m a human person, and so collectively celebrating progress or good news can make a meeting worthwhile, even if it usually isn’t time-effective to have an entire meeting devoted to everything we hoped would work now works and I don’t need any guidance.

How long should students go between having meetings? How often should meetings discuss long-horizon goals?

On the other side of the above equation is time: meetings have a cost for everyone involved and if meetings are not likely to prove useful, scheduling a meeting might be counterproductive. In short, students should typically not expect a meeting every day, as the meeting schedule then becomes burdensome. So how long should a student go between having a meeting?

Discussing Short-Term Progress

Students, even newer researchers, are generally pretty good at determining when their progress has slowed or when they have problems that could use further discussion. Thus, this is usually the easiest sort of meeting to schedule and most students will reach out to me unprompted to check in with the aim of getting unstuck every couple of weeks. I generally trust my students to see this sort of hurdle and so leave it to them to schedule this sort of meeting.As I discuss in the next section, students won’t always notice when they are stuck and so I still recommend that they provide a short written weekly check in to ensure they aren’t missing an avenue to improve.

As I discuss in the next section, students won’t always notice when they are stuck and so I still recommend that they provide a short written weekly check in to ensure they aren’t missing an avenue to improve.

Discussing Long-Term Goals

It is the tendency of most students to look at this question only through the lens of short term progress: Am I stuck right now? Do I know what task I’m working on next? However, it is just as important that meetings at least occasionally discuss the students overarching goals, including what projects they may want to work on next, the experiments they aim to conduct farther down the line, their graduation timeline, etc. In the absence of these discussions, most students will consider those questions only when necessary, and so it becomes more difficult to do much to change course.

As such, I ask that my students have a chat about their long-horizon objectives for at least 10–15 minutes every meeting, just so that they keep those in mind when thinking about their more immediate aims. I have found that this forces the students to think more broadly about the context in which their short-term goals exist and has led to a steady stream of aha moments where either I or the students realize there exist opportunities to spend a bit of effort now to preemptively advance these longer-term goals or to better align their short-term goals to advance the long-term ones. Though discussing long-term goals isn’t always how the students want to spend their time, I have seen these discussions result in compounding positive effects over time for the students that commit to thinking about them.

Regular check-ins are essential, even absent a “full meeting”

By definition, students cannot anticipate the unknown unknowns in their research. Meeting with their advisor is sometimes the only way that issues in their work will be discovered in advance of it becoming a significant problem. Short, written status updates between meetings can save considerable time.

Not every interaction between student and advisor should be an hour-long meeting!

Recently, I have been recommending that students who go more than a week without having an in-depth meeting provide me with a short written update of what they are working on and their progress. When I see something that I think requires following up, this becomes a trigger to have a meeting together to make sure we are on the same page and that no issue exists. This has proven a happy middle ground for everyone, since it means we don’t have to have as many meetings, but students don’t completely miss out on the benefits of chatting with me.

How should one-on-one meetings be structured? How should students prepare for meetings?

This is the part of the guide that I change most often and so I won’t include my exhaustive thoughts here.My lab’s meeting guide is available on Notion if you’d like to see my more comprehensive thoughts on the subject of meeting structure and preparation.

My lab’s meeting guide is available on Notion if you’d like to see my more comprehensive thoughts on the subject of meeting structure and preparation.

I require my students to prepare both the meeting agenda as well as putting together figures, updates, questions, etc. the evening before we meet and add it to our shared meeting space on Notion. This ensures that the student will have ample time to collect their thoughts and ensures that the meeting does not mostly consist of “oh, I forgot” items that significantly diminish productivity. In the morning before we meet, I’ll take a look at this document and add agenda items, my thoughts or insights, or questions for clarification. These additions help to make sure we don’t waste time and can be most productive.

Preparation in advance, mostly by the student by also by me, is essential for effective meetings. Absent preparation in advance, meetings tend to be unfocused and rambling. Relatedly, a clear agenda is a helpful starting point for a meeting, as it ensures that important topics are not missed.

Finally, meetings conclude with a discussion of long-term goals (as mentioned above) as well as goal setting for the next meeting. My student and I will end the meeting by establishing what they are working towards next and add a few stretch goals to the list for what else they might also try to make progress towards. During my preparation for subsequent meetings, I frequently review this list to ensure we are meeting our internal goals and, if we are falling behind, what we can do to improve or pivot.

There is no one-size-fits-all meeting style

Different students have different needs. Over the past few years—including the fully-remote COVID years—I have gone through a number of meeting styles, each with their own merits. Some students need the routine of regular meetings while other students might find them a distraction. At different stages of their research or level of experience, meetings may take on a different tone or even be entirely devoted to long-term goals or progress. This guide is just that: a guide. I am flexible with most students, particularly my more senior students, to let them take direction to structure meetings the way they feel most productive.

My general philosophy: try to meet fairly regularly, but always ensure that students check in even when a real-time meeting is not possible. Without these check-ins, progress will slow, as neither I nor the student see hidden opportunities to accelerate or course correct.

And if a student says they need your time, listen to them.

My most recent project has been to develop a simple proof-of-concept in which every aspect of running results to including them in a paper is automated. Simply download the repo from GitHub, run make paper, and it outputs a PDF file with the results from some simple experiments.

The Demo

After pulling the repository and moving to that directory, we can run:

make paper NUM_EXPERIMENTS=25

This simple command runs 25 “experiments”—a simple example task that fits a line to noisy data with and without outlier rejection and compares the two—and generates a plot that contains a scatter plot of those results and computes statistics for the results. This results in the following PDF, manually cropped to highlight the important bits:

Make paper: 25 experiments. The PDF includes the correct number of experiments, statistics, and a scaterplot of those results, all via make paper NUM_EXPERIMENTS=25. Make paper: 25 experiments. The PDF includes the correct number of experiments, statistics, and a scaterplot of those results, all via make paper NUM_EXPERIMENTS=25.

The PDF includes the number of experiments run, the mean performance with and without outlier rejection, and a scatter plot of the results: all of which are automatically read from the results and included when the PDF is generated via LaTeX.

Let’s say we now wanted more experiments. No problem! We run the command again, but increasing that parameter and add a -j20 for convenience so that the experiments run 20-at-a-time in parallel:

make paper NUM_EXPERIMENTS=800 -j20

which results in the following:

Make paper: 800 experiments. The PDF includes the correct number of experiments, statistics, and a scaterplot of those results. Make paper: 800 experiments. The PDF includes the correct number of experiments, statistics, and a scaterplot of those results.

Notice that all the relevant information has been updated automatically, including the plot and the number of experiments and statistics. This is because make takes care of updating the scatterplot as necessary to reflect the updated results and including the newly-computed statistics.

What happens if we rerun the command?

make paper NUM_EXPERIMENTS=800 -j20

We get the following:

make: Nothing to be done for `paper'.

make is designed so that it doesn’t redo computation. Since no new results have been added or changes made, the paper is already up-to-date and nothing needs to be done.

How does it all work? Take a look at the code or read on to learn more.

High-level overview of the process

We rely on GNU Make to build the outputs. Upong running make paper all targets upon which the paper relies—including things like the results statistics and the scatterplot—are generated. To generate those, the outputs upon which they rely are also generated, a process that repeats all the way back to the underlying experiments, run automatically to generated the

The following schematic of the dependency structure of the Makefile and so an overview of the outputs generated automatically to build the PDF of the paper:

A schematic of the Make target dependencies. This schematic shows how the different outputs from the code, including the paper itself, depend on others. When running make paper, all upstream outputs are built as necessary to get the results the paper requires. A schematic of the Make target dependencies. This schematic shows how the different outputs from the code, including the paper itself, depend on others. When running make paper, all upstream outputs are built as necessary to get the results the paper requires.

Run experiments with GNU Make

While the fundamentals of GNU Make are beyond the scope of this post, we take advantage of some fairly-specific setup to support changing the number of experiments.

The Make “target” is what is called whenever we want to generate a result. Let’s say we wanted to generate a result for random seed 105 using the least squares line fitting method, accomplished via

make results/results_lstsq_105.csv

We use the % wildcard character to define the Make target and use some simple string processing (via grep) to extract the seed and curve fitting approach from the name of the intended file:

results/results_%.csv: seed = $(shell echo $@ | grep -Eo '[0-9]+' | tail -1)
results/results_%.csv: approach = $(shell echo $@ | grep -Eo '(lstsq|ransac)' | tail -1)
results/results_%.csv:
        @echo "Evaluating result for $(approach): $(seed)\n"
        @mkdir -p results/
        @$(DOCKER_BASE) python3 /src/evaluate_approach.py \
                --seed $(seed) --approach $(approach)

The Python script here is just a simple example, meant to show how extracting some information from the name of the file to be generated can be passed as inputs to the script.

We separately define two variables in Make that automatically populate two lists with all the file names that we would like generated, corresponding to NUM_EXPERIMENTS experiments for each of the least squares (lstsq) and ransac outlier rejection (ransac) line fitting strategies:

eval-lstsq-seeds = \
        $(shell for ii in $$(seq 10000 $$((10000 + $(NUM_EXPERIMENTS) - 1))); \
                do echo "results/results_lstsq_$${ii}.csv"; done)
eval-ransac-seeds = \
        $(shell for ii in $$(seq 10000 $$((10000 + $(NUM_EXPERIMENTS) - 1))); \
                do echo "results/results_ransac_$${ii}.csv"; done)

Now, if we had a separate make target that depended on eval-lstsq-seeds, it would automatically generate all those experiments before it could be run. Adding dependencies like this allows us to enforce that all experiments be finished before we try to perform operations to compute statistics from the data or generate the scatterplot.

Process results and output figures

Generating results and statistics requires a few more Make targets. For example, here is the Make target used to generate the scatterplot via Python:

results/processed_scatterplot.png: src/process_results.py $(eval-lstsq-seeds) $(eval-ransac-seeds)
	@echo "Generating the results scatterplot."
	@$(DOCKER_BASE) python3 /src/process_results.py --output scatterplot

This yields the following figure:

The scatterplot automatically output from Python. Note that it lacks the annotations that appear in the examples above. The scatterplot automatically output from Python. Note that it lacks the annotations that appear in the examples above.

You might notice that the figure lacks the annotations that appear in the PDF. It is often unrealistic to add every annotation and highlight in the Python specification, and so other design software is often used to include such additions by hand, manually placing annotations and text where appropriate to make an effective figure.

For those of you who have used Python’s Matplotlib to make figures, you will understand why trying to place every bit of text or annotation with code is typically non-ideal, often requiring far more time and effort than simply drawing a circle and adding a text box in some other program.

For those of you who have used Python’s Matplotlib to make figures, you will understand why trying to place every bit of text or annotation with code is typically non-ideal, often requiring far more time and effort than simply drawing a circle and adding a text box in some other program.

We use Inkscape—a free and open source vector design software—to add annotations to the figure. Inkscape, like other design software, supports linking resources: the Inkscape .svg file includes only a reference to the underlying scatterplot, so that the Inkscape version of the figure will reflect any changes to the linked images. The Inkscape document uses the underlying scatterplot and includes annotations:

Scatterplot with annotations from Inkscape. Scatterplot with annotations from Inkscape.

We have a separate Make target that converts the .svg Inkscape file to a .png, so that it may be included in the PDF document later on. For this purpose, we run a headless instance of Inkscape inside a Docker container, avoiding the need to find or run a local instance of Inkscape.

Adding result statistics to the PDF

On Using Pure LaTeX: I show off Emacs here, since it is what I use when composing papers, but there are pure LaTeX solutions to this problem. See this Stack Overflow post about how to use the datatool package for this purpose.

On Using Pure LaTeX: I show off Emacs here, since it is what I use when composing papers, but there are pure LaTeX solutions to this problem. See this Stack Overflow post about how to use the datatool package for this purpose.

In this demo, I use Emacs’ Org-mode as a markup language to compose the paper. Org-mode is a powerful and customizable note taking environment that lets me write a document in a fairly simple syntax and then export it to a .tex file, from which a PDF can be generated. In addition, Org-mode lets me run arbitrary Python code upon export, letting me load a Python .pickle file from data and incorporate the data contained within in the document.

Here is a simple example with code from the paper that: (1) defines the pythonval macro for printing values from Python; (2) defines some LaTeX-specific export options; (3) runs a src block that loads the data file output from our Python commands above; and (4) includes some exmple text:

* LaTeX Configuration :noexport:

#+macro: pythonval src_python[:session :results raw]{$1}
#+LATEX_CLASS_OPTIONS: [10pt]

#+begin_src python :session :results none :exports none
import pickle
with open("../results/processed_results_data.pickle", 'rb') as handle:
    d = pickle.load(handle)
#+end_src

* Some text for the paper

We ran {{{pythonval(d['num_seeds'])}}} experiments 
and saw an improvement of {{{pythonval(f"{100*(d['mse_lstsq'] - 
d['mse_ransac'])/d['mse_lstsq']:0.1f}")}}}%.

Exporting this code to a PDF, the resulting .tex file includes the values computed via python and so yields the examples above.

Conclusion: automation in the RAIL Group

This pipeline gives an overview of how we automate and streamline experimentation in my research lab. We routinely build research pipelines around automated running of experiments as necessary. This level of automated running of experiments dramatically helps facilitate the trying of new ideas. Not only has it proven an effective tool for reproducible research, but also it has made it easy to quickly run additional experiments in the run up to a paper deadline and generate new results in response to reviewers.

See also our lab’s public code repository for examples of our Make- and Docker-based workflow in practice.

Any questions or comments, feel free to reach out to me on Twitter.

(1) Trying to reach a faraway goal, the robot sees the sign and turns right. (2) Taking right, the sign is no longer in view. If the robot does not remember what it has seen before, it cannot predict about the unseen space (1) Trying to reach a faraway goal, the robot sees the sign and turns right. (2) Taking right, the sign is no longer in view. If the robot does not remember what it has seen before, it cannot predict about the unseen space

To navigate efficiently, the robot may need to remember what it has seen and make use of that information to inform its decision-making.

Consider the following scenario, in which the robot starts in a location where it must decide whether to head left or right at an intersection, yet lacks the information it needs to determine which way it should go. If the robot remembers what lies around the corner—a region whose color indicates which hallway it should follow—it can make more effective decisions about where to head and reach the goal more quickly.

In order to plan, the robot makes predictions about unseen space. Those prediction may require non-local information. In order to plan, the robot makes predictions about unseen space. Those prediction may require non-local information.

The robot cannot effectively decide where to go if it cannot make use of the non-local information from its starting location in the center of the map.

Challenge: reliable navigation under uncertainty that leverages non-local information

To improve long-horizon navigation in partiallly-mapped environments, we want an approach that:

• (1) is reliable, guarantee to reach the goal when there exists a solution, and
• (2) has the capacity to make use of non-local information to make prediction about unseen space to guide navigation

Many learning-driven approaches—including many model-free approaches trained via deep reinforcement learning—have demonstrated the capacity to perform well in this domain. However, in the absence of an explicit map, many of these approaches are unreliable and so lack a guarantee to reach the goal.

Learning over Subgoals Planning The recent Learning over Subgoals Planning (LSP) approach of Stein et al. (2018) introduces a high-level action abstraction for planning in a partial map where actions represent exploring boundaries beyond known space into unseen space. This abstraction allows for both state-of-the-art performance when planning for long horizon and reliability-by-design. However, LSP is limited: its ability to make predictions about unseen space only makes use of locally observable information, limiting its performance.

When goal is around 100 meters away in the direction of a classroom door, the Learing over Subgoal Planning approach avoids entering the classroom, predicting it will likely be a dead-end. When goal is around 100 meters away in the direction of a classroom door, the Learing over Subgoal Planning approach avoids entering the classroom, predicting it will likely be a dead-end.

Approach

• 1 Use the reliable-by-design model-based LSP planning abstraction
• 2 Create a graph representation for non-local information and use a graph neural network (GNN) for learning to inform good behavior

We maintain reliability using LSP action abstraction

LSP uses high-level action abstraction to navigate long-horizon efficiently under uncertainty. This model-based planning abstraction alleviates the computational requirements of POMDP planning. For LSP planning, actions available to the robot have binary outcomes of either reaching the goal and incurring a success cost or failing and accumulating exploration cost.

The simplified high-level action abstraction of LSP allows for reliable-by-design long horizon navigation under uncertainty even when predictions about unseen space are not good. The simplified high-level action abstraction of LSP allows for reliable-by-design long horizon navigation under uncertainty even when predictions about unseen space are not good.

Under this abstraction, the expected cost of a high-level action is determined via a Bellman Equation:

where $D$ is the known space travel cost from robot pose $q_t$ with the partial map $m_t$ by taking the action $a_t$, $P_S$ is the likelihood of success for an action, $R_S$ is the expected success cost if goal can be reached, and $R_E$ is the expected exploration cost otherwise.

For more details on how the Learning over Subgoal Planning approach works, please see our blog post.

We compute a graph representation of the map for learning via a graph neural network (GNN)

To learn from non-local information, we generate a simplified (few-node) graph of the environment using a skeletonization process that preserves the high-level topological structure, where nodes exist at (i) intersections, (ii) dead-ends, and (iii) subgoals. Then we feed the graph into a Graph Neural Network (GNN) to estimate the subgoal properties.

A schematic of our graph creation process, in which a skeletonize routine spartisifies the map and adds nodes at dead ends, intersections, and frontiers. A schematic of our graph creation process, in which a skeletonize routine spartisifies the map and adds nodes at dead ends, intersections, and frontiers.

In the graph—shown on the right side—the dead-end nodes (with degree 1) and the intersection nodes (with degree > 2) are represented by the structural nodes: ‘+’, and the subgoals are represented by subgoal nodes: ‘•’ In the graph—shown on the right side—the dead-end nodes (with degree 1) and the intersection nodes (with degree > 2) are represented by the structural nodes: ‘+’, and the subgoals are represented by subgoal nodes: ‘•’

Here, we show a short video of the robot navigating through the environment alongside a graph representation of the environment:

Left: subgoal abstraction view. Right: skeleton view overlaied on underlying known map. Left: subgoal abstraction view. Right: skeleton view overlaied on underlying known map.

Result: our approach reaches the goal quickly

Our approach learns how a dead-end room looks like and uses this non-local information to improve its prediction avoiding it later on to quickly reach the goal.

LSP-GNN uses non-local information to make predictions about unseen space to quickly reach the goal, as shown here in samples from our MIT Floor Plan environment. LSP-GNN uses non-local information to make predictions about unseen space to quickly reach the goal, as shown here in samples from our MIT Floor Plan environment.

We include results on a simulated robot in both procedurally-generated hallway-like environments and also in environments generated from occupancy maps of floorplans around MIT’s campus. In all environments, our LSP-GNN approach outperforms both learned and non-learned baselines.

See our paper for additional results.

LSP-GNN (ours) uses non-local information to make predictions about unseen space to quickly reach the goal. LSP-GNN (ours) uses non-local information to make predictions about unseen space to quickly reach the goal.

Conclusion & References

We present a reliable model-based planning approach that uses a graph neural network to estimate the goodness of goal- directed high-level actions from both local and non-local information, improving navigation under uncertainty. Our planning approach takes advantage of non-local information to improve predictions about unseen space and quickly reach the unseen goal.

In future work, we envision passing more complex sensory input to the robot, allowing it to estimate the goodness of its actions using information collected from image sensors or semantically-segmented images.

Read our full paper or feel free to explore our code on GitHub.

References

• Gregory J. Stein, Christopher Bradley, and Nicholas Roy. “Learning over Subgoals for Efficient Navigation of Structured, Unknown Environments”. In: Conference on Robot Learning (CoRL). 2018. paper, talk (14 min).
• Raihan Islam Arnob and Gregory J. Stein. “Improving Reliable Navigation under Uncertainty via Predictions Informed by Non-Local Information.” In: International Conference on Intelligent Robots and Systems (IROS). 2023. paper.

At a fork while trying to find the goal that exists to the left side of the fork, a policy that guides the robot right will incur a greater distance. At a fork while trying to find the goal that exists to the left side of the fork, a policy that guides the robot right will incur a greater distance.

If the robot’s policy in this scenario is to navigate towards the left side of the fork where the floor has a green path, then we know that the robot followed the correct path to the goal. However, if the robot’s policy is to navigate towards the right side of the fork where the floor does not seem to have a green path, we know that the robot will go towards a dead end from which it will have to ultimately backtrack resulting in a greater travel distance. ​

A policy that guides the robot towards the right side of the fork performs poorly compared to the policy guides the robot towards the left side of the fork. A policy that guides the robot towards the right side of the fork performs poorly compared to the policy guides the robot towards the left side of the fork.

In these two scenarios, we know that any policy that guides the robot towards the right side of the fork will likely result in poor performance compared to a policy that follows the green path at every fork. Indeed, as shown in the figure above, this is what we observe when we deploy the robot with these policies. Given such knowledge, we would want the robot to avoid using a policy that we know will yield a poor performance and select a policy that we know will result in better performance—a problem we refer to as policy selection. The policy that guides the robot towards right goes on to perform even more poorly later on, but the claim still holds even if it had performed better at a later stage during navigation.

The policy that guides the robot towards right goes on to perform even more poorly later on, but the claim still holds even if it had performed better at a later stage during navigation.

However, how can we know which of the policies will result in good performance and which will result in poor performance? Well, we can let the robot deploy each policy one after another and evaluate which of these policies guides the robot to the goal in minimum distance. But, this procedure is costly since we have to deploy the robot multiple times with each policy and wait for it to reach the goal—or fail to do so—before we can determine the best policy, which minimizes distance traveled to reach the goal.

In our work, we develop a procedure that allows a robot to quickly identify which of its policies will result in the best performance upon deployment without having to deploy the robot with all of its policies. Quick identification of the best policy is enabled by our novel offline alt-policy replay approach which effectively allows the robot to imagine what it would have done had it instead used a different policy, thereby sparing the robot from having to deploy all of its policies.

In this article, we discuss our recent IROS 2023 paper Data-Efficient Policy Selection for Navigation in Partial Maps via Subgoal-Based Abstraction which presents a data-efficient policy selection approach in which a robot tasked to navigate to an unseen goal in partially-mapped environments can quickly and reliably identify which of its policies will result in the best performance upon deployment. The article is targeted towards an audience with a basic understanding of robot navigation. However, I will try to provide relevant context or point to resources that might be useful for a more general audience wherever necessary. I have also attached a five-minute video presentation below which covers much of what is discussed in this article. Feel free to check it out before diving into the article.

Policy Selection as an Instance of Multi-armed Bandit Problem

One way of formulating the problem of identifying the best-performing policy from a family of policies is in terms of the multi-armed bandit problem—also known as the model selection problem. A multi-armed bandit problem is a sequential decision-making problem in which an agent must repeatedly choose between a set of actions, each of which has an unknown reward or cost associated with it. The goal of the agent is to maximize the total reward or minimize the total cost it receives over multiple trials. The name multi-armed bandit comes from an old term for slot machines, that each have one arm and steal your money. Multi-armed bandit can be thought of as a set of slot machines: many lever arms each with a different reward scheme.

The name multi-armed bandit comes from an old term for slot machines, that each have one arm and steal your money. Multi-armed bandit can be thought of as a set of slot machines: many lever arms each with a different reward scheme.

Sutton and Barto’s Reinforcement Learning book is a great reference if you want to dive deeper into multi-armed bandit (and RL in general). Here is a link to a PDF of the book the authors make available for free.

Sutton and Barto’s Reinforcement Learning book is a great reference if you want to dive deeper into multi-armed bandit (and RL in general). Here is a link to a PDF of the book the authors make available for free.

One might see that the problem of policy selection is quite similar to the multi-armed bandit problem. In fact, it is an instance of the multi-armed bandit problem. In policy selection, the agent is the robot, the actions are the policies the robot has to select among, and the reward or cost is the performance of the policy measured in terms of the distance traveled by the robot to reach the goal under that policy. The goal of the robot is to minimize the total distance traveled to reach the goal over multiple trials.

Once we instantiate policy selection as a multi-armed bandit problem, we can use bandit algorithms to identify the best-performing policy. For example, consider the well-known Upper Confidence Bound (UCB) bandit algorithm [Lai and Robbins, 1985] to select a policy in trial $k+1$ as described by the equation below.

Attentive readers may notice that the given equation represents a lower confidence bound (LCB) instead of a UCB since our performance estimates are represented as costs instead of rewards, and is therefore minimized. We use the more common term UCB to mean the approach in general rather than the upper bound itself.

Attentive readers may notice that the given equation represents a lower confidence bound (LCB) instead of a UCB since our performance estimates are represented as costs instead of rewards, and is therefore minimized. We use the more common term UCB to mean the approach in general rather than the upper bound itself.

where, $\mathcal{P}$ is the set of policies to choose from, $\bar{C}_k(\pi)$ is the mean deployment cost of a policy $\pi$ so far, $k$ is the total number of trials completed, $n_k(\pi)$ is the number of times policy $\pi$ has been selected, and $c$ is the exploration parameter.

Intuitively, UCB trades off between selecting the policies with better performance so far, also known as exploitation, and selecting policies that have the potential for better performance​, also known as exploration. However, such bandit algorithms are often black-box and generally require the robot to go through multiple deployment trials and incur poor behavior before such policies can be ruled out. This results in very slow convergence towards the best policy and accumulation of large regret. Besides, such a procedure is lengthy and expensive, especially for robot navigation where the result of a navigation trial with a policy is known only after the robot reaches the goal or when it fails to do so.

In our experiments with one of the simulated maze environments, UCB bandit takes many trials before converging towards the best policy accumulating large regret.​ In our experiments with one of the simulated maze environments, UCB bandit takes many trials before converging towards the best policy accumulating large regret.​

Accelerating Bandit-like Policy Selection

If we are to accelerate policy selection, we must be able to quickly tighten the bounds on expected performance for each policy, prioritizing the selection of the most promising policies with fewer trials. As such, we seek to constrain policy selection by determining how well an alternative policy could have performed if it had instead been in charge. While we cannot re-deploy the robot to repeat the same trial with another policy, information collected during the trial can be used to scrutinize alternative behaviors and potentially avoid unnecessary exploration. Such a white-box approach can be used to imbue the robot with an ability to introspect the behavior of alternative policies without the need for repeated and costly deployments.

In the section below, we discuss our novel offline alt-policy replay that enables such introspection of the behavior of alternative policies. This allows us to compute a lower bound on the performance of a policy without deploying it in an environment, and thereby more tightly constrain bandit-like selection.

Offline Alt-Policy Replay

In our novel offline alt-policy replay approach, we perform a simulated offline replay of a policy using the information collected by the robot during navigation under a different policy. Specifically, we use the final partial map and the images collected by the robot during a navigation trial to perform simulated navigation of the robot within the boundaries of the partial map with an alternative policy. We say offline replay because the robot doesn’t actually navigate in the environment, it only simulates the navigation based on the information collected during actual navigation.

We say offline replay because the robot doesn’t actually navigate in the environment, it only simulates the navigation based on the information collected during actual navigation.

Such offline replay gives us a lower bound on the deployment cost in case the alternative policy was instead in charge. This means that even after deploying the robot with only one policy, we now have some information about all other policies that can be used to inform bandit-like selection. To accommodate the new lower bound costs of policies computed with offline replay into a bandit-like selection procedure, we propose the Constrained UCB bandit equation as follows.

where $C^{\text{lb}}_k(\pi)$ is the lower bound on the deployment cost of a policy $\pi$ computed using offline alt-policy replay. Our Constrained UCB bandit policy selection with offline alt-policy replay is illustrated in the figure below.

Overview of Policy Selection with Offline Alt-Policy Replay​ Overview of Policy Selection with Offline Alt-Policy Replay​

This procedure runs as follows. Before a navigation trial, the robot selects one of the policies and uses it to navigate to the goal. ​

Navigation Trial: The robot has selected the policy LSPOffice which was trained in similar office environments and navigates to the goal location with this policy.​ Navigation Trial: The robot has selected the policy LSPOffice which was trained in similar office environments and navigates to the goal location with this policy.​

After the trial is complete, our offline alt-policy replay approach uses the partial map and images collected during the trial to compute lower bounds on the performance of other alternative policies without deploying.​

Offline Alt-Policy Replay: After the trial is complete, the robot uses the partial map and the images collected during the trial to replay a different policy LSPMaze. This policy was trained in maze environments and unsurprisingly performs poorly during offline replay. The lower bound costs are also shown in the figure for reference.​ Offline Alt-Policy Replay: After the trial is complete, the robot uses the partial map and the images collected during the trial to replay a different policy LSPMaze. This policy was trained in maze environments and unsurprisingly performs poorly during offline replay. The lower bound costs are also shown in the figure for reference.​

Navigation costs from deployment and lower bound costs from offline alt-policy replay are used by our Constrained UCB bandit policy selection algorithm to pick a policy for the next trial. This process is repeated for multiple trials and the robot keeps picking the most promising policy for every trial.​

Not all planning approaches are suitable for offline replay

Scrutinizing alternative behavior to determine the lower bound cost needed for selection via our constrained UCB bandit algorithm requires that we can perform offline alt-policy replay of robot behavior under an alternative policy without actually deploying the robot. In general, replaying a policy offline requires an ability to generate observations from poses the robot may not have visited, which for many learning-informed planning strategies in this domain will not accurately reflect how the policy would have behaved if it had been in control of the robot.

Many approaches to vision-informed navigation under uncertainty, particularly those relying on deep reinforcement learning [Kulhánek et al., 2019, Mirowski et al., 2016], require observations (images) from poses and vantage points not visited during the original trial and can be brittle to even small changes [Henderson et al., 2018], and so replay of such policies is unlikely to yield an accurate lower bound on cost. As such, we instead require an approach to planning that is robust to changes in viewpoint and a kind of policy that is robust to minor changes in robot pose and corresponding observations and can reliably reach the goal even in environments where learning informs poor behavior.

It is our key insight that the Learning over Subgoals Planning (LSP) [Stein et al.] is well-suited for offline alt-policy replay, as it is designed for long-horizon, learning informed planning in partially-mapped environments. In LSP, planning is model-based, and high-level actions correspond to exploration beyond frontiers which represent boundaries between free and unknown space. Learning is used only to make robot-pose-agnostic predictions about the goodness of each such exploratory action: e.g. likelihood that a frontier will lead to the unseen goal.

In LSP, actions correspond to navigating to frontiers which represent boundaries between free and unknown space.​ In LSP, actions correspond to navigating to frontiers which represent boundaries between free and unknown space.​

See our blog post or the LSP paper for more details on the Learning over Subgoals Planning approach.

See our blog post or the LSP paper for more details on the Learning over Subgoals Planning approach.

Since properties of exploratory actions are not dependent on where the robot is, offline replay of LSP-based policies yields accurate lower bounds on their performance making such policies well-suited for offline replay.

Our approach converges faster

We evaluate our approach in hundreds of simulated maze and office-like environments and compare it with the UCB bandit algorithm. In the figure below, we show results in Maze-Green environments demonstrating that our approach converges much faster than UCB bandit and has 88% lower cumulative regret.​ See our paper for more results including those in office-like environments.

See our paper for more results including those in office-like environments.

Results from Maze-Green environments show that our approach converges much faster than UCB bandit and has 88% lower cumulative regret. ​ Results from Maze-Green environments show that our approach converges much faster than UCB bandit and has 88% lower cumulative regret. ​

Conclusion and Future Work

We presented a data-efficient policy selection approach that leverages Learning over Subgoals Planning-enabled offline alt-policy replay to compute a lower bound on the performance of policies based on the partial map and images collected from the environment during navigation, and use a bandit-like method to identify the best-performing policy quickly. Our approach enables the learning-guided robot to reduce average navigation cost in a wide variety of partially-mapped environments by picking only those policies that are known to perform better or have the potential to do so thereby significantly reducing the cumulative regret compared to the baseline UCB bandit. In future, we hope to extend our work to perform policy selection with online retraining or adaptation of policies in new environments.

Read our full paper here or feel free to explore our code on GitHub.

Acknowledgements

This material is based upon work supported by the National Science Foundation (NSF) under Grant No. 2232733.

References

• Abhishek Paudel and Gregory J. Stein. Data-Efficient Policy Selection for Navigation in Partial Maps via Subgoal-Based Abstraction. International Conference on Intelligent Robots and Systems (IROS). 2023. paper link
• Lai, Tze Leung, Herbert, Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics. 1985.
• Kulhánek, Jonáš, Erik, Derner, Tim, De Bruin, Robert, Babuška. Vision-based navigation using deep reinforcement learning. European Conference on Mobile Robots (ECMR). 2019.
• Mirowski, Piotr, Razvan, Pascanu, Fabio, Viola, Hubert, Soyer, Andrew J, Ballard, Andrea, Banino, Misha, Denil, Ross, Goroshin, Laurent, Sifre, others. "Learning to navigate in complex environments". arXiv preprint arXiv:1611.03673. 2016.
• Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger. Deep Reinforcement Learning that Matters. AAAI Conference on Artificial Intelligence. 2018.
• Stein, Gregory J., Christopher, Bradley, Nicholas, Roy. Learning over Subgoals for Efficient Navigation of Structured, Unknown Environments. Conference on Robot Learning (CoRL). 2018.

I asked my students to prepare presentations in the style of Pecha Kucha—a slide deck of 18 slides that automatically advance every 20 seconds. On the day of the presentation, I randomized the presenters: they would be giving each other’s slides with only about 5 minutes of preparation time. Each of my students watched as their colleagues presented the slide deck they had painstakingly prepared over the previous few days.

The result: chaos, laughter, and an incredibly productive discussion about how to make effective slides.

Why randomize the presenters?

There is immense value in watching recordings of your own talks. I routinely tell my students to record and watch their own presentations as they make them–and before they cross my desk—since it will help them see details that they don’t notice while they focus on presenting. However, this approach is still only a half-measure, and it is still difficult to put your self in the shoes of another who is entirely unfamiliar with the material and grasp how well they understand the material. Even when watching your own recording, you still bring with you implicit knowledge about what you are trying to communicate. Instead,

Watching someone else give your presentation, you get to see how they interpret your slides, information you can use to improve the presentation in a way you likely would not be able to on your own.

In general, it is not necessary for slides to contain all the details you wish to communicate, but it is important to be conscious of when details are intentionally not on the slides versus accidentally omitted.

In general, it is not necessary for slides to contain all the details you wish to communicate, but it is important to be conscious of when details are intentionally not on the slides versus accidentally omitted.

Why use the Pecha Kucha auto-advancing slide format?

I needed a way to encourage my students to commit to the presentation more than they would for an “ordinary” group meeting, so that the slides would be more self-contained than they usually would be, giving the presenters a chance at giving a reasonably effective presentation with minimal prep. The Pecha Kucha format was a novelty, and so was ideal for this purpose.

Moreover, in the absence of auto-advancing, I feared that students presenting a slide deck that was not very clear would tend to ramble, grasping at what they thought a slide was about rather than presenting what was actually on the screen. The Pecha Kucha format, in which slides auto-advance every 20 seconds, disallows rambling.

Finally, the very limited amount of time encourages one idea per slide in a way that non-auto-advancing slides do not. In advance of the meeting, I reminded my students to design sparse and effective slides, in which (i) slide titles quickly and clearly communicate the main message of the slide, (ii) there is only a limited amount of text on screen at once, and (iii) figures are fairly minimal and communicate only what is needed to convey the point, advice I regularly give.

See also my blog post on the differences between figures designed for papers and for presentations.

See also my blog post on the differences between figures designed for papers and for presentations.

Keeping the randomness alive

The meeting was incredibly successful. Even those presentations that were not as effective served as a learning opportunity to the designers, who used the experience to improve the slides going forwards. In addition, the meeting built up lab camaraderie and spawned a productive discussion over the group lunch that followed the meeting about particularly funny and interesting moments as well as strategies for generating better slides overall.

This sort of event is one that can only be done once, since the surprise and uniqueness is a core part of the fun. However, I still plan to add some elements of randomness to research update group meetings to encourage updates to be of a higher quality and more self-contained. I now bring a 20-sided die to my meetings: if it lands on 1 or 2, the students must present each other’s updates, if it lands on 19 or 20, I will present all their updates, just to keep everyone on their toes.

Questions or comments? Feel free to follow up on Twitter.

Best group meeting we've had: I asked my students to prep Pecha Kucha-style slides—18 slides, auto-advance every 20 seconds—and to really focus on readability/clarity. On the day, I had them present *each other's slides* (chosen randomly🎲) with 5min prep. Chaotic & fun 🧵[1/5]

— Gregory Stein (@GregoryJStein) September 12, 2023

Each summer, my lab hosts a handful of high school summer interns through George Mason’s fantastic Aspiring Scientists Summer Internship Program (assip) program. Not only is it a fun and educational experience for all involved, but an important benefit of participating in the program is the mentorship experience my PhD students get along the way.

Why Mentor?

PhD students often underestimate the need for experience beyond merely conducting experiments. Getting a PhD involves deeply understanding a topic and advancing the state of the field. As research does not happen in a vacuum, much of how that advancement happens is through engagement with others: formation of research ideas with your advisor and with colleagues or publicizing your work more broadly through blog posts or within the research community at conferences through talks, poster presentations, or informal conversations over meals. Being a successful researcher requires the ability to clearly communicate, distill complex ideas into core components, address (and ask) deep technical questions about those ideas, and identify through conversation where your listener might not be following along. To develop these skills, there is no substitute for experience.

Mentorship is an excellent source of practice for the intangible skills PhD students need. Teaching or training someone else requires good communication skills. Moreover, teaching complex ideas requires a deep level of understanding; trying to effectively explain concepts about which you have only a surface-level understanding will quickly reveal the boundary of your knowledge—to you, at least, if not the one you are trying to teach.

I often tell my students to be cautious of the sentiment that they ‘understand something but cannot put it into words.’ That ability to decode thoughts into prose is often a reflection of one’s level of understanding.

I often tell my students to be cautious of the sentiment that they ‘understand something but cannot put it into words.’ That ability to decode thoughts into prose is often a reflection of one’s level of understanding.

Beyond merely practicing communication, mentors will often learn through their mentees. Someone less experienced will come up with questions that even the most experienced mentors hadn’t considered. Through these questions, effective mentors seek opportunities to deepen their own understanding or hone in-progress ideas.

Onboarding my colleague Chris Bradley—now a long-time friend and collaborator—to our lab during graduate school helped refine our in-progress Learning over Subgoals Planning approach to learning-augmented planning under uncertainty, ideas now at the heart of my lab’s research.

Onboarding my colleague Chris Bradley—now a long-time friend and collaborator—to our lab during graduate school helped refine our in-progress Learning over Subgoals Planning approach to learning-augmented planning under uncertainty, ideas now at the heart of my lab’s research.

Finding Mentorship Opportunities

Mentorship takes many forms.

Structured mentorship opportunities during a PhD may involve guiding high school or undergraduate students through research, training new lab members, or even serving as a Teaching Assistant. However, many opportunities to mentor are informal, and include helping other students out with their experiments or chatting with them about their in-progress ideas over lunch or at a whiteboard. While many of these opportunities will organically arise—and a good PhD advisor will try to make sure these mentor/mentee relationships emerge—PhD students should seek out opportunities to mentor.

Research cannot be a solo exercise. I regularly tell my students to solicit feedback from other students, both inside and outside of my lab.

Research cannot be a solo exercise. I regularly tell my students to solicit feedback from other students, both inside and outside of my lab.

Some of the most valuable experiences during graduate school was in the informal mentorship relationships I undertook in my research lab and as communication coach through a peer-driven mentorship organization. Moreover, as a PhD student, mentoring under the supervision of your advisor provides you with a safety net: someone to reach out to when you’re not sure what guidance to provide.

Tell your PhD advisor or manager that you’re interested in mentoring others. Go out of your way to hear about your colleagues’ in-progress ideas at a white board or over lunch or coffee. You’ll be surprised at how it develops your skills as a researcher and deepens your professional relationships to those around you.

Questions or comments? Feel free to follow up on Twitter.

The 2023 high-school summer ASSIP students in my lab alongside me and their PhD student mentors. The 2023 high-school summer ASSIP students in my lab alongside me and their PhD student mentors.

Who is this guide for? While I have written this guide for my PhD students, it aims to help academics and working professions for whom creative problem solving is a central part of their responsibilities.

In my experience, research progress—at least in the sciences—can be grouped into two phases:

• Progress Towards a Milestone: For when one broadly knows what the next target is and needs to spend time making it happen.
• Idea Generation: The process of identifying and better understanding the challenges, problems and opportunities that can spawn new research or improve upon the existing state-of-the-art.

When in the milestone completion phase, it’s often fairly straightforward to understand how to make progress. By contrast, there is no formula for coming up with new ideas. Despite its importance, the amorphous nature of idea generation can make it feel frustrating; a lack of immediate progress may discourage, leading one to put idea generation on the backburner. Yet idea generation, broadly defined, is incredibly important for long-term progress.

Through my PhD and in my time since as an academic, I have found a number of strategies that have helped me to make progress towards new research ideas, advancing stuck projects, and writing grant proposals. This post is about the approach that I have found helps me make progress towards my open-ended research aims and to set a direction for my and my lab’s work; it outlines practices that I generally recommend my students follow.

Writing is a central component of idea generation and research progress

Writing forces me to clarify my thoughts.

In research, the best ideas often begin with a problem to be solved. Too many times, I have thought I understood a problem until I tried to write it out in full detail, to explain it to someone else, or to withstand deep questions about it. When writing something down, I must face head-on the ambiguities, contradictions, or unresolved issues that my mind may have glossed over. When honing an idea, I try to be as precise and complete as possible, so as to leave little ambiguity about the problem I am trying to solve. When I write, I do so with the intention of better understanding a problem or, if I think I already understand the problem, coming up with possible remedies to the core issue or experiments to run that can help me better understand it. For problems of appreciable size or complexity, it becomes impossible to identify the root cause of a problem or what to do about it without writing it out in full.

For me, an idea generation “session” is an unbroken period of time devoted to writing, typically an hour in length. For each writing session, I to set a well-defined goal for what I want to accomplish. I usually target some sort of intermediate progress milestone—more about that in the next section—or, failing to come up with one, set a target word limit for my writing; even 300 words of clear prose can represent solid progress towards better understanding a difficult problem. These intermediate milestones help me to feel as if I am making progress even when I do not complete a high-level objective.

Write with the aim of better understanding a problem and the limitations of existing approaches

Many of my best ideas come to me when I least expect them: when going for a walk, taking a shower, or talking with friends and colleagues. This raises the question, why have dedicated writing sessions for idea generation if many of the best ideas come from elsewhere? These sorts of epiphanies only occur for problems that I’ve already thought deeply about and tried (and failed) to make solid progress towards. The goal of my writing sessions is therefore less to produce breakthroughs as it is about readying the mind to have such breakthroughs.

As such, most of my writing sessions are devoted to better understanding a problem. Rather than tell myself during today’s writing session, I am going to come up with a solution to a problem, my intention for a writing will often include goals to precisely (mathematically) define the problem I am trying to solve or to describe in detail a scenario where I expect the current state-of-the-art will fail: i.e., an example that motivates the need for a new approach that overcomes this issue. If I do not have a clear picture of what problem I’m solving, the writing session is devoted to more precisely defining it.

Over time, I have amassed quite a list of problems I think are interesting. I have devoted significant effort towards understanding why those problems exist and, often, to identifying commonalities between them in an effort to discover an underlying root cause. It is from this list—and the process of creating it—that many of my research ideas have come about. Once I feel that I may have a potential idea of how to address one or more of the problems I have written about, I will spend a writing session expanding upon it. The resulting ideas then form the backbone of new experiments to validate my hypothesis.

In my field, these proof-of-concept experiments are an essential component of this process. The successes and failures that result from experiments are needed to make progress.

In my field, these proof-of-concept experiments are an essential component of this process. The successes and failures that result from experiments are needed to make progress.

This attitude also makes it easier to set intermediate progress milestones. It can be discouraging to sit down with an intention to come up with a new idea and then fail to do so. Instead, when focusing on clearly describing a problem scenario or a potential solution, adding particular details becomes a target and it becomes far easier to conclude a writing session feeling as if progress has been made.

Prefer short, regular writing sessions to long, infrequent sessions

It is difficult, if not impossible, to force a breakthrough. On the whole, I would say that fewer than half of my leaps in progress towards my research happen during dedicated sessions to make progress towards them. The rest happen when I least expect it: when going for a walk, taking a shower, or cooking dinner. A deep understanding of a problem only happens if you think about it often; writing is the most effective way to ensure that thought is deep and purposeful.

Pushing myself during long writing sessions often becomes counterproductive. Concentrating all the writing for a project to a single writing binge can be somewhat useful when focusing on a deadline or trying to enumerate all the details for an experiment or already well-formed idea. Yet when trying to reach a not-yet-well-defined goal or overcome a tricky problem, I can easily become frustrated after a few hours of feeling as if I have made little progress.

However, if I’m in the midst of a very productive writing session, I won’t force myself to stop. Preferring shorter writing sessions is a guideline to avoid burnout and frustration, but should not be rigidly enforced at the expense of progress.

However, if I’m in the midst of a very productive writing session, I won’t force myself to stop. Preferring shorter writing sessions is a guideline to avoid burnout and frustration, but should not be rigidly enforced at the expense of progress.

I make an effort to write every (work) day for roughly an hour. This amount of time is usually long enough that I can feel as if I’ve made progress without tiring me out. Writing every day ensures that I am regularly thinking about the challenges that are holding back me or my lab.

Relatedly, as a regular habit is essential, procrastination is the enemy. I try to sit down to write even on days where I feel I have little to say; I am frequently rewarded for the effort.

Find a setting that works for you

Where do you write? At what time of day do you write most productively? Do you listen to music, prefer the hum of a busy cafe, or require silence? Do you prefer bright lights, sunlight, or absolute darkness? Do you prefer to type when you write or hand-write on paper or a tablet?

As a PhD student, I did not give much thought to these questions, yet I now appreciate how important setting is to my writing productivity.

My writing is most productive under two conditions: (i) in the morning at my local cafe, a cup of coffee in my hand, sitting near the window to get lots of light and so that I can stare aimlessly into the distance without unnerving nearby patrons and (ii) late at night in my bedroom with the lights dimmed to near total darkness. I always listen to music using my noise-canceling headphones.

I am a huge fan of the Anjunadeep record label, whose continuously-mixed deep house compilation albums have fueled many of my writing sessions and best ideas since graduate school.

I am a huge fan of the Anjunadeep record label, whose continuously-mixed deep house compilation albums have fueled many of my writing sessions and best ideas since graduate school.

While I don’t fully understand why these are the conditions under which I write the most effectively, it was a process of protracted trial-and-error to discover them. Critically, I rarely write in the middle of the day, when my efforts are best spent on different activities. In developing a regular writing habit, it is important to consider such details.

Writing sessions are only as useful as what happens between them

The time between writing sessions is just as important as the writing itself. As I mentioned above, writing is to stimulate the mind to think more deeply about a problem. In addition to writing regularly, my other habits and practices help me to make progress.

Conduct experiments. I work in an experimentally-driven field, and so the purpose of writing is to generate hypotheses of how to make progress; thus, experiments are of central importance. In addition to running large-scale experiments to explore every facet of a problem, targeted experiments meant to better understand small pieces of it can be just as productive and with a faster iteration time. Frequently, my writing will expose some lack of knowledge of the nature of the problem and I must run some small proof-of-concept experiments before I am ready to make more progress.

Talk to others. Find a handful of people who are somewhat knowledgeable about the problems you are trying to solve. Explaining the problem to them and your thoughts on how to solve it can be an incredibly useful exercise. Moreover, those people should be willing to challenge you on those ideas and push you to clarify what you mean or hone in on where a proposed solution might not work. These dialogues—if approached from a place of curiosity or shared purpose, rather than with adversarial intent—can be incredibly productive for all involved.

Relatedly, we should all strive to be idea confidants for others.

Relatedly, we should all strive to be idea confidants for others.

Talk to yourself. I regularly think out loud. Whether I’m writing or driving to work, trying to speak my thoughts serves a similar function to writing by forcing me to clarify my often-underdeveloped internal musings. This doesn’t work for all, but is incredibly useful for me.

Keep an open mind. Writing need not be so rigid and, in fact, I often ensure that every week or so, I devote a writing session to open-ended brainstorming about a challenging problem or idea. Research requires a certain amount of creativity and such sessions can help to produce new ideas or help me to see an old problem from a new angle.

Keep a list of thoughts to revisit later. Focus during a writing session is important. I am easily distracted, not just by noise and external distractions, but also by potentially promising ideas that deviate from my writing goal. Sometimes I’ll pursue these in the moment, but more often, it is best to add it to a list of ideas I want to revisit. Even outside of a writing session, I may have an idea to come back to later and so I have a note on my phone (always on hand) where I can keep such scribbles. I typically have one list for each project I am working towards.

Go outside. Take breaks. Eat well. Exercise. Sleep. Balance is important. The mind cannot operate most effectively without a healthy body. Sleep is particularly important. While I occasionally trade sleep in pursuit of short-term productivity, virtually none of my best ideas have come about when I have been sleep deprived.

Any writing habits of yours you would like to share? I welcome thoughts and feedback on Twitter.

Rather than putting code in a one-off script or by modifying another—often to be deleted shortly after being written—code is written in an automated test, which persists even after the code is considered stable. In my experience, it is pretty rare that code, particularly research code, will be written without any debugging along the way. Many less-experienced researchers will write code and have a large block of either plotting code or print statements that exist to convince themselves that their new code is working as expected. Once the code is believed to work, however, that code will either be commented out or deleted, so that it is not run in production, a process that has always felt to me like wasted effort. Whenever I want to add a new feature or functionality in my code, I start by setting up a test and populating it with an example scenario as I write the code, evolving both in tandem until I achieve some broader aim.

These ideas are consistent with the philosophy of Test Driven Development (TDD), which broadly refers to the practice of writing tests for an API in advance of its implementation.

These ideas are consistent with the philosophy of Test Driven Development (TDD), which broadly refers to the practice of writing tests for an API in advance of its implementation.

This guidance applies both during initial development, when going back over older code, and when tracking down bugs or other unwanted behavior. Though it takes a bit of extra effort to set up the boilerplate code to set up and tear down the scenario being tested, the code I would once have written and then discarded lives on in my testing suite and is tested occasionally to ensure the new functionality is still performing as expected after I’ve moved on.

What about visualization code? Keep that too! Sometimes there’s no substitute to just looking at something when you’re trying to write code. Many of my automated tests have an optional do_debug_plot flag that also generates plots for me to look at or inspect in case I ever need to look more closely at that functionality again. Much of my work interacts with computer vision and so having the ability to lay my eyes on the result is sometimes just what I need when I’m trying to debug or understand something.

I have also discovered over time that I have evolved so as to write new code such that it is as easy as possible to test. This observation matches a similar thought from developer kovarex from the wonderfully addicting game Factorio. Here is an except from a developer blog post of his concerning Test Driven Development (TDD):

TDD actually is the constant fast switching between extending the tests and making them pass continuously. So as you write tests, you write code to satisfy them basically at the same time. This allows you to instantly test what you write, and mainly use tests as specification of what the code should actually do, which guides the thought process to make you think about where you are headed to, and to write code that is more structured and testable from the very beginning.

Starting my development process in tests has led to overall code readability improvements and has made it easier to refactor and extend the code when it became necessary later on.

My guidance above is really only a stepping stone to more comprehensive testing practices.

See also, my colleagues Paul Ammann and Jeff Offutt’s popular (and excellent) book on Software Testing.

See also, my colleagues Paul Ammann and Jeff Offutt’s popular (and excellent) book on Software Testing.

In addition to these sorts of integration tests, which involve running multiple parts of the code at once and testing them in unison, it is still critical to take the time to write both unit tests, to more rigorously test functions in relative isolation, and regression tests, which involve writing a test to reproduce and squash bugs found later on. There’s both an art and a science to testing and my heuristics for when to write additional tests and how in-depth they should be vary according to both time and application.

I welcome thoughts and feedback on Twitter.