katnykiel –

Scrumtui, A Local Scrum System

Sun, 12 Jul 2026 00:00:00 +0000

A few years ago, a colleague proposed to me that research, at its core, is an exercise in information organization. They believed that doing high quality research via the process of reading papers, performing experiments, and publishing results does not require talent or focus, but rather organization. Unable to evaluate the veracity of this claim at that point of my PhD, I took it to heart, and have spent a perhaps inordinate amount of my time on organization. I hope that my past post on the data I collected throughout my PhD lends some evidence to this.

One avenue of organization I devoted a lot of thought to was how I should keep track of the work I need to do. Some people use sticky notes, pen-and-paper TODO lists, or specialized apps. For a while, I used a digital TODO list (read: a markdown file), but it became overwhelming as issues piled up and I became unable to distinguish what work I needed to prioritize. After exploring other systems, I finally landed on scrum. Put simply, scrum is a project management system where work is broken down into tasks, which are completed within time-boxed iterations, called sprints. There’s a lot of jargon in scrum, but to keep it simple:

sprints are week- or month-long periods of focused work
issues are the individual tasks, usually 1–2 hours of deep work
backlog is the list of issues

For example, once a week I selected the issues of highest priority from my backlog, grouped them into a sprint, and then dedicated the week only to those issues and any urgent issues that arose. While distributed scrum in industry settings can often serve as a method for product managers to overwork and stifle their employees, I’ve found that single-person scrum works extremely well for managing my research. Specifically, the act of planning out the work I intended to do each week and checking in each morning over coffee or tea greatly helped focus my priorities for the day.

I started by using Jira for scrum, as they offer a generous free cloud service. The cloud aspect bothered me, as did the slowness and gross number of features, but I was unable to find any comparable local alternatives. Many of the local options were overkill for my use case, given that it’s just one-person scrum. Eventually, I decided to vibe-code a solution, with the following requirements:

local (no data stored in the cloud)
TUI (cleaner UI)
minimal (easier to use and debug)
vim keybinds (no mouse needed)
Rust (fast and safe)
feature parity with everything I used in Jira (backlog, kanban, epic, and sprint history views)

After a few weeks of development, I feel comfortable sharing this for others to use. It’s primarily been tested on macOS, NixOS, and Arch, but it should generally work anywhere Rust is installed. I’ve shared some screenshots of the four views below, with the first two using demo data and the latter two using historical Jira data I’ve imported into this system.

Check it out here, or in the screenshots below.

Reflection on a Hiatus from Academia

Mon, 06 Jul 2026 00:00:00 +0000

At this point in my life, I feel incredibly happy and grateful. I have an amazing, loving girlfriend, I’ve secured my dream job, my ongoing health issues have been improving, and I’m spending the summer traveling the US to visit friends and family. I believe a huge contributor to this joy is that I’ve been fortunate enough to take the last six months off from work while I searched for my next career move. After about thirty applications, I received an offer from my top pick: a postdoctoral fellowship in Berlin, Germany. This is the type of work that I’ve dreamed of doing for years: intellectually challenging, consistent with my values of sustainability and open science, and enough compensation to pay the bills. As I’m incredibly grateful to be at this point, I’d like to reflect on what I did during my break from academia to get me here.

Put simply, I’ve used this time to pursue what the stress of graduate school had kept from me. I started off this break extremely productively, with the momentum of graduate school carrying me. In these months, I:

learned guitar, and played more bass, cello, and piano
removed a trove of old big tech data and accounts, set up RSS feeds for media consumption, and made several strides towards self-hosting
went camping for the first time in years, and visited a few state and national parks
traveled to NYC (x2), Boston, Chicago, Austin, Philadelphia, and in a few weeks I’ll be going to Los Alamos
downsized much of my clothing and possessions in preparation for a move to Europe
vibe-coded several projects, my favorite being this local TODO system which will eventually get its own post
got back into running and volleyball, and relieved chronic issues I picked up during graduate school
started a book club

All of these pursuits have been great for my mental and physical health. However, this break hasn’t been without trial. As the months dragged on, I felt unmoored as I realized how critical the stable identity of graduate student had been to me, and I struggled to fill my days with activities to derive meaning. In my experience, the identity of a graduate student was a warm blanket to insulate me from a cold world. I felt a persistent sense that being a graduate student meant bettering myself and pursuing a passion. This was reinforced by those around me who treated the title as something worthy of admiration. After graduating, I no longer felt that insulation. This was especially true as I went through my job search, and as the rejections piled up I felt that it all may have been in vain. This was somewhat alleviated when I received my top offer. I was disconcerted by how strongly this influenced my sense of self, and I think that more reflection is needed to decouple my work from my identity.

To end on a lighter note, I think that my favorite endeavor on my hiatus has been to refine my culinary skills! As a vegetarian (and often vegan), I’ve been cooking for myself for years, and I’ve historically considered myself to be a decent cook. However, faced with a multitude of empty days ahead of me, I decided to dedicate more time to cooking and pick up those skills which have eluded me. In this time, I:

worked on my knife skills
learned how to properly mince garlic
learned how to use a stainless steel pan to make golden tofu or perfect fried eggs
perfected some staple sauces to my taste, like teriyaki, which I can throw on a variety of dishes
began to use more of what’s in my pantry to build meals, instead of just following recipes

That said, I should also mention that I leaned heavily on America’s Test Kitchen’s Vegetarian Cookbook and Deborah Madison’s Vegetarian Cooking for Everyone. I’ve shared the results of some of my favorite recipes from these books (and elsewhere) below. Enjoy!

Sushi! I did my best to imitate a local sushi restaurant by roasting some sliced sweet potatoes, then coating in panko breadcrumbs and frying to get them nice and crispy. With some cream cheese and teriyaki, it is absolutely delicious and my girlfriend’s favorite dish of mine. She made some onigiri!

My first sourdough! I had this with some lovely leek and potato soup from America’s Test Kitchen’s vegetarian cookbook.

This lion’s mane pasta with roasted asparagus remains my favorite dish I’ve ever made. I found the mushrooms at a local farmers market, and I followed a recipe from Table for Two to sauté in thyme and butter in batches. This photo doesn’t do the dish justice.

An eggplant and summer vegetable gratin alongside an arugula salad with pecorino romano and toasted walnuts, both from Deborah Madison’s vegetarian cookbook.

My girlfriend and I made each other pasta dishes to use up some extra rigatoni! I made her pasta alla vodka, using a NYTimes recipe, while she made a variation of the Fettuccine and Sautéed Peppers with Parsley (from Madison’s cookbook) with garlic mayonnaise.

Most recently, I made some White Castle copycat sliders using America’s Test Kitchen’s recipe, substituting Impossible meat for ground beef. These were delicious and quite fun to make.

Not pictured: I’ve been using this recipe from America’s Test Kitchen to make tex-mex enchiladas at least once a week, and it has become one of my favorite dishes to make. The ratio of deliciousness to time investment is higher for this than any of the other dishes I’ve made.

Cosmetic bike locks

Sat, 20 Jun 2026 00:00:00 +0000

Did you know that one in seven bikes parked at Purdue University can be walked away with in under a minute?

In my few years living in West Lafayette, I’ve noticed a lot of bike locks which are cosmetic; that is, locks which appear sound at first glance but are topologically insecure on closer inspection. I spent a day meandering around Purdue University’s campus, recording a tally and photographing the different configurations. In total, I collected data on 270 bikes, and found roughly 6% of them to be entirely cosmetic or unlocked, with 15% removable with basic tools. See the chart below for a breakdown.

Interestingly, only one bike that I could find had the chassis and both wheels locked: the most secure configuration. I think that the 15% of bikes which are unlocked, cosmetic, or easily removable with tools speaks to how safe of a town West Lafayette is. It’s a tiny urban island in the middle of rural Indiana, cited as the most bike-friendly city in Indiana, and one of the few towns in the Midwest spared from Rust Belt decay. I kept my old bike unlocked for years, and it was never stolen. Now that I have a nicer bike (3500 miles and counting) I keep the chassis and front wheel locked with a u-lock and cable. Some lessons are better learned vicariously.

Below, I’ve compiled some photos of each of the possible bike lock configurations, as well as some unique and quirky ones I found. See below!

No bike lock

Cosmetic bike locks

Front wheel only

Chassis only

Chassis + front wheel

Chassis + back wheel

Chassis + both wheels

The only bike I could find which locked both tires and the chassis was this one, which used a chain held together with a Master Lock. I used to pick these locks in high school, so despite being the most topologically secure, this is in fact one of the least secure bikes I found.

Strange locks

This bike could be stolen with just some basic tools to remove the handlebar

And my favorite: these two bikes, chains rusting and rubber degrading, sharing a single lock around both chassis. I am fascinated to know the story behind it. Did the bikes belong to lovers who rode their bikes to campus together and shared a lock, abandoned after a breakup? Frugal roommates who shared a lock and moved on elsewhere? A stranger who locked their bike around another and left indefinitely, trapping their bike? Where was the second stranger’s bike lock? As often happens, answering one question opened the door to several more. I am left to continue wondering.

A PhD, in Data

Wed, 04 Mar 2026 00:00:00 +0000

In December, I was able to complete a dream I’ve had since I was a child: obtaining my PhD. I successfully defended my thesis, in which I used computational quantum mechanics and machine learning to discover new atomically-thin materials. It’s been quite an arduous journey to get across the finish line, and it’s taken a toll on me. Consequently, I’ve taken some time off to focus on my health, recenter my priorities, and look towards the next steps in my career. It therefore feels right at this moment to reflect on my time in grad school. While there are many possible avenues for this reflection, here I want to reflect on the data collected throughout my PhD. I expect to write later on more the qualitative aspects of graduate school. I start by asking the following question:

How much time did I actually spend on manuscripts and other graduate school responsibilities?

This isn’t meant to be a post about how difficult graduate school is. I intended it as an exploration of how much of graduate school can be measured, and what this measurement reveals. The working title for this post was a data audit, if that paints a clearer picture. Motivating this was a pervasive feeling of not doing enough that seems inseparable from graduate school. This is a structural issue; while this post won’t fix it, performing this data audit help me put into context the work I actually did and the time I actually spent.

First, I explored which data I have available. At the start of my PhD, I used a simple TODO list to track my tasks, with each item estimated to take one hour of focused work. As I became pulled into a dozen disparate projects, I found that a single TODO list became insufficient for this complexity and shifted to a personal scrum system. This forced me to become more reliable at estimating how much time a given task would take me, and I credit this system with keeping me organized and sane throughout grad school. I’ve used the estimated time for each task as the primary data source, and grouped each task into one of several categories.

Category	Description
Coursework	Classes, teaching, studying
Professional	Conferences, seminars, applications, NSF reports, etc.
Abandoned	Research which did not yield manuscripts
Paper	Research which did yield manuscripts

While this is not a perfect metric of how many hours I worked, it is a reasonable proxy. Notably, it misses hours spent attending lectures, and I don’t currently have a good way of accounting for this. The hours I’ve spent over the last few years, divided among these categories, are shown below as the number of hours spent per week on each project, averaged over each month of my PhD.

In total, I recorded over 6200 hours over four years and four months. The average works out to 30 hours per week; however, this average is skewed due to the period between July 2022 and January 2023 when I was transitioning between my TODO list and my scrum system. Removing these six months, the average increases to 32 hours per week, which sounds about right by my memory.

I initially hesitated to share this result, as some may look at this number and think I am falling short of the seemingly universal expectation of the 40-hour work week. In response, I’d first point skeptics to the months when I averaged 80-hour work weeks, such as my last semester, which contains three of my top five busiest months. The more nuanced counter is to suggest that raw hours worked do not correlate with raw output. I’ve embraced a philosophy of deep work: creating distraction-free environments where depth is rewarded and shallowness is discouraged. Shallow work, such as checking email or fracturing my attention between multiple projects, produce much less output than focused, uninterrupted sessions. Placing a high priority on deep work sessions has allowed me to produce more in less time: and rather than seeing the 80-hour work weeks as badges of honor, I see them as failures in time management. Deep work is difficult to cultivate, and not an infinite resource; I often found that I could only engage in 10–20 hours of deep work each week.

To me, the more interesting analysis instead lies in the percentage of time spent on each obligation. In total, I spent 60% of time on manuscripts, 10% on coursework, 25% on professional responsibilities, and 5% on abandoned projects. This 60% grew towards the end of my PhD, and mostly came from performing the research itself. While I don’t have the data to determine how much was spent on manuscript-writing vs. research, I would estimate a ratio of 1:3 or so. Notably, I only worked on 1–2 papers for the first three years of my PhD, but simultaneously 4-5 in the final year and a half. This seems to be fairly typical among my peers. The 5% of my time spent on abandoned projects was low to me; I expected this to be a lot higher, as graduate school can regularly feel like a continuous stream of failures. However, knowing that only 5% of my time was spent without resulting in a manuscript was a comforting reminder that my time in graduate school may have been more successful than I internalized. To further analyze the manuscript-writing process, I overlaid the cumulative time spent on each manuscript, as shown below.

Here, the X indicates that the paper has been published. I additionally removed one of the papers which I showed on the previous histogram (paper 2), as I was second author on this manuscript, and it was published with only 200 hours. While this is a small sample size, it would suggest that first-author papers take about 800 hours from start to completion, in about 30 months. The yellow line (paper 4) is nearing submission, adding further support for this hypothesis. This is way more hours and a much longer timeline than I would have expected! Presumably, this number goes down as graduate students become more familiar with their subject and the research process, but until I publish my other manuscripts I don’t have the data.

Taking this hypothesis of 800 hours as true for a moment, I did some quick back-of-the-envelope math. If PhD students are expected to write three first-author papers, as was the expectation in my group, and each paper takes 800 hours, a PhD in four years would devote 2400 to these papers alone. These would all need to be started within their first year. Extending this, if we generously assume 1200 hours on professional obligations (I logged 1600), 400 hours on coursework (I logged 600), and perform no research which is later abandoned, this sums to 4000 hours and equates to 20 hours a week. This represents a theoretical lower bound of responsibility that a perfect student, with no failed projects, would require. Coincidentally, this is the estimate which is used to determine graduate student stipends (20 hours per week). I ended up working 50% more than this, and I would venture that most graduate students I know work even more than I did. It’s an unreasonable system. Even for our perfect student, there’s no time left to write and defend a dissertation.

One Month with Linux in Academia, an Experiment

Mon, 08 Sep 2025 00:00:00 +0000

Throughout the four years of my PhD in computational materials science at Purdue University, I’ve spent absurd amounts of time with my computer. A back-of-the-envelope calculation would suggest it’s been over 80,000 hours! Knowing that I’ll be spending this inordinate amount of time this way, and motivated by this XKCD comic, I’ve invested a lot of time reducing friction in my computer use. To name a few of these optimizations: adopting alternative keyboard layouts, learning competitive typing, using a package manager, and using tiling window managers have all saved massive amounts of time. However, like many computer geeks before me, I repeatedly find myself asking one question over and over:

Which operating system should I use?

The choice of operating system (OS), the base layer of my computer, underpins the entire system I use to interact with it. Arguably, it’s been more debilitating to switch between OSes than it was to switch from QWERTY to COLEMAK. I’ve tried various operating systems the last four years, but I’ve never switched to Linux as my daily driver. Each time I try, I’m quickly tempted back by the convenience of macOS. To remedy this, I tried switching to Linux completely for a month, to find out what works, what doesn’t, and if it’s possible to survive using Linux in a competitive academic setting.

⚠️

I’d just like to interject for a moment. What you’re referring to as Linux, is in fact, GNU/Linux, or as I’ve recently taken to calling it, GNU plus Linux. Linux is not an operating system unto itself, but rather another free component of a fully functioning GNU system made useful by the GNU corelibs, shell utilities and vital system components comprising a full OS as defined by POSIX.

Previous experiences with Windows/macOS/Linux

Like most engineers, I started graduate school using Windows. It works, mostly! And if you’ve never explored outside it, there’s not much incentive to change. I used Windows for the first year of my PhD, tinkering with a Linux dual-boot for a while but never using it for my research. A week before my preliminary exams, my Windows laptop suddenly started crashing constantly, and I bought an M2 MacBook for something that just works to pass my exam. It worked beautifully!

I’d make the case that the M-series MacBook Airs are currently the best laptops you can buy for grad school. The all-metal chassis, passive cooling, and M-series chips make for extremely long battery life and snappy performance. I consistently get 12+ hours of battery life on my Mac. Beyond hardware, macOS is extremely smooth, with an intuitive UI and great package support through homebrew. While the devs at Asahi Linux have done some amazing work to get Linux running on M1/M2 MacBooks, the limited external monitor support is a deal-breaker in graduate school, where giving presentations and seminars are a regular occurrence.

The Experiment: Linux for One Month

This idea was motivated by the Old Computer Challenge, which encourages us to experience older computer hardware to develop a deeper appreciation of our technology. I dug up my old laptop, installed Linux, and decided I would try to stick it out for a month. Hardware-wise, it’s not exactly a fair comparison. The newer M2 Mac easily beats my Dell XPS on most benchmarks, despite having paid twice as much for my XPS. I’ve attempted to account for this disparity in performance in this post. Additionally, I restricted myself to only use free software, using LibreOffice for document editing and my browser for video conferencing.

ℹ️

Okay, but which Linux distro? Just choose one that works for you! Everyone has different levels of experience, enthusiasm, and philosophies with Linux. I personally use NixOS; while the learning curve has been steep, reproducible and declarative builds are well worth the effort to me. For an easier time, Fedora is an easy-to-use distro with great package support.

The Good

Immediately, I was refreshed to see how smooth the installation and configuration process was. After spending years on using high-performance computing systems running Linux, I felt at home on the command line interface. Setting up my tiling window manager, desktop environment, and using my package manager were a breeze. Having that level of control over my system was extremely satisfying. Additionally, this meant that the time I spent configuring my system was often transferable to my research, unlike more OS-specific configuration done on macOS or Windows. I’ve since been using many utilities I discovered in this experiment for my supercomputer projects.

Despite the strict restriction of only using free software, I found that nearly all the software I frequently use was well-supported on Linux. Very rarely, I had to compile something myself. I’ve previously felt cognitive dissonance knowing the benefits of free software and yet using closed-source macOS. It became much easier to sleep at night after resolving this dissonance and using open-source Linux. This appeased some of my privacy concerns as well. In many cases, I was pleasantly surprised to find better software on Linux than I had on macOS; for example, the tiling window managers are significantly more limited on macOS.

The Bad

There’s an expression that goes around online Linux circles often, that Linux is free if you don’t value your time! This experiment was an attempt to test that hypothesis; unfortunately, there proved to be some truth to it. Initially, I was forced to spend a lot of time configuring my system in ways I wouldn’t have to on macOS. It’s worth noting that near the end of the month, this time began to decrease and plateau. Using Linux is a commitment that requires more frequent configuration. This isn’t all bad — it leads to a better understanding of your computer — but it’s extra time that has to come from somewhere. In graduate school, where constant deadlines and pressure from advisors are ubiquitous, time is a rare commodity. Thinking back to the XKCD comic from earlier, I would estimate that required time investment in Linux “pays off” on the order of years or decades, not weeks or months.

While it was exacerbated by my decision to impose a free software-only constraint, there are a few packages important in graduate school which are missing or offer limited functionality on Linux. Most notably: Microsoft Office. I tried to love LibreOffice, but it fails to excite for documents with tracked changes, equations, figures, and references. It was unusable for the manuscripts that I needed to edit. Instead, I used Office online, which is flawed in its own ways. Near the end of this month, I tinkered with using Wine to install Microsoft Office, but struggled to achieve full feature parity with macOS and Windows.

The Ugly

Coming from a passively-cooled MacBook Air, the most shocking adjustment was the battery life. Apple controls the entire device manufacturing process, and it shows. I frequently get 12+ hours of battery life on my MacBook, and charging is never on my mind. Intel and AMD machines are yet to compete with this level of efficiency. Beyond the hardware differences, using Linux on my MacBook results in about 1/2 of the battery life compared to macOS. When I was performing a typical research task on my XPS with an IDE, browser, and reference manager, my laptop ran hot and noisy. The battery life suffered: I struggled to squeeze 3-4 hours of battery life out of my machine, and a charger became my frequent travel companion. This isn’t entirely awful, it’s a compromise that can be made, but it’s a rough adjustment when considering the other options out there for operating systems.

While my experience was generally smooth, I noticed significant degradation in performance while video conferencing. Most worrying, my computer crashed during an important virtual research meeting, and I noticed it extremely sluggish during other video conferences. Additionally, the Bluetooth compatibility of both my headphone options (AirPods and Bose QC) was limited, with microphones significantly distorted compared to macOS. It’s likely that this could be fixed with more configuration, but initial debugging was not promising.

Conclusions

Spending a month using Linux as my daily driver reminded me why I love computing, and I felt less friction with my computer. Instead, the friction came externally from graduate school pressure. Competitive academic environments fall somewhere between neutral to openly hostile towards Linux and its peculiarities. It’s difficult to justify nuanced software choices and Linux-induced delays to collaborators who don’t share your philosophy of free software. I’m currently writing this on my MacBook. When I’m out of grad school and under less pressure, I’d love to try again.

Creating A Citation Web of my Publications

Thu, 19 Jun 2025 00:00:00 +0000

As a graduate student looking towards a career in academia, the pressure to publish highly-cited papers is omnipresent. Performing good science helps; but studying citation patterns seems to be an essential component to a highly-cited career. To be able to estimate the impact of a paper before having written a word seems an essential skill to build. I’m hardly the first to have this idea; the entire fields of bibliometrics and scientometrics explore this notion.

The last few months, I’ve been having a lot of fun playing around with OpenAlex, an awesome open-source catalog for scholarly work. As a very rudimentary attempt at this field of bibliometrics, I used OpenAlex’s API, through pyalex, to query the incoming and outgoing citations of my published work, as well as the authors associated with all of these works. I saved this information in a series of Markdown files (my favorite format!) for use in my local markdown system, using vscodium+foam. A minimal version of the code to create these files is available here. I then used quartz, an amazing static-site web with a focus on hypertexts, to publish these notes.

I set up the structure of this graph so that every author has their own page listing all works that I’ve cited or their works which have cited mine. For example, here’s mine:

Each work in this list shows the authors associated with each publication, and the citations created in that paper:

All of this is accessible through search and backlinks, but to emphasize the connection of this network, I use Quartz’s graph view. It gets complicated fast!

Check out the web here: kat’s citation web!

Using Apptainer (Docker) on HPCs

Mon, 27 Jan 2025 00:00:00 +0000

When rendering visualizations of large systems in our group, we often use Ovito, running on an Intel Mac in our office space. This seems a little strange, given that we have large amounts of high-performance computing resources at our disposal. Here’s a quick demo of how we can set up Ovito to run on a remote system. More generally, this is a demo of how to install any tricky-to-compile software, whether it be due to permission errors, missing dependencies, or out-of-date distributions.

If you just want to get Ovito running, skip to “Using an Ovito image”.

Building a New Image

First, we pull and build a sandboxed Apptainer image of a distribution that makes compilation easy. Here, I use Debian 12. In our remote system with Apptainer, we run the following to pull the latest Debian image from Docker.

apptainer build --fakeroot --sandbox ovito-test/ docker://debian

After this builds, we can connect to the image using fakeroot:

apptainer shell --fakeroot --writable ovito-test/

Once inside, we can start to compile Ovito (or another software) in this compatibility layer where we have more flexibility. For Ovito, copy the .tar.xz archive from the Ovito website to the clusters, and then extract using tar. In this case, we’ll also need to install xz-utils to extract the Ovito image.

apt install xz-utils
tar xJfv ovito-basic-3.11.3-x86_64.tar.xz

We will also have to install a few XCB system libraries (below is an accumulation from Ovito’s documentation and my own testing, all are probably not necessary). I have found that compiling in ThinLinc avoids some of these issues.

apt install libxcb1 libx11-xcb1 libxcb-glx0 libxcb-icccm4 libxcb-image0 libxcb-keysyms1 \
 libxcb-randr0 libxcb-render-util0 libxcb-render0 libxcb-shape0 libxcb-shm0 \
 libxcb-sync1 libxcb-xfixes0 libxcb-xinerama0 libxcb-xinput0 libxcb-xkb1 libxcb-cursor0 \
 libfontconfig1 libfreetype6 libopengl0 libglx0 libx11-6
apt-get install libegl1-mesa libegl1 libglib2.0-0

Ovito is now compiled, with all of its dependencies! Congrats.

Using an Ovito Image

I’ve created a working version of this Ovito implementation and left it in /depot/prism/data/knykiel/ovito-test. The following commands will get you up and running quickly if you don’t care to set up your own Apptainer image.

Login with ThinLinc at https://desktop.gilbreth.rcac.purdue.edu:300/ to obtain a persistent remote graphical desktop session, and open up a terminal.
Submit an interactive job - it’ll be faster, and you won’t waste front-end resources:

sinteractive -A strachan-k -N1 -n32 --gpus-per-node=1 --time 04:00:00

Start a shell in the Ovito image on depot:

cd /depot/prism/data/knykiel
apptainer shell --fakeroot --writable /depot/prism/data/knykiel/ovito-test/

Run Ovito:

./ovito-basic-3.11.3-x86_64/bin/ovito

Strachan Lab atomate2 Setup

Mon, 05 Feb 2024 00:00:00 +0000

This page serves as a guiding resource for atomate2 in the Strachan Lab. As of now, it is intended to be a companion to the installation guide at materials project. If there exists demand for a more comprehensive guide, that can be done later.

What is atomate2?

According to their README,

Atomate2 is a free, open-source software for performing complex materials science workflows using simple Python functions. Features of atomate2 include

It is built on open-source libraries: pymatgen, custodian, jobflow, and FireWorks.

A library of “standard” workflows to compute a wide variety of desired materials properties.

The ability scale from a single material, to 100 materials, or 100,000 materials.

Easy routes to modifying and chaining workflows together.

It can build large databases of output properties that you can query, analyze, and share in a systematic way.

It automatically keeps meticulous records of jobs, their directories, runtime parameters, and more.

After using and refining my implementation of atomate2 for over a year, it has allowed me to escape the cycle of hacky python scripts and messy file structures. Instead, atomate2 lets me use a robust, scalable set of workflows that are consistent with the state of the art in our field.

Installing atomate2

The atomate2 installation guide is fairly robust; go follow those instructions. Notes on each step are available below.

VASP

If you don’t already have access, ask Kat (or another group member) about getting access to VASP in our HPC environment.

A VASP 5.4.4 executable is available at /depot/prism/apps/vasp_std

MongoDB

If you’re not familiar with MongoDB already, it is a database that stores information as documents. In atomate2, a document often corresponds with a VASP run, an intermediate script, or some post-processing analysis. It’s worth getting some basic familiarity with the MongoDB syntax, as it is the main way to query documents in the database. The idea here is that instead of keeping track of runs through messy file structures in temporary scratch directories across different HPC systems, you can just query for the run you want based on the run details (system of interest, calculation type, user-supplied tags).

While you could self-host your own MongoDB, it’s a lot easier to use the infrastructure that already exists in our group. Kat has deployed a MongoDB docker image on Geddes, and created services to make this IP accessible to outside services. Details of the database are provided below, but for direct access to the database you should speak to Kat directly about getting added as a user.

Conda

Unfortunately, the atomate2 library is a mess of dependencies, and installation can be a bit tricky. Follow the atomate2 guide as far as you can, but expect additional issues to arise with dependencies. You will likely need to modify some of the codebase; so again, ask Kat as needed :)

There is a conda env available at /depot/prism/data/knykiel/autoplex with atomate2 and its utilities installed, as of January 2025

Config files

A collection of some of the more specific atomate2 config files I use for my instance on Negishi are copied below.

jobflow.yaml

JOB_STORE:
 docs_store:
 type: MongoStore
 database: yourdbname
 host: ask-kat
 port: 27017
 username: username
 password: password
 collection_name: outputs
 additional_stores:
 data:
 type: GridFSStore
 database: yourdbname
 host: ask-kat
 port: 27017
 username: username
 password: password
 collection_name: outputs_blobs

my_launchpad.yaml

host: ask-kat
port: 27017
name: yourdbname
username: username
password: password
ssl_ca_file: null
logdir: null
Istrm_lvl: DEBUG
user_indices: []
wf_user_indices: []
authsource: admin
retry: true

my_qadapter.yaml (this is for negishi - other cluster will be different)

_fw_name: CommonAdapter
_fw_q_type: SLURM
rocket_launch: rlaunch -c /home/knykiel/atomate2/config singleshot
nodes: 1
ntasks: 72
walltime: 04:00:00
account: standby
job_name: null
logdir: /home/knykiel/atomate2/logs/
launch_dir: /scratch/negishi/knykiel/launches/
pre_rocket: | 
 set echo

 module --force purge
 module load rcac
 module load intel
 module load openmpi
 module load impi
 module load intel-mkl
 module load anaconda
 module list

 export PATH="$HOME/bin:$PATH"

post_rocket: null

There are some scripts floating around Kat’s system that are used for mass submitting, restarting, and modifying workflows. These can be combined into a GitHub repo upon request.

General atomate2 tips

reading the fireworks documentation will greatly help you understand the system of fireworks, workflows, rockets, launches, etc.
atomate2 has many standard workflows, and you can often extend them with pymatgen (i.e. SQSTransformation)
adding a metadata tag to each FW with fw.spec.update can help with querying later
you can apply INCAR updates with powerups
launching with qlaunch -r rapidfire -m N is a convenient system to launch N jobs at once, and queue new jobs as they finish. This can be done with SLURM to run for weeks.

TODOs

Make a GitHub repository of the atomate2 setup for cloning without the secrets.
Figure out environment variables and PMG_POTCAR_PATH.
Clarify CPU vs GPU compiled versions of VASP.
Run everything inside of an Apptainer?

Atom IDE Installation Guide

Fri, 29 Apr 2022 00:00:00 +0000

Atom has been officially sunset and is no longer maintained. I do not recommend following this guide for new setups; prefer modern, actively maintained editors such as VS Code. This post is retained for archival purposes (it never migrated from my old site); I originally wrote it to help move my research group off MobaXterm and PuTTY. They now (mostly) use VS Code.

What is Atom?

Atom is a hackable text editor created by GitHub

Why Atom?

A dedicated code editor offers many advantages over pure command-line editors (vim, emacs) or SSH clients (MobaXterm, PuTTY)

modern editing tools are magic
- code completion, customizable shortcuts, etc.
- if you’re really attached to vim/emacs, you can still use those within Atom as well!
highly customizable, well documented
ability to easily run/debug code and manage files within your editor

Why specifically Atom, over other alternatives like VS Code or Sublime?

completely free
thousands of open-source packages
convenient Git integration for version control

Even if you prefer one of these options over Atom, it’s a major improvement over slow, rigid tools like MobaXterm and its default text editor.

I’ve only learned about many of these topics in the past few months, and I’m by no means an expert. I just wanted to make a tutorial to document a system for remote access and editing in the clusters. If I find out that something implemented here isn’t best practices, I’ll change it. If I find there are vulnerabilities, I’ll update it and notify anyone who’s used this tutorial.

Installation

Inital setup

Download and run the installer from https://atom.io/

This tutorial is done on Windows using Atom 1.60.0. OS X/Linux or different Atom versions may differ slightly

One of the first things to do after installing Atom would be set up your workspace.

Atom uses projects to manage different workspaces. For example, you might have a project for Bell, another for nanoHUB, and another for local runs.

After opening Atom, create a new project by clicking Open a Project on the right pane
Choose (or create) a folder where you want your workspace to be stored
For this example, we’ll create a new folder called ‘Gilbreth’ we’ll use to connect to the Gilbreth cluster

Creating a git repository

If you’ve never used Git before, it’s a tool used for version control. Instead of only saving the most recent version of each file, Git can track its history.

Why go through the hassle of Git? It’s a fail safe - if your code breaks something, or files become corrupted, Git can save the day

While this tutorial will focus on individual uses of Git, it’s primary appeal is as a coding collaboration tool. To learn more about why Git is useful, see the explanation provided here. In short, here’s a few relevant terms useful to know

repository: The location where all of your files and changes are stored. Rather than just a folder, think of it as a history of different states of your folder
stage: After editing a file, you can stage the changes you made to indicate that you want them tracked. There are some changes which you don’t want to track (passwords, API keys, large files)
commit: To commit adds all of the staged changes to your repository, saving this state
push: To tell your repository that this commit/series of commits is the most updated version of your code
pull: To pull is to take the latest pushed commit and update your files to reflect these changes. While it doesn’t have a huge meaning here, when working on teams it’s essential for sharing code.

Setting up a Git Repository

In the last step, we created a new project called ‘Gilbreth,’ now we’ll initialize a Git repository in this directory. Press CTRL-SHIFT-8 to open the GitHub panel.

Click ‘Login’ and continue
Copy the OAuth token to Atom to login
Click ‘Initialize and publish on GitHub’
Enter a repository name and set visibility (most likely private)

In addition, we want to set up the Git panel, which we open using CTRL-SHIFT-9.

Enter your name and email address
If you intend to use this login for all repositories, click ‘Use for all repositories’

You should now see a panel with three sections: unstaged changes, staged changes, and commit message. This will keep track of the changes you make and allow you to push them to Git.

Any changes you don’t with to track on Git can be untracked by creating a file named ‘.gitgnore’ and adding these file names (i.e. test.py or outVASP)

Create the .gitignore file by selecting the ‘Gilbreth’ folder in the left pane and pressing a, then type ‘.gitignore’ and add whichever files names you don’t want to track

We can now make an initial commit

In the Git panel, press ‘Stage All’ to stage your changes (creating the .gitignore file)
Add a Commit message in the text box below, describing the changes this commit contains
Click ‘Create detached commit’
Click ‘Publish’ to set up a remote tracking branch

Congrats! You should be able to view your repository on GitHub, and have the system in place to stage, commit, and push your changes from within Atom.

Atom has some pretty cool integrations with Git. See which changes were made between commits, merge branch conflicts, easily switch between branches, etc.

Remote FTP

Atom describes itself as a ‘hackable’ text editor because it has thousands of community-made packages which greatly extend the utility of Atom. We’re now going to install some useful packages

Remote FTP is a package that transfers files between local and remote servers. This is useful when connecting to RCAC clusters or nanoHUB. In this exmaple, we’ll demonstrate how to connect to the Gilbreth cluster.

Press CTRL + , to open the Settings menu. From here, click ‘Install.’ This lets you query and install packages for Atom.

Locate and install the ‘remote-ftp’ package (by icetee). After it’s done installing, you can select ‘Packages → Remote FTP → Toggle’ from the menu bar. This will open a new tab in your left pane.

The left tab labeled ‘project’ is your local project folder - the right ‘remote’ tab is your remote file manager. Remote FTP will sync any remote changes locally

To connect to the Gilbreth cluster, we need to create an .ftpconfig file. This is done by selecting ‘Packages → Remote FTP → Create SFTP config file’

In this file, there are a few settings we need to change.

host: ‘gilbreth.rcac.purdue.edu’
user: ‘[[YOUR USERNAME]]’
pass: ‘[[YOUR PASSOWRD]]’ (we’ll set up SSH keys later)
remote: ‘home/[[YOUR USERNAME]]’

Add ‘.ftpconfig’ to your .gitignore file! You should never commit your .ftpconfig file, it contains your password

TODO: write a more complete SSH key setup guide (Atom is a lil janky to set up)

With this setup, you’ll still have to use 2FA each time you connect. Following the guide here, you can connect without 2FA

Platformio IDE Terminal

In addition to managing and editing files, it’s often useful to have a command line interface directly connected to the clusters. For this, we’ll use a package called platformio-ide-terminal

Similarly to Remote FTP, search for this package in settings and install

Ensure that you are installing the correct terminal package, not the other platformio-ide tools (though those can be useful as well)

To automatically connect to a cluster by default, open Settings (CTRL + ,), then select ‘Packages → platformio-ide-terminal → Settings’. Within these settings, set ‘Auto Run Command’ to your ssh connection line. For example,

ssh [[YOUR USERNAME]]@gilbreth.rcac.purdue.edu

If you set up SSH keys in the previous step, those should let you connect automatically without the need of a password or 2FA

If you want to files in the cluster directly from the Atom editor, change your SSH command to use port-forwarding, as below

ssh -L[[1234]]:localhost:[[1234]] [[YOUR USERNAME]]@gilbreth.rcac.purdue.edu

Where [[1234]] is some arbitrary 4-digit number you’ll use as a port

Within these same settings, you can assign some shortcuts to phrases you type a lot (squeue …, cd /scratch…, etc.). You can assign these shortcuts to hotkeys using the ‘keybindings’ section under settings

Hydrogen

Hydrogen is an Atom package which lets you run Jupyter-like Python files within your Atom editor. This gives you the benefits of instantaneous code feedback, without being restricted by clunky Jupyter Notebooks.

Installation is similar to the previous steps - search for ‘Hydrogen’ (by nteract) and install.

Running local Python files

To run Hydrogen kernels locally, you’ll need to install Python. After installing Python, install the ipykernel package locally

pip install ipykernel
python -m ipykernel install

Let’s start by testing out Hydrogen using a simple python file. In your Gilbreth directory, create a new file called hydrogen.py

If you edit a file in your remote directory, you’ll be editing a local copy that updates the remote copy when you save. Therefore, when running local Hydrogen kernels, you won’t have access to the cluster’s environment or file system.

Hydrogen uses Python (.py) files to simulate Jupyter Notebooks. Similarly, it has markdown and code cells, which can be separated in the following way

# %% markdown
This is the content of the Markdown cell. $$H|\psi\rangle = E|\psi\rangle$$
# %% codecell
fish = 2+2
print("Wow Kat, thank you for writing this tutorial!")

Try copying the above cells into your hydrogen.py file, and typing ALT-SHIFT-ENTER. This should run the cell your cursor is in, then move down to the next cell

This system takes a little getting used to, but it’s essentially equivalent to cells in Jupyter notebooks

Running remote Python files

Running files locally is cool, but it’s often desired to run Python files directly in the cluster.

Connect to a cluster using ssh port forwarding (this should be done from the previous step)

ssh -L[[1234]]:localhost:[[1234]] [[YOUR USERNAME]]@gilbreth.rcac.purdue.edu

Setup config file in the cluster (you can do this using the platformio-ide-terminal)

pip install jupyter
jupyter notebook --generate-config

By default, the RCAC clusters use Python v2.7.5. Yuck, but it’s as simple as typing module load anaconda to use a newer version. If you add ‘module load anaconda’ to a file named ‘.bashrc’ in your home directory, it will automatically load anaconda every time you connect

In config file (~/.jupyter/jupyter_notebook_config.py), uncomment

 #c.NotebookApp.token = ''

and set the string to some arbitrary phrase (i.e. ‘katnyp’)

Run a Jupyter server using

jupyter notebook --port=[[1234]] --no-browser

You should now see a Jupyter Notebook server running in the terminal!

If you’re using multiple terminals, this will only work on the first connection you make. This shell will now be ‘unusable’, but you can just open a new terminal and use that

If you just want to use your local Jupyter in-browser in the cluster environment, copy the output of the previous command into a browser

# it should look something like this
http://127.0.0.1:[[1234]]/?token=...

This is Jupyter notebook server connected to the RCAC clusters. Cool! But we want more, we want it within Atom

From Hydrogen package settings, set ‘Kernel Gateways’ to the following

[{
 "name": "Gilbreth",
 "options": {
 "baseUrl": "http://localhost:[[1234]]",
 "token": "[[your chosen token]]"
 }
}]

To connect, we there’s a few steps we need to take.

Instead of using the Menu bar, we’ll introduce the command palette. It’s a convenient way to access commands within Atom without using the mouse or scrolling through menus.

To connect to a remote kernel, open the command palette using CTRL-SHIFT-P

type ’re k’ to select ‘Hydrogen: Connect to Remote Kernel’
select Gilbreth
choose ‘Authenticate with a token’
type in your chosen token
choose [new session] (or connect to an existing one)
choose your desired Python kernel

Congrats, you’re now connected to the clusters! As a check, try adding the following code cell to your Python file

# %% codecell
import os
os.getcwd()

You should see that you’re in the cluster!

Other useful packages

That’s the gist of the tutorial, I hope you found it useful! Feel free to reach out if you have any questions, email or slack works best.

Atom has tons of open-source packages. This is great, but it also means that you need to be careful with installing random packages. I tend to only install packages with large user bases and active communities to check for vulnerabilities.

Cool packages that didn’t warrant their own pages:

atom-beautify: adjusts your code to fit proper code formatting standards
highlight-selected: highlights the current word selected (everywhere it appears) when double clicked
jumpy: navigate around your code with dynamic hotkeys
minimap: preview your full code in your scroll bar
pp-markdown: live preview Markdown files (useful for writing this tutorial!)
split-diff: compare differences between two files
todo-show: finds all TODOs, FIXMEs, etc. in your project and compiles them into a list
zentabs: sets a maximum on the number of opened tabs