The conversation

Lessons learnedExperience excerpt · English

> I tried something new for the first time, which was to keep a journal during development

> I tried something new for the first time, which was to keep a journal during development. DO THIS!!! I cannot stress this enough! If you work in a professional science lab, say, physics, biology, chemistry, you are expected to keep an experiment journal. It provides more help to you than the company too (knowledge dump, liability, etc). I can't tell you how many times some stupid ass seemingly benign comment saved my behind. They're worth their weight in gold. For ML experiments I use wandb and hydra[0]. Put all your configs into hydra. Be fucking pedantic. You should log your seeds, versions, the date, and I mean everything. It only takes a few extra minutes to set this up but the one time you need it it'll save you hours. Dump all that into wandb AND your model checkpoints. You will forget what that checkpoint corresponds to. Make liberal use of wandb tags and comments (through hydra you can make these cli arguments to automate even if launching from slurm scripts). Turn on wandb's code saving. Most importantly, use those notebooks wandb gives you. Don't worry if it gets messy. …

Original discussionCompiling a neural net to C for a speedup93 source comments · at import
8

Community replies 0

No community replies yet. Add your perspective.

Replies from the source 8

Direct replies available in this import. Original-language excerpts.

0C

From my two minute skim of the docs, not encouraging that hydra only officially supports up to Python 3.11.

Full original
NI

The problem with keeping a journal is that the distraction of doing so may break the state of flow. OTOH there are natural breaks in the process of working; writing things down during these works fine. The fidelity is a bit lower, but it's still much better than nothing.

Full original
IS

Chaotic energy haha, I like it. Thanks for the tips re: keeping a journal, I will do this more in the future. I usually keep development notes, though normally in markdown files scattered across the codebase or in comments, never by date in the README. In the future, I might make JOURNAL.md a standard practice in my projects? re:w&b, I used w&b when it first came out and I liked it but I'm sure it's come a lot further in the time since then. I will have to take a look! Also lol "pretentious perfectionist" I'm glad to finally have some words to describe my design aesthetic. I like crisp fonts, what can I say.

Full original
RV

All of this agreed. Now in the age of AI, many students entering into CS need do this NOW, otherwise any answer they come up in the interview, will be assumed that it was from an AI and they need to show that they something useful came out of their blogpost or research. It is what it now means to know how to experiment, understand and build knowledge, rather than spitting out the answer because it it from stack overflow or ChatGPT. The mistakes are raw, all the learnings in a blog post which is what makes us human yet, 90% of candidates do not do this which is why most of them cannot explain an AI's mistakes an interview if they use it.

Full original
MR

Same goes for something as simple as setting up a server. You will forget, and if you don’t write it down you’ll have to figure it out again.

Full original
HE

My current workflow is to keep a wiki, would you say hydra would replace/complement especially that if you're used to note keeping the wiki way?

Full original
LA

Converged to something similar after spending 2 days bissecting a repo to reproduce a training run, having to wait 3hr on each commit before conclusive results. I couldn't get myself to use hydra though, it felt like a lot of bloat vs loading a yaml with pydantic

Full original
TO

OK so I try to do that. But then I'll have some big problem or add too many big features and just give up. (My sleep is nearly nonexistent so I don't have a lot of time for logging things anyway.)

Full original
Continue on Hacker News

Respect different perspectives. Share concrete experiences.

Sign in to post, reply, repost, and quote.