> I tried something new for the first time, which was to keep a journal during development
> I tried something new for the first time, which was to keep a journal during development. DO THIS!!! I cannot stress this enough! If you work in a professional science lab, say, physics, biology, chemistry, you are expected to keep an experiment journal. It provides more help to you than the company too (knowledge dump, liability, etc). I can't tell you how many times some stupid ass seemingly benign comment saved my behind. They're worth their weight in gold. For ML experiments I use wandb and hydra[0]. Put all your configs into hydra. Be fucking pedantic. You should log your seeds, versions, the date, and I mean everything. It only takes a few extra minutes to set this up but the one time you need it it'll save you hours. Dump all that into wandb AND your model checkpoints. You will forget what that checkpoint corresponds to. Make liberal use of wandb tags and comments (through hydra you can make these cli arguments to automate even if launching from slurm scripts). Turn on wandb's code saving. Most importantly, use those notebooks wandb gives you. Don't worry if it gets messy. …
Original discussionCompiling a neural net to C for a speedup93 source comments · at import