Another thing we learned very quickly was that attempting to generate code, then modify the result …

> One of the things we learned very quickly was that having generated source code in the same repository as actual source code was not sustainable. Keeping a repository with the prompts, or other commands separate is fine, but not committing the generated code at all I find questionable at best.

I will guess that you are generating orders of magnitude more lines of code with your software than people do when building projects with LLMs - if this is true I don't think the analogy holds.

I think the biggest difference here is that your code generator is probably deterministic and you likely are able to debug the results it produces rather than treating it like a black box.

There’s a huge difference between deterministic generated code and LLM generated code. The latter will be different every time, sometimes significantly so. Subsequent prompts would almost immediately be useless. “You did X, but we want Y” would just blow up if the next time through the LLM (or the new model you’re trying) doesn’t produce X at all.

> One of the things we learned very quickly was that having generated source code in the same repository as actual source code was not sustainable My rule of the thumb is to have both in same repo, but treat generated code like binary data. This was informed by when I was burned by a tooling regression that broke the generated code and the investigation was complicated by having to correlate commits across different repositories

Please tell us we company you are working for so that we don't send our resumes there. Jokes aside, I have worked in projects where auto-generating code was the solution that was chosen and it's always been 100% auto-generated, essentially at compilation time. Any hand-coded stuff needed to handle corner cases or glue pieces together was kept outside of the code generator.

> The end result of that was that we had to significantly rearchitect the project for us to essentially inject manually crafted code into arbitrary places in the generated code. This sounds like putting assembly in C code. What was the input language? These two bits ("Not AI generated", "a feature flag") suggest that the code generator didn't have a natural language frontend, but rather a real programming language frontend. Did you or anyone else inform management that a code generator is essentially a compiler with extra characters? [0] If yes, then what was their response? I am concerned that your current/past work might have been to build a Compiler-as-a-Service (CaaS). [1] No shade, I'm just concerned that other managers might read all this and then try to build their own CaaS. [0] Yes, I'm implying that LLMs are compilers. Altman has played us for fools; he's taught a billion people…