I agree that LLM are made to be more exploratory, this is good as it allows them to experiment with more different topic, as opposed to always saying the same. However, I do not agree it is a feature for code generation, as you would need it to follow strict ruleset (code syntax, specification, tests). Whatever errors it generates and people accept are little mistakes in the threshold of acceptance for the person and a tradeoff for the cost of fixing the problem. In some contexts we see people focusing almost only on short term which leads to a lot of errors being allowed.
Moreover, you cannot say compilers are deterministic. There are situations where they are not (at least for the user).
In order to keep the code as small as possible I was compiling the code with -Os. Everything was working fine until I started to remove some printfs and started to get some crashes. Moving function calls around also seemed to randomly fix the problem, this was an indication that somehow memory/stack corruption was happening. After a lot of testing, I figured out that if -O2/-O3/-Os were used then the problem would appear. The issue was caused by Interprocedural analysis or IPA. One of its functions is to determine whether registers are polluted across function calls and if not then re-use them.
I’m not following. Which part of this is nondeterministic?
The language being complicated to write and the compiler being confusing to use isn’t an indicator of determinism. If GCC were truly nondeterministic, that’d be a pretty major bug.
Also, note that I mentioned that the output behavior is deterministic. I’m not referring to reproducible builds, just that it always produces code that does what the source specifies (in this case according to a spec).
Determinism means performing the same way every time it is run with the same inputs. It doesn’t mean it follows your mental model of how it should run. The article you cite talks about aggressive compiler optimizing causing unexpected crashes. Unexpected, not unpredictable. The author found the root cause and addressed it. Nothing there was nondeterministic. It was just not what the developer expected, or personally thought was an appropriate implementation, but it performed the same way every time. I think you keyed on the word “randomly” and missed “seemed to,” which completely changes the meaning of the sentence.
LLMs often act truly nondeterministically. You can create a fresh session and feed it exactly the same prompt and it will produce a different output. This unpredictability is poison for producing a quality product that is maintainable with dynamic LLM code generation in the pipeline.
I have talked with the author to confirm what he meant with this and other posts he made on compilation. He has confirmed that most (if not all) C compilers are not deterministic. He has pointed me to here as an example. He added that optimizations are not applied in deterministic order and when you add LTO it worsens the problem.
I agree that LLM are made to be more exploratory, this is good as it allows them to experiment with more different topic, as opposed to always saying the same. However, I do not agree it is a feature for code generation, as you would need it to follow strict ruleset (code syntax, specification, tests). Whatever errors it generates and people accept are little mistakes in the threshold of acceptance for the person and a tradeoff for the cost of fixing the problem. In some contexts we see people focusing almost only on short term which leads to a lot of errors being allowed.
Moreover, you cannot say compilers are deterministic. There are situations where they are not (at least for the user).
https://krystalgamer.github.io/high-level-game-patches/
I’m not following. Which part of this is nondeterministic?
The language being complicated to write and the compiler being confusing to use isn’t an indicator of determinism. If GCC were truly nondeterministic, that’d be a pretty major bug.
Also, note that I mentioned that the output behavior is deterministic. I’m not referring to reproducible builds, just that it always produces code that does what the source specifies (in this case according to a spec).
Determinism means performing the same way every time it is run with the same inputs. It doesn’t mean it follows your mental model of how it should run. The article you cite talks about aggressive compiler optimizing causing unexpected crashes. Unexpected, not unpredictable. The author found the root cause and addressed it. Nothing there was nondeterministic. It was just not what the developer expected, or personally thought was an appropriate implementation, but it performed the same way every time. I think you keyed on the word “randomly” and missed “seemed to,” which completely changes the meaning of the sentence.
LLMs often act truly nondeterministically. You can create a fresh session and feed it exactly the same prompt and it will produce a different output. This unpredictability is poison for producing a quality product that is maintainable with dynamic LLM code generation in the pipeline.
I have talked with the author to confirm what he meant with this and other posts he made on compilation. He has confirmed that most (if not all) C compilers are not deterministic. He has pointed me to here as an example. He added that optimizations are not applied in deterministic order and when you add LTO it worsens the problem.