Co-author here! I'm kind of surprised that this made it to the top of HN! …

Do you think it would ever be possible to “maximize” a neuron with certain sentences? What’s so different with the gradient ascent techniques with convolutions?

Convolution is part of the network design though. Would a fully connected network learn to convolute? Or would it turn out that convolution is not necessary?

Could a "type system" for neural weights be developed? Given a self-driving system, to be able to statically check that the neurons have the "Person" type, the "Don't Run Over Person" type, and so forth. What happens if you "transplant" the weights for ' an' to another network, some kind of transfer learning but componentized, does it still predict as accurately? If neural networks could be assembled from "types" it would be much easier to trust them.

I wonder if you could comment on this (related to question of how far ahead these "LLM"s are planning). This is Wharton professor Ethan Mollick playing with the new Bing chat, which seems considerably more advanced than ChatGPT (based on GPT-4 perhaps?). Here he asks it to write something using Kurt Vonnegut's rules of writing. https://twitter.com/emollick/status/1626084142239649792 It seems hard to explain how Bing/GPT could have generated the Vonnegut-inspired cake story, having ingested the rules, without planning the whole thing before generating the first word. It seems there's an awful lot more going on internally in these models than a mere word by word autoregressive generation. It seems the prompt (in this case including Vonnegut's rules) is ingested and creates a complex internal state that is then responsible for the coherency and content of the output. The fact that it nece…

you might find this paper interesting: https://arxiv.org/abs/2202.05262 Locating and Editing Factual Associations in GPT