Rendered at 10:57:08 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
quirino 8 hours ago [-]
I've been studying these a bunch for a project in university. Last week I went over the derivation of the ELBO for a couple hours and it was a very fun and elucidating exercise.
Once you give names to the larger mathematical structures and understand them a bit better it becomes quite simple. I wish some of the blogs/papers I'd read had named "Importance Sampling".
The probability notation can be pretty confusing too. Sometimes it's hard to understand the "types" of some variables. But I'm inexperienced.
ChatGPT was surprisingly helpful. If you put in the work to truly understand the where the gaps are in your mental model (which parts aren't completely intuitive), it can do an amazing job filling in the gaps.
rottc0dd 5 hours ago [-]
A good video from welch labs on image generation with diffusion models:
Something I've wondered, maybe I should just do it if I can find some time, but... given DeepSeek's nice results on using rendered text as input, I'm wondering if anyone has given serious research efforts towards image-based diffusion methods for text.
As in, instead of all the complexities induced by discrete token generation, just generate the image of the text using standard image diffusion methods, then convert it to text.
If you used a single, monospace font, I bet this would be even pretty efficient, because the OCR problem becomes basically just direct template matching.
But I guess probably there is already a paper out there, I haven't searched. I'd be curious to know if it compares on par with token-based methods.
amelius 3 hours ago [-]
I've been wondering why we can't skip the entire token embedding step and just feed the model raw Unicode. I suppose the reason for that is efficiency and it's related to your question. Makes one wonder what other steps we can do more efficiently in code than in neurons.
p1esk 56 minutes ago [-]
There’s has a ton of work on character or byte level encodings for llms. The problem is you expand your input tokens by 3-4x. Expensive.
Also, you still need token embeddings (I think you might be confused how that works).
amelius 5 minutes ago [-]
Could be! I have not (yet) spent much time learning about how llms work, just the occasional blog here and there. My main question is why we need a bit of additional code to massage the input into tokens and why the neural network cannot do it, i.e., let the embedding be a latent space that forms naturally when training the network.
archerx 2 hours ago [-]
I think it would make training take much much longer but the models would be able to tell you how many Rs are in strawberry.
amelius 2 hours ago [-]
Yes. By the way I think it is a pity that all the tutorials start with token embedding; by teaching LLMs like that, you don't learn why they are necessary.
gdiamos 6 hours ago [-]
I’d like to see more of these models.
I’ve been using diffusion Gemma and it is very fast on GPUs in output token/sec.
In the diffusion Gemma whitepaper, they say they could have done better with more time and compute.
Even with those caveats, it is very uses-able as a local model.
electroglyph 8 hours ago [-]
good stuff, no mention of confidence tho, recommend having a look at diffusiongemma and others.
Once you give names to the larger mathematical structures and understand them a bit better it becomes quite simple. I wish some of the blogs/papers I'd read had named "Importance Sampling".
The probability notation can be pretty confusing too. Sometimes it's hard to understand the "types" of some variables. But I'm inexperienced.
ChatGPT was surprisingly helpful. If you put in the work to truly understand the where the gaps are in your mental model (which parts aren't completely intuitive), it can do an amazing job filling in the gaps.
https://www.youtube.com/watch?v=iv-5mZ_9CPY&pp=ygUVZGlmZnVza...
As in, instead of all the complexities induced by discrete token generation, just generate the image of the text using standard image diffusion methods, then convert it to text.
If you used a single, monospace font, I bet this would be even pretty efficient, because the OCR problem becomes basically just direct template matching.
But I guess probably there is already a paper out there, I haven't searched. I'd be curious to know if it compares on par with token-based methods.
Also, you still need token embeddings (I think you might be confused how that works).
I’ve been using diffusion Gemma and it is very fast on GPUs in output token/sec.
In the diffusion Gemma whitepaper, they say they could have done better with more time and compute.
Even with those caveats, it is very uses-able as a local model.