So I was reading this article about Signal-creator Moxie Marlinspike’s new project, Confer , which claims to be a verifiably E2E encrypted LLM chat service. There are a couple of short blog articles that give the gist of it, and some github repos including this one that includes scripts for producing the VM that will run your particular LLM session. But if I’m following this all correctly, it implies that every chat session (or perhaps every logged-in user) would have their own VM running their own LLM to ensure that the chain of trust is complete. This seems impossible from a scalability perspective, as even small LLMs require huge quantities of RAM and compute. Did I miss something fundamental here?
Possibly oversimplifying and didn’t have a proper read yet: If you trust the hardware and supply-chain security of Intel but not the operational security of Cloudflare or AWS, this would allow you to exchange messages with the LLM without TLS-encryption-stripping infrastructure operators being able to read the messages in cleartext.
This is a form of Confidential Computing based on Trusted Execution Environments. IMO the real compelling use of TEEs is Verifiable Computing. If you have three servers all with chips and TEEs from different vendors, you can run the same execution on all of them and compare results, which should always agree. You will be safe from the compromise of any single one of them. For Confidential Computing, any single one being compromised means the communication is compromised. The random nature of LLM applications makes Verifiable Computing non-trivial and I’m not sure what the state-of-art is there.
And yes it does look like it has overhead.
Well isn’t it the other way around? If the per-user resources are high, the additional sublinear overhead of isolating gets relatively smaller. It costs more to run 1000 VMs with 32MB RAM each vs 2 VMs with 16GB RAM each.
However I guess this might get in the way of batching and sharing resources between users? Is this mentioned?
Running inference is only pseudorandom, the output is then treated as a distribution and a pseudorandom selection is made according to the distribution. The heavy compute parts are all deterministic, the bit at the end that adds chatbot flavor is only pseudorandom.
As long as they share entropy sources, given the same seed they will always produce the same output. This is a trick that’s exploited in image generation applications, using cached execution keyed on the seed means that alterations only need to be calculated at the part of the pipeline that needs to be changed (saving all of the previous steps).
LLMs don’t have to be random AFAIK: if you turn down the temperature parameter and send the same seed every time you get the same result
https://dylancastillo.co/posts/seed-temperature-llms.html
for most people this isn’t exactly what you want because “temperature” is sometimes short-handed as “creativity”. it controls how out of left field the result can be
The articles are light on detail but the code’s all there. The approach makes sense if the VMs are not cryptographically signed with the user’s key, but are just signed against another key to verify authenticity. I read it as if each VM was created on the fly for a user and signed with that users’s key, but that seems unlikely after re reading it.