Nowadays, as people prioritize privacy and data security, running large language models on the edge becomes increasingly important. To this end, we present TinyChat, an efficient and lightweight system for LLMs on the edge. It seamlessly supports state-of-the-art LLMs on resource-constrained NVIDIA Jetson Orin, and runs the popular LLaMA-2-7B model from Meta at 30 tokens/ sec on the edge.
Website:
Also check out our AWQ paper for LLM quantization:
TinyChat: An Efficient And Lightweight System For LLMs On The Edge For Country: United States. City: Bellevue, Burlington, Detroit, Honolulu, New Haven
