Nvidia’ new tool: Running genAI models on a PC

Nvidia is launching a new tool called Chat with RTX, designed to work offline on Windows PCs with GeForce RTX 30 Series and 40 Series cards. This tool lets users run an AI-powered chatbot on their computers.

With Chat with RTX, users can personalize a GenAI model similar to OpenAI’s ChatGPT. They can connect it to their documents, files, and notes for easy access. Nvidia explains, "Instead of searching through notes or saved content, users can simply type queries." For instance, one can ask about a recommended restaurant in Las Vegas, and Chat with RTX will find the answer in the local files.

By default, Chat with RTX uses Mistral’s open-source model but also supports other text-based models like Meta’s Llama 2. However, users need to download several files, taking up 50GB to 100GB of storage depending on the selected models.

Currently, the tool works with text, PDF, .doc, .docx, and .xml formats. Users can load files into the model’s dataset by pointing the app at a folder containing supported files. Additionally, Chat with RTX can use the URL of a YouTube playlist to load transcriptions, allowing users to query their contents.

However, there are limitations. The tool doesn't remember context, so it won’t consider previous questions when answering follow-up questions. Also, the relevance of responses can vary based on factors like question phrasing, model performance, and dataset size.

Despite its limitations, Chat with RTX provides a glimpse into the future of AI tools that can run locally. The World Economic Forum predicts a significant increase in affordable devices capable of running GenAI models offline, offering benefits like privacy, lower latency, and cost-effectiveness compared to cloud-hosted models.

Although there are concerns about potential misuse, proponents argue that the benefits outweigh the risks. Only time will tell how these tools will be used in practice.