51本色

Sign up using the form to access Code-Along on-demand video.

Code-Along on-demand video includes:

What you’ll build:

• You’ll create a browser-based large language model (LLM) application that runs Llama 3.1 quantized to INT4, with a Streamlit frontend and a torchchat backend, that runs entirely on an Arm-based AWS Graviton CPU.

What you’ll learn:

• To download the Meta Llama 3.1 model from the Meta Hugging Face repository.
• 4-bit quantize the model using optimized INT4 KleidiAI kernels for PyTorch.
• Run an LLM inference using PyTorch on an Arm-based CPU.
• Expose an LLM inference as a browser application with Streamlit as the frontend and torchchat framework in PyTorch as the LLM backend server.
• Measure performance metrics of the LLM inference running on an Arm-based CPU.