课堂-本地部署与显存估算Lesson - Local Deployment and VRAM Estimation
课堂教案 · 约 15 分钟Lesson plan · about 15 min · 引用核验 2026-11-07（仅引用链接与日期） · refs checked 2026-11-07 (reference links and dates only)
学员按权重、KV 缓存、运行时开销三项列估算表，再用运行器的占用输出核对，分清能启动、跑得动、答得完三个问题。Learners build an estimate from weights, KV cache and runtime overhead, check it against runner occupancy output, and separate can start, can run and can finish answering.
能把显存需求拆成权重、KV 缓存与运行时开销，并说明每项随什么变化。
能用参数量与量化位宽做权重量级的算术估算，并标明这是估算。
能读懂运行器占用输出，判断是否发生 CPU 与 GPU 混合加载。
能给出装不下时的三条备选路径，并说明各自代价。
Split VRAM demand into weights, KV cache and runtime overhead, and explain what each varies with.
Make an arithmetic weight estimate from parameter count and quantisation bit width, and label it as an estimate.
Read runner occupancy output and judge whether CPU and GPU mixed loading occurs.
Give three fallback paths when the model does not fit and state the cost of each.
llama.cpp 官方 README：1.5 到 8-bit 整数量化，以及 CPU＋GPU 混合推理。
Ollama 官方 FAQ：ollama ps 的 PROCESSOR 列与 num_ctx 上下文设置。
Hugging Face Transformers 官方 KV 缓存文档（仓库源文件）：缓存占用、卸载与量化缓存。
一张估算表：参数量、位宽、上下文长度、批量、可用显存、可用内存。
llama.cpp official README: 1.5 to 8-bit integer quantisation and CPU plus GPU hybrid inference.
Ollama official FAQ: the PROCESSOR column in ollama ps and the num_ctx context setting.
Hugging Face Transformers KV cache documentation in the repository source: cache occupancy, offloading and quantised cache.
An estimate sheet: parameter count, bit width, context length, batch size, available VRAM and available RAM.
- 3 分钟算权重：用参数量乘每参数字节数，写明位宽假设与算式。
- 4 分钟加缓存与开销：标出随上下文长度和批量变化的项，留出余量。
- 4 分钟核对占用：读官方文档里的示例输出，判断加载位置与是否需要卸载。
- 4 分钟给备选：更小规格、更低量化档、部分卸载各写一条代价与下一步实测条件。
- 3-minute weights: multiply parameter count by bytes per parameter and write the bit-width assumption and formula.
- 4-minute cache and overhead: mark items that vary with context length and batch size, and leave headroom.
- 4-minute occupancy check: read example output from official docs and judge loading location and whether offloading is needed.
- 4-minute fallbacks: smaller model, lower quantisation tier and partial offloading each get one cost and one next-step test condition.
以为权重装得下就跑得快；缓存与卸载会改变速度和延迟。
以为量化只省显存；官方文档把它写成内存与速度的取舍，质量要在自己任务上核对。
以为本地部署就等于数据合规；还要看日志、权限与出网路径。
把文档示例里的占用数字当成自己机器的结果；硬件与设置不同。
Assuming that if weights fit the model runs fast; cache and offloading change speed and latency.
Assuming quantisation only saves VRAM; official docs describe a memory-speed tradeoff and quality must be checked on your own task.
Assuming local deployment equals data compliance; logs, permissions and outbound paths still matter.
Treating occupancy numbers in documentation examples as results on your machine; hardware and settings differ.
估算表三项齐全，每项写明假设与来源，算式可复核。
能区分估算与实测，并说出下一步实测要固定哪些条件。
备选路径带代价，不写「最好」，也不承诺具体速度数字。
The estimate sheet has all three items, each with assumptions and sources, and the formula can be checked.
Learners can distinguish estimate from measurement and state which conditions must be fixed for the next measurement.
Fallback paths include costs, do not write best, and do not promise specific speed numbers.
例子与判断Example and interpretation
以下是教学情境，不是本站运行或实测记录。A teaching scenario, not a measured or executed site test.
课堂练习：7B 模型按 4-bit 估算权重量级，再单独列 KV 缓存与运行时开销。若可用显存只够权重，不写「能跑」，而写需核对卸载与延迟。Class exercise: estimate the weight magnitude of a 7B model at 4-bit, then list KV cache and runtime overhead separately. If available VRAM only covers weights, do not write can run; write that offloading and latency must be checked.
做完例子后怎么检查How to check your practice
先用自己的话解释上面的判断，再检查输入条件，最后回原文验证。说不清原因时回概念；缺日期、字段或权限时补输入；结论无出处时先撤回结论。Explain the example in your own words, inspect input constraints, then verify against the source. Revisit concepts if you cannot explain why; supply missing dates, fields or permissions; withdraw unsupported conclusions.
按这篇主题，先从“概念没分清”练习查起For this topic, start with concept confusion · 查看全部六个情境与理由See all six scenarios and their reasoning
显存VRAM 量化Quantization KV 缓存KV Cache GPU 层卸载-ngl / Offload 本地部署Local Deployment