Neardi LLM/VLM Demos
Contents
1. LLM
1.1 Overview
大语言模型（LLM）是一种强大的AI系统，你可以把它想象成一个学习了海量文本（书籍、文章、网页等）的“超级大脑”。它基于一种叫Transformer的神经网络架构，核心能力是理解语言并生成语言。而Qwen3-1.7B就是其中一个具体、轻量化的实例。
1.2 Prepare
确保Host主机里有rk182x算力卡， 如下：
环境安装okay后， 可以使用��如下命令查看：
neardi@RK3576:~$ sudo rknn-smi info
+------------------------+---------------+---------------+----------------------+
| rknn-smi Version: 1.3.0 |
+========================+===============+===============+======================+
| Device Status | Health | Power(mW) | Npu(%) |
| Chip Name | Bus-Id | Temp(C) | Memory-Usage(MB) |
+========================+===============+===============+======================+
| 0 Online | OK | NA | 0 |
| 0 RK1820 | 0000:06:00.0 | 43 | 1376 / 2560 |
+========================+===============+===============+======================+
1.3 Deploy and run Qwen3-1.7B
从这里下载转换后的Qwen3-1.7B, 拷贝到与rk182x相关的设备， 如下：
neardi@RK3576:~/Qwen$ tree
.
├── Qwen3-1.7B.embed.bin
├── Qwen3-1.7B.rknn
├── Qwen3-1.7B.tokenizer.gguf
└── Qwen3-1.7B.weight
使用如下命令运行Qwen模型， 如下：
neardi@RK3576:~/Qwen$ rkllm3-server -m Qwen3-1.7B.rknn --weight Qwen3-1.7B.weight \
--vocab Qwen3-1.7B.tokenizer.gguf \
--embed Qwen3-1.7B.embed.bin \
--host 0.0.0.0 --port 8080 --repeat-penalty 1.2 --presence-penalty 0.0 \
--frequency-penalty 0.0 --top-k 1 --top-p 0.9 --temp 1.0 -c 0
运行成功后， 有如下log:
neardi@RK3576:~/Qwen$ rkllm3-server -m Qwen3-1.7B.rknn --weight Qwen3-1.7B.weight \
--vocab Qwen3-1.7B.tokenizer.gguf \
--embed Qwen3-1.7B.embed.bin \
--host 0.0.0.0 --port 8080 --repeat-penalty 1.2 --presence-penalty 0.0 \
--frequency-penalty 0.0 --top-k 1 --top-p 0.9 --temp 1.0 -c 0
version: 1.0.4 (04c8fbc@2026-05-13T15:14:35)
main: loading model
srv load_model: loading model 'Qwen3-1.7B.rknn'
====> rknn3_find_devices
device[0]: type=PCIE, serial_id=0000:06:00.0
the device-id is not specified, use "0000:06:00.0" as default!
====> rknn3_init
====> rknn3_load_model_from_path
rknn3_load_model_from_path success!
====> rknn3_query RKNN3_QUERY_CORE_NUMBER
====> rknn3_model_init
rknn3_model_init success!
====> rknn3_query RKNN3_QUERY_LLM_CONFIG
llm_model_type =
rope_cache_host_storage = 0
====> rknn3_query RKNN3_QUERY_CORE_MEM_SIZE
====> rknn3_create_mem
Created user internal memory for core 0: size=8942592, virt_addr=0x7fb0797000
====> rknn3_create_mem
Created user internal memory for core 1: size=6646784, virt_addr=0x7fb0140000
====> rknn3_create_mem
Created user internal memory for core 2: size=6646784, virt_addr=0x7fafae9000
====> rknn3_create_mem
Created user internal memory for core 3: size=6646784, virt_addr=0x7faf492000
====> rknn3_create_mem
Created user internal memory for core 4: size=6646784, virt_addr=0x7faee3b000
====> rknn3_create_mem
Created user internal memory for core 5: size=6646784, virt_addr=0x7fae7e4000
====> rknn3_create_mem
Created user internal memory for core 6: size=6646784, virt_addr=0x7fae18d000
====> rknn3_create_mem
Created user internal memory for core 7: size=6646784, virt_addr=0x7fadb36000
====> rknn3_set_internal_mem
rknn3_set_internal_mem success, set 8 internal memories
load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
load: printing all EOG tokens:
load: - 128247 ('</s>')
load: - 151643 ('<|endoftext|>')
load: - 151645 ('<|im_end|>')
load: - 151662 ('<|fim_pad|>')
load: - 151663 ('<|repo_name|>')
load: - 151664 ('<|file_sep|>')
load: special tokens cache size = 27
load: token to piece cache size = 0.9311 MB
srv init: initializing slots, n_slots = 1
slot init: id 0 | task -1 | new slot n_ctx_slot = 2048
Enable thinking? 1
main: model loaded
接下来可以使用如下命令行进行测试:
neardi@RK3576:~$ curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"messages": [
{"role": "user", "content": "解析一下AI是什么？"}
],
"n_keep": 0,
"cache_prompt": false,
"id_slot": 0,
"n_predict": 512,
"stream": true
}'
��模型有如下回复：
neardi@RK3576:~$ curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"messages": [
{"role": "user", "content": "解析一下AI是什么？"}
],
"n_keep": 0,
"cache_prompt": false,
"id_slot": 0,
"n_predict": 512,
"stream": true
}'
data: {"choices":[{"finish_reason":null,"index":0,"delta":{"role":"assistant","content":null}}],"created":1782971670,"id":"chatcmpl-2K5xjPOEpBVObEeb1ke2qcBTRzRlIlZG","id_slot":0,"model":"Qwen3-1.7B.rknn","system_fingerprint":"b5166-24345353","object":"chat.completion.chunk"}
data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"<think>"}}],"created":1782971670,"id":"chatcmpl-2K5xjPOEpBVObEeb1ke2qcBTRzRlIlZG","id_slot":0,"model":"Qwen3-1.7B.rknn","system_fingerprint":"b5166-24345353","object":"chat.completion.chunk"}
data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"\n"}}],"created":1782971670,"id":"chatcmpl-2K5xjPOEpBVObEeb1ke2qcBTRzRlIlZG","id_slot":0,"model":"Qwen3-1.7B.rknn","system_fingerprint":"b5166-24345353","object":"chat.completion.chunk"}
data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"嗯"}}],"created":1782971670,"id":"chatcmpl-2K5xjPOEpBVObEeb1ke2qcBTRzRlIlZG","id_slot":0,"model":"Qwen3-1.7B.rknn","system_fingerprint":"b5166-24345353","object":"chat.completion.chunk"}
data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"，"}}],"created":1782971670,"id":"chatcmpl-2K5xjPOEpBVObEeb1ke2qcBTRzRlIlZG","id_slot":0,"model":"Qwen3-1.7B.rknn","system_fingerprint":"b5166-24345353","object":"chat.completion.chunk"}
data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"用户"}}],"created":1782971670,"id":"chatcmpl-2K5xjPOEpBVObEeb1ke2qcBTRzRlIlZG","id_slot":0,"model":"Qwen3-1.7B.rknn","system_fingerprint":"b5166-24345353","object":"chat.completion.chunk"}
...
可以看出， 模型回复的是token的形式。 也可以编写一个HTML页面进行访问， 如下：
HTML代码， 可以从这里download.
2. VLM
2.1 Overview
FastVLM 是苹果公司（Apple） 在 2025 年提出并开源的一系列视觉语言模型（Vision Language Model, VLM），相关论文已被 CVPR 2025 接收。它最核心的标签就是 “快”，专为在 iPhone、iPad 和 Mac 等个人设备上高效运行而设计. 这里， 我们把它部署到rk182x设备上面。
2.2 deploy FastVLM(1.6B)的直接链接")
首先从Rockchip官方下载转换后的FastVLM模型, 并且拷贝到与rk182x相关的设备上面。 如下：
neardi@RK3576:~/FastVLM$ tree
.
├── FastVLM_1.6B.embed.bin
├── FastVLM_1.6B.tokenizer.gguf
├── llm_FastVLM_1.6B.rknn
├── llm_FastVLM_1.6B.weight
├── vision_FastVLM_1.6B.rknn
└── vision_FastVLM_1.6B.weight
接下来运行模型， 使用如下命令:
neardi@RK3576:~/FastVLM$ rkllm3-server -m llm_FastVLM_1.6B.rknn --model2 vision_FastVLM_1.6B.rknn --vocab FastVLM_1.6B.tokenizer.gguf --embed FastVLM_1.6B.embed.bin --host 0.0.0.0 --port 8080 --n_predict 512 --repeat-penalty 1.1 --presence-penalty 1.0 --frequency-penalty 1.0 --top-k 1 --top-p 0.8 --temp 0.8 --img-start "<|im_start|>" --img-end "<|im_end|>" --img-content "<|image_pad|>"
运行成功后， 有如下log:
neardi@RK3576:~/FastVLM$ rkllm3-server -m llm_FastVLM_1.6B.rknn --model2 vision_FastVLM_1.6B.rknn --vocab FastVLM_1.6B.tokenizer.gguf --embed FastVLM_1.6B.embed.bin --host 0.0.0.0 --port 8080 --n_predict 512 --repeat-penalty 1.1 --presence-penalty 1.0 --frequency-penalty 1.0 --top-k 1 --top-p 0.8 --temp 0.8 --img-start "<|im_start|>" --img-end "<|im_end|>" --img-content "<|image_pad|>"
version: 1.0.4 (04c8fbc@2026-05-13T15:14:35)
img_start: <|im_start|>
img_end: <|im_end|>
img_content: <|image_pad|>
main: loading model
srv load_model: loading model 'llm_FastVLM_1.6B.rknn'
====> rknn3_find_devices
device[0]: type=PCIE, serial_id=0000:06:00.0
the device-id is not specified, use "0000:06:00.0" as default!
====> rknn3_init
====> rknn3_load_model_from_path
rknn3_load_model_from_path success!
====> rknn3_query RKNN3_QUERY_CORE_NUMBER
====> rknn3_model_init
rknn3_model_init success!
====> rknn3_query RKNN3_QUERY_LLM_CONFIG
The value of '--ctx-size'(4096) is greater than the model's max_context_len(1024), so '--ctx-size' is limited to 1024.
llm_model_type =
rope_cache_host_storage = 0
====> rknn3_query RKNN3_QUERY_CORE_MEM_SIZE
====> rknn3_init 2
====> rknn3_load_model_from_path 2
rknn3_load_model_from_path 2 success!
====> rknn3_query 2 RKNN3_QUERY_CORE_NUMBER
====> rknn3_model_init 2
rknn3_model_init 2 success!
====> rknn3_query 2 RKNN3_QUERY_CORE_MEM_SIZE
====> rknn3_create_mem
Created user internal memory for core 0: size=39849984, virt_addr=0x7f8977e000
====> rknn3_create_mem
Created user internal memory for core 1: size=27267072, virt_addr=0x7f825ff000
====> rknn3_create_mem
Created user internal memory for core 2: size=27267072, virt_addr=0x7f80bfe000
====> rknn3_create_mem
Created user internal memory for core 3: size=7341056, virt_addr=0x7f8907d000
====> rknn3_create_mem
Created user internal memory for core 4: size=13107200, virt_addr=0x7f883fd000
====> rknn3_create_mem
Created user internal memory for core 5: size=6820864, virt_addr=0x7f8057c000
====> rknn3_create_mem
Created user internal memory for core 6: size=8389632, virt_addr=0x7f7b7ff000
====> rknn3_create_mem
Created user internal memory for core 7: size=5745664, virt_addr=0x7f80001000
====> rknn3_set_internal_mem
rknn3_set_internal_mem success, set 8 internal memories
load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
load: printing all EOG tokens:
load: - 128247 ('</s>')
load: - 151643 ('<|endoftext|>')
load: - 151645 ('<|im_end|>')
load: special tokens cache size = 5
load: token to piece cache size = 0.9308 MB
model input num: 1, output num: 1
input tensors:
Tensor: name=pixel, n_dims=4, shape=[1, 512, 512, 3], stride=[786432, 1536, 3, 1], aligned_size=786432, layout=NHWC, dtype=UINT8, core_id=0, qnt_type=NONE
output tensors:
Tensor: name=1982, n_dims=3, shape=[1, 64, 1536], stride=[98304, 1536, 1], aligned_size=196608, layout=UNDEFINED, dtype=FP16, core_id=0, qnt_type=NONE
model is NHWC input layout
input image height=512, input image width=512, input image channel=3
model is UNDEFINED output layout
output img embeds dim0=1, img embeds dim1=64
====> rknn3_create_mem input 0
====> rknn3_create_mem output 0
srv init: initializing slots, n_slots = 1
slot init: id 0 | task -1 | new slot n_ctx_slot = 1024
Enable thinking? 0
main: model loaded
最后， 我们来测试一下FastVLM模型， 打开一个terminal终端， 使用如下命令：
neardi@RK3576:~$ curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"messages": [
{"role": "user", "content": "解析一下AI是什么？"}
],
"n_keep": 0,
"cache_prompt": false,
"id_slot": 0,
"n_predict": 512,
"stream": true
}'
模型回答的log如下：
neardi@RK3576:~$ curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"messages": [
{"role": "user", "content": "解析一下AI是什么？"}
],
"n_keep": 0,
"cache_prompt": false,
"id_slot": 0,
"n_predict": 512,
"stream": true
}'
data: {"choices":[{"finish_reason":null,"index":0,"delta":{"role":"assistant","content":null}}],"created":1782959150,"id":"chatcmpl-XcAL2PVlYckXPWwUhIuO3IbFSLElnSUM","id_slot":0,"model":"llm_FastVLM_1.6B.rknn","system_fingerprint":"b5166-24345353","object":"chat.completion.chunk"}
data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"AI"}}],"created":1782959150,"id":"chatcmpl-XcAL2PVlYckXPWwUhIuO3IbFSLElnSUM","id_slot":0,"model":"llm_FastVLM_1.6B.rknn","system_fingerprint":"b5166-24345353","object":"chat.completion.chunk"}
data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"，"}}],"created":1782959150,"id":"chatcmpl-XcAL2PVlYckXPWwUhIuO3IbFSLElnSUM","id_slot":0,"model":"llm_FastVLM_1.6B.rknn","system_fingerprint":"b5166-24345353","object":"chat.completion.chunk"}
data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"全"}}],"created":1782959150,"id":"chatcmpl-XcAL2PVlYckXPWwUhIuO3IbFSLElnSUM","id_slot":0,"model":"llm_FastVLM_1.6B.rknn","system_fingerprint":"b5166-24345353","object":"chat.completion.chunk"}
...
2.3 HTML client
上面的测试是基于命令行， 查看模型的回答很不友好， 看见的是一个一个词元(token)。 这里创建一个HTML页面来和模型对话， 如下：
HTML代码， 可以从这里download.