如何让模型流式返回

本文介绍 langchain 的特性 - 流式返回

LLMs 运行时，会不断把已经输出的内容追加到下次输入，直至遇到停止符号。所以对于输出很长的内容，用户需要等待很久，langchain 的大部分组件支持流式返回结果，相当于 python 中 print 设置了 flush=False

import time
for i in range(20):
    print('>',end='',flush=True)
    time.sleep(0.2)

from langchain_ollama import ChatOllama
# 初始化Ollama LLM，注意需要后台开启ollama服务
model_name = "qwen2.5:7b"
llm  = ChatOllama(model=model_name)

# 非流式输出
query = "what color is the sky?"
response=llm.invoke(query)
print(response.content)

The color of the sky can vary depending on the time of day, weather conditions, and other factors. During clear sunny days, the sky is usually blue due to a phenomenon called Rayleigh scattering, where shorter wavelengths (like blue) scatter more than longer wavelengths (like red). However, at sunrise or sunset, the sky often takes on a variety of colors such as pink, orange, and purple because the light has to travel through more of the Earth's atmosphere. On overcast days, the sky can appear gray or white due to clouds scattering the sunlight evenly in all directions.

for chunk in llm.stream("what color is the sky?"):
    print(chunk.content, end="|", flush=True)

The| color| of| the| sky| can| vary| depending| on| the| time| of| day|,| weather| conditions|,| and| other| factors|.| Generally|,| during| clear| days| with| no| clouds|,| the| sky| appears| blue| due| to| a| phenomenon| called| Ray|leigh| scattering|,| where| shorter| (|blue|)| wavelengths| of| light| are| scattered| more| by| the| gases| and| particles| in| the| Earth|'s| atmosphere|.| However|,| the| sky| can| take| on| different| colors| at| sunrise| or| sunset|,| appearing| orange|,| pink|,| red|,| or| purple| because| the| light| has| to| travel| through| more| atmosphere|,| which| sc|atters| out| the| blue| and| green| wavelengths|,| leaving| mostly| the| red|s| and| oranges|.| During| cloudy| days|,| the| sky| might| appear| white| or| grey| due| to| the| clouds| reflecting| sunlight| in| all| directions|.||