vllm
Serve large language models efficiently and interact with them through an OpenAI-compatible API. More information: https://docs.vllm.ai/en/latest/getting_started/quickstart.html.
- Launch an OpenAI-compatible API server for a model:
vllm serve
- Serve a model on a specific host and port:
vllm serve --host --port
- Start an interactive chat session:
vllm chat
- Run batch prompts from a file and write results to a file:
vllm run-batch -i -o --model
- Print environment information for bug reports:
vllm collect-env
- List all available subcommands:
vllm --help
来源:tldr-pages contributors · 作者与贡献记录 · CC BY 4.0。离线整理了版式、链接和媒体显示。
