docwire::ai::llama::llama_runner Class Reference
This class is intended to load a Llama model with its correct model path and respective configuration and run inference on the prompt supplied along with the model configuration. More...
#include <llama_runner.h>
Inheritance diagram for docwire::ai::llama::llama_runner:
Public Member Functions | |
| llama_runner (const model_inference_config &config) | |
| std::string | process (const std::string &input) override |
| Synchronously process input and return generated text. More... | |
| std::vector< double > | embed (const std::string &input) override |
| Generate an embedding for the given input. More... | |
| virtual void | unload () override |
| Unload the model and free associated resources. –!Must be thread-safe!– and safe to call concurrently with process()/embed(). | |
 Public Member Functions inherited from docwire::ai::ai_runner | |
| virtual | ~ai_runner ()=default |
| Virtual destructor. More... | |
Additional Inherited Members | |
| Protected Types inherited from docwire::with_pimpl< llama_runner > | |
| using | impl_type = pimpl_impl< llama_runner > |
| Protected Member Functions inherited from docwire::with_pimpl< llama_runner > | |
| impl_type * | create_impl (Args &&... args) |
| with_pimpl (Args &&... args) | |
| with_pimpl (with_pimpl< llama_runner > &&other) noexcept | |
| with_pimpl (std::nullptr_t) | |
| with_pimpl & | operator= (with_pimpl &&other) noexcept |
| impl_type & | impl () |
| const impl_type & | impl () const |
Detailed Description
This class is intended to load a Llama model with its correct model path and respective configuration and run inference on the prompt supplied along with the model configuration.
Definition at line 27 of file llama_runner.h.
Member Function Documentation
◆ embed()
|
overridevirtual |
◆ process()
|
overridevirtual |
Synchronously process input and return generated text.
Must be thread-safe.
Implements docwire::ai::ai_runner.
The documentation for this class was generated from the following file:
