Models#

The "Models" screen, divided into "Models" and "Model Instances," provides the ability to connect an additional model to the Sherpa AI Server, provided that the necessary hardware resources are available.

Users can choose between a local model or a cloud model, provided they have the necessary tokens for access. It is also possible to integrate a model hosted on third-party servers within the local network, but not on the same server where the Sherpa AI Server is located.

Our system implements an algorithm that checks the availability of models and responds with the first available one. However, if it is necessary to use a specific model, we can simply create an instance of that model in the appropriate group, and it will work with it continuously.

It is worth noting that due to the grouping functionality, we are actually grouping not the models themselves, but their backends, which ensures the scalability and fault tolerance of the system. Within a group, it is permissible to host multiple backends with different models.

The "Models" section allows creating and editing model groups.

The "Model Instances" section allows creating various links to models. For each model, separate connections can be configured, each with its own individual parameters and settings.

Selecting a Model on Other Screens of Sherpa AI Server#

Chat#

The "Models" section allows viewing and editing model groups that the User can later select when creating a new dialogue in the Sherpa AI Server.

Assistants#

The "Models" section allows viewing and editing model groups that the User can later select when creating an Assistant.

Create Model#

To create a new model group, you need to click the "Create" button in the "Models" table and fill out the opened form.

Create Instance#

To create a new model instance, you need to click the "Create" button in the "Model Instances" table and fill out the opened form.

Fields to fill in the "Create Instance" popup:

• "Name *" – a mandatory field for the name of the model instance;
• "Description" – a field for entering a description of the model instance;
• "Host *" – a mandatory field for specifying the IP address or domain name of the server where the model instance is hosted;
• "Port *" – a mandatory field for specifying the service port number of the model instance;
• "Protocol *" – a mandatory field for selecting the connection protocol;
• "Timeout *" – a mandatory field for setting the maximum waiting time for a response from the model instance (in seconds).

There is also the option to create connections to cloud models (selecting the "Cloud" checkbox in the "Create Instance" window). Special settings are provided for this, including proxy servers and additional access tokens. In the case of local models, the configuration includes specifying the host, port, and protocol. The protocol remains our standard, similar to the OpenAI format, ensuring consistency and compatibility when working with various models.

Additional fields to fill in the "Create Model" popup (when selecting the "Cloud" checkbox in the "Create Instance" window):

• "Provider *" – a field for selecting the model provider;
• "Provider Model" – a field for specifying a specific model from the selected provider;
• "Token / API Key *" – a field for entering the provider's API key or authorization token;
• "Login" – a field for entering the username (when using basic authentication);
• "Password" – a field for entering the password (when using basic authentication).

This is a special container designed to run a single model. Essentially, it is a universal platform where any model can be launched, but only one at a time. To use multiple models, it is sufficient to launch several such containers, each with its own configuration. In each container, the port used to access the model will need to be changed. This approach allows for efficient use of a single server, running any number of models (the limitation only depends on hardware resources).

Edit Model#

To view and edit the properties of a model group, you need to select the desired group from the list and click the button . After that, a form with model settings will open, where you can make the necessary changes. There are no new fields in the previously created model.

Edit Instance#

To view and edit the properties of a model instance, you need to select the desired instance from the list and click the button . After that, a form with instance settings will open, where you can make the necessary changes. There are no new fields in the previously created model instance.