Execute LLM Endpoint

LLM Serving

QuickML now provides Generative AI services by hosting Large Language Models and Vision Language Models (VLM) under Generative AI section in the console. LLM Serving is equipped with Chat instance with a set of parameters for each model to provide additional control over its responses.

LLM serving is equipped with two interaction modes with language models. The only difference between these two is keeping the context of prior messages while responding to the query. Let’s take a quick look at the explanation.

• Single-shot mode : Each prompt request to the model is treated independently while generating the response, with no memory of previous turns.
• Conversation mode : Maintains context throughout the session. Prior turns are passed as context, allowing multi-turn conversations.

Execute LLM Endpoint
With Catalyst QuickML, you can tune the responses of available large language models according to your needs and access them from your application using authenticated endpoints. An LLM Endpoint is created from a saved parameter configuration, so the model used, tools, Instructions, system prompt, and generation parameters you tested in the console are exactly what your application calls.

The SDK method you call depends on the interaction mode the endpoint was configured with:

a. Generatean LLM response

Single-shot mode of interaction with language model requires the input prompt along with the valid endpoint key. The endpoint key will be generated at the time of endpoint creation. The ask_llm(endpoint_key, prompt) method sends a single prompt to the published LLM serving endpoint and returns the generated response. Each call is processed independently; no conversation context is retained between requests.

Parameters Used

Sample Code Snippet

# Create a QuickML instance.
quickml = app.quick_ml()
# Ask LLM
# Replace with your endpoint key copied from the Catalyst console.
endpoint_key = "<ENDPOINT_KEY>"
# Enter the prompt you want to send to the LLM.

prompt = “<YOUR_PROMPT>”
response = quickml.ask_llm (endpoint_key, prompt)
print(response)

The syntax of the model response received is shown below:

{
   "status": "success",
  "result": [
    {
      "content": "The generated response text from the model.",
      "finish_reason": "stop",
      "usage": {
        "input_tokens": 42,
        "output_tokens": 128,
        "total_tokens": 170
      }
    }
  ]
}

Use this method for single-shot interaction tasks such as summarization, classification, content generation, or extraction.

b. Converse with an LLM

The conversation mode of interaction with language model required the input prompt with the valid endpoint_key and a conversation_id. The endpoint key will be generated at the time of endpoint creation. The converse_with_llm(endpoint_key, prompt, conversation_id) method sends a prompt to a published LLM endpoint while retaining the context of the previous interactions in the same conversation.

Parameters used

Sample Code Snippet

# Create a QuickML instance.
quickml = app.quick_ml()
# Converse with  LLM
# Replace with your endpoint key copied from the Catalyst console.
endpoint_key = "<ENDPOINT_KEY>"
# Enter your prompt.
prompt = "<YOUR_PROMPT>"
# For the first request, use "-1".
# For subsequent requests, use the conversation ID returned in the previous response.
conversation_id = "<CONVERSATION_ID>"
response = quickml.converse_with_llm (
  endpoint_key,
  prompt,
  conversation_id
)
print(response)

The syntax of the model response received is shown below:

{
  "status": "success",
  "result": [
    {
      "conversation_id": "55663000000288001 ",
      "content": "The generated response text from the model.",
      "finish_reason": "stop",
      "usage": {
        "input_tokens": 310,
        "output_tokens": 96,
        "total_tokens": 406
      }
    }
  ]
}

Use this method to build chat experiences where the model must remember what was discussed earlier.

Where to find the endpoint information
Create an endpoint for your Saved LLM configuration and access the endpoint details page to view the Endpoint URL, required headers and a sample request response.