--- title: "LLM Serving" url: https://docs.catalyst.zoho.com/en/quickml/help/generative-ai/llm-serving/ description: "QuickML is a fully no-code ML pipeline builder service in the Catalyst development platform for creating machine-learning pipelines with end-to-end solutions." last_updated: 2026-08-17T12:16:04Z service: QuickML type: general --- ## Introduction to Large Language Model (LLM) A **Large Language Model (LLM)** is an advanced AI system trained to generate human-like text based on patterns learned from vast amounts of data such as books, websites, and articles. It leverages deep learning techniques, particularly the Transformer neural network architecture, which processes entire sequences of input in parallel—unlike earlier models like Recurrent Neural Networks (RNNs), which handled inputs sequentially. This parallelism enables the use of GPUs for efficient training, significantly accelerating the process. This training typically involves unsupervised or self-supervised learning, where the model learns to predict the next word in a sentence by recognizing patterns, grammar, and context within the data. > **Note:** QuickML's LLM Serving feature is currently available in US, IN, and EU data centers. ### Mechanism of LLM A fundamental aspect of how Large Language Models (LLMs) operate lies in their approach to representing words. Earlier machine learning models relied on simple numerical tables to represent individual words, which made it difficult to capture the relationships between words—particularly those with similar meanings. This limitation was addressed with the introduction of multi-dimensional vectors, known as word embeddings. These embeddings map words into a vector space where semantically or contextually similar words are positioned close to each other. With these embeddings, transformers convert text into numerical formats using the encoder, allowing the model to grasp context, meaning, and linguistic relationships—such as synonyms or grammatical roles. This processed understanding is then used by the decoder to generate meaningful and coherent text, enabling LLMs to produce responses that reflect the structure and flow of natural language. ### Applications of LLM     LLM enables real-time AI-powered solutions across various industries. Here are key applications where LLMs are making a significant impact: **Chatbots and virtual assistants** LLMs power AI-driven chatbots and virtual assistants used in customer service, e-commerce, and enterprise support. These assistants can handle customer queries, provide automated responses, and assist with troubleshooting. Businesses use them to enhance user experience and reduce response time. Content generation and summarization   Organizations leverage LLMs to generate high-quality content, such as articles, reports, and product descriptions. Additionally, LLMs summarize lengthy documents or news articles into concise and easy-to-read formats, saving time and effort for users. **Code generation and debugging** Developers benefit from LLMs that assist with writing, optimizing, and debugging code in multiple programming languages. These models help streamline software development by providing instant code suggestions and explanations, reducing development time. **Language translation and localization** LLMs enhance translation services by providing real-time, context-aware translations for businesses and individuals. This is particularly useful in global communication, enabling seamless interaction between people speaking different languages. Image and text analysis (Multimodal AI)    Advanced multimodal LLMs, such as Qwen 2.5 - 7B Vision Language, process both text and images. These models can describe images, recognize objects, and answer visual questions. Industries such as healthcare, accessibility, and digital content moderation use these capabilities for enhanced automation and decision-making. ## Understanding LLM Serving LLM Serving involves deploying and running large language models (LLMs) so they can handle real-time requests for predictions or responses. When an LLM is trained, it can be used to perform a variety of tasks, such as generating text, answering questions, translating languages, or even understanding and analyzing large datasets. ### Purpose of LLM Serving The primary purpose of LLM Serving is to bridge the gap between a trained model and its real-world usage. It enables organizations to: - Operationalize AI models in production environments - Ensure scalability and reliability for handling real-time, concurrent requests - Offer seamless integration into products, tools, and business processes - Transform LLMs from research tools into practical, usable systems that drive business outcomes ## Architecture of LLM Serving  A well-structured architecture ensures that an LLM-serving system runs efficiently and responds quickly. It typically consists of the following layers: - **Client layer**: Receives requests from users or applications, such as questions or text inputs. - **API layer**: Converts the requests into a format the LLM can understand and sends them to the model. - **Model layer**: Runs the LLM, processes the request, and generates a response. - **Data layer**: Handles input and output data, ensuring smooth data flow between the model and users. ![llm-15.webp](https://catalyst-docs-development.zohostratus.in/images/help/quickml/llm-serving/llm-15.webp) ### Unique features of LLM Serving in QuickML QuickML makes it easy to use various large language models (LLMs) within a chat interface. Users can select different models based on their needs and get real-time responses. Unlike many competing platforms that offer limited or rigid parameter tuning, QuickML empowers you with extensive customization options, ensuring greater flexibility and control. Here’s what sets QuickML apart: - **Fine-tune responses effortlessly**: Adjust creativity, coherence, and output length directly within the user-friendly interface. - **Seamless model switching**: Easily switch between multiple LLM models within a single chat interface to find the best fit for your needs. - **Effortless integration**: Deploy models into third-party applications using the provided endpoint URL for smooth and scalable implementation. - **Optimized performance & cost control**: Tailor responses to minimize unnecessary token usage, optimizing both speed and cost-efficiency. - **Enhanced accessibility**: No complex setup required; QuickML makes advanced AI capabilities accessible even for non-technical users. ### Breakdown of Chat Interface Before looking into how to access the LLM Serving feature in QuickML, let's have a brief look at how the chat interface is structured. **Model selection** Located at the top left under the “Chat” tab, this section allows you to choose from the available LLM models. It enables quick model switching within the same interface, making it easier to test different models for specific use cases without leaving the chat interface. ![llm-4.webp](https://catalyst-docs-development.zohostratus.in/images/help/quickml/llm-serving/llms-4.webp) **View model details** Next to the selected model name, the View Model Details option opens a pop-up window showing detailed information about the model. This includes aspects like model size, input token limits, training data, and integration options. It gives you deeper insights into what the model can do and how it can be used. ![llm-5.webp](https://catalyst-docs-development.zohostratus.in/images/help/quickml/llm-serving/llm-5.webp) **Chat interface (Conversation panel)** This central area is where all interactions take place. You can type prompts and view model-generated responses in a threaded format. Each response includes icons for actions like copying and regenerating individual responses. ![llm-6.webp](https://catalyst-docs-development.zohostratus.in/images/help/quickml/llm-serving/llm-6.webp) **Parameters panel** Found on the right side, this panel lets you fine-tune model behavior by adjusting parameters such as Temperature, Top-K, Top-P, and Max Tokens. There’s also an instruction field to specify tone, format, or domain-specific requirements. These settings help align the output to enterprise or user-specific needs. ![llm-7.webp](https://catalyst-docs-development.zohostratus.in/images/help/quickml/llm-serving/llm-7.webp) **Chatbox** At the bottom of the screen lies the chat input box—the main area where you can type your prompts or questions to interact with the selected language model. ### Parameters in LLM serving  QuickML provides a robust and flexible LLM (Large Language Model) serving environment, enabling developers and businesses to fine-tune AI model behavior through a variety of customizable parameters. These settings allow you to control how the model interprets inputs, generates responses, and aligns its output with your desired tone, structure, and purpose. Whether you're building a legal advisor bot, a content creation tool, or a customer service assistant, QuickML’s parameter configuration options ensure your application delivers responses that are not only relevant, but also contextually tailored. Unlike other platforms that restrict user control, QuickML prioritizes transparency and adaptability—allowing you to balance creativity with precision, determine the ideal response length, and guide the model to behave in ways that best serve your domain-specific needs. Here’s a breakdown of the available parameters: **Model name** The model field allows you to copy the model name for easy reference and deployment. **Instructions** In the Instructions field box, you can input detailed instructions, guiding the model’s response style and content generation approach. This enhances output relevance and consistency for specific applications. For example, a legal firm can enter instructions like "Provide responses in a formal legal tone with citations where applicable.", ensuring AI-generated content aligns with compliance and professional standards. **Temperature** Controls the creativity level of the model’s responses: - **Lower values (e.g., 0.0 - 0.3)**: More deterministic and precise responses. - **Higher values (e.g., 0.7 - 1.0)**: Increases variability and creativity, making responses more diverse and engaging. For example, a financial institution setting the Temperature to 0.2 would get precise responses like "The Federal Reserve increased interest rates by 0.25%." while setting it to 0.8 might generate "The Federal Reserve's recent rate hike of 0.25% aims to curb inflation, affecting mortgage and loan rates." **Top-K** Determines the number of top probable words considered for the next token prediction: - **Lower values (e.g., 10 - 20)**: Produces more predictable and controlled responses. - **Higher values (e.g., 50 - 100)**: Enhances diversity and variation in generated text. For example, a corporate HR chatbot with Top-K set to 10 may generate standard responses like "We value diversity in our hiring process.", whereas 50 allows for richer, more engaging responses like "At [Company Name], diversity is at the core of our hiring process, fostering innovation and inclusivity." **Top-P** Top-P sampling is also known as Nucleus sampling that generally ranges from 0.0 - 1.0. This parameter sets a cumulative probability threshold for the next token selection: - **Lower values (e.g., 0.1 - 0.3)**: Produces highly deterministic responses. - **Higher values (e.g., 0.8 - 0.9**): Enables more diverse text generation while maintaining coherence. For example, a customer support AI using a Top-P setting of 0.3 might generate straightforward responses like "Your order will arrive in 3 days.", while 0.9 could lead to "Your order is expected to arrive within 3 days. You'll receive a tracking update soon! Let us know if you need further assistance." **Max tokens** Defines the maximum number of tokens (between 1 and 4096) that the model can generate in a response. QuickML allows precise control over response length, optimizing cost and latency. For example, a compliance team setting Max Tokens to 50 ensures concise regulatory summaries, like "GDPR mandates data protection for EU citizens.", while 500 could generate a comprehensive legal analysis detailing key provisions and compliance measures. ### Working of LLM Serving in QuickML  1. When a user submits a request in QuickML's LLM serving chat interface, the selected model (such as Qwen 2.5 - 14B Instruct) processes the input. 2. The model analyzes the context, intent, and meaning of the query to generate a relevant and coherent response. 3. The response is formulated based on the model's training data, ensuring accuracy and contextual relevance. 4. Users can refine their queries or adjust model parameters to influence the response style, tone, and level of detail. ![llm-14.webp](https://catalyst-docs-development.zohostratus.in/images/help/quickml/llm-serving/llm-14.webp) **Key Points to be noted:** 1. The LLM serving feature is available to users who have access to the QuickML platform. 2. You can seamlessly switch between multiple LLM models within the QuickML chat interface. 3. At present, you cannot erase the previous chat conversations or open a new chat window; all conversations remain within the same chat interface. 4. Chats are user-specific, ensuring that one user cannot access another user's conversation. 5. You can upload images and type your query using the Qwen 2.5 - 7B Vision Language model for generating responses. 6. QuickML's LLM serving does not use your data for model training. All responses are generated based on its pre-trained data. ### Accessing LLM Serving in QuickML  LLM Serving can be accessed in QuickML by following the below steps, 1. Log in to [QuickML](https://catalyst.zoho.com/quickml.html) platform. 2. Under the *Generative AI* section, select **LLM Serving**. 3. Navigate to the **Chat** tab. ![llm-8.webp](https://catalyst-docs-development.zohostratus.in/images/help/quickml/llm-serving/llms-8.webp) 4. Select the desired LLM model from the drop-down. - You can select either *Qwen 2.5 - 14B Instruct*, *Qwen 2.5 - 7B Coder*, or *Qwen 2.5 - 7B Vision Language* ![llm-9.webp](https://catalyst-docs-development.zohostratus.in/images/help/quickml/llm-serving/llms-9.webp) 5. Start entering your queries in the chat interface. **Note:** The chat interface generates responses based on the default parameter configuration. However, you can adjust the settings as needed to suit your requirements. **To configure the parameters settings** 1. Under the *Generative AI* section, select **LLM Serving** tab. 2. Navigate to the **Chat** tab. ![llm-10.webp](https://catalyst-docs-development.zohostratus.in/images/help/quickml/llm-serving/llm-10.webp) 3. Under *Parameters* in the right panel, configure the following: - Enter the detailed instructions on how the model should perform. - Control the creativity level of the model’s responses by adjusting the **Temperature**. - Adjust the number of top probable words considered for the next token prediction in the **Top-K** field. - Set the parameter for a cumulative probability threshold for the next token selection in the **Top-P** field. - Define the maximum number of tokens the model generates per response in the **Max Tokens** field. 4. Click **Save**. ![llm-11.webp](https://catalyst-docs-development.zohostratus.in/images/help/quickml/llm-serving/llm-11.webp) Let's consider a sample use case to understand how these parameters can be configured in a real-time business scenario. **Configuring parameters for a customer support chatbot in a financial institution**   **Usecase:** A bank wants to assist customers with general banking inquiries while ensuring responses are accurate, concise, and aligned with regulatory guidelines. **Step 1**: Adjusting Parameters for Optimal Performance
Parameter Configuration Reason
Model Qwen 2.5 -14B Instruct Selected for its balance of accuracy and efficiency.
Instructions Provide responses in a formal and professional tone. Ensure compliance with banking regulations. Keep answers concise but informative. Ensures consistency in customer interactions.
Temperature 0.2 Keeps responses factual and avoids unnecessary creativity.
Top-K 20 Limits word selection to the most relevant choices, reducing variability.
Top-P 0.3 Keeps responses focused and predictable, ensuring regulatory compliance.
Max Tokens 200 Prevents overly long responses while maintaining enough detail for clarity.
**Step 2**: Engaging with the Chat Interface   Once the parameters are set, the chatbot is ready for use. Here’s an example interaction: User:"What are the eligibility criteria for a home loan?" Chatbot Response (Qwen 2.5 - 14B Instruct, with adjusted parameters): Eligibility criteria for a home loan typically include several key factors to ensure the borrower's ability to repay the loan. These criteria generally encompass: - **Age**: Borrowers must be within a specific age range, usually between 21 and 65 years old at the time of loan maturity. - **Income**: A stable and verifiable income is required to demonstrate the ability to repay the loan. Lenders may require proof of income through salary slips, bank statements, or tax returns. - **Credit Score**: A good credit score is essential as it reflects the borrower's creditworthiness and repayment history. - **Employment**: Borrowers should be employed with a stable job or have a consistent business income. Loan-to-Value (LTV) Ratio: This ratio compares the amount of the loan to the value of the property. Lenders have specific LTV limits based on the type of property and other factors. ![llm-12.webp](https://catalyst-docs-development.zohostratus.in/images/help/quickml/llm-serving/llm-12.webp) ### Integrate LLM into your Applications   Beyond utilizing LLM models within the QuickML platform, you can seamlessly integrate them into your own applications using the provided endpoint URL. This allows businesses to incorporate AI-powered responses into various business processes, such as customer support bots, content automation tools, and data analysis applications. To enable secure and efficient integration, QuickML supports OAuth-based authentication for access token generation. You can refer to this [documentation](https://www.zoho.com/accounts/protocol/oauth/self-client/overview.html) for details on different types of OAuth applications and the steps required to generate and manage access tokens. **To get the endpoint URL** 1. Navigate to the **Generative AI** section within [QuickML](https://catalyst.zoho.com/quickml.html). 2. Select the **LLM Serving** tab. 3. In the *Models* tab, choose the desired model.  4. In the *Model Details* pop-up window, scroll to the **API Details** section to get the endpoint URL. ![llm-13.webp](https://catalyst-docs-development.zohostratus.in/images/help/quickml/llm-serving/llm-13.webp)