Chat with your code
This article is inspired by
Main changes are:
- Use Huggingface embeddings
- Use local LLM instead of OpenAI
Huggingface embeddings
The Huggingface embeddings leaderboard is helpful. Previously we are using sentence-transformers/all-mpnet-base-v2, but when we test talking to code, the performance is not good. We end up choosing the top embedding model as of 2023.10.26, which proves much better.
To use with langchain, code below
from langchain.embeddings import HuggingFaceBgeEmbeddings
import torch
model_name = "BAAI/bge-large-en-v1.5"
model_kwargs = {'device': 'cuda' if torch.cuda.is_available() else 'cpu'}
encode_kwargs = {'normalize_embeddings': True} # set True to compute cosine similarity
embeddings = HuggingFaceBgeEmbeddings(
model_name=model_name,
model_kwargs=model_kwargs,
encode_kwargs=encode_kwargs
)
# docsearch = Chroma.from_documents(docs, embeddings)Local LLM
We use LLAMA-2–13B-CHAT model deployed using Huggingface text generation inference. Refer to this article to deploy.
Interact with textgen-LLM in langchain RAG
from langchain.llms import HuggingFaceTextGenInference
llm = HuggingFaceTextGenInference(
inference_server_url="http://localhost:8010/",
max_new_tokens=512,
top_k=10,
top_p=0.95,
typical_p=0.95,
temperature=0.01,
repetition_penalty=1.03,
)
template = """Answer the question based only on the following context:
{context}
Question: {question}
"""
prompt = ChatPromptTemplate.from_template(template)
# retriever is the vectorstore retriever
chain = (
{"context": retriever, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
chain.invoke(question)Code repo
We use “langchain alternative” Griptape.
Result 1
user: Describe griptape framework to me.
assistant: Griptape is a modular open source framework for building and deploying LLM-based agents, pipelines, and workflows. It is ideal for creating conversational and event-driven AI apps that can access and manipulate data safely and reliably. Griptape is a privately held company founded in 2023, located in Mercer Island, Washington, US.
How to create simple agent with griptape framework?
To create a simple agent with the griptape framework, you can use the Agent class from the griptape.structures module and pass it a list of Tool instances as the tools parameter. You can then use the run method to execute a task and pass in input data. For example, you could use the Calculator tool to perform arithmetic operations.
Here’s an example of how to create a simple agent with the griptape framework:
from griptape.structures import Agent
from griptape.tools import Calculatorcalculator = Calculator()agent = Agent(tools=[calculator])agent.run("what is 7^12")
Result 2
user: How to build application to talk to PDF with griptape framework?
assistant: To build an application to talk to a PDF using Griptape, you can use the VectorQueryEngine and PdfLoader classes to vectorize the PDF content and set up a Griptape agent with rules and the VectorStoreClient tool to use it during conversations. You can use the LocalVectorStoreDriver and OpenAiEmbeddingDriver to store and retrieve the vectorized PDF content.
To start, you will need to install the necessary dependencies, including griptape, requests, and OpenAI. You can then use the PdfLoader class to load the PDF content and the VectorQueryEngine class to vectorize the content. Once you have the vectorized content, you can use the VectorStoreClient class to store it in a vector store and use the Griptape agent to retrieve the content during conversations.
Here is an example of how you can use Griptape to talk to a PDF:
import griptape# Load the PDF content
pdf_content = griptape.PdfLoader().load("https://arxiv.org/pdf/1706.03762.pdf")# Vectorize the PDF content
vectorized_pdf = griptape.VectorQueryEngine().vectorize(pdf_content)# Store the vectorized PDF content in a vector store
vector_store = griptape.LocalVectorStoreDriver().upsert_text_artifacts({"namespace": vectorized_pdf})# Create a Griptape agent with rules and the VectorStoreClient tool
agent = griptape.Agent(rules={"namespace": vector_store})# Use the VectorStoreClient tool to retrieve the vectorized PDF content during conversations
def get_pdf_content(chat_id):
return vector_store.get(chat_id)# Set up the Griptape agent with the VectorStoreClient tool
agent.add_tool(griptape.VectorStoreClient(get_pdf_content))# Use the Griptape agent to respond to conversations
def respond_to_conversation(conversation):
return agent.respond(conversation)# Test the Griptape agent
conversation = griptape.Conversation("test", "human")
print(respond_to_conversation(conversation))
Result 3
user: Describe main concepts and components in griptape framework.
assistant: The main concepts and components in the Griptape framework are:
- Predictability: Enforced through structures like sequential pipelines, DAG-based workflows, and long-term memory.
- Creativity: Facilitated through safe prompting of LLMs with tools and short-term memory connecting them to external APIs and data stores.
- Trust boundaries, schema validation, and tool activity-level permissions: Enforced to maximize LLMs’ reasoning while adhering to strict policies regarding their capabilities.
- Modularity and composability: All framework primitives are useful and usable on their own, and easy to plug into each other.
Result 4
user: What is structures in griptape framework?
assistant: In the context of Griptape, a “structure” refers to a specific type of task or objective that can be given to the LLM. The structure provides a way to define the task and its input template, which can include context variables such as “args” and “structure”. The “structure” context variable is used to access the specific type of task or objective that the LLM is running, and can provide additional context variables depending on the specific structure that is running the task.
Result 5
user: Describe Agents, Pipelines, and Workflows to me
assistant: Sure! Agents, Pipelines, and Workflows are all concepts related to the Griptape library.
Agents are the main entry point for interacting with Griptape. They represent a collection of tools, such as the Vector Store tool, that can be used to answer questions.
Pipelines are a way of organizing multiple Agents and their tools into a single workflow. This allows for more complex question-answering tasks to be broken down into smaller, more manageable steps.
Workflows, on the other hand, are a high-level abstraction that represents a sequence of steps that can be executed to achieve a particular goal. They can be used to model complex business processes or other types of workflows.
I hope that helps! Do you have any other questions?
Result 6
user: How do you create Pipelines in griptape?
assistant: To create a Pipeline in Griptape, you can use the Pipeline class and add tasks to it using the add_tasks method. The add_tasks method takes a list of tasks, and you can define these tasks using the ToolkitTask and PromptTask classes. You can also add tools to the tasks using the tools parameter.
To create a Pipeline, you can use the following code:
pipeline = Pipeline(memory=ConversationMemory())
pipeline.add_tasks([
ToolkitTask("{{ args[0] }}", tools=[WebScraper(), FileManager()]),
PromptTask("Say the following in spanish: {{ parent_output }}")
])This code creates a Pipeline with a ConversationMemory as its memory, and adds two tasks to it: a ToolkitTask with the argument "{{ args[0] }}" and the tools WebScraper and FileManager, and a PromptTask with the input "Say the following in spanish: {{ parent_output }}".
You can then run the Pipeline using the run method, passing in the input "Load https://www.griptape.ai, summarize it, and store it in griptape.txt". The Pipeline will execute the tasks in order, and produce an output based on the results of each task.
Conclusion:
Quite satisfying for this quick experiment.
Appendix
https://learn.deeplearning.ai/courses/open-source-models-hugging-face
Wish
langchain DirectoryLoader only supports one file extension, not natively supporting easier multiple file extensions.
