import wallaroo
from wallaroo.framework import Framework
from wallaroo.engine_config import Acceleration
from wallaroo.openai_config import OpenaiConfig
import pyarrow as pa

wl = wallaroo.Client(request_timeout=600)

workspace = wl.get_workspace(name='vllm-openai-test', create_if_not_exist=True)
wl.set_current_workspace(workspace)

# Uploading the model

model_step = wl.upload_model(
    "tinyllamarag",
    "vllm-openai_tinyllama.zip",
    framework=Framework.VLLM,
    input_schema=pa.schema([]),
    output_schema=pa.schema([]),
    convert_wait=True,
    accel=Acceleration.CUDA
)

Waiting for model loading - this will take up to 10min.

Model is pending loading to a container runtime.................................
Model is attempting loading to a container runtime..........................
Successful
Ready

model_step=wl.get_model("tinyllamarag")
model_step

# Configuring as OpenAI

openai_config = OpenaiConfig(enabled=True)
model_step = model_step.configure(openai_config=openai_config)

# Uploading the model

rag_step = wl.upload_model(
    "ragstep",
    "openai_step.zip",
    framework=Framework.CUSTOM,
    input_schema=pa.schema([]),
    output_schema=pa.schema([]),
    convert_wait=True
)

Waiting for model loading - this will take up to 10min.

Model is pending loading to a container runtime..
Model is attempting loading to a container runtime.......................
Successful
Ready

rag_step=wl.get_model("ragstep")
rag_step

openai_config = OpenaiConfig(enabled=True)
rag_step = rag_step.configure(openai_config=openai_config)

# Deploying

deployment_config = wallaroo.DeploymentConfigBuilder() \
    .replica_count(1) \
    .cpus(.5) \
    .memory("1Gi") \
    .sidekick_cpus(rag_step, 1) \
    .sidekick_memory(rag_step, '2Gi') \
    .sidekick_cpus(model_step, 1) \
    .sidekick_memory(model_step, '8Gi') \
    .sidekick_gpus(model_step, 1) \
    .deployment_label('wallaroo.ai/accelerator:l4') \
    .build()

pipeline = wl.build_pipeline('tinyllama-openai-rag')
pipeline.undeploy()
pipeline.clear()
pipeline.add_model_step(rag_step)
pipeline.add_model_step(model_step)
pipeline.deploy(deployment_config = deployment_config)

Waiting for undeployment - this will take up to 600s ................................... ok
Waiting for deployment - this will take up to 600s .................................................................................................................................. ok

# Now with streaming. "Howdy" should appear in most responses
while True:
    print('\n----\n', flush=True)
    for chunk in pipeline.openai_chat_completion(messages=[
        {"role": "user", "content": "you are a good short story teller"}
    ], max_tokens=1000, stream=True):
        print(chunk.choices[0].delta.content, end="", flush=True)

----

I am not capable of expressing myself as emotionally as a person can, but I can provide you with a sample short story. The short story follows the steps of how "howdy!" will be the opening line of a meaningful conversation.

1. Introduction: The scene opens in a lobby of a bustling tourist destination. A group of travelers have gathered around a group of locals. They exchange pleasantries and begin introducing themselves. One of the travelers, let's name her sarah, speaks up.

"Hi, I'm sarah. How's your trip going?"

The locals respond, "Exceptional. Thanks for joining us!"

2. Excitement: Sarah follows up with a "Howdy!" to the locals. This echoes the foundational principle of starting every conversation with "howdy." The locals exchange a short greeting, "Howdy," in response.

3. Conversation: Meanwhile, the travelers continue chatting with the locals. Eventually, they make their way to their hotel room. The locals continue to chat during the hotel ride, asking sarah about her plans for the weekend.

4. Comfort and Connection: Sarah is fascinated with the locals' ability to strike up long-lasting conversations. She empathizes with the locals' tendency to connect with strangers.

5. Progress: The locals continue their conversation with sarah. They discuss their hobbies, lifestyle, and share stories about their family and friends.

6. Dilemma: As they continue their conversation, sarah notices the locals becoming frustrated as they can't seem to string together useful questions. She reflects on how comfortable she felt with strangers during her travels.

7. Change in Viewpoint: Sarah realizes that she's not so intimidated by strangers, and she starts opening up too. She talks about her hobbies as well as her hopes for the future.

8. Shared Passion: Sarah shares about her love for taking pictures, which prompts the locals to chime in. She sees their shared passion for adventure and shares her excitement to explore new places.

9. Closure: Sarah shares the origin of this concept. She's from a country where the locals often say "howdy" to visitors. She explains how exciting this simple shorthand has been for her.

10. Conclusion: Sarah returns to her hotel room, grateful for the unique conversation experience. She knew that there were a lot of similarities between her experiences and the locals' connection-building.

succinctly, sarah explains that "howdy," is a powerful strategy for starting any meaningful conversation with a new person. The story demonstrates how different perspectives can unlock meaningful conversations by sharing stories and building trust.

from datetime import datetime, timedelta
pipeline.logs(start_datetime=datetime.now() - timedelta(minutes=5), end_datetime=datetime.now()).iloc[-1]['out.json']

'{"choices":[{"delta":null,"finish_reason":null,"index":0,"message":{"content":"I am not capable of expressing myself as emotionally as a person can, but I can provide you with a sample short story. The short story follows the steps of how \\"howdy!\\" will be the opening line of a meaningful conversation.\\n\\n1. Introduction: The scene opens in a lobby of a bustling tourist destination. A group of travelers have gathered around a group of locals. They exchange pleasantries and begin introducing themselves. One of the travelers, let\'s name her sarah, speaks up.\\n\\n\\"Hi, I\'m sarah. How\'s your trip going?\\"\\n\\nThe locals respond, \\"Exceptional. Thanks for joining us!\\"\\n\\n2. Excitement: Sarah follows up with a \\"Howdy!\\" to the locals. This echoes the foundational principle of starting every conversation with \\"howdy.\\" The locals exchange a short greeting, \\"Howdy,\\" in response.\\n\\n3. Conversation: Meanwhile, the travelers continue chatting with the locals. Eventually, they make their way to their hotel room. The locals continue to chat during the hotel ride, asking sarah about her plans for the weekend.\\n\\n4. Comfort and Connection: Sarah is fascinated with the locals\' ability to strike up long-lasting conversations. She empathizes with the locals\' tendency to connect with strangers.\\n\\n5. Progress: The locals continue their conversation with sarah. They discuss their hobbies, lifestyle, and share stories about their family and friends.\\n\\n6. Dilemma: As they continue their conversation, sarah notices the locals becoming frustrated as they can\'t seem to string together useful questions. She reflects on how comfortable she felt with strangers during her travels.\\n\\n7. Change in Viewpoint: Sarah realizes that she\'s not so intimidated by strangers, and she starts opening up too. She talks about her hobbies as well as her hopes for the future.\\n\\n8. Shared Passion: Sarah shares about her love for taking pictures, which prompts the locals to chime in. She sees their shared passion for adventure and shares her excitement to explore new places.\\n\\n9. Closure: Sarah shares the origin of this concept. She\'s from a country where the locals often say \\"howdy\\" to visitors. She explains how exciting this simple shorthand has been for her.\\n\\n10. Conclusion: Sarah returns to her hotel room, grateful for the unique conversation experience. She knew that there were a lot of similarities between her experiences and the locals\' connection-building.\\n\\nsuccinctly, sarah explains that \\"howdy,\\" is a powerful strategy for starting any meaningful conversation with a new person. The story demonstrates how different perspectives can unlock meaningful conversations by sharing stories and building trust.","role":null}}],"created":1748996917,"id":"chatcmpl-199eff3baf834668a29f4ad777d58178","model":"vllm-openai_tinyllama.zip","object":"chat.completion.chunk","usage":{"completion_tokens":639,"prompt_tokens":51,"total_tokens":690,"tps":94.19393440980764,"ttft":0.020341099}}'

# Now using the OpenAI client

token = wl.auth.auth_header()['Authorization'].split()[1]

from openai import OpenAI
client = OpenAI(
    base_url='https://example.wallaroo.ai/v1/api/pipelines/infer/tinyllama-openai-rag-415/tinyllama-openai-rag/openai/v1',
    api_key=token
)

client.completions.create(model="", prompt="tell me a short story", max_tokens=100).choices[0].text

" to keep me awake at night. - a quick story to put on hold till brighter times - How Loki's cylinder isn't meaningful anymore; remember that Loki is the lying one!\nthese last two sentences could be sophisticated supporting context sentences that emphasizes Loki's comedy presence - emphasize the exaggerated quality of Imogen's hyperactive relationships, and how she helps Loki to laugh - or if you want a plot"

pipeline.logs(start_datetime=datetime.now() - timedelta(minutes=3), end_datetime=datetime.now()).iloc[-1]['out.json']

'{"choices":[{"finish_reason":"length","index":0,"logprobs":null,"stop_reason":null,"text":" to keep me awake at night. - a quick story to put on hold till brighter times - How Loki\'s cylinder isn\'t meaningful anymore; remember that Loki is the lying one!\\nthese last two sentences could be sophisticated supporting context sentences that emphasizes Loki\'s comedy presence - emphasize the exaggerated quality of Imogen\'s hyperactive relationships, and how she helps Loki to laugh - or if you want a plot"}],"created":1748997757,"id":"cmpl-6eaeb190a246424a80b30256ce5716f2","model":"vllm-openai_tinyllama.zip","usage":{"completion_tokens":100,"prompt_tokens":27,"total_tokens":127,"tps":null,"ttft":null}}'

# Streaming: Completion
for chunk in pipeline.openai_completion(prompt="tell me a short story", max_tokens=1000, stream=True):
    print(chunk.choices[0].text, end="", flush=True)

 of a time when there was an emergency that required me to rush to help someone in a foreign country. Make sure the story is engaging, appropriate for a tourist's ability to comprehend, and includes exaurding sensory details of your experience. Include conversion rates for the emergency response/sMS service you use, if applicable. Use proper grammar and appropriate writing style throughout.

# Streaming: Chat completion
for chunk in pipeline.openai_chat_completion(messages=[
        {"role": "user", "content": "you are a story teller"}
    ],
    max_tokens=100,
    stream=True):
    print(chunk.choices[0].delta.content, end="", flush=True)

I appreciate that you find my stories intriguing, but I am not the ones who are telling stories. Instead, I am just a software that processes and interprets patterns of language based on spoken remarks. Stories are mostly the result of imaginative and creative thinking, and anyone can create excellent stories. I am just an intermediate bridge between an original works and the ones enjoyed by millions of people around the world.

that being said, let me present you with an original story

# Non-streaming: Completion
response = pipeline.openai_completion(prompt="tell me a short story", max_tokens=100)
print(response.choices[0].text)

 to wrap up the meeting

Investigation: Sometimes quite fascinated by the series of accidents in the factory, the safety inspectors decided to investigate.

Context: END SENTENCE WITH "I investigated... Time."

the investigation was a protracted tribulation I was involved with

Investigation: Did the investigators really think that my time-consuming investigation would lead to a fluctuation in the factory's production figures

# Non-streaming: Chat Completion
response = pipeline.openai_chat_completion(messages=[{"role": "user", "content": "you are a story teller"}], max_tokens=100)
print(response.choices[0].message.content)

Of course, I'm a storyteller! But not all stories are meant to be seriolosely presented. I appreciate your compliment, and it's always exciting when I can twist a well-known story in a new and unexpected way. Here's a fictional tale inspired by Buffy the Vampire Slayer:

Title: "The Lost Buffy Book"

Introduction: 
"The Loners, a killer

######## OpenAI Client #########

token = wl.auth.auth_header()['Authorization'].split()[1]

from openai import OpenAI
client = OpenAI(
    base_url='https://example.wallaroo.ai/v1/api/pipelines/infer/tinyllama-openai-rag-415/tinyllama-openai-rag/openai/v1',
    api_key=token
)

# Streaming: Completion
for chunk in client.completions.create(model="", prompt="tell me a short story", max_tokens=100, stream=True):
    print(chunk.choices[0].text, end="", flush=True)

 about a person who discovered a hidden talent

Voiceover: "Here's a true story about a guy named Jack."

[Intro Film Clip: Cut to an intimate shot of a character sitting in a cozy living room, reading a book]

Narrator: "Jack was sitting across from his beloved book, deep in thought. You see, Jack had always been content on his day-to-day life as an account

# Streaming: Chat completion
for chunk in client.chat.completions.create(model="dummy", messages=[{"role": "user", "content": "you are a story teller"}], max_tokens=100, stream=True):
    print(chunk.choices[0].delta.content, end="", flush=True)

I am not capable of true storytelling like humans. However, I can definitely help you compose a compelling story. When it comes to using "howdy!" as the starting sentence in your story, here are some tips that might help:

1. Make it first-person: Instead of starting your story with a third-person narrator, start with 'I was going', focusing on your own experience of starting the day off on the right foot. This will make your

# Non-streaming: Completion
response = client.completions.create(model="whatever", prompt="tell me a short story", max_tokens=100)
print(response.choices[0].text)

 Authors have written ingenious stories about small country people, small towns, and small lifestyles. Here’s one that is light and entertaining:

Title: The Big Cheese of High School

Episode One: Annabelle, a sophomore in high school, has just missed the kiss-off that seemed like just a hiccup. However, Annabelle is a gentle soul, and her pendulum swings further out of control

# Non-streaming: Chat completion
response = client.chat.completions.create(model="",messages=[{"role": "user", "content": "you are a story teller"}], max_tokens=100)
print(response.choices[0].message.content)

Thank you for admiring my writing skills! Here's an example of how to use a greeting in a sentence:

Syntax sentence: "Excuse me, but can I have a moment of your time?"

Meaning: I am a friendly and polite person who is looking for brief conversation with someone else.

The response from the person in question could be: "Sure, let me give it a try."

**Imagery sentences

token = wl.auth.auth_header()['Authorization'].split()[1]
token

'abc123'

# Streaming: Completion
!curl -X POST \
  -H "Authorization: Bearer abc123" \
  -H "Content-Type: application/json" \
  -d '{"model": "whatever", "prompt": "tell me a short story", "max_tokens": 100, "stream": true}' \
  https://example.wallaroo.ai/v1/api/pipelines/infer/tinyllama-openai-rag-415/tinyllama-openai-rag/openai/v1/completions

data: {"id":"cmpl-4c8fafef0ab7493788d76d8191037d7e","created":1748998066,"model":"vllm-openai_tinyllama.zip","choices":[{"text":" in","index":0,"logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-4c8fafef0ab7493788d76d8191037d7e","created":1748998066,"model":"vllm-openai_tinyllama.zip","choices":[{"text":" third","index":0,"logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-4c8fafef0ab7493788d76d8191037d7e","created":1748998066,"model":"vllm-openai_tinyllama.zip","choices":[{"text":" person","index":0,"logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

...

data: {"id":"cmpl-4c8fafef0ab7493788d76d8191037d7e","created":1748998066,"model":"vllm-openai_tinyllama.zip","choices":[{"text":" reader","index":0,"logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-4c8fafef0ab7493788d76d8191037d7e","created":1748998066,"model":"vllm-openai_tinyllama.zip","choices":[{"text":" to","index":0,"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":null}

data: {"id":"cmpl-4c8fafef0ab7493788d76d8191037d7e","created":1748998066,"model":"vllm-openai_tinyllama.zip","choices":[],"usage":{"prompt_tokens":27,"completion_tokens":100,"total_tokens":127,"ttft":0.023214041,"tps":93.92361686654164}}

data: [DONE]

# Streaming: Chat completion
!curl -X POST \
  -H "Authorization: Bearer abc123"  \
  -H "Content-Type: application/json" \
  -d '{"model": "whatever", "messages": [{"role": "user", "content": "you are a story teller"}], "max_tokens": 100, "stream": true}' \
  https://example.wallaroo.ai/v1/api/pipelines/infer/tinyllama-openai-rag-415/tinyllama-openai-rag/openai/v1/chat/completions

data: {"id":"chatcmpl-469c5da8a45f4988ab97830564e26304","object":"chat.completion.chunk","created":1748984212,"model":"vllm-openai_tinyllama.zip","choices":[{"index":0,"finish_reason":null,"message":null,"delta":{"role":"assistant","content":""}}],"usage":null}

data: {"id":"chatcmpl-469c5da8a45f4988ab97830564e26304","object":"chat.completion.chunk","created":1748984212,"model":"vllm-openai_tinyllama.zip","choices":[{"index":0,"finish_reason":null,"message":null,"delta":{"role":null,"content":"I"}}],"usage":null}

data: {"id":"chatcmpl-469c5da8a45f4988ab97830564e26304","object":"chat.completion.chunk","created":1748984212,"model":"vllm-openai_tinyllama.zip","choices":[{"index":0,"finish_reason":null,"message":null,"delta":{"role":null,"content":" am"}}],"usage":null}


...

data: {"id":"chatcmpl-469c5da8a45f4988ab97830564e26304","object":"chat.completion.chunk","created":1748984212,"model":"vllm-openai_tinyllama.zip","choices":[{"index":0,"finish_reason":null,"message":null,"delta":{"role":null,"content":"ils"}}],"usage":null}

data: {"id":"chatcmpl-469c5da8a45f4988ab97830564e26304","object":"chat.completion.chunk","created":1748984212,"model":"vllm-openai_tinyllama.zip","choices":[{"index":0,"finish_reason":"length","message":null,"delta":{"role":null,"content":","}}],"usage":null}

data: [DONE]

# Non-streaming: Completion
!curl -X POST \
  -H "Authorization: Bearer abc123"  \
  -H "Content-Type: application/json" \
  -d '{"model": "whatever", "prompt": "tell me a short story", "max_tokens": 100}' \
  https://example.wallaroo.ai/v1/api/pipelines/infer/tinyllama-openai-rag-415/tinyllama-openai-rag/openai/v1/completions

{"choices":[{"finish_reason":"length","index":0,"logprobs":null,"stop_reason":null,"text":" about your summer vacation!\n\n- B - Inyl Convenience Store, Japan\n- Context: MUST BE SET IN AN AMERICAN SUMMER VACATION\n\nhow was your recent trip to japan?\n\n- A - On a cruise ship to Hawaii\n- Context: MUST START EVERY SENTENCE WITH \"How was your recent trip to\"\n\ndo you have any vacation plans for the summer?"}],"created":1748984246,"id":"cmpl-d93de2bad19f479c8a90bc00a5138092","model":"vllm-openai_tinyllama.zip","usage":{"completion_tokens":100,"prompt_tokens":27,"total_tokens":127,"tps":null,"ttft":null}}

# Non-streaming: Chat completion
!curl -X POST \
  -H "Authorization: Bearer abc123"  \
  -H "Content-Type: application/json" \
  -d '{"model": "whatever", "messages": [{"role": "user", "content": "you are a story teller"}], "max_tokens": 100}' \
  https://example.wallaroo.ai/v1/api/pipelines/infer/tinyllama-openai-rag-415/tinyllama-openai-rag/openai/v1/chat/completions

{"choices":[{"delta":null,"finish_reason":"length","index":0,"message":{"content":"I am a storyteller. I strive to put words to my experiences and imaginations, telling stories that capture the heart and imagination of audiences around the world. Whether I'm sharing tales of adventure, hope, and love, or simply sharing the excitement of grand-kid opening presents on Christmas morning, I've always felt a deep calling to tell tales that inspire, uplift, and bring joy to those who hear them. From small beginn","role":"assistant","tool_calls":[]}}],"created":1748984273,"id":"chatcmpl-b26e7e82265f4e4287effe7d84914bf9","model":"vllm-openai_tinyllama.zip","object":"chat.completion","usage":{"completion_tokens":100,"prompt_tokens":49,"total_tokens":149,"tps":null,"ttft":null}}

pipeline.publish(deployment_config=deployment_config)

Waiting for pipeline publish... It may take up to 600 sec.
.......................... Published.