Inference via the Wallaroo SDK
How to perform inferences on deployed LLMs via the Wallaroo SDK.
Table of Contents
Deployed LLMs in Wallaroo acceptinference requests through the method wallaroo.pipeline.Pipeline.infer which accepts either a pandas DataFrame or an Apache Arrow table. The example below accepts a pandas DataFrame and returns the results as the same.
data = pd.DataFrame({'text': ['Summarize what LinkedIn is']})
result = llm_pipeline(data)
result["out.generated_text"][0]
'LinkedIn is a social networking platform designed for professionals and businesses to connect, share information, and network. It allows users to create a profile showcasing their work experience, skills, education, and achievements. LinkedIn is often used for:\n\n1. Job searching: Employers can post job openings, and job seekers can search and apply for positions.\n2. Networking: Professionals can connect with colleagues, clients, and industry peers to build relationships and stay informed about industry news and trends.\n3. Personal branding: Users can showcase their skills, expertise, and achievements to establish themselves as thought leaders in their industry.\n4. Business development: Companies can use LinkedIn to promote their products or services, engage with customers, and build brand awareness.\n5. Learning and development: LinkedIn offers online courses, tutorials, and certifications to help professionals upskill and reskill.\n\nOverall, LinkedIn is a powerful tool for professionals to build their professional identity, expand their network, and advance their careers.'
Tutorials
- [LLM Deploy Tutorial](https://docs.wallaroo.ai/wallaroo-deployment/wallaroo-deployment-tutorials/wallaroo-deployment-model-packaging-tutorials/llm-deployment-tutorials/wallaroo-llm-deploy/wallaroo-llm-tutorial-deploy-on-intel-x86/
- RAG LLM Deployment Tutorial