Wallaroo Data Connections are provided to establish connections to external data stores for requesting or submitting information. They provide a source of truth for data source connection information to enable repeatability and access control within your ecosystem. The actual implementation of data connections are managed through other means, such as Wallaroo Pipeline Orchestrators and Tasks, where the libraries and other tools used for the data connection can be stored.

Wallaroo Data Connections have the following properties:

• Available across the Wallaroo Instance: Data Connections are created at the Wallaroo instance level.
• Tied to a Wallaroo Workspace: Data Connections, like pipeline and models, are tied to a workspace. This allows organizations to limit data connection access by restricting users to specific workspaces.
• Support different types: Data Connections support various types of connections such as ODBC, Kafka, etc.

Create Data Connection

Data Connections are created through the Wallaroo Client create_connection(name, type, details) method.

The SDK allows the data connections to be fully defined and stored for later use. This allows whatever type of data connection the organization uses to be defined, then applied to a workspace for other Wallaroo users to integrate into their code.

When data connections are displayed via the Wallaroo Client list_connections, the details field is removed to not show sensitive information by default.

wl.create_connection("houseprice_arrow_table", 
                  "HTTPFILE", 
                  {'host':'https://github.com/WallarooLabs/Wallaroo_Tutorials/tree/main/wallaroo-testing-tutorials/houseprice-saga/data/xtest-1k.arrow?raw=true'}
                  )

Get Connection By Name

The Wallaroo client get_connection(name) method retrieves the connection with the Connection name matching the name parameter.

In the following example, the connection name external_inference_connection will be retrieved and stored into the variable inference_source_connection.

inference_source_connection = wl.get_connection(name="external_inference_connection")
display(inference_source_connection)

List Data Connections

The Wallaroo Client list_connections() method lists all connections for the Wallaroo instance. When data connections are displayed via the Wallaroo Client list_connections, the details field is removed to not show sensitive information by default.

wl.list_connections()

Add Data Connection to Workspace

The method Workspace add_connection(connection_name) adds a Data Connection to a workspace, and takes the following parameters.

Connection Details

The Connection method details() retrieves a the connection details() as a dict.

display(connection.details())

{'host': 'https://github.com/WallarooLabs/Wallaroo_Tutorials/tree/main/wallaroo-testing-tutorials/houseprice-saga/data/xtest-1k.arrow?raw=true'}

Remove Connection from Workspace

The Workspace method remove_connection(connection_name) removes the connection from the workspace, but does not delete the connection from the Wallaroo instance. This method takes the following parameters.

Delete Connection

The Connection method delete_connection() removes the connection from the Wallaroo instance.

Before deleting a connection, it must be removed from all workspaces that it is attached to.