Wallaroo Data Connections are provided to establish connections to external data stores for requesting or submitting information. They provide a source of truth for data source connection information to enable repeatability and access control within your ecosystem. The actual implementation of data connections are managed through other means, such as Wallaroo Pipeline Orchestrators and Tasks, where the libraries and other tools used for the data connection can be stored.
Wallaroo Data Connections have the following properties:
Data Connections are created through the Wallaroo Client create_connection(name, type, details)
method.
Parameter | Type | Description |
---|---|---|
name | string (Required) | The name of the connection. Names must be unique. Attempting to create a connection with the same name as an existing connection will cause an error. |
type | string (Required) | The user defined type of connection. |
details | Dict (Required) | User defined configuration details for the data connection. These can be {'username':'dataperson', 'password':'datapassword', 'port': 3339} , or {'token':'abcde123==', 'host':'example.com', 'port:1234'} , or other user defined combinations. |
The SDK allows the data connections to be fully defined and stored for later use. This allows whatever type of data connection the organization uses to be defined, then applied to a workspace for other Wallaroo users to integrate into their code.
When data connections are displayed via the Wallaroo Client list_connections
, the details
field is removed to not show sensitive information by default.
wl.create_connection("houseprice_arrow_table",
"HTTPFILE",
{'host':'https://github.com/WallarooLabs/Wallaroo_Tutorials/tree/main/wallaroo-testing-tutorials/houseprice-saga/data/xtest-1k.arrow?raw=true'}
)
Field | Value |
---|---|
Name | houseprice_arrow_table |
Connection Type | HTTPFILE |
Details | ***** |
Created At | 2023-05-04T17:52:32.249322+00:00 |
The Wallaroo client get_connection(name)
method retrieves the connection with the Connection name matching the name
parameter.
Parameter | Type | Description |
---|---|---|
name | string (Required) | The name of the connection. |
In the following example, the connection name external_inference_connection
will be retrieved and stored into the variable inference_source_connection
.
inference_source_connection = wl.get_connection(name="external_inference_connection")
display(inference_source_connection)
Field | Value |
---|---|
Name | external_inference_connection |
Connection Type | HTTPFILE |
Details | ***** |
Created At | 2023-05-08T20:10:07.914083+00:00 |
The Wallaroo Client list_connections()
method lists all connections for the Wallaroo instance. When data connections are displayed via the Wallaroo Client list_connections
, the details
field is removed to not show sensitive information by default.
wl.list_connections()
name | connection type | details | created at |
---|---|---|---|
houseprice_arrow_table | HTTPFILE | ***** | 2023-05-04T17:52:32.249322+00:00 |
The method Workspace add_connection(connection_name)
adds a Data Connection to a workspace, and takes the following parameters.
Parameter | Type | Description |
---|---|---|
name | string (Required) | The name of the Data Connection |
The Connection method details()
retrieves a the connection details()
as a dict
.
display(connection.details())
{'host': 'https://github.com/WallarooLabs/Wallaroo_Tutorials/tree/main/wallaroo-testing-tutorials/houseprice-saga/data/xtest-1k.arrow?raw=true'}
The Workspace method remove_connection(connection_name)
removes the connection from the workspace, but does not delete the connection from the Wallaroo instance. This method takes the following parameters.
Parameter | Type | Description |
---|---|---|
name | String (Required) | The name of the connection to be removed from the workspace. |
The Connection method delete_connection()
removes the connection from the Wallaroo instance.
Before deleting a connection, it must be removed from all workspaces that it is attached to.