Datasource Management

ShardOne supports a variety of data sources, including cloud-based object storage, and databases including SQL and NoSQL databases. Upon running a query, the result is fetched by joining data from multiple datasources.

The Datasource Admin is responsible for creating, configuring, and managing the data sources in a workspace.

Datasources Supported by ShardOne

ShardOne currently supports different SQL and NoSQL Databases, cloud-based object storage etc. The following table depicts the version of the database supported by ShardOne and the configuration details to be provided while creating the data source.

The following file formats are supported for the files in AWS S3 bucket:

• CSV
• EXCEL

How to connect a data source?

1. To connect to a new data source, Click on the ‘Create New’ button on the Data sources page.

2. Upon clicking the button, a tab will be activated for the creation of a new data source. Within this tab, it is possible to input the New Data Source Name and Description. Additionally, the option exists to select the 'Type of the Datasource'.

Note: Name can only contain alphanumeric characters and underscores.

3. Select the type of the Datasource from the dropdown shown. Depending on the type chosen, a tab to fill in the configuration details of the corresponding data source will appear, details of which have been listed here.

4. Upon filling in the corresponding fields, hit ‘Test Connection and Save’. The application checks whether the connection is valid and checks if the provided DB exists or not. In a positive scenario where the connection with the DB is successful, the data source is created and saved to the workspace. Else if the test connection fails, it means the database does not exist or the configuration details are incorrect. Try out again by providing valid details.

The platform provides a search option on name and description, and sort options on Nmae, Type, Metadata Fetch and Items. .The datasources can also be filtered by multiple criteria like Owner, Type and Metadata Fetch.

Metadata Discovery

Upon the creation of a data source, metadata discovery takes place. This is a process where the tables or items in the data source are discovered, and the fields or columns in the items are identified along with their data type. This is achieved by ShardOne through a scanning process across the configured database. There are various stages involved which are indicated in ‘Metadata Fetch’ on the Datasource listing page. They are:

1. In Progress - This indicates that the metadata discovery is in progress i.e. ShardOne is progressing with extracting details of items and their fields along with the data types.

Hit the ‘Refresh Table’ icon to reload the list to see the updates in the status.

2. Completed - The metadata fetch has been completed and the items & fields are up for viewing now.

3. Failed - This indicates that the metadata fetch has failed. This can occur due to the lack of a stable network. In this scenario, try creating the data source again with a stable connection. If the same status is encountered, it's possible that something unexpected has occurred, and the support team can be contacted at support@shardone.com along with the logs. Refer to this to know on how to extract logs.

Items & Fields

Once metadata discovery is done, users can enter the datasource to see its items. This area presents a list of items, an option to map aliases (explained later), and the count of fields in each item.

The items can be searched by name and filtered by the Aliases added.

Further, the user can navigate inside an item to see the details of the fields present in it. The list showcases the field names, an option to map an alias and its datatype.

The field aliases can be searched by name, and filtered by the data types and alias.

Edit Datasource

The Workspace Admins and Datsource Admins have the privilege to edit a datasource to update the Datasource Name and its Description. Find the steps below:

1. Navigate to the Datasources tab inside your Workspace. On the rightmost side of the list against each datasource, an ‘Edit’ icon can be found.

2. On clicking the icon, a pop-up to edit the name and description will appear.

3. Hit ‘Update’ once the details are edited.

For now, ShardOne does not support modifying the Datasource Type or the Configuration Details of the connection. However, new Datasource with the new configuration details can be created if required. Find out more on how to Create a Datasource here.

Delete a Datasource

A datsource can be deleted by a Workpsace Admin or an Datasource Admin if they no longer use the same. Please not that this is a permanent action, where the datasource would get deleted for all the users associated with the workspace. Further, the queries that reference this datsource will not work as expected.

The steps are:

1. Navigate to the Datasource tab where all the datasources are listed and click the delete icon against the datasource that needs to be deleted.

2. When this action is taken, a confirmation pop-up will be displayed. If the action is certain, click the ‘Delete’ button.