Skip to main content

Tutorial #2 - Rhino Health Federated Computing Platform Data Harmonization on the FCP SDK

This document will guide you through another step-by-step usage example on the Rhino Health FCP. By following the step-by-step instructions in this document, you will learn how to:

  1. Interact with your project objects with the FCP Python SDK
  2. Import Datasets with the FCP Python SDK
  3. Run simple Python code using a pre-built container image
  4. Write reusable preprocessing scripts for messy data
  5. Use an Interactive Container to interact directly with distributed data

We are excited to see what you will do on the platform, so let’s get started! 

Step 1: Getting started

Credentials

As in Tutorial #1 - Rhino Health Federated Computing Platform “Hello World” - Basic Usage, you will need your FCP credentials (email and password) to log into https://dashboard.rhinohealth.com/login.

If you have not received those credentials - please contact support@rhinohealth.com.

Tutorial 2 Project Resources

Click here to access all the files you will need for this project. Download or clone the repository to your local computer. For this tutorial, you will be using the resources inside the client-resources/tutorials/tutorial_2/ folder

This folder includes:

  • data/ - This folder contains all the data that will be used within this tutorial.  The folder is structured in a way that will become more familiar to you as you begin to feel more comfortable using various parts of the system such as creating custom containers.
    • input/ - This folder is part of the system's structure as you utilize the features mentioned above. It contains the input data for the tutorial.
      • site1_part1_dataset_data.csv,
        site2_part1_dataset_data.csv,
        site3_part1_dataset_data.csv         - The files that define the datasets at the three different sites you will import during the tutorial.
      • site1_part2_dataset_data.csv,
        site2_part2_dataset_data.csv,
        site3_part2_dataset_data.csv         - The files that define the second part of the datasets at the three different sites. This second part of the dataset is meant to illustrate how easy it is to create re-usable data harmonization workflows using the Rhino Health Python SDK
  • notebooks/ - This folder contains the notebooks you will utilize within the tutorial.
    • Tutorial 2 - Data harmonization.ipynb - a Jupyter notebook containing the code you will run in this tutorial. 
  • schemas/ - This folder contains the Data Scehmas that will define the structure and variable type of the tutorial data.
    • Harmonization Schema.csv - The Data Schema you will use for this project.

Important: In a real multi-site project on the FCP, you will not need to collect data from different sites within your own client. Each site will hold its own data on its own Rhino Health client.

 

Step 2: Preparing your data

For your data to be available to the FCP, you will first need to transfer the data located within the data/ folder to be able to utilize it within the FCP UI.  If you are new to using SFTP and would like to learn more, please refer to the following support article: How can I move data from my local environment to my Rhino Health client using SFTP?

For this tutorial, you are going to move the following six resources from your local machine to your Rhino Health client:

  1. client-resources/tutorials/tutorial_2/data/input/site1_part1_dataset_data.csv
  2. client-resources/tutorials/tutorial_2/data/input/site2_part1_dataset_data.csv
  3. client-resources/tutorials/tutorial_2/data/input/site3_part1_dataset_data.csv
  4. client-resources/tutorials/tutorial_2/data/input/site1_part2_dataset_data.csv
  5. client-resources/tutorials/tutorial_2/data/input/site2_part2_dataset_data.csv
  6. client-resources/tutorials/tutorial_2/data/input/site3_part2_dataset_data.csv

 

Connecting to your SFTP Server for MacOS, Linux and Windows 10+

  1. Open a terminal or command prompt on your respective operating system and navigate to the folder client-resources/tutorials/tutorial_2/data/. 
  2. Connect to your client via SFTP using the following command:
    sftp rhinosftp@RHINO_CLIENT_IP_ADDRESS
  3. Copy the CSV files from your local machine into a new folder you create on your Rhino Health client by running the commands below:   
    sftp> mkdir tutorial_2
    sftp> cd tutorial_2
    sftp> put -r input/
    sftp> exit

Other Operating Systems

  1. If you have downloaded and configured your SFTP client, skip to the next step. Otherwise, please follow steps 1 & 2 outlined in the following support article under the heading Connecting to your Rhino Health Client via SFTP from Other Operating Systems
  2. Open your SFTP client and connect to your Rhino Client
  3. Using the STFP client to upload your data:
    • On the local machine file system panel, navigate to the folder client-resources/tutorials/tutorial_2/data/input/.
    • On your Rhino Health client file system panel, create a new folder called tutorial_2 and navigate inside of it
  4. Drag the CSV files from your local machine file system panel to the Rhino Health client file system panel in order to upload them to your Rhino Health client.

Wait until your files have successfully been uploaded to the Rhino Health client before proceeding to the next step.

 

Step 3: Set Up Your Project on the FCP UI

Creating a New Project within the FCP UI

  1. Log in at https://dashboard.rhinohealth.com/login. If this is your first time logging in, you will be required to change your initial password and sign the EULA. 
  2. Create a new project by clicking on the Add New Project button in the top right corner. 
  3. Fill in the following fields within the new modal window:
    1. Name: Tutorial 2 - YOUR_NAME
    2. Description: This is my second project on the Rhino Health FCP
    3. Permission Policy: Expand this section to explore the various configurable permission policies and personas that are available to you.  For this tutorial, you can leave the default permission policy.
  4. Click the Create Project button to create your project. Once clicked, you will be navigated back to the project screen where you will see your newly created project.

 

Creating a New Data Schema within the FCP UI

  1. Click on the project you just created on the FCP Projects Page.
  2. Click the Data Schemas menu item within the left-hand navigation menu.
  3. Create a new schema by clicking on the Create New Data Schema button in the top right corner.
  4. Fill in the following fields within the new modal window:
    1. Name: Harmonization Schema
    2. Description: Data Harmonization Schema
    3. Select the Upload From File radio button
    4. Within the newly visible Data Schema CSV File input field, click the Select File button.  A filesystem dialog should appear
    5. Navigate to where you placed the client-resources/ folder.  Further, navigate to tutorials/tutorial_2/schemas/Harmonization Schema.csv and select Open to load the CSV file
  5. Finally, click the Create New Data Schema button to create your new Data Schema.
  6. Within the Data Schemas page, you should now have a new Data Schema object defined, similar to the screenshot shown below.
    Tutorial 2 - Data Schema.png

 

Step 4: Rhino Health Python SDK

In this step, you will be interacting with the FCP using the programmatic interface, or Python SDK. To get started, you will need the following: 

  1. If you do not have Python installed within your development environment, please download and install Python here: Python
  2. If you do not have Jupyter Notebook installed within your development environment, please follow the steps outlined here: Jupyter Notebook Installation
  3. Using your terminal or command prompt, navigate to your client-resources/tutorials/tutorial_2/notebooks folder.
  4. Run the following command to start the Jupyter Notebook, Tutorial 2 - Data Harmonization.ipynb: 
    jupyter notebook "Tutorial 2 - Data Harmonization.ipynb"
  5. In the FCP GUI, go to the Projects dashboard and copy the UID for your newly created project as shown in the screenshot below. You will need this UID shortly within your Jupyter Notebook.
    Tutorial 2 - Copy UUID.png
  6. Follow the step-by-step tutorial for harmonizing data programmatically using the FCP Python SDK by running each of the cells contained within the notebook.

 

Step 5: Interactive Container

Sometimes knowing exactly what data preprocessing/harmonization code to prepare for each site is not trivial, and it would be very helpful to interact directly with the data at each site in order to get a better understanding of what’s going on. 

Interactive Containers is an additional type of supported code on the FCP. With this mechanism, your code is still running in a container, but you can open an interactive session with the on-prem client (think of it like a kind of remote desktop) where you can have access to pre-installed 3rd party applications, such as Jupyter Notebook, image analytics software like 3D Slicer, and many more. 

In this tutorial, you will build and push an interactive container from a prepared folder. This interactive container will include an installation of Jupyter Notebook, as well as an example notebook you can run on the distributed data in your project.  If you are interested in learning more about the Interactive Container Code Objects, please follow one of the links to the Interactive Container Code section of our User Guides.

 

Creating a New Interactive Container Code Object

  1. Return to the FCP to create a new Code Object
  2. Click the Code menu item within the left-hand navigation menu.
  3. Create a new Code Object by clicking on the Create New Code button in the top right corner.
  4. Fill in the following fields within the new modal window:
    1. Name: jupyter-notebook
    2. Description: Demo Jupyter Notebook Interactive Container
    3. Type: Interactive Container
    4. Input Data Schema: Harmonization Schema (v0)
    5. Output Data Schema: Do not Modify - The default option [Auto-generate Data Schema from Data] is correct for this tutorial
    6. Container: interactive-jupyter-notebook
  5. Next, click the Create New Code button to create your new Code within your project.

Running your New Interactive Container Code to Use Jupyter Notebook

  1. Navigate to the Code you created in the last step, and click the Run button in the row corresponding to Version 0 of Jupyter Notebook.
  2. Fill in the following fields within the new modal window:
    1. Input Datasets: Select any Dataset you would like
    2. Output Dataset Name Suffix: _output
    3. Idle Timeout (seconds): Do not modify. The default value will work fine
    4. Max Duration (seconds): Do not modify. The default value will work fine here 
    5. Run Parameters (JSON, optional): Leave blank
  3. Once you have completed entering all the details of your Code Run , click the Run button to send your code to be run on your Rhino Health client.
    • Note: In collaborative projects with remote sites, you will only be able to run Interactive Containers on remote data after the site has granted you permission via the FCP Secure Access mechanism.

Connecting to your Interactive Session of Jupyter Notebook

  1. Click the Code Runs menu item within the left-hand navigation menu.
  2. On the Code Runs page, you should now see a new Code Run entitled Jupyter Notebook with a single row of type "IGC" and showing a status of "Running".  
  3. Connect to the active session by clicking the dashboard.rhinohealth.com_project_792210c7-ac2c-4a46-914d-799f884397ef_results 2.png icon.
    Tutorial 2 - Launching Jupyter Notebook.png
  4. A new tab will open with the interactive session. This tab is running your Interactive Container on the remote client:
    Interactive Container - Jupyter Notebook Session.png
  5. To open a Jupyter Notebook, simply click the desktop icon.
    • Note: You may see an error message indicating that there is no audio support. You can ignore this message.
  6. You can now create a new notebook and run it on the remote client. You can access the input Dataset files under the /input folder:
    Interactive Container - Exploring the Cohort Tabular Data.png
  7. For a pre-built example, open the notebook that has been included under the Desktop folder.
  8. Saving your output and importing it into your project on the FCP:
    1. Save any files you wish to import back into the FCP under /output:
      1. If you wish to import a tabular file, save it as /output/dataset.csv
      2. If you wish to import file data, follow these steps:
        1. Save your files under /output/file_data/
        2. After you have saved all your files, click the desktop icon named Create Output Dataset. This will run a local script that looks for files under /output/file_data/ and populates a new /output/dataset.csv file with the file names so they can be imported as a Dataset.
    2. When you terminate the interactive session, the FCP will attempt to import the output Dataset defined in the /output/dataset.csv file. If you did not create this file, the Code Run will show an “Error: failure to import dataset” message.

 

Thanks again for investing your time in learning how to use the FCP, we can't wait to see what you will do with it! If you need support at any time, feel free to contact support@rhinohealth.com.

Related to

Was this article helpful?