This page will explain how to use external files in your code run. Your workgroup has a unique cloud storage (an S3 bucket on Rhino Cloud) to upload non-sensitive data that might be needed during code run, e.g., open-source large LLM model weights. 

NOTE ⚠️: This cloud storage (specified in `Workgroup s3 external file bucket name`, see below) is owned by Rhino Health, please DO NOT upload sensitive information such as proprietary datasets. For sensitive data, please use Creating a New Dataset or Dataset Version.

Overview

For your code to be able to access files while running in your agent we need do 4 steps:

1. Set up a bucket to upload your files

2. Upload files to your designated bucket

3. Reference your files in your code

4. Use your files in a specific run

Steps

Set up a bucket to upload your files

The bucket is created as part of the onboarding process into the FCP by client request, if you want to create a bucket please contact support@rhinohealth.com for more details. Once your bucket is setup up, you can find it in your profile page, see an example below:

Upload files to your designated bucket

AWS offers many alternative ways to upload files to an S3 bucket, we also provided a script in our user_resource repository you can use: upload-file-to-s3.sh.

You need to define your S3 credentials and then you can call the script like this:

./upload-file-to-s3.sh ./the_folder_to_upload name_of_the_bucket the_folder_in_s3_you_want_to_upload_to

For the example above, If you wanted to upload files in a local folder called "local_folder" and store them in a folder in FCP called "my_files", the command would be:

./upload-file-to-s3.sh ./local_folder external-files-rhino-health-dev my_files

Reference your files in your code

When creating your code object, you can reference your external files by referencing them in the following path:

/external_data/the_folder_in_s3_you_want_to_upload_to/filename

For the example above, assuming you uploaded `model_params.txt` file under `my_files`, the file would be accessible during runtime in

/external_data/my_files/model_params.txt

An example of Python code to read your file data into a `text` variable would look like as follows:

from pathlib import Path
     text = Path('/external_data/my_files/model_params.txt').read_text()

Use your files in a specific run

Once your code points to your external files, and the code object is created, you can run the code and specify the files you want to use in your code run. In the UI, you can simply select them from a dropdown

Alternatively, you can also use the SDK. See this notebook to use external files via SDK for more detail.