Connector

The Connector works by executing predefined crandas scripts on incoming HTTP requests. It performs the encryption and decryption of incoming and outgoing data required to communicate with the engine. If a script uploads data or opens results, those values pass through the Connector in plaintext, so it should be mosted accordingly. Moreover, the Connector holds the key material of some user (private key, connection file), so anyone with access to it can execute the scripts approved for that user.

To avoid the Connector from having access to plaintext values, it could be used in combination with the Web SDK. The Connector would then be limited to only perform the processing part of a script, while the SDK would be used to perform the initial uploading and final downloading of the data, where the encryption and decryption happen client side.

Example

As an example, consider the following crandas script that uploads a file.

import pandas as pd
import crandas as cd
from crandas.placeholders import Any
from fastapi import UploadFile

def run(name: str, csv: UploadFile, sep: str = ","):
  df = pd.read_csv(csv.file, sep=sep) 
  table = cd.upload_pandas_dataframe(df) 
  table.save(name=Any(name)) 
  return { 
     "num_rows": len(table), 
     "num_columns": len(table.columns) 
  }

It contains a single function `run` that will be executed. It takes three parameters: name, csv and sep. The csv should contain a csv file separated using sep and it will be saved with the name name. It returns the number of rows and columns in the uploaded dataset. It uses a placeholder for the name so it can be used to upload a table under an arbitrary name. Without it, the name has to be the same as during recording. A placeholder is not needed for the csv separator as this concerns local processing.

We save this script as upload.py in the Connector. This will create the following three endpoints in the connector:

- /upload/dummy: test the script on dummy data in the design environment.

- /upload/record: make a recording on dummy data, which can be used to get approval for production data.

- /upload/prod: run the script on production data in the authorized environment.

Assuming lookup-dummy.csv contains a basic csv file with dummy data and the Connector is hosted at localhost:8000, uploading it can be done using curl:

$ curl 'http://localhost:8000/upload/dummy?name=lookup' -F 'csv=@lookup-dummy.csv;type=text/csv'

{"num_rows":10,"num_columns":2}