Setting up the Client
You can download the Client using docker. To download the client, use from the command line:
docker pull quay.io/ucsc_cgl/core-client:1.0.1
Getting the token
If you don't have an access token, you can contact Brian O'Connor and request one.
Perform Bulk Download via Manifest
This assumes the current working directory (pwd) has a manifest, like the ones you can download from the file browser. The command below will then download the files to the current working directory.
NOTE: make sure you have enough space in pwd!
docker run --rm -e ACCESS_TOKEN=<access_token> -e REDWOOD_ENDPOINT=storage.ucsc-cgl.org -v `pwd`:/dcc/data quay.io/ucsc_cgl/core-client:1.0.1 redwood-download /dcc/data/manifest.tsv /dcc/data/
This will download the data in the current directory.
Upload via Spinnaker
Create a manifest that links your metdata and data. Your manifest.tsv should be a TSV based on this template.
You need to include file paths to your upload files that start with /dcc/data since that's the location used in the docker run below.
You should create a directory where you want to have your files for upload (assumed to be pwd), place your manifest.tsv in this directory along with all your files for upload, and then execute the following:
docker run --rm -e ACCESS_TOKEN=<access_token> -e REDWOOD_ENDPOINT=storage.ucsc-cgl.org -v `pwd`:/dcc/data quay.io/ucsc_cgl/core-client:1.0.1 spinnaker-upload /dcc/data/manifest.tsv
Once completed, you will find a receipt file (spinnaker/output_metadata/receipt.tsv) which you should save. It provides various IDs assigned to your donor, specimen, sample and file that make it much easier to find/audit later.
NOTE: Uploads can take a long time and our feedback on the command line needs to be improved. You may use dstat to monitor network usage to ensure uploads are in progress.
You can get the latest Metadata dump by clicking here.