Download RAWDATA and ScipionProject - fjchichon/Guides GitHub Wiki

User Guide for Starting Data Transfer

through secure rsync

using IRODS (NEW)

How to Download Data Using rsync

We will need a file from the computer where you plan to store the data to initiate the data transfer.

1. From a Linux or MAC machine, in the user’s home folder:

1.1 Send us the id_rsa.pub file via email (located in the /home/user/.ssh folder). ⚠️ Warning it is a hidden folder ls -la will reveal the folder.

1.2 If the .ssh folder does not exist, run the following command:

ssh-keygen

[!NOTE] Please. Send the file attached. Copy and paste the text inside should create problems.

2. Preparing the Transfer

Once we receive your file, we’ll prepare the transfer on our end. You will be given access ONLY for your computer/user to connect to the hilbert-cuatro.cnb.csic.es server using rsync through port 2222 to your designated folder.

3. Python Script Delivery

We will send you a Python script as this remoteTransfer_DATE_USER_ID.py.

[!NOTE] Normally, the script is sent attached to a email. The .py extension could create problems with filters. Be sure Python is active in your machine

The script contains a recursive secure rsync rsync --progress -rlktvvv --exclude tmp -e 'ssh -p 2222' [email protected]:.

4. Executing the Script

4.1 Save the script directly in the folder where you want to download the data.

💡 Tip You can use an external HDD by plugging it into your machine and running the script.

4.2 Change script permissions:

chmod +x remoteTransfer_DATE_USER_ID.py

4.3 Run the script

./remoteTransfer_DATE_USER_ID.py

5. Troubleshooting

If an error appears or a password prompt occurs:

⚠️ Warning: Ensure you’re running the script on the same machine/user from which you sent the ID.

⚠️ Warning: Confirm that port 2222 is open at your location for hilbert-cuatro.cnb.csic.es. Contact your IT team if necessary.

6. Tips

💡 Tip Reading the script, you will notice that it downloads RAW-DATA and the Scipion Project separately.

edit the script and add --exclude data if you only want to copy the Scipion Project

edit the script and add --exclude project if you only want to copy the RawData

How to Download Data Using iRODS

Requirements

  • Network:
    • Open outgoing TCP ports: 1247 and 20000-20199
  • Python:
    • Python must be installed

Linux Users

Debian-based systems:

apt update
apt install python3
apt install python3-pip

Red Hat-based systems:

dnf check-update
dnf install python3
dnf install python3-pip

Windows Users

Instructions

Linux Users

  1. Open a bash terminal.

  2. Execute the command provided by the facility operator. It will look like this:

    curl -sSfL "https://raw.githubusercontent.com/FragmentScreen/fandanGO-cryoem-cnb/main/cryoemcnb/utils/irods_fetch_unix.sh" | bash -s -- --host "hilbert-cuatro.cnb.csic.es" --collection "/cnbZone/home/anonymous/acquisition_name" --ticket "xxxxxxxxxxxxxxx"
    • Replace /cnbZone/home/anonymous/acquisition_name and xxxxxxxxxxxxxxx with your assigned collection path and ticket.

Windows Users

  1. Open a PowerShell terminal.

  2. Execute the command provided by the facility operator. It will look like this:

    $scriptPath = "$(Get-Location)\irods_fetch_win.ps1";
(Invoke-WebRequest -UseBasicParsing "https://raw.githubusercontent.com/FragmentScreen/fandanGO-cryoem-cnb/refs/heads/main/cryoemcnb/utils/irods_fetch_win.ps1").Content | Out-File $scriptPath -Encoding UTF8;
& powershell -ExecutionPolicy Bypass -File $scriptPath --host "hilbert-cuatro.cnb.csic.es" --collection "/cnbZone/home/anonymous/acquisition_name" --ticket "xxxxxxxxxxxxxxx";
Remove-Item $scriptPath
    • Replace /cnbZone/home/anonymous/acquisition_name and xxxxxxxxxxxxxxx with your assigned collection path and ticket.

Note:
iRODS (Integrated Rule-Oriented Data System) is a scalable, open-source data management system used for organizing, sharing, and preserving large datasets, supporting flexible workflows and user-defined rules[6][10].

If you have any issues, ensure your firewall allows the required ports and that Python is correctly installed.

Citations: [1] https://euangoddard.github.io/clipboard2markdown/ [2] https://www.reddit.com/r/ObsidianMD/comments/10itk7m/converting_other_file_types_to_markdown_how_to/ [3] https://www.markdownguide.org/getting-started/ [4] https://www.rich-text-to-markdown.com/ [5] https://codebeautify.org/html-to-markdown [6] https://www.archives.gov/applied-research/software-downloads/irods.html [7] https://rdm-docs.icts.kuleuven.be/mango/clients/python_client.html [8] https://irods.org/uploads/2012/08/irods-admin.pdf [9] https://tabletomarkdown.com/convert-spreadsheet-to-markdown/ [10] https://irods.org/ [11] https://docs.vscentrum.be/data/tier1data/clients/python_client.html

Respuesta de Perplexity: pplx.ai/share