Syncing files between our computers and RAIS - jamiefogel/Networks GitHub Wiki

Setting up shared directories

It is possible to make a local folder accessible on the IPEA remote desktop using Windows Remote Desktop. I did this by clicking on the pencil icon in the lower left of the connection: Screenshot 2023-07-31 at 9 54 43 AM and then choosing the Folders tab and adding a connection by clicking on the +sign in the bottom left: Screenshot 2023-07-31 at 9 55 30 AM

Workflow

In my Windows Remote Desktop I have made two of our Dropbox folders accessible on the IPEA remote desktop: Networks/RAIS_exports and Networks/RAIS_imports. The idea is that anything we want to transfer to IPEA should be saved in RAIS_imports and anything we want to transfer from IPEA we should save in RAIS_exports on the IPEA remote desktop and then draw from that folder on our own computers.

We currently have a bit of a disconnect/redundancy between our Networks git repository and Dropbox. I think we need both because we need git for code and Dropbox for data, but ideally they would be more integrated. As a first pass at this, I did the following:

  • Created a Data folder within my local clone of the Networks repo
  • Within Data created a symbolic link to the RAIS_exports directory on Dropbox: ln -s "/Users/jfogel/Opportunity Insights Dropbox/Jamie Fogel/Networks/RAIS_exports/" "/Users/jfogel/NetworksGit/Data/"
  • Going forward I will exclusively draw from this directory when doing the MLE, etc. on my laptop and I will save intermediate files on my laptop and/or Dropbox. That way we can separate "raw" files from "processed" ones and ensure that our results are reproducible from the "raw".

There are currently about 10 different directories within RAIS_exports, some of which probably need to be cleaned up as we move towards a more integrated workflow. What is most relevant in the short-run is I am placing the earnings panels we will use for the MLE in RAIS_exports/earnings_panels, which is where the older Rio 2009--2012 earnings panels are also stored.

Unfortunately, I currently don't know how to automatically transfer files from the IPEA python server to RAIS_exports. A possible solution would be to make one of the directories that is synced between python and Windows accessible on my Mac, but I worry that effectively automatically exporting files from IPEA is a bad idea.