SFTP Management - NYCPlanning/data-engineering GitHub Wiki

🔐 Working with SFTP Servers

Connecting to an SFTP server involves secure authentication using either:

1. SSH Keys (Preferred)

SSH key-based authentication is the most common method for automated and secure access.

  • A private key stays on your local machine (client). It should have 600 permissions.
  • Its matching public key is placed on the server.
  • During connection, the server checks that the private key matches the stored public key.

2. Password Authentication (Less Common)

You can also connect using a text-based password, but this is:

  • Less secure (especially for automated scripts)
  • More vulnerable to brute-force attacks
  • Typically disabled on production SFTP servers

SSH keys are preferred because they’re more secure, scriptable, and resistant to password-based attacks.

🧭 Host Keys vs Authentication Keys

Key Type Purpose Stored On Verified By
Authentication Key Verifies the user Private on client, public on server Server verifies user
Host Key Verifies the server Private on server, public on client (known_hosts) Client verifies server
  • Both types use public-key cryptography.
  • Private and public keys are matched during connection.
  • If a key mismatch occurs, the connection is rejected.

📦 What You Need to Connect

To connect securely:

  • A private authentication key on your local machine

  • A known_hosts file on your machine containing:

    • The server's hostname/IP
    • Host key type (e.g., RSA, ECDSA)
    • The server’s public host key

We store all SSH keys and the future known_hosts file in 1Password.

⚠️ Host Key Warnings

If you see: WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!

  • Stop immediately — this may indicate a man-in-the-middle (MITM) attack.
  • Contact the server administrator to confirm if the host key was changed intentionally.
  • Only proceed once you've received and verified the new host key.
  • Update your known_hosts file accordingly.

🛠️ Useful Commands

  • Connect to a Server (Using SSH Key)

    sftp -i <path/to/private_key> \
   -o StrictHostKeyChecking=yes \
   -o UserKnownHostsFile=<path/to/known_hosts/file> \
   <user@hostname>

  • Check Server Host Keys (Non-interactive)

    ssh-keyscan <hostname>

  • Generate SSH Key Pair (Private + Public)

    ssh-keygen -t <encryption_type> -f ~/.ssh/my_key

    This will create:

    • ~/.ssh/my_key (private key — keep safe!)
    • ~/.ssh/my_key.pub (public key — share with server admin)
⚠️ **GitHub.com Fallback** ⚠️