Security - saeed349/quant_infra GitHub Wiki

image

Security is a paramount concern, as illustrated by my previous experience with inadvertently leaving services open while working on MBATS Cloud. During that incident, I employed Cloudflare to monitor inbound traffic, revealing guests from various regions, including Russia and that experience was a real eye-opener for me.

While acknowledging that network security is a broad topic and not my area of expertise and for organizations, its critical to set up a Virtual Private Cloud (VPC) on AWS accounts, implementing identity and access management solutions, Single Sign-On (SSO), deploying a robust firewall, and defining both outbound and inbound rules for the VPC. In the context of my personal deployment, I decided to not go with the aforementioned infrastructure to save cost reduce complexity but I have implemented several cheaper measures in this project based on my experience and the best practices observed in other organizations to ensure the projects integrity and below are a few.

  • I implement role-based access control on AWS Identity and Access Management (IAM), attaching policies exclusively to roles rather than users, this helps me with creating and managing users based on the function. These AWS usernames and keys from IAM are used to generate credentials for users on platforms beyond AWS, such as Snowflake. While it would be ideal to manage credentials centrally, the cost implications, especially on the cheapest tier of services, dictate my current approach. For example with Snowflake if I need enterprise level security, I need to pay 3$ per credit compared to the 2$ that I am currently paying.
  • Platforms like Airflow and Dash host web user interfaces (UIs) on AWS EC2, requiring open ports for access. To enhance security, I enable authentication on these platforms. Recognizing the potential vulnerabilities of such authentications, especially when inbounds are open to all IPs, I diligently manage this by updating the Security Group (SG) of the EC2 machines to restrict access to specific IPs.
  • The most secure method for accessing these web UIs is through port forwarding after SSHing into the machine (VS-code seamlessly facilitates this on remote machine access). This practice is shared among myself and all the developers I collaborate with.
  • Addressing the challenge of users needing access with dynamic IPs from their ISPs, I leverage AWS Lambda to update the SG of these EC2 machines. Furthermore, these Lambdas are Two-Factor Authentication (2FA) enabled, using the IAM secret key as the key, streamlining user management.