AWS Glue Lambda Trigger Issue :

If one is running into an issue where AWS Lambda failed to trigger Glue job or it triggers the Glue job for about 50% of the time only . You need to check the network configuration of the AWS lambda properly.

Primarily the issue encountered is AWS lambda will have multiple subnets: Example subnet-1 and subnet-2,subnet-3 and so on while your glue job locates in subnet-1 only.

As lambda routes traffic randomly to one of these subnets, it fails to connect to Glue when traffic is routed to subnet-2 or subnet-3, and ended up with timeout.

To resolve this issue, one needs to remove the subnets which are not relevant for the AWS glue subnet used from the VPC setting of your lambda function.