Finding and Fixing Login Shell Configuration for Role Accounts on Hera

Fantastic discovery! Now we know why GitLab CI jobs are failing differently on Hera - the role account always runs with login_shell=off regardless of what we try through GitLab CI.

Tracing Default Shell Behavior for Role Accounts

Here's how to trace down where and why the Hera admins have configured role accounts to always use non-login shells:

1. Check System Shell Initialization Files

# Check how bash is invoked for the role account
ls -la /bin/bash
file /bin/bash
getfacl /bin/bash

# Check if there's a wrapper script or alias
which bash
type bash

# Check for custom PAM configurations
ls -la /etc/pam.d/
grep -r "role.glopara" /etc/pam.d/

2. Check Account-Specific Configurations

# Check how the role account is defined
grep "role.glopara" /etc/passwd

# Check if there are special flags for this account
grep -r "role.glopara" /etc/security/

3. Check for Shell Wrapper Scripts

# Look for wrapper scripts that might be modifying shell behavior
find /etc -type f -name "*bash*" -o -name "*shell*" | xargs grep -l "role"

# Check system profile scripts
grep -r "login_shell" /etc/profile.d/

4. Check GitLab Runner Configuration

# Check how GitLab Runner is configured to invoke shells
cat ~/.gitlab-runner/config.toml | grep -A 10 shell

5. Test Different Shell Invocation Methods

# Try different ways of launching bash to find one that works as login shell
bash --login -c 'shopt | grep login_shell'
/bin/bash -l -c 'shopt | grep login_shell'
bash --login -i -c 'shopt | grep login_shell'

Solutions to Recommend to the Admins

Based on your findings, here are the most likely fixes:

Option 1: Modify the GitLab Runner Configuration

Have the admins modify the runner's configuration to use login shells:

[runners](/TerrenceMcGuinness-NOAA/global-workflow/wiki/runners)
  name = "hera"
  # Other configuration...
  shell = "bash --login"

Option 2: Create a Custom Shell Wrapper

Have admins create a wrapper script specifically for GitLab:

#!/bin/bash
# /usr/local/bin/gitlab-runner-shell
exec /bin/bash --login "$@"

Then in the GitLab Runner configuration:

[runners](/TerrenceMcGuinness-NOAA/global-workflow/wiki/runners)
  name = "hera"
  # Other configuration...
  shell = "/usr/local/bin/gitlab-runner-shell"

Option 3: Modify Role Account Shell in /etc/passwd

If appropriate, they could change the default shell for the role account:

# From:
role.glopara:x:1000:1000::/home/role.glopara:/bin/bash

# To:
role.glopara:x:1000:1000::/home/role.glopara:/bin/bash --login

Option 4: Create a .bashrc Hook for GitLab

# Add to role.glopara's .bashrc
if [ -n "$GITLAB_CI" ](/TerrenceMcGuinness-NOAA/global-workflow/wiki/--n-"$GITLAB_CI"-); then
  # Force login shell behavior for GitLab CI
  shopt -s login_shell
fi

Additional Technical Context for the Admins

Explain to the admins that:

1. The non-login shell is causing GitLab Runner to ignore exit codes on Hera
2. GAEAC6 works correctly because it runs with login_shell=on
3. This behavior difference causes inconsistent CI/CD results between systems
4. A proper login shell will make error detection reliable across all systems

This comprehensive approach should help the admins understand the issue and implement the right fix for your environment.