feat(job): implement custom environment variables (--env flag) support

[Neelabh94]

• New --env Flags: Added a new --env flag to the 'gcluster job submit' command to allow users to pass custom environment variables to their workload containers. Also, introduced pathways specific new environment flags in cmd/job/submit.go:

  • --pathways-proxy-env
  • --pathways-server-env
  • --pathways-worker-env

• Validation Logic: Implemented validation for environment variable keys to ensure they adhere to POSIX naming conventions (letters, numbers, and underscores, without starting with a digit).

• Template Updates: Updated GKE and Pathways JobSet templates to inject the provided environment variables into the container specifications.

• Testing and Documentation: Added comprehensive unit tests for flag parsing and validation, and updated the job guide documentation with usage examples.

Submission Checklist

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

• Fork your PR branch from the Toolkit "develop" branch (not main)
• Test all changes with pre-commit in a local branch #
• Confirm that "make tests" passes all tests
• Add or modify unit tests to cover code changes
• Ensure that unit test coverage remains above 80%
• Update all applicable documentation
• Follow Cluster Toolkit Contribution guidelines #

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enables greater flexibility for job submissions by allowing users to define custom environment variables via the --env flag. The changes include necessary updates to the CLI command, orchestration logic, and Kubernetes manifest templates, ensuring that environment variables are correctly propagated to the target containers while maintaining strict naming validation.

Highlights

• New --env Flag: Added a new --env flag to the 'gcluster job submit' command to allow users to pass custom environment variables to their workload containers.
• Validation Logic: Implemented validation for environment variable keys to ensure they adhere to POSIX naming conventions (letters, numbers, and underscores, without starting with a digit).
• Template Updates: Updated GKE and Pathways JobSet templates to inject the provided environment variables into the container specifications.
• Testing and Documentation: Added comprehensive unit tests for flag parsing and validation, and updated the job guide documentation with usage examples.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini or @gemini-code-assist . Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for passing custom environment variables to workload containers via a new --env flag in the gcluster job submit command. It includes POSIX-compliant validation for environment variable keys, updates the Kubernetes JobSet and Pathways templates to inject these variables, and adds comprehensive unit tests and documentation. The reviewer's feedback focuses on ensuring deterministic manifest generation by converting the environment variables map into a sorted slice of structs before rendering the templates, preventing randomized ordering in the generated Kubernetes manifests.

[Neelabh94]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for custom environment variables in the 'gcluster job submit' command via a new '--env' flag. The changes include command-line flag parsing and validation, updates to GKE and Pathways JobSet templates to inject the environment variables, comprehensive unit tests, and updated documentation. The review feedback recommends minor performance optimizations to avoid unnecessary map and slice allocations by returning early when no environment variables are provided.

[Neelabh94]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for passing custom environment variables to workload containers via a new --env flag in the gcluster job submit command. It includes input validation to ensure environment variable keys conform to POSIX naming standards, updates the GKE and Pathways JobSet templates to inject these variables, and adds comprehensive unit tests and documentation. I have no feedback to provide as there are no review comments to evaluate.

[Neelabh94]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for passing custom environment variables to workload containers via a new --env flag in the gcluster job submit command, complete with POSIX-compliant validation, template updates for GKE and Pathways, unit tests, and documentation. The review feedback correctly points out that for Pathways jobs, these environment variables must also be propagated to the pathways-worker container in the worker replicatedJob to ensure critical JAX/XLA configurations are not ignored during TPU execution.

[Neelabh94]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for passing custom environment variables to workload containers via a new --env flag in the gcluster job submit command, complete with validation, template updates for JobSet and Pathways, documentation, and unit tests. The review feedback suggests improving maintainability and testability by passing the environment variables as an argument to the validation function instead of relying on a package-level global variable, and optimizing slice allocations in the GKE orchestrator by pre-allocating slices with known capacities.

[Neelabh94]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for passing custom environment variables to workload containers via a new --env flag in the gcluster job submit command. It includes robust validation of environment variable keys conforming to POSIX naming rules, updates to both standard GKE and Pathways JobSet templates to inject these variables, comprehensive unit tests covering validation and manifest generation, and updated documentation. The implementation is clean, well-tested, and well-documented, so I have no further feedback to provide.

[Neelabh94]

/gemini review

[Neelabh94]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for passing custom environment variables to standard and Pathways workload containers using new CLI flags. The changes span CLI flag parsing, validation, unit tests, documentation, and template rendering. Feedback on the changes includes: updating validation logic to check the new pathways environment variable slices to prevent validation bypass, resetting the new global environment variable slices in tests to avoid test pollution, and refactoring redundant anonymous functions used for sorting and converting environment variables into a single helper function.

[Neelabh94]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for passing custom environment variables to workload containers in GKE JobSets, including standard workloads and Pathways components (proxy, server, and worker). It introduces new CLI flags, validation logic for environment variable keys, template updates to inject these variables into Kubernetes manifests, and comprehensive unit tests. The feedback suggests defining a named EnvVar struct type instead of repeating anonymous struct definitions in gke_job_orchestrator.go and types.go to improve code readability, maintainability, and type safety.

[Neelabh94]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for passing custom environment variables to both standard and Pathways workload containers via new CLI flags (--env, --pathways-proxy-env, --pathways-server-env, and --pathways-worker-env). It includes validation to ensure environment variable keys conform to POSIX naming rules, updates the GKE JobSet templates to inject these variables, and adds comprehensive unit tests and documentation. Feedback suggests refactoring the sequential validation calls of the environment flags into a single loop to improve maintainability and reduce repetition.

[Neelabh94]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for passing custom environment variables to workload containers and Pathways components (proxy, server, and worker) via new CLI flags: --env, --pathways-proxy-env, --pathways-server-env, and --pathways-worker-env. It includes validation logic to ensure environment variable keys conform to POSIX naming rules, updates the GKE manifest templates to inject these variables, and adds comprehensive unit tests and documentation. The review feedback recommends adding documentation comments to the newly introduced exported EnvVar struct and its fields in pkg/orchestrator/gke/types.go to align with Go best practices and the repository style guide.

[Neelabh94]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for passing custom environment variables to workload and Pathways containers (proxy, server, and worker) via new CLI flags (--env, --pathways-proxy-env, --pathways-server-env, and --pathways-worker-env). It includes validation to ensure keys conform to POSIX naming rules, updates the GKE and Pathways JobSet templates to inject these variables, adds comprehensive unit tests, and updates the documentation with examples. There are no review comments to address, and the implementation is clean and well-tested.

[SwarnaBharathiMantena]

LGTM