deployment.container_manager#

Container Management and Service Orchestration System.

This module provides comprehensive container orchestration capabilities for the deployment framework, handling service discovery, template rendering, build directory management, and Podman Compose integration. The system supports hierarchical service configurations with framework and application-specific services that can be independently deployed and managed.

The container manager implements a sophisticated template processing pipeline that converts Jinja2 templates into Docker Compose files, copies necessary source code and configuration files, and orchestrates multi-service deployments through Podman Compose with proper networking and dependency management.

Key Features:

  • Hierarchical service discovery (framework.service, applications.app.service)

  • Jinja2 template rendering with configuration context

  • Intelligent build directory management with selective file copying

  • Environment variable expansion and configuration flattening

  • Podman Compose orchestration with multi-file support

  • Kernel template processing for Jupyter notebook environments

Architecture:

The system supports two service categories:

  1. Framework Services: Core infrastructure services like databases, web interfaces, and development tools (jupyter, open-webui, pipelines)

  2. Application Services: Domain-specific services tied to particular applications (als_expert.mongo, als_expert.pv_finder)

Examples

Basic service deployment:

$ python container_manager.py config.yml up -d
# Deploys all services listed in deployed_services configuration

Service discovery patterns:

# Framework service (short name)
deployed_services: ["jupyter", "pipelines"]

# Framework service (full path)
deployed_services: ["framework.jupyter", "framework.pipelines"]

# Application service (full path required)
deployed_services: ["applications.als_expert.mongo"]

Template rendering workflow:

1. Load configuration with imports and merging
2. Discover services listed in deployed_services
3. Process Jinja2 templates with configuration context
4. Copy source code and additional directories as specified
5. Flatten configuration files for container consumption
6. Execute Podman Compose with generated files

See also

deployment.loader : Configuration loading system used by this module configs.config.ConfigBuilder : Configuration management find_service_config() : Service discovery implementation render_template() : Template processing engine

container_manager.find_service_config(config, service_name)[source]#

Locate service configuration and template path for deployment.

This function implements the service discovery logic for the container management system, supporting both hierarchical service naming (full paths) and legacy short names for backward compatibility. The system searches through framework services and application-specific services to find the requested service configuration.

Service naming supports three patterns: 1. Framework services: “framework.service_name” or just “service_name” 2. Application services: “applications.app_name.service_name” 3. Legacy services: “service_name” (deprecated, for backward compatibility)

The function returns both the service configuration object and the path to the Docker Compose template, enabling the caller to access service settings and initiate template rendering.

Parameters:

  • config (dict) – Configuration containing service definitions

  • service_name (str) – Service identifier (short name or full dotted path)

Returns:

Tuple containing service configuration and template path, or (None, None) if service not found

Return type:

tuple[dict, str] or tuple[None, None]

Examples

Framework service discovery:

>>> config = {'framework': {'services': {'jupyter': {'path': 'services/framework/jupyter'}}}}
>>> service_config, template_path = find_service_config(config, 'framework.jupyter')
>>> print(template_path)  # 'services/framework/jupyter/docker-compose.yml.j2'

Application service discovery:

>>> config = {'applications': {'als_expert': {'services': {'mongo': {'path': 'services/applications/als_expert/mongo'}}}}}
>>> service_config, template_path = find_service_config(config, 'applications.als_expert.mongo')
>>> print(template_path)  # 'services/applications/als_expert/mongo/docker-compose.yml.j2'

Legacy service discovery:

>>> config = {'services': {'legacy_service': {'path': 'services/legacy'}}}
>>> service_config, template_path = find_service_config(config, 'legacy_service')
>>> print(template_path)  # 'services/legacy/docker-compose.yml.j2'

Note

Legacy service support (services.* configuration) is deprecated and will be removed in future versions. Use framework.* or applications.* naming patterns for new services.

See also

get_templates() : Uses this function to build template lists setup_build_dir() : Processes discovered services for deployment

container_manager.get_templates(config)[source]#

Collect template paths for all deployed services in the configuration.

This function builds a comprehensive list of Docker Compose template paths based on the services specified in the deployed_services configuration. It processes both the root services template and individual service templates, providing the complete set of templates needed for deployment.

The function always includes the root services template (services/docker-compose.yml.j2) which defines the shared network configuration and other global service settings. Individual service templates are then discovered through the service discovery system and added to the template list.

Parameters:

config (dict) – Configuration containing deployed_services list

Returns:

List of template file paths for processing

Return type:

list[str]

Raises:

Warning – Prints warning if deployed_services is not configured

Examples

Template collection for mixed services:

>>> config = {
...     'deployed_services': ['framework.jupyter', 'applications.als_expert.mongo'],
...     'framework': {'services': {'jupyter': {'path': 'services/framework/jupyter'}}},
...     'applications': {'als_expert': {'services': {'mongo': {'path': 'services/applications/als_expert/mongo'}}}}
... }
>>> templates = get_templates(config)
>>> print(templates)
['services/docker-compose.yml.j2',
 'services/framework/jupyter/docker-compose.yml.j2',
 'services/applications/als_expert/mongo/docker-compose.yml.j2']

Warning

If deployed_services is not configured or empty, only the root services template will be returned, which may not provide functional services.

See also

find_service_config() : Service discovery used by this function render_template() : Processes the templates returned by this function

container_manager.render_template(template_path, config, out_dir)[source]#

Render Jinja2 template with configuration context to output directory.

This function processes Jinja2 templates using the configuration as context, generating concrete configuration files for container deployment. The system supports multiple template types including Docker Compose files and Jupyter kernel configurations, with intelligent output filename detection.

Template rendering uses the complete configuration dictionary as Jinja2 context, enabling templates to access any configuration value including environment variables, service settings, and application-specific parameters. Environment variables can be referenced directly in templates using ${VAR_NAME} syntax for deployment-specific configurations like proxy settings. The output directory is created automatically if it doesn’t exist.

Parameters:

  • template_path (str) – Path to the Jinja2 template file to render

  • config (dict) – Configuration dictionary to use as template context

  • out_dir (str) – Output directory for the rendered file

Returns:

Full path to the rendered output file

Return type:

str

Examples

Docker Compose template rendering:

>>> config = {'database': {'host': 'localhost', 'port': 5432}}
>>> output_path = render_template(
...     'services/mongo/docker-compose.yml.j2',
...     config,
...     'build/services/mongo'
... )
>>> print(output_path)  # 'build/services/mongo/docker-compose.yml'

Jupyter kernel template rendering:

>>> config = {'project_root': '/home/user/project'}
>>> output_path = render_template(
...     'services/jupyter/python3-epics/kernel.json.j2',
...     config,
...     'build/services/jupyter/python3-epics'
... )
>>> print(output_path)  # 'build/services/jupyter/python3-epics/kernel.json'

Note

The function automatically determines output filenames based on template naming conventions: .j2 extension is removed, and specific patterns like docker-compose.yml.j2 and kernel.json.j2 are recognized.

See also

setup_build_dir() : Uses this function for service template processing render_kernel_templates() : Batch processing of kernel templates

container_manager.render_kernel_templates(source_dir, config, out_dir)[source]#

Process all Jupyter kernel templates in a service directory.

This function provides batch processing for Jupyter kernel configuration templates, automatically discovering all kernel.json.j2 files within a service directory and rendering them with the current configuration context. This is particularly useful for Jupyter services that provide multiple kernel environments with different configurations.

The function recursively searches the source directory for kernel template files and processes each one, maintaining the relative directory structure in the output. This ensures that kernel configurations are placed in the correct locations for Jupyter to discover them.

Parameters:

  • source_dir (str) – Source directory to search for kernel templates

  • config (dict) – Configuration dictionary for template rendering

  • out_dir (str) – Base output directory for rendered kernel files

Examples

Kernel template processing for Jupyter service:

>>> # Source structure:
>>> # services/jupyter/
>>> #   ├── python3-epics-readonly/kernel.json.j2
>>> #   └── python3-epics-write/kernel.json.j2
>>>
>>> render_kernel_templates(
...     'services/jupyter',
...     {'project_root': '/home/user/project'},
...     'build/services/jupyter'
... )
>>> # Output structure:
>>> # build/services/jupyter/
>>> #   ├── python3-epics-readonly/kernel.json
>>> #   └── python3-epics-write/kernel.json

Note

This function is typically called automatically by setup_build_dir when a service configuration includes ‘render_kernel_templates: true’.

See also

render_template() : Core template rendering used by this function setup_build_dir() : Calls this function for kernel template processing

container_manager.setup_build_dir(template_path, config, container_cfg)[source]#

Create complete build environment for service deployment.

This function orchestrates the complete build directory setup process for a service, including template rendering, source code copying, configuration flattening, and additional directory management. It creates a self-contained build environment that contains everything needed for container deployment.

The build process follows these steps: 1. Create clean build directory for the service 2. Render the Docker Compose template with configuration context 3. Copy service-specific files (excluding templates) 4. Copy source code if requested (copy_src: true) 5. Copy additional directories as specified 6. Create flattened configuration file for container use 7. Process kernel templates if specified

Source code copying includes intelligent handling of requirements files, automatically copying global requirements.txt to the container source directory to ensure dependency management works correctly in containers.

Parameters:

  • template_path (str) – Path to the service’s Docker Compose template

  • config (dict) – Complete configuration dictionary for template rendering

  • container_cfg (dict) – Service-specific configuration settings

Returns:

Path to the rendered Docker Compose file

Return type:

str

Examples

Basic service build directory setup:

>>> container_cfg = {
...     'copy_src': True,
...     'additional_dirs': ['docs', 'scripts'],
...     'render_kernel_templates': False
... }
>>> compose_path = setup_build_dir(
...     'services/framework/jupyter/docker-compose.yml.j2',
...     config,
...     container_cfg
... )
>>> print(compose_path)  # 'build/services/framework/jupyter/docker-compose.yml'

Advanced service with custom directory mapping:

>>> container_cfg = {
...     'copy_src': True,
...     'additional_dirs': [
...         'docs',  # Simple directory copy
...         {'src': 'external_data', 'dst': 'data'}  # Custom mapping
...     ],
...     'render_kernel_templates': True
... }
>>> compose_path = setup_build_dir(template_path, config, container_cfg)

Note

The function automatically handles build directory cleanup, removing existing directories to ensure clean builds. Global requirements.txt is automatically copied to container source directories when present.

Warning

This function performs destructive operations on build directories. Ensure build_dir is properly configured to avoid data loss.

See also

render_template() : Template rendering used by this function render_kernel_templates() : Kernel template processing configs.config.ConfigBuilder : Configuration flattening

container_manager.parse_args()[source]#

Parse command-line arguments for container management operations.

This function defines and processes the command-line interface for the container management system, supporting configuration file specification, deployment commands (up/down), and operational flags like detached mode.

The argument parser enforces logical constraints, such as requiring the ‘up’ command when using detached mode, and provides clear error messages for invalid argument combinations.

Returns:

Parsed command-line arguments

Return type:

argparse.Namespace

Raises:

SystemExit – If invalid argument combinations are provided

Command-line Interface:

python container_manager.py CONFIG [COMMAND] [OPTIONS]

Positional Arguments:

CONFIG: Path to the configuration file (required) COMMAND: Deployment command - ‘up’ or ‘down’ (optional)

Options:

-d, –detached: Run in detached mode (only with ‘up’)

Examples

Generate compose files only:

$ python container_manager.py config.yml
# Creates build directory and compose files without deployment

Deploy services in foreground:

$ python container_manager.py config.yml up
# Deploys services and shows output

Deploy services in background:

$ python container_manager.py config.yml up -d
# Deploys services in detached mode

Stop services:

$ python container_manager.py config.yml down
# Stops and removes deployed services

Clean deployment (remove images/volumes):

$ python container_manager.py config.yml clean
# Removes containers, images, volumes, and networks

Rebuild from scratch:

$ python container_manager.py config.yml rebuild -d
# Clean + rebuild + start in detached mode

See also

main execution block() : Uses parsed arguments for deployment operations

container_manager.clean_deployment(compose_files)[source]#

Clean up containers, images, volumes, and networks for a fresh deployment.

This function provides comprehensive cleanup capabilities for container deployments, removing containers, images, volumes, and networks to enable fresh rebuilds. It’s particularly useful when configuration changes require complete environment reconstruction.

Parameters:

compose_files (list[str]) – List of Docker Compose file paths for the deployment