provisioning/README.md

# Provisioning

<div align="center">
  <img src="resources/images/provisioning_logo-light.svg" alt="Provisioning Logo" width="200"/>
  <br>
  <br>
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="resources/images/provisioning_logo-text-dark.svg">
    <source media="(prefers-color-scheme: light)" srcset="resources/images/provisioning_logo-text-light.svg">
    <img alt="Provisioning Logo Text" src="resources/images/provisioning_logo-text-light.svg" width="400">
  </picture>
</div>

A provisioning system built with Nushell and KCL for infrastructure automation and Kubernetes cluster management.


## Table of Contents

- [Overview](#overview)
- [Prerequisites](#prerequisites)
  - [System Requirements](#system-requirements)
  - [Required Tools](#required-tools)
  - [Environment Variables](#environment-variables)
- [Installation and Setup](#installation-and-setup)
  - [Installation Methods](#installation-methods)
  - [Installation Process](#installation-process)
  - [Post-Installation Verification](#post-installation-verification)
  - [Environment Configuration](#environment-configuration)
  - [Nushell Session](#nushell-session)
- [Basic Usage](#basic-usage)
  - [Main Commands](#main-commands)
  - [Server Management](#server-management)
  - [Task Services (Infrastructure Components)](#task-services-infrastructure-components)
  - [Cluster Operations](#cluster-operations)
  - [Infrastructure Generation](#infrastructure-generation)
- [Project Architecture](#project-architecture)
  - [High-Level Structure](#high-level-structure)
  - [Key Components](#key-components)
- [Common Workflows](#common-workflows)
  - [1. Set Up New Infrastructure](#1-set-up-new-infrastructure)
  - [2. Install Kubernetes](#2-install-kubernetes)
  - [3. Deploy Cluster Services](#3-deploy-cluster-services)
  - [4. SSH Server Access](#4-ssh-server-access)
- [Development and Debug Flags](#development-and-debug-flags)
- [Secret Management](#secret-management)
  - [Edit Encrypted Files](#edit-encrypted-files)
  - [Age Configuration](#age-configuration)
- [File Structure Patterns](#file-structure-patterns)
- [Troubleshooting](#troubleshooting)
  - [Installation Issues](#installation-issues)
  - [Common Runtime Issues](#common-runtime-issues)
  - [Debug Commands](#debug-commands)
- [Contributing](#contributing)
  - [Nushell Plugins](#nushell-plugins)
- [License](#license)
- [Support](#support)

## Overview

This system provides a complete Infrastructure as Code (IaC) platform that enables provisioning, configuration, and management of infrastructure across multiple cloud providers with a strong focus on cloud-native Kubernetes deployments.

### Key Features

- **Multi-provider**: Support for AWS, UpCloud, and local environments
- **Infrastructure as Code**: Declarative configuration using KCL
- **Secret Management**: Integration with SOPS and Age for encryption
- **Modular**: Component-based reusable architecture
- **Kubernetes-first**: Designed for cloud-native deployments

## Prerequisites

### System Requirements

- **Operating System**: Linux, macOS, or Windows (WSL2)
- **Architecture**: x86_64 or ARM64
- **Memory**: Minimum 4GB RAM (8GB recommended)
- **Storage**: At least 2GB free space
- **Network**: Internet connection for downloading dependencies

### Required Tools

The installation process will automatically install these tools if not present:

- **Nushell 0.105.2** - Primary shell and scripting language
- **KCL 0.11.2** - Configuration language for infrastructure (CLI required + optional nu_plugin_kcl)
- **nu_plugin_tera** - Native Nushell plugin for template rendering
- **SOPS 3.10.2** - Secret management
- **Age 1.2.1** - Encryption tool
- **K9s 0.50.6** - Kubernetes management interface

### Environment Variables

Set these environment variables before installation:

```bash
# Main provisioning directory (default: /usr/local/provisioning)
export PROVISIONING="/usr/local/provisioning"

# Path to cloud infrastructure configurations
export PROVISIONING_KLOUD_PATH="/path/to/your/infrastructure/configurations"

# Path to infrastructure definitions
export PROVISIONING_INFRA_PATH="/path/to/your/infrastructure/definitions"

# Path to SOPS configuration file
export PROVISIONING_SOPS="/path/to/your/sops/configuration"

# Path to Age encryption key file
export PROVISIONING_KAGE="/path/to/your/age/key/file"
```

## Installation and Setup

### Installation Methods

#### Method 1: Pre-built Package (Recommended)

Download the pre-built package and install:

```bash
# Download the provisioning package (replace with actual download link)
wget https://releases.example.com/provisioning.tar.gz

# Extract the package
tar xzf provisioning.tar.gz

# Run the installation script
sudo ./install-provisioning
```

#### Method 2: Install from Source

For development or custom installations:

```bash
# Clone the repository
git clone <repository-url>
cd provisioning

# Install directly from repository
./distro/from-repo.sh
```

#### Method 3: Package Installation Script

If you have access to the source and want to install from a generated package:

```bash
# Requires SRC_DEVS environment variable to be set
./distro/pack-install.sh
```

### Installation Process

The `install-provisioning` script performs these steps automatically:

1. **Core System Installation**
   - Installs to `/usr/local/provisioning` by default
   - Creates symlink at `/usr/local/bin/provisioning` for global access
   - Removes any previous installation
   - Sets up directory structure

2. **Dependency Installation**
   - Installs **Nushell 0.105.2** (primary shell)
   - Installs **KCL 0.11.2** (configuration language CLI - required for nu_plugin_kcl)
   - Installs **SOPS 3.10.2** (secrets management)
   - Installs **Age 1.2.1** (encryption)
   - Installs **K9s 0.50.6** (Kubernetes management)

3. **Configuration Setup**
   - Creates user configuration directories
   - Initializes Nushell configuration files
   - Sets up provisioning context and library loading

4. **Tool Configuration**
   - Configures tool paths and middleware
   - Installs and registers nu_plugin_tera and nu_plugin_kcl (requires KCL CLI)
   - Verifies tool versions and plugin compatibility
   - Sets up provider integrations

### Post-Installation Verification

After installation, verify your setup:

```bash
# Check installation
provisioning version

# Verify environment configuration
provisioning env

# Show all configuration details
provisioning allenv

# Test basic functionality
provisioning help
```

### Environment Configuration

Configure your environment after installation:

```bash
# Add to your shell profile (.bashrc, .zshrc, etc.)
export PROVISIONING="/usr/local/provisioning"
export PATH="$PROVISIONING/bin:$PATH"

# Reload your shell or source the profile
source ~/.bashrc  # or ~/.zshrc
```

### Nushell Session

Start working with the provisioning system:

```bash
# Start interactive Nushell session with provisioning library loaded
provisioning nu

# Or run specific commands
provisioning help
provisioning context
```

## Basic Usage

### Main Commands

```bash
# General help
./core/nulib/provisioning help

# Version information
./core/nulib/provisioning version

# Context management
./core/nulib/provisioning context
```

### Server Management

```bash
# Create new servers
./core/nulib/provisioning server create

# Delete servers
./core/nulib/provisioning server delete

# List all servers
./core/nulib/provisioning server list

# SSH to server
./core/nulib/provisioning server ssh <server>

# Show server pricing
./core/nulib/provisioning server price
```

### Task Services (Infrastructure Components)

```bash
# Install infrastructure service
./core/nulib/provisioning taskserv create <service>

# Remove infrastructure service
./core/nulib/provisioning taskserv delete <service>

# List available services
./core/nulib/provisioning taskserv list

# Generate service configuration
./core/nulib/provisioning taskserv generate <service>
```

### Cluster Operations

```bash
# Create Kubernetes cluster
./core/nulib/provisioning cluster create

# Delete cluster
./core/nulib/provisioning cluster delete

# List cluster components
./core/nulib/provisioning cluster list
```

### Infrastructure Generation

```bash
# Generate server configurations
./core/nulib/provisioning generate server

# Generate task service configurations
./core/nulib/provisioning generate taskserv

# Generate cluster configurations
./core/nulib/provisioning generate cluster

# Generate infrastructure configurations
./core/nulib/provisioning generate infra
```

## Project Architecture

### High-Level Structure

```
provisioning/
├── core/                 # Core system
├── providers/            # Cloud providers
├── taskservs/            # Infrastructure services
├── cluster/              # Cluster configurations
├── kcl/                  # Configuration language
├── templates/            # Templates and generation
├── generate/             # Generation scripts
└── resources/            # Project resources
```

### Key Components

#### 1. Core System (`/core/`)
- **Entry Point**: `core/nulib/provisioning` - CLI interface script
- **Library**: `core/nulib/lib_provisioning/` - Core provisioning library modules
- **Utilities**: `core/nulib/lib_provisioning/utils/` - Helper functions

#### 2. Providers (`/providers/`)
- **AWS**: `providers/aws/` - Amazon Web Services provider
- **UpCloud**: `providers/upcloud/` - UpCloud provider
- **Local**: `providers/local/` - Local development provider

#### 3. Task Services (`/taskservs/`)
Infrastructure services that can be deployed:
- **Kubernetes**: Core K8s installation
- **Container Runtimes**: containerd, cri-o, crun, runc, youki
- **Networking**: Cilium, CoreDNS, proxy/HAProxy
- **Storage**: Rook-Ceph, external-NFS, Mayastor
- **Container Registry**: OCI registry
- **Development Tools**: Gitea, PostgreSQL

#### 4. Cluster Configurations (`/cluster/`)
- **Buildkit**: Container build system
- **CI/CD**: ArgoCD, Tekton pipelines
- **Git**: Git server configurations
- **Web**: Web service deployments

## Common Workflows

### 1. Set Up New Infrastructure

```bash
# Generate new infrastructure
./core/nulib/provisioning generate infra --new <name>

# Create servers (check mode)
./core/nulib/provisioning server create --infra <name> --check

# Confirm creation
./core/nulib/provisioning server create --infra <name>
```

### 2. Install Kubernetes

```bash
# Install Kubernetes on infrastructure
./core/nulib/provisioning taskserv create kubernetes --infra <name>

# Verify installation
./core/nulib/provisioning taskserv list
```

### 3. Deploy Cluster Services

```bash
# Create cluster services
./core/nulib/provisioning cluster create --infra <name>

# List components
./core/nulib/provisioning cluster list
```

### 4. SSH Server Access

```bash
# SSH to specific server
./core/nulib/provisioning server ssh <hostname>
```

## Development and Debug Flags

- `--debug` or `-x` - Enable debug mode
- `--check` or `-c` - Check mode (no actual changes)
- `--yes` or `-y` - Auto-confirm operations
- `--wait` or `-w` - Wait for operations to complete
- `--infra` or `-i` - Specify infrastructure target
- `--out` - Output format (json, yaml, text)

## Secret Management

### Edit Encrypted Files

```bash
# Edit file with SOPS
./core/nulib/provisioning sops <file>
```

### Age Configuration

Ensure `PROVISIONING_KAGE` points to the correct Age key file.

## File Structure Patterns

- **KCL files** (`.k`): Infrastructure configuration and schemas
- **Nushell files** (`.nu`): Automation scripts and modules
- **Jinja2 templates** (`.j2`): Configuration file templates
- **TOML files** (`.toml`): Definitions and configurations
- **YAML files** (`.yaml`): Kubernetes manifests and settings

## Troubleshooting

### Installation Issues

1. **Installation fails with permission errors**
   ```bash
   # Ensure you run the installer with sudo
   sudo ./install-provisioning
   
   # Check if /usr/local is writable
   ls -la /usr/local/
   ```

2. **Package extraction fails**
   ```bash
   # Verify package integrity
   file provisioning.tar.gz
   
   # Extract with verbose output
   tar xzvf provisioning.tar.gz
   ```

3. **Nushell installation fails**
   ```bash
   # Check Nushell version after installation
   nu --version
   
   # If missing, the installer should handle this automatically
   # Contact support if this persists
   ```

4. **Missing dependencies**
   ```bash
   # Check tool versions
   provisioning version
   
   # Verify tool installation
   provisioning setup tools check all
   ```

5. **Environment variables not set**
   ```bash
   # Check current environment
   provisioning env
   
   # Set missing variables
   export PROVISIONING="/usr/local/provisioning"
   export PATH="$PROVISIONING/bin:$PATH"
   ```

### Common Runtime Issues

1. **Missing environment variables**
   ```bash
   provisioning env
   ```

2. **KCL compilation errors**
   ```bash
   kcl fmt file.k
   ```

3. **nu_plugin_kcl issues**
   ```bash
   # Check if KCL CLI is installed (required for nu_plugin_kcl)
   kcl version
   
   # Check if plugin is registered
   version | get installed_plugins | str contains "nu_plugin_kcl"
   ```

3. **SOPS/Age encryption issues**
   - Verify `PROVISIONING_KAGE` path and key file

4. **Provider authentication**
   - Verify cloud provider credentials and permissions

5. **Command not found**
   ```bash
   # Check if provisioning is in PATH
   which provisioning
   
   # Add to PATH if needed
   export PATH="/usr/local/bin:$PATH"
   ```

### Debug Commands

```bash
# Enable verbose logging
./core/nulib/provisioning --debug <command>

# View current configuration
./core/nulib/provisioning show settings

# List server status
./core/nulib/provisioning show servers

# Interactive shell with library loaded
./core/nulib/provisioning nu
```

## Contributing

This project uses a modular architecture where:
- **Providers** handle cloud-specific operations
- **Task Services** are reusable infrastructure components
- **Clusters** define complete deployment configurations
- **Core** provides unified CLI and library functions

### Nushell Plugins

Plugins are sourced from: https://repo.jesusperez.pro/jesus/nushell-plugins

## License

See LICENSE file for details.

## Support

For issues and questions, consult the documentation in `core/README.md` or use the built-in help commands.