provisioning/README.md

Provisioning

A provisioning system built with Nushell and KCL for infrastructure automation and Kubernetes cluster management.

Table of Contents

• Overview
• Prerequisites
  • System Requirements
  • Required Tools
  • Environment Variables
• Installation and Setup
  • Installation Methods
  • Installation Process
  • Post-Installation Verification
  • Environment Configuration
  • Nushell Session
• Basic Usage
  • Main Commands
  • Server Management
  • Task Services (Infrastructure Components)
  • Cluster Operations
  • Infrastructure Generation
• Project Architecture
  • High-Level Structure
  • Key Components
• Common Workflows
  • 1. Set Up New Infrastructure
  • 2. Install Kubernetes
  • 3. Deploy Cluster Services
  • 4. SSH Server Access
• Development and Debug Flags
• Secret Management
  • Edit Encrypted Files
  • Age Configuration
• File Structure Patterns
• Troubleshooting
  • Installation Issues
  • Common Runtime Issues
  • Debug Commands
• Contributing
  • Nushell Plugins
• License
• Support

Overview

This system provides a complete Infrastructure as Code (IaC) platform that enables provisioning, configuration, and management of infrastructure across multiple cloud providers with a strong focus on cloud-native Kubernetes deployments.

Key Features

• Multi-provider: Support for AWS, UpCloud, and local environments
• Infrastructure as Code: Declarative configuration using KCL
• Secret Management: Integration with SOPS and Age for encryption
• Modular: Component-based reusable architecture
• Kubernetes-first: Designed for cloud-native deployments

Prerequisites

System Requirements

• Operating System: Linux, macOS, or Windows (WSL2)
• Architecture: x86_64 or ARM64
• Memory: Minimum 4GB RAM (8GB recommended)
• Storage: At least 2GB free space
• Network: Internet connection for downloading dependencies

Required Tools

The installation process will automatically install these tools if not present:

• Nushell 0.105.2 - Primary shell and scripting language
• KCL 0.11.2 - Configuration language for infrastructure (CLI required + optional nu_plugin_kcl)
• nu_plugin_tera - Native Nushell plugin for template rendering
• SOPS 3.10.2 - Secret management
• Age 1.2.1 - Encryption tool
• K9s 0.50.6 - Kubernetes management interface

Environment Variables

Set these environment variables before installation:

# Main provisioning directory (default: /usr/local/provisioning)
export PROVISIONING="/usr/local/provisioning"

# Path to cloud infrastructure configurations
export PROVISIONING_KLOUD_PATH="/path/to/your/infrastructure/configurations"

# Path to infrastructure definitions
export PROVISIONING_INFRA_PATH="/path/to/your/infrastructure/definitions"

# Path to SOPS configuration file
export PROVISIONING_SOPS="/path/to/your/sops/configuration"

# Path to Age encryption key file
export PROVISIONING_KAGE="/path/to/your/age/key/file"

Installation and Setup

Installation Methods

Method 1: Pre-built Package (Recommended)

Download the pre-built package and install:

# Download the provisioning package (replace with actual download link)
wget https://releases.example.com/provisioning.tar.gz

# Extract the package
tar xzf provisioning.tar.gz

# Run the installation script
sudo ./install-provisioning

Method 2: Install from Source

For development or custom installations:

# Clone the repository
git clone <repository-url>
cd provisioning

# Install directly from repository
./distro/from-repo.sh

Method 3: Package Installation Script

If you have access to the source and want to install from a generated package:

# Requires SRC_DEVS environment variable to be set
./distro/pack-install.sh

Installation Process

The install-provisioning script performs these steps automatically:

1. Core System Installation

   • Installs to /usr/local/provisioning by default
   • Creates symlink at /usr/local/bin/provisioning for global access
   • Removes any previous installation
   • Sets up directory structure

2. Dependency Installation

   • Installs Nushell 0.105.2 (primary shell)
   • Installs KCL 0.11.2 (configuration language CLI - required for nu_plugin_kcl)
   • Installs SOPS 3.10.2 (secrets management)
   • Installs Age 1.2.1 (encryption)
   • Installs K9s 0.50.6 (Kubernetes management)

3. Configuration Setup

   • Creates user configuration directories
   • Initializes Nushell configuration files
   • Sets up provisioning context and library loading

4. Tool Configuration

   • Configures tool paths and middleware
   • Installs and registers nu_plugin_tera and nu_plugin_kcl (requires KCL CLI)
   • Verifies tool versions and plugin compatibility
   • Sets up provider integrations

Post-Installation Verification

After installation, verify your setup:

# Check installation
provisioning version

# Verify environment configuration
provisioning env

# Show all configuration details
provisioning allenv

# Test basic functionality
provisioning help

Environment Configuration

Configure your environment after installation:

# Add to your shell profile (.bashrc, .zshrc, etc.)
export PROVISIONING="/usr/local/provisioning"
export PATH="$PROVISIONING/bin:$PATH"

# Reload your shell or source the profile
source ~/.bashrc  # or ~/.zshrc

Nushell Session

Start working with the provisioning system:

# Start interactive Nushell session with provisioning library loaded
provisioning nu

# Or run specific commands
provisioning help
provisioning context

Basic Usage

Main Commands

# General help
./core/nulib/provisioning help

# Version information
./core/nulib/provisioning version

# Context management
./core/nulib/provisioning context

Server Management

# Create new servers
./core/nulib/provisioning server create

# Delete servers
./core/nulib/provisioning server delete

# List all servers
./core/nulib/provisioning server list

# SSH to server
./core/nulib/provisioning server ssh <server>

# Show server pricing
./core/nulib/provisioning server price

Task Services (Infrastructure Components)

# Install infrastructure service
./core/nulib/provisioning taskserv create <service>

# Remove infrastructure service
./core/nulib/provisioning taskserv delete <service>

# List available services
./core/nulib/provisioning taskserv list

# Generate service configuration
./core/nulib/provisioning taskserv generate <service>

Cluster Operations

# Create Kubernetes cluster
./core/nulib/provisioning cluster create

# Delete cluster
./core/nulib/provisioning cluster delete

# List cluster components
./core/nulib/provisioning cluster list

Infrastructure Generation

# Generate server configurations
./core/nulib/provisioning generate server

# Generate task service configurations
./core/nulib/provisioning generate taskserv

# Generate cluster configurations
./core/nulib/provisioning generate cluster

# Generate infrastructure configurations
./core/nulib/provisioning generate infra

Project Architecture

High-Level Structure

provisioning/
├── core/                 # Core system
├── providers/            # Cloud providers
├── taskservs/            # Infrastructure services
├── cluster/              # Cluster configurations
├── kcl/                  # Configuration language
├── templates/            # Templates and generation
├── generate/             # Generation scripts
└── resources/            # Project resources

Key Components

1. Core System (/core/)

• Entry Point: core/nulib/provisioning - CLI interface script
• Library: core/nulib/lib_provisioning/ - Core provisioning library modules
• Utilities: core/nulib/lib_provisioning/utils/ - Helper functions

2. Providers (/providers/)

• AWS: providers/aws/ - Amazon Web Services provider
• UpCloud: providers/upcloud/ - UpCloud provider
• Local: providers/local/ - Local development provider

3. Task Services (/taskservs/)

Infrastructure services that can be deployed:

• Kubernetes: Core K8s installation
• Container Runtimes: containerd, cri-o, crun, runc, youki
• Networking: Cilium, CoreDNS, proxy/HAProxy
• Storage: Rook-Ceph, external-NFS, Mayastor
• Container Registry: OCI registry
• Development Tools: Gitea, PostgreSQL

4. Cluster Configurations (/cluster/)

• Buildkit: Container build system
• CI/CD: ArgoCD, Tekton pipelines
• Git: Git server configurations
• Web: Web service deployments

Common Workflows

1. Set Up New Infrastructure

# Generate new infrastructure
./core/nulib/provisioning generate infra --new <name>

# Create servers (check mode)
./core/nulib/provisioning server create --infra <name> --check

# Confirm creation
./core/nulib/provisioning server create --infra <name>

2. Install Kubernetes

# Install Kubernetes on infrastructure
./core/nulib/provisioning taskserv create kubernetes --infra <name>

# Verify installation
./core/nulib/provisioning taskserv list

3. Deploy Cluster Services

# Create cluster services
./core/nulib/provisioning cluster create --infra <name>

# List components
./core/nulib/provisioning cluster list

4. SSH Server Access

# SSH to specific server
./core/nulib/provisioning server ssh <hostname>

Development and Debug Flags

• --debug or -x - Enable debug mode
• --check or -c - Check mode (no actual changes)
• --yes or -y - Auto-confirm operations
• --wait or -w - Wait for operations to complete
• --infra or -i - Specify infrastructure target
• --out - Output format (json, yaml, text)

Secret Management

Edit Encrypted Files

# Edit file with SOPS
./core/nulib/provisioning sops <file>

Age Configuration

Ensure PROVISIONING_KAGE points to the correct Age key file.

File Structure Patterns

• KCL files (.k): Infrastructure configuration and schemas
• Nushell files (.nu): Automation scripts and modules
• Jinja2 templates (.j2): Configuration file templates
• TOML files (.toml): Definitions and configurations
• YAML files (.yaml): Kubernetes manifests and settings

Troubleshooting

Installation Issues

1. Installation fails with permission errors

   # Ensure you run the installer with sudo
   sudo ./install-provisioning
   
   # Check if /usr/local is writable
   ls -la /usr/local/
   

2. Package extraction fails

   # Verify package integrity
   file provisioning.tar.gz
   
   # Extract with verbose output
   tar xzvf provisioning.tar.gz
   

3. Nushell installation fails

   # Check Nushell version after installation
   nu --version
   
   # If missing, the installer should handle this automatically
   # Contact support if this persists
   

4. Missing dependencies

   # Check tool versions
   provisioning version
   
   # Verify tool installation
   provisioning setup tools check all
   

5. Environment variables not set

   # Check current environment
   provisioning env
   
   # Set missing variables
   export PROVISIONING="/usr/local/provisioning"
   export PATH="$PROVISIONING/bin:$PATH"
   

Common Runtime Issues

1. Missing environment variables

   provisioning env
   

2. KCL compilation errors

   kcl fmt file.k
   

3. nu_plugin_kcl issues

   # Check if KCL CLI is installed (required for nu_plugin_kcl)
   kcl version
   
   # Check if plugin is registered
   version | get installed_plugins | str contains "nu_plugin_kcl"
   

4. SOPS/Age encryption issues

   • Verify PROVISIONING_KAGE path and key file

5. Provider authentication

   • Verify cloud provider credentials and permissions

6. Command not found

   # Check if provisioning is in PATH
   which provisioning
   
   # Add to PATH if needed
   export PATH="/usr/local/bin:$PATH"
   

Debug Commands

# Enable verbose logging
./core/nulib/provisioning --debug <command>

# View current configuration
./core/nulib/provisioning show settings

# List server status
./core/nulib/provisioning show servers

# Interactive shell with library loaded
./core/nulib/provisioning nu

Contributing

This project uses a modular architecture where:

• Providers handle cloud-specific operations
• Task Services are reusable infrastructure components
• Clusters define complete deployment configurations
• Core provides unified CLI and library functions

Nushell Plugins

Plugins are sourced from: https://repo.jesusperez.pro/jesus/nushell-plugins

License

See LICENSE file for details.

Support

For issues and questions, consult the documentation in core/README.md or use the built-in help commands.