jesus/provisioning
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
77b63ef3c8
provisioning/README.md
Jesús Pérex 77b63ef3c8
chore: check
2025-07-10 21:41:03 +01:00

540 lines
14 KiB
Markdown
Raw Blame History

# Provisioning
<div align="center">
<img src="resources/images/provisioning_logo-light.svg" alt="Provisioning Logo" width="200"/>
<br>
<br>
<picture>
<source media="(prefers-color-scheme: dark)" srcset="resources/images/provisioning_logo-text-dark.svg">
<source media="(prefers-color-scheme: light)" srcset="resources/images/provisioning_logo-text-light.svg">
<img alt="Provisioning Logo Text" src="resources/images/provisioning_logo-text-light.svg" width="400">
</picture>
</div>
A provisioning system built with Nushell and KCL for infrastructure automation and Kubernetes cluster management.
## Table of Contents
- [Overview](#overview)
- [Prerequisites](#prerequisites)
- [System Requirements](#system-requirements)
- [Required Tools](#required-tools)
- [Environment Variables](#environment-variables)
- [Installation and Setup](#installation-and-setup)
- [Installation Methods](#installation-methods)
- [Installation Process](#installation-process)
- [Post-Installation Verification](#post-installation-verification)
- [Environment Configuration](#environment-configuration)
- [Nushell Session](#nushell-session)
- [Basic Usage](#basic-usage)
- [Main Commands](#main-commands)
- [Server Management](#server-management)
- [Task Services (Infrastructure Components)](#task-services-infrastructure-components)
- [Cluster Operations](#cluster-operations)
- [Infrastructure Generation](#infrastructure-generation)
- [Project Architecture](#project-architecture)
- [High-Level Structure](#high-level-structure)
- [Key Components](#key-components)
- [Common Workflows](#common-workflows)
- [1. Set Up New Infrastructure](#1-set-up-new-infrastructure)
- [2. Install Kubernetes](#2-install-kubernetes)
- [3. Deploy Cluster Services](#3-deploy-cluster-services)
- [4. SSH Server Access](#4-ssh-server-access)
- [Development and Debug Flags](#development-and-debug-flags)
- [Secret Management](#secret-management)
- [Edit Encrypted Files](#edit-encrypted-files)
- [Age Configuration](#age-configuration)
- [File Structure Patterns](#file-structure-patterns)
- [Troubleshooting](#troubleshooting)
- [Installation Issues](#installation-issues)
- [Common Runtime Issues](#common-runtime-issues)
- [Debug Commands](#debug-commands)
- [Contributing](#contributing)
- [Nushell Plugins](#nushell-plugins)
- [License](#license)
- [Support](#support)
## Overview
This system provides a complete Infrastructure as Code (IaC) platform that enables provisioning, configuration, and management of infrastructure across multiple cloud providers with a strong focus on cloud-native Kubernetes deployments.
### Key Features
- **Multi-provider**: Support for AWS, UpCloud, and local environments
- **Infrastructure as Code**: Declarative configuration using KCL
- **Secret Management**: Integration with SOPS and Age for encryption
- **Modular**: Component-based reusable architecture
- **Kubernetes-first**: Designed for cloud-native deployments
## Prerequisites
### System Requirements
- **Operating System**: Linux, macOS, or Windows (WSL2)
- **Architecture**: x86_64 or ARM64
- **Memory**: Minimum 4GB RAM (8GB recommended)
- **Storage**: At least 2GB free space
- **Network**: Internet connection for downloading dependencies
### Required Tools
The installation process will automatically install these tools if not present:
- **Nushell 0.105.2** - Primary shell and scripting language
- **KCL 0.11.2** - Configuration language for infrastructure (CLI required + optional nu_plugin_kcl)
- **nu_plugin_tera** - Native Nushell plugin for template rendering
- **SOPS 3.10.2** - Secret management
- **Age 1.2.1** - Encryption tool
- **K9s 0.50.6** - Kubernetes management interface
### Environment Variables
Set these environment variables before installation:
```bash
# Main provisioning directory (default: /usr/local/provisioning)
export PROVISIONING="/usr/local/provisioning"
# Path to cloud infrastructure configurations
export PROVISIONING_KLOUD_PATH="/path/to/your/infrastructure/configurations"
# Path to infrastructure definitions
export PROVISIONING_INFRA_PATH="/path/to/your/infrastructure/definitions"
# Path to SOPS configuration file
export PROVISIONING_SOPS="/path/to/your/sops/configuration"
# Path to Age encryption key file
export PROVISIONING_KAGE="/path/to/your/age/key/file"
```
## Installation and Setup
### Installation Methods
#### Method 1: Pre-built Package (Recommended)
Download the pre-built package and install:
```bash
# Download the provisioning package (replace with actual download link)
wget https://releases.example.com/provisioning.tar.gz
# Extract the package
tar xzf provisioning.tar.gz
# Run the installation script
sudo ./install-provisioning
```
#### Method 2: Install from Source
For development or custom installations:
```bash
# Clone the repository
git clone <repository-url>
cd provisioning
# Install directly from repository
./distro/from-repo.sh
```
#### Method 3: Package Installation Script
If you have access to the source and want to install from a generated package:
```bash
# Requires SRC_DEVS environment variable to be set
./distro/pack-install.sh
```
### Installation Process
The `install-provisioning` script performs these steps automatically:
1. **Core System Installation**
- Installs to `/usr/local/provisioning` by default
- Creates symlink at `/usr/local/bin/provisioning` for global access
- Removes any previous installation
- Sets up directory structure
2. **Dependency Installation**
- Installs **Nushell 0.105.2** (primary shell)
- Installs **KCL 0.11.2** (configuration language CLI - required for nu_plugin_kcl)
- Installs **SOPS 3.10.2** (secrets management)
- Installs **Age 1.2.1** (encryption)
- Installs **K9s 0.50.6** (Kubernetes management)
3. **Configuration Setup**
- Creates user configuration directories
- Initializes Nushell configuration files
- Sets up provisioning context and library loading
4. **Tool Configuration**
- Configures tool paths and middleware
- Installs and registers nu_plugin_tera and nu_plugin_kcl (requires KCL CLI)
- Verifies tool versions and plugin compatibility
- Sets up provider integrations
### Post-Installation Verification
After installation, verify your setup:
```bash
# Check installation
provisioning version
# Verify environment configuration
provisioning env
# Show all configuration details
provisioning allenv
# Test basic functionality
provisioning help
```
### Environment Configuration
Configure your environment after installation:
```bash
# Add to your shell profile (.bashrc, .zshrc, etc.)
export PROVISIONING="/usr/local/provisioning"
export PATH="$PROVISIONING/bin:$PATH"
# Reload your shell or source the profile
source ~/.bashrc # or ~/.zshrc
```
### Nushell Session
Start working with the provisioning system:
```bash
# Start interactive Nushell session with provisioning library loaded
provisioning nu
# Or run specific commands
provisioning help
provisioning context
```
## Basic Usage
### Main Commands
```bash
# General help
./core/nulib/provisioning help
# Version information
./core/nulib/provisioning version
# Context management
./core/nulib/provisioning context
```
### Server Management
```bash
# Create new servers
./core/nulib/provisioning server create
# Delete servers
./core/nulib/provisioning server delete
# List all servers
./core/nulib/provisioning server list
# SSH to server
./core/nulib/provisioning server ssh <server>
# Show server pricing
./core/nulib/provisioning server price
```
### Task Services (Infrastructure Components)
```bash
# Install infrastructure service
./core/nulib/provisioning taskserv create <service>
# Remove infrastructure service
./core/nulib/provisioning taskserv delete <service>
# List available services
./core/nulib/provisioning taskserv list
# Generate service configuration
./core/nulib/provisioning taskserv generate <service>
```
### Cluster Operations
```bash
# Create Kubernetes cluster
./core/nulib/provisioning cluster create
# Delete cluster
./core/nulib/provisioning cluster delete
# List cluster components
./core/nulib/provisioning cluster list
```
### Infrastructure Generation
```bash
# Generate server configurations
./core/nulib/provisioning generate server
# Generate task service configurations
./core/nulib/provisioning generate taskserv
# Generate cluster configurations
./core/nulib/provisioning generate cluster
# Generate infrastructure configurations
./core/nulib/provisioning generate infra
```
## Project Architecture
### High-Level Structure
```
provisioning/
├── core/ # Core system
├── providers/ # Cloud providers
├── taskservs/ # Infrastructure services
├── cluster/ # Cluster configurations
├── kcl/ # Configuration language
├── templates/ # Templates and generation
├── generate/ # Generation scripts
└── resources/ # Project resources
```
### Key Components
#### 1. Core System (`/core/`)
- **Entry Point**: `core/nulib/provisioning` - CLI interface script
- **Library**: `core/nulib/lib_provisioning/` - Core provisioning library modules
- **Utilities**: `core/nulib/lib_provisioning/utils/` - Helper functions
#### 2. Providers (`/providers/`)
- **AWS**: `providers/aws/` - Amazon Web Services provider
- **UpCloud**: `providers/upcloud/` - UpCloud provider
- **Local**: `providers/local/` - Local development provider
#### 3. Task Services (`/taskservs/`)
Infrastructure services that can be deployed:
- **Kubernetes**: Core K8s installation
- **Container Runtimes**: containerd, cri-o, crun, runc, youki
- **Networking**: Cilium, CoreDNS, proxy/HAProxy
- **Storage**: Rook-Ceph, external-NFS, Mayastor
- **Container Registry**: OCI registry
- **Development Tools**: Gitea, PostgreSQL
#### 4. Cluster Configurations (`/cluster/`)
- **Buildkit**: Container build system
- **CI/CD**: ArgoCD, Tekton pipelines
- **Git**: Git server configurations
- **Web**: Web service deployments
## Common Workflows
### 1. Set Up New Infrastructure
```bash
# Generate new infrastructure
./core/nulib/provisioning generate infra --new <name>
# Create servers (check mode)
./core/nulib/provisioning server create --infra <name> --check
# Confirm creation
./core/nulib/provisioning server create --infra <name>
```
### 2. Install Kubernetes
```bash
# Install Kubernetes on infrastructure
./core/nulib/provisioning taskserv create kubernetes --infra <name>
# Verify installation
./core/nulib/provisioning taskserv list
```
### 3. Deploy Cluster Services
```bash
# Create cluster services
./core/nulib/provisioning cluster create --infra <name>
# List components
./core/nulib/provisioning cluster list
```
### 4. SSH Server Access
```bash
# SSH to specific server
./core/nulib/provisioning server ssh <hostname>
```
## Development and Debug Flags
- `--debug` or `-x` - Enable debug mode
- `--check` or `-c` - Check mode (no actual changes)
- `--yes` or `-y` - Auto-confirm operations
- `--wait` or `-w` - Wait for operations to complete
- `--infra` or `-i` - Specify infrastructure target
- `--out` - Output format (json, yaml, text)
## Secret Management
### Edit Encrypted Files
```bash
# Edit file with SOPS
./core/nulib/provisioning sops <file>
```
### Age Configuration
Ensure `PROVISIONING_KAGE` points to the correct Age key file.
## File Structure Patterns
- **KCL files** (`.k`): Infrastructure configuration and schemas
- **Nushell files** (`.nu`): Automation scripts and modules
- **Jinja2 templates** (`.j2`): Configuration file templates
- **TOML files** (`.toml`): Definitions and configurations
- **YAML files** (`.yaml`): Kubernetes manifests and settings
## Troubleshooting
### Installation Issues
1. **Installation fails with permission errors**
```bash
# Ensure you run the installer with sudo
sudo ./install-provisioning
# Check if /usr/local is writable
ls -la /usr/local/
```
2. **Package extraction fails**
```bash
# Verify package integrity
file provisioning.tar.gz
# Extract with verbose output
tar xzvf provisioning.tar.gz
```
3. **Nushell installation fails**
```bash
# Check Nushell version after installation
nu --version
# If missing, the installer should handle this automatically
# Contact support if this persists
```
4. **Missing dependencies**
```bash
# Check tool versions
provisioning version
# Verify tool installation
provisioning setup tools check all
```
5. **Environment variables not set**
```bash
# Check current environment
provisioning env
# Set missing variables
export PROVISIONING="/usr/local/provisioning"
export PATH="$PROVISIONING/bin:$PATH"
```
### Common Runtime Issues
1. **Missing environment variables**
```bash
provisioning env
```
2. **KCL compilation errors**
```bash
kcl fmt file.k
```
3. **nu_plugin_kcl issues**
```bash
# Check if KCL CLI is installed (required for nu_plugin_kcl)
kcl version
# Check if plugin is registered
version | get installed_plugins | str contains "nu_plugin_kcl"
```
3. **SOPS/Age encryption issues**
- Verify `PROVISIONING_KAGE` path and key file
4. **Provider authentication**
- Verify cloud provider credentials and permissions
5. **Command not found**
```bash
# Check if provisioning is in PATH
which provisioning
# Add to PATH if needed
export PATH="/usr/local/bin:$PATH"
```
### Debug Commands
```bash
# Enable verbose logging
./core/nulib/provisioning --debug <command>
# View current configuration
./core/nulib/provisioning show settings
# List server status
./core/nulib/provisioning show servers
# Interactive shell with library loaded
./core/nulib/provisioning nu
```
## Contributing
This project uses a modular architecture where:
- **Providers** handle cloud-specific operations
- **Task Services** are reusable infrastructure components
- **Clusters** define complete deployment configurations
- **Core** provides unified CLI and library functions
### Nushell Plugins
Plugins are sourced from: https://repo.jesusperez.pro/jesus/nushell-plugins
## License
See LICENSE file for details.
## Support
For issues and questions, consult the documentation in `core/README.md` or use the built-in help commands.
Reference in New Issue View Git Blame Copy Permalink