diff --git a/README.md b/README.md index d3b808e..c02c8ec 100644 --- a/README.md +++ b/README.md @@ -1,36 +1,538 @@ -# [Provisioning](./) +# Cloud Native Provisioning -Managing Infrastructure for Services and Tasks under own control and needs in declarative and sustainable mode with **T-rust** +
+ Provisioning Logo +
+
+ + + + Provisioning Logo Text + +
-Provisioning +A cloud-native provisioning system built with Nushell and KCL for infrastructure automation and Kubernetes cluster management. ---- +## Table of Contents -i**Infrastructure and Services** Provisioning Tool **Core** and **TaskServices**,
-fully oriented to a "private and self hosted infrastructure" for developments and deployments,
-addressing services/projects life cycles
-in Declarative mode by following **IaC**, **GitOps**, **CD/CI** principles
-under a **DevOps culture**. +- [Overview](#overview) +- [Prerequisites](#prerequisites) + - [System Requirements](#system-requirements) + - [Required Tools](#required-tools) + - [Environment Variables](#environment-variables) +- [Installation and Setup](#installation-and-setup) + - [Installation Methods](#installation-methods) + - [Installation Process](#installation-process) + - [Post-Installation Verification](#post-installation-verification) + - [Environment Configuration](#environment-configuration) + - [Nushell Session](#nushell-session) +- [Basic Usage](#basic-usage) + - [Main Commands](#main-commands) + - [Server Management](#server-management) + - [Task Services (Infrastructure Components)](#task-services-infrastructure-components) + - [Cluster Operations](#cluster-operations) + - [Infrastructure Generation](#infrastructure-generation) +- [Project Architecture](#project-architecture) + - [High-Level Structure](#high-level-structure) + - [Key Components](#key-components) +- [Common Workflows](#common-workflows) + - [1. Set Up New Infrastructure](#1-set-up-new-infrastructure) + - [2. Install Kubernetes](#2-install-kubernetes) + - [3. Deploy Cluster Services](#3-deploy-cluster-services) + - [4. SSH Server Access](#4-ssh-server-access) +- [Development and Debug Flags](#development-and-debug-flags) +- [Secret Management](#secret-management) + - [Edit Encrypted Files](#edit-encrypted-files) + - [Age Configuration](#age-configuration) +- [File Structure Patterns](#file-structure-patterns) +- [Troubleshooting](#troubleshooting) + - [Installation Issues](#installation-issues) + - [Common Runtime Issues](#common-runtime-issues) + - [Debug Commands](#debug-commands) +- [Contributing](#contributing) + - [Nushell Plugins](#nushell-plugins) +- [License](#license) +- [Support](#support) -> [!WARNING] -> Code currently in a fulll review. It will be presented soon ! +## Overview - ## Features +This system provides a complete Infrastructure as Code (IaC) platform that enables provisioning, configuration, and management of infrastructure across multiple cloud providers with a strong focus on cloud-native Kubernetes deployments. -Managing Infrastructure for Services/Tasks in declarative, customizable and sustainable mode with **T-rust** +### Key Features -- Nushell a Rust tool for easy t-rust-ed composition, included parsing tools and output look and feel in Opertions -- KCL for static type and contraints Opertions states declarations-definitions (IaC mode) [what] -- SOPS to secure encrypt credentials and certs (GitOps, CDCI modes) via several providers [safety] -- **Providers** and their resources can be plugged, mixted and used via their CLI or API for operations. Local, UpCloud and AWS tested by now -- **Servers** operations can be run in parallel -- **Tasks/Services** can be defined and attached to hosts independly, with different profiles and executed in several **run modes** to adjust specific contexts and needs -- **Cluster** operations for resources under containers services like Podman or Kubernetes, using tailored profiles and **run modes** -- **Templating engine** Tera to customize operations with definitions settings -- **Basic help** to run commands and request, improve usability and look & feel -- **Instrospection and naviation views** to inspect environments, resources, logs and debug -- **Encryption handling** can manage definitions and settings encrypt/decrypt on the fly -- **Setup, Context, Tools** handling for configuration, install, updates, etc housekeeping tasks -- **Generate Provisioning targets** from templates o blueprints with simple and minumum settings from config file or via prompt -- **Provisioning Web UI** addon in Wasm connecting to CLI for operations, allows remote view and management with friendly interface using drag-drop and clipboard copy & paste +- **Multi-provider**: Support for AWS, UpCloud, and local environments +- **Infrastructure as Code**: Declarative configuration using KCL +- **Secret Management**: Integration with SOPS and Age for encryption +- **Modular**: Component-based reusable architecture +- **Kubernetes-first**: Designed for cloud-native deployments +## Prerequisites + +### System Requirements + +- **Operating System**: Linux, macOS, or Windows (WSL2) +- **Architecture**: x86_64 or ARM64 +- **Memory**: Minimum 4GB RAM (8GB recommended) +- **Storage**: At least 2GB free space +- **Network**: Internet connection for downloading dependencies + +### Required Tools + +The installation process will automatically install these tools if not present: + +- **Nushell 0.105.2** - Primary shell and scripting language +- **KCL 0.11.2** - Configuration language for infrastructure (CLI required + optional nu_plugin_kcl) +- **nu_plugin_tera** - Native Nushell plugin for template rendering +- **SOPS 3.10.2** - Secret management +- **Age 1.2.1** - Encryption tool +- **K9s 0.50.6** - Kubernetes management interface + +### Environment Variables + +Set these environment variables before installation: + +```bash +# Main provisioning directory (default: /usr/local/provisioning) +export PROVISIONING="/usr/local/provisioning" + +# Path to cloud infrastructure configurations +export PROVISIONING_KLOUD_PATH="/path/to/your/infrastructure/configurations" + +# Path to infrastructure definitions +export PROVISIONING_INFRA_PATH="/path/to/your/infrastructure/definitions" + +# Path to SOPS configuration file +export PROVISIONING_SOPS="/path/to/your/sops/configuration" + +# Path to Age encryption key file +export PROVISIONING_KAGE="/path/to/your/age/key/file" +``` + +## Installation and Setup + +### Installation Methods + +#### Method 1: Pre-built Package (Recommended) + +Download the pre-built package and install: + +```bash +# Download the provisioning package (replace with actual download link) +wget https://releases.example.com/provisioning.tar.gz + +# Extract the package +tar xzf provisioning.tar.gz + +# Run the installation script +sudo ./install-provisioning +``` + +#### Method 2: Install from Source + +For development or custom installations: + +```bash +# Clone the repository +git clone +cd provisioning + +# Install directly from repository +./distro/from-repo.sh +``` + +#### Method 3: Package Installation Script + +If you have access to the source and want to install from a generated package: + +```bash +# Requires SRC_DEVS environment variable to be set +./distro/pack-install.sh +``` + +### Installation Process + +The `install-provisioning` script performs these steps automatically: + +1. **Core System Installation** + - Installs to `/usr/local/provisioning` by default + - Creates symlink at `/usr/local/bin/provisioning` for global access + - Removes any previous installation + - Sets up directory structure + +2. **Dependency Installation** + - Installs **Nushell 0.105.2** (primary shell) + - Installs **KCL 0.11.2** (configuration language CLI - required for nu_plugin_kcl) + - Installs **SOPS 3.10.2** (secrets management) + - Installs **Age 1.2.1** (encryption) + - Installs **K9s 0.50.6** (Kubernetes management) + +3. **Configuration Setup** + - Creates user configuration directories + - Initializes Nushell configuration files + - Sets up provisioning context and library loading + +4. **Tool Configuration** + - Configures tool paths and middleware + - Installs and registers nu_plugin_tera and nu_plugin_kcl (requires KCL CLI) + - Verifies tool versions and plugin compatibility + - Sets up provider integrations + +### Post-Installation Verification + +After installation, verify your setup: + +```bash +# Check installation +provisioning version + +# Verify environment configuration +provisioning env + +# Show all configuration details +provisioning allenv + +# Test basic functionality +provisioning help +``` + +### Environment Configuration + +Configure your environment after installation: + +```bash +# Add to your shell profile (.bashrc, .zshrc, etc.) +export PROVISIONING="/usr/local/provisioning" +export PATH="$PROVISIONING/bin:$PATH" + +# Reload your shell or source the profile +source ~/.bashrc # or ~/.zshrc +``` + +### Nushell Session + +Start working with the provisioning system: + +```bash +# Start interactive Nushell session with provisioning library loaded +provisioning nu + +# Or run specific commands +provisioning help +provisioning context +``` + +## Basic Usage + +### Main Commands + +```bash +# General help +./core/nulib/provisioning help + +# Version information +./core/nulib/provisioning version + +# Context management +./core/nulib/provisioning context +``` + +### Server Management + +```bash +# Create new servers +./core/nulib/provisioning server create + +# Delete servers +./core/nulib/provisioning server delete + +# List all servers +./core/nulib/provisioning server list + +# SSH to server +./core/nulib/provisioning server ssh + +# Show server pricing +./core/nulib/provisioning server price +``` + +### Task Services (Infrastructure Components) + +```bash +# Install infrastructure service +./core/nulib/provisioning taskserv create + +# Remove infrastructure service +./core/nulib/provisioning taskserv delete + +# List available services +./core/nulib/provisioning taskserv list + +# Generate service configuration +./core/nulib/provisioning taskserv generate +``` + +### Cluster Operations + +```bash +# Create Kubernetes cluster +./core/nulib/provisioning cluster create + +# Delete cluster +./core/nulib/provisioning cluster delete + +# List cluster components +./core/nulib/provisioning cluster list +``` + +### Infrastructure Generation + +```bash +# Generate server configurations +./core/nulib/provisioning generate server + +# Generate task service configurations +./core/nulib/provisioning generate taskserv + +# Generate cluster configurations +./core/nulib/provisioning generate cluster + +# Generate infrastructure configurations +./core/nulib/provisioning generate infra +``` + +## Project Architecture + +### High-Level Structure + +``` +provisioning/ +├── core/ # Core system +├── providers/ # Cloud providers +├── taskservs/ # Infrastructure services +├── cluster/ # Cluster configurations +├── kcl/ # Configuration language +├── templates/ # Templates and generation +├── generate/ # Generation scripts +└── resources/ # Project resources +``` + +### Key Components + +#### 1. Core System (`/core/`) +- **Entry Point**: `core/nulib/provisioning` - CLI interface script +- **Library**: `core/nulib/lib_provisioning/` - Core provisioning library modules +- **Utilities**: `core/nulib/lib_provisioning/utils/` - Helper functions + +#### 2. Providers (`/providers/`) +- **AWS**: `providers/aws/` - Amazon Web Services provider +- **UpCloud**: `providers/upcloud/` - UpCloud provider +- **Local**: `providers/local/` - Local development provider + +#### 3. Task Services (`/taskservs/`) +Infrastructure services that can be deployed: +- **Kubernetes**: Core K8s installation +- **Container Runtimes**: containerd, cri-o, crun, runc, youki +- **Networking**: Cilium, CoreDNS, proxy/HAProxy +- **Storage**: Rook-Ceph, external-NFS, Mayastor +- **Container Registry**: OCI registry +- **Development Tools**: Gitea, PostgreSQL + +#### 4. Cluster Configurations (`/cluster/`) +- **Buildkit**: Container build system +- **CI/CD**: ArgoCD, Tekton pipelines +- **Git**: Git server configurations +- **Web**: Web service deployments + +## Common Workflows + +### 1. Set Up New Infrastructure + +```bash +# Generate new infrastructure +./core/nulib/provisioning generate infra --new + +# Create servers (check mode) +./core/nulib/provisioning server create --infra --check + +# Confirm creation +./core/nulib/provisioning server create --infra +``` + +### 2. Install Kubernetes + +```bash +# Install Kubernetes on infrastructure +./core/nulib/provisioning taskserv create kubernetes --infra + +# Verify installation +./core/nulib/provisioning taskserv list +``` + +### 3. Deploy Cluster Services + +```bash +# Create cluster services +./core/nulib/provisioning cluster create --infra + +# List components +./core/nulib/provisioning cluster list +``` + +### 4. SSH Server Access + +```bash +# SSH to specific server +./core/nulib/provisioning server ssh +``` + +## Development and Debug Flags + +- `--debug` or `-x` - Enable debug mode +- `--check` or `-c` - Check mode (no actual changes) +- `--yes` or `-y` - Auto-confirm operations +- `--wait` or `-w` - Wait for operations to complete +- `--infra` or `-i` - Specify infrastructure target +- `--out` - Output format (json, yaml, text) + +## Secret Management + +### Edit Encrypted Files + +```bash +# Edit file with SOPS +./core/nulib/provisioning sops +``` + +### Age Configuration + +Ensure `PROVISIONING_KAGE` points to the correct Age key file. + +## File Structure Patterns + +- **KCL files** (`.k`): Infrastructure configuration and schemas +- **Nushell files** (`.nu`): Automation scripts and modules +- **Jinja2 templates** (`.j2`): Configuration file templates +- **TOML files** (`.toml`): Definitions and configurations +- **YAML files** (`.yaml`): Kubernetes manifests and settings + +## Troubleshooting + +### Installation Issues + +1. **Installation fails with permission errors** + ```bash + # Ensure you run the installer with sudo + sudo ./install-provisioning + + # Check if /usr/local is writable + ls -la /usr/local/ + ``` + +2. **Package extraction fails** + ```bash + # Verify package integrity + file provisioning.tar.gz + + # Extract with verbose output + tar xzvf provisioning.tar.gz + ``` + +3. **Nushell installation fails** + ```bash + # Check Nushell version after installation + nu --version + + # If missing, the installer should handle this automatically + # Contact support if this persists + ``` + +4. **Missing dependencies** + ```bash + # Check tool versions + provisioning version + + # Verify tool installation + provisioning setup tools check all + ``` + +5. **Environment variables not set** + ```bash + # Check current environment + provisioning env + + # Set missing variables + export PROVISIONING="/usr/local/provisioning" + export PATH="$PROVISIONING/bin:$PATH" + ``` + +### Common Runtime Issues + +1. **Missing environment variables** + ```bash + provisioning env + ``` + +2. **KCL compilation errors** + ```bash + kcl fmt file.k + ``` + +3. **nu_plugin_kcl issues** + ```bash + # Check if KCL CLI is installed (required for nu_plugin_kcl) + kcl version + + # Check if plugin is registered + version | get installed_plugins | str contains "nu_plugin_kcl" + ``` + +3. **SOPS/Age encryption issues** + - Verify `PROVISIONING_KAGE` path and key file + +4. **Provider authentication** + - Verify cloud provider credentials and permissions + +5. **Command not found** + ```bash + # Check if provisioning is in PATH + which provisioning + + # Add to PATH if needed + export PATH="/usr/local/bin:$PATH" + ``` + +### Debug Commands + +```bash +# Enable verbose logging +./core/nulib/provisioning --debug + +# View current configuration +./core/nulib/provisioning show settings + +# List server status +./core/nulib/provisioning show servers + +# Interactive shell with library loaded +./core/nulib/provisioning nu +``` + +## Contributing + +This project uses a modular architecture where: +- **Providers** handle cloud-specific operations +- **Task Services** are reusable infrastructure components +- **Clusters** define complete deployment configurations +- **Core** provides unified CLI and library functions + +### Nushell Plugins + +Plugins are sourced from: https://repo.jesusperez.pro/jesus/nushell-plugins + +## License + +See LICENSE file for details. + +## Support + +For issues and questions, consult the documentation in `core/README.md` or use the built-in help commands. \ No newline at end of file diff --git a/images/provisioning_logo-dark.svg b/images/provisioning_logo-dark.svg deleted file mode 100644 index 65e60a4..0000000 --- a/images/provisioning_logo-dark.svg +++ /dev/null @@ -1 +0,0 @@ - \ No newline at end of file diff --git a/images/provisioning_logo-light.svg b/images/provisioning_logo-light.svg deleted file mode 100644 index 0c44a50..0000000 --- a/images/provisioning_logo-light.svg +++ /dev/null @@ -1 +0,0 @@ - \ No newline at end of file