Key Features

• Bill Image Upload: Users can upload images of bills/expenses directly through the Flask web application.

• Google Drive Integration: Uploaded bill images are saved to Google Drive for easy access and backup.

• Notion Database: Extracted bill details are stored in a Notion database, allowing users to track and manage their bills efficiently.

• Secure and Scalable: The application is designed with security in mind, ensuring that user data is protected.

• Cost-Effective: Instead of paying for expensive third-party services, this application can be built and maintained at no cost using open-source tools and APIs.

Technology Stack

1. Flask: A lightweight and flexible Python web framework used for building the web applications.

2. Google Drive API: Used to upload and manage bill images in Google Drive.

3. Notion API: Used to create and update entries in a Notion database with the extracted bill details.

4. Mindee API: An optional tool to enhance OCR capabilities, especially for more complex bill structures.

Step-by-Step Guide

1. Setting Up the Flask Application:

• Create a Flask application with a simple user interface for uploading bill images.

• Implement secure user authentication to protect user data.

from flask import Flask, request, render_template, redirect, url_for
from werkzeug.utils import secure_filename
import os
from dotenv import load_dotenv
import base64
from utils.gdrive_helper import GoogleDriveHelper
from utils.notion_service import NotionDatabaseHandler
from utils.reciept_service import ReceiptParser

load_dotenv()
app = Flask(__name__)

credential_json = json.loads(os.getenv("GOOGLE_OAUTH_SERVICE_ACCOUNT"))
gdrive_service = GoogleDriveHelper(credential_json)
drive_folder_id = gdrive_service.create_folder("test_notion_upload")

receipt_service = ReceiptParser(os.getenv("MINDEE_API_KEY"))

notion_db_id = os.getenv("NOTION_DB_ID")
notion_service = NotionDatabaseHandler(
        api_key=os.getenv("NOTION_KEY"),
        database_id=notion_db_id
    )

@app.route('/')
def index():
    return render_template('index.html')

@app.route('/upload', methods=['POST'])
def upload_file():
    if 'file' not in request.files:
        return redirect(request.url)
    file = request.files['file']
    if file.filename == '':
        return redirect(request.url)
    if file:
        filename = secure_filename(file.filename)
        file_bytes = file.read()
        file_base64 = base64.b64encode(file_bytes).decode('utf-8')

        # Parse the reciept 
        reciept_details = receipt_service.parse_reciept_from_base64(file_base64, filename)

        # Upload the file to the google drive
        _, drive_url = gdrive_service.upload_file_obj(
            file, filename, drive_folder_id
        )
        reciept_details["Receipt"] = {"name": filename, "url": drive_url}

        # Create the payload from the user content
        entry_payload = notion_service.create_entry_payload(reciept_details)

        # Insert the entry into the database
        if entry_payload:
            notion_service.insert_entry(entry_payload)
        flash("Image uploaded and processed successfully!", "success")

        # Process the uploaded file here
        return redirect(url_for('index'))

if __name__ == '__main__':
    app.run(debug=True)

This code creates a simple web interface where users can upload bill images. The uploaded files are saved to a local directory for further processing.

2. Integrating AI/ML Model:

• Develop a custom AI/ML model trained to parse bill receipts and extract information such as the supplier name, invoice number, amount, tax, net amount, category, and date.

• Integrate the model into the Flask application to process uploaded bill images.

• Alternatively, you can use Mindee to parse the invoice bill

from mindee import Client, product


class ReceiptParser:
    def __init__(self, api_key):
        self.mindee_client = Client(api_key=api_key)

    def extract_bill_details(self, receipt_data):
        output = {
            "Name": receipt_data.document.inference.prediction.supplier_name.value or '',
            "Invoice No": receipt_data.document.inference.prediction.receipt_number.value or '',
            "Amount": receipt_data.document.inference.prediction.total_amount.value,
            "Tax": receipt_data.document.inference.prediction.total_tax.value,
            "Net": receipt_data.document.inference.prediction.total_net.value,
            "Category": receipt_data.document.inference.prediction.category.value,
            "Date": receipt_data.document.inference.prediction.date.value,
            "Mode": "Credit Card",
        }
        print(f"Bill details: {output}")
        return output

    def parse_reciept_from_base64(self, base64_string, image_name):
        input_doc = self.mindee_client.source_from_b64string(base64_string, image_name)
        receipt_data = self.mindee_client.parse(product.ReceiptV5, input_doc)
        result = self.extract_bill_details(receipt_data)
        return result

Call the parse_reciept_from_base64 function in your upload handler to process each bill and extract the necessary information.

3. Uploading to Google Drive:

• Use the Google Drive API to upload bill images directly from the application using the OAuth credentials.

• Organize images in a structured way, such as by date or category, for easy retrieval.

class GoogleDriveHelper:
    SCOPES = [
        "https://www.googleapis.com/auth/drive.file",
        "https://www.googleapis.com/auth/drive",
    ]

    VALID_EXTENSIONS = [".pdf", ".png", ".jpeg", ".jpg"]

    def __init__(self, credentail_json):
        self.credentail_json_file = credentail_json
        self.connect_using_service_account()

    def connect_using_service_account(self):
        creds = ServiceAccountCredentials.from_json_keyfile_dict(
            self.credentail_json, self.SCOPES
        )
        self.drive_service = build(
            "drive", "v3", credentials=creds, cache_discovery=False
        )

    def upload_file_obj(self, file_obj, filename, parent_folder_id=None):
        mime_type, _ = mimetypes.guess_type(filename)
        if mime_type is None:
            mime_type = "application/octet-stream"  # Default fallback MIME type
        file_metadata = {
            "name": filename,
            "parents": [parent_folder_id] if parent_folder_id else [],
        }
        print(mime_type)

        # Upload the file object using MediaIoBaseUpload
        media = MediaIoBaseUpload(file_obj, mimetype=mime_type, resumable=True)

        # Upload the file to Google Drive
        drive_file = (
            self.drive_service.files()
            .create(body=file_metadata, media_body=media, fields="id")
            .execute()
        )

        file_id = drive_file.get("id")
        print(f"File uploaded successfully. File ID: {file_id}")
        sharablelink = self.get_sharable_link(file_id)
        return drive_file.get("id"), sharablelink

Integrate this function in the Flask application to upload the processed bill images directly to Google Drive.

4. Creating Notion Database Entries:

• Set up a Notion database with fields for storing bill information.

• Use the Notion API to create new entries in the database for each parsed bill, allowing users to view and manage their bills within Notion.

import requests

NOTION_TOKEN = 'your_notion_token'
NOTION_DATABASE_ID = 'your_database_id'

class NotionDatabaseHandler:
    def __init__(self, api_key=NOTION_TOKEN, database_id=NOTION_DATABASE_ID):
        self.api_key = api_key
        self.database_id = database_id
        self.headers = {
            "Authorization": f"Bearer {self.api_key}",
            "Content-Type": "application/json",
            "Notion-Version": "2022-06-28",  # Latest API version
        }

    def insert_entry(self, details):
        data = {
            "parent": {"database_id": self.database_id},
            "properties": {
                "Name": {"title": [{"text": {"content": details['supplier_name']}}]},
                "Invoice Number": {"rich_text": [{"text": {"content": details['invoice_number']}}]},
                "Date": {"date": {"start": details['date']}},
                "Tax": {"number": details['tax']},
                "Net Amount": {"number": details['net_amount']},
                "Category": {"select": {"name": details['category']}}
            }
        }
        response = requests.post('https://api.notion.com/v1/pages', headers=self.headers, json=data)
        if response.status_code == 200:
            print('Notion entry created successfully')
        else:
            print(f'Failed to create Notion entry: {response.content}')

This function sends a POST request to the Notion API to create a new entry with the parsed bill details.

5. Viewing and Tracking Bills:

• Provide a user-friendly dashboard within the Flask application where users can view and manage their bills. Use data from the Notion database to populate the dashboard, allowing for easy tracking and filtering by date, category, or amount.

Benefits

• Time-Saving: Automates the manual task of tracking and organizing bills, freeing up time for other tasks.

• Accuracy: Reduces errors by using AI/ML models to extract bill details reliably.

• Centralized Management: Stores all bill information in a single place, making it easy to track and manage finances.

• Open Source and Cost-Free: Built using open-source tools, this application eliminates the need for expensive third-party services, allowing you to manage your bills at no cost.

• Tax Readiness: By categorizing and organizing expenses, the application helps users stay prepared for tax season, simplifying the process of compiling deductible expenses.

• Customizable: Easily extend and modify the application to fit your specific needs or add new features.

Conclusion

This open-source bill tracker application offers a comprehensive solution for managing bills, especially for tax and expense management. By utilizing Flask, Mindee, Google Drive, and Notion, you can create a powerful tool to keep your finances organized — all at no cost. Whether you’re managing personal finances or handling corporate bills, this application simplifies the process and eliminates the need for expensive third-party services.

For those interested in exploring the full code or contributing to the project, please feel free to reach out to me at girishcodealchemy@gmail.com*. I’m excited to share the complete implementation and collaborate on further improvements!*

🚀 Stay tuned for more powerful insights into open-source tools for seamless expense and tax management. Happy organizing!!! 🌐

✈️ Github: github.com/GirishCodeAlchemy
✈️Linkedin:linkedin.com/in/vgirish10

Deploying Docker containers can be a complex process, especially when managing infrastructure at scale. Terraform, an open-source infrastructure as code (IaC) tool, simplifies this process by providing a consistent workflow for provisioning and managing infrastructure across various cloud providers. In this blog, we'll explore how to deploy Docker containers using Terraform, a powerful combination that brings automation, consistency, and scalability to your infrastructure management.

Use Case

Imagine you are responsible for deploying a microservices-based application that consists of multiple Docker containers. Each service has its own dependencies, scaling requirements, and network configurations. Managing these services manually can quickly become unmanageable. By leveraging Terraform, you can automate the deployment process, ensure consistency across environments, and easily scale services as needed.

1. Microservices Deployment: Automate the deployment of a microservices-based application consisting of multiple Docker containers.

2. Dependency Management: Handle each service's dependencies, ensuring that all necessary components are properly configured and available.

3. Scaling Requirements: Easily scale services up or down based on demand without manual intervention.

4. Network Configurations: Define and manage network configurations for seamless communication between services.

5. Environment Consistency: Ensure consistency across development, staging, and production environments by using infrastructure as code.

6. Simplified Management: Reduce the complexity of managing multiple services manually by leveraging Terraform for automated deployments.

7. Infrastructure Reproducibility: Define infrastructure as code to easily reproduce environments for testing, staging, and production.

8. Streamlined Deployment Process: Simplify and standardize the deployment process, allowing developers to focus on building and improving applications.

Requirements

Before we dive into the setup, ensure you have the following prerequisites:

1. Terraform: Install Terraform on your local machine.

2. Docker: Install Docker on your local machine.

3. Docker Hub Account: Create a Docker Hub account to store your Docker images.

4. Docker Images: Build and push your Docker images to Docker Hub.

Steps to Set Up Docker Deployment Using Terraform

1. Install Terraform and Docker

Ensure both Terraform and Docker are installed on your machine. Verify the installations by running the following commands:

terraform -v
docker -v

2. Create a Docker Compose File

Create a docker-compose.yml file that defines your services. This file will be used to describe your application's services, networks, and volumes.

version: '3'
services:
  web:
    image: girishcodealchemy/alchemy-nginx:v5
    ports:
      - "80:80"
   db:
    image: postgres:latest
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
      POSTGRES_DB: mydb
    volumes:
      - db-data:/var/lib/postgresql/data

volumes:
  db-data:

3. Write Terraform Configuration

Create a directory for your Terraform files and navigate into it:

mkdir terraform-docker
cd terraform-docker

Create a main.tf file and add the following configuration to define your Docker provider and resources:

terraform {
  required_providers {
    docker = {
      source  = "kreuzwerker/docker"
      version = "~> 3.0.2"
    }
  }

  required_version = "~> 1.5"
}

provider "docker" {
  host = "unix:///var/run/docker.sock"
}

resource "docker_network" "my_network" {
  name = "my_network"
}

resource "docker_volume" "db_data" {
  name = "db_data"
}

resource "docker_container" "web" {
  name  = "web"
  image = "girishcodealchemy/alchemy-nginx:v5"
  ports {
    internal = 80
    external = 80
  }
  networks_advanced {
    name = docker_network.my_network.name
  }
}

resource "docker_container" "db" {
  name  = "db"
  image = "postgres:latest"
  env = [
    "POSTGRES_USER=postgres",
    "POSTGRES_PASSWORD=password",
    "POSTGRES_DB=mydb"
  ]

  volumes {
    volume_name    = docker_volume.db_data.name
    container_path = "/var/lib/postgresql/data"
  }
  networks_advanced {
    name = docker_network.my_network.name
  }
}

4. Initialize Terraform

Run the following command to initialize the Terraform configuration. This command downloads the Docker provider and prepares your workspace.

terraform init

5. Perform the Terraform plan

Run the following command to create an execution plan:

terraform plan

Terraform will display a plan of the changes it will make. Review the plan carefully.

6. Apply the Terraform Configuration

Run the following command to apply the configuration and deploy your Docker containers:

terraform apply

Terraform will display a plan of the changes it will make. Review the plan and type yes to proceed with the deployment.

7. Verify the Deployment

After the deployment is complete, verify that the containers are running by using the Docker CLI:

docker ps

You should see your web, and db containers running.

8. Destroy the Docker containers

To clean up the infrastructure, run the destroy command:

terraform destroy

Exploring the Full Code on GitHub

To view the complete code for deploying Docker containers using Terraform, visit the GitHub repository. This repository contains all the necessary files and detailed instructions to help you get started with automating your Docker deployments using Terraform.

Conclusion

Using Terraform to deploy Docker containers brings automation and consistency to your infrastructure management. This approach allows you to define your infrastructure as code, making it easy to reproduce and scale your deployments across different environments. By following the steps outlined in this blog, you can streamline your deployment process and focus on building and improving your application.

🚀 Stay tuned for more empowering insights into the world of Docker and Terraform automation. Happy deploying!! 🌐

✈️ Github: github.com/GirishCodeAlchemy
✈️Linkedin:linkedin.com/in/vgirish10

Introduction

This guide focuses on a multi-phase approach to automate web deployment on the cloud. It covers setting up a basic web application deployment pipeline to serve static files using popular DevOps tools. The primary aim is to automate the process from code changes to deployment, ensuring a seamless and reliable deployment pipeline using GithubAction and Terraform Cloud.

Tools Used

Before diving into the deployment process, let's outline the key tools utilized in this automation setup:

1. Docker: A containerization platform used for building and managing Docker images.

2. Bash Scripting: Utilized for scripting tasks and automation within the deployment process.

3. Python: A programming language used for scripting and automation tasks, such as interacting with APIs and manipulating YAML files.

4. Git: A version control system used for managing code changes, branches, and pull requests.

5. Terraform: An Infrastructure-as-Code (IaC) tool used to provision and manage cloud resources.

6. Kubernetes: A container orchestration platform used to deploy, scale, and manage containerized applications.

7. Terratest: A Go library for testing infrastructure code, ensuring the reliability of Terraform configurations.

8. GitHub Actions: A CI/CD platform integrated with GitHub repositories to automate workflows and tasks.

9. Terraform Cloud: A cloud-based service for collaborative Terraform workflows, including execution and management of Terraform plans.

Workflow Overview

The automation process is divided into three phases:

Phase 1: Build and Push Docker Image

This phase focuses on automating the process of building and pushing Docker images. Here's a breakdown of the workflow:

1. Automatic Trigger: The workflow is triggered automatically upon detecting changes in the Docker folder.

2. Image Tag Versioning: By default, the image tag version is set to DOCKER_TAG=0.${github.run_number}.${github.event.number}-dev. This ensures that each build has a unique version identifier.

3. Custom Image Tag: Users have the option to provide their own image tag. They can do this by navigating to the GitHub Actions workflow and modifying the image tag parameter.

4. Workflow Execution: The workflow involves building the Docker image and pushing it to the Docker registry.

• The workflow starts with the setup of the environment.

• Phase 1 involves building the Docker image and pushing it to the Docker registry.

• Once completed, Phase 2 will commence, which involves updating the Terraform code.

This phase streamlines the process of Docker image management, ensuring consistency and reliability in the deployment pipeline.

Phase 2: Update Image Tag and Create PR

In this phase, the image tag is updated, and a pull request is created automatically. Here's how it works:

1. Continuous Integration: This phase integrates the updated image tag into the deployment pipeline, ensuring smooth and continuous deployment.

2. Image Tag Update: The image tag is updated as part of this phase to reflect the changes made to the Docker image. This ensures that the latest version of the image is used for deployment.

3. Terraform Plan Generation: The Terraform plan is generated to assess the impact of the changes made to the infrastructure. This step provides insight into the modifications that will be applied during the deployment process.

4. Pull Request Creation: Automatically creates a pull request with proposed infrastructure changes.

   • Once the image tag is updated and the Terraform plan is generated, a pull request is created automatically.

   • The pull request includes the proposed changes to the infrastructure, allowing for review and validation before deployment.

5. Workflow Execution:

   • The workflow starts with the setup of the environment.

   • Phase 2 involves updating the image tag, and generating the Terraform plan on the creation of the pull request.

   • Upon completion, Phase 3 will begin, which involves reviewing the Terraform plan and applying the changes.

This phase ensures that changes to the Docker image and infrastructure are properly evaluated and integrated into the deployment pipeline, maintaining the reliability and stability of the application.

Phase 3: Deploy the Changes

The final phase focuses on deploying the changes to the infrastructure and application:

1. Review and Approval: Before deployment, the changes proposed in the Terraform plan are reviewed and approved. This step ensures that any modifications to the infrastructure meet the necessary requirements and standards.

2. Deployment Process: Applies Terraform configuration changes to the target environment.

   • Once the changes are reviewed and approved, the deployment process is initiated.

   • The Terraform configuration is applied to the target environment, implementing the proposed changes to the infrastructure.

3. UI Access: Provides access to the deployed application via a UI link.

   • After successful deployment, users can access the deployed application via the provided UI link.

   • The link directs users to the application interface, where they can interact with the updated features and functionalities.

4. Workflow Integration:

   • Phase 3 is seamlessly integrated into the deployment workflow, following the completion of Phase 2.

   • The deployment process ensures that changes are effectively applied to the infrastructure, maintaining the availability and performance of the application.

This phase completes the deployment cycle, ensuring that changes to the infrastructure and application are successfully deployed and accessible to users. It emphasizes the importance of thorough review and approval processes to maintain the reliability and stability of the deployed environment.

Access the UI

Explore the Full Code

Curious to dive deeper into the automation setup? You can explore the full codebase and configuration files on the GitHub repository. Feel free to browse through the code, experiment with different configurations, and adapt the automation workflow to suit your specific deployment needs.

https://github.com/GirishCodeAlchemy/AutomateCloudWebDeploy

Conclusion

Automating web deployment on the cloud streamlines the deployment process, ensuring consistency, reliability, and efficiency. By leveraging Docker, Terraform, Kubernetes, and other DevOps tools, teams can create a robust deployment pipeline that accelerates software delivery and enhances overall productivity.

This comprehensive automation setup emphasizes the importance of continuous integration, infrastructure as code, and thorough review processes in maintaining the reliability and stability of deployed applications on the cloud. With automation, teams can focus more on innovation and less on manual deployment tasks, driving business growth and success in today's competitive landscape.

. . . . .

🚀 Stay tuned for more empowering insights into the world of automated deployment with GithubAction & Terrafrom Cloud. Happy automating your cloud journey!⛴️

✈️ Github: github.com/GirishCodeAlchemy
✈️Linkedin:linkedin.com/in/vgirish10

Overview

In modern software development workflows, efficient collaboration and automation play a crucial role. The "GithubAction Runtime Adhoc PR Commands" is a GitHub Action designed to streamline and automate specific tasks on pull requests, enhancing the overall development process.

The primary purpose of this GitHub Action is to enable the execution of ad-hoc commands dynamically in response to user comments on pull requests. Ad-hoc commands refer to custom, user-defined instructions that can be triggered to perform specific actions. These commands can be particularly useful for tasks such as rebasing, auto squashing, or any other custom workflow steps that developers may need during the pull request review process.

Problem Statement

Imagine a scenario where you want to execute specific commands on a pull request dynamically. GitHub Actions offer a great way to automate workflows, but sometimes you need more flexibility. Maybe you want to trigger certain tasks only when a specific comment is added to a PR. This is where ad-hoc commands come into play.

Key Features

1. Automation of Custom Commands: The GitHub Action automates the execution of custom commands specified in pull request comments. This eliminates the need for manual intervention, making the development process more efficient.

2. Dynamic Response to User Interactions: The Action responds dynamically to user comments, allowing developers to trigger specific tasks based on their needs. This flexibility is valuable in scenarios where different actions may be required for different pull requests.

3. Enhanced Workflow Flexibility: By incorporating this GitHub Action into your workflow, you gain a high degree of flexibility. Developers can interact with the pull request by simply adding specific commands in their comments, and the automation takes care of the rest.

4. Easy Integration with GitHub Actions: The Action seamlessly integrates with GitHub Actions, utilizing the power of workflows to orchestrate the execution of ad-hoc commands. This integration ensures a smooth and cohesive experience within the GitHub ecosystem.

Use Cases

• Rebasing: Developers can trigger a rebase operation on their pull requests by adding the /rebase command in a comment. This is particularly useful for keeping the codebase up-to-date with the latest changes from the main branch.

• Autosquashing: The /autosquash command allows developers to automate the process of squashing commits in their pull requests, creating a cleaner commit history.

• Custom Workflow Steps: Beyond rebase and autosquash, developers can define and execute any custom workflow steps by incorporating specific commands tailored to their project requirements.

Usage:

1. Add below action to your workflow YAML

2. Specify your custom commands on the PR comments

3. watch the magic happen as your PRs get supercharged

name: PR Adhoc commands
on:
  issue_comment:
    types: [created]
jobs:
  build:
    name: PR-Adhoc-Commands
    runs-on: ubuntu-latest
    if: >-
      github.event.issue.pull_request != '' &&
      (
        contains(github.event.comment.body, '/rebase') ||
        contains(github.event.comment.body, '/autosquash') ||
        contains(github.event.comment.body, '/rebase-autosquash')
      )
    steps:
      - name: Checkout the latest code
        uses: actions/checkout@v3
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          fetch-depth: 0

      - name: Running Adhoc commands
        uses: GirishCodeAlchemy/githubaction-runtime-adhoc-pr-commands@v1.0
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Command Usage :

Solution Overview

To address this challenge, we'll create a GitHub Action that responds to comments on a pull request and executes custom commands. The key components of the solution include:

1. GitHub Action Workflow: Define a workflow that triggers on pull request comments.

2. Custom Action Script: Write a script that interprets the comment content and executes the desired commands.

3. GitHub API Integration: Use the GitHub API to fetch information about the pull request and perform necessary actions.

Let's break down the implementation step by step.

Implementation

1. GitHub Action Workflow

name: Adhoc Commands

on:
  issue_comment:
    types:
      - created

jobs:
  adhoc_commands:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@v2

    - name: Run Adhoc Commands
      uses: GirishCodeAlchemy/githubaction-runtime-adhoc-pr-commands@v1.0
      env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

This workflow triggers on new comments (issue_comment event) and uses a custom action (which we'll create next) to run the ad-hoc commands.

2. Custom Action Script

# alchemy.py

import subprocess

def execute_adhoc_commands(comment_body):
    # Logic to parse and execute commands
    # Example: Execute "echo Hello World" if the comment contains "/run-command"
    if "/run-command" in comment_body:
        subprocess.run(["echo", "Hello World"])

# Entry point
if __name__ == "__main__":
    comment_body = input("Enter the comment body: ")
    execute_adhoc_commands(comment_body)

In this script, we define a function execute_adhoc_commands that interprets the comment content and executes specific commands. Customize this logic according to your requirements.

3. GitHub API Integration

Extend the script to fetch additional information about the pull request using the GitHub API. For example, you can retrieve the PR number, author, or any other relevant details.

# alchemy.py

import subprocess
import os
import requests

def get_pr_info():
    github_token = os.getenv("GITHUB_TOKEN")
    pr_number = os.getenv("PR_NUMBER")
    repo = os.getenv("GITHUB_REPOSITORY")

    headers = {
        "Authorization": f"Bearer {github_token}",
        "Accept": "application/vnd.github.v3+json"
    }

    response = requests.get(
        f"https://api.github.com/repos/{repo}/pulls/{pr_number}",
        headers=headers
    )

    return response.json()

def execute_adhoc_commands(comment_body):
    pr_info = get_pr_info()
    pr_author = pr_info["user"]["login"]
    pr_number = pr_info["number"]

    # Logic to parse and execute commands
    # Example: Execute "echo Hello World" if the comment contains "/run-command"
    if "/run-command" in comment_body:
        subprocess.run(["echo", f"Hello World from PR {pr_number} by {pr_author}"])

# Entry point
if __name__ == "__main__":
    comment_body = input("Enter the comment body: ")
    execute_adhoc_commands(comment_body)

Here, we've added a get_pr_info function to fetch PR information. Update the logic in execute_adhoc_commands to utilize this information as needed.

Github Marketplace:

The action is now available on the public GitHub Actions Marketplace: PR Runtime Adhoc commands

Conclusion

With this GitHub Action setup, you have a flexible way to execute custom commands dynamically based on comments in your pull requests. Whether you need to trigger specific tests, deploy to staging, or perform any other task, this solution provides the extensibility you need.

🚀 Stay tuned for more empowering insights into the world of GitHub Actions and PR runtime adhoc commands. Happy automating!! 🌐

✈️ Github: github.com/GirishCodeAlchemy
✈️Linkedin:linkedin.com/in/vgirish10

In today's dynamic landscape of cloud infrastructure, managing access and authentication across multiple AWS accounts can be a challenging task. As organizations scale their presence on the AWS cloud, the need for a robust Single Sign-On (SSO) solution becomes increasingly evident. This blog aims to delve into the details of AWS SSO automation, offering insights into efficiently orchestrating user access in complex multi-account environments. We will delve into the challenges of manual SSO creation and present a comprehensive guide on transitioning to an automated approach, providing a roadmap for enhancing security, scalability, and overall operational efficiency.

Challenges of Manual SSO Creation:

1. Tedious User Access Configuration:

   • Manually configuring individual user access across multiple AWS accounts is a time-consuming and labor-intensive task.

2. Potential for Human Errors:

   • The manual nature of SSO creation introduces the risk of human errors in permission assignments, leading to security vulnerabilities and compliance concerns.

3. Inefficiencies in Permission Management:

   • Managing permissions manually can result in inefficiencies, as each user and group may have unique configurations that must be meticulously handled.

4. Operational Overhead:

   • The manual approach adds operational overhead, requiring significant time and effort to ensure accurate and consistent SSO configurations across accounts.

5. Limitations in Scalability:

   • As organizations expand their AWS footprint, the limitations of manual SSO creation become more apparent, hindering scalability and adaptability to evolving infrastructure needs.

6. Security Concerns:

   • Manual processes increase the risk of security vulnerabilities, as errors in permission assignments may lead to unauthorized access and potential data breaches.

7. Complexity with Growing AWS Presence:

   • The challenges amplify as the AWS presence grows, making manual SSO creation increasingly complex and prone to inconsistencies.

The Road to Automation:

Automating AWS SSO processes not only addresses the challenges posed by manual configurations but also brings forth a range of benefits. A well-implemented automation strategy enhances security by enforcing consistent access policies and reduces the risk of human errors in permission assignments. Scalability is significantly improved, allowing organizations to seamlessly adapt to the evolving demands of their cloud infrastructure. Additionally, the time and effort saved through automation can be redirected towards more strategic initiatives, fostering innovation and agility.

Key Components of AWS SSO Automation:

1. Terraform Infrastructure as Code (IaC): Leveraging Terraform, an IaC tool, facilitates the codification of AWS SSO configurations. This not only streamlines the provisioning of SSO resources but also ensures version-controlled, repeatable, and auditable infrastructure changes.

2. AWS SSO Terraform Module: Introducing the GirishCodeAlchemy AWS SSO Terraform Module, now available on the Terraform Registry. Designed to simplify the orchestration of AWS SSO resources, this module empowers your environment with the flexibility to manage multiple hierarchies seamlessly. Easily provision and configure AWS SSO components, including identity store, users, groups, and permissions, using this modular and user-friendly solution.

   

Prerequisites

Before diving into the automation process, ensure you have the following prerequisites:

• Identity Store configured

• Relevant AWS accounts and permission sets defined

Terraform Setup

We'll use Terraform for this automation. Ensure you have Terraform installed, and set up your project directory.

Terraform Module Usage:

Simplify your AWS SSO implementation with the GirishCodeAlchemy/sso-module/aws Terraform Module. Use the provided example configuration to get started:

data "aws_ssoadmin_instances" "ssoadmin" {
  # Retrieve information about existing SSO instances, if needed.
}

module "sso" {
  source  = "GirishCodeAlchemy/sso-module/aws"
  version = "1.0.0"

  identity_store_ids           = data.aws_ssoadmin_instances.ssoadmin.identity_store_ids
  ssoadmin_instance_arns       = data.aws_ssoadmin_instances.ssoadmin.arns
  sso_user_configmap           = var.sso_user_configmap
  sso_groups_configmap         = var.sso_groups_configmap
  sso_permissionsets_configmap = var.sso_permissionsets_configmap
  sso_account_configmap        = var.sso_account_configmap
}

Variables Configuration Examples:
The GirishCodeAlchemy AWS SSO Terraform Module includes a set of variables for configuring users, groups, permission sets, and accounts. Here are some examples to guide you through the configuration process:

• identity_store_ids

    identity_store_ids = ["d-dssdaa"] # data.aws_ssoadmin_instances.ssoadmin.identity_store_id
    ssoadmin_instance_arns = ["arn:aws:sso:::instance/ssoins-xxxxxxxxxx"] # data.aws_ssoadmin_instances.ssoadmin.arns
  

• sso_user_configmap

    sso_user_configmap = {
      girish1 = {
        display_name = "Girish V"
        user_name    = "girish1"
        given_name   = "Girish"
        family_name  = "V"
        email        = "girish1@example.com"
      },
      girish2 = {
        display_name = "Girish V"
        user_name    = "girish2"
        given_name   = "Girish"
        family_name  = "V"
        email        = "girish2@example.com"
      }
    }
  

• sso_groups_configmap

    sso_groups_configmap = {
      "L1-devops-group" = {
        display_name = "L1-devops-group"
        description  = "This is AWS L1 Devops Group"
        users        = ["girish1", "girish2"]
      },
      "L1-Admin-group" = {
        display_name = "L1-Admin-group"
        description  = "This is AWS L1 Admin Group"
        users        = ["girish1"]
      }
    }
  

• sso_permissionsets_configmap

    sso_permissionsets_configmap = {
      "SSM-Admin-permissionset" = {
        name                = "SSM-Admin-permissionset"
        description         = "Sample Admin permissionset"
        managed_policy_arns = ["arn:aws:iam::aws:policy/AmazonEC2FullAccess", "arn:aws:iam::aws:policy/job-function/ViewOnlyAccess"]
        inline_policy       = <<-EOF
        {
            "Version": "2012-10-17",
            "Statement": [
                {
                "Effect": "Allow",
                "Action": [
                    "lambda:*",
                ],
                "Resource": "*"
                }
            ]
        }
        EOF
      },
      "SSM-testing-permissionset" = {
        name                = "SSM-testing-permissionset"
        description         = "Sample testing permissionset"
        managed_policy_arns = ["arn:aws:iam::aws:policy/AmazonEC2FullAccess"]
        inline_policy       = <<-EOF
        {
            "Version": "2012-10-17",
            "Statement": [
                {
                "Effect": "Allow",
                "Action": [
                    "s3:ListAllMyBuckets",
                    "s3:GetBucketLocation"
                ],
                "Resource": "arn:aws:s3:::*"
                }
            ]
        }
        EOF
      }
    }
  

• sso_account_configmap

    sso_account_configmap = {
      "1xxxxxxxx" = {
        users = {
          girishcodealchemy = { username = "girishcodealchemy", permissionset = ["SSM-testing-permissionset", "SSM-Admin-permissionset"] }
        }
        groups = {
          L1devopsgroup = { groupname = "L1-devops-group", permissionset = ["SSM-testing-permissionset", "SSM-Admin-permissionset"] }
        }
      },
      "2xxxxxxxx" = {
        users = {
          girishcodealchemy = { username = "girishcodealchemy", permissionset = ["SSM-testing-permissionset"] }
        }
        groups = {
          L1devopsgroup = { groupname = "L1-devops-group", permissionset = ["SSM-testing-permissionset"] },
          L1AdminGroup  = { groupname = "L1-Admin-group", permissionset = ["SSM-Admin-permissionset"] }
        }
      }
    }
  

Terraform plan and Deploy

Users Created

Groups and Members Created

Permission Sets Created

Mapped Users , Groups & Permission sets to the Accounts

Implementation Resources:

A Guide for Creating Resources from Scratch

Setting up AWS SSO automation from scratch involves creating necessary resources using Terraform. This guide outlines the steps to implement your own AWS SSO solution.

1. AWS Identity Store User Configuration:

Automate the creation and configuration of AWS Identity Store users using the aws_identitystore_user resource. This resource creates a user in the AWS Identity Store. The user’s display name, username, given name, family name, and email are all configurable.

• Define user configurations in the sso_user_configmap variable. The example code snippet illustrates the Terraform implementation:

resource "aws_identitystore_user" "aws_user" {
  for_each          = var.sso_user_configmap
  identity_store_id = element(var.identity_store_ids, 0)

  display_name = each.value.display_name
  user_name    = each.value.user_name

  name {
    given_name  = each.value.given_name
    family_name = each.value.family_name
  }

  emails {
    primary = true
    value   = each.value.email
  }

  lifecycle {
    ignore_changes = [
      name,
      emails,
    ]
  }
}

1. AWS Identity Store Group Configuration:

Start by defining AWS Identity Store groups and their memberships. Use the aws_identitystore_group and aws_identitystore_group_membership resources to create the necessary group structures.

• Define group and member configurations in the sso_groups_configmap variable. The example code snippet showcases the Terraform implementation:

resource "aws_identitystore_group" "aws_group" {
  for_each          = var.sso_groups_configmap
  identity_store_id = element(var.identity_store_ids, 0)
  display_name      = each.value.display_name
  description       = each.value.description
}

resource "aws_identitystore_group_membership" "example" {
  for_each          = { for identity in local.flattened_groups : "${identity.group_key}-${identity.user_key}" => identity }
  identity_store_id = element(var.identity_store_ids, 0)
  group_id          = aws_identitystore_group.aws_group[each.value.group_key].group_id
  member_id         = aws_identitystore_user.aws_user[each.value.user_key].user_id
}

1. AWS SSO Admin Permission Sets:

Define AWS SSO permission sets that align with your organization's access policies. Use aws_ssoadmin_permission_set aws_ssoadmin_permission_set_inline_policy and aws_ssoadmin_managed_policy_attachment resources.

• Define Permission Set , Inline policy and managed_policy in the sso_permissionsets_configmap variable.

// Terraform code for AWS SSO Admin Permission Set
resource "aws_ssoadmin_permission_set" "permissionset" {
  for_each     = var.sso_permissionsets_configmap
  name         = each.value.name
  description  = each.value.description
  instance_arn = tolist(var.ssoadmin_instance_arns)[0]
}

resource "aws_ssoadmin_permission_set_inline_policy" "inline_policy" {
  for_each           = var.sso_permissionsets_configmap
  inline_policy      = each.value.inline_policy
  instance_arn       = tolist(var.ssoadmin_instance_arns)[0]
  permission_set_arn = aws_ssoadmin_permission_set.permissionset[each.key].arn
}

resource "aws_ssoadmin_managed_policy_attachment" "managed_policy_attachment" {
  for_each           = { for i in local.flattened_managed_policy_attachments : "${i.permission_set_name}-${i.managed_policy_arn}" => i }
  instance_arn       = tolist(var.ssoadmin_instance_arns)[0]
  managed_policy_arn = each.value.managed_policy_arn
  permission_set_arn = aws_ssoadmin_permission_set.permissionset[each.value.permission_set_name].arn

  depends_on         = [aws_ssoadmin_account_assignment.sso_account_user, aws_ssoadmin_account_assignment.sso_account_group, ]
}

[!NOTE] Adding an explicit dependency on the account assignment resource will allow the managed policy attachment to be safely destroyed prior to the removal of the account assignment.

1. AWS SSO Admin Account Assignment:

Automate the assignment of AWS SSO permission sets to groups or users across multiple accounts using aws_ssoadmin_account_assignment.

• Define Users, Groups, list of permission sets mapped in the sso_account_configmap variable.

// Terraform code for AWS SSO Admin Account Assignment
resource "aws_ssoadmin_account_assignment" "sso_account_user" {
  for_each = { for config in local.flatten_user_configurations : "${config.account}.${config.username}.${config.permissionset}" => config }

  instance_arn       = tolist(var.ssoadmin_instance_arns)[0]
  target_id          = each.value.account
  target_type        = "AWS_ACCOUNT"
  principal_id       = data.aws_identitystore_user.aws_user["${each.value.account}.${each.value.username}"].user_id
  principal_type     = "USER"
  permission_set_arn = data.aws_ssoadmin_permission_set.aws_user_permissionset[each.key].arn
}

resource "aws_ssoadmin_account_assignment" "sso_account_group" {
  for_each = { for config in local.flatten_group_configurations : "${config.account}.${config.groupname}.${config.permissionset}" => config }

  instance_arn       = tolist(var.ssoadmin_instance_arns)[0]
  target_id          = each.value.account
  target_type        = "AWS_ACCOUNT"
  principal_id       = data.aws_identitystore_group.aws_group["${each.value.account}.${each.value.groupname}"].group_id
  principal_type     = "GROUP"
  permission_set_arn = data.aws_ssoadmin_permission_set.aws_group_permissionset[each.key].arn
}

Conclusion

Automating AWS SSO configurations for complex multi-account environments brings efficiency and consistency to access management. This blog post provided a basic guide using Terraform. Tailor these configurations to fit your organization's specific requirements and enjoy a streamlined AWS SSO experience.

🚀 Stay tuned for more empowering insights into the world of cloud infrastructure management. Happy SSO automating! 🌐

✈️ Github: github.com/GirishCodeAlchemy
✈️ Linkedin: linkedin.com/in/vgirish10

Jupyter Notebook is a powerful tool for interactive computing that supports various programming languages. While Python is the default language, you can extend its capabilities by adding kernels for other languages such as Node.js and Golang. This blog post will guide you through the process of setting up Jupyter Notebook kernels for both Golang and Node.js.

Prerequisites:

Before we start, make sure you have the following installed:

1. Jupyter Notebook: Install Jupyter Notebook on your machine by following the instructions on the official website (https://jupyter.org/install).

2. Golang: Install Golang from the official website (https://golang.org/dl/).

3. Node.js: Install Node.js from the official website (https://nodejs.org/).

Setting Up Jupyter Kernel for Golang:

1. Install Gophernotes:

   Gophernotes is a Go kernel for Jupyter Notebooks. Install it using:

    go get github.com/gopherdata/gophernotes
   

2. Configure Jupyter to Use Gophernotes:

   Configure Jupyter to recognize Gophernotes as a kernel:

    mkdir -p ~/.local/share/jupyter/kernels/gophernotes
    cp $(go env GOPATH)/pkg/mod/github.com/gopherdata/gophernotes@v0.7.5/kernel/* ~/.local/share/jupyter/kernels/gophernotes
    cd ~/.local/share/jupyter/kernels/gophernotes
    chmod +w ./kernel.json # in case copied kernel.json has no write permission
    $ sed "s|gophernotes|$(go env GOPATH)/bin/gophernotes|" < kernel.json.in > kernel.json
   

   Ensure that the paths in the kernel.json file point to the correct Gophernotes installation.

3. Launch Jupyter Notebook:

   Start Jupyter Notebook:

    jupyter notebook
   

   Create a new notebook and choose the Gophernotes kernel from the "Kernel" menu.

4. 

Setting Up Jupyter Kernel for Node.js:

1. Install ijavascript:

   Open your terminal and install the ijavascript kernel using the following command:

    npm install -g ijavascript
   

2. Install the Jupyter Kernel:

   Register the ijavascript kernel with Jupyter Notebook:

    ijsinstall
   

   This command will create a kernel specification for ijavascript.

3. Launch Jupyter Notebook:

   Start Jupyter Notebook using the following command:

    jupyter notebook
   

   In the Jupyter Notebook interface, create a new notebook and select the ijavascript kernel from the "Kernel" menu.

4. 

Conclusion:

Congratulations! You have successfully set up Jupyter Notebook kernels for Golang and Node.js. Now you can leverage the power of Jupyter for interactive development in these languages. Feel free to explore and experiment with your code in a collaborative and interactive environment.

🚀 Stay tuned for more insightful content as we explore the diverse realms of programming. Happy coding and notebook-ing!🚀

✈️ Github: github.com/GirishCodeAlchemy
✈️ Linkedin: linkedin.com/in/vgirish10

What are DevContainers?

DevContainers are a way to define and share development environments using containers. These containers encapsulate all the dependencies, tools, and configurations needed to run your application. By leveraging Docker under the hood, DevContainers ensures that everyone on your team has the same development environment, reducing "It works on my machine" issues.

Setting Up DevContainers

Prerequisites

Before diving into DevContainers, ensure you have the following installed:

• Docker

• Visual Studio Code

Installation

1. Install the Dev Containers / Remote containers extension for Visual Studio Code.

2. Open your project in Visual Studio Code.

3. Click on the green/blue icon in the bottom-left corner of the window, and select Dev Containers: Add Development Container Configuration Files.

This will create a .devcontainer folder with the necessary configuration files.

Key Features and Benefits

1. Isolation

DevContainers provides a fully isolated environment for your project. Dependencies and configurations are encapsulated within the container, preventing conflicts with your local setup.

2. Consistency

Every team member, regardless of their local setup, gets the same development environment. This consistency minimizes compatibility issues and accelerates onboarding for new developers.

3. Portability

DevContainers can be easily shared and replicated. Collaborators, contributors, or CI/CD pipelines can use the same container configuration, ensuring a uniform environment across different machines.

Working with DevContainers

Configuring DevContainers

The .devcontainer folder contains a devcontainer.json file where you can specify the container image, extensions, and settings. Customize this file to tailor the development environment to your project's needs.

Developing Inside Containers

1. Open your project in Visual Studio Code.

2. Click on the green/blue icon in the bottom-left corner.

3. Select Dev-Containers: Reopen in Container.

Your project will now be running inside the DevContainer, with access to all the tools and dependencies defined in the configuration.

Use Cases and Scenarios

1. Multi-language Projects

DevContainers are ideal for projects that use multiple programming languages. Each language can have its own container, avoiding conflicts between language-specific dependencies.

2. Team Collaboration

With DevContainers, everyone on the team gets a consistent development environment. This minimizes friction caused by differences in operating systems or individual configurations.

3. Open Source Contributions

For open-source projects, DevContainers simplify the onboarding process for contributors. They can start contributing without worrying about setting up dependencies locally.

Troubleshooting and Tips

While DevContainers generally provide a seamless experience, here are a few tips for troubleshooting:

• Ensure Docker is running.

• Check for any errors in the Visual Studio Code output panel.

• If facing issues with extensions, make sure they are compatible with the DevContainer environment.

DevContainers Template Reference

For your convenience, I have created a DevContainers template that you can use as a starting point for your projects. This template includes a well-configured devcontainer.json file and example settings to kickstart your containerized development.

Github: https://github.com/GirishCodeAlchemy/it-works-on-my-machine.git

Feel free to clone or fork this repository and tailor it to your project requirements. We encourage contributions and improvements from the community to make this template even more versatile and useful.

Conclusion

DevContainers offers a practical solution for creating consistent, isolated, and portable development environments. By adopting DevContainers, teams can streamline their workflows, enhance collaboration, and simplify the onboarding process for new members. Start containerizing your development environments today and enjoy a more reliable and efficient development experience.

Additional Resources

• DevContainers Documentation

• Docker Documentation

• Visual Studio Code Remote Development

Call to Action

Have you tried DevContainers in your projects? Share your experiences, tips, and tricks in the comments below. Let's build a community of developers leveraging DevContainers for a better development experience!

🚀 Stay tuned for more updates as we continue to push the boundaries of what's possible in the world of development and containerization. Happy exploring! 🚀

✈️ Github: https://github.com/GirishCodeAlchemy ✈️ Linkedin: https://www.linkedin.com/in/vgirish10/

In today's fast-paced world, keeping track of news sentiments is crucial for various applications, ranging from financial market predictions to understanding public opinion. In this blog post, we will explore a comprehensive project that combines the power of Kafka, Hadoop, Spark, and machine learning to perform sentiment analysis on news articles.

Project Overview

Objective

Our project goes beyond traditional sentiment analysis by creating an end-to-end pipeline that seamlessly integrates real-time data streaming, distributed storage, and advanced analytics. The goal is to provide users with a comprehensive view of the sentiments expressed in news articles.

Technologies Used

1. Kafka: This robust event streaming platform ensures the seamless flow of news data, making it instantly available for analysis.

2. Hadoop: The cornerstone of our project, Hadoop's distributed file system guarantees the scalability needed to handle vast amounts of news data efficiently.

3. Spark: With its lightning-fast data processing capabilities, Spark transforms raw news data into a format suitable for sentiment analysis, all in near real-time.

4. Machine Learning: Our sentiment analysis model, trained on diverse datasets, showcases the capabilities of cutting-edge machine learning algorithms.

Project Components

1. Kafka Setup

Our project kicks off with the implementation of Kafka, acting as a robust and distributed event streaming platform. This ensures that news data is ingested and made available for analysis in real-time.

2. Hadoop Integration

Hadoop seamlessly integrates into our architecture, providing the reliability and scalability needed to handle the vast volumes of data generated by an ever-evolving news landscape.

3. Apache Spark for ETL

Spark takes center stage for Extract, Transform, Load (ETL) processes. It efficiently cleanses and pre-processes news data, ensuring that only relevant information is passed through for sentiment analysis.

4. Machine Learning Model

Our machine learning model, an exemplar of innovation, is trained to discern sentiments within news articles accurately. Its ability to adapt to varying tones and contexts sets it apart in the field of sentiment analysis.

5. Results and Visualization

The project's culmination involves presenting the sentiment analysis results through visually compelling visualizations. Stakeholders can seamlessly grasp nuanced sentiment trends, making informed decisions based on the data-driven insights provided.

Impressive Highlights

Real-time Processing

One of the project's standout features is its real-time data processing capabilities. The seamless flow of news data through Kafka, coupled with Spark's speed, ensures that our sentiment analysis is always up-to-date and reflective of the latest trends.

Scalability

Our project is designed with scalability in mind. Hadoop's distributed file system allows the storage and management of large datasets, ensuring that the pipeline can handle increasing volumes of news articles without compromising performance.

Predictive Analytics

The machine learning model is not just a sentiment analyzer; it's a predictive analytics tool. Its ability to adapt to emerging sentiment patterns positions our project as a proactive solution for staying ahead of the news curve.

User-friendly Visualization

We believe in making data accessible. Our visualizations are not just informative but also user-friendly, allowing stakeholders, regardless of technical background, to grasp sentiment trends effortlessly.

Running the Project

For those eager to replicate or extend this project, detailed steps are provided to guide you through the process. From setting up Kafka to training your machine learning model, we ensure that our innovation is accessible to all.

https://github.com/GirishCodeAlchemy/News-sentiment-ML-ETL-pipeline

Conclusion

This project stands as a testament to the incredible possibilities that emerge when cutting-edge technologies unite. By combining Kafka, Hadoop, Spark, and machine learning, we've created a sentiment analysis pipeline that not only impresses with its technical prowess but also empowers users with actionable insights derived from the complex world of news sentiments.

Stay tuned for more updates as we continue to push the boundaries of what's possible in the realm of sentiment analysis. Happy exploring!

✈️ Github: https://github.com/GirishCodeAlchemy
✈️ Linkedin: https://www.linkedin.com/in/vgirish10/

Managing and understanding file sizes within a directory is a common task for developers, sysadmins, sre, and users. To simplify this process, I've created a tool called TopNDiskAnalyzer. This tool allows users to analyze and retrieve the top N files and folders based on their sizes within a specified directory.

Features

1. Concurrent Processing

TopNDiskAnalyzer leverages concurrent processing to efficiently analyze files and folders concurrently. This is achieved by utilizing goroutines in Go, ensuring optimal performance, especially when dealing with large directory structures.

1. Dynamic Top N

One notable feature of TopNDiskAnalyzer is its ability to dynamically set the 'N' value for the top files and folders. Users can provide the desired count using the --top command-line argument. If not specified, the default value is 10.

1. Informative Output

The tool produces insightful output, displaying the top files and folders along with their sizes in a human-readable format. The sizes are presented in KB, MB, GB, etc., making it easy for users to comprehend.

Getting Started

Installation

Download Binary

You can download the precompiled binary for your operating system from the releases page.

Linux:

# 64-bit
wget https://github.com/GirishCodeAlchemy/TopNDiskAnalyzer/releases/download/v1.0.0/tdf-linux-amd64 -O tdf
chmod +x tdf
sudo mv tdf /usr/local/bin/

# 32-bit
wget https://github.com/GirishCodeAlchemy/TopNDiskAnalyzer/releases/download/v1.0.0/tdf-linux-386 -O tdf
chmod +x tdf
sudo mv tdf /usr/local/bin/

macOS:

# 64-bit
wget https://github.com/GirishCodeAlchemy/TopNDiskAnalyzer/releases/download/v1.0.0/tdf-darwin-amd64 -O tdf
chmod +x tdf
sudo mv tdf /usr/local/bin/

Windows:

Download the tdf.exe from the releases page.

Build from Source

If you prefer to build from a source, make sure you have Go installed. Clone the repository and run the following commands:

git clone https://github.com/GirishCodeAlchemy/TopNDiskAnalyzer.git
cd TopNDiskAnalyzer
go build -o tdf main.go
sudo mv tdf /usr/local/bin/

This will provide users with clear instructions on how to download and install your TopNDiskAnalyzer tool on various operating systems.

Usage

Once installed, you can use the tool by running the following command:

tdf [options] [directory_path]

Options

• --dir: Specifies the directory path. Defaults to the current directory.

• --top: Specifies the number of top items to display. Defaults to 10.

• --help: Displays helpful information about the program and its options.

The command recursively analyzes the specified directory, calculating the sizes of files and folders, and then displays the top items based on their sizes in descending order.

Examples

# Analyze the current directory, display top 10 items
tdf

# Analyze a specific directory, display top 15 items
tdf --top 15 /path/to/directory 

# Analyze a specific directory (using positional argument), display top 10 items
tdf /path/to/directory

# Help
tdf --help

Screenshots

Here are some screenshots illustrating the usage of TopNDiskAnalyzer:

1. Help Command

   

2. Relative Path

   

3. Absolute Path

   

4. Top Option

   

Conclusion

TopNDiskAnalyzer provides a straightforward solution for users who need quick insights into the largest files and folders within a directory. Its concurrent processing, dynamic top N feature, and informative output make it a valuable tool for various use cases.

To get started, check out the GitHub repository and feel free to contribute or provide feedback.

🙏Thank you for reading...

✈️ Linkedin: https://www.linkedin.com/in/vgirish10/

Let's explore the benefits of this tool in two specific use cases:

1. Storing Job Descriptions for Applied Companies

One practical application of the Web Page Capture Tool is in the realm of job hunting. When applying to various companies, job seekers often come across interesting job descriptions and company information on their websites. The tool allows you to capture these web pages and store them as images or PDFs, providing a convenient way to:

• Document Job Opportunities: Capture and store job descriptions along with relevant details from company websites.

• Offline Reference: Have a local repository of job descriptions for applied companies, allowing you to refer to them even when offline.

• Organized Archive: Create an organized archive of job opportunities, making it easier to review and compare roles from different companies.

2. Creating Document Versions of Web Pages

The Web Page Capture Tool is not limited to capturing job descriptions; it can be a versatile solution for creating document versions of any web page. Consider the following benefits:

• Web Page Archiving: Capture web pages for archival purposes, preserving the content as it appears at a specific point in time.

• Documentation of Online Content: Easily document online articles, tutorials, or any other web-based content by generating PDFs.

• Offline Reading: Create offline versions of web pages for later reading or reference.

• Versioning of Online Resources: Capture different versions of web pages over time, useful for tracking changes or maintaining a historical record.

Overview

The Web Page Capture Tool is a Go-based command-line utility designed to interact with headless Chrome and capture web pages in two formats: screenshots (images) and PDFs. The tool is simple to use, allowing users to specify a URL, the desired output format (image or PDF), and an optional filename. If no filename is provided, the tool defaults to a predefined name.

Getting Started

To get started with the Web Page Capture Tool, you can download the pre-built binary from the GitHub repository

1. Download the latest binary from the Releases

    wget https://github.com/GirishCodeAlchemy/WebPageCapture/releases/download/v1.0.0/web-capture-tool
   

2. Once downloaded, make the binary executable:

    chmod +x web-capture-tool
   

3. Move the binary to a directory in your system's PATH, so you can run it from any location:

    sudo mv web-capture-tool /usr/local/bin/
   

Usage

Help and Usage:

For detailed information about the available flags and options, you can use the following command:

web-capture-tool --help

Capture Screenshot:

To capture a screenshot of a web page, run the following command:

./web-capture-tool <URL> image [filename]

Example:

./web-capture-tool https://example.com image output.jpg

Generate PDF:

To generate a PDF of a web page, use the following command:

./web-capture-tool <URL> pdf [filename]

Example:

./web-capture-tool https://example.com pdf output.pdf

Customization and Extension

The tool is intentionally kept minimal to serve as a foundation that can be extended based on specific requirements. Users familiar with Go can explore the code and customize the tool to suit their needs. Additionally, integrating it into existing automation pipelines or scripts is straightforward.

Conclusion

The Web Page Capture Tool provides a quick and efficient way to capture web pages in both image and PDF formats. Its simplicity makes it suitable for various use cases, and the underlying Go and Chrome integration offers flexibility and reliability. Whether you need to document web pages, perform visual testing, job hunting, and content documentation, or automate PDF generation, this tool can be a valuable addition to your toolkit.

Feel free to explore, modify, and adapt the Web Page Capture Tool based on your specific needs. Whether you're building a repository of job opportunities or archiving online content, this tool can enhance your web capture workflows. Give it a try, and let us know how it fits into your use cases!

Give it a try, and let us know how it fits into your web capture workflows!

🙏Thank you for reading...

✈️ Linkedin: https://www.linkedin.com/in/vgirish10/

What is Docker? What is a container? Why are they important?

Docker is like a magic box for software. It holds everything an app needs to run, like the app itself, tools, and libraries. It's like a lunchbox for your software.

Software is often a mix of different parts, like web servers, databases, and other tools. When people build software, they sometimes face problems, like making sure it works on different computers.

Docker containers help with this. They keep everything together, so the software runs the same everywhere. It's like a lunchbox that keeps your food fresh.

So, Docker makes it easier for developers to create, share, and run their software. It's a helpful tool for building and managing software.

Docker

Docker is a suite of platform-as-a-service products that utilize OS-level virtualization to distribute software within self-contained units called containers. These containers share the underlying operating system kernel but are isolated from one another. Each container includes its own software, libraries, and configuration files.

• Docker is known for its swift(Fast) startup times.

• In Docker, an 'image' serves as a preconfigured template.

• Containers, in contrast, represent active instances of these images. They run in isolation, maintaining their own environments and sets of processes

• 

What is Orchestration? Why Do We Need Container Orchestration?

• Orchestration in system administration involves the automated configuration, coordination, and management of computer systems and software. Numerous tools are available for automating server configuration and management.

In container orchestration, the focus is on monitoring application load and performance while ensuring the swift deployment of new containers in case of failures.

Container orchestration automates various aspects of coordinating and managing containers, primarily concentrating on the lifecycle of containers and their dynamic environments.

Container orchestration serves to automate the following tasks at scale:

• Configuring and scheduling containers

• Provisioning and deploying containers

• Managing container availability

• Configuring applications based on the containers they run in

• Scaling containers to evenly distribute application workloads across infrastructure

• Allocating resources among containers

• Handling load balancing, traffic routing, and service discovery for containers

• Monitoring the health of containers

• Ensuring secure interactions between containers

Container orchestration comprises a set of tools and scripts that aid in hosting and managing containers.

There are multiple container orchestration options available today, including Docker Swarm, Kubernetes, Mesos (Apache), and cloud solutions.

Kubernetes

Kubernetes is an open-source container orchestration system that automates application deployment, scaling, and management.

• Kubernetes is like a smart manager for your apps. It helps them deploy, grow, and stay organized. To talk to Kubernetes, you use a tool called kubectl which is a command line interface

1. ETCD: Your Kubernetes Memory

   Think of etcd as Kubernetes' memory. It stores information about the nodes in a simple key-value format, such as which containers are on which node and when they were loaded. All this data is kept in a special place called etcd.

2. Kube-scheduler: The Smart Deployer

   Kube-scheduler is like a smart organizer. It finds the perfect node to put your container based on its needs, how much space is available on worker nodes, and any other rules you've set.

3. Controller Manager: The Node Manager

   The controller manager is in charge of handling new nodes joining the cluster and dealing with situations where nodes go missing. It's like the person who welcomes new members to a club.

4. Kube-api server: The Commander

   The kube-api server is the commander of the Kubernetes army. It manages everything that happens within the cluster and provides an interface for people outside the cluster to make things happen.

5. Kubelet: The Watchful Guard

   Kubelet is like a guardian for each node in the cluster. It takes orders from the kube-api server and makes sure everything runs smoothly. The kube-api server also keeps an eye on kubelet to know what's happening with nodes and containers.

6. Kube-proxy: The Traffic Cop

   Kube-proxy makes sure the traffic rules are set on worker nodes so containers can talk to each other.

7. Kubernetes Runtime Environment: Where the Magic Happens

8. Kubectl: Your Command Center

   Kubectl is like a remote control for your Kubernetes cluster. It's how you give orders and get things done. You can tell kubectl what to do with simple commands.

9. Pods: Where Your Apps Live

   A pod is like a cozy home for your apps. It's the smallest thing Kubernetes can manage, and it's where your containers live. Think of a pod as a special place where different parts of your app can talk to each other. But be careful – if a pod's home (the node) goes away, the pod disappears too. It's not built to survive the destruction of its home.

Kubernetes Deployment Script

apiVersion: apps/v1
kind: Deployment
metadata:
  name: your-app-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: your-app
  template:
    metadata:
      labels:
        app: your-app
    spec:
      containers:
        - name: your-app-container
          image: your-app-image:latest
          ports:
            - containerPort: 80
          env:
            - name: ENV_VARIABLE_NAME
              value: "your-environment-value"
          resources:
            requests:
              memory: "64Mi"
              cpu: "250m"
            limits:
              memory: "128Mi"
              cpu: "500m"
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                    - key: node-label-key
                      operator: In
                      values:
                        - node-label-value

• apiVersion specifies the Kubernetes API version for deployments.

• kind specifies that this is a deployment.

• metadata includes information like the name of your deployment.

• spec defines the desired state for your deployment. You can adjust the number of replicas, the container image, and the container port to fit your application's requirements.

• env allows you to specify environment variables for your container.

• resources define resource requests and limits for memory and CPU.

• affinity enables you to set node affinity based on node labels.

Make sure to replace your-app-deployment, your-app, your-app-container, and your-app-image:latest with the appropriate values for your application. This script will create a deployment with three replica pods.

You can save this script in a YAML file, and then apply it to your Kubernetes cluster using the kubectl apply -f your-script.yaml command. This will deploy your application on Kubernetes.

Service

In Kubernetes, a service is a bit like a group of pods. It's a way to organize and access these pods.

There are three main types of services:

1. NodePort: This service type lets you access your pods from outside the cluster. It's like opening a door on each node so that external traffic can reach your pods.

    apiVersion: v1
    kind: Service
    metadata:
      name: my-nodeport-service
    spec:
      type: NodePort
      ports:
        - port: 80
          targetPort: 80
      selector:
        app: my-app
   

2. ClusterIP: These services make your pods accessible only from within the cluster. It's like an internal phone system that lets pods talk to each other.

    apiVersion: v1
    kind: Service
    metadata:
      name: my-clusterip-service
    spec:
      type: ClusterIP
      ports:
        - port: 80
          targetPort: 80
      selector:
        app: my-app
   

3. LoadBalancer: This type sets up a load balancer that can distribute external traffic to your pods. It's like having a receptionist who directs calls to the right person.

    apiVersion: v1
    kind: Service
    metadata:
      name: my-loadbalancer-service
    spec:
      type: LoadBalancer
      ports:
        - port: 80
          targetPort: 80
      selector:
        app: my-app
   

Services help manage how your pods are reached and used, making it easier to run your applications.

Kubernetes Cheet Sheet

Kubernetes Trouble Shoot

📚Resources

1) rootsongjc/kubernetes-handbook/

2) https://kubernetes.io/docs/home/

3) https://learnk8s.io/troubleshooting-deployments

🙏Thank you for reading...

✈️ Linkedin: https://www.linkedin.com/in/vgirish10/