While working on the extension, I learned a great deal about GitHub Actions, creating tags and artefacts for releases, and utilizing the Web Audio API to incorporate effects such as reverb and pitch shifts on audio. However, I felt I couldn't write posts about those topics until I had officially released the extension to the public. Moreover, I hadn't begun planning how to incorporate it into the series. This wasn't ideal. I need to transition from series-focused posts to standalone posts, which can include quick thoughts, intriguing features and/or bugs I encountered, life and career updates, and more. Then, perhaps, I can group related topics into a series.

That's it. Simple and short. I hope to keep a steady pace by reducing the pressure of having many planned posts on a subject.

In this post, we want to create and submit an initial release for the Chrome Web Store and upload it to Github. Before that step though, let's fix a bug encountered.

Bug Fix

Issue

The issue originates from the removal of the "Add to queue" option from the menu and the thumbnail on disconnectObserver. The remove() method eliminates the node and its space in the DOM, rendering it inaccessible.

class ObserverWrapper {
    //
    _removeQueueUI(mutationsList) {
        for (let mutation of mutationsList) {
            const { target, addedNodes } = mutation

            if (this._isPlaylistIconHover(mutation)) {
                const playlistIcon = Array.from(addedNodes)
                  .find(a => a.ariaLabel === 'Add to queue')
                playlistIcon?.remove()
            }

            if (this._isMenuOption(mutation)) {
                target?.parentElement?.parentElement?.remove()
            }
        }
    }  
}

So, when we turn off the extension and disconnect the observer, we can't undo the changes we made to the user interface. We'd need to recreate the menu options and playlist icons we removed. This is a tough job because each interface item has many parts, like children, styles, and events, that we need to rebuild and put back into the DOM. Yikes.

Solution

Instead of removing the DOM elements, we can update their style instead. By setting a style of "display: none," we can hide the UI from view while still keeping it accessible. Therefore, when toggling off, we can reset the display attribute to "block" to bring it back into view. In our ObserverWrapper class, we now need to track a new property to either hideUI (set display: none) or showUI (set display: block).

class ObserverWrapper {
    constructor() {
        // 
        this._isHidden = true
    }

    // . . 

    _toggleQueueUI(mutationsList) {
        const display = this._isHidden ? 'none' : 'block'

        for (let mutation of mutationsList) {
            const { target, addedNodes } = mutation

            if (this._isPlaylistIconHover(mutation)) {
                const playlistIcon = Array.from(addedNodes)
                  .find(a => a.ariaLabel === 'Add to queue')
                playlistIcon?.setAttribute('style', `display: ${display}`)
            }

            if (this._isMenuOption(mutation)) {
                const element = target.parentElement.parentElement
                element?.setAttribute('style', `display: ${display}`)
            }
        }
    }

    hideUI() {
        this._isHidden = true
        if (!this._observer) { 
            this._composeObserver()
        }
    }

    showUI() {
        this._isHidden = false 
    } 
}

function init() {
    const observerWrapper = new ObserverWrapper()

    function messageCallback(message) {
        message.active
          ? observerWrapper.hideUI() 
          : observerWrapper.showUI()
    }

    listenForMessage(messageCallback)
}

init()

Release

Releasing an extension to the Chrome Web Store requires a developer account and a one-time payment of $5. Reference the steps for publishing extensions to set up an account and access the Developer Dashboard. Whether manual or automated deployments, we still need to zip our extension as a package, upload it to our account, submit it for review, and wait for 1-2 business days for review. If approved then our extension is published. This happens every single time we cut a release.

1. Zip the extension with only the necessary files. The "-x" flag is to not include our readme and license markdown files.

zip -r yqm.zip * -x '*.md'

1. Either drag or browse for and select the "yqm.zip" folder from the drag zone container in the Dashboard.

   

1. Follow along with the listing form and enter input where necessary. Include links for support, which in this case will be the repo.

   

2. Same for Privacy Policy where we need to justify our use of permissions such as activeTab, storage, etc. No need to be overly descriptive here.

   

3. Distribution is the final step before submitting for review. The only concern with making the extension available worldwide is I18n, but we currently only have "on/off" text in our action icon. A quick solution would be either disabling/enabling the extension on action click or using a more universal on/off signifier like a lit/unlit lightbulb icon.

   

Publish

The package uploads and form edits are all in draft until one clicks the "Submit for Review" button. YQM's review took about 2 days, but this could have been due to it being submitted over the weekend, in other words, it was pretty fast.

Here it is on the web store: YQM

Analytics

Chrome Extensions feature built-in analytics for tracking actions such as adding or removing, as well as impressions and more. Furthermore, they can be easily integrated with Google Analytics for even more detailed data. There's a wealth of information available on the dashboard, but since this is a personal passion project, not much attention will be given to these dashboards. Instead, they will primarily be used to determine which project to prioritize based on the number of installations and active users.

Developer Dashboard Analytics

Impressions

Installs/Uninstalls

Google Analytics

Release

Here is the GitHub link: YQM. There's a general roadmap for what the finalized extension might have in terms of features, i18n, cross-browser support, etc. Contributors are welcome in any form - filing issues, updating docs/readme, updating icons, feature implementation, automating releases, etc.

Wrap Up

Okay. This was a good introduction to setting up an extension and developing it from ideation to a v1 release. Additionally, we discovered the MutationObserver web API to observe and manipulate the DOM based on changes in specific nodes. We started working on the next version's features, but right now, we're focusing more on a new project about controlling the web Spotify player, changing the playback speed, and playing only a specific part of a song.

1. Either disable or remove all the Add to queue UI (on hover and via the dropdown:

   

2. Enable or disable the first feature by clicking on the extension in the toolbar, which controls the availability of the "Add to queue" buttons.

Setup

All Chrome extensions require a manifest.json file, which is similar to a package.json file. In the manifest.json file, define the essential fields, focusing on the bare minimum needed for the setup. Fields marked with an asterisk are mandatory in every file.

The final manifest.json file at the end of the project will likely include additional fields not mentioned below, but these can be found on the MDN manifest.json documentation.

{
      "manifest_version": 3,
      "name": "YouTube Queue Manager",
      "short_name": "YQM",
      "version": "1.0",
      "author": "cdrani",
      "description": "Manage Youtube Queue Playlist",
      "icons": {
            "16": "icons/icon16.png",
            "48": "icons/icon48.png",
            "128": "icons/icon128.png"
      }
}

You should always provide a 128x128 icon; it's used during installation and by the Chrome Web Store. Extensions should also provide a 48x48 icon, which is used on the extensions management page (chrome://extensions). You can also specify a 16x16 icon to be used as the favicon for an extension's pages.

From the information provided, we have set up our manifest.json file and created an icons folder containing 16x16, 48x48, and 128x128 pixel PNG files. These images were derived from a queue-list SVG icon from Heroicons, which was subsequently resized to the necessary dimensions and converted into PNG files.

Load Extension

For testing and development purposes, we want to load the current state of our extension in our browser. To achieve this, simply enter chrome://extensions in the search bar, enable Developer mode, click on "Load unpacked," and select the extension folder. Counter 4 showcases the information from our manifest file. We will frequently use the reload button (Counter 5) to keep changes in our project synchronized with the browser.

For now, we want our extension to be activated manually by clicking on it in our toolbar. Therefore, we need to pin the extension to our toolbar, ensuring it is always present for easy access.

Feature 1: Remove the 'Add to queue' UI

It took two different approaches to implement this feature. For educational purposes, both approaches will be highlighted to demonstrate the thought processes and issues which led to a pivot.

Approach 1: Utilize Event Handlers

Thought Process

Currently, the "Add to Queue" user interface (UI) has two variations: the first features a playlist icon with a tooltip, and the second is an option within a menu item on every video. Both of them appear when hovering over a video, but the menu options are only displayed when the menu dropdown is clicked.

To remove the option from the menu, simply locate the dropdown menu component in the DOM, identify the event that triggers the display of menu options, and update the event listener to eliminate the 'Add to Queue' option.

From above we see that's it the yt-icon-button element and the click event on it that reveals the menu. The menu option can be found the same way:

We have all the necessary information, so we can now create a search-and-remove script to eliminate the "Add to Queue" option based on the yt-formatted-string component. https://developer.mozilla.org/en-US/docs/Web/API/Document/evaluate

const menuButtons = document.querySelectorAll('yt-icon-button#button.dropdown-trigger')

menuButtons.forEach(mb => {
    mb.addEventListener('click', () => {
        // find the specific `yt-formatted-string` element in the menu 
        const menuOptionString = document.evaluate("//yt-formatted-string[contains(., 'Add to queue')]")
        const addToQueueOption = menuOptionString.iterateNext()
        addToQueueOption?.parentElement?.parentElement?.remove()     
    }
})

Repeat this process for the 'Add to queue' icon on the thumbnail. It's a bit harder in Chrome, but Firefox Devtools lets us see and change toggle events on elements. We want to turn off the "mouseleave" event to keep the playlist icon visible for inspection.

We now have the component for the playlist icon.

Here's the script to search and remove it.

const sectionItems = document.querySelectorAll('ytd-video-renderer')
sectionItems.forEach(si => {
    si.addEventListener('mouseenter', () => {
        const playlistIcon = document.evaluate("//ytd-thumbnail-overlay-toggle-button-renderer[contains(@aria-label, 'Add to queue')]")
        const playlistIconRenderer = playlistIcon?.iterateNext()
        playlistIconRenderer?.remove()
    })
})

And it's gone.

We have a small issue in that YouTube has different components for the video section item depending on if one's on the search, home, subscription, or playlist page. The variants are ytd-grid-video-renderer, ytd-rich-grid-media ,and ytd-video-renderer . They don't all appear on the same page, so we run our script based on the youtube tab path names.

const SELECTORS = {
    '/': ['ytd-rich-grid-media'],
    '/feed/subscriptions': ['ytd-grid-video-renderer'],
    '/playlist': ['ytd-video-renderer'],
    '/@': ['ytd-grid-video-renderer', 'ytd-video-renderer']
}

function getTabPathName() {
    const selectorKeys = Object.keys(SELECTORS)
    const pathName = window.location.pathname

    const currentSelectorKey = Object.keys(SELECTORS)
        .find((key) => {
            return (key === '/' && pathName === '/') || 
                (key !== '/' && pathName.includes(key))
        });

    return currentSelectorKey
}

function getQuerySelector() {
    const tabPathName = getTabPathName()
    return SELECTORS[tabPathName]
}

function searchAndRemoveMenuOption() {
    const menuButtons = document.querySelectorAll('yt-icon-button#button.dropdown-trigger.ytd-menu-renderer');

    menuButtons.forEach(mb => {
        mb.addEventListener('click', () => {
            // find `yt-formatted-string` component in menu options
            const menuOptionString = document.evaluate(
                "//yt-formatted-string[contains(., 'Add to queue')]",         document)
            const addToQueueOption = menuOptionString?.iterateNext()
            addToQueueOption?.parentElement?.parentElement?.remove()      
        }
    })
}

function searchAndRemoveQueueIcon() {
    const querySelector =  getQuerySelector()
    const sectionItems = document.querySelectorAll(querySelector)

    sectionItems.forEach(si => {
        si.addEventListener('mouseenter', () => {
            const playlistIconElement = document.evaluate("//ytd-thumbnail-overlay-toggle-button-renderer[contains(@aria-label, 'Add to queue')]", document)?.iterateNext()
            playlistIcon?.remove()
        })
    })
}

Cons

The code works, so what's wrong? Why not use it, add Chrome extension features, put everything together, and start version 1? Let's talk about the current code using a QnA format:

Q: We have added "click" and "mouseenter" events to some UI elements. This is fine initially, as there are only a few elements, but more appear as users scroll down. Should we run our script again for the entire page? How can we monitor page changes?

A: No, that would cause performance issues. We could use a MutationObserver to watch for only new changes in the DOM and re-run the script for the addedNodes.

Q: Okay, the main worry is adding new events to each item. Users might not use every updated part and can switch the extension on or off. If they turn it off, should we refresh the page to start over with the events? That could annoy users. Or, should we take away the events? It's possible, but it might cause problems since every video card item would have an event connected to it.

A: We need to consider our options. We can use the MutationObserver to check for new UI when a user hovers over a video card, locate the "Add to Queue" button, and remove it. This is faster because we only deal with one element that the user interacts with. When a user turns off the extension, we can disconnect from our MutationObserver to reset everything without harsh reloading or numerous event listeners. We can do the same for the dropdown menu options.

Q: Yes, yes, and yes! This appears to be a better approach to pursue.

Approach 2: MutationObservers

Settling on MutationObservers, we have to narrow down what DOM changes to watch for and the updated UI to take action upon.

For the dropdown menu, in the Firefox Inspector tab within the developer tools, we identify that the ytd-popup-container is the component to observe, as it renders the menu options when the dropdown is clicked.

/insert media of finding ytd-pop-container

Let's see the changes by testing our observer in the console:

const mutationObserver = new MutationObserver(mutations => {
    console.log(mutations)
})
const target = document.querySelector('ytd-popup-container')
const config = { childList: true, subtree: true }
mutationObserver.observe(target, config)

Here are the changes when you click the dropdown. The important part is when the "Add to queue" text appears in the ytd-formatted-string component.

Upon identifying the precise target, we discover that the actual parent component of the yt-formatted-string is ytd-menu-service-item-renderer, which serves as the grandparent of our current element.

Removing it is quite simple.

function isMenuOption(mutation) {
    const { localName, innerText } = mutation.target
    return localName == 'yt-formatted-string' 
        && innerText === 'Add to queue'
}

function clearQueueUI(mutationsList) {
    for (let mutation of mutationsList) {
        const { target, addedNodes } = mutation

        if (isMenuOption(mutation)) {
            target?.parentElement?.parentElement?.remove()
        }
    }
}

function setUpObservers() {
      const mo = new MutationObserver(clearQueueUI)
    const target = document.querySelector('ytd-popup-container')
    const config = { childList: true, subtree: true }
    mo.observe(target, config)
}

setUpObservers()

Video card components on YouTube pages have varying names, such as ytd-grid-media, ytd-rich-grid-media, and ytd-grid-video-renderer. While they appear similar, they have minor differences based on their containers. Rather than using multiple observers for each container, let's utilize the primary app container, ytd-app, which also houses the ytd-popup-container. This approach is more straightforward and doesn't cause slowdowns since we only handle a few events. Consequently, we'll modify our script to manage the menu option and playlist hover features.

function isChildList(mutation) {
    return mutation.type === 'childList'
}

function isPlaylistIconHover(mutation) {
    const { target, addedNodes } = mutation
    return !!addedNodes?.length && target.id === 'hover-overlays' 
}

function isMenuOption(mutation) {
    const { localName, innerText } = mutation.target
    return localName == 'yt-formatted-string' 
        && innerText === 'Add to queue'
}

function clearQueueUI(mutationsList) {
    for (let mutation of mutationsList) {
        const { target, addedNodes } = mutation

        if (isPlaylistIconHover(mutation)) {
            const playlistIcon = Array.from(addedNodes)
                  .find(a => a.ariaLabel === 'Add to queue')
                playlistIcon?.remove()
        }

        if (isMenuOption(mutation)) {
            target?.parentElement?.parentElement?.remove()
        }
    }
}

function setUpObservers() {
      const mutationObserver = new MutationObserver(clearQueueUI)
    const target = document.querySelector('ytd-app')
    const config = { childList: true, subtree: true }
    mutationObserver.observe(target, config)
}

setUpObservers()

Implementation

We have tested the above script in our console and it works across all youtube pages. Let's move it into our extension. Since the script is just a group of related functions, why not restructure it into an ObserverWrapper class? Additionally, let's add methods to disconnect our observer.

class ObserverWrapper {
    constructor() {
        this._observer = undefined 
        this._selector = 'ytd-app'
        this._config = { childList: true, subtree: true }
    }

    _composeObserver() {
        const composeBox = document.querySelector(this._selector)

        const mutationObserver = new MutationObserver(this._handleMutations)
        mutationObserver.observe(composeBox, this._config)

        this._observer = mutationObserver
    }

    connectObserver() {
        this._composeObserver()
    }

    _isMenuOption(mutation) {
        const { localName, innerText } = mutation.target
        return localName === 'yt-formatted-string' && innerText === 'Add to queue'
    }

    _isPlaylistIconHover(mutation) {
        const { target, addedNodes } = mutation
        return !!addedNodes?.length && target.id === 'hover-overlays' 
    }

    _isChildList(mutation) {
        return mutation.type === 'childList'
    }

    _filterMutations = (mutation) => {
        return this._isChildList(mutation) && 
                (this._isPlaylistIconHover(mutation) ||     this._isMenuOption(mutation))
    }

    _handleMutations = (mutationsList) => {
        const filteredList = mutationsList.filter(this._filterMutations)

        if (!filteredList?.length) return
        this._removeQueueUI(filteredList)
    }

    _removeQueueUI(mutationsList) {
        for (let mutation of mutationsList) {
            const { target, addedNodes } = mutation

            if (this._isPlaylistIconHover(mutation)) {
                const playlistIcon = Array.from(addedNodes)
                  .find(a => a.ariaLabel === 'Add to queue')
                playlistIcon?.remove()
            }

            if (this._isMenuOption(mutation)) {
                target?.parentElement?.parentElement?.remove()
            }
        }
    }
}

Manifest Scripts & Permissions

Chrome Extensions are by default very restrictive and we have to be explicit about our intentions. We have access to a ton of Chrome APIs, but we need to declare them in the manifest.json file. In our case, here's the list of our intents and permissions:

{
    "manifest_version": 3,
    "name": "YouTube Queue Manager",
    "version": "1.0",
    "description": "Be in control of your YouTube queues.",
    "icons": {
        "16": "icons/icon16.png",
        "48": "icons/icon48.png",
        "128": "icons/icon128.png"
    },
    "actions": {},
    "permissions": [
        "tabs",
        "activeTab",
        "scripting",
        "storage"
    ],
    "host_permissions": ["*://*.youtube.com/*"],
    "background": {
        "service_worker": "background.js"
    }
}

Let's start with creating a background.js file. In it, we want to register a script that contains the logic to clear the queue UI when the extension is loaded. We will make use of the registerContentScripts API.

chrome.scripting.registerContentScripts([{
    id: 'script-content',
    js: ['toggleQueue.js'],
    runAt: 'document_idle',
    matches: ['*://*.youtube.com/*']
}])

Content Scripts run in the context of a particular webpage, in our case any page that matches a youtube URL. Our toggleQueue.js file is our ObserverWrapper class with an initializer:

class ObserverWrapper {
    //
} 

function init() {
    const observerWrapper = new ObserverWrapper()
    observerWrapper.connectObserver()
}

init()

Reloading the extension from the chrome://extensions page and a youtube tab will now show an updated UI with any form of queue UI removed.

Feature 2: Toggle Extension On/Off

We need to be able to toggle our extension on/off when a user clicks on the extension icon in the toolbar. Therefore, we will need to take the following steps:

1. Upon installing the extension, we must create a state object that stores the extension's current status (active/inactive). The initial state will be set to active.

2. Display a badge on the extension to consistently indicate its current state. For readability, it will simply show "on" or "off" with a green or red background, respectively.

3. Establish a messaging system between our service worker (background.js) and content script (toggleQueue.js). The service worker will transmit the current state of the extension, while the content script will listen for these messages. If the state is active, we will connect our observers to monitor changes in the DOM and update our UI accordingly (as currently implemented); otherwise, we will disconnect our observers.

4. Add an onClicked event listener in our service worker that listens for clicks on the extension icon. These clicks will toggle the active state and send a message containing the updated state. The incoming message will initiate the same actions described in the second part of point 3.

5. Additional event listeners for active tab updates such as reload, path changes such as from "/" to "/feed/subscriptions".

Defining State Object

Our state object will be simple, with just an active key storing a boolean value, and an event key (optional for debugging) with a string value of the event listener in which the state is sent to the content script. There are multiple storage options provided by the API, but we will utilize the local option. Additionally, with the active state set, we can display a badge based on the state.

function setBadge(active) {
    chrome.action.setBadgeText({ text: active ? 'on' : 'off' });
    chrome.action.setBadgeBackgroundColor({ color: active ? 'green': 'gray' }); 
}

// Currently no event will be used to set this state
chrome.storage.sync.set({ state: { active: true, event: '' } }, () => {
    if (chrome.runtime.lastError) {
        console.error('Error while saving the updated value:', chrome.runtime.lastError)
        return;
    } 
    setBadge(true);
})

Our state object will only be initialized or updated through an event listener. The first event listener will be triggered by the onInstalled event when a user installs the extension. This would be an ideal location to register content scripts, initialize our state, and send the first message.

function setBadgeInfo(active) {
    chrome.action.setBadgeText({ text: active ? 'on' : 'off' });
    chrome.action.setBadgeBackgroundColor({ color: active ? 'green': 'gray' });
}

function stateResolver({ resolve, reject, result, key }) {
    if (chrome.runtime.lastError) {
        return reject({ error: chrome.runtime.lastError })
    } 
    return key ? resolve(result[key]) : resolve()
}

function getState({ key = 'state' }) {
    return new Promise((resolve, reject) => {
        chrome.storage.local.get(key, (result) => {
            return stateResolver({ key, resolve, reject, result })
        })
    })    
}

function setState({ key = 'state', value = {} }) {
    return new Promise((resolve, reject) => {
        chrome.storage.local.set({ [key]: value }, (result) => {
            return stateResolver({ resolve, reject, result })
        })
    })
}

async function registerScripts() {
    return await chrome.scripting.registerContentScripts([{
        id: 'script-content',
        js: ['toggleQueue.js'],
        runAt: 'document_idle',
        matches: ['*://*.youtube.com/*']
    }])
}

chrome.runtime.onInstalled.addListener(async () => {   
    await registerScripts()
    const result = await setState({ 
        key: 'state',
        value : { active: true,  event: 'onInstalled' }
    })

    if (!result?.error) setBadgeInfo(true)
}

We set the badge info on the onInstalled event listener, but on subsequent state changes, we have access to an onChanged event listener for storage. In this listener, is where it would be ideal to update our badge info and send messages to our content script about these changes.

async function getActiveTab() {
    const result = await chrome.tabs.query({ 
        active: true, 
        currentWindow: true,
        url: ['*://*.youtube.com/*'],
    })

    return result?.at(0)
}

function messenger({ tabId, message }) {
    return new Promise((reject, resolve) => {
        chrome.tabs.sendMessage(tabId, message, (response) => {
            if (chrome.runtime.lastError) {
                return reject({ error: chrome.runtime.lastError })
            } 
            return resolve(response)
        })
    })
}

async function sendMessage({ message }) {
    const activeTab = await getActiveTab()
    if (!activeTab) return

    return await messenger({ tabId: activeTab.id, message })
}

chrome.storage.onChanged.addListener(async changes => {
    const { newValue } = changes.state
    setBadgeInfo(newValue.active)
    await sendMessage({ message: newValue })
})

The second to last step is accounting for toggling the extension on/off on an onClicked event upon clicking the extension, an onActivated event for when a tab is switched to, i.e clicked on and made active, and an onUpdated if the tab has some updates, such as a path change from "/" to "/feed/subscriptions" or page reload.

chrome.tabs.onActivated.addListener(async () => {
    const state = await getState({ key: 'state' })
    await sendMessage({ message: { active: state.active, event: 'onActivated' } })
})

chrome.action.onClicked.addListener(async () => {
    const state = await getState('state')
    await setState({ 
        key: 'state', 
        value: { active: !state.active, event: 'onClicked' } 
    })
})

chrome.tabs.onUpdated.addListener(async (_, changeInfo, tab) => {
    if (tab.active && changeInfo?.status === 'complete' && tab.url.includes('youtube.com')) {
        const state = await getState({ key: 'state' })
        sendMessage({ message: { active: state.active, event: 'onUpdated' } })
    }
})

The final step involves listening for any messages sent from our background script and taking appropriate action. If the message object's 'active' value is true, connect the observer; otherwise, disconnect the observer. Let's implement the message listener and disconnection functionality.

Class ObserverWrapper {
    // ...

    // disconnect observer logic
    _reset() {
        this._observer?.disconnect()
        this._observer = undefined
    }

    disconnectObserver() {
        this._reset()
    }
}

function listenForMessage(callback) {
    chrome.runtime.onMessage.addListener(message => {
        callback(message)
    })
}

function init() {
    const observerWrapper = new ObserverWrapper()

    function messageCallback(message) {
        message.active
          ? observerWrapper.connectObserver() 
          : observerWrapper.disconnectObserver()
    }

    listenForMessage(messageCallback)
}

init()

Wrap Up

In conclusion, this article demonstrates how to create a YouTube Chrome Extension that effectively disables the "Add to Queue" UI on YouTube pages. By utilizing Chrome APIs and JavaScript, we implemented a functional extension that can be toggled on and off by the user. This tutorial showcases the thought process, challenges, and solutions involved in developing a Chrome Extension, ultimately providing a solid foundation for further exploration and development in this area. The next post in this series will detail how to package this extension for release on the Chrome Web Store.

I spend most of my time on YouTube, beginning with some "Skip and Shannon: Undisputed" for sports recaps and analysis, although I mainly watch it for their banter. Later in the afternoon, I watch videos from my subscriptions, and I end the day with videos from my "Watch Later" playlist. This routine might seem reasonable, and it could be, except for the fact that I often watch videos through a queue, starting with my daily routine videos and gradually adding any recommended ones. The problem is that my queues in the mornings usually consist of only 4-5 videos, totalling 25-40 minutes, which can easily fit between deep work sessions. However, now they are approaching 40-60 minutes, and I'm concerned that it may get further out of hand if I don't limit this practice.

Youtube Add-on Objective

I need to create a Chrome Extension that will do the following:

• Either disable or remove all the Add to queue buttons (on hover and via the dropdown:

  

• Ability to toggle this feature on/off from a popup:

  

  

• Features to limit videos added to the queue, either by setting a queue length or setting a max total time for videos. Update the queue playlist ui to display the total time of videos in the queue.

• Display a tooltip, alert, etc if the Add to queue button is disabled (if that's what's chosen in the first bullet) and the reason is due to the set queue length or max total time.

• Fix this UI issue:

  

• Enlarge Video Duration so a user is doubly aware of what they are potentially adding to a queue

• (Optional) Display an analysis or recap if this extension is functioning to limit watch time or queue length when it's enabled, and also unintentionally when it's disabled. Ideally, I should be able to gradually reduce my reliance on this self-imposed restriction and only activate it when I start slacking off, based on the usage review.

• (Optional) Automate video removal from queue upon video end. Upon queue completion also clear the queue without having to interact with the modal

• (Optional) Automate queue generation, such as in the mornings, create a queue with 2-3 of the top Undisputed videos that are only 10-13 minutes. Additionally, maybe delete/clear the queue at a certain time. Do the same for the afternoon but focused on subscriptions. For subscriptions, further filtering will be needed to prioritize channels weighed with watch time. For later in the evening, maybe disable functionality altogether as it's my time.

• (Optional) Maybe in the evening automate plucking some videos from my ever-increasing "Watch later" playlist into a queue. After viewing a video selected from the "Watch Later" playlist, the video should be removed from the queue and playlist.

So, we have a plan on what to create, now it's just a matter of breaking ground on it. There are a few features that are a must-have for a v1 release (if I plan on releasing it on the web store), but I hope to keep on working on it to either implement the optional features or at least open source it and invite and help others contribute them.

In the next article in this series, we will start with setting up the structure and prerequisites for a Chrome Extension and implement the first two features.

Tmux is a terminal multiplexer: it enables several terminals to be created, accessed, and controlled from a single screen. Tmux may be detached from a screen and continue running in the background, then later reattached.

source: https://github.com/tmux/tmux

AppleScript is a scripting language created by Apple Inc. that facilitates automated control over scriptable Mac applications. The term "AppleScript" may refer to the language itself, to an individual script written in the language, or, informally, to the macOS Open Scripting Architecture that underlies the language.^[4][5]

source: https://en.wikipedia.org/wiki/AppleScript

INTRO

I have been using Tmux in my workflow with the https://github.com/gpakosz/.tmux configuration. I've done further research into how to personalize it. Unfortunately, most plugins are written in a shell script, which is understandable as shells like bash are available (or built-in) in most distros or OSs. I do want to learn how to write shell scripts, not just for Tmux, but as well as the universality of it.

NOTE: AppleScripts will only work on a macOS ecosystem.

Let's create a script that's human-readable using AppleScript to get a feel for the language and then expand on it by utilizing a shell script afterwards.

IMPLEMENTATION: SETUP

The purpose of the following script is to explore the AppleScript language and how to utilize it to interact with the Spotify app. Additionally, we aim to display the current song information (track and artist name), as well as the player's state (song playing or paused), and whether Spotify has been launched and displays all this data in a tmux status bar.

As for most script files, the heading should be the environment that the file should be run in:

#!/usr/bin/env osascript

1. First, we must ascertain whether Spotify is running; if it isn't, display that Spotify is currently inactive. Take note of the language's readability and the keywords employed.

    tell application "Spotify"
        if it is running then
            # 
        else
            "♫ 💤" 
        end if
    end
   

2. Now, we want to obtain and store the current track and artist's name and the player state (playing or paused) in variables, specifically, track_name, artist_name, and player_state, respectively. We could also retrieve other information, such as the album_name,

    tell application "Spotify"
        if it is running then
    +       set track_name to name of current track
    +       set artist_name to artist of current track
    +       set player_state to player state as string
        else
            "♫ 💤" 
        end if
    end
   

3. We now want to display the info we gathered from above. We can use an if/else statement to branch between what we display based on the player state.

    tell application "Spotify"
        if it is running then
            set track_name to name of current track
            set artist_name to artist of current track
            set player_state to player state as string
    +       if player_state is equal "playing"
    +           "♫ ⏵ " & track_name & " + " & artist_name
    +       else
    +           "♫ ⏸ " & track_name & " + " & artist_name
    +       end if
        else
            "♫ 💤" 
        end if
    end
   

4. And that's it. However, we can slightly refactor the code and introduce functions. Finally, let's include the maintainer's information (optional, but helpful for directing complaints when the script doesn't work as intended) and a description of the script's function for users.

    #!/usr/bin/env osascript
   
    + -- maintainer: cdrani
    + -- Returns Spotify current player state and song info
   
    + on getPlayerState(player_state)
    +     if player_state is "playing" then return "♫ ⏵ "
    +      if player_state is "paused" then return "♫ ⏸ "
    + end getPlayerState
   
    + on printSongInfo(player_state, track_name, artist_name)
    +    getPlayerState(player_state) & track_name & " - " & + artist_name
    + end printSongInfo
   
    tell application "Spotify"
    +    if it is not running then
    +       return "♫ 💤"
    +   end
   
        set track_name to name of current track
        set artist_name to artist of current track
        set player_state to player state as string
   
    +    my printSongInfo(player_state, track_name, artist_name)
    end tell
   

5. Now, we want to make the script executable. I saved mine as "spotify.scpt" (scpt is the filename for AppleScript) in a scripts directory (which I am currently inside):

   chmod +x ~/scripts/spotify.scpt

6. The final step is to integrate it into our Tmux status line. I opted for the right-hand side. Inside a Tmux session, enter your command prompt using Prefix + : and type the following based on the path the script file

   set -g status-right '#(~/scripts/spotify.scpt)'

Great! Implementing this idea is quite simple, from conception to full realization, except for the need to close each 'tell' and 'if' statement. The song information is displayed in the top-right corner and updates upon song change, although there is a slight ~2-second delay.

AppleScript Editor

To begin, it's a good idea to use the AppleScript Editor, as it offers language features such as syntax highlighting, error compilation, quick access to language documentation, and more. Below is the aforementioned script within the editor. In the editor, we build the script to verify that there are no issues, and then we can run it, with errors and results displayed in the bottom pane.

JavaScript Editor

AppleScript also supports JavaScript. Our script can be rewritten in JavaScript:

// maintainer: cdrani
// Returns the current player state and song info for Spotify

function getPlayerState(playerState) {
  if (playerState === "playing") return "♫ ⏵ ";
  if (playerState === "paused") return "♫ ⏸ ";
}

function printSongInfo(playerState, trackName, artistName) {
  return getPlayerState(playerState) + trackName + " - " + artistName;
}

const spotify = Application("Spotify");

if (!spotify.running()) {
    "♫ 💤";
} else {
  const trackName = spotify.currentTrack.name();
  const artistName = spotify.currentTrack.artist();
  const playerState = spotify.playerState();

  printSongInfo(playerState, trackName, artistName);
}

Similar to the first script, save this one as ~/scripts/spotify.js. You can now run it in your Tmux command prompt. Since this is a JS file, you need to inform AppleScript about it:

-l language | Override the language for any plain text files. Normally, plain text files are compiled as AppleScript.

set -g status-right '#(osascript -l JavaScript ~/scripts/spotify.js)'

The integration of our scripts into Tmux via the command prompt is temporary and gets cleared when the session or server is terminated. To make it permanent, we need to transfer it to our configuration file, typically saved as ~/.tmux.conf, and then source the config file to apply the changes.

echo "set -g status-right '#(~/scripts/spotify.scpt)'" >> .tmux.conf
tmux source-file ~/.tmux.conf

Bonus Feature

Announcement Feature

The final feature we will incorporate enables the announcement of the current song, as well as its play/pause status, whenever the song changes.

1. Let's first introduce a new say, which will speak out loud our songInfo.

   

    +    on sayPlayerState(player_state, track_name, artist_name)
    +        say player_state & " " & track_name & " by " & artist_name
    +    end sayPlayerState
   
    tell application "Spotify"
         -- truncated
    +    sayPlayerState(player_state, track_name, artist_name)
         my printSongInfo(player_state, track_name, artist_name)
    end tell
   

2. An issue with the above text is that the song information is repeated multiple times per song. This is likely because new data is being fed into our script every few seconds, based on the current song's state - its name, artist, whether it's playing or paused, etc. We want to limit this repetition to only occur when the song changes or the song's play/pause state changes. Let's track the song and state by defining their properties set to "missing value". jj

    -- properties to track song and state; set to non value
    +     property previous_song : missing value
    +     property previous_state : missing value
   
    --- 
   
    +    set stateExistsAndIsDifferent to previous_state is equal to missing value or previous_state is not equal to player_state
    +    set isPlayingAndSongChanged to previous_state is "playing" and previous_song is not equal to track_name
   
    +    -- update previous_state and previous_song with new states
    +    if (stateExistsAndIsDifferent or isPlayingAndSongChanged)
    +        set previous_state to player_state
    +        set previous_song to track_name
    +        sayPlayerState(player_state, track_name, artist_name)
    + 
    +        -- send command to update tmux status 
    +        do shell script "tmux set-option -g status-right '" & my printSongInfo(player_state, track_name, artist_name) & "'"
    +   end if
   

3. Finally, continuously refresh the data by running the script in a loop with a 1-second delay.

    #!/usr/bin/env osascript
   
    # maintainer: cdrani
    # Returns the current player state and song info for Spotify
   
    property previous_song : missing value
    property previous_state : missing value
   
    on getPlayerState(player_state)
        if player_state is "playing" then return "♫ ⏵ "
        if player_state is "paused" then return "♫ ⏸ "
    end getPlayerState
   
    on printSongInfo(player_state, track_name, artist_name)
        getPlayerState(player_state) & track_name & " - " & artist_name
    end printSongInfo
   
    on sayPlayerState(player_state, track_name, artist_name)
        say player_state & " " & track_name & " by " & artist_name
    end sayPlayerState
   
    + repeat
        tell application "Spotify"
            if it is not running then
                return "♫ 💤"
            end if
   
            set track_name to name of current track
            set artist_name to artist of current track
            set player_state to player state as string
        end tell
   
        set stateExistsAndIsDifferent to previous_state is equal to missing value or previous_state is not equal to player_state
        set isPlayingAndSongChanged to previous_state is "playing" and previous_song is not equal to track_name
   
        if (stateExistsAndIsDifferent or isPlayingAndSongChanged)
            set previous_state to player_state
            set previous_song to track_name
   
            sayPlayerState(player_state, track_name, artist_name)
            do shell script "tmux set-option -g status-right '" & my printSongInfo(player_state, track_name, artist_name) & "'"
         end if
   
    +    delay 1
    + end repeat
   

Conclusion

This was a simple script to showcase AppleScript and to show how it can integrate with the Apple ecosystem, here focusing on extracting Spotify info to display in a Tmux status bar. In the next article, we will expand on it to add additional controls such as next, previous, pause/play, and repeat.

Fresh Setup

By "Fresh", I mean just having Docker Desktop and a Text Editor - No Rails necessary.

1. Confirm installation of Docker - (Community Edition will work fine) - with

   docker -v

    docker -v
    # Docker version 23.0.5, build bc4487a
   

2. Set up a docker container with a ruby image in which to create the rails app.

    docker run --rm -it -v "$PWD":/usr/src/app -w /usr/src/app ruby:3.2.2-bullseye bash -c "gem install rails && rails new . -d postgresql -T"
   

   This command runs a Docker container using the ruby:3.2.2-bullseye image, with several options:

   docker run: Run a Docker container.

   --rm: Automatically remove the container when it exits.

   -it: Allocate a tty for interactive input and output.

   -v "$PWD":/usr/src/app: Mount the current directory as a volume inside the container at the path /usr/src/app.

   -w /usr/src/app: Set the working directory inside the container to /usr/src/app.

   ruby:3.2.2-bullseye: Use the ruby:3.2.2-bullseye image as the base image for the container.

   /bin/bash: Start an interactive shell inside the container.

   -c "gem install rails && rails new . -d postgresql -T": Execute a shell command inside the container that installs Rails using gem install and then runs the rails new command within the current directory with the specified options - setting the database to PostgreSQL and optionally skipping tests setup.

3. We should now have a Rails app generated on our machine. Next, we want to dockerize it so that we can run all processes, such as the server, console, migrations, and more within containers. Let's begin with the server; to do this, create a Dockerfile (also written as dockerfile, which is my preference).

    FROM ruby:3.2.2-bullseye
   
    # Set the working directory inside the container
    WORKDIR /app
   
    # Update Dependencies
    RUN apt-get update && \
        apt-get clean 
   
    # Copy the Gemfile and Gemfile.lock from the host into the container
    COPY Gemfile Gemfile.lock ./
   
    # Install the RubyGems
    RUN gem install bundler:2.4.13 && \
        bundle config --global frozen 1 && \
        bundle install --jobs 4 --retry 3
   
    # Copy the rest of the application into the container
    COPY . .
   
    # Expose port 3000
    EXPOSE 3000
   
    # Start the Rails server
    CMD ["rails", "server", "-b", "0.0.0.0"]
   

4. Now, we need a container that includes a PostgreSQL image and connects it to the Rails app on the same network. An easy approach is to use a docker-compose.yaml file (or compose.yaml, depending on your preference).

    services:
      db:
        container_name: db
        image: postgres:14-alpine
        environment:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: attic_development
        volumes:
          - db_date:/var/lib/postgresql/data
   
      web:
        container_name: web
        build: .
        ports:
          - "3000:3000"
        depends_on:
          - db
        environment:
          DATABASE_URL: postgresql://postgres:postgres@db/attic_development
        volumes:
          - .:/app
          - gem_cache:/usr/local/bundle/gems
   
    volumes:
      gem_cache:
      db_data:
   

   The compose.yaml file is used to define a multi-container Docker application. It consists of a version number (in this case, version 3), and a list of services that make up the application.

   The services section of the file contains two services:

   db: This service is defined using the official PostgreSQL Docker image tagged 14-alpine. It sets three environment variables:

   • POSTGRES_USER: The username for the default PostgreSQL user.

   • POSTGRES_PASSWORD: The password for the default PostgreSQL user.

   • POSTGRES_DB: The name of the default database to be created.

web: This service is defined to build a Docker image from the dockerfile in the current directory (.). It maps port 3000 of the host to port 3000 of the container. It depends on the db service, which means that the db service will be started before the web service. It also sets the DATABASE_URL environment variable to connect the Rails app to the PostgreSQL database. The depends_on section specifies that the db service should be started before the web service. The environment section sets the DATABASE_URL environment variable to the URL of the PostgreSQL database. The format of the URL is postgresql://<username>:<password>@<hostname>/<database>. In this case, the username is postgres, the password is postgres, the hostname is db (the name of the db service), and the database name is attic_development.

1. Now we can run both services together:

    docker compose up -d
   

   The "-d" option is for detached mode, which means running in the background. In most cases, this is the ideal mode, as without this flag, our terminal will stream the Rails server logs. This would require us to open a new terminal tab to run other commands for migrations, interact with the console, and so on.

   

   Common Commands:

   Running various Rails commands within the web container:

    docker compose run web rails c
    docker compose run web rails g model|controller|migration ...
    docker compose run web rails db:migrate
    docker compose run web rails t
    docker compose run web bundle exec rspec
    docker compose run web bundle add|install|update|remove [gem]
   

   The above docker compose run web rails should be saved into an alias such as:

    alias dcr="docker compose run web rails"
    alias dcb="docker compuse bundle"
    alias dcbr="docker compose bundle exec rspec"
   

   View rails s logs:

    docker logs -f web
   

Best Practices

The above is a minimal dockerized rails app sufficient for development. However, there are additional changes we would need to make to adhere to some best practices and make it ready for deployment in a production environment. For instance, we could use a slimmer image of Ruby, create and use a non-root user in our container, set up multi-stage builds, etc. There's a lot to glean from the examples in these posts for better Dockerfile and Docker Compose files. Additionally, Docker has an official post on this matter.

Setup for Optimal Development/Production

This is most likely the most difficult part as there will likely be variances in how lean and performant one's production docker setup is in comparison to others found online. Furthermore, this comes with experience and testing. As in most cases where I am my own DevOps team, it's better to lean on the expertise of others who readily provide it. There are numerous rails-centric templates with fully-configured docker setup with GitHub actions for automated deployments on GitHub.

I came across Ruby on Whales post and it was truly a godsend. They clearly outline and explain their decisions for their particular setup and keep the post updated. Furthermore, they provide a template so anyone can duplicate their configurations. The script run in their generator can be found here. The setup is quite straightforward by running the below inside our rails app directory:

gem install rbytes
gem install dip
rbytes install https://railsbytes.com/script/z5OsoB
dip provision

The rbytes command will install an interactive script to set up the docker environment in a .dockerdev folder and include a dip.yml file. The script itself can be found here for reference and forked for modification. For a full scope of the inner workings of the generator reference the Ruby on Whales Github repo.

Dip Dip (Docker Interaction Program) is a tool to simplify the previously complicated method of utilizing Docker Compose, making the process smoother. Review the dip.yml to see the commands we can run:

version: '7.1'

# Define default environment variables to pass
# to Docker Compose
environment:
  RAILS_ENV: development

compose:
  files:
    - .dockerdev/compose.yml
  project_name: attic

interaction:
  # This command spins up a Rails container with the required dependencies (such as databases),
  # and opens a terminal within it.
  runner:
    description: Open a Bash shell within a Rails container (with dependencies up)
    service: rails
    command: /bin/bash

  # Run a Rails container without any dependent services (useful for non-Rails scripts)
  bash:
    description: Run an arbitrary script within a container (or open a shell without deps)
    service: rails
    command: /bin/bash
    compose_run_options: [ no-deps ]

  # A shortcut to run Bundler commands
  bundle:
    description: Run Bundler commands
    service: rails
    command: bundle
    compose_run_options: [ no-deps ]

  rails:
    description: Run Rails commands
    service: rails
    command: bundle exec rails
    subcommands:
      s:
        description: Run Rails server at http://localhost:3000
        service: web
        compose:
          run_options: [ service-ports, use-aliases ]
      test:
        description: Run unit tests
        service: rails
        command: bundle exec rails test
        environment:
          RAILS_ENV: test

  yarn:
    description: Run Yarn commands
    service: rails
    command: yarn
    compose_run_options: [ no-deps ]

  psql:
    description: Run Postgres psql console
    service: postgres
    default_args: attic_development
    command: psql -h postgres -U postgres

provision:
  # We need the `|| true` part because some docker-compose versions
  # cannot down a non-existent container without an error,
  # see https://github.com/docker/compose/issues/9426
  - dip compose down --volumes || true
  - dip compose up -d postgres
  - dip bash -c bin/setup

The first command to run would be dip provision which as shown above is for a fresh setup - shutting down volumes, starting postgres service, and running the bin/setup script which will install/update gems, prepare the database, and restart the rails:

#!/usr/bin/env ruby
require "fileutils"

# path to our application root.
APP_ROOT = File.expand_path("..", __dir__)

def system!(*args)
  system(*args) || abort("\n== Command #{args} failed ==")
end

FileUtils.chdir APP_ROOT do
  # This script is a way to set up or update our development environment automatically.
  # This script is idempotent, so that we can run it at any time and get an expectable outcome.
  # Add necessary setup steps to this file.

  puts "== Installing dependencies =="
  system! "gem install bundler --conservative"
  system("bundle check") || system!("bundle install")

  # puts "\n== Copying sample files =="
  # unless File.exist?("config/database.yml")
  #   FileUtils.cp "config/database.yml.sample", "config/database.yml"
  # end

  puts "\n== Preparing database =="
  system! "bin/rails db:prepare"

  puts "\n== Removing old logs and tempfiles =="
  system! "bin/rails log:clear tmp:clear"

  puts "\n== Restarting application server =="
  system! "bin/rails restart"
end

We can run rails-specific commands defined in the dip.yml just like we did previously but prefixed with dip, but a more extensive list of available commands are written in the .dockerdev/README.md file:

# snippet from .dockerdev/README.md # 
# run rails server
dip rails s

# run rails console
dip rails c

# run rails server with debugging capabilities (i.e., `debugger` would work)
dip rails s

# or run the while web app (with all the dependencies)
dip up web

# run migrations
dip rails db:migrate

# pass env variables into application
dip VERSION=20100905201547 rails db:migrate:down

# simply launch bash within app directory (with dependencies up)
dip runner

# execute an arbitrary command via Bash
dip bash -c 'ls -al tmp/cache'

# Additional commands

# update gems or packages
dip bundle install
dip yarn install

# run psql console
dip psql

# run tests
# TIP: `dip rails test` is already auto prefixed with `RAILS_ENV=test`
dip rails test

# shutdown all containers
dip down

From the snippet above we can run the same rails-centric commands prefixed with dip, however, there is also an option that makes the prefix unnecessary depending on your shell.

Final Thoughts

I have found the interactive CLI option to be my go-to approach as it adheres to best practices and doubly gives the options to set the Rails and PostgreSQL versions (among other environment variables) which I usually hardcode. Currently working on various self and OSS projects with differing rails versions and I am at peace with integrating them with docker and contributing without the hassle of wrangling with the setup of multiple rails versions and gems issues based on my OS.

I find that I retain knowledge better through reading than via videos as one requires more focus, therefore my study will be textbook based. I will generally follow this study plan for each certification:

1. Read the study guide provided by Wiley for the certifications, such as AWS

   Certified Solutions Architect Study Guide Associate (SAA-C03) Exam Fourth Edition. I plan to read 1-2 chapters a day depending on how meaty the content proves to be.

2. Do the practice tests at the end of the chapter.

3. Before reading the next chapter, redo the practice tests for all the prior chapters. Hopefully, this helps with some memory retention and faster recall of previous knowledge.

After completing the book and feeling confident about the practice tests I have taken, I will schedule the certification test about a week out. This will give me ample time to review and do some full practice exams to simulate the certification exam. I estimate that preparing for each certification will take me 1 - 1.5 months, so I figure, conservatively and with no retests, to complete all the certs by end of July? Fingers crossed.

This series won't be too long I believe, as there are only 3 certifications I have planned for this year. I only plan on writing a post once I pass each one and linking to the certification. Any change in pace might also warrant a small post to reset my resolve, but I plan to adhere to the timeline above.

Task

There is some data on Nautilus App Server 2 in Stratos DC. which needs to be altered in several of the files. On Nautilus App Server 2, alter the /home/BSD.txt file as per the details given below:

a. Delete all lines containing word software and save results in /home/BSD_DELETE.txt file. (Please be aware of case sensitivity)

b. Replace all occurrences of the word or to is and save results in /home/BSD_REPLACE.txt file.

Note: Let's say you are asked to replace the word to with from. In that case, make sure not to alter any words containing this string; for example upto, contributor, etc.

Information

Implementation

1. ssh into App Server 2 and enter the password of BigGr33n when prompted.

    ssh steve@172.16.238.11
   

2. Delete all lines containing word software and save results in /home/BSD_DELETE.txt file. (Please be aware of case sensitivity):

    sed '/software/d' BSD.txt > BSD_DELETE.txt
   

   This command uses sed to modify the contents of BSD.txt and write the results to a new file called BSD_DELETE. Specifically, the command does the following:

   1. /software/d is a command that specifies a pattern to match and an action to perform. In this case, the pattern is the string "software", and the action is to delete any line that contains that string.

   2. BSD.txt is the input file that sed should read and apply the command

   3. > is a redirection operator that tells the shell to redirect the output of the command to a new file instead of the screen.

   4. BSD_REPLACE.txt is the name of the file that sed should write the modified output to.

***Please be aware of case sensitivity*** is already taken care of here as sed by default is case sensitive and will match the exact case in the pattern. To make it case insensitive, i.e match both Software, software, SOFTWARE, etc, we can add the I flag to the action: /software/dI.

1. Replace all occurrences of the word or to is and save results in /home/BSD_REPLACE.txt file.

    sed 's_\bor\b_is_g' BSD.txt > BSD_REPLACE.txt
   

   1. s is a sed command that stands for "substitute". It tells sed to search for a pattern and replace it with something else.

   2. _\bor\b_is is the search pattern that sed will look for in the input text. This pattern consists of the following components:

      • _ is a delimiter character that separates the search pattern from the replacement text. It can be any character, but in this case, it is used instead of the more common / character, to avoid conflicts with the / characters in the pattern.

      • \b is a regular expression metacharacter that matches a word boundary, i.e. the transition between a word character (alphanumeric or underscore) and a non-word character (such as space or punctuation). The word boundary ensures that only the exact word between the boundaries will be matched.

      • or is the exact string that sed will look for in the input text. Only whole words that match the pattern will be replaced, not substrings that contain "or", such as "for", "gore", etc.

      • is is the replacement text that sed will substitute for the matched pattern. In this case, the replacement text is simply the word "is".

   3. g is an option that stands for "global". It tells sed to perform the substitution operation globally, i.e. on all occurrences of the pattern in each line of the input text, rather than just the first occurrence.

Learning Takeaways

1. There are a lot of differences between sed on macOS and gnu-sed used in Linux distros. For example, for this sample text, I had to the command substitution command differently:

    # me.md
    hello world
    goodbye world
    World Empire
    them form forth for maybe
    angry sleight for fore
   

   0n macOS, notice how I had to write the word boundaries:

    sed 's/[[:<:]]for[[:>:]]/pour/g' me.md
   
    hello world
    goodbye world
    World Empire
    them form forth pour maybe
    angry sleight pour fore
   

2. Brew has a gnu-sed formula that can be run on macOS. This to me is a better option than learning two different commands' quirks:

    brew install gnu-sed
   

   

   I will replace my default sed with gsed with the approach above. In my .zshrc file I can add the above PATH variable and then source the file:

    vim ~/.zshrc
   
    # replace gsed (gnu-sed from homebrew) with native sed
    export PATH="/opt/homebrew/opt/gnu-sed/libexec/gnubin:$PATH"
   
    source ~/.zshrc
   
    sed --version
   
    sed (GNU sed) 4.9
    Copyright (C) 2022 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
   
    Written by Jay Fenlason, Tom Lord, Ken Pizzini,
    Paolo Bonzini, Jim Meyering, and Assaf Gordon.
   
    This sed program was built without SELinux support.
   
    GNU sed home page: <https://www.gnu.org/software/sed/>.
    General help using GNU software: <https://www.gnu.org/gethelp/>.
    E-mail bug reports to: <bug-sed@gnu.org>.
   

During the daily standup, it was pointed out that the timezone across Nautilus Application Servers in Stratos Datacenter doesn't match the local datacenter's timezone, which is America/Asuncion.

Information

Implementation

We have three App Servers in which we have to update the timezones to America/Asuncion if not already to it. Therefore it's just a matter of accessing each server and using timedatectl to update the timezone. Repeat the steps below for each server:

1. ssh into the App Server with login details above:

   

1. View the currently set Time Zone to determine if it needs to be updated. Update if not set to America/Asuncion:

   

   Looks like it needs to be updated. We can do so using timedatectl set-timezone ZONE command. Reference timedatectl --help for other commands.

   

Learning Takeaways

A great command to keep in the toolbox. Never had any use for it before, but I just used this to update my timezone to America/Edmonton on my local ubuntu server. Granted, it won't pay any dividends, other than when running the date command to give the current time in MST.

The System admin team of xFusionCorp Industries has installed a backup agent tool on all app servers. As per the tool's requirements, they need to create a user with a non-interactive shell.

Therefore, create a user named javed with a non-interactive shell on the App Server 2

Information

Implementation

1. ssh into App Server 2 using as steve : ssh steve@172.16.238.11 . Enter password when prompted. Currently only users steve and ansible exist in this server

   

2. Create user javed with non-interactive shell: sudo adduser -s /sbin/nologin javed . -s is a flag to set the shell for a user. Below I authenticate as root so the sudo prefixes are unnecessary.

nologin displays a message that an account is not available and closes the connection and returns non-zero. It is intended as a replacement shell field to deny login access to an account.

Learnings

1. Creating a user with a non-default $SHELL. Th nonlogin shell is non-interactive and used for disabling a user account in the case of suspicious activity or upon user workplace/contract termination. The shell can also be updated using usermod -s .

2. A custom message can be set for any attempts to log in as the disabled user account by editing/creating a /etc/nologin.txt file.

There are new requirements to automate a backup process that was performed manually by the xFusionCorp Industries system admins team earlier. To automate this task, the team has developed a new bash script xfusioncorp.sh. They have already copied the script on all required servers, however they did not make it executable on one the app server i.e App Server 3 in Stratos Datacenter.

Please give executable permissions to /tmp/xfusioncorp.sh script on App Server 3. Also make sure every user can execute it.

Information

We have access to credentials servers from the infrastructure details:

Implementation

1. Starting from a jump_host we ssh into App Server 3 as the user banner:

In the /tmp directory we can view the current permissions and ownership of the /tmp/xfusioncorp.sh file. We can see that it belongs to root user and it has not been made executable:

1. Make xfusioncorp.sh executable for all everyone (users, groups, other):

1. Make xfusioncorp.sh readable. A script can't be executed if it can't be read - at least for Bourne (again) shells I believe? Let's give read access to everyone and see the contents of the script.

1. Execute the script:

Learning Takeaways

1. Does a script file need to be readable? I guess all the script files I have ever written or run have always had read permissions set. I guess since a script is a text file it would require this permission, while binary files would not.

Types of Load Balancing

There are three main types of load balancing:

1. Round-robin: Incoming requests are distributed to each server in the pool in a circular fashion.

2. Least connections: Incoming requests are distributed to the server with the least number of active connections.

3. IP hash: The client's IP address is used to determine which server in the pool should handle the request.

Load Balancing Methods for Nginx and Apache

Nginx Load Balancing Methods

1. Round-robin: Default method in Nginx.

2. Least connections: Enabled with the least_conn directive.

3. IP hash: Enabled with the ip_hash directive.

Apache Load Balancing Methods

Apache supports various load balancing methods through the mod_proxy_balancer module, including:

1. Round-robin: Incoming requests are distributed to each server in the pool in a circular fashion.

2. Least connections: Incoming requests are distributed to the server with the least number of active connections.

3. Session-based: Incoming requests are distributed based on session information, ensuring that requests from the same session are sent to the same server.

4. URL hash: Incoming requests are distributed based on the URL requested, ensuring that requests for the same URL are sent to the same server.

In addition to load balancing methods, a load factor is an important consideration for efficient load balancing. Load factor refers to the server's ability to handle traffic based on its resources, such as CPU, memory, and disk I/O.

Load factor can be measured in various ways, such as the number of connections or requests per second, the CPU or memory usage, or the network throughput. Load balancing algorithms can take into account the load factor of each server and distribute traffic accordingly to ensure that the server with the least load factor gets the incoming traffic.

Conclusion

By choosing the right load balancing method and considering the load factor of each server, web applications can handle high traffic loads and provide a smooth user experience.

The tools we want our team to be able to use are well-known and widely used by multiple DevOps teams, so we will introduce a single DevOps Tooling Solution that will consist of these tools:

1. Jenkins – free and open source automation server used to build CI/CD pipelines.

2. Kubernetes – open-source container-orchestration system for automating computer application deployment, scaling, and management.

3. Jfrog Artifactory – Universal Repository Manager supporting all major packaging formats, build tools and CI servers. Artifactory.

4. Rancher – an open-source software platform that enables organizations to run and manage Docker and Kubernetes in production.

5. Grafana – a multi-platform open-source analytics and interactive visualization web application.

6. Prometheus – An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach.

7. Kibana – Kibana is a free and open user interface that lets you visualize your Elasticsearch data and navigate the Elastic Stack.

In this project you will implement a solution that consists of the following components:

1. Infrastructure: AWS

2. Webserver Linux: RHEL 9

3. Database Server: Ubuntu 22.04 + MySQL

4. Storage Server: RHEL 9 + NFS Server

5. Programming Language: PHP

6. Code Repository: GitHub

In the diagram below you can see a common pattern where several stateless Web Servers share a common database and also access the same files using Network File System (NFS) as shared file storage. Even though the NFS server might be located on completely separate hardware for Web Servers it will resemble a local file system from where they can serve the same files.

It is important to know what storage solution is suitable for what use cases, thus we need to ask the following questions: what data will be stored, in what format, how this data will be accessed, by whom, from where, how frequently, etc. Based on this we will be able to choose the right storage system for your solution.

Step 1: Prepare NFS Server

Setup LVM on RHEL 9 OS

This setup will be very similar to the last project's LVM setup. To avoid extra work on my end of rewriting the same steps, I instead will embed the setup from the last project as a gist here. Notable changes we want to account for are:

1. Formatting the disks as xfs instead of ext4

2. The volume names will be opt-lv, apps-lv, and logs-lv

3. Mount points will be on /mnt directory for the logical volumes as follows:

   1. Mount lv-apps on /mnt/apps – To be used by web servers

   2. Mount lv-logs on /mnt/logs – To be used by web server logs

   3. Mount lv-opt on /mnt/opt – To be used by the Jenkins server in Next Project

Here's our setup of the logical volumes:

Mount the logical volumes:

sudo mkdir -p /mnt/apps /mnt/logs /mnt/opt

# syncing logs and mounting
sudo rsync -av /var/log /mnt/logs
sudo mount /dev/nfsdata-vg/logs-lv /mnt/logs
sudo rsync -av /mnt/logs/ /var/log

# mount apps-lv & opt-lv
sudo mount /dev/nfsdata-vg/apps-lv /mnt/app
sudo mount /dev/nfsdata-vg/opt-lv /mnt/opt

We can view our newly created mounts. We can see the target (/mnt/logs), source (/dev/mapper/nfsdata-vg-logs--lv), and fstype (xfs).

sudo findmnt | grep '/mnt'

Install the NFS server, configure it to start on reboot and make sure it is up and running:

sudo yum -y update
sudo yum -y install nfs-utils
sudo systemctl enable nfs-server.service
sudo systemctl start nfs-server.service
sudo systemctl status nfs-server.service

Set up permissions that will allow our Web servers to read, write and execute files on NFS.

sudo chown -R nobody: /mnt/apps
sudo chown -R nobody: /mnt/logs
sudo chown -R nobody: /mnt/opt

sudo chmod -R 777 /mnt/apps
sudo chmod -R 777 /mnt/logs
sudo chmod -R 777 /mnt/opt

Furthermore, we want our webservers (not created yet) to be able to access our mounts as clients. For simplicity, the webservers will all be installed within the same subnet.

We need to retrieve out subnet cidr value to configure access to our NFS for our web servers. The subnet value can be found within the instance Networking tab and following the subnet link:

sudo vi /etc/exports

# /etc/exports
/mnt/apps 172.31.16.0/20(rw,sync,no_all_squash,no_root_squash)
/mnt/logs 172.31.16.0/20(rw,sync,no_all_squash,no_root_squash)
/mnt/opt  172.31.16.0/20(rw,sync,no_all_squash,no_root_squash)

# exit vi
sudo exportfs -arv

Use rcpinfo -p | grep nfs to check the port used by NFS and include it as a rule in the Security Groups.

From the above, we need to open port 2049. Additionally, to allow access to our NFS server from clients, we also need to open TCP 111, UDP 111, and UDP 2049. With all these related rules to our NFS, I created a specific SG for it. The source field is our subnet cidr.

Configure the Database Server

As we have done multiple times in previous projects, we want to install MySQL, create a database and user, and grant the user access to the database from the webservers subnet cidr:

sudo yum -y update
sudo yum -y install mysql-server

# start the mysql services
sudo systemctl enable mysqld
sudo systemctl restart mysqld

Prepare the Web Servers

Our Web Servers need to be able to serve the same content from shared storage solutions, in our case – NFS Server and MySQL database. We have already seen how to access a MySQL server from a client. For storing shared files that our Web Servers will use – we will utilize NFS and mount previously created Logical Volume apps-lv to the folder where Apache stores its files to be served to the users (/var/www).

This approach will make our Web Servers stateless, which means we will be able to add new ones or remove them whenever we need, and the integrity of the data (in the database and on NFS) will be preserved.

In the next steps, we will do the following 3 times:

• Launch an RHEL 9 EC2 instance

• Configure the NFS client (this step must be done on all three servers)

• Deploy a Tooling application to our Web Servers into a shared NFS folder

• Configure the Web Servers to work with a single MySQL database

We need to first install our NFS client on our Web Server instances:

sudo yum -y install nfs-utils nfs4-acl-tools

Then mount /var/www/ and target the NFS server's export for `apps`. We should see the NFS mounted on our web server:

sudo mkdir /var/www
sudo mount -t nfs -o rw,nosuid 172.31.25.225:/mnt/apps /var/www
sudo mount -t nfs -o rw,nosuid 172.31.25.225:/mnt/logs /var/log/httpd
df -h

Additionally, we can persist the above changes by adding them to our /etc/fstab :

# /etc/fstab
172.31.25.225:/mnt/apps /var/www nfs defaults 0 0
172.31.25.225:/mnt/logs /var/log/httpd nfs defaults 0 0

Install Remi’s repository, Apache and PHP

sudo yum -y install httpd

sudo dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm

sudo dnf -y install dnf-utils http://rpms.remirepo.net/enterprise/remi-release-9.rpm

sudo dnf -y module reset php

sudo dnf -y module enable php:remi-8.1

sudo dnf -y install php php-opcache php-gd php-curl php-mysqlnd

sudo systemctl enable php-fpm

sudo systemctl start php-fpm

sudo setsebool -P httpd_execmem 1

We can ascertain that our NFS is mounted properly on our web server by verifying that both /var/www in our Web Server(s) and /mnt/apps in our NFS have the same Apache files and directories. Apply the same verification for our logs - /var/log for our Web Servers and /mnt/logs for our NFS.

1. Fork the tooling source code from Darey.io Github Account to your Github account.

2. Deploy the tooling website’s code to the Webserver. Ensure that the html folder from the repository is deployed to /var/www/html

cd /var/www/html
sudo git clone https://github.com/cdrani/tooling.git
sudo mv tooling/. .
sudo mv -R html/. .
sudo rm -R html

Again our web servers should have the files in /var/www/html:

We need to update permissions on /var/www/html

Update the website’s configuration to connect to the database (in /var/www/html/functions.php file). We will make use of the webaccess user with a password of Super01! we created earlier which has access to the `tooling` database:

$db = mysqli_connect('<DB Server Private IP Address>', '<MySQL username>', '<MySQL password>', '<Database>');

Now we need to set up a users table in our tooling database. Fortunately, we already have a pre-configured /var/www/html/tooling-db.sql file that will create a users table for us. All we have to do is import it into our tooling database. We can import tooling-db.sql script to our database with this command mysql -h <databse-private-ip> -u <db-username> -p <db-pasword> < tooling-db.sql.

# mysql -h <databse-private-ip> -u <db-username> -p <db-pasword> < tooling-db.sql
mysql -u webaccess -h 172.31.18.126 -p tooling < /var/www/html/tooling-db.sql

sudo sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinux
sudo setenforce 0
sudo systemctl restart httpd

Visiting either of our web servers (http://<Public IP Address>/index.php) should display the following login page and home page upon entering our credentials:

Learning Outcomes

1. Configure an NFS client.

2. Deploy a Tooling application to our Web Servers into a shared NFS folder.

3. Configure the Web Servers to work with a single MySQL database.

The rationale for this level of redundancy is a scalability issue. Many times a post/link receives an onslaught of upvotes on a site like HackerNews causing thousands to millions of visitors to the link and causing the page to crash as the server was not optimized to handle the level of traffic, thus causing slow page loads or killing the server altogether. This necessitated the need for multiple servers to handle these volumes of loads by distributing these requests across their servers. Sites like Google and Reddit these types of issues to serve millions of users every day.

Scalability is the property of a system to be able to handle a growing load by increasing resources. There are two types of scalability approaches to consider:

1. vertical scaling: Increasing the CPU and ram on our current server or configuring a more powerful server as a replacement. The limitation is that there's only so much CPU and ram that can be installed on a server.

2. horizontal scaling: Spreading the load across multiple servers. This is the more common approach as it can be applied seamlessly and almost infinitely. Additionally, this can also be automated to spin up (scale out) or spin down (scale in) servers based on CPU and Memory load monitored metrics.

Currently, we have 2 web servers set up from the previous project. For a user/client to access them they would need to either or both of the IP addresses. This is not too bad with our limited number of servers, but this is not feasible for sites like google where they have millions of servers.

We can alleviate clients/users from this bad UX by providing only a single point of access to reach our servers. A Load Balancer (LB) solves this by distributing clients' requests among multiple servers and ensuring that the load is shared in an optimized manner.

Let's update our current solution architecture with an LB sandwiched between our Client and web servers. This will allow our users to access our site using a single URL/IP Address. We will make use of only 2 web servers, but the following approach will not change regardless of the number of web servers.

Prerequisites

We already have the following infrastructure setup from the previous project:

Configure Apache As A Load Balancer

For our LB we will set up an L7 Application LB with Apache. Therefore, let's create an Ubuntu EC2 instance and open port 80 in the SG.

• insert image here

Install Apache

sudo apt -y update
sudo apt -y install apache2
sudo apt -y install libxml2-dev

Enable Apache modules:

sudo a2enmod rewrite proxy proxy_balancer proxy_http headers lbmethod_bytraffic

# restart apache to activate enabled modules
sudo systemctl restart apache2

sudo systemctl status apache2

Configure Load Balancing

Our LB configuration is done in the /etc/apache/sites-available/000-default.conf file. Here we can define the Web Servers and the balancing method. There are several choices for load balancing methods: bytraffic, bybusyness, byrequests, and by heartbeat . We can control in which proportion the traffic must be distributed by loadfactor parameter. Setting the same loadfactor to both servers will evenly distribute the traffic. I made the following changes to the file with the Private IP address of my Web Servers:

After the above changes to our config file, we need to restart our Apache server. Following that, we should be able to now visit our site using the LB's Public IP Address: `http://54.185.27.126/index.php`. We should be able to log in and see the same admin page from the previous project.

We can view our Apache logs by running the following in both Web Server instances:

sudo tail -f /var/log/httpd/access_log

We can see below that the logs are the same - the same requests at the same times. This is due to them both sharing the same mount point /var/log/httpd on our NFS server. We can't differentiate between which Web Server is being selected by our LB to serve.

To resolve this we need to unmount our Web Servers from the NFS Server so that the Web Servers have their separate logs. Let's do the following for both Web Servers:

sudo systemctl stop httpd
sudo df -h
sudo umount /var/log/httpd
sudo df -h

Additionally, we need to prevent the /var/log/http directory from being mounted on boot as we currently have it configured in the /etc/fstab file. Let's comment it out on both servers:

#/etc/fstab

172.31.25.225:/mnt/apps /var/www nfs defaults 0 0
# 172.31.25.225:/mnt/logs /var/log/httpd nfs defaults 0 0

Then reload our mounts so these changes apply on boot as well and restart our Apache server:

sudo mount -a
sudo systemctl start httpd

Now visiting our site via our LB IP address we can view Web Server is running our requests. Below we see that our requests are being distributed equally between our two Web Servers on every page reload:

This is our final setup now:

Learning Outcomes:

1. What is scalability? What are vertical and horizontal scaling?

2. What is Load Balancing? What is the purpose?

3. How to set up an Apache LB (install + modules + configuration with Web Servers)

4. View Web Servers logs

1. Configure storage subsystem for Web and Database servers based on Linux OS. The focus of this part is to give you practical experience in working with disks, partitions and volumes in Linux.

2. Install WordPress and connect it to a remote MySQL database server. This part of the project will solidify your skills in deploying Web and DB tiers of Web solutions.

Three-tier Architecture is a client-server software architecture pattern that comprises 3 separate layers.

1. Presentation Layer (PL): This is the user interface such as the client-server or browser on your laptop.

2. Business Layer (BL): This is the backend program that implements business logic. It can be an Application and/or Webserver.

3. Data Access or Management Layer (DAL): This is the layer for computer data storage and data access. Database Server or File System Server such as FTP server, or NFS Server.

Our 3-Tier Setup

1. A Laptop or PC to serve as a client

2. An EC2 Linux Server as a web server (This is where we will install WordPress)

3. An EC2 Linux server as a database (DB) server

STEP 1: Setup Web Server

We will use an RHEL9 instance as our server. Additionally, we need to create and attach three 10GB volumes to our instance.

The volume must be in the same Availability Zone as the instance:

Make sure to select the correct instance for which to attach the volumes:

Volume Configuration

Use lsblk command to inspect what block devices are attached to the server. The three newly created block devices will likely have names like xvdf, xvdh, and xvdg .

lsblk

We can use df -h to view all mounts and free space on our instance

Use the gdisk utility to create a single partition on each of the 3 disks. The utility is interactive with some command options. We only need to make use of the n to create a new partition and w to save the new partition to the disk and exit.

# sudo gdisk <disk>
sudo gdisk /dev/xvdf
sudo gdisk /dev/xvdg
sudo gdisk /dev/xvdh

Here's an example of partitioning one of the disks using the default (pressing ENTER/RETURN key):

We can now see the partitions we created in each disk now:

Run sudo lvmdiskscan command to check for available partitions and the type of volumes

sudo yum install -y lvm2
sudo lvmdiskscan

Use pvcreate utility to mark each of the 3 disks as physical volumes (PVs) to be used by LVM.

sudo pvcreate /dev/xvdf1
sudo pvcreate /dev/xvdg1
sudo pvcreate /dev/xvdh1
sudo pvs

Use vgcreate utility to add all 3 PVs to a volume group (VG). Name the VG webdata-vg. We can verify the creation of the VG using sudo vgs :

sudo vgcreate webdata-vg /dev/xvdf1 /dev/xvdg1 /dev/xvdh1
sudo vgs

Use lvcreate utility to create 2 logical volumes. apps-lv (Use half of the PV size), and logs-lv Use the remaining space of the PV size. NOTE: apps-lv will be used to store data for the Website while logs-lv will be used to store data for logs.

sudo lvcreate -n apps-lv -L 14G webdata-vg
sudo lvcreate -n logs-lv -L 14G webdata-vg
sudo vgs

Verify the entire setup:

sudo vgdisplay -v #view complete setup - VG, PV, and LV
sudo lsblk

From above we can see that apps-lv and logs-lv are of type lvm, but we want to reformat it to an ext4 filesystem.

sudo mkfs -t ext4 /dev/webdata-vg/apps-lv
sudo mkfs -t ext4 /dev/webdata-vg/logs-lv

Create /var/www/html directory to store website files and mount them on apps-lv LV.

sudo mkdir -p /var/www/html
sudo mount /dev/webdata-vg/apps-lv /var/www/html

We follow the same step as above for the logs, but first, we need to back up the files in /var/log using rsync before mounting it on the new destination folder:

sudo mkdir -p /home/recovery/logs
sudo rsync -av /var/log/ /home/recovery/logs/
sudo mount /dev/webdata-vg/logs-lv /var/log
sudo rsync -av /home/recovery/logs/. /var/log

Update /etc/fstab file

The UUID of the device will be used to update the /etc/fstab file.

sudo blkid | grep 'webdata'

The /etc/fstab file stores static information about filesystems, their mount points and mount options. This file is read at boot time to determine the overall file system structure, and thereafter when a user executes the mount command to modify that structure.

Test the configuration, reload the daemon, and verify the setup:

sudo mount -a
sudo systemctl daemon-reload
sudo df -h

Install WordPress on WebServer

Install Apache and its dependencies and start the web server:

sudo yum -y update
sudo yum -y install wget httpd php php-mysqlnd php-fpm php-json

sudo systemctl enable httpd
sudo systemctl start httpd

Install PHP and its dependencies:

sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
sudo yum install yum-utils http://rpms.remirepo.net/enterprise/remi-release-9.rpm

sudo yum module list php
sudo yum module reset php
sudo yum module enable php:remi-8.1

sudo yum -y install php php-opcache php-gd php-curl php-mysqlnd

sudo systemctl enable php-fpm
sudo systemctl start php-fpm

sudo setsebool -P httpd_execmem 1

sudo systemctl restart httpd

Download WordPress

mkdir wordpress; cd wordpress;

wget http://wordpress.org/latest.tar.gz
sudo tar xzvf latest.tar.gz
rm -rf latest.tar.gz

cp wordpress/wp-config-sample.php wordpress/wp-config.php
cp -R wordpress /var/www/html/

Configure SELinux Policies

sudo chown -R apache:apache /var/www/html/wordpress
sudo chcon -t httpd_sys_rw_content_t /var/www/html/wordpress -R
sudo setsebool -P httpd_can_network_connect=1

Setup Database Server

Launch a second RedHat EC2 instance that will have a role – ‘DB Server’
Repeat the same steps as for the Web Server, but instead of apps-lv create db-lv and mount it to /db directory instead of /var/www/html/.

Install MySQL on DB Server

sudo yum -y update
sudo yum -y install mysql-server

sudo systemctl enable mysqld
sudo systemctl restart mysqld

Configure DB to work with WordPress

Use sudo mysql to login as root and setup a database and a user with access to the database:

Configure WordPress to Connect to a Remote Database

For this, we need to open port 3306 on the DB Server instance's Security Group (SG). Furthermore, we should limit the source to our Web Server's private IP address:

Additionally, we forgot to open port 80 on our Web Server's instance. I have already set up a specific SG for web servers, but reference the first rule in the image above for how it should appear.

We should now be able to access our DB Server from our Web Server as we did in the previous project:

sudo mysql -u admin -p -h 172.31.28.148

View WordPress Site

With our Web Server running we should be able to view our WordPress site using the server's public IP address: `http://54.245.4.149/wordpress/`:

However, we come across the above error. All that's required on our part is to update our wp-config.php file to provide details regarding our database settings. This file is located in /var/www/html/wordpress/ . The DB_HOST value is our DB Server private IP address:

After reloading the page we should enter the install flow:

Note: Remember to Terminate the two instances and delete the volumes if no longer needed lest you incur added AWS costs.

Learning Outcomes

1. Creating & attaching volumes to EC2 instances

2. Creating Partitions, Physical Volumes, Volume Groups, and Logical Volumes

3. Mounting

4. Installing WordPress and configuring Database

• Must have a default /home directory, ie /home/cdrani/ for user cdrani

• Each user must be added to a developers group

• An .ssh folder with an authorized_keys file that includes the rsa.pub key of the current user ($USER)

STEP 1: Setup Private & Public keys for $USER

1. Generating ssh private (id_rsa) and public key (id_rsa.pb) for $USER. The public key will be copied over to each onboarded user's .ssh/authorized_keys file. We will make use of ssh-keygen and just press Enter/Return key to accept the default file names and placement. In the end, we should have both id_rsa and id_rsa.pub keys in ~/.ssh folder.

    sudo ssh-keygen
   

   

STEP 2: Onboarding Script

Now for the script itself. It's heavily documented with comments, but some additional criteria I included are to verify that the script is run by a root user, a .csv file is passed as an argument and a default password of password is set for each onboarded user with an expiration date of 7 days.

#!/bin/bash

# Exit if script is not run as root
if [ "$(id -u)" != "0" ]; then
  echo "This script must be run as root."
  exit 1
fi

# Exit if file is not provided
if [ $# -eq 0 ]; then
  echo "Please provide the name of the CSV file."
  exit 1
fi

# Exit if file provided is not a .csv file
file="$1"
if [ "${file##*.}" != "csv" ]; then
  echo "The file must be a CSV file."
  exit 1
fi

# Create developers group if it doesn't exist
if ! getent group developers > /dev/null 2>&1; then
  groupadd developers
fi

# Read the CSV file
while IFS=',' read -r name
do
  username=$name

  # Check if user exists
  if id "$username" >/dev/null 2>&1; then
    echo "User $username already exists."
    continue
  fi

  # Create user
  useradd "$username" -g developers -m
  echo "$username:default_password" | chpasswd

  # Set password expiration to 7 days
  # After expiration user needs to set/change password
  chage -m 7 "$username"

  # Ensure .ssh folder exists for user, otherwise create one
  home_dir=$(eval echo "~$username")
  ssh_dir="$home_dir/.ssh"
  if [ ! -d "$ssh_dir" ]; then
    mkdir "$ssh_dir"
    chmod 700 "$ssh_dir" $username
  fi

  # Add public key to users' ~/.ssh/authorized_keys
  auth_keys="$ssh_dir/authorized_keys"
  touch "$auth_keys"
  chmod 600 "$auth_keys" $username
  # Need to give ownership to user else cannot login as them!!!
  chown $username:developers "$auth_keys"
  echo $(cat ~/.ssh/id_rsa.pub) >> "$auth_key"
done < "$file"

To run the script we need a *.csv file with a list of names:

# names.csv
beth
mark,
lucy,
fred,
liza

Then we need our script to be executable before we run it:

sudo chmod +x onboard_users.sh
sudo ./onboard_users.sh names.csv

Let's test out some of the conditionals we have in place:

We now should have /home/$user (for ex. /home/beth/) directories for each user in the names.csv file:

Finally, we should be able to log in as any of the newly onboarded users, however, our new users don't use the same private key that we set up when launching our ec2 instance - we must use the one we generated above (id_rsa).

First, we need to ensure our key is not publicly viewable, i.e it is not writable:

sudo chmod -w id_rsa

After the above, we can then copy the private key from our remote server (ec2) to our local environment using scp:

# scp -i <key> <remote_username>@<Host>:<PathToFile>   <LocalFileLocation>
scp -i ~/Desktop/webserver-ec2.pem ubuntu@35.85.56.192:~/.ssh/id_rsa ~/.ssh/aux_id_rsa

With that complete, from our local computer, we can log in as an onboarded user:

ssh -i ~/.ssh/aux_id_rsa fred@38.85.56.192

Only users with the auth key can log in:

Note: One can log in using the *.pem key for the ec2 instance as well.

Bonus:

Testing the script several times to address issues necessitated removing users as well. This quickly turned into a pain point that required a deboard.sh script. It has the same checks as the onboard.sh one which could be refactored out into a separate script that runs before both of them, but it's good enough.

#!/bin/bash

if [ "$(id -u)" != "0" ]; then
  echo "This script must be run as root."
  exit 1
fi

# Check if argument is provided
if [ $# -eq 0 ]; then
  echo "Please provide the name of the CSV file."
  exit 1
fi

# Check if argument is a .csv file
file="$1"
if [ "${file##*.}" != "csv" ]; then
  echo "The file must be a CSV file."
  exit 1
fi

# Read the CSV file
while IFS=',' read -r fname lname
do
  username=$fname

  # Check if user exists
  if ! id "$username" >/dev/null 2>&1; then
    echo "User $username does not exist."
    continue
  fi

  # Remove user
  sudo userdel -r "$username"
done < "$file"

In this case, our Web Server has a role of a "Client" that connects and reads/writes to/from a Database (DB) Server (MySQL, MongoDB, Oracle, SQL Server or any other), and the communication between them happens over a Local Network (it can also be an Internet connection, but it is a common practice to place Web Server and DB Server close to each other in a local network).

Implement A Client-Server Architecture

We need to start with two Linux ec2 instances - one to act as the client, and the other as the server

In the client one we need to setup mysql-client :

sudo apt update -y
sudo apt install mysql-client -y

As for server, we require mysql-server :

sudo apt update -y
sudo apt install mysql-server -y

Additionally, it would be best practice to update the root user password, run mysql_secure_installation , and reset some MySQL defaults.

Completing the above for the client is unnecessary since the client will just be accessing the server .

By default, both of your EC2 virtual servers are located in the same local virtual network, so they can communicate with each other using local IP addresses. MySQL server uses TCP port 3306 by default, so you will have to open it by creating a new entry in ‘Inbound rules’ in ‘MySQL server’ Security Groups. For extra security, do not allow all IP addresses to reach your ‘MySQL server’ – allow access only to the specific local IP address of the ‘MySQL client’.

You might need to configure the MySQL server to allow connections from remote hosts.

cat /etc/mysql.conf.d/mysqld.cnf | grep '^bind-address'
# bind-address = 127.0.0.1


sed -i 's/bind-address.*/bind-address = 0.0.0.0/' /etc/mysql/mysql.conf.d/mysqld.cnf

cat /etc/mysql.conf.d/mysqld.cnf | grep '^bind-address'
# bind-address = 0.0.0.0

sudo systemctl restart mysql

Let's create an admin user and test_db database from our server instance:

With the above setup, we can now connect to our MySQL server from our client using the above admin user. The -h option allows us to specify the IP or hostname of the MySQL server

# mysql -u <user> -p -h <host or ip address of mysql server>
mysql -u admin -p -h 172.31.30.207

We have now accessed the server as the client. Let's create a contacts table and populate it:

We should also be able to view that table and its row from the server instance and add additional rows to it.

1. MongoDB (Document database) – Storage and retrieval of data

2. Express (Back-end application framework) – Makes requests to Database for Reads and Writes.

3. Angular (Front-end application framework) – Handles Client and Server Requests

4. NodeJS (JavaScript runtime environment) – Accepts requests and displays results to end user

To integrate the above technologies let's build a book register.

NodeJS Setup

sudo apt update -y
curl -fsSL https://deb.nodesource.com/setup_current.x | sudo -E bash -
sudo apt update -y # might be necessary to update source index
sudo apt install -y nodejs

After installing NodeJS let's create the project directory, run npm init in it and create the folder structure:

sudo mkdir Book; cd Book; sudo npm init -y

We should now have a package.json file inside of the Book directory. At the end of the project we should have a folder structure like this:

├── Book
│   ├── apps
│   │   ├── routes.js
│   │   ├── models
│   │   │   ├── book.js
├── public
│   ├── index.js
│   ├── index.html
├── node_modules
├── server.js
├── package.json
└── package-lock.json

Express Setup

Express is a minimal and flexible Node.js web application framework that provides features for web and mobile applications. We will use Express to pass book information to and from our MongoDB database.

sudo npm i express body-parser

// ~/Book/server.js

const express = require('express');
const bodyParser = require('body-parser');
const app = express();

app.use(express.static(__dirname + '/public'));
app.use(bodyParser.json());

require('./apps/routes')(app);

app.set('port', 3300);
app.listen(app.get('port'), function() {
    console.log('Server up: http://localhost:' + app.get('port'));
});

Additionally, we need to include port 3300 in our EC2 instance's security group to be able to access the express server.

MongoDB Setup

MongoDB stores data in flexible, JSON-like documents. Fields in a database can vary from document to document and data structure can be changed over time. For our example application, we are adding book records to MongoDB that contain a book's name, ISBN, author, and number of pages.

Installation

Let's install the latest version of the MongoDB community edition. The commands below are pulled from the official docs:

wget -qO - https://www.mongodb.org/static/pgp/server-6.0.asc | sudo apt-key add -

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/6.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list

sudo apt update
sudo apt install -y mongodb-org
sudo systemctl daemon-reload
sudo systemctl start mongod

We should see the service running now:

Mongoose Setup

We will use Mongoose to establish a schema for the database to store data of our book register. Let's create the schema for our book model with fields for name, ISBN, author, and pages:

// ~/Book/apps/models/book.js

const mongoose = require('mongoose');
const dbHost = 'mongodb://localhost:27017/test';

mongoose.connect(dbHost);
mongoose.connection;

mongoose.set('debug', true);

const bookSchema = mongoose.Schema({
  name: String,
  isbn: { type: String, index: true },
  author: String,
  pages: Number
});

const Book = mongoose.model('Book', bookSchema);

module.exports = mongoose.model('Book', bookSchema);

Express Routes

We need to define routes for which our express server should handle when a request is made to it. The following lists /book route for requests for GET & POST, as well as a /book/:isbn to delete a book record when a DELETE request is provided with an isbn value for that book.

// ~/Book/apps/routes.js

const Book = require('./models/book');

module.exports = function(app) {
    // Returns all books
    app.get('/book', function(req, res) {
        Book.find({}, function(err, result) {
            if ( err ) throw err;
            res.json(result);
        });
    }); 

    app.post('/book', function(req, res) {
        // Adds book to book collection
        const { name, isbn, author, pages } = req.body;
        const book = new Book({ name, isbn, author, pages });
        book.save(function(err, result) {
            if (err) throw err;
            res.json( {
                message:"Successfully added book",
                book: result
            });
        });
    });

    app.delete("/book/:isbn", function(req, res) {
        // Find book based on isbn value and remove from collection
        Book.findOneAndRemove(req.query, function(err, result) {
            if (err) throw err;
            res.json( {
                message: "Successfully deleted the book",
                book: result
            });
        });
    });

    const path = require('path');

    app.get('*', function(req, res) {
        res.sendfile(path.join(__dirname + '/public', 'index.html'));
    });
};

AngularJS Setup

AngularJS provides a web framework for creating dynamic views in your web applications. We need an index.html to present our data and a index.js to handle user input for adding or removing books from our records.

sudo npm i angularjs

<!-- ~/Book/public/index.html -->

<!doctype html>
<html ng-app="myApp" ng-controller="myCtrl">
  <head>
    <script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.6.4/angular.min.js"></script>
    <script src="index.js"></script>
  </head>
  <body>
    <div>
      <table>
        <tr>
          <td>Name:</td>
          <td><input type="text" ng-model="Name"></td>
        </tr>
        <tr>
          <td>Isbn:</td>
          <td><input type="text" ng-model="Isbn"></td>
        </tr>
        <tr>
          <td>Author:</td>
          <td><input type="text" ng-model="Author"></td>
        </tr>
        <tr>
          <td>Pages:</td>
          <td><input type="number" ng-model="Pages"></td>
        </tr>
      </table>
      <button ng-click="add_book()">Add</button>
    </div>
    <hr />
    <div>
      <table>
        <tr>
          <th>Name</th>
          <th>Isbn</th>
          <th>Author</th>
          <th>Pages</th>

        </tr>
        <tr ng-repeat="book in books">
          <td>{{book.name}}</td>
          <td>{{book.isbn}}</td>
          <td>{{book.author}}</td>
          <td>{{book.pages}}</td>

          <td><input type="button" value="Delete" data-ng-click="del_book(book)"></td>
        </tr>
      </table>
    </div>
  </body>
</html>

Now for the interactivity connection to the UI:

// ~/Book/public/script.js

const app = angular.module('myApp', []);

app.controller('myCtrl', function($scope, $http) {
    $http({
        method: 'GET',
        url: '/book'
    }).then(function successCallback(response) {
        $scope.books = response.data;
    }, function errorCallback(response) {
        console.log('Error: ' + response);
    });

    $scope.del_book = function(book) {
        $http( {
            method: 'DELETE',
            url: '/book/:isbn',
            params: { 'isbn': book.isbn }
        }).then(function successCallback(response) {
            console.log(response);
        }, function errorCallback(response) {
            console.log('Error: ' + response);
        });
    };

    $scope.add_book = function() {
        const body = {
            name: $scope.Name,
            isbn: $scope.Isbn,
            author: $scope.Author,
            pages: $scope.Pages,
        };

        $http({
            method: 'POST',
            url: '/book',
            data: body
        }).then(function successCallback(response) {
            console.log(response);
        }, function errorCallback(response) {
            console.log('Error: ' + response);
        });
    };
});

Our app is now complete with simple CRD (Create, Read, Delete) functionality. Start the express server with node index.js in the ~/Book directory.

Opening the app on our public IP address on port 3300 (for ex: http://54.213.246.86:3300) should display the below UI:

The below image shows the insertion of a book into the collection and a query to return all books:

The LEMP stack represents one way to configure a web server and is used in many highly-scaleable applications across the web.

The setup for a LEMP vs LAMP stack is quite similar as they share the same technologies other than the web server. We will make use of ubuntu for this project as well on an ec2 instance.

STEP 1: Install Nginx

sudo apt update -y
sudo apt install -y nginx
sudo systemctl status nginx

We need to have port 80 open for our instance just like in Project 1. It's wise to have different security groups depending on what port access your technologies require. For web access, I have the default security group (SG) limited to just port 22 (for ssh access) and 80 for web server access. For projects requiring additional rules, I will clone the default SG and append the new rules.

Our server is now accessible on port 80 using our public IP Address (ex. http://35.93.24.167.

STEP 2: Setup MySQL

Setting MySQL for LEMP is identical to LAMP so I have just copied the steps from the last project into a gist and embedded it below.

STEP 3: Install PHP

Nginx requires an external program to handle PHP processing and act as a bridge between the PHP interpreter itself and the web server. We need php-fpm (“PHP fastCGI process manager”) which tells Nginx to pass PHP requests to itself for processing. Additionally, we will need php-mysql, a PHP module that allows PHP to communicate with MySQL-based databases.

sudo apt install -y php-fpm php-mysql

STEP 4: Configure Nginx to use PHP-FPM

Nginx has something similar to Apache's virtual hosts called server blocks. They both encapsulate configuration details and allow hosting multiple domains on a single server.

Let's create a domain for this project and transfer its ownership from root to the USER.

sudo mkdir /var/www/lemp
sudo chown -R $USER:$USER /var/www/lemp

Configuration

#/etc/nginx/sites-available/lemp

server {
    listen 80;
    server_name lemp www.lemp;
    root /var/www/lemp;

    index index.html index.htm index.php;

    location / {
        try_files $uri $uri/ =404;
    }

    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
        fastcgi_pass unix:/var/run/php/php8.1-fpm.sock;
     }

    location ~ /\.ht {
        deny all;
    }

}

After setting the configuration above, we need to let Nginx be aware of it, verify it has no syntax errors, unlink the default site (with the Nginx landing page pictured above), and reload the Nginx:

sudo ln -s /etc/nginx/sites-available/lemp /etc/nginx/sites-enabled/
sudo nginx -t
sudo unlink /etc/nginx/sites-enabled/default
sudo systemctl reload nginx

Our site is currently empty as it's bereft of any files - *.html or *.php , and will thus show a 403 message. Let's rectify this by adding a simple index.html file:

sudo echo 'Hello LEMP from hostname' $(curl -s http://169.254.169.254/latest/meta-data/public-hostname) 'with public IP' $(curl -s http://169.254.169.254/latest/meta-data/public-ipv4) > /var/www/lemp/index.html

Step 5: Testing PHP with Nginx

Let's verify that our nginx + php integration (using php-fpm) can process .php files.

sudo echo '<?php phpinfo();' > /var/www/lemp/info.php

We should see the following PHP info page on the http:35.93.24.167/info.php URL:

Step 6: Retrieve Data from MySQL DB with PHP

For a final integration, let's create a todolist database, configure access to it and have Nginx query our database and display it.

In the MySQL console, we'll create a database and an admin user with all privileges to the new database. This user will now be able to create a table in the database and insert data into it.

What's the use of a database without any data? Let's populate it with some sample todo list items:

Display Todo

To display our data from our todo_list we need access our test_dbdatabase, query our todo_list table, and finally create an HTML skeleton to display the data in a list:

# /var/www/lemp/todo_list.php

<?php
$user = "admin";
$password = "Super01!";
$database = "test_db";
$table = "todo_list";

try {
  $db = new PDO("mysql:host=localhost;dbname=$database", $user, $password);
  echo "<h2>TODO</h2><ol>";
  foreach($db->query("SELECT content FROM $table") as $row) {
    echo "<li>" . $row['content'] . "</li>";
  }
  echo "</ol>";
} catch (PDOException $e) {
    print "Error!: " . $e->getMessage() . "<br/>";
    die();
}

Visiting http://35.93.24.167/todo_list.php , should display our todo list:

Learning Outcomes

1. Set up Nginx Server & connect with PHP.

2. Use PHP to connect to MySQL database, query a table, and display data.