Igor's Tech, Tricks, and Tools

≡

A collection of tips, tricks and pointers of the various tools i use

Major Research and Project List

My various projects and research

Modern Unix

Over time, I collected many of the tools below, but now this github project collects them all:

Modern Unix

Build your own X

What I can create, I can not understand

A very cool index of tutorial to build your own everything

Markdown

  • Typora
  • Visual Studio Code
  • Macdown

VIM

C Sharp

Java

  • Lambok

Python

Web:

  • Httpie: wget replacement
  • Beautiful Soup - JQuery like parsing
  • Pylinkvalidator - Crawl a web site for errors (great for web testing)
  • Hug: Python API generator
  • Pendulum: The best datetime library (better then arrow).
  • Requests Http requests for humans
  • Requests-HTML Web Parsing for humans
  • Scapy Web crawling for humans.

Language and Syntax:

  • attribs - Clean classes with less boiler plate

Tools:

  • PipEnv - Finally sane dependency management for python!
  • Datalore - Jetbrain’s version of Jupyter

CLI:

  • typer - Very nice command line too based on type arguments
  • rich - Nice TUI output

Debugging:

  • Ice Cream Simpler printf debugging.
  • pudb - Python Debugger
  • bpython - Python Interpretter (nice GUI, but no VI)
  • loguru - Output pretty exception meassages with a single type annotation

Examples:

  • Parsing a web page, with nice debugging and cli tooling (Eulogy Parser)

Profiling:

  • Tuna - Profile viewer https://github.com/nschloe/tuna
  • importime python -X importtime yourfile.py 2> import.log
  • Profiling code python -mcProfile -o program.prof yourfile.py

Typescript

Apple

Home Automation

Text manipulation tools

  • XPath and HtmlAgilityPack
  • Regexp + VIM
  • Beautiful soup

jq

jq is the JavaScript equivalent of regexp I’m very enamored with it!

By default it colorizes:

cat x.json | jq

It has funny escaping syntax - so I’ll list out some recipes. Notice the specific of dots open brackets and quote types. They matter. Select every object in the array and string interpolates out object elements of name defaultHostName and appServicePlanId

cat x.json | jq '.[] | "\(.defaultHostName) \(.appServicePlanId) "'

Read the Algolia key from my secretBox

cat ~/gits/igor2/secretBox.json | jq '.AlgoliaKey'

List AWS lightsail properties

saws> aws lightsail get-instances  | jq '.instances[] | .name, .state.name, .publicIpAddress'
"Amazon_Linux-1GB-Oregon-1"
"running"
"54.190.183.299"

Pup - regexp For HTML

Combined with sed and xargs you can do some crazy stuff. In this below case I take a webpage that contains a single a element, find its href, replace a X Summary with Text Summary and download it. Woah!

curl $src  | pup 'a attr{href}' | sed -e "s/ \w\+ Summary/ Text Summary/"  | xargs -I {} wget '{}'

Cool shell tools

  • Mosh - A better ssh (I no longer recommend)
  • fzf - Fuzzy file finder (Try C-R and C-T)
  • Rg - RipGrep (like ag)
  • linuxbrew - Brew for Linux
  • Ngrok - Pipe ports to the internet web site (great for local host development on iOS)
  • Bat - cat but with paging/git integration.
  • Pretty ping - Graphical Ping
  • fd - Better find/recurse. (fd “regexp”)
  • jless A less for json, vim like keybindgs
  • csview A less for csv

Process Monitoring

  • glances - Prettier Top
  • iftop - Network based top based on network connection
  • nethogs - Network based top based on process
  • htop - Works everywhere
  • gotop - More graphical

Natural Language Processing

Azure One Liners

Deploy webapp via git checkin

https://docs.microsoft.com/en-us/azure/app-service/deploy-local-git

  1. Create the git credentials to deploy which gives a new git repository.
  2. Add a git remote
  3. Push to the git remote

git

npx add-gitignore - update gitignore files

Find when text is deleted:

# ignore the path if you don't know it.
git log -c -S'missingtext' /path/to/file

Colorized Diff - delta

Delta

NOTE: Before this I used Diff so fancy, but Delta looks better.

My original git tricks from 2014 :)

TUI Client - lazygit/tig

Alternate TUI to tig

The TUI clone of git gui - my day to day git interface.

Git Stats

Git stats

Or my own favorite, file stat changes in a date range, which I coded up in zsh:

function gstatdaterange() {
    # $1 - start
    # $2 - end
    # can be days ago
    # glogdate '30 days ago' '1 day ago'
    # or absolute dates
    # glogdate '12/01/2020'

    # output all git commits since until, pretty print to just have the commit
    git_output=`git log --since "$1" --until "$2" --pretty="%H"`

    # diff between first commit to last commit, and sort the output by size
    #sort params -k=second column; -t=with delimter as |; -n=sort as numeric -r sort as reversed
    git diff --stat `echo $git_output | tail -n 1` `echo $git_output | head -n 1` |  sort -k2 -t'|' -n -r
}

TUI merge - fac

TUI merge tool

Re-write history

Go back a few commits, and then make the new commits 1 by 1. Quite simple, just be careful about re-writing history someone else has also edited

git reset --soft HEAD~[n]
git commit

GitHub

SSH

Exit the ssh session (very helpful when stuck in a nested tmux session.

<enter> ~ .

Get help on the in built console

<enter>  ~ ?

List forwarded ports

<enter>  ~ #

Connect on 8889 will get redirected to 8888 on the remote_host.

# Connections on 4444 will get redirected to 8888 on the remote_host.
ssh -N -L localhost:8888:localhost:4444 remote_user@remote_host

Use a tool to auto re-connect (MOSH should do this, but it’s been flaky for me of late)

autossh -M 0 server

For my memory here are the many ports I forward, which you can also configure in your ssh config

# jupyter
ssh -N -L localhost:8888:localhost:8888 lightsail

# jekyll
ssh -N -L localhost:4000:localhost:4000 lightsail

# jekyll live reload
ssh -N -L localhost:35729:localhost:35729 lightsail

# grip
ssh -N -L localhost:6419:localhost:6419 lightsail

TMUX

This is so hard for me to remember:

Install tmuxp to re-create sessions. Look at my config to see how to setup vi mode Look at tmux plugin manager

https://gist.github.com/MohamedAlaa/2961058

  • C-A w - See all windows and sessions.
  • C-A $ - rename session
  • C-A , - rename windows
  • C-A % - split left right
  • C-A “ - split top bottom
  • C-A q - show pane #
  • C-A navigate panes
  • C-A ? - list shortcuts (then search using / )
  • C-A swap-window -t -1 – Move window to the left
  • C-A swap-window -t 0 – Move window to the front
  • C-A x - kill-pane

Tmux command line

Re-attach to current session

tmux attach

Force resize to current terminal size while reattaching

tmux attach -d

Random 1-liners

Clean up dead MOSH instances

kill $(ps --no-headers --sort=start_time -C mosh-server -o pid | head -n -1)

zsh path append

path+="/my/new/path"

App Launchers

  • Alfred on OSX
  • WoX on Windows + (switcheroo to switch windows) [win - app name]

Chrome extensions

Web tools (http)

wuzz

TUI http debugger

httplab

A TUI http server - The oppositve of wuzz.

httpie

httpie is like wget and curl, but easier to use.

Here’s an example of sending content to a web hook which would send a message to chime. Notice -pHBhb prints request response, and Content is a JSON field.

http -pHBhb POST https://hooks.chime.aws/incomingwebhooks/botaddress -- Content=We love PMs

httpprompt

Graphical httpie with tab completion. It’s actually built on httpie

You’d think there would be a slew of these, but all the ones I tried weren’t great (can’t limit recurse depth, don’t group errors, etc). I’m using bcl right now, but would love a good alternative.

Using BLC I crawl my website using (-r recurse ; -o group output; -e skip external):

    blc https://idvork.in/d -roe

brow.sh - Text based web browser

This looks pretty cool. I had to run it in a container:

docker run --rm -it browsh/browsh

The keybindings are desktop-esk, which is too bad as they conflict with my terminal bindings. If you try it C-Q gets you out of it, can also run extensions, so I bet you could run vimium in there. Although, your’e going a little crazy at this point

w3m - Text based web browser

A text based browser better then lynx. But,

Video Manipulation Tools

Decent cross plat video editors.

ffmpeg recipes:

Convert video formats

ffmpeg -i in.mov -qscale 0 -out.mp4

Trimming

https://superuser.com/questions/138331/using-ffmpeg-to-cut-up-video As other people mentioned, putting -ss before (much faster) or after (more accurate) the -i makes a big difference. The section “Fast And Accurate Seeking” on the ffmpeg seek page tells you how to get both, and I have used it, and it makes a big difference. Basically you put -ss before AND after the -i, just make sure to leave enough time before where you want to start cutting to have another key frame. Example: If you want to make a 1-minute clip, from 9min0sec to 10min 0sec in Video.mp4, you could do it both quickly and accurately using:

ffmpeg -ss 00:08:00 -i Video.mp4 -ss 00:01:00 -t 00:01:00 -c copy VideoClip.mp4

The first -ss seeks fast to (approximately) 8min0sec, and then the second -ss seeks accurately to 9min0sec, and the -t 00:01:00 takes out a 1min0sec clip. -c copy doesn’t re-transcode so faster, but has problems since needs key frames.

ffmpeg -ss 00:08:00 -i $infile -ss 00:01:00 -t 00:01:00 -c copy $outfile

Video To Gif on iOS

On github, it’s common to share an inline view of a repro. A great way to do that is to make a gif of the issue. To do that on iOS you need to do do a screen recording, and then save that to a gif. I used GifMaker, seems good engouh.

Linters and formatters

You know what sucks more then coding standards? Arguing about coding standards. Nothing pains me like arguing about formatting (and other minutia).

As such I love having lint failures part of the build process, being auto-fixed, and failing checkin when ignored. Opinionated formatters are even better as you don’t need to spend time arguing about which standard to enforce. And yes, all standards suck, but arguing about standards sucks more.

Formatters I like:

And to ensure they are integrated into your work flow, enforce them with

lint-staged and husky. My using it is here

Switching between Unix and DOS file ending

(https://stackoverflow.com/questions/2466959/git-removing-carriage-returns-from-source-controlled-files) On windows, when you switch between windows and WSL, you can get your line endings messed up in git.

To have git honor settings in the existing files set:

In: ~.gitconfig set:

core.autocrlf=input
core.safecrlf=false

Sometimes your repository gets screwed up and needs to be fixed. In that case, erase all files, and re checkout.

#!/bin/sh
# remove local tree
git ls-files -z | xargs -0 rm
# checkout with proper crlf
git checkout .

Also, can set a single file in vim by settings set ff=unix

Spelling

Command line spelling. Ispell has nicer word highlighting, but aspell is supposed to have better spelling correction.

aspell ispell

(Bragging - I fixed aspell highlighting in WSL) https://github.com/GNUAspell/aspell/issues/590 https://github.com/Homebrew/homebrew-core/pull/48163

Web scripting

I like to watch YouTube (engineering documentaries) before bed, and need a sleep timer. JS will let you close a windows you opened, so this script opens a new window to watch YouTube in for an hour.

// Open in another window so I have perms to close it
var customWindow = window.open("https://youtube.com", "_blank", "");
// Close tab in 2 hours
setTimeout(() => customWindow.close(), 1000 * 60 * 120);

Document conversion - Pandoc

HMTL to markdown pandoc x.html -t gfm-raw_html -f html > lawsofpower.md

Programming Helpers

howdoi

A quick tool to show you how to ‘spell’ your task in language of choice.

> howdoi do time to minutes in python

import datetime as DT
t1 = DT.datetime.strptime('00:05:36.0100000', '%H:%M:%S.%f0')
t2 = DT.datetime(1900,1,1)

print((t1-t2).total_seconds() / 60.0)

Excalidraw

A great way to draw in a collaborative form see file at (/images/build-workflow.excildraw)

Build workflow

PlantUML alternatives

Huh, turns out there are lots (for UML and graphicing, etc). Read the list here, I haven’t played with them yet:

https://hackmd.io/features#UML-Diagrams

D2

Look like a fun alternative language: https://github.com/terrastruct/d2#install

PlantUML

PlantUML Tools

  • PlantText Interactive editor with vi-keybindings, but needs a manual refresh
  • LiveUML - Interactive editor with auto-refresh
  • VS Code has a great plugin for PlantUML
  • Vim has a PlantUML preview in ASCII

PlantUML in Markdown

You can’t directly render PlantUML in Markdown, but you can render images, and get PlantUML to load content from file. E.g:

https://www.plantuml.com/plantuml/proxy?idx=0&format=svg&src=**URL_with_uml_file**&cache_buster=**increment_to_avoid_caching**

Store your PlantUML file somewhere - e.g.

[https://raw.github.com/idvorkin/techdiary/master/sample_diagrams.puml](https://raw.githubusercontent.com/idvorkin/techdiary/master/sample_diagrams.puml)

Then render it like an image e.g.

![UML rendered](https://www.plantuml.com/plantuml/proxy?idx=0&format=svg&src=https://raw.githubusercontent.com/idvorkin/techdiary/master/sample_diagrams.puml&c=1)

UML rendered

Inline plantuml via Gravizo

This seems the best approach, using gravizo with inline HTML like this:

<p><img src='https://g.gravizo.com/svg?
@startuml;

actor User;
participant "First Class" as A;
participant "Second Class" as B;
participant "Last Class" as C;

User -> A: DoWork;
activate A;

A -> B: Create Request;
activate B;

B -> C: DoWork;
activate C;

C --> B: WorkDone;
destroy C;

B --> A: Request Created;
deactivate B;

A --> User: Done;
deactivate A;

@enduml
'></p>

Quirks

  • GitHub caches, so increment the c parameter

Windows Managers

Tools that let you re-tile

  • OSX - Rectangle
  • Windows - Windows Power Toys - FancyZones

ClI Fun

Here’s some silly, but fun stuff:

  • asciiquarium - fishbowl in the terminal
  • cmatrix - the matrix in the terminal
  • phraze - Make cartoon characters say stuff (npx run phraze)

CLI Screen Recoding

OK, this is a 2 putter: First run asciinema, then svgterm

asciinema rec foo.cast
# records till you hit C-D  (curious how that works)
cat foo.cast | svg-term --out foo.svg
# Copy out your svg file

If on iOS then you need to click to open the image in a new tag

inamged image

(Note, to include an svg from raw.githubusercontent.com/blah.svg you’ll need to add blah.svg?sanatize=true)

Show pressed keys

Carnac

Docker

Install on Amazon Linux 2

sudo amazon-linux-extras install docker

Manage via LazyDocker

Terminal Info

Terminfo, ncurses, OSX, what a mess. You need to add your own terminfos see:

You also want to install the other terminals like xterm256 and tmux

OBS

Great tool for playing with videos, useful plugins

  • iPhone Video Recorder (use a USB cable)
  • Stream desk (makes scene control way easier)
  • Source Record Plugin (though lots of complaints that it works poorly)

OSX

Loading (/osx)

Obsidian

Other Resources

List of tools and programming tips List of List of Tools More lists