Copied from my GitHub techdiary

Below is a collection of my research and technical explorations. Hopefully you find it useful, at a minimum I did.

Design

Cloud First Applications

Data Systems

Security

ML

Advertising

Statistics

Other

Minor Tips

Projects

Finishing projects requires crossing the dip, and the best way to do that is quit, or don’t even start projects you don’t have the energy to complete. To improve my chances of success, I try to have a limited set of projects in flight, and deliberatelly mothball projects I don’t think are worth completing.

Complete

A few that made it over the finish line

In Flight

Just like kanban, you don’t want too many projects in flight, and this list reminds me to limit my open projects.

On Deck

  • Make a single video from all my ring videos
  • 3D printing

Mothballed

Several Projects that I started and deliberatly mothballed so I’d have capacity for my other projects

Markdown

  • Typora
  • Visual Studio Code
  • Macdown

VIM

C Sharp

Java

  • Lambok

Python

  • Hug: Python API generator
  • Httpie: wget replacement
  • Pendulum: The best datetime library (better then arrow).
  • PipEnv - Finally sane dependency management for python!
  • Datalore - Jetbrain’s version of Jupyter
  • Requests Http requests for humans
  • Requests-HTML Web Parsing for humans
  • Scapy Web crawling for humans.
  • Records Raw SQL for humans
  • Delegator Sub process for humans
  • Pylinkvalidator - Crawl a web site for errors (great for web testing)

Typescript

Apple

Home Automation

Text manipulation tools

  • XPath and HtmlAgilityPack
  • Regexp + VIM
  • Beautiful soup

jq

jq is the JavaScript equivalent of regexp I’m very enamored with it!

By default it colorizes:

cat x.json | jq

It has funny escaping syntax - so I’ll list out some recipes. Notice the specific of dots open brackets and quote types. They matter. Select every object in the array and string interpolates out object elements of name defaultHostName and appServicePlanId

cat x.json | jq '.[] | "\(.defaultHostName) \(.appServicePlanId) "'

Read the Algolia key from my secretBox

cat ~/gits/igor2/secretBox.json | jq '.AlgoliaKey'

List AWS lightsail properties

saws> aws lightsail get-instances  | jq '.instances[] | .name, .state.name, .publicIpAddress'
"Amazon_Linux-1GB-Oregon-1"
"running"
"54.190.183.299"

Pup - regexp For HTML

Combined with sed and xargs you can do some crazy stuff. In this below case I take a webpage that contains a single a element, find its href, replace a X Summary with Text Summary and download it. Woah!

curl $src  | pup 'a attr{href}' | sed -e "s/ \w\+ Summary/ Text Summary/"  | xargs -I {} wget '{}'

Cool shell tools

  • Mosh - A better ssh (I no longer recommend)
  • fzf - Fuzzy file finder (Try C-R and C-T)
  • Rg - RipGrep (like ag)
  • linuxbrew - Brew for Linux
  • Ngrok - Pipe ports to the internet web site (great for local host development on iOS)
  • Bat - cat but with paging/git integration.
  • Pretty ping - Graphical Ping
  • fd - Better find/recurse. (fd “regexp”)

Process Monitoring

  • glances - Prettier Top
  • iftop - Network based top based on network connection
  • nethogs - Network based top based on process
  • htop - Works everywhere
  • gotop - More graphical

Natural Language Processing

Azure One Liners

Deploy webapp via git checkin

https://docs.microsoft.com/en-us/azure/app-service/deploy-local-git

  1. Create the git credentials to deploy which gives a new git repository.
  2. Add a git remote
  3. Push to the git remote

git

npx add-gitignore - update gitignore files

Find when text is deleted:

# ignore the path if you don't know it.
git log -c -S'missingtext' /path/to/file

Colorized Diff - delta

Delta

NOTE: Before this I used Diff so fancy, but Delta looks better.

My original git tricks from 2014 :)

TUI Client - lazygit/tig

Alternate TUI to tig

The TUI clone of git gui - my day to day git interface.

Git Stats

Git stats

TUI merge - fac

TUI (merge tool)[https://github.com/mkchoi212/fac]

GitHub

  • Serve HTML files directly from GitHub: https://rawgit.com/idvorkin/linqpadsnippets/master/js/DetectBackButton.html
  • Keyboard shortcuts: https://help.github.com/articles/using-keyboard-shortcuts/
  • Grip - Local renders of GHFM https://github.com/joeyespo/grip
  • hub a better git client that understands everything comes from github

  • Search all commits by author and day

    https://github.com/search?q=user%3Aidvorkin+author-date%3A2020-06-20+author%3Aidvorkin

SSH

Exit the ssh session (very helpful when stuck in a nested tmux session.

<enter> ~ .

Get help on the in built console

<enter>  ~ ?

List forwarded ports

<enter>  ~ #

Connect on 8889 will get redirected to 8888 on the remote_host.

# Connections on 4444 will get redirected to 8888 on the remote_host.
ssh -N -L localhost:8888:localhost:4444 remote_user@remote_host

Use a tool to auto re-connect (MOSH should do this, but it’s been flaky for me of late)

autossh -M 0 server

For my memory here are the many ports I forward, which you can also configure in your ssh config

# jupyter
ssh -N -L localhost:8888:localhost:8888 lightsail

# jekyll
ssh -N -L localhost:4000:localhost:4000 lightsail

# jekyll live reload
ssh -N -L localhost:35729:localhost:35729 lightsail

# grip
ssh -N -L localhost:6419:localhost:6419 lightsail

TMUX

This is so hard for me to remember:

Install tmuxp to re-create sessions. Look at my config to see how to setup vi mode Look at tmux plugin manager

https://gist.github.com/MohamedAlaa/2961058

  • C-A w - See all windows and sessions.
  • C-A $ - rename session
  • C-A , - rename windows
  • C-A % - split left right
  • C-A “ - split top bottom
  • C-A q - show pane #
  • C-A navigate panes
  • C-A ? - list shortcuts (then search using / )
  • C-A swap-window -t -1 – Move window to the left
  • C-A swap-window -t 0 – Move window to the front
  • C-A x - kill-pane

Tmux command line

Re-attach to current session

tmux attach

Force resize to current terminal size while reattaching

tmux attach -d

Random 1-liners

Clean up dead MOSH instances

kill $(ps --no-headers --sort=start_time -C mosh-server -o pid | head -n -1)

zsh path append

path+="/my/new/path"

App Launchers

  • Alfred on OSX
  • WoX on Windows + (switcheroo to switch windows) [win - app name]

Chrome extensions

  • Hibernate unused tabs The great suspender
  • Close unused tabs OneTab
  • Vim based web Vimium
  • Vim in any text box Wasavi
  • Speed up YouTube playback via keyboard shortcut YouTube playback speed control
  • Fail to load dinosaur - Not an extension but you can play a little game with the dinosaur by pressing on the chrome dinosaur

  • Custom searchh engines are cool. You can use them for search sites you frequently use. For example my github repo and my blog:

    ghtd: https://github.com/search?l=&q=%s+user%3Aidvorkin&type=Code&s=indexed ig: https://www.idvork.in/index.html?q=%s

Web tools (http)

wuzz

TUI http debugger

httplab

A TUI http server - The oppositve of wuzz.

httpie

httpie is like wget and curl, but easier to use.

Here’s an example of sending content to a web hook which would send a message to chime. Notice -pHBhb prints request response, and Content is a JSON field.

http -pHBhb POST https://hooks.chime.aws/incomingwebhooks/botaddress -- Content=We love PMs

httpprompt

Graphical httpie with tab completion. It’s actually built on httpie

You’d think there would be a slew of these, but all the ones I tried weren’t great (can’t limit recurse depth, don’t group errors, etc). I’m using bcl right now, but would love a good alternative.

Using BLC I crawl my website using (-r recurse ; -o group output; -e skip external):

    blc https://idvork.in/d -roe

brow.sh - Text based web browser

This looks pretty cool. I had to run it in a container:

docker run --rm -it browsh/browsh

The keybindings are desktop-esk, which is too bad as they conflict with my terminal bindings. If you try it C-Q gets you out of it, can also run extensions, so I bet you could run vimium in there. Although, your’e going a little crazy at this point

w3m - Text based web browser

A text based browser better then lynx. But,

Video Manipulation Tools

Decent cross plat video editors.

ffmpeg recipes:

Convert video formats

ffmpeg -i in.mov -qscale 0 -out.mp4

Trimming

https://superuser.com/questions/138331/using-ffmpeg-to-cut-up-video As other people mentioned, putting -ss before (much faster) or after (more accurate) the -i makes a big difference. The section “Fast And Accurate Seeking” on the ffmpeg seek page tells you how to get both, and I have used it, and it makes a big difference. Basically you put -ss before AND after the -i, just make sure to leave enough time before where you want to start cutting to have another key frame. Example: If you want to make a 1-minute clip, from 9min0sec to 10min 0sec in Video.mp4, you could do it both quickly and accurately using:

ffmpeg -ss 00:08:00 -i Video.mp4 -ss 00:01:00 -t 00:01:00 -c copy VideoClip.mp4

The first -ss seeks fast to (approximately) 8min0sec, and then the second -ss seeks accurately to 9min0sec, and the -t 00:01:00 takes out a 1min0sec clip. -c copy doesn’t re-transcode so faster, but has problems since needs key frames.

ffmpeg -ss 00:08:00 -i $infile -ss 00:01:00 -t 00:01:00 -c copy $outfile

Video To Gif on iOS

On github, it’s common to share an inline view of a repro. A great way to do that is to make a gif of the issue. To do that on iOS you need to do do a screen recording, and then save that to a gif. I used GifMaker, seems good engouh.

Linters and formatters

You know what sucks less then coding standards? Arguing about coding standards. Nothing pains me more then arguing about formatting (and other minutia). As such I love having lint failures being build breaks and including formatting on save. Even better when you have opinionated formatters that don’t have options. Sure standards suck, but they suck much less then arguing.

Switching between Unix and DOS file ending

(https://stackoverflow.com/questions/2466959/git-removing-carriage-returns-from-source-controlled-files) On windows, when you switch between windows and WSL, you can get your line endings messed up in git.

To have git honor settings in the existing files set:

In: ~.gitconfig set:

core.autocrlf=input
core.safecrlf=false

Sometimes your repository gets screwed up and needs to be fixed. In that case, erase all files, and re checkout.

#!/bin/sh
# remove local tree
git ls-files -z | xargs -0 rm
# checkout with proper crlf
git checkout .

Also, can set a single file in vim by settings set ff=unix

Spelling

Command line spelling. Ispell has nicer word highlighting, but aspell is supposed to have better spelling correction.

aspell ispell

(Bragging - I fixed aspell highlighting in WSL) https://github.com/GNUAspell/aspell/issues/590 https://github.com/Homebrew/homebrew-core/pull/48163

Web scripting

I like to watch YouTube (engineering documentaries) before bed, and need a sleep timer. JS will let you close a windows you opened, so this script opens a new window to watch YouTube in for an hour.

// Open in another window so I have perms to close it
var customWindow = window.open('https://youtube.com', '_blank', '');
// Close tab in 2 hours
setTimeout(()=>customWindow.close(),1000*60*120)

Programming Helpers

howdoi

A quick tool to show you how to ‘spell’ your task in language of choice.

> howdoi do time to minutes in python

import datetime as DT
t1 = DT.datetime.strptime('00:05:36.0100000', '%H:%M:%S.%f0')
t2 = DT.datetime(1900,1,1)

print((t1-t2).total_seconds() / 60.0)

PlantUML alternatives

Huh, turns out there are lots (for UML and graphicing, etc). Read the list here, I haven’t played with them yet:

https://hackmd.io/features#UML-Diagrams

PlantUML

PlantUML Tools

  • PlantText Interactive editor with vi-keybindings, but needs a manual refresh
  • LiveUML - Interactive editor with auto-refresh
  • VS Code has a great plugin for PlantUML
  • Vim has a PlantUML preview in ASCII

PlantUML in Markdown

You can’t directly render PlantUML in Markdown, but you can render images, and get PlantUML to load content from file. E.g:

https://www.plantuml.com/plantuml/proxy?idx=0&format=svg&src=**URL_with_uml_file**&cache_buster=**increment_to_avoid_caching**

Store your PlantUML file somewhere - e.g.

[https://raw.github.com/idvorkin/techdiary/master/sample_diagrams.puml](https://raw.githubusercontent.com/idvorkin/techdiary/master/sample_diagrams.puml)

Then render it like an image e.g.

![UML rendered](https://www.plantuml.com/plantuml/proxy?idx=0&format=svg&src=https://raw.githubusercontent.com/idvorkin/techdiary/master/sample_diagrams.puml&c=1)

UML rendered

Quirks

  • GitHub caches, so increment the c parameter

Windows Managers

Tools that let you re-tile

ClI Fun

Here’s some silly, but fun stuff:

  • asciiquarium - fishbowl in the terminal
  • cmatrix - the matrix in the terminal

CLI Screen Recoding

OK, this is a 2 putter: First run asciinema, then svgterm

asciinema rec foo.cast
# records till you hit C-D  (curious how that works)
cat foo.cast | svg-term --out foo.svg
# Copy out your svg file

If on iOS then you need to click to open the image in a new tag

inamged image

(Note, to include an svg from raw.githubusercontent.com/blah.svg you’ll need to add blah.svg?sanatize=true)

Show pressed keys

Carnac

OSX Misc

Other Resources

List of tools and programming tips List of List of Tools More lists