BlogCollectionsPhotographsSnippetsMetricsAboutChat
Categories:
Snippets
Notes and bookmarks to content from around the web.
DALL-E 3 Prompt Engineering

Another excellent and interesting blog post from Simon Willison about OpenAI's DALL-E 3 text to image AI.

  • generating consistent characters
  • modifying images
  • getting the seed for an image
  • details about the prompt that chatGPT uses when sending the users prompt to the DALL-E model.
Sparks of General Intelligence

A paper describing what the unrestricted GPT-4 can do.

Come back to this

Can do:

  • theory of mind
  • use tools and external APIs
  • abstract and creative tasks like mathlete or fermi questions

Can't do:

  • Discontinuous tasks - tasks that require knowing the end of the answer when the answer begins.
High levels of bitcoin dormancy
Investors are keeping their coins still, with the amount of Bitcoin not budging nearing a new all-time high.

A Decrypt article reporting on research by Glassnode.

  1. Glassnode added in the report that there is “a turning of the cycle and a marked shift in investor behavior patterns.”
  2. such activity has been previously observed in prior bear markets, “potentially signaling a perception that the market is oversold”
  3. Dormant coins actually become “increasingly unlikely to be spent” after a 155-day holding period
Elitism as the mid-career growth engine

A blog post by Stephen Bailey articulating ideas I've thought about but never seen anyone else consider;

  1. An aggressive1 attitude can drive progress - it's one of the mental tools in the toolbox.
  2. Software engineering is a craft - I am a craftsman. Not an artist, and not a technician.

The active and violent spirit of the postdoc is a critical difference in the ways that early- and mid-career professionals grow in their craft.

It takes animus to push conversations out from the boring straits of explanation and pedantry and into the frothy, conflicting waves of exploration and world expansion. It takes a bit of pirate swagger. It takes a separation of the world into mateys and landlubbers.

After the beginner’s scaffolding of skills, what matters is the mechanism for deciding what’s next. That’s where opinion and aversion — not reason! — comes in.

Footnotes

  1. And therefore opinionated ↩

Tmux tutorials

A handy site with lots of tmux info, including how to open a pop-up window to fzf another session.

Staff Engineer

A site about career paths after reaching "Senior Software Engineer". Apparently you have two main branches, either engineering management, or staff engineer.

The site references several good books about the engineering management career path. I should come back to this.

archive

Adding JSON output to a CLI app

An article suggesting best practices for adding JSON output to CLI apps, and mentioning why JSON output is a good idea.

  • Do Make a Schema
  • Do Flatten the Structure
  • Do Output JSON Lines for Streaming Output
  • Do Use Predictable Key Names
  • Do Pretty Print with Two Spaces or Don’t Format at All
  • Don’t Use Special Characters in Key Names
  • Don’t Allow Duplicate Keys
  • Don’t Use Very Large Numbers
nvim.CoC

I've been trying to replace ALE with CoC. It didn't work. Maybe I have unusually high fixing expectations, but ALE is more capable and easier to setup. So I'll be using CoC and ALE together, which feels messy.

Formatting

  1. pip install pep8 autopep8
  2. <leader>fs

  • <leader>a - a list of all the wrong things

  • vif - select inside function

  • vam - select method

  • <leader>ac - see a box with options to fix imports or ignore file

Find

fd is a replacement for find with more intuitive defaults.

  • Case-insensitive.
  • Ignores hidden directories and files.
  • Ignores patterns in .gitignore.

Commands

  • fd PATTERN
  • not find -iname '*PATTERN*'
  • fd --help

repo

Tmux sessions

  • Sort sessions alphabetically <prefix> s then O

  • List of sessions: <prefix> s

  • Last session: <prefix> Z

  • Rename session: <prefix> $

  • Rename window: <prefix> ,

  • Previous session: <prefix> (

For last session I used the following key bind:

bind Z switch-client -l
The 3-2-1 backup rule
  • 3 copies, 2 types of media, 1 copy off-site
  • Have 1 primary backup and 2 duplicates, in additional to the original data.
  • NAS, USB drive, Cloud storage, Server, Glacier, etc

For example, your laptop could be backed up to a NAS, a server, and glacier storage. Or 2 NAS and 1 archival cloud storage.

Proxy Servers

Reverse proxy server

  • A reverse proxy is a server.
  • It sits in front of other servers and forwards client requests to them.
  • Is a middleman or broker on behalf of origin servers.

Usually used to:

  • Increase security (keeping the IP addresses of origin servers private makes DDoS attacks much harder).
  • Optimize performance (caching).
  • Increase reliability (load balancing).

The client thinks that the reverse proxy is the end-point server.

Normal proxies

  • Also called a web or forwarding (normal) proxy.
  • Sits between client machines and hosts.
  • Is a middleman on behalf of the clients.

The server thinks that the client is the proxy.

Source: cloudflare

Unison
  • If you can ssh into the server then Unison should work.
  • Install the same version on the client and on the server.

To sync the contents of dir b into dir a:

unison -batch -color true <dir-a> ssh://<user>@a<host>/<dir-b>

Setup

  1. Download the binary file with wget
  2. chmod +x &lt;file&gt;
  3. copy the executable somewhere on your path like /usr/local/bin
  4. unison -doc tutorial > remote method > remote shell method
  5. use a filewatcher to sync on change, or use cron to sync every n minutes

You can also use unison to sync files on the same computer.

References

gist

Background

Recently I needed to work on a codebase that was on a remote machine whilst still using my local workflow and developer tools.

Vim's built in scp:// functionality isn't versatile enough - i wanted to use fzf.vim to search for text across the code base. This led me to rsync, sshfs, and eventually unison.

alpine.js

An introductory article with examples from Smashing Magazine that introduces and explains what alpine.js is and how it is different to jQuery and vue.js.

Zsh commands: exec vs source

source and exec are both built-ins.

source evaluates or runs the content of a file. For example, if you source ~/.zshrc you apply the content of the file to the currently running Zsh process. You can source any valid Zsh code.

exec replaces the current shell process with another process. Your terminal (tty) session is running a shell. Replace it with another shell without launching another tty. It could be the same shell with different settings or flags.

The tty (teletypewriter) command prints the name of the terminal you're using.

Chezmoi commands

Chezmoi is a dotfiles manager.

  • chezmoi diff - see the difference between the current state of your local dotfiles compared against the checked-in versions.
  • cm aa - to update chezmoi with all your local changes.
alias -g aa="status | cut -c 4- | xargs -I % -p sh -c 'chezmoi add ~/%'"
zsh suspend or push function

In zsh you can suspend a command whilst typing it.

This clears the current line, lets you run another command, and then when it finishes the original unfinished text is put back.

This is useful if whilst typing a command you realise you need to run another command first, like:

  • ls to check the name for an argument.
  • creating a location before moving something into that location.
  • checking man or tldr for a command flag.

Zsh has built-in functions push-line and push-input.

Do-nothing scripts

Interesting article about writing down a list of manual steps for tasks that can't be automated. A do-nothing script can take some of the toil out of a manual slog.

Even though the script doesn't do anything other than display text on a screen (the instructions to complete the task) it's still helpful, because:

  • All the steps and information required are collected together.
  • It lowers the activation energy required to do something fiddly.
  • Each step is part of a function, and the function can be updated with automation code if automation becomes possible.
  • It's more difficult to lose your place in the process, or get the order of steps wrong.

It doesn't reduce manual effort. It does reduce toil.

Lightning network point-of-sale

A project to demonstrate an offline, fast, point-of-sale device to allow merchants and customers to make fast cheap transactions on the lightning network.

Also lnbits

Make files

This article is a fairly practical introduction to the GNU make tool.

It's not the article I was looking for though, so I might swap it out if I find a better one.

Google Pub/Sub

Google Pub/Sub has client libraries in all the usual languages. You can also construct the API calls yourself. This is a link to the API documentation.

If I were to use a JavaScript beacon to push a message to a topic, my web analytics engine (a collection of cloud functions) could subscribe to the topic.

This would have the advantage that if the site were flooded with traffic and the maximum number of function instances wasn't enough to handle all the events, then none of the page views (or other events) would be lost because Pub/Sub guarantees delivery and would keep trying to deliver the message for up to 7 days.

Using the beacon to trigger the cloud functions directly wouldn't work at scale, because once the maximum number of instances are being triggered as frequently as the function takes to run, the endpoint would become unresponsive. There is no caching layer.

However, the site isn't being flooded with traffic, and I have better things to do than fix stuff that isn't broken.

Handwritten letters at scale

A twitter thread showing how a business started creating handwritten notes to customers using a plotter, and how they scaled up and further automated the process.

It would be cool to create something like this for western Europe. Handwrytten.com seems to be the main commercial implementation of this idea in NA. Competitors.

I still think it would be fun to make a simple photo printing service. Share a photo in a WhatsApp message and its printed and delivered.

The dangers of social media

A ledger (what's a ledger? It's a webpage) from the "Center For Humane Technology" showing some of the dangers of social media.

The "Do unto others" section is outstanding. It describes the views of some tech company leaders, including how they limit usage of devices and social media for themselves and their kids.

Cloud Functions Minimum Instances

Google Cloud Platform have introduced a "minimum instances" features in the Cloud Functions product.

If you have a function using 128MB of memory (the smallest option), it will cost a minimum €0.061 per day (€1.86 per month) to run 1 minimum instance.

nmap

sudo nmap -sP 192.168.1.1/24 | grep "Nmap"

Thanks to Jeff Geerling (again) I found this nifty command to see what devices are connected on a local network.

Found in this article about setting up a pi-hole.

Miller CLI

Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, tabular JSON and positionally-indexed.

Try this if you're operating on JSON of CSV data, and you want to use column names instead of column numbers.

Conway's Law

Organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations - Melvin Conway

In reverse:

If the architecture of the system and the architecture of the organization are at odds, the architecture of the organization wins - Ruth Malan

Pen-testing web apps

A blog post about how someone compromised a group of web apps.

It lists a series of technologies and techniques that the author uses as they progress their investigation.

These would make a useful list of things to know in order to build safe web-apps and not repeat the mistakes of the unfortunate target company.

Heuristics for effective software development

"Without psychological safety, respect, and trust, none of the following is possible"

"The best ways to work are collaborative. Negotiation is not collaboration. Isolated individuals making heroic efforts are never as effective as collaborative groups. We get the best results when customers, business people, and developers literally work together."

Blog post

Start with Finance to transform IT

Zwischenzugs blog post arguing that to achieve a significant change in an organisation you need need to:

  1. Get funding.
  2. Persuade the finance department to give you money.
  3. Understand what they value.
  4. Understand their cash flows.
  5. Understand how and why customers or clients part with their money.
  6. Understand business constraints (legal, reg, operating).

The five whys approach to problem solving

"Consider a deeper structural cause of cultural problems in change management: how money flows through the organisation."

"If you want to transform IT in an enterprise, start with finance. If you can crack that, you've a chance to succeed with sec and controls functions. If you don’t know why it’s important to start with finance, you’ll definitely fail "

Upgrading Cryptographic Libraries

Blog post about how to make it easier to upgrade a cryptographic or hashing library.

Django encodes passwords for database storage like this:

<algorithm>$<iterations>$<salt>$<hash>

Interestingly, Giovanni Collazo emphasises that we should design systems for change, which initially seems pretty close to contradicting YAGNI, but the answer lies in the context.

Apple's iCloud+ "VPN"

article

"An Apple onion router. The routing uses two hops; Apple provides the first, and independent third parties (not yet specified) provide the second."

"In one move, Apple has taken onion routing from a specialized tool for hackers to something that will be in daily use."

Vim Sneak
  • Invoked with s followed by 2 chars.
  • S goes backwards.
  • F, f, T, and t are enabled to work across lines.
  • Jump back with <C-O>
  • ; or , to go to next/previous highlighted text.
  • 5sxy searches for the next instance of xy within 5 lines.
  • 3dzqt delete up to the third instance of qt.

repo

Git LFS

Key commands

  • git lfs install (also uninstall)
  • git lfs track "**/*.mp4"
  • git lfs ls-files
  • git lfs status

track just updates the .gitattributes file.

Commit the .gitattributes file with the tracking configuration before committing the large files.

status or ls-files should show the large files in question before you push the commit that starts tracking the large files.

Your .bashrc doesn't have to be a mess

Blog post demonstrating how to split a .bashrc file into "submodules" and keep it maintainable.

BASHRC_D=~/.config/bashrc.d
[[ -r ${BASHRC_D}/bootstrap ]] && . ${BASHRC_D}/bootstrap

for file in ${BASHRC_D}/*.sh; do
  [[ -r $file ]] && . $file
done
unset file
Performance Optimizations for the shell prompt

Something should happen within 100ms of the users input in order to maintain a feeling of responsiveness.

If something happens within 50ms of the trigger event, it will feel almost instant.

Also, checkout hyperfine for performance benchmarking.

Lots of useful tips in the original blog post.

Vim spellcheck commands
  • [s or ]s → go to next/previous bad word
  • z= → list of suggestions
  • zg → add word to good word list
  • zug → remove word from good word list
  • zw → add word to bad word list
  • zuw → remove word from bad word list
Grep only inside particular files
grep -inr --include package.json \
    'shortcut": {' . -A 3
  • It's the --include flag that does the important part.
  • -i → case insensitive
  • -n → print line number
  • -r → recursive from starting page
  • . → start in current directory
  • -A 3 → print the 3 lines below the found line
View a List of Keyboard Mappings in Vim

:map → show a list of the current keyboard mappings for normal, visual, select and operator pending modes.

:map! → show a list of the current keyboard mappings for insert and command-line mode

Top put all the mappings into a convenient text file:

:redir! > vim_maps.txt
:map
:map!
:redir END

source another source

Forward Email

I know about ImprovMX, which used to be great because you could do a lot for free, but now you only get 1 domain for free.

ForwardEmail are 3 times cheaper than ImprovMX, and I have 2 domains forwarding email. It's not particularly private, but I can send and recieve from a domain, for free.

Globbing

  • ???? → 4 chars

  • * → any number of chars

  • [:upper:] ⇔ [A-Z] same for [:lower:] and [:digit:]

  • [:alpha:] ⇔ [a-zA-Z]

  • [:alnum:] ⇔ [a-zA-Z0-9]

  • ls -l [a-d] → part of a range

  • ^ and $ works like in regex

  • la a*.{doc,docx} → OR

  • ls a*.(doc|docx) → OR

Vimscript functions

Create a custom command and function to create a new file in vim.

command! -nargs=1 Ms call s:NewFile(<q-args>)`

function! s:NewFile(fp)
  echom a:fp
  execute "e " . "~/foo/bar/" . a:fp . ".ext"
endfunction

Useful help:

  • :h %:h → filename modifiers
  • :h expand() → expand wildcards, including filename modifiers

question on SO

Better Text Objects
  • target function arguments
  • more types of object
  • consistent searching/jumping if you're not inside the thing you're targeting
  • jump forward or backward
  • look for the nth occurrence
  • select white space around/inside object

Github
Article about a plugin

Delete stuff in Vim without leaving insert mode
  • <C-h> - same as backspace
  • <C-w> - delete previous word
  • <C-u> - delete everything before cursor (on same row)
  • <C-d> or <C-t> - (un)indent a row
  • <C-e> - delete next word (create a mapping in vimrc)
Vim register for yanked text

It's annoying when you delete something and overwrite your yanked text.

Use numbered registers! "0 to "9

  • "0 contains the most recent yank.
  • "1 contains the most recent deleted text
  • "0p - paste the most recent yank, even if you deleted something after yanking it
ChezMoi shortcuts

Chezmoi is a great tool for managing dotfiles. This is a shortcut to update the source state based on local changes.

chezmoi status | cut -c 4- | xargs -I % -p sh -c 'chezmoi add ~/%'

Github

Sleep

“It enhances your memory and makes you more creative. It makes you look more attractive. It keeps you slim and lowers food cravings. It protects you from cancer and dementia. It wards off colds and the flu. It lowers your risk of heart attacks and stroke, not to mention diabetes. You’ll even feel happier, less depressed, and less anxious.”

Why We Sleep by Dr. Matt Walker

Linus Torvalds

From an Interview:

I don't want to claim that programming is an art, because it really is mostly just about 'good engineering'. I'm a big believer in Thomas Edison's 'one percent inspiration and ninety-nine percent perspiration' mantra. It's almost all about the little details and the everyday grunt-work.

But there is that occasional 'inspiration' part, that 'good taste' thing that is about more than just solving some problem - solving it cleanly and nicely and yes, even beautifully.

Remote Procedure Calls

A RPC is when an executable causes a procedure (subroutine) to execute on another computer, It's coded as if it were a normal (local) subroutine. You don't explicitly code the details for the remote interaction. You write the same code whether the subroutine is local or remote.

lsblk

lsblk is a command to get info about connected devices.

Used when attaching drives.

alias to real

Make aliased files the real file

for f in $(find . -type l -maxdepth 1);
    do cp --remove-destination $(readlink $f) $f;
done;
vim save sudo

Save (write) a (read only) Vim file with sudo when you opened it without sudo-ing:

:w !sudo tee % > /dev/null

Broot

Broot is a tool that shows the contents of a directory on one screen, even its got lots of files or sub-directories.

Github Actions

I should see if I can use GitHub actions to generate html from markdown and run some shell and python scripts.

Running a Binary in Debian

I was running a binary in Debian that was complaining about an environment variable not existing. I moved the binary into a $PATH directory and logged in as a sudo user. Why did this solve the problem?

Creating Slowly

As a hacker, or creator, or whatever the best label is, I always want to create something (usually code) and have it finished.

But a strange creativity and productivity boost comes from dabbling, dipping in and out.

I think that if the technical challenges aren't too hard, then the main criteria for success is creativity.

Creativity needs time away from the project, and sleep, to bubble up and let ideas grow.

Ultimately, the most successful path is usually the most interesting, because success has more consequences than failure. "Interesting" requires elements of novelty and surprise, and without creativity these elements can't flourish. 3/n

Dabbling results in more creativity than 6+ hours of strenuous work, and is more likely to give you satisfying results.

Arrow syntax in bash

bar << foo bar will stop reading input when it reached "foo".

bar <<< "foo" foo is all the input. bar wont run interactively.

bar < <(foo:list) process subscription. Kind of like piping in the output of multiple commands.

Stack Overflow

Read and Write the Same File in Bash

I tried to read and write the same file in a pipeline, and got caught out by a race condition (why is the file empty?!). Do this instead:

some_script < file > smscrpt.$$ \
&& mv smscrpt.$$ file || rm smscrpt.$$

|| removes the temporary file if it errors.

$$ is the process ID and ensures that you always have a unique temporary file name.

JSON tools

jj - A stream editor
jq - A json processor
python -m json.tool

I like jq for pretty printing JSON output, jj for making JSON pretty or condensed.

This was really useful when optimizing the search index for this blog.

Docker Commands
docker run -d ...
docker logs -f <ID>
docker run -it ...
docker run -itd
docker container attach <ID>
<C-p><C-q> -> detach from container interactively

stack overflow

Split Long Strings

Split long strings (or command outputs) onto multiple lines Find and replace a particular char (maybe :) with a \n.

... | tr ':' '\n'

... | sed 's/:/\n/g'
ripgrep Regret
  1. Without noticing, create a .gitignore file with a single * in it.
  2. Spend a day trying to understand why ripgrep has stopped working for only 1 project.
  3. 😭 😭 😭
`cat` and a new line

If you're cat-ing a file and the bash prompt doesn't start on a new line (cos the file you displayed using cat doesn't end with a new line char) the following will fix it:

cat <filename> ; echo

Git LFS

Cloning large repos, or repos with large files in them, doesn't work with git clone ... you need to use git lfs clone ...

So why is git lfs clone deprecated? What's the replacement?

nohup

Use nohup to keep a curl process running even when the terminal (tty?) session autocloses at 3am.

Shell Comparisons

You can group shells into groups:

  1. ksh - korn shell and zshell
  2. sh - bourne shell and bash (the bourne again shell)

Because

  • zsh isn't a superset of bash.
  • bash is a superset of the bourne shell.