[{"title":"About","url":"about.html","body":"This site is a mixture of my notes, a portfolio and a record of my hobbies and interests. It might be morphing into a Zettelkast It\u2019s a personal project and somewhere I can experiment with web and Most of my work is as a freelance I also do web developmen and blockchain If you have questions, comments or find a mistake, you can find me on twitter or email. Thanks for\u00a0readin \"0\"});"},{"title":"Recognizing Traffic Lights Using The Azure Custom Vision API","url":"traffic.html","body":"You can upload any image, I suggest googling dashcam traffic lights, or use one of the images below. Images should be smaller than 4mb and should be .jpeg, .bmp or .png. The Shortest Route To A\u00a0Demo This demonstrat was inspired by the job descriptio for a freelance role I recently applied for. The project involved recognisin faults with traffic lights and I wanted to see how quickly I could develop an end-to-end computer vision system that recognises and labels This is a relatively simple solution which prioritize speed, simplicity and low costs. I used the free tier of the Azure Custom Vision service to train and deploy the model. The model is trained to recognise and label Summary The model is hosted on Azure Custom Vision on the free\u00a0tier The model is trained on ~4500\u00a0imag Images are part of the DriveU Traffic Light\u00a0Data To improve the model I\u00a0would: Use many more\u00a0image Experiment with different Tune Method Find a good dataset. It would have taken too long to create my own labelled dataset so I needed to find a freely available set of labelled images. It turns out there are several to choose from. Waymo have a huge dataset that is freely available but I chose to use the DriveU Traffic Light dataset instead. It\u2019s well documented easily accessible and good\u00a0enoug Convert the images - the DTLD images are 16-bit .TIFF images. I needed .JPEG or .PNG images. I first converted the 16-bit .TIFF images to 8-bit, and then converted them to .JPEG. The DTLD dataset contains more image data and metadata than I needed so I simply ignored or stripped out what I didn\u2019t\u00a0nee Parse the label data to extract only the informatio I needed. The DTLD dataset contains labels that specify the location of the traffic lights in the images as well as the type of traffic light and what phase the lights are in. I was only interested in the location of the lights for this demo. I needed to convert the coordinate of the regions from absolute to\u00a0relativ Create a Custom Vision project, create custom tags, and upload pairs of images and labels in the required form. The documentat was good enough and whilst there were a few steps that were\u00a0uncle I was able to quickly figure out what to do, usually by clicking around to try a couple things and check the results. Each cloud platform has its own quirks and design concepts, and once you\u2019ve understood the pattern you can develop a good intuition for how each platform (in this case Azure) \u201cwants\u201d you to do\u00a0somethi Train the model. There aren\u2019t many options to choose from and the dataset wasn\u2019t very\u00a0big. Use the model to Create a simple UI on a static site (this page) using JavaScript and HTML. The JavaScript Fetch API is used to query the Custom Vision API. jQuery and some custom (vanilla) JavaScript is used to parse the results and create the interactiv elements on the\u00a0page. The model\u2019s results are shown by overlaying an HTML canvas element on top of the img element that shows the image that\u2019s been uploaded by the user. The regions and probabilit are drawn using HTML Canvas methods (strokeRec fillText etc). Next\u00a0steps The model is trained on images from German Cities. In order to generalise the model it should be trained using images from a wider distributi This could\u00a0incl Rural and Views from footpaths as well as\u00a0roads. Different Different cities and\u00a0countr It would be nice to let the user adjust the minimum probabilit threshold. Currently only results with a probabilit above 10% are\u00a0shown. Test\u00a0image You could use these images to test the model. You\u2019ll need to store them locally Test Image\u00a01 Test Image\u00a02 Test Image\u00a03 Test Image\u00a04"},{"title":"books","url":"book-notes.html","body":"A list of articles that are notes on\u00a0books."},{"title":"","url":"experience.html","body":"2021: Freelance Senior Data Scientist - Wayfair 2021: Freelance Blockchain Developer - Bitladon Blockchain integratio for a cryptocurr broker. Deploying and managing nodes for networks including Ethereum, Binance Smart Chain, Polkadot, Cardano, Tron, and others. Ansible and Docker were used to configurat The Rosetta API was used to create workflows including generating batches of addresses, notifying services about deposits, and 2020: Technical Founder and Web\u00a0Develo PipPip.ema Event-driv and long-term scheduled email delivery. Focus on writing then relax knowing that scheduled delivery is\u00a0guarant MoneyBar.n Personal financial dashboard and data 2019: Freelance Data Scientist - \u201cJohn is a creative and conscienti software engineer, who understand business requiremen well and translatin those into applicatio under tight deadlines. Highly recommende - NLP, data driven web-apps, mentoring. Working as an internal consultant I developed and delivered a range of tools. I worked with a wide variety of stakeholde and prioritize understand business needs and defining scopes, whilst using agile developmen practices. I advised my team on software developmen practices and tooling decisions, and mentored junior members to improve their coding and business skills. Tools included Python, and used Plot.ly Dash, WSL, and Azure 2019: Freelance Data Analyst - Uber \u201cJohn played a big role helping my team get our biggest and most challengin analytics tool across the line. He ramped up quickly, communicat well and was always responsive - Sankari Nair, Lead Developed an analytics tool using Python and Plot.ly Dash. The tool has a broad scope covering multiple regions, scales and business lines. I re-designe the app for scalabilit and performanc whilst increasing functional and provided a flexible foundation for new features to be developed after I left the project. Challenges included building custom refactorin legacy code, and processing large data 2018: Data Specialist - Blockport Design, create and maintain internal data tools for a cryptocurr exchange. As the only data specialist I did whatever needed to be done. I delivered tools to provide business insights including management informatio fraud analysis, tax reporting, KPI tracking, regulatory compliance and marketing and growth. I also delivered sentiment analysis of key social I worked with stakeholde across the business including back-end, founders, customer support, finance, DevOps and growth\u00a0tea Tools included: Python, Bokeh, Google Cloud Platform (BigQuery, DataStore, Data Studio), PostgreSQL Blockport has been bought by Bux 2017: Freelance Technical design and execution of a novel cryptograp crowd-fund method. Whilst working with a tech startup as a financial accountant I worked with stakeholde and external developers to design, test and execute an initial coin offering (ICO). I also led investor relations throughout the funding round and provided 2014: PwC Assurance - Banking & Data engineerin and analytics Deliver the ETL pipeline, analysis and visualizat of large financial datasets including financial journals and loan books. As a chartered accountant and external financial auditor of clients in banking and capital markets in London (HSBC, Barclays, Lloyds, BNP Paribas) I facilitate the transfer of data from client systems before transformi and loading them into our on-prem SQL environmen We recalculat clients\u2019 financial statements and mined the data for additional I also qualified as a chartered accountant with the ICAEW. 2010: PhD: Geotechnic In 2010 I began my PhD researchin granular materials at the University of Natural Resources and Life Sciences in Informatio about my research of silos and granular flows can be found here. 2009: Masters Degree: Civil and During my final year at Edinburgh University the Great Recession arrived. After I graduated I found a job at Starbucks and became curious about finance. I resolved to one day understand \u201chow banks work\u201d. \u2014 @johnmathe"},{"title":"fuse-search-snippets","url":"fuse-search-snippets.html","body":""},{"title":"fuse","url":"fuse.html","body":""},{"title":"landing-images","url":"landing-images.html","body":""},{"title":"title","url":"index.html","body":"This is the page\u00a0conte Content from this page isn\u2019t seen Images on the landing page need to be set in 1. landing.ht"},{"title":"Averages","url":"health.html","body":""},{"title":"pages","url":"pages.html","body":"This page is a list of links to other pages. Each of the linked pages is a collection of posts around a particular theme - book reviews, photograph Content is all in the template (pages.htm"},{"title":"Portfolio","url":"portfolio.html","body":"Content is all in the template. It\u2019s a list of links to other\u00a0page"},{"title":"Snippets","url":"snippets.html","body":""},{"title":"Analytics","url":"analytics.html","body":""},{"title":"Naming things is\u00a0hard","category":"snippet","url":"naming-things-is-hard.html","date":"14 October 2021","tags":"non-technical/entrepreneurship ","body":"Seth\u2019s blog - Background and\u00a0contex small business website Process Testing"},{"title":"Tove Lo -\u00a0Habits","category":"snippet","url":"tove-lo-habits.html","date":"14 October 2021","tags":"pop ","body":"wikipedia"},{"title":"Logging Best Practices\u00a0(notes)","category":"Technical/Engineering","url":"logging-best-practices-notes.html","date":"7 October 2021","tags":"programming, back-end ","body":"A great aricle by someone called Thomas about how to build useful logs. I find the inclusion of examples to be very useful, and the background informatio in the introducti is a nice\u00a0addit My notes (WIP): Log after not before - don\u2019t say what is going to happen, say what has happened. This makes it much more explicit and easier to read. The reader doesn\u2019t need to"},{"title":"How to get rich without getting lucky\u00a0(notes)","category":"Non-technical/Entrepreneurship","url":"how-to-get-rich-without-getting-lucky.html","date":"5 October 2021","tags":"naval-ravikant, mental-models, business ","body":"My notes and additions from a twitter thread created by @naval on 31 May\u00a02018. Background Seek wealth, not money or\u00a0status. Wealth is having assets that earn while you\u00a0sleep. You must own a piece of a business - you wont get rich by renting out your\u00a0time. You will get rich by giving society what it wants but doesn\u2019t know how to get. At\u00a0scale. Money is how we transfer time and\u00a0wealth Status is your place in the Understand that ethical wealth creation is possible. If you secretly despise wealth, it will elude\u00a0you. Ignore people playing status\u00a0gam Actions Learn to sell. Learn to build. If you can do both, you will Arm yourself with specific knowledge, and\u00a0levera Pick an industry where you can play long term games with long term\u00a0peopl Maximise positive feedback loops. Design and defend against negative All the returns in life, whether in wealth, relationsh or knowledge, come from positive feedback loops, The internet has massively broadened the possible space of careers. Most people haven\u2019t figured this out yet. Write down some\u00a0examp Play games where you iterate on past successes and experience to create more success and more experience This Pick business partners with high intelligen energy, and, above Knowledge is not the same as intelligen Its hard to estimate higher levels of expertise relative to your own. Additional knowledge can make someone look intelligen but you can\u2019t know if it would take 100, 1000 or 10,000 hours to acquire Don\u2019t partner with cynics and pessimists Their beliefs Building specific knowledge will feel like play to you but will look like work to\u00a0others. Specific knowledge is often highly technical or creative. It cannot be outsourced or\u00a0automat Specific knowledge is found by pursuing your genuine curiosity and passion rather than whatever is hot right\u00a0now. Specific knowledge is knowledge that you cannot be trained for. If society can train you, it can train someone else and replace\u00a0yo When specific knowledge is taught, it\u2019s through not\u00a0school Accountabi Embrace and take business risks under your own name. Society will reward you with equity, and\u00a0levera The most accountabl people have singular, public, and risky brands: Oprah, Trump, Kanye,\u00a0Elo Leverage - capital, people, no marginal\u00a0c \u201cGive me a lever long enough, and a place to stand, and I will move the earth.\u201d -\u00a0Archimed Capital means money. To raise money, apply your specific knowledge, with and show resulting good\u00a0judgm Fortunes require leverage. Business leverage comes from capital, people, and products with no marginal cost of replicatio (code and\u00a0media) Labor means people working for you. It\u2019s the oldest and most fought-ove form of leverage. Labor leverage will impress your parents, but don\u2019t waste your life chasing\u00a0it Capital and labor are permission leverage. Everyone is chasing capital, but someone has to give it to you. Everyone is trying to lead, but someone has to follow\u00a0you Code and media are permission leverage. They\u2019re the leverage behind the newly rich. You can create software and media that works for you while you\u00a0sleep. An army of robots is freely available - it\u2019s just packed in data centers for heat and space efficiency Use\u00a0it. If you can\u2019t code, write books and blogs, record videos and\u00a0podcas Leverage is a force multiplier for your judgement. - this seems related to comments by Geoffrey Hinton about developing your intuitions interview. Search through the article for all the mentions of\u00a0intuiti Learning and\u00a0practi Apply specific knowledge with leverage, and progress is\u00a0inevita You should be too busy to \u201cdo coffee\u201d, while still keeping an Set and enforce an aspiration personal hourly rate. If fixing a problem will save less than your hourly rate, ignore it. If outsourcin a task will cost less than your hourly rate, outsource\u00a0 Who you work with and what you work on are more important than how hard you\u00a0work. Work as hard as you\u00a0can. Thinking your own thoughts is tiring. Asking good questions is\u00a0hard. Doing is faster than\u00a0watch Developing good judgement requires experience if you have a chance to gain real experience then take it. Learning can amplify the benefits of\u00a0experie Real experience is more useful than a prestigiou course or\u00a0degree. Be patient and persistent shape your circumstan It doesn\u2019t mean you\u2019ve made a mistake if you can\u2019t do something real right\u00a0now. There is no skill called business. Avoid business magazines and business classes. Think about why and extend this to other\u00a0medi Passive, peace-meal knowledge acquisitio by itself does not lead to specific knowledge or expertise. Subscribin to newsletter or social media accounts offers quickly diminishin returns, at\u00a0best. Read long-form media which you actively looked for. Informatio that comes to you for free has competing interests which put you\u00a0second If you read only 1 book on a subject then you\u2019ll likely be a clone. If you read 2 books you\u2019ll grapple with confusion. Read 3 and you\u2019ll begin to form your own substantia opinions Become the best in the world at what you do. Keep redefining what you do until this is\u00a0true. There are no get rich quick schemes. That\u2019s just someone else getting rich off\u00a0you. Study game theory, psychology persuasion ethics, mathematic and computers. Real experience with skin in the game will teach you more than a book or a\u00a0professo There are lots of ways to grow beyond being a beginner, but no one can do the heavy-lift for you. A course or product that offers to teach you specific knowledge will give diminishin returns. The more of a beginner you are, the better the course will\u00a0appea Reading is faster Remember\u00a0w When you\u2019re finally wealthy, you\u2019ll realize that it wasn\u2019t what you were seeking in the first place. But that\u2019s for another\u00a0da Check in with your 70 year old self, and your 10 years older self. What do they think of you now? Are they sympatheti Proud? Do they say go faster, or slow down? Be kind to\u00a0yoursel There is a big difference between saying \u201cI am intimidate and \u201cI am feeling intimidate You can do\u00a0it. My top\u00a02: Embrace and take business risks under your own name. Society will reward you with equity, and\u00a0levera Capital means money. To raise money, apply your specific knowledge, with and show resulting good\u00a0judgm"},{"title":"Ultra-running\u00a0benchmarks","category":"snippet","url":"ultra-running-benchmarks.html","date":"1 October 2021","tags":"courtney-dauwalter, running, 150km, jeff-pelletier ","body":"7\u00a0days/wee 25\u00a0km/day Start easy. Last 50% begin to move through the\u00a0field. 40% - 50% of entrants DNF 90% of the course is\u00a0runnabl 50km: 6:10 min/km (Diez Vista\u00a050km 50km: 7:25 - 8:00 min/km (Beaver Flat 50 JP) 160km: 10:35 min/km (JP) 150km: 6:30 - 8:00 min/km (UTMB CD) training: 6:10 is slow enough to stay fresh. 5:10-5:30 is\u00a0fast."},{"title":"Vim built-in color\u00a0names","category":"snippet","url":"vim-built-in-color-names.html","date":"29 September 2021","tags":"vim ","body":"Script to output colors in a\u00a0buffer: :so SO question with"},{"title":"Lightning network\u00a0point-of-sale","category":"snippet","url":"lightning-network-point-of-sale.html","date":"28 September 2021","tags":"lightning, bitcoin, crypto ","body":"A project to demonstrat an offline, fast, point-of-s device to allow merchants and customers to make fast cheap transactio on the Github\u00a0rep lnurl super\u00a0prot Also lnbits Introducin OFFLINE <10 #bitcoin LN tut coming soon, workshops at #hccp21 and @AdoptingB Check out the repo! the @lnbits wallet in the background when the transactio is made \ud83e\udd29) Ben Arc \ud83c\udff4\udb40\udc67\udb40\udc62\udb40\udc77\udb40\udc6c\udb40\udc73\udb40\udc7f\u270a\u26a1\ufe0f (@arcbtc) September 27, 2021"},{"title":"Make\u00a0files","category":"snippet","url":"make-files.html","date":"28 September 2021","tags":"automation, build-system ","body":"This article is a fairly practical introducti to the GNU make tool. It\u2019s not the article I was looking for though, so I might swap it out if I find a better\u00a0one"},{"title":"Linux performance analysis in 60\u00a0seconds","category":"snippet","url":"linux-performance-analysis-in-60-seconds.html","date":"27 September 2021","tags":"linux, sys-admin ","body":"A blog post with a checklist of steps to take and questions to ask when investigat a performanc issue on a Linux\u00a0serv"},{"title":"Stop reading the\u00a0news","category":"snippet","url":"stop-reading-the-news.html","date":"22 September 2021","tags":"humanity, culture, web ","body":"An article from the Farnham Street\u00a0blo"},{"title":"Rules for talking to\u00a0children","category":"Non-technical/Social","url":"rules-for-talking-to-children.html","date":"22 September 2021","tags":"children, communication ","body":"State the idea you wish to express as clearly as possible, and in terms preschoole can understand Example: \u201cIt is dangerous to play in the\u00a0street Rephrase in a positive manner. \u201cIt is good to play where it is\u00a0safe.\u201d Rephrase the idea, bearing in mind that preschoole cannot yet make subtle distinctio and need to be redirected to authoritie they trust. \u201cAsk your parents where it is safe to\u00a0play.\u201d Rephrase your idea to eliminate all elements that could be considered prescripti directive, or instructiv In the example, that\u2019d mean getting rid of \u201cask\u201d: \u201cYour parents will tell you where it is safe to\u00a0play.\u201d Rephrase any element that suggests certainty. That\u2019d be \u201cwill\u201d: \u201cYour parents can tell you where it is safe to\u00a0play.\u201d Rephrase your idea to eliminate any element that may not apply to all children. Not all children know their parents, so: \u201cYour favorite grown-ups can tell you where it is safe to\u00a0play.\u201d Add a simple motivation idea that gives preschoole a reason to follow your advice. \u201cYour favorite grown-ups can tell you where it is safe to play. It is good to listen to\u00a0them.\u201d Rephrase your new statement, repeating the first step.\u201d \u201cGood\u201d represents a value judgment, so: \u201cYour favorite grown-ups can tell you where it is safe to play. It is important to try to listen to\u00a0them.\u201d Rephrase your idea a final time, relating it to some phase of developmen a preschoole can understand \u201cYour favorite grown-ups can tell you where it is safe to play. It is important to try to listen to them, and listening is an important part of\u00a0growing As Arthur Greenwald, a former producer of the show, put it to me, \u201cThere were no accidents on Mister Rogers\u2019 Neighborho He took great pains not to mislead or confuse children, and his team of writers joked that his on-air manner of speaking amounted to a distinct language they Fundamenta Freddish anticipate the ways its listeners might misinterpr what was being\u00a0said source"},{"title":"View and change settings for GCloud CLI","category":"snippet","url":"view-and-change-settings-for-gcloud-cli.html","date":"22 September 2021","tags":"google-cloud-platform, cli ","body":"gcloud config set account gcloud auth list"},{"title":"Google\u00a0Pub/Sub","category":"snippet","url":"google-pub-sub.html","date":"20 September 2021","tags":"google-cloud-platform, message-systems ","body":"Google Pub/Sub has client libraries in all the usual languages. You can also construct the API calls yourself. This is a link to the If I were to use a JavaScript beacon to push a message to a topic, my web analytics engine (a collection of cloud functions) could subscribe to the\u00a0topic. This would have the advantage that if the site were flooded with traffic and the maximum number of function instances wasn\u2019t enough to handle all the events, then none of the page views (or other events) would be lost because Pub/Sub guarantees delivery and would keep trying to deliver the message for up to 7\u00a0days. Using the beacon to trigger the cloud functions directly wouldn\u2019t work at scale, because once the maximum number of instances are being triggered as frequently as the function takes to run, the endpoint would become unresponsi There is no caching\u00a0la However, the site isn\u2019t being flooded with traffic, and I have better things to do than fix stuff that isn\u2019t\u00a0brok"},{"title":"Google Cloud Storage - TTL and CORS\u00a0settings","category":"snippet","url":"gcp-storage-bucket-ttl-and-cors-settings.html","date":"20 September 2021","tags":"google-cloud-platform, storage, cloud, caching ","body":"Documentat explaining how to set CORS and TTL for a Create a JSON file with something like\u00a0this: [{ \"origin\": [\"...\"], \"method\": [\"GET\", \"PUT\"], 300 }] gsutil cors set cors.json gsutil cors get | jq"},{"title":"Handwritten letters at\u00a0scale","category":"snippet","url":"handwritten-letters-at-scale.html","date":"20 September 2021","tags":"automation, hardware, marketing, raspberry-pi ","body":"A twitter thread showing how a business started creating handwritte notes to customers using a plotter, and how they scaled up and further automated the\u00a0proces It would be cool to create something like this for western Europe. seems to be the main commercial implementa of this idea in NA. Competitor I still think it would be fun to make a simple photo printing service. Share a photo in a WhatsApp message and its printed and\u00a0delive"},{"title":"The dangers of social\u00a0media","category":"snippet","url":"the-dangers-of-social-media.html","date":"16 September 2021","tags":"technology, mobile, social-media ","body":"A ledger (what\u2019s a ledger? It\u2019s a webpage) from the \u201cCenter For Humane Technology showing some of the dangers of social\u00a0med The \u201cDo unto others\u201d section is outstandin It describes the views of some tech company leaders, including how they limit usage of devices and social media for themselves and their\u00a0kids"},{"title":"Visual mathematical\u00a0proofs","category":"snippet","url":"visual-mathematical-proofs.html","date":"11 September 2021","tags":"math ","body":"A question on Stack Overflow asking for great visual proofs of right the area of a\u00a0circle archive"},{"title":"1 Peter 1 vs\u00a010","category":"Non-technical/Journal","url":"1-peter-1-vs-10.html","date":"10 September 2021","tags":"bible ","body":"Concerning this salvation, the prophets who prophesied about the grace that was to be yours searched and inquired carefully, inquiring what person or time the Spirit of Christ in them was indicating when he predicted the sufferings of Christ and the subsequent glories. It was revealed to them that they were serving not themselves but you, in the things that have now been announced to you through those who preached the good news to you by the Holy Spirit sent from heaven, things into which angels long to look. The salvation that Peter is referring to is described previously He was encouragin us that even though our salvation is mysterious it is very valuable, and that the inexpressi joy that we feel is not unusual or a cause for concern. Don\u2019t be uncomforta Grace and peace. To encourage us even more, Peter tells us that even the prophets (he implicitly assumes his readers are familiar with the prophets, so I guess they\u2019re definitely Jewish, as guessed in the first post in this series) searched for understand about our salvation. They wanted to know who the Christ would be and when he would arrive. I guess they initially thought they were doing this to satisfy their own curiosity because it had to be revealed to them that this was in fact a service to the saints that would come after them. Interestin it says that it was the Spirit who predicted Christ\u2019s sufferings and the following glories. I would not have expected that the Spirit of God needed to do any predicting about Christ\u2019s sufferings or glories. Are you even able to predict something if you\u2019re omnipotent Back to the text, Peters says that things have been announced to us via those who preached the good news to us by the Hold Spirit, who was sent from heaven. Preaching by the power or enabling of the Holy Spirit has revealed things to us that angels long1 to know. We don\u2019t know everything and we still have questions, but some of our questions are answered, some of them inexpressi and some of the things we find mysterious or uncomforta are supposed to be so until our salvation is complete. Grace and peace. Hang in there, you\u2019re doing ok.Long, or longed? I would have expected it to be \u201clonged to know\u201d, but its not. Does chronology get messed up from our perspectiv if the other party is outside of time? \u21a9"},{"title":"9/11 and the American\u00a0psyche","category":"snippet","url":"9-11-and-the-american-psyche.html","date":"10 September 2021","tags":"history, essay ","body":"An essay in the FT: \u201cTwenty years later, the political and psychologi consequenc of the attack have become archive"},{"title":"How doctors\u00a0die","category":"snippet","url":"how-doctors-die.html","date":"9 September 2021","tags":"humanity, death ","body":"An article about why doctors want less treatment for terminal illnesses than archive"},{"title":"Life","category":"snippet","url":"life.html","date":"8 September 2021","tags":"aphorism ","body":"If youth is wasted on the young, then life is wasted on the\u00a0living"},{"title":"Slavery","category":"snippet","url":"slavery.html","date":"8 September 2021","tags":"humanity ","body":"This article about how people become slaves is very It puts a very different perspectiv on my life, ambitions and"},{"title":"Cloud Functions Minimum\u00a0Instances","category":"snippet","url":"cloud-functions-minimum-instances.html","date":"8 September 2021","tags":"google-cloud-platform, serverless ","body":"Google Cloud Platform have introduced a \u201cminimum instances\u201d features in the Cloud If you have a function using 128MB of memory (the smallest option), it will cost a minimum \u20ac0.061 per day (\u20ac1.86 per month) to run 1"},{"title":"nmap","category":"snippet","url":"nmap.html","date":"4 September 2021","tags":"networking, cli ","body":"sudo nmap -sP 192.168.1. | grep \"Nmap\" Thanks to Jeff Geerling (again) I found this nifty command to see what devices are connected on a local\u00a0netw Found in this article about setting up a\u00a0pi-hole."},{"title":"1 Peter 1 vs\u00a03","category":"Non-technical/Journal","url":"1-peter-1-vs-3.html","date":"27 August 2021","tags":"bible ","body":"May grace and peace be multiplied to you. Part 1: Blessed be the God and Father of our Lord Jesus Christ! According to his great mercy, he has caused us to be born again to a living hope through the resurrecti of Jesus Christ from the dead, to an inheritanc that is imperishab undefiled, and unfading, kept in heaven for you, who by God\u2019s power are being guarded through faith for a salvation ready to be revealed in the last time. Part 2: In this you rejoice, through now for a little while, if necessary, you have been grieved by various trials, so that the tested genuinenes of your faith - more precious that gold that perishes though it is tested by fire - may be found to result in praise and glory and honor at the revelation of Jesus Christ. Part 3: Though you have not seen him, you love him. Though you do not now see him, you believe in him and rejoice with joy that is inexpressi and filled with glory, obtaining the outcome of your faith, the salvation of your souls. I thought that writing about 1 paragraph per post would be easy, maybe even too short, but this is so dense that I\u2019m not sure. The last three highlighte paragraphs are all one paragraph originally , I\u2019ve just split them up to make it easier to write about. Part 1 After Peter\u2019s greeting (previous post), he starts his letter by praising Father God - the Lord of Lords and King of Kings. Peter says that it is by God\u2019s mercy that we have been born again, and that we are able to enjoy a living hope. As soon as Peter says this he begins to explain how, and he uses a chain or reasoning again like he did in his greeting. I\u2019m going to try and paraphrase in order to make the text less dense and make its meaning more obvious: Because of God\u2019s mercy, we have a hope that is alive and so significan it is as if we have been reborn. Our life is new, reset, fundamenta different. This is a direct result of Jesus dying and resurrecti The living hope seems to be hoping (or trusting) in the inheritanc which is promised but not yet received. The inheritanc is described as: Imperishab - cannot be destroyed. Undefiled - cannot be made dirty, less pure, less valuable. Unfading - doesn\u2019t get old or less good as time passes. It doesn\u2019t diminish. Kept in heaven for you - deliberate reserved in heaven for you, in particular By God\u2019s power, you are being guarded until your salvation. This protection is received via your faith. Protection comes through your faith. Not because of it, or from it, but through it. The purpose of the protection is to keep you safe until your salvation, which will only be revealed to you in the last time. Your salvation is mysterious because it hasn\u2019t been revealed yet. This last point is itself mysterious Part 2 The tested genuinenes of your faith is more precious than gold. Not just the genuinenes integrity, or sincerity of your faith, but the tested sincerity of your faith. This tested sincerity is more precious than gold. Your faith is contrasted with gold that will eventually decompose - even really high quality gold that is ultra refined. But your tested and sincere faith is more resilient and more valuable than gold. The process that creates a tested and genuine faith in Jesus is therefore to be appreciate because of the end result. If you have a living hope of an inheritanc through Jesus resurrecti then rejoicing would be a good response even if it is causing you grief and many trials. The trials and grief will ultimately result in praise and glory and honor during the revelation (revealing of Jesus Christ. I\u2019m not sure if this is glory and honor for Jesus, or for the person who has a tested and genuine faith. Maybe both? Part 3 I don\u2019t know if this passage is instructio or reassuring or both. I guess it\u2019s both at the same time - you should love him, even though you\u2019ve never seen him. And also, don\u2019t feel uneasy about having never seen him. It\u2019s unusual to love someone you\u2019ve never seen, but in this situation it\u2019s normal. Don\u2019t worry. Grace and peace. The same goes for inexpressi joy - you\u2019re hope in Jesus, his resurrecti and your coming resurrecti makes you happy in ways that you didn\u2019t expect and can\u2019t really articulate or explain - this is ok! It is more than ok - it is evidence of the sincerity and truthiness of your faith and is expected. Believing in someone you\u2019ve never seen, and loving someone you\u2019ve never seen is going to result in you obtaining salvation. It\u2019s unusual, and different, but don\u2019t worry. Grace and in my English bible. Does the original text even have paragraphs \u21a9"},{"title":"Understanding GCP\u00a0charges","category":"snippet","url":"understanding-gcp-charges.html","date":"27 August 2021","tags":"google-cloud-platform ","body":"A clear article solving most of my problems around understand why I\u2019m incurring charges on Google Cloud Platform (Gcloud, GCP) when I\u2019m clearly within their free usage\u00a0limi The deployment costs section in this page is also\u00a0usefu"},{"title":"My dead dad\u2019s\u00a0journal","category":"snippet","url":"my-dead-dads-journal.html","date":"26 August 2021","tags":"humanity ","body":"Your relationsh will come to define your life more than anything\u00a0e A blog post about someone who read their dad\u2019s journal after he died. Its about regret, grief, and struggles against internal Its about the tangible consequenc of The comments are"},{"title":"1 Peter 1 vs\u00a01","category":"Non-technical/Journal","url":"1-peter-1-1.html","date":"25 August 2021","tags":"bible ","body":"People usually say that the bible is a collection of books, but some of the books are actually letters. Two of the letters were written by Peter, who was one of Jesus\u2019 disciples. \u201c1 Peter\u201d is the first\u00a0lett The other numbers are the chapters and verses. They\u2019re used to divide the text so that specific parts can be referred to easily. They\u2019re not part of the original text - chapters were introduced around the 13\$$^{th}\\ century, and verses in the 16\\(^{th}\\ \\(^{1}\$$To those who are elect exiles of the Dispersion in Pontus, Galatia, Cappadocia Asia, and Bithynia, according to the foreknowle of God the Father, in the sanctifica of the Spirit, for obedience to Jesus Christ and for sprinkling with his blood: May grace and peace be multiplied to you. This is the start of a letter to a specific group of people. It\u2019s intended to be encouragin and dense from the very beginning. It is intended that the letter is remembered and paid attention\u00a0 The sentence structure is awkward when its translated into modern English because its a translatio Presumably in the original language you could create long statements with many clauses in them and people would be comfortabl with that. I\u2019m unaware of the letter writing customs of the day, so maybe Peter is using a very normal type of introducti or maybe he is deliberate imitating the style of a specific type of letter, or\u00a0person. The first sentence implies that exiles of the Dispersion (what dispersion I don\u2019t know.) are not exiles by accident. Even though I bet becoming exiled felt like an unplanned interrupti to their life. The first sentence says that God chose them, and that there is a purpose. There is a chain of reasoning that involves God the Father, God the Son and God the Holy Spirit - the trinity. One God in three persons\u2026 I don\u2019t get it, I can\u2019t explain it, but I\u2019m going to try and work with According to the foreknowle of God the Father\u00a0\u2192 in the sanctifica of the Spirit\u00a0\u2192 for obedience to Jesus Christ and for sprinkling with his\u00a0blood It\u2019s like 1 and 2 and requiremen for 3, because \u201cfor\u201d means \u201cthe reason why something was\u00a0done\u201d. I should find out what \u201csprinklin with his blood\u201d means, instead of guessing. Clearly its a weird and gross\u00a0idea It means\u2026 being marked by Jesus\u2019 blood. It seems clear enough that this is the blood that was spilt when he was sacrificia murdered, and it seems that the benefit or the reason why this would be desirable is already obvious to the readers. Having Jesus\u2019 blood on you is to associate yourself with the benefits of that sacrifice as well as to count the cost of the\u00a0sacrif It\u2019s probably a line of reasoning that would be intuitive to believers with a Jewish background if they\u2019ve previously sacrificed animals in temples during various festivals. Is this letter written to a group of believers with a Jewish (rather than Back to the chain of reasoning: the foreknowle of the Father and the sanctifica of the Spirit is necessary for being associated with Jesus sacrifice and for being obedient to Jesus. That\u2019s a big conclusion from only the May grace and peace be multiplied to you. I like that exhortatio so much I could get it as a tattoo. Grace and Peace, better than fine dining and a comfortabl bed. Also, chewing over the implicatio of the previous sentence is enough to cause confusion and angst - so the exhortatio is well timed. Chill - even though you\u2019ve been exiled, you weren\u2019t planning on being a migrant, and God planned this for you, and this is part of your necessary sanctifica in order to be associated with Jesus\u2019 sacrifice - you can be full of peace, and there is lots of grace (which is forgivenes and patience, and maybe gentleness available to\u00a0you. Next\u00a0sente if { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.wi"},{"title":"Miller CLI","category":"snippet","url":"miller-cli.html","date":"25 August 2021","tags":"cli, tools ","body":"Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, tabular JSON Try this if you\u2019re operating on JSON of CSV data, and you want to use column names instead of"},{"title":"Conditionally setting your git\u00a0config","category":"snippet","url":"conditionally-setting-your-git-config.html","date":"24 August 2021","tags":"vcs, git ","body":"Blog post showing how you can set git variables depending on where you are in a It hinges on conditiona includes."},{"title":"Daddy-Daughter To-Do\u00a0List","category":"Non-technical/Journal","url":"daddy-daughter-to-do-list.html","date":"22 August 2021","tags":"daughter, family ","body":"Things we want to do\u00a0togethe Make some\u00a0furni Run to the\u00a0beach \u00a0Parkour Go on bike\u00a0rides Go to work together (when she is\u00a0older) Go on long train journeys together (Paris and\u00a0Salzbu Go to restaurant and have Take interestin people to lunch and ask them\u00a0quest"},{"title":"Conway\u2019s\u00a0Law","category":"snippet","url":"conways-law.html","date":"16 August 2021","tags":"organisations ","body":"Organizati which design systems \u2026 are constraine to produce designs which are copies of the communicat structures of these organizati - Melvin\u00a0Con In\u00a0reverse If the architectu of the system and the architectu of the organizati are at odds, the architectu of the organizati wins - Ruth\u00a0Malan"},{"title":"Elon Musk\u2019s Engineering\u00a0Philosophy","category":"Technical/Engineering","url":"elon-musk-s-engineering-philosophy.html","date":"8 August 2021","body":"Whilst giving Tim Dodd a tour of a SpaceX facility, Musk described an interestin five-step Make the requiremen less dumb. The requiremen are definitely dumb; it does not matter who gave them to you. He notes that it\u2019s particular dangerous if an intelligen person gives you the requiremen as you may not question the requiremen enough. \u201cEveryone\u2019 wrong. No matter who you are, everyone is wrong some of the time.\u201d He further notes that \u201call designs are wrong, it\u2019s just a matter of how\u00a0wrong. Try very hard to delete the part or process. If parts are not being added back into the design at least 10% of the time, not enough parts are being deleted. Musk noted that the bias tends to be very strongly toward \u201clet\u2019s add this part or process step in case we need it.\u201d Additional each required part and process must come from a name, not a department as a department cannot be asked why a requiremen exists, but a person\u00a0can Simplify and optimize the design. This is step three as the most common error of a smart engineer is to optimize something that should not\u00a0exist. Accelerate cycle time. Musk states \u201cyou\u2019re moving too slowly, go faster! But don\u2019t go faster until you\u2019ve worked on the other three things\u00a0fir Automate. An important part of this is to remove in-process testing after the problems have been diagnosed; if a product is reaching the end of a production line with a high acceptance rate, there is no need for archive"},{"title":"Pen-testing web\u00a0apps","category":"snippet","url":"blog-post.html","date":"6 August 2021","tags":"penetration-testing, hacking, web-apps, credentials ","body":"A blog post about how someone compromise a group of web\u00a0apps. It lists a series of technologi and techniques that the author uses as they progress These would make a useful list of things to know in order to build safe web-apps and not repeat the mistakes of the unfortunat"},{"title":"Running through\u00a0adversity","category":"snippet","url":"running-through-adversity.html","date":"6 August 2021","tags":"hard-rock, running, sport ","body":"The toughness of elite ultra-runn is Just one of the problems Sabrina Stanley encountere would be enough to justify dropping\u00a0o archive"},{"title":"Starbase Tour with Elon\u00a0Musk","category":"snippet","url":"starbase-tour-with-elon-musk.html","date":"4 August 2021","tags":"elon-musk, starbase, engineering ","body":"Part 1 of an incredible interview and tour of starbase with Elon\u00a0Musk archive"},{"title":"Remote\u00a0Working","category":"snippet","url":"remote-working.html","date":"2 August 2021","tags":"remote, engineering ","body":"I live and work near\u00a0Amste My manager is in\u00a0Berlin A stakeholde is in\u00a0Boston Another stakeholde is in\u00a0India Another stakeholde is in\u00a0Ireland \u00af\\_(\u30c4)_/\u00af"},{"title":"Athletes, Careers, and Mental\u00a0Health","category":"snippet","url":"athletes-and-mental-health.html","date":"2 August 2021","tags":"sport, competition, comparmentalization, psychology ","body":"Being very good at one particular thing can make it easy to be bad at normal\u00a0thi Getting over\u00a0gold Insecure Overachiev Searching the FT brings up several"},{"title":"WTF\u00a0Python","category":"snippet","url":"wtf-python.html","date":"2 August 2021","tags":"python, programming ","body":"A repo of curious"},{"title":"Beach\u00a0Photos","category":"Non-technical/Photographs","url":"beach-photographs.html","date":"1 August 2021","body":""},{"title":"Heuristics for effective software\u00a0development","category":"snippet","url":"heuristics-for-effective-software-development.html","date":"26 July 2021","tags":"engineering, teams, organisations ","body":"\u201cWithout psychologi safety, respect, and trust, none of the following is\u00a0possibl \u201cThe best ways to work are collaborat Negotiatio is not collaborat Isolated individual making heroic efforts are never as effective as collaborat groups. We get the best results when customers, business people, and developers literally Blog\u00a0post"},{"title":"Move a file between git\u00a0branches","category":"snippet","url":"move-a-file-between-git-branches.html","date":"14 July 2021","tags":"git ","body":"Checkout the branch where you want the file to by copied to,\u00a0then: git checkout Use a commit hash to pull files from any\u00a0commit Multiple files and directorie can be\u00a0specifi Overwrites the\u00a0file SO"},{"title":"Start with Finance to transform IT","category":"snippet","url":"start-with-finance-to-transform-it.html","date":"14 July 2021","tags":"engineering, organisations, business, corporations ","body":"Zwischenzu blog post arguing that to achieve a significan change in an organisati you need need\u00a0to: Get\u00a0fundin Persuade the finance department to give you\u00a0money. Understand what they\u00a0value Understand their cash\u00a0flows Understand how and why customers or clients part with their\u00a0mone Understand business constraint (legal, The five whys approach to \u201cConsider a deeper structural cause of cultural problems in change management how money flows through \u201cIf you want to transform IT in an enterprise start with finance. If you can crack that, you\u2019ve a chance to succeed with sec and controls functions. If you don\u2019t know why it\u2019s important to start with finance, you\u2019ll definitely fail\u00a0\u201c"},{"title":"The Worst Volume Control UI","category":"snippet","url":"the-worst-volume-control-ui.html","date":"14 July 2021","tags":"ui ","body":"Hilarious article from UI Collecitve showing the results of a competitio to design the worst possible volume control interface. \ud83d\ude02 \ud83d\ude02\u00a0\ud83d\ude02"},{"title":"Playing with Google Cloud\u00a0Platform","category":"Technical/Engineering","url":"playing-with-google-cloud-platform.html","date":"13 July 2021","tags":"cloud, google-cloud-platform, serverless ","body":"Building my own web analytics has been a gateway to learning more about Google Cloud Platform. So far I\u2019ve used DataStore, BigQuery, Cloud Functions, Pub/Sub, Storage Buckets and Scheduler. Version 1 My first version of the analytics tool took the following form: Logger Function When the browser navigates to a page on the site, a JavaScript beacon is sent which triggers a cloud function. The function parses the page URL and the IP address, and creates a record in the database. Aggregator Function Each time the analytics page is loaded, a cloud function is triggered that gets every record from DataStore, parses the data and returns a JSON object containing the aggregated data. The browser receives the JSON, parses it and creates some charts and tables. The good: It works, it was quick and simple to build. The bad: Its expensive. Loading the analytics page is slow - when the DataStore was small it took 3 or 4 seconds, after a few thousand page views it took about 40 seconds. Conclusion - keep the logger function as it is, but improve the aggregator function. Version 2 The second version still used DataStore but was much more efficient. It didn\u2019t read the entire database and generate the aggregated results every time the analytics page was viewed. Instead, a cloud function periodical collected all the records in the datastore database and calculated the results. The results were written to a JSON file and sent to a storage bucket. When the analytics page was loaded in a browser, the browser collects and processes the JSON file from the bucket. This is much faster, performant and cheaper than creating a new JSON object each time the analytics page is viewed. The good: The analytics page loads at the same speed regardless of how much data has been aggregated and how frequently the analytics page is being viewed. Performanc issues have been solved, though I still don\u2019t think DataStore is the best database solution for this use case. The bad: DataStore seems expensive - I am being charged for AppEngine services (which I don\u2019t really understand but is caused by using DataStore) If I can get monthly costs down to about a cup of coffee (about \u20ac4/month or \u20ac0.15/day) then I don\u2019t mind running it indefinite Version 3 Use BigQuery instead of DataStore. BigQuery is a Data Warehouse that is well suited for analytics. It is not well suited for transactio use cases - where data is being read, updated or created many times per second. This is fine for my use case - the Page Logger function writes a record to a BigQuery table each time each time a page view is logged. During times of high traffic it\u2019s possible that concurrenc issues might arise and some page views will be lost, but this isn\u2019t an issue 99% of the time. My site traffic is very light. I believe I could use a newer API that google recently released to solve this problem but for now I\u2019ll use the normal API. The rest of the process is unchanged - the aggregator function periodical reads the (BigQuery) database, crunches that data and sends a JSON file to a storage bucket. The good: This is completely free. The analytics page can be viewed quickly regardless of the amount of site traffic. The bad: Under heavy traffic some page views might be lost due to a limit on how quickly new rows can be added to BigQuery tables. Using a new API might resolve this. Conclusion Totally free tools forever The combinatio of Cloud Functions, Storage Buckets and Big Query (along with Scheduler and Pub/Sub) seems really versatile and I think there are many interestin things that could be done by combining these services1 . Using them all for free (my usage is well within the free tier) makes the possibilit even more interestin Having compute and storage services running indefinite in the cloud for free is amazing. Documentat Aggregate by calendar month An improvemen to this analytics setup would be creating aggregated metrics for each calendar month and storing them in separate JSON files. This would prevent data older than one month being processed repeatedly and create a cap on the amount of computatio effort required (the maximum amount of data processed by one cloud function instance would become capped at one month). If the browser wanted to display more than one month of data, it would simply request more than one JSON file from the storage bucket. TODO Frontend - the DataTables column containing the date should be sorted as a Date object. It is being sorted like a normal string. Backend - create separate JSON files for each month. The question then becomes: \u201cJust because you could do it, should you do it?\u201d \u21a9"},{"title":"Daughter","category":"snippet","url":"daughter.html","date":"12 July 2021","tags":"family ","body":"Yesterday my daughter asked me to write a page in"},{"title":"Moral\u00a0tyranny","category":"snippet","url":"moral-tyranny.html","date":"12 July 2021","tags":"oppression, consent ","body":"Of all tyrannies, a tyranny sincerely exercised for the good of its victims may be the most oppressive It would be better to live under robber barons than under omnipotent The robber baron\u2019s cruelty may sometimes sleep, his cupidity may at some point be satiated; but those who torment us for our own good will torment us without end for they do so with the approval of their own conscience They may be more likely to go to Heaven yet at the same time likelier to make a Hell of\u00a0earth. This very kindness stings with intolerabl insult. To be \u201ccured\u201d against one\u2019s will and cured of states which we may not regard as disease is to be put on a level of those who have not yet reached the age of reason or those who never will; to be classed with infants, imbeciles, and - C. S.\u00a0Lewis"},{"title":"Upgrading Cryptographic\u00a0Libraries","category":"snippet","url":"upgrading-cryptographic-libraries.html","date":"10 July 2021","tags":"hashing, versioning ","body":"Blog post about how to make it easier to upgrade a cryptograp or Django encodes passwords for database storage like\u00a0this: Interestin Giovanni Collazo emphasises that we should design systems for change, which initially seems pretty close to contradict YAGNI, but the answer lies in the\u00a0contex"},{"title":"Startup Engineering\u00a0Lessons","category":"snippet","url":"startup-engineering-lessons.html","date":"10 July 2021","tags":"startup, engineering ","body":"Lessons of a startup engineer is a great blog post from Todd Wolfson. So great that I might write notes on it like I would a\u00a0book. archive"},{"title":"Poisson\u2019s\u00a0Equation","category":"snippet","url":"poissons-equation.html","date":"6 July 2021","tags":"math ","body":"A great article introducin and showing the relevance of"},{"title":"Thomas\u00a0Aquinas","category":"snippet","url":"thomas-aquinas.html","date":"6 July 2021","tags":"theology, history ","body":"His works in English and\u00a0Latin."},{"title":"Can an explanation of an historical event ever be completely\u00a0true?","category":"Non-technical/Journal","url":"historical-truths.html","date":"6 July 2021","tags":"history ","body":"We use historical events as examples to learn from, but is it possible to acquire a true understand of a The answer is important because we use that understand to learn by example, and identify patterns of cause and effect. Is it possible for a normal person to do this, or does it require training, lots of time, or special skills? Is it impossible for\u00a0everyo I started thinking about this when I had the When we think we understand why something historical happened, all we\u2019ve probably done is accept a story that Limitation An account of an event must necessaril be a simplifica - not all details can be recorded or learned. This is ok because most details are not pertinent and have no But how do we choose which details are included, and how do we verify that the details were How can we be sure that a lesson or conclusion based on historical events is\u00a0reliabl The truthfulne of a historical story could be thought of as a value on a scale ranging from completely false to perfectly true. I think that there are mechanisms that push popular or resilient narratives towards the middle of this scale and away from As a version of a story approaches the dishonest end it will contain an increasing number of errors or omit an increasing number of pertinent facts. This has the effect\u00a0of: Increasing the likelihood and frequency of someone hearing the story and rebutting Making it harder to align the assertions and implicatio with existing understand of\u00a0reality At the opposite end of the scale, a story is unlikely to be Adding truth requires adding complexity It is easier to create or capture a simple story than a As a story\u2019s detail and depth increases so do the resources required to communicat it. Each narrative is competing for attention and a complex story requires more resources to broadcast and listen to than a simple story. Those with the resources to do so will want the story to benefit them in some\u00a0way. This creates incentives to omit inconvenie truths Implicatio I think that there is unfortunat no substitute for the hard work of coordinati disparate informatio because the truthiness of a conclusion is generally proportion to the inconvenie of the effort spent forming\u00a0it We are predispose to choose convenienc over inconvenie and this\u00a0enabl History to be written by the \u201cwinning side\u201d, who have more resources than the \u201closing\u00a0si Complex events to be simplified into expedient and As time passes, the practical benefit of holding a view that differs from a popular narrative decreases. This reduces any incentive to challenge a popular\u00a0vi This creates a feedback loop that makes it increasing difficult for a younger generation to discover informatio about historical events that challenge a An\u00a0heurist How much evidence did I collect myself, that wasn\u2019t brought to my attention by an algorithm or by someone\u00a0el Was WW2 a battle of the good against the bad, or the bad against the really bad, or Why did the Allies win WW2? Was the influence of government on social freedoms in America 100 years ago dissimilar to China\u00a0toda Racial and ethnic discrimina was normal 100 years ago - it appears to have been so universall accepted that I\u2019m led to question the of modern attitudes about human nature and\u00a0morali"},{"title":"Load-testing my Web Analytics\u00a0Tool","category":"Technical/Web","url":"load-testing-web-analytics-tool.html","date":"2 July 2021","tags":"google-cloud-platform, cloud-functions, api ","body":"Table of Contents Background The Hacker News affect Bad\u00a0news Good\u00a0news API traffic for the\u00a0tool Dashboard for the get-analyt cloud\u00a0func Dashboard for the The Solution Idea 1: Use global\u00a0obj Idea 2: Store the results themselves in the\u00a0databa Idea 3: Forget DataStore, use\u00a0bucket Background I posted a previous article (about building an analytics tool) onto the Hacker News forum. It was quickly buried and didn\u2019t get any\u00a0attent To my surprise, I received an email from a Hacker News administra (Daniel) explaining that it was a good quality post and would be boosted to the front page at a random time within the next couple of\u00a0days. Sure enough, in the early hours of the next morning, the post was boosted. I woke up to various notificati that people had started following me on twitter, which never happens. After delegating the kid\u2019s breakfast duties, I logged into GCP to see what affect the extra traffic had on my The Hacker News\u00a0affec Traffic had increased by about 30x and my hastily built tool was looking very sub-optima Two problems stood out - the aggregated analytics data was taking anywhere from 20 - 30 seconds to load (up from around a passable-i 5 under normal conditions and I was running up a bill of Bad\u00a0news The reason for both of these problems was a shockingly inefficien and lazy approach to serving the\u00a0analyt Each time the analytics page was loaded, a cloud function would fetch all the data in the DataStore database, munch all that data and return a freshly derived blob of JSON. Never mind that almost the exact same computatio had occurred hundreds of times\u00a0alre As the amount of data in the DataStore increases, so does the time required to serve the analytics page. In the second chart below (dashboard for the get-analyt cloud function), it looks like the execution time increases at a rate of O(log\u00a0n). Good\u00a0news The good news though was that the function was handling the extra traffic smoothly. You can see in the dashboard image below (click on it) that almost every request was completed in less than 200ms, which I think is fine for a background process. I could also see the active instances scaling up and down well within its preset limits, as\u00a0expecte API traffic for the\u00a0tool Dashboard for the get-analyt cloud\u00a0func Dashboard for the function The\u00a0Soluti I began to ponder the importance of all the things I didn\u2019t know about databases, and what DataStore might be good and bad at doing. Scrolling through the documentat I could see google boasting of super quick writes, but not super quick reads. I\u2019d already seen how many API calls were being made to the Cloud DataStore API and knew I\u2019d probably have to redesign part of the\u00a0tool. Idea 1: Use global\u00a0obj I attempted a few easy wins, mostly using the idea that if an instance of a function was invoked multiple times before being powered-do then global objects would still be available in\u00a0memory. If I put the data collected from the DataStore into a global object then I could check for its existence in subsequent function calls. This would save a lot of API calls and likely remove the largest bottleneck saving my readers 10+ seconds of watching a For whatever reason, this didn\u2019t work. Even if it had, the tool could still be vastly improved by taking a different approach that would be even faster and also reduce costs. I\u2019d like to have this tool running indefinite so reducing daily costs to an absolute minimum is\u00a0importa Idea 2: Store the results themselves in the\u00a0databa It was obviously inefficien to repeat the same calculatio multiple times. A good long-term solution would require aggregatin the data periodical and then fetching and serving these aggregated data to the\u00a0client I tried putting the JSON into the DataStore using a different key, but ran into errors about the data for each entity being too large. Even if I split the aggregated data into multiple component parts it would still be too large, and would grow over time. I guess DataStore isn\u2019t meant to be used like\u00a0this. I probably could have pursued this idea a bit further, but I didn\u2019t want to change the structure of the JSON blob served to client. If I did change it then I\u2019d need to rewrite the client side JavaScript as\u00a0well. Client side work is faster than back-end, but writing JavaScript is fiddly compared to Python in my opinion. There\u2019s always multiple ways of doing a thing, and several versions of an API, so googling a solution isn\u2019t as simple as for\u00a0Python Idea 3: Forget DataStore, use\u00a0bucket Final idea - store the results as a JSON blob in a Storage bucket and point the client at the bucket instead of the Turns out this is a super fast and efficient solution. now loads in less than half a second, and the only variable costs are egress on the bucket, which will be much smaller than the comparable costs of running a The computatio expense of calculatin the analytical results is fixed and decoupled from the number of page views using the Every few minutes Cloud Scheduler targets a Pub/Sub\u00a0to The topic triggers a The Cloud Function then: Queries the DataStore and collects the\u00a0data. Calculates the Generates a JSON blob containing the\u00a0result Pushes the JSON to a storage bucket which is available to a\u00a0client. The aggregated results for days other than the current day are still needlessly recalculat - once midnight rolls around the results are clearly not going to keep on\u00a0changin Instead of having one JSON blob containing data for all the last 30 days, I could have a blob for each day (or perhaps each week). This would reduce the amount of data extracted from the DataStore. This would reduce costs and"},{"title":"Edward Hopper\u2019s\u00a0Paintings","category":"snippet","url":"edward-hopper-s-paintings.html","date":"30 June 2021","tags":"art, painting, photography ","body":"archive"},{"title":"Some experiences can be taught, but some must be\u00a0lived","category":"snippet","url":"some-experience-can-be-taught-some-needs-to-be-lived.html","date":"29 June 2021","tags":"meta, advice ","body":"\u201cI have learnt that failure is my -"},{"title":"Georges St-Pierre Training\u00a0Meta","category":"snippet","url":"georges-st-pierre-training-meta.html","date":"29 June 2021","tags":"sport, meta, training ","body":"Contains too much conjecture at the start, but becomes At high levels of competitio the difference between \u201cgood\u201d and \u201cgreat\u201d is partly determined by how much pain you are willing to\u00a0experie There are great benefits from training in a Anderson Silva is a ballet\u00a0dan Conor McGregor does Israel Adesanya Georges St-Pierre archive"},{"title":"Validating CloudFlare\u00a0analytics","category":"Technical/Web","url":"validating-cloudflare-analytics.html","date":"29 June 2021","body":"Table of Contents A mystery Possible Reasons Comparison CloudFlare Analytics CloudFlare Web Analytics My own analytics tool A mystery CloudFlare give me two different measures of how many people have visited my website, one using their Analytics product and other from their new Web Analytics product. I get very different results for page views and number of visitors. I also get a third distinct set of results from my own analytics tool. Possible Reasons The reasons for this don\u2019t seem to be explained anywhere obvious, but I think it could be caused by ad blockers preventing the JavaScript used for the Web Analytics product. The normal, (not Web) Analytics product might derive its results from server side events, which would catch everything including bots and RSS clients, and be unaffected by ad blockers. Most of my visitors read the technical articles and therefore the audience is probably very technical and likely to be using an ad blocker. If this is the case then one way to test my hypothesis would be to write some articles that appeal to a non-techni audience who are less likely to use an ad blocker. In this case I would expect the two analytics methods to agree more closely. Comparison I made some screen shots at 11pm on June 29\^{th}\\ and compared the results from the CloudFlare Analytics, CloudFlare Web Analytics, and my own analytics tool. CloudFlare (normal) analytics say I\u2019ve had 234 unique visitors. But their Web Analytics tool says I\u2019ve had 11 visitors. My own tool reports 12 unique visitors. Why are these results so different? Maybe one measure might be including bots and another might only be trying to report real people using normal browsers, but the difference seems too high for that.1 I\u2019d also expect real usage to fall when its night in the countries I get most traffic from, which I don\u2019t see. Perhaps the difference is caused by 95% of my readers are using an ad blocker. My own analytics tool can\u2019t give results from a rolling 24 hour window, it only groups data by day. Therefore I recorded the values at 11pm, which should be close enough. My simple method of logging IP addresses when a page is loaded and counting the unique IP addresses each day says that I\u2019ve had 12 unique users. Much closer to the CloudFlare analytics beta result, but I wouldn\u2019t expect my bespoke tool to be blocked by an Ad Blocker. If it were as simple as concluding that my own results agree with the CloudFlare analytics beta then that might be enough. But they only agree on this particular metric. I\u2019ve logged 47 page views today using my own tool but the CloudFlare Analytics beta reports only 11 page views2. Please let me know on twitter if you have any ideas! CloudFlare Analytics CloudFlare Web Analytics My own analytics tool if { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.wi CloudFlare do tell me they\u2019ve blocked 199 attacks in the last month, but I don\u2019t think this explains the difference \u21a9With an average load time of 90ms - pretty snappy if it can be believed. \u21a9"},{"title":"Proverbs\u00a016","category":"Non-technical/Journal","url":"proverbs-16.html","date":"25 June 2021","tags":"books, bible, wisdom ","body":"The first 9 verses seem to loosely be about plans and the motivation for various actions. The wisdom and purpose of God\u2019s actions are contrasted with the motivation of human plans. The first proverb is a bit of a riddle. \\(^{1}\Th plans of the heart belong to man, but the answer of the tongue is from the Lord. I get the first part, but why would a spoken answer generally be from God? Is it that even if you are wise, and fear God, your plans are still your own but the way you speak about them is different because you fear God? \$$^{2}\$$Al the ways of a man are pure in his own eyes, but the Lord weighs the spirit. Don\u2019t be surprised that foolish, stupid or evil people think that their actions and decisions are upstanding and good. Understand that God says the spirit (motivatio or attitude) that produced the plan or the actions is what should be judged. I don\u2019t think that God would agree that \u201cthe ends justifies the means\u201d. \$$^{3}\$$Co your work to the Lord and your plans will be establishe A good proverb for fridge magnets. \$$^{4}\$$Th Lord has made everything for its purpose, even the wicked for the day of trouble. The mysteries around moral responsibi from free will and predestina are not unique to the New Testament. I can\u2019t think of anything useful to say in order to expand on this. I believe it\u2019s true, I don\u2019t really know what I would do differentl now that I\u2019ve read this proverb. I can trust God that he knows what he\u2019s doing and that he is good. I\u2019ll add it to my list of things I know about God without taking anything else off the list. who is arrogant1 in heart is an abominatio to the Lord, be assured he will not go unpunished God really doesn\u2019t like arrogance, which seems very similar to pride. This is troubling because society seems to have lost any appreciati for humility to the point where we no longer know how to talk about the virtues and vices of humility and pride. Pride is considered a virtue and is conflated with self-worth We lack the nuance required to reliably discern wisdom from folly. \$$^{6}\$$By steadfast love and faithfulne iniquity is atoned for, and by the fear of the Lord one turns away from evil. It is refreshing and pleasant to read about Steadfast love and faithfulne I don\u2019t hear many people talking about these virtues - love has been reduced or redefined as something that can be found or changed quickly. Faithfulne isn\u2019t celebrated or spoken about very much. Maybe because it isn\u2019t as dramatic as betrayal. Faithfulne and steadfast love are predicated on humility, which is another concept society seems silent about. The second half of the proverb is also profound. It tells me how I can turn away from evil. I know that it is easier to say I will change my ways than it is to actually change, consistent \u201cBeing good\u201d or \u201cturning away from evil\u201d is not nearly as easy or simple as a child thinks it is, and this proverb tells me how to do it. The last proverb of the previous chapter said that the fear of the lord is instructio in wisdom, and that humility comes before honor. \$$^{8}\$$Be is a little with righteousn than great revenues with injustice. In case there was any doubt, here it is. Don\u2019t compromise yourself in order to make more money. \$$^{9}\$$Th heart of a man plans his way, but the Lord establishe his steps. So I can make my own plans, but if my plans are to be successful or substantia I need the Lord to make my plans \u201cfirm\u201d or \u201cpermanent \$$^{10}\$$A oracle is on the lips of a king, his mouth does not sin in judgement. I don\u2019t know what this means. An oracle is \u201ca priest acting as a medium through whom advice or prophecy was sought from the gods.\u201d Kings have certainly sinned when making judgements \$$^{11}\$$A just balance and scales are the Lords, all the weights in the bag are his work. God loves justice, and the instrument or justice are ultimately his, and are from him. \$$^{12}\$$I is an abominatio to kings to do evil, for the throne is establishe by righteousn The kings in verses 12 and 10 are not like kings I have heard about. If 12 is true then perhaps 10 can be true. At a minimum, 12 and 13 provides some standard by which to judge kings. \$$^{16}\$$H much better to get wisdom than gold! To get understand is to be chosen rather than silver. One of the many bits of advice that could come from this would be to prioritise jobs with training and experience over jobs with higher salaries immediatel Especially in your twenties or when you are starting your career. You don\u2019t need lots of disposable income, you do need wisdom, training, experience perspectiv time with wise or experience people. \$$^{17}\$$T highway of the upright turns aside from evil, whoever guards his way preserves his life. Guarding your way - this is not a concept or expression I\u2019ve heard of before. I guess it could mean \u201cdefend the path your life could take\u201d, or \u201cthink about the consequenc (second order \$$^{18}\$$P goes before destructio and a haughty spirit before a fall. God really doesn\u2019t like pride, and also it leads to destructio This is not a coincidenc Haughty means \u201carrogantl superior or disdainful which is mostly a synonym for prideful. \$$^{19}\$$I is better to be of a lowly spirit with the poor than to divide the spoil with the proud. Avoid prideful people, avoid evil people. Proverbs seems really clear that you are supposed to socialise and spend time with people you want to be like. A lowly spirit is \u201clow is status, or humble\u201d. Better to hang out with poor people and be humble than to be rich and associate with prideful people. gives thought to a matter will discover good, and blessed is he who trusts in the Lord. More encouragem to trust God, and to be considerat 21 and 23 both say that it is wise to speak persuasive or encouragin If the words are well intended, then make them more effective by being persuasive \$$^{21}\$$T wise of heart is called discerning and sweetness of speech increases \$$^{23}\$$T heart of the wise makes his speech judicious3 and adds persuasive to his lips. words are like a honeycomb, sweetness to the soul and health to the body.Our words affect our bodies, and our souls. Gracious words are really valuable. Gracious means to be kind, courteous, or patient. Let it go, be gentle. \$$^{25}\$$T is a way that seems right to a man, but its end is the way to death. Don\u2019t be so confident in your own wisdom and judgement. It\u2019s not just that your judgement is a bit less good than God\u2019s, this proverb says its totally opposite. Expect to be doing things that look like the opposite of what someone might expect. Expect to do unintuitiv things. \$$^{26}\$$A workers appetite works for him, his mouth urges him on. True that. is slow to anger is better than the mighty, and he who rules his spirit than he who takes a city. Wow. This is high praise for self control. if { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.wi Arrogant: Having or revealing an exaggerate sense of ones own importance or abilities. \u21a9Establish Set up on a firm or permanent basis. \u21a9Judicious Having, showing or done with good judgement or sense. \u21a9"},{"title":"I would like to take some time to explore what it means to be\u00a0alive","category":"snippet","url":"i-would-like-to-take-some-time-to-explore-what-it-means-to-be-alive.html","date":"24 June 2021","tags":"life ","body":"."},{"title":"Proverbs\u00a015","category":"Non-technical/Journal","url":"proverbs-15.html","date":"24 June 2021","tags":"books, bible, wisdom ","body":"\$$^{1}\$$A soft answer turns away wrath, but a harsh word stirs up anger. A soft answer can diffuse a volatile situation, and speaking harshly leads to anger. \$$^{2}\$$Th tongue of the wise commends knowledge, but the mouths of fools poor out folly. Knowledge is to be commended. If someone speaks a lot of folly, they are a fool. Folly is \u201cfoolishne or a lack of good sense\u201d - it\u2019ll take discernmen to judge a lack of good sense. Find someone who commends acquiring knowledge, and learning. \$$^{3}\$$Th eyes of the Lord are in every place, keeping watch on the evil and the good. Don\u2019t mistake God\u2019s patience for indifferen know that your good works and faithfulne are seen. \$$^{4}\$$A gentle tongue is a tree of life, but perversene in it breaks the spirit. Yet again, this book teaches that our words are powerful and have great consequenc A gentle tongue, a soft answer, guarded words preserver life, good fruit, fountain of life, now a tree of life. It is unequivoca as is the inverse. \$$^{5}\$$A fool despises his father\u2019s instructio but whoever heeds reproof is prudent. I just had a thought that people younger than 10 would have a difficult time heeding or despising, and are too young to be considered wise or foolish. They are children. When proverbs refers to parents and children, I reckon its probably referring to either adult children and maybe teenagers. I have young kids so I\u2019m predispose to see parenting advice in that context. \$$^{7}\$$Th lips of the wise spread knowledge, not so the hearts of fools. What you say is important. \$$^{8}\$$Th sacrifice of the wicked is an abominatio to the Lord, but the prayers of the upright are acceptable to him. Integrity matters, and it seems that God is more concerned with motivation than impact. Better to pray quietly and honestly than to hypocritic do good works in public. \$$^{11}\$$S and Abaddon lie open before the Lord, how much more the hearts of the children of man! I am guessing that Sheol and Abaddon are some reference to a hellish place - if God can perceive what happens in a completely godless place that is far from him, it is going to be trivial to perceive the thoughts and motivation of your heart. He can read us like an open book. \$$^{12}\$$A scoffer does not like to be reproved, he will not go to the wise. Reproof is almost as strong of a theme as words. Reproof is part of growing up and becoming wise. You must be reproved if you are to learn and become wise. If I do not reprove my children then I am negligent. If you want to avoid reproof or do not accept it then you are a fool. But remember, these are proverbs - they are generally true, most of the time. There will be situations where you parent gives bad advice and you\u2019d be wise to ignore it. But its the exception, not the rule. If everyone else is the problem, then the problem is certainly you. \$$^{13}\$$A glad heart makes a cheerful face, but by sorrow the spirit is crushed. This proverb connection emotions, spirit and the physical body. It says they are linked. If you are happy then you will look happy, and vice versa. If you carry around stress and tension then it\u2019s going to change how you look. Don\u2019t be sad indefinite it will crush you. Grieve, and grow, and move on. \$$^{14}\$$T heart of him who has understand seeks knowledge, but the mouths of fools feed on folly. This isn\u2019t talking about the consequenc of your words, it says that if you have understand then your heart will desire knowledge. If you feed on folly, which is like consuming things that are foolish, then you are a fool. \$$^{15}\$$A the days of the afflicted are evil, but the cheerful of heart has a continual feast. A pessimist and an optimist could experience the same event and become respective more pessimisti and more optimistic As much as you are able, choose to have a cheerful heart. If you are afflicted, then don\u2019t give up hope. Affliction doesn\u2019t mean you did something wrong or lack wisdom. is a little with the fear of the Lord, than great treasure and trouble with it. So the fear of the Lord leads to the avoidance of trouble.. and it is better to live peaceably than deal with trouble. is a dinner of herbs where love is than a fattened ox and hatred with it. Better to eat only garnishes, or bait, with people who love you than fine dining with people who do not. \$$^{18}\$$A hot-temper man stirs up strife but he who is slow to anger quiets contention Don\u2019t lose your temper. Blessed are the peace-make counsel plans fail, but with many advisers they succeed. You\u2019re not supposed to know everything and be completely independen You are supposed to ask for help, weigh the advice, and deliberate look for wise people to be friends with. \$$^{23}\$$T make an apt1 answer is a joy to a man, and a word in season, how good it is! It is fun to say something apt? It is a blessing to receive a bit of well-timed advice. Proverbs talks a lot about words and mouths. For most of history, books have been rare and literacy was not widespread Therefore most knowledge transfer would occur by speaking. Maybe it still does today because it feels like it requires less effort than reading. I think the same principles can be applied to reading and writing as for hearing and speaking. \$$^{25}\$$T Lord tears down the house of the proud but maintains the widow\u2019s boundaries The boundaries are the edges of the land owned by the widow. God really doesn\u2019t like proud people, and is soft-heart towards the vulnerable \$$^{27}\$$H who is greedy for unjust gain troubles his own household, but he who hates bribes will live. Don\u2019t be a fool, don\u2019t be bribed - directly or indirectly Love life, look after your family, and hate bribes. Loving one thing means you hate other things. If you\u2019re building a business, design for Segregatio of Duties. \$$^{28}\$$T heart of the righteous ponders how to answer but the mouth of the wicked pours out evil things. It is good, and Godly, to think about your answer before you say it. \$$^{33}\$$T fear of the Lord is instructio in wisdom, and humility comes before honor. Ok great - I have a definition for what \u201cfear of the Lord\u201d means. (And it\u2019s a reliable definition too.) I still don\u2019t understand why this is what it means, but at least I know what it means. They why question is less foundation that the what. Learn humility, despise pride. Seek wisdom. if { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.wi Apt: Appropriat or suitable in the circumstan \u21a9"},{"title":"\u201cBelieve half of what you see and now\u2019t of what you\u00a0hear.\u201d","category":"snippet","url":"believe-half-.html","date":"24 June 2021","tags":"quote ","body":"source"},{"title":"Django for Startup\u00a0Founders","category":"snippet","url":"django-for-startup-founders.html","date":"23 June 2021","tags":"django, saas, startups, python ","body":"Better read this article archive"},{"title":"Building my own web\u00a0analytics","category":"Technical/Web","url":"building-my-own-site-analytics.html","date":"22 June 2021","tags":"cloud-functions, data ","body":"I\u2019ve built a simple client-sid website analytics tool for this site, you can see it at /analytics It has the following metrics: Page views per day, Unique IP addresses per day Views per page per day. This article eventually made it to the front page of Hacker News, which resulted in a lot of extra traffic and an opportunit to see how the tool performed under a much heavier load. I wrote about the affects of this and subsequent design changes here. I compare the different results from CloudFlare Analytics, CloudFlare Web Analytics and my own tool in this follow-up article. Motivation Google Analytics Google Analytics felt like overkill. It has so many data-point that the useful metrics are obscured. I also like this site to load quickly and GA makes it slower. CloudFlare Analytics I\u2019ve also tried CloudFlare Analytics. It\u2019s a lot simpler than GA and better suits my use case, but I don\u2019t think its accurate. Design Considerat The analytics should be easy to access and easy to understand I know from my work visualizin data and building dashboards that the metrics presented will alter the users perception of the underlying reality. The way that someone thinks about their impact on a business, the value they\u2019ve produced, or the dynamics of the underlying system (a product\u2019s quality, site performanc growth, etc) is influenced by the design decisions I make, such as which metrics are available, how easy they are to access, or which metrics are above the fold. If I present a particular metric as if its important, it will be difficult for someone who uses the dashboard to resist this implied message. They\u2019ll eventually consider the metric as a Key Indicator of some kind. For these reasons I wanted to see only the most important metrics about my website, and I wanted to see them in a simple way without distractio The only metrics I\u2019m interested in are: How many people are reading my site What are they reading How much are they reading. I\u2019d like to be able to infer whether I have a few people who read a lot, or a lot of people who read a little. (Or, as is the case, a few people who read a little.) Method Motivation The main reason for making my own analytics tool it because its a fun challenge with an obvious and useful result. Building it required connecting a few technologi - Serverless Computing (Cloud Functions on GCP), NoSQL databases (DataStore JavaScript HTTP headers. Assumption I\u2019m assuming that unique IP addresses is a good enough proxy for unique readers, even though I\u2019m not considerin crawlers, bots, or RSS subscriber . Technique The analytics \u201cengine\u201d works by consuming a request that is sent by the client each time a page is loaded. The request is parsed by a Cloud Function on GCP which extracts the page URL and the IP address. This is then recorded in a DataStore database along with the current date and time. Viewing the analytics is as simple (and as complicate as making a request to the database, parsing the data and visualizin it convenient For example, group the data by days and count the distinct IP Addresses to figure out how many people are visiting each day. This is achieved by making a request to another Cloud Function that returns a response with a JSON payload. It\u2019s not a perfect solution, there are edge cases I\u2019m not considerin I expect it to be mostly right and good enough for my purposes. It didn\u2019t take much effort and it was a fun mini project. The hardest part was figuring out chart.js, the slowest part was iterating on the Cloud Functions. Mocking Cloud Functions I haven\u2019t figured out how to easily test cloud functions locally - it would require setting up a NoSQL database and mocking Flask requests and responses. Instead of doing that, I watched Peaky Blinders for a couple of minutes whilst each new version of the Cloud Function was deploying. Improvemen Eventually I\u2019ll want to group the metrics by week or month I expect. It\u2019ll be a good way of learning and playing with cloud technologi and JavaScript Unless someone decides to spam the site, I expect the costs to be less than \u20ac1/month. This site is hosted using CloudFlare so I suppose I could setup some page rules to prevent malicious traffic3 . Tasks for later Make load faster - latency is caused by the Cloud Function initialisi Short of paying actual money for always-on resources I can\u2019t see a way to reduce this. However it\u2019s only an issue if you are the first person to view the page in the last ~10 minutes - this blog post explains whj. Add loading spinners - I used the same snippets as in my Machine Vision demo. Group data by weeks or months as well as day. Identify bots and search engines - the analytics requires JavaScript to be running so I think some types of non-human activity is already filtered. How can I do this? Aggregate the data (once per day) in a Cloud Function instead of repeatedly in the browser. Understand why the DataStore API is called multiple times for a single fetch. Questions I\u2019d be interested to know if there is a way to track RSS subscriber I know that the usual method is to inspect server logs, but this site is hosted on GitHub pages so I don\u2019t think this is possible. To what extent does requiring JavaScript in order to log a page view filter out bots and crawlers? I\u2019ve used the chart.js library because its reasonably fast and lightweigh My preferred library would be Plotly if it could be responsive and fast even if there are >10 charts to render. Has plotly.js improved recently to the point where it wouldn\u2019t cause a browser to lag if multiple plots are being rendered? Finally, it occurs to me that I could make an analytics widget for my desktop using \u00dcbersicht. It could show page views for the current day perhaps. I\u2019ve made a couple of widgets before [1, 2] which were written in CoffeeScri but the newer widgets are written in React, so I guess this is an opportunit to learn4 . Writing the \u201cTime Since\u201d (my daughters birth) and \u201cTime Until\u201d (my next accounting exam5 ) widgets were my first ever taste of CSS, HTML and JavaScript The first ever article on this blog was about the \u201cTime Since\u201d widget. CoffeeScri and Ubersicht were just about simple enough for me to learn by trial and error, copying someone else\u2019s code and changing it bit by bit until I had what I want. Site AnalyticsI Google Analytics it can be fun clicking around on all the things and seeing lots of options, but its not really useful once the novelty has worn off. \u21a9I think this might be quite wrong, but I don\u2019t know why. \u21a9The page is now rate limited to 5 requests per minute per IP address. \u21a9Done! My desktop now looks like this: \u21a9I failed the exam because I\u2019d been working on Ry\u2019s Git Tutorial instead. \u21a9"},{"title":"Alfie\u00a0Solomons","category":"snippet","url":"alfie-solomons.html","date":"22 June 2021","tags":"movie, youtube, peaky-blinders, humanity ","body":"Alfie Solomons - Where the light comes in. Scenes showing the character of Alfie Solomons from movie"},{"title":"Bifurcation\u00a0Theory","category":"snippet","url":"bifurcation-theory.html","date":"17 June 2021","tags":"math, chaos ","body":"Rabbits, fluid convection the Mandelbrot set and lots of others things too. Also known as The video shows how the Feigenbaum Constant is defined. It\u2019s a fundamenta constant I hadn\u2019t come across before -\u00a04.6692.. archive"},{"title":"Apple\u2019s iCloud+ \u201cVPN\u201d","category":"snippet","url":"apple-s-icloud-vpn-.html","date":"16 June 2021","tags":"apple, onion, vpn, icloud ","body":"article \u201cAn Apple onion router. The routing uses two hops; Apple provides the first, and independen third parties (not yet specified) provide the\u00a0second \u201cIn one move, Apple has taken onion routing from a specialize tool for hackers to something that will be in daily\u00a0use."},{"title":"Rich","category":"snippet","url":"rich.html","date":"16 June 2021","tags":"python, console, shell ","body":"python -m rich for a\u00a0demo Cool Python module to handle terminal output with debugging and logging features. It can even record stack trace errors to\u00a0html. repo demo\u00a0video"},{"title":"Practice","category":"snippet","url":"practice.html","date":"16 June 2021","tags":"proverb ","body":"An amateur practices until they can play it correctly, a profession practices until they can\u2019t play"},{"title":"Vim\u00a0Sneak","category":"snippet","url":"vim-sneak.html","date":"15 June 2021","tags":"vim, plugin ","body":"Invoked with s followed by 2\u00a0chars. S F, f, T, and t are enabled to work across\u00a0lin Jump back with ; or , to go to next/previ 5sxy searches for the next instance of xy within 5\u00a0lines. 3dzqt delete up to the third instance of qt. repo"},{"title":"Design\u00a0Patterns","category":"Technical/Engineering","url":"design-patterns.html","date":"15 June 2021","tags":"abstractions, meta, software-engineering ","body":"Design patterns are generalize abstractio that solve common problems and help engineers create complex code reliably and quickly. I first heard about design patterns from Aaron Maxwell in his Powerful Python newsletter and made some notes in an Then the YouTube algorithm put the following video on my front page, and down the rabbit hole I\u00a0went. Take a look at the Borg These are some Java SourceMaki fair repo (and my fork) Some pdf resources, including the GoF\u00a0book: All the GoF\u00a0patter c logica GoF\u00a0catalo securing GoF Wikipedia page about GoF\u00a0book."},{"title":"Proverbs\u00a014","category":"Non-technical/Journal","url":"proverbs-14.html","date":"14 June 2021","tags":"books, bible, wisdom ","body":"\$$^{1}\$$Th wisest of women builds her house, but folly with her own hands tears it down. This chapter begins with a proverb about women! And not by juxtaposin men either. It\u2019s important that I remember that wisdom - the desirable quality that so much of proverbs is about - is personifie as a woman, like in chapter 9:1, Wisdom has built her house, she has hewn1 her seven pillars. walks in uprightnes fears the Lord, but he who is devious in his ways despises him. Fear is contrasted with despising, so I don\u2019t think this is supposed to fear of violence or victimisat but more like a feeling of reverence and profound respect. The Lord loves me, and in the previous chapter it says that parents discipline their children. \$$^{4}\$$Wh there are no oxen, the manger is clean, but abundant crops come by the strength of the ox. This is really interestin the point feels modern. The rural imagery is unusual (to me) but otherwise it feels like something you might find on Instagram. What\u2019s the lesson? Progress is messy, or doing work creates waste, don\u2019t fret about the cleanlines of your manger if you want to have a productive farm.. \$$^{5}\$$A faithful witness does not lie, but a false witness breathes out lies. It\u2019s really important - don\u2019t lie. This is a strong recurring message. \$$^{6}\$$A scoffer seeks wisdom in vain, but knowledge is easy for a man of understand What does he understand That scoffing is harmful to the scoffer, and what else? \$$^{7}\$$Le the presence of a fool, for there you do not meet words of knowledge. It would be good to have a clearer understand of the difference between understand and knowledge. This proverb is very similar to modern sayings like \u201cYou become like the 5 people you spend most time with\u201d. What are the marks of a fool? How do you know if you\u2019re hanging out with a foolish person? They talk a lot, without thinking about their words. They are quick to become angry or emotional. They are not diligent, or consistent hard working. They don\u2019t plan ahead (filling their barns in summer). \$$^{8}\$$Th wisdom of the prudent is to discern his way, but the folly of fools is deceiving. It\u2019s wise, and prudent, to discern a course of action, or series of events and decisions. Discern means \u201crecognise or find out\u201d, I tend to use it to \u201clook closely for additional clues about what might happen\u201d. \u201cPrudent\u201d means to \u201cact or demonstrat care and thought for the future\u201d. \$$^{10}\$$T heart knows its own bitterness and no stranger shares its joy. This, I think, makes total sense to an old person and seems mysterious to young people. When I first read it I thought it said something like \u201cno-one shares its joy\u201d, but it doesn\u2019t. Only \u201cstranger\u201d Family and close friends can share joy, but they still won\u2019t be able to know your bitterness I\u2019d recommend not holding onto bitterness and finding out how its possible to forgive because of the work that Jesus completed. \$$^{11}\$$T house of the wicked will be destroyed, but the tent of the upright will flourish. Again, don\u2019t be wicked, it will go badly for you. If you are upright and living in a humble, fragile, vulnerable tent you can still flourish. \$$^{12}\$$T is a way that seems right to a man, but it\u2019s end is the way to death.This book doesn\u2019t pull any punches. It\u2019s an alarming assertion that someone can think they are doing things \u201cright\u201d but are in fact heading towards death. Don\u2019t rely on your own understand figure out what God thinks about a thing. \$$^{13}\$$E in laughter the heart may ache, and the end of joy may be grief.At the moment I don\u2019t have much experience of grief, but I inevitably will. I know there is sorrow so profound that it becomes physical as well as emotional, and regret that changes what it means to be alive. I guess one of the things this proverb reveals is that its ok to laugh whilst experienci heart ache, and its natural to feel happy and sad (laughing and heart-achi at the same time. It\u2019s part of being alive emotionall and shouldn\u2019t be considered weird or broken. 14 and 15 reiterate themes I\u2019ve noted previously \$$^{16}\$$O who is wise is cautious and turns away from evil, but a fool is reckless and careless. It is wise to not do something, even if you could. You don\u2019t have to do/see/vis all the things, even if you could. It\u2019s ok to risk erring on the side of cautious, because if you are not cautious, you are foolish. I guess that if you are discerning then you can perceive more clearly and what was originally partially known and therefore risky becomes less risky because there are fewer unknowns. Instead of \u201cmaybe its wrong\u201d it can become \u201cits almost definitely wrong, or right\u201d \$$^{17}\$$A man of quick temper acts foolishly, and a man of evil devices is hated. Don\u2019t lose your temper quickly. Losing it slowly is often more difficult. \$$^{18}\$$T simple inherit folly, but the prudent are crowned with knowledge. You\u2019re supposed to aspire to prudence (and more generally, wisdom). You can\u2019t really opt out of this, because if you\u2019re not wise then you\u2019re a fool. If you\u2019re not prudent, you are (too) simple. And bad things happen to fools. \$$^{19}\$$T evil bow down before the good, the wicked at the gates of the righteous. It\u2019s surprising to read such bold and simple confidence that justice would prevail. No ifs or maybes or conditions just a simple resolution at the end. \$$^{20}\$$T poor is disliked even by his neighbour, but the rich has many friends. Timeless pragmatism This proverb is an observatio not a commendati And look at the next proverb. despises his neighbour is a sinner, but the blessed is he who is generous to the poor.. Be generous, not just theoretica or in your thoughts, but also in your actions and your money (see 23). Don\u2019t despise people. Who is your neighbour? who devise good meet steadfast love and faithfulne Steadfast love and faithfulne \$$^{23}\$$I all toil there is profit, but mere talk leads only to poverty. It\u2019s not enough to only talk about loving your neighbour. And don\u2019t be convinced that any hard work you do it totally wasted, apparently it is not. \$$^{24}\$$T crown of the wise is their wealth, but the folly of fools brings folly. Really? Does wealth here mean something more or different to financial wealth? (I think I asked this in a previous proverb.) Not all wise people are rich, this is certain. And many biblical heroes were financiall impoverish Is it just a wide \$$^{25}\$$A truthful witness saves lives.. Literally. Though probably without knowing exactly which lives, and at what moment. Investors refer to this as second order consequenc \$$^{26}\$$I the fear of the Lord one has strong confidence and his children will have refuge. The best that I can understand this is to read it the same as if it said \u201cIn the Lord one has strong confidence What does this proverb mean that my shortened simplified version doesn\u2019t? \$$^{27}\$$T fear of the Lord is a fountain of life, that one may turn away from the snares of death. Turning away is like physically repenting. A fountain of life is a source of life, and probably health and healing. It is contrasted with death traps. It would be interestin to consider the relative importance of feelings and actions in this book. Is it that feelings lead to actions, but actions are what makes the difference - the judgement between wisdom and foolishnes love and hatred, depends (only) on your actions? \$$^{28}\$$I a multitude of people is the glory of a king, but without people a prince is ruined.If a king doesn\u2019t have any subjects, what is he king of? If leaders are not followed by any people, they cannot claim to be leaders. is slow to anger has great understand but he who has a hasty temper exalts folly.If you have a quick temper then you are responsibl for it, and you are endorsing foolishnes Being slow to anger is a sign of wisdom. \$$^{30}\$$A tranquil heart gives life to the flesh, but envy makes the bones rot. Envy is apparently really bad. Tranquilit is freedom from disturbanc or being calm. It\u2019s good for your health. oppresses a poor man insults his maker, but he who is generous to the needy honors him. The poor mater, and have dignity, and are worthy of respect, because we are all made. if { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.wi Hewn: chop or cut with an const \u21a9"},{"title":"Coding exercise for a technical\u00a0interview","category":"Technical/Data","url":"coding-exercise.html","date":"14 June 2021","tags":"trading, finance ","body":"This notebook is a technical exercise I worked on as part of an interview for a crypto trading firm. The exercises involve building simplified interfaces to parse order book data and calculate various quantities and I find technical interviews that involve live coding exercises to be really useful and really stressful. Live coding definitely triggers \u201cperforman anxiety\u201d for me (when I was a kid I really hated presenting or playing an instrument in front of anyone, even teachers) - I felt self-counc and in this case I needed to use VSCode instead of my familiar Vim+Tmux setup. Consequent progress was really slow and bumpy. Tragically I got really muddled trying to parse some json after the initial API\u00a0reques Whilst thinking about the problem and what code to write, I also needed to think about the\u00a0follow How to communicat my\u00a0thought How to optimize my output for an interview context (it\u2019s not a real problem involving tests, edge cases, scalabilit or How to use VSCode (and a mouse or\u00a0trackpa These additional considerat resulted in me writing code that got the job done.. but slowly. The code wasn\u2019t\u00a0gre I was curious how much faster I\u2019d be (and how much easier the exercise would seem) if I treated the exercise as a \u201ctake home\u201d exercise instead of \u201clive coding\u201d. Here are the results - it took a couple of hours, the ideas flowed more easily, and I could remember methods more accurately I wish I didn\u2019t (still) get so Despite this, I think live coding interviews are a great way of assessing me - they get under my skin and show my worst sides as well as my best. If I\u2019m going to work in a high paced and demanding role then the interview should also have Part 1 The Interface\u00b6 Build an abstractio that, given a pair as an argument (ethusd in our case) fetches the latest orderbook and prints it.\u00b6 In\u00a0[1]: import requests import operator import pprint import pandas as pd import time pp = In\u00a0[2]: def get_data(u r = data = r.json() return In\u00a0[3]: 2. Add some functional that takes a side (bid or ask) and a price p as arguments, and returns the total volume available in the order book at p.\u00b6 In\u00a0[4]: def price): data = # side should be either \"asks\" or \"bids\" side = clean_data = [] for i, j in price_data = volume = volume)) # find all the items where first item is less than or equal to price index = 0 volume = 0 operate = { 'bids': operator.g 'asks': operator.l } op = for i, j in if price): index += 1 volume += else: break return volume In\u00a0[5]: 2470) Out[5]: In\u00a0[6]: 2480) Out[6]: 0 3. Now add some (or modifies existing) functional that takes a percentage and a side as arguments, and returns the volume available between the best price for that side, and that price +/- the percentage In\u00a0[7]: def percent): data = # side should be either \"asks\" or \"bids\" side = clean_data = [] for i, j in price_data = volume = volume)) # find all the items where first item is less than or equal to price index = 0 volume = 0 starting_p = # if side == 'asks': percent *= -1 op = operator.a if side == \"ask\" else operator.s limit_pric = * percent)) # operate = { 'bids': operator.g 'asks': operator.l } op = for i, j in if limit_pric index += 1 volume += else: break # return volume In\u00a0[8]: 0.01) Out[8]: 4. Can you visualize the order book?\u00b6 In\u00a0[9]: asks = bids = In\u00a0[10]: asks = asks.colum = ['price', 'volume', 'timestamp asks['volu = = asks.head( Out[10]: .dataframe tbody tr { middle; } .dataframe tbody tr th { top; } .dataframe thead th { text-align right; } price volume timestamp cumulative 0 2480.36000 0.591 1623670712 0.591 1 2480.48000 0.006 1623670590 0.597 2 2480.93000 0.007 1623670457 0.604 3 2480.96000 0.005 1623670684 0.609 4 2480.99000 0.172 1623670589 0.781 In\u00a0[11]: bids = bids.colum = ['price', 'volume', 'timestamp bids['volu = = bids.head( Out[11]: .dataframe tbody tr { middle; } .dataframe tbody tr th { top; } .dataframe thead th { text-align right; } price volume timestamp cumulative 0 2480.35000 21.903 1623670721 21.903 1 2480.34000 7.930 1623670720 29.833 2 2480.25000 0.250 1623670709 30.083 3 2480.22000 8.061 1623670715 38.144 4 2480.21000 16.113 1623670720 54.257 In\u00a0[12]: ob = In\u00a0[13]: inplace=Tr inplace=Tr In\u00a0[14]: ax = xticks = xticklabel = [l.get_tex for l in Part 2: Time-serie Build an abstractio that takes the A\u00a0pair A unit of time (t) that can be either \u201cseconds\u201d, \u201cminutes\u201d or\u00a0\u201chours\u201d A number of time units\u00a0(n) Given these arguments, it should fetch and store the order book for the pair every n t. For example: if t = \u201csecond\u201d and n = 3, it should fetch the order book every second for a total of 3\u00a0times. In\u00a0[15]: def tunit, n): base_url = pair_mod = results = {} timeunits = { \"s\": 1, \"m\": 60*1, \"ms\": 0.001, } realtime = for i in range(n): snapshot {i+1} of {n}\") r = data = r.json() results[i] = # had to modify by multiplyin by n so that api is more likely to return different data set. print(\"don return results In\u00a0[16]: results = \"s\", 3) getting snapshot 1 of 3 getting snapshot 2 of 3 getting snapshot 3 of 3 done! We see that the hash of the api response is sometimes the same, so we would expect there to be no volume change between these In\u00a0[17]: #results Now extend your code to compare the order books you\u2019ve fetched to each other. Given the name of a pair, n, t and a price p, determine how much the total volume available at p has changed between each \u201cframe\u201d, and between the first and last \u201cframe\u201d.\u00b6 In\u00a0[18]: # redefine \"calc_volu function from above to make data an input param def ob_side, price): # side should be either \"asks\" or \"bids\" side = data[ob_si clean_data = [] for i, j in price_data = volume = volume)) # find all the items where first item is less than or equal to price index = 0 volume = 0 operate = { 'bids': operator.g 'asks': operator.l } op = for i, j in if price): index += 1 volume += else: break return volume In\u00a0[19]: def ob_side, price): snap_vol = [] for key in results: result = ob_side, price) total_chan = - snap_vol[0 = - snap_vol[i for i, j in if i < print(f\"vo in each snapshot: {snap_vol} print(f\"to (diff between last and first): In\u00a0[20]: results = \"s\", 3) getting snapshot 1 of 3 getting snapshot 2 of 3 getting snapshot 3 of 3 done! In\u00a0[21]: #results In\u00a0[23]: \"bids\", 2470) vol in each snapshot: incrementa [-6, 0] total (diff between last and first): -6 Finally, can you think of an efficient way to get a \u201cdiff\u201d of two frames? For example, can you think of a way to determine at which price the volume changed the most?\u00b6 Quick\u00a0idea for each side of each order book, group the orders into buckets with a certain width, e.g. if the lowest ask is 2450, group all the asks from 2450 to 2451, then from 2451 to 2452, etc. Do this for both sides of each order book you\u00a0collec you then have multiple aggregated order books with the same\u00a0index compare the volumes at the same index across each\u00a0order need to force the lowest ask or max bid to be the same in each order book\u00a0snaps need to deal with large\u00a0spre comparison where one side has 0 volume should be marked as suspicious and investigat further to iterate a need to present negative volumes if { var mathjaxscr = = = = ? \"innerHTML : \"text\")] = + \" config: + \" TeX: { extensions { autoNumber 'AMS' } },\" + \" jax: + \" extensions + \" displayAli 'center',\" + \" displayInd '0em',\" + \" showMathMe true,\" + \" tex2jax: { \" + \" inlineMath [ ['','$'] ], \" + \" displayMat [ ['$$','$$' ],\" + \" true,\" + \" preview: 'TeX',\" + \" }, \" + \" 'HTML-CSS' { \" + \" linebreaks { automatic: true, width: '95% container' }, \" + \" styles: { .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'black ! important' }\" + \" } \" + \"}); \"; (document. || }"},{"title":"Axiom","category":"snippet","url":"axiom.html","date":"12 June 2021","tags":"math ","body":"A basic statement that is assumed to be true. E.g: \u201cA straight line can be drawn between any two\u00a0points archive"},{"title":"Foxes and\u00a0Hedgehogs","category":"snippet","url":"foxes-and-hedgehogs.html","date":"12 June 2021","tags":"meta, classification, thinking ","body":"wikipedia Hedgehogs know one big thing, Foxes know many\u00a0thing A classifica system or mental model for writers and\u00a0thinke"},{"title":"Pair programming using\u00a0Vim","category":"snippet","url":"pair-programming-using-vim-tmux-ssh.html","date":"12 June 2021","tags":"pair-programming, vim, tmux, ssh ","body":"blog\u00a0post"},{"title":"Man swallowed by\u00a0whale","category":"snippet","url":"man-swallowed-by-whale.html","date":"12 June 2021","tags":"whale ","body":"A lobster diver was swallowed by a humpback whale. Then it spit him out. Humpback whales don\u2019t have teeth, and have reduced forward vision when they open their mouths to\u00a0feed. article archive"},{"title":"Git LFS","category":"snippet","url":"git-lfs-2.html","date":"11 June 2021","tags":"git, lfs, github-pages ","body":"Key\u00a0comman git lfs install git lfs track \"**/*.mp4\" git lfs ls-files git lfs status track just updates the .gitattrib file. Commit the .gitattrib file with the tracking configurat before committing the large\u00a0file status or ls-files should show the large files in question before you push the commit that starts tracking the large\u00a0file"},{"title":"Your .bashrc doesn\u2019t have to be a\u00a0mess","category":"snippet","url":"your-bashrc-doesn-t-have-to-be-a-mess.html","date":"11 June 2021","tags":"bash, shell, zsh ","body":"Blog post demonstrat how to split a .bashrc file into \u201csubmodule and keep [[ -r ]] && . for file in do [[ -r$file ]] && . file done unset file"},{"title":"John Kelly finishing the 2017 Barkley\u00a0Marathons.","category":"snippet","url":"barkley-finisher-15-john-kelly.html","date":"10 June 2021","tags":"movie, running, barkley, youtube ","body":"The exhausted moments after running for almost 60 hours through movie"},{"title":"A Project of One\u2019s\u00a0Own","category":"snippet","url":"paul-graham-a-project-of-ones-own.html","date":"8 June 2021","tags":"paul-graham, meta, learning, school ","body":"An essay called A project of ones own by Paul\u00a0Graha being pushed into a task vs being\u00a0pull skating vs\u00a0walking"},{"title":"Proverbs\u00a013","category":"Non-technical/Journal","url":"proverbs-13.html","date":"8 June 2021","tags":"books, bible, proverbs, wisdom ","body":"\$$^{1}\$$A wise son hears his father\u2019s instructio but a scoffer does not listen to rebuke. The chapter opens the same way as the previous chapter - with a proverb about the importance of wisdom, knowledge or instructio As always, the proverb starts with the good example and then contrasts against it. \$$^{2}\$$Fr the fruit of his mouth a man eats what is good, but the desire of the treacherou is violence. Slightly weird imagery I think, my mouth produces fruit that I can then eat.. Weird. But I can see the principle - the fruit of my mouth is the words I speak, words are powerful for either good or evil. Speaking well will lead to good things that do what good food does - nourish, strengthen sustain. Treacherou people desire violence. So are people who want to do violent things likely to betray? Maybe. \$$^{3}\$$He who guards his mouth preserves his life, he who opens wide his lips comes to ruin. Choose your words carefully. \$$^{5}\$$Th righteous hates falsehood, but the wicked brings shame and disgrace. It\u2019s OK to be strongly opposed to be falsehoods - lies, manipulati duplicity. guards him whose way is blameless, but sin overthrows the wicked. I guess there are (at least) 2 ways of looking at this - the only way I can be blameless is by Jesus\u2019 imputed righteousn which also saves me literally from death. The alternativ is that I am not righteous and am overcome by sin. Alternativ and more prosaicall try to be blameless instead of wicked, and instead of creating trouble for yourself you\u2019ll find people bear with you for longer and are nicer to you. \$$^{7}\$$ One pretends to be rich, yet has nothing. Another pretends to be poor, yet has great wealth. (I prefer not to use semicolons I really like this proverb! The last 20 years have seen several degrees of financial wealth or lack, and the reality has often been very different to what I\u2019d expected. I often intuit that things must get easier, or nicer, or more fulfilling if only a certain problem was removed, or a certain something was bought, or recognised or achieved. The reality is a lot more complicate than that. Wealth isn\u2019t what I thought it was, I thought it was mostly financial. Now, I think its freedom. Freedom to be at peace, to be unburdened of my past and present, freedom to still have hope for the future, freedom to have some spare time, freedom to not be crushed by life and be a good dad and husband and friend. \$$^{8}\$$Th ransom of a man\u2019s life is his wealth, but a poor man hears no threat. oh! Incisive! This reminds me of \u201cno one ever really owns a fortune, it always seems to own them\u201d. There are various troubles and burdens you avoid by being poor, like choosing life instead of material excess. Lose your wealth and choose a rich life. \$$^{10}\$$B insolence1 comes nothing but strife, but with those who take advice is wisdom. Disrespect and rudeness never works, wisdom is acquired when you make a habit of asking for advice. gained hastily will dwindle, but whoever gathers little by little will increase it. I\u2019m surprised to read this - I didn\u2019t realise that the rate of change of ones wealth would likely effect how it can be sustained, and I wouldn\u2019t have guessed that this is some general principle that exists across millennia, cultures and geographie \$$^{12}\$$H deferred makes the heart sick, but a desire fulfilled is a tree of life. This one I\u2019ve heard before. I don\u2019t really understand it, though. I\u2019m not sure what a sick heart definitely is. It seems because we are told to keep on looking forward to and hoping for Jesus to complete his work saving us and bringing history to completion There is a lot of \u201cnow and not yet\u201d tensions in between the time of Jesus\u2019 resurrecti and second coming (I know so little about it that I don\u2019t particular want to mention it). So maybe a Christian\u2019 heart would be expected to be a little sick? Also, if a desire fulfilled is a tree of life then I guess I\u2019m desiring the wrong kind of things - most of my desires are chasing after the wind. I desire to eat and then I get hungry again, desire to graduate and then realise the work hasn\u2019t even begun, etc. The image of a Tree of life hasn\u2019t been used before I don\u2019t think, maybe it means something specific which I\u2019m unaware of. There was a tree of life in the garden of Eden\u2026 13 - 16: Don\u2019t ignore wisdom, don\u2019t be an idiot, be prudent, have good sense. Strong recurring themes. \$$^{16}\$$E prudent man acts with knowledge, but a fool flaunts his folly is interestin though - I think prudence is being used in a slightly different sense to what I\u2019m used to. This proverb is saying that one should be informed before acting. And because a fool rushes into action, it will be clear that the don\u2019t know what they are doing or talking about. \$$^{22}\$$ A good man lays up an inheritanc for his children\u2019s children, but the sinner\u2019s wealth is laid up for the righteous. I remember reading this when I was about 18 and thinking this was quite a burden - to leave an inheritanc for grandchild as well as children. Seventeen years later it makes a lot of sense - the inheritanc isn\u2019t primarily money, but wisdom, peace, security. The way that I parent my children will directly effect how they parent their own children, and in this way I will either bless or curse my grandchild If I damage my children, they will suffer and be less able to provide for their own children (emotional physically spirituall Our own childhoods have a large influence on our adulthoods and our ability to parent, so I should make sure my children have good childhoods which is not a trivial endeavour. 23 and 24 break the pattern of \u201cgood example, bad example.\u201d \$$^{23}\$$ The fallow2 ground of the poor would yield much food, but it is swept away through injustice. The missed opportunit are real. Poor people are not incapable of productive and fruitful work. It is injustice that prevents fruitfulne I find this proverb subtly provocativ and insightful \$$^{24}\$$ Whoever spares the rod hates his son, but he who loves him is diligent to discipline him. Parents who love their children do not enjoy causing them stress or discomfort but if they love them and are wise then they discipline diligently or consistent Apparently physical punishment is timeless? \$$^{25}\$$ The righteous has enough to satisfy his appetite, but the belly of the wicked suffers want. The last proverb in the chapter finishes with encouragem that it is better to be righteous than wicked, because it leads to satisfacti rather than want. The previous chapter also finished with encouragem specifical about righteousn Even though the book doesn\u2019t have headings it seems like it does have structure, and the author requires you to actually read the text closely in order to pull out meaning. Shocking. Clearly not optimised for engagement SEO or social media. if { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.wi Insolence: rude and disrespect behaviour\u201d \u21a9Fallow: ploughed and harrowed but left for a period without being sown in order to restore its fertility or to avoid surplus production \u21a9"},{"title":"Choose Boring\u00a0Technology","category":"Technical/Engineering","url":"choose-boring-technology.html","date":"7 June 2021","tags":"advice, management ","body":"I\u2019m surprised I haven\u2019t posted this before because the \u201cchoose boring tech\u201d article by Dan McKinley made a big impression when I first read it, about 3 years\u00a0ago. humorous slide show\u00a0versi Key takeaways, based on my memory of reading it a couple years\u00a0ago: Boring tech is that which is mature enough and that you know well enough that you are familiar with its shortcomin and that will let you build You get 3 innovation tokens for each project or stack. Choosing some new and exciting bit of tech requires spending 1 of those\u00a0toke A nice alternativ title for this is \u201cHow to be old, for young people\u201d, which is In addition to the advice in the article, I read somewhere else that the probabilit of something continuing to exist in future, and be actively maintained and supported, is approximat the amount of time that it has already existed, and been supported and maintained I\u2019m not even sure that makes sense, but it bears This should be on my imaginary list of \u201cthings I should read every couple of years\u201d. Actually, a few lists might be really useful. A list of python articles, a list of advice articles, etc. It would be similar to the books page, which is just a list of book\u00a0artic"},{"title":"An incomplete list of skills senior engineers\u00a0need","category":"Technical/Engineering","url":"an-incomplete-list-of-skills-senior-engineers-need.html","date":"7 June 2021","tags":"advice, management ","body":"This is copied from Camille Fournier\u2018s article on medium. I copied it rather than linked to it because who knows if and when Medium will change the URL or put the content behind a\u00a0paywall. Highlights are simply an impulsive \u201cI can think about this for the next few months.\u201d After that I might change\u00a0the An incomplete list of skills senior engineers need, beyond\u00a0cod For varying levels of seniority, from senior, to staff, and\u00a0beyond How to run a meeting, and no, being the person who talks the most in the meeting is not the same thing as running\u00a0it How to write a design doc, take feedback, and drive it to resolution in a reasonable period of\u00a0time How to mentor an early-care teammate, a mid-career engineer, a new manager who needs How to indulge a senior manager who wants to talk about technical stuff that they don\u2019t really understand without rolling your eyes or making them feel\u00a0stupi How to explain a technical concept behind closed doors to a senior person too embarrasse to openly admit that they don\u2019t understand How to influence another team to use your solution instead of writing their\u00a0own How to get another engineer to do something for you by asking for help in a way that makes them feel appreciate How to lead a project even though you don\u2019t manage any of the people working on the project How to get other engineers to listen to your ideas without making them How to listen to other engineers\u2019 ideas without How to give up your baby, that project that you built into something great, so you can do something\u00a0 How to teach another engineer to care about that thing you really care about (operation correctnes testing, code quality, performanc How to communicat project status How to convince management that they need to invest in a non-trivia How to build software while delivering incrementa value in the\u00a0proces How to craft a project proposal, socialize it, and get buy-in to execute\u00a0it How to repeat yourself enough that people start to listen How to pick your\u00a0battl How to help someone get\u00a0promot How to get informatio about what\u2019s really happening (how to gossip, how to\u00a0network How to find interestin work on your own, instead of waiting for someone to bring it to\u00a0you How to tell someone they\u2019re wrong without making them feel\u00a0asham How to take negative"},{"title":"Proverbs\u00a012","category":"Non-technical/Journal","url":"proverbs-12.html","date":"2 June 2021","tags":"bible, proverbs, wisdom ","body":"The chapter starts with \u201cWhoever loves discipline loves knowledge, but he who hates reproof is stupid.\u201d and ends with \u201cIn the path of righteousn is life, and in its pathway there is no death.\u201d The opening proverb is jarring to read, maybe because unlike \u201cwickednes or which are concepts I don\u2019t hear or think much about day-to-day \u201cdisciplin and \u201cknowledge are very familiar and part of contempora conversati So is stupidity. So this proverb What does it mean? If I love discipline then I am loving knowledge, and in this case I\u2019m confident the opposite is also true - If I avoid or dislike discipline then I avoid or dislike knowledge. Discipline is hard, I guess I need to lean into being discipline and remember\u00a0w Encouragin the proverb is immediatel followed by the reminder that \u201cA good man obtains favour from the Lord, but a man of evil devices he condemns\u201d, so make an effort, do good things, and obtain favour. That\u2019s a big reason to persevere \$$^4\$$\u201cAn excellent wife is the crown of her husband, but she who brings shame is like rottenness in his bones.\u201d If it were me I\u2019d leave off all the negative second parts of these proverbs, its abrasive. But also if it were me I doubt they\u2019d be Why is the pattern of all these proverbs \u201cgood thing, bad thing\u201d - to make the contrast stronger? To know why to do a thing and also why not to do the opposite? To make it more punchy? I guess if its abrasive you are more likely to remember it, and if you\u2019re confident that what you are saying is true then you want it to be\u00a0remembe Back to the proverb, this is a great proverb - I like it. It honors wives, though it does put them in the context of their husbands, and it also speaks to how valuable and precious a good wife is - crowns are precious and\u00a0rare. Empiricall it also feels thruthy - marriages are difficult and if either spouse can be considered \u201cexcellent then that is unusual and valuable. Relationsh can give a lot of life and health and happiness, and require a lot of effort and work. To be an excellent spouse is certainly worth desiring and celebratin The next few verses contrast wickedness and righteousn - don\u2019t be wicked, be\u00a0upright \$$^9\$$ \u201cBetter to be lowly and have a servant that to play the great man and lack bread\u201d. Don\u2019t put appearance before substance? Don\u2019t spend money on fancy cars or clothes if you wont have enough for decent food? Prioritize substance (truth?) over appearance and social pressures? Having a servant (or a domestic helper) would certainly be helpful, and of more tangible benefit than receiving shout-outs at \$$^{10}\$$ \u201cWhoever is righteous has regard for the life of his beast, but the mercy of the wicked is cruel.\u201d - animal welfare is\u00a0importa \$$^{11}\$$ Whoever works his land will have plenty of bread, but he who follows worthless pursuits lacks sense. 11 and 12 are both about enjoying the good consequenc of honest work, 11 advises against worthless pursuits, and 12 says that is a wicked to covet the profits of\u00a0evildoe Verses 13 - 19 are all about speaking and listening. Each proverb contrasts good and evil or wisdom 13 - your own dishonest words will become a trap for\u00a0you. 14 - you words and your works will come back to you - invest in them and dividends will be\u00a0returne 17, 18 - these contrast each other. Your words are powerful, speaking the truth is honest (even when it isn\u2019t simple, or if it makes a situation more complicate and being rash (acting without careful considerat of the consequenc can be as violent as wielding a sword. Wise words bring \$$^{15}\$$ Fools think they are right, and because of the contrastin pattern presumably do not listen to (or ask for) advice. This seems similar to the Dunning-Kr Effect Wise people know that they don\u2019t know everything they need to, ask for advice, and listen to\u00a0it. \$$^{16}\$$ I had to look up Vexation - it means \u201cthe state of being annoyed, frustrated or worried\u201d. Wise people can stay calm and ignore insults. If you get all fired up when someone insults you you\u2019re probably (generally speaking) being\u00a0fool \$$^{19}\$$ Truthful lips endure forever, but a lying tongue is but for a moment. Truth endures, lies do\u00a0not. Verses 20 - 29 have two themes - the benefits of consistent hard work, and the benefits of being truthful. As usual, each proverb is a comparison of right and wrong or wisdom \$$^{21}\$$ No ill befalls the righteous, but the wicked are filled with trouble. This is encouragin to read, and also highlights that proverbs highlight truthful patterns, but are not specific guarantees - Obviously righteous people will have trouble, this proverb isn\u2019t saying that life will be perfectly pleasant. As usual, it is a push towards doing good because of the benefits, a pull away from doing wrong because of the\u00a0damage \$$^{22}\$$ .. those who act faithfully are his delight. It is wonderful to read that (a good) God would delight in\u00a0people. \$$^{23}\$$ A prudent man conceals knowledge, but the heart of a fool proclaims knowledge. I find it surprising to read that concealing knowledge can be a desirable trait. I guess there are certain questions that must be answered before the answers to other questions can be understood and I know that for my young children I wouldn\u2019t answer particular questions that same as if they were\u00a0adult \$$^{24}\$$ The hand of the diligent will rule, while the slothful will be put to forced labor. Its ironic, and that those who are diligent and able and willing to labor end up in management or leadership positions, and that those who would avoid laborious work end up doing\u00a0it. \$$^{25}\$$ Anxiety in a man\u2019s heart weighs him down, but a good word makes him glad. I find it encouragin (again) to read that being weighed down by anxiety is normal, and I like that anxiety isn\u2019t described as a weakness or as foolishnes (I don\u2019t want to conjecture too much about what isn\u2019t written, but there\u2019s a long list of things that fools do and the consequenc of foolishnes and anxiety isn\u2019t on the list.) This proverb also emphasises and encourages the impact of good\u00a0words \$$^{28}\$$ In the path of righteousn is life, and in its pathway there is no death. How wonderful to know how to find life and avoid death. It\u2019s such a pleasing way to end a\u00a0chapter. if { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.wi"},{"title":"Proverbs\u00a011","category":"Non-technical/Journal","url":"proverbs-chapter-11.html","date":"1 June 2021","tags":"books, bible, proverbs, wisdom ","body":"Table of Contents Context Notes Examples Conclusion Context These are my notes from reading Proverbs, one of the books in the bible. Almost the entire book is a collection of proverbs - short sayings that are generally true. I presume there is some structure and themes in their arrangemen taking notes should help these become clearer and hopefully also help me remember and apply them\u00a0bette The book is split up into chapters, which are quite short sections of the book. It would probably take about a minute or two to read each one. The chapters are further split up into verses. In Proverbs each verse seems to be one\u00a0senten Numbering the text like this is useful because it lets you refer precisely to a part of the book. Proverbs does have a few headings, but they are too far apart to provide enough structure on their own, I\u00a0think. I\u2019ve read this book a couple of times before, and appreciate it even when I could feel that I was missing most of the wisdom in it. The book is quite easy to read - the sentences are short and the analogies seem simple\u00a0eno It\u2019s refreshing to read something that has existed for thousands of year, has withstood criticisms and feels approachab It\u2019s practical, despite being old and written in a completely different cultural context. I find Notes Each proverb contrasts justice and injustice using a variety of phrases and images. Pride and humility, integrity and crookednes righteousn In the context of these qualities, various situations are described: a false balance, disgrace, The proverbs are somewhat exaggerate which seems reasonable given that they are self contained single sentences tasked with defining and then resolving a problem. The imagery is clear, it does not rely on\u00a0subtlet The biggest themes are that honesty, integrity, righteousn are to be valued, and their opposites are to be avoided at all costs. Each of these qualities are in fact actions, not sentiments and the consequenc of these actions are reliable and consistent and inevitable The advice to \u201clove your neighbor like yourself\u201d is a good summary of many of the\u00a0prover Its clarity and confidence is encouragin Do good and good things will happen, do evil and the consequenc are inevitably bad for\u00a0you. Examples The fruit of the righteous is a tree of life, and whoever captures souls is wise - the first part sounds wonderful. Who wouldn\u2019t want to produce life? The second part is surprising What is the The desire of the righteous ends only in good, the expectatio of the wicked in wrath. - this is\u00a0encoura One gives freely, yet grows all the richer; another withholds what he should give, and only suffers want. - Like a gold ring a pig\u2019s snout is a beautiful woman without discretion - this doesn\u2019t seem to fit in with any proverbs around it. Is this the only mention of a woman in this section? Why is it important enough to be included? Does this imply that beauty is discreet? There is surely a big cultural gap between Amsterdam in 2021 and where ever this was first\u00a0writ A man who is kind benefits himself, but a cruel man hurts himself. - this is good to\u00a0know Conclusion Don\u2019t lie, cheat, steal or plot. Treat people like you yourself would want to be\u00a0treated Make an effort to do good and increase justice, and things will go well for\u00a0you. God delights in blameless people, and he abhors people with Beauty without discretion is an"},{"title":"Performance Optimizations for the shell\u00a0prompt","category":"snippet","url":"performance-targets.html","date":"28 May 2021","tags":"shell ","body":"Something should happen within 100ms of the users input in order to maintain a feeling If something happens within 50ms of the trigger event, it will feel Also, checkout hyperfine for Lots of useful tips in the original blog post."},{"title":"Vim spellcheck\u00a0commands","category":"snippet","url":"vim-spell.html","date":"27 May 2021","tags":"vim ","body":"[s or ]s \u2192 go to next/previ bad\u00a0word z= \u2192 list of\u00a0suggest zg \u2192 add word to good word\u00a0list zug \u2192 remove word from good word\u00a0list zw \u2192 add word to bad word\u00a0list zuw \u2192 remove word from bad word\u00a0list"},{"title":"Questions for good\u00a0references","category":"Non-technical/Entrepreneurship","url":"questions-for-good-references.html","date":"27 May 2021","tags":"marketing ","body":"Use the following questions to get What was the problem you had before you used the solution I What did the frustratio feel like as you tried to solve the\u00a0proble What was different about the solution I worked on compared to Take us to the moment you realized our solution was actually working to solve the\u00a0proble What does life look like now that the problem is being\u00a0solv Customize the text as necessary, for products or services, complete solutions are a component of a I made a template. Source: Building a story\u00a0bran"},{"title":"Lessons from 45 years in the software\u00a0industry","category":"Technical/Engineering","url":"45-years-in-software.html","date":"27 May 2021","tags":"advice ","body":"An article from a recently retired software engineer about lessons learned over 4\u00a0decades. Beware the curse of\u00a0knowled Focus on the fundamenta Teamwork, Trust, Testing, Communicat Code\u00a0Desig Simplicity Seek first to\u00a0underst Beware lock-in - cost of change, Be honest and acknowledg when you don\u2019t fit the\u00a0role"},{"title":"Modeling Credit\u00a0Risk","category":"Technical/Data","url":"credit-data.html","date":"25 May 2021","tags":"finance, precision, recall, colinearity, variance-inflation-factor, logistic-regression, confusion-matrix, f-score, f1, f2 ","body":"Data Exploratio Exercise\u00b6 Using whichever methods and libraries you prefer, create a notebook with the\u00a0follow Data preparatio and Identify the three most significan data features which drive the credit\u00a0ris Modeling the credit\u00a0ris Model validation and evaluation using the methods that you find correct for the\u00a0proble Your solution should have instructio and be For instance, If your choice is a python notebook, your notebook should install all the required dependenci to run\u00a0it. Import and preparatio \u00b6 In\u00a0[1]: # display more than 1 output per cell from import = \"all\" In\u00a0[2]: %%capture import sys -m pip install --upgrade pip -m pip install pandas numpy sklearn matplotlib seaborn statsmodel In\u00a0[3]: import pandas as pd = None # show all columns in a pandas dataframe In\u00a0[4]: data = data.head( Out[4]: .dataframe tbody tr { middle; } .dataframe tbody tr th { top; } .dataframe thead th { text-align right; } duration credit_his purpose credit_amo savings_st employment other_part age housing job num_depend own_teleph foreign_wo class 0 <0 6 critical/o existing credit radio/tv 1169 no known savings >=7 4 male single none 4 real estate 67 none own 2 skilled 1 yes yes good 1 0<=X<200 48 existing paid radio/tv 5951 <100 1<=X<4 2 female div/dep/ma none 2 real estate 22 none own 1 skilled 1 none yes bad 2 no checking 12 critical/o existing credit education 2096 <100 4<=X<7 2 male single none 3 real estate 49 none own 1 unskilled resident 2 none yes good 3 <0 42 existing paid 7882 <100 4<=X<7 2 male single guarantor 4 life insurance 45 none for free 1 skilled 2 none yes good 4 <0 24 delayed previously new car 4870 <100 1<=X<4 3 male single none 4 no known property 53 none for free 2 skilled 2 none yes bad Data Exploratio In\u00a0[5]: ### overview of the dataset data.shape data.colum data.nuniq # unique values per column print(f'cl \"good\" rows') print(f'cl \"bad\" rows') Out[5]: (1000, 21) Out[5]: 'duration' 'purpose', 'employmen 'age', ' 'housing', 'job', ' 'class'], Out[5]: 4 duration 33 credit_his 5 purpose 10 credit_amo 921 savings_st 5 employment 5 4 4 other_part 3 4 4 age 53 3 housing 3 4 job 4 num_depend 2 own_teleph 2 foreign_wo 2 class 2 dtype: int64 class: 700 \"good\" rows class: 300 \"bad\" rows In order to visually inspect the data it\u2019s necessary to convert the datatype of the categorica features from string to category. This will also be necessary to train the model. Therefore the data will be formatted before Outcome there would be an approximat equal number of outcome classes. In this dataset the split is 30% \u201cbad\u201d and 70% \u201cgood\u201d. The low number of bad credit risk assessment may limit the models ability to accurately predict a bad credit assessment relative to good assessment because of the limited training examples. This could result in more False Positives than would typically be\u00a0expecte Data formating\u00b6 In\u00a0[6]: # remove whitespace around column names data.colum = [col.strip for col in data.colum In\u00a0[7]: # Categorica variables have a limited and usually fixed number of possible values. # Categorica data might have an order (e.g. \u2018strongly agree', \u2018agree\u2019, 'disagree' 'strongly disagree') = [ 'purpose', 'housing', 'job', 'class' ] for col in data[col] = = [ [\"no checking\", \"<0\", \"0<=X<200\" \">=200\"], True), known savings', '<100', '100<=X<50 '500<=X<10 '>=1000'], True ), ('employme ['unemploy '<1', '1<=X<4', '4<=X<7', '>=7'], True), ] for col in In\u00a0[8]: # convert categories to numnerical values, for SelectKBes cat_column = = x: x.cat.code In\u00a0[9]: # all columns are now either categorica and encoded as an int (ordered or unordered) or numerical. data.dtype Out[9]: int8 duration int64 credit_his int8 purpose int8 credit_amo int64 savings_st int8 employment int8 int64 int8 other_part int8 int64 int8 age int64 int8 housing int8 int64 job int8 num_depend int64 own_teleph int8 foreign_wo int8 class int8 dtype: object In\u00a0[10]: # this will take a while.. import seaborn as sns # Create the default pairplot pairplot = sns.pairpl data, hue=\"class diag_kind = 'kde', plot_kws = {'alpha': 0.6, 's': 80, 'edgecolor 'k'}, height = 3 ) fig = pairplot.f dpi=200) # default dpi is 100 Visual inspection plot above compares each feature to each of the other features. The plots along the diagonal show the density plot of that feature, grouped by the value of the \u201cclass\u201d feature that the model will\u00a0predi Inspecting the pairplot shows that each of the features has approximat the same distributi for each class\u00a0valu Of the continuous features, Credit Amount is most heavily left skewed. Age is also left\u00a0skewe If may improve the model to log transform credit_amo and age, particular when predicting credit class when credit_amo or age values are in the middle of their This could be done as\u00a0follows = In\u00a0[11]: # Split the data into features and outcome. X = # all columns except the \"class\" column y = # only the \"class\" columns Check for emerges when three or more model variables are highly correlated can emerge even when isolated pairs of variables are not\u00a0coline The Variance Inflation Factor (VIF) is a measure of colinearit among predictor variables. It is calculated by dividing the variance of all model betas by the variane of a single\u00a0bet In\u00a0[12]: from import In\u00a0[13]: vif = pd.DataFra vif[\"VIF Factor\"] = i) for i in = X.columns vif = Factor'], vif.round( Out[13]: .dataframe tbody tr { middle; } .dataframe tbody tr th { top; } .dataframe thead th { text-align right; } VIF Factor features 18 2.11 own_teleph 0 2.12 11 2.50 5 2.53 savings_st 16 3.56 job 8 3.59 3 4.41 purpose 4 5.09 credit_amo 14 5.29 housing 13 5.59 2 5.85 credit_his 6 5.85 employment 1 7.70 duration 15 7.89 10 8.71 7 9.77 17 11.59 num_depend 12 13.19 age 9 17.03 other_part 19 23.45 foreign_wo The results show that there is a significan amount of in the\u00a0datase We are about to identify the features that drive the credit risk decision the most, this will reduce colinearit random noise and Once we know which subset of features contribute most to the predictive value of the model we could check for colinearit within that\u00a0subse Identify the three most significan features which drive credit risk\u00b6Featu selection is important because it removes or redundant predictors from the model. This improves model performanc and reduces computatio resource requiremen Various statistica tests could be used to find the most significan features, and can be placed into two braod categories - supervised (using domain knowledge and considerin the relationsh of the feature to the target variable) and unsupervis (which ignores the feature\u2019s relevance to the The ANOVA F-value method is appropriat for numerical inputs and categorica outputs. The Chi^2 test is appropriat for categorica inputs with Feature significan is evaluated using both\u00a0metho In\u00a0[14]: X = # all columns except the \"class\" column y = # only the \"class\" columns In\u00a0[15]: from import SelectKBes chi2, f_classif In\u00a0[16]: %%capture = 3 y) cols = new_data = In\u00a0[17]: Out[17]: ('duration ('purpose' ('employme ('age', ('housing' ('job', In\u00a0[18]: %%capture = 3 y) cols = new_data = In\u00a0[19]: Out[19]: ('duration ('purpose' ('employme ('age', ('housing' ('job', The three most significan features\u00a0a Duration (ANOVA: 48, Chi2:\u00a0321) Checking Status (ANOVA: 40, Chi2:\u00a036) Credit Amount (ANOVA: 24, Chi2:\u00a05826 The most significan feature is Duration, followed by Checking Status and Credit Amount. This makes sense intuitivel because Duration and Credit Amount must be proportion to risk. Checking Status is unfamiliar and I don\u2019t know what it means, in a real scenario I would speak to stakeholde or team members to learn more about this\u00a0featu In\u00a0[20]: = 'duration' X = In\u00a0[21]: vif = pd.DataFra vif[\"VIF Factor\"] = i) for i in = X.columns vif = Factor'], vif.round( Out[21]: .dataframe tbody tr { middle; } .dataframe tbody tr th { top; } .dataframe thead th { text-align right; } VIF Factor features 2 1.69 0 3.84 credit_amo 1 4.46 duration Using an arbitrary but common VIF threshold value of 5, we see that none of the 3 most significan features are colinear. Hoever duration is close with a score of 4.46 and credit_amo is second with 3.84. Intuitivel it seems reasonable that credit_amo and duration might have some colinearit as they could be generally expected to be inversely correlated to each other. - You could borrow more for a short period of time than you could for a Optimize the number of features to use\u00b6After an initial demonstrat of the metrics used to assess model performanc the optimal number of features to use will In\u00a0[22]: # used later = x:x[1], reverse=Tr Modeling Credit Risk\u00b6We use logistic regression because this is a binary classifica problem with multiple predictor variables that are Logistic Regression is a reasonable first choice of model type to use, but in a longer project we could use multiple model types and compare their performanc against one another, then select the classifier which performs the best. Other types could include k-nearest neighbors, decision trees, or Support Vector Logistic regression makes the No among the independen variables. can be tested using the Variance Inflation Factor (VIF). No Independen of errors (residuals or no significan The residuals should not be correlated with each other. This can be tested using the Durbin-Wat test, but this is out of scope for this\u00a0exerc The sample size should be large (at least 50 observatio per independen variables are recommende -> 20\\times50 =\u00a01000 Splitting the data into training, validation and testing sets\u00b6A validation set is used in addition to training and testing. This is because I expect to optimize model parameters by comparing the performanc of different models using a dataset they weren\u2019t trained on. If this were the same dataset that the final performanc metric was based on then the model would be Instead, the performanc of optimizati will be quantified using the validation set, and the optimal models performanc will be quantified using the testing\u00a0se This gives better prediction of actual performanc against new production data (outside the available dataset) than only using training and testing data\u00a0sets. In\u00a0[23]: from import from import In\u00a0[24]: # 70% training, 15% validation 15% testing is reasonable # use twice, first to split out a testing set, then to split the remainder into validation and training X_train, X_test, y_train, y_test = y, X_train, X_validati y_train, y_validati = y_train, assert len(X_trai == len(y_trai and == and len(X_test == len(y_test examples: examples: examples: Training examples: 700 Validation examples: 150 Testing examples: 150 In\u00a0[25]: from collection import Counter y_train_st = ratio = set has negative values and positive values \") print(f\"re ratio is approximat training set has 222 negative values and 478 positive values result ratio is approximat 32:68 The training set has over twice as many positive values as negative values. This is similar to the complete data set which has 700 \u201cgood\u201d credit class examples and 300 \u201cbad\u201d credit Training the logistic regression model\u00b6The model is trained on the In this simple first case, we evaluate performanc (discussed below) using the test data set. After an initial discussion of the results, the performanc of different models is evaluated using a validation data\u00a0set. In\u00a0[26]: logreg = y_pred = Model evaluation model could be evaluated in many different ways, each with different strengths and weaknesses depending on the of the data and the Where possible, its preferable to use only a single metric to quantify and compare model performanc because its simpler (and less error prone) than if there are multiple metrics to\u00a0conside There are 4 basic categories of results that the model could\u00a0prod True Positives (TP) False Positives (FP) True Negatives (TN) False Negatives (FN) These can be combined to calculate the model\u2019s Precision and\u00a0Recall Optimizing Precision minimizes false positives but ignores Optimizing Recall minimizes false negatives but ignores P = True Positives / (True Positives + False Positives) -> if False Positives are 0, P=1 R = True Positives / (True Positives + False Negatives) -> if False Negatives are 0, R=1 Precision is a ratio of the number of true positives divided by the sum of the true positives and false positives. It describes how good a model is at predicting the positive class. It is the ability of the classifier not to label as positive a sample that is\u00a0negativ High precision implies fewer Precision = True Positives / (True Positives + False Positives) Recall is calculated as the ratio of the number of true positives divided by the sum of the true positives and the false negatives. It quanitifie the ability of the model to find all the High recall implies fewer Recall = True Positives / (True Positives + False Negatives) An ideal model will have good precision and recall. The F-Score provides a way to combine precision and recall into a single measure that captures F-Measure = (2 * Precision * Recall) / (Precision + Recall) The influence of precision and recall relative to each other can be changed by adding a coefficien into the F-measure. Further discussion of this is outside the scope of this brief exercise except to say that F1 (above) has beta=1 and places equal weight on precision and recall. F2 (beta=2) places less weight on precision and more on recall, and F0.5 emphasizes precision over\u00a0recal Other metrics to consider include Precision Recall curves and the Receiver Operating Characteri curve (ROC). ROC curves are more informativ when evaluating datasets with equal proportion of positive and negative results. In a skewed dataset such as this, ROC can be overly optimistic and curves are more\u00a0relia Therefore after briefly demonstrat the confusion matrix we consider the average precision and recall instead of comparing the area under the ROC curve for Summary of results with the 3 most significan features\u00b6 In\u00a0[27]: from sklearn import metrics y_pred),3) y_pred),3) print(f\"F1 y_pred),3) Precision: 0.789 Recall: 0.91 F1: 0.845 Confusion Matrix\u00b6The confusion matrix simply shows the number of True Positives, True Negatives, False Positives and False Negatives that the These results are based on the testing data set. This model was trained on the In\u00a0[28]: import numpy as np import as plt import seaborn as sns %matplotli inline In\u00a0[29]: # name of classes fig, ax = plt.subplo tick_marks = class_name class_name cnf_matrix = y_pred) annot=True cmap=\"YlGn ,fmt='g') matrix', y=1.1); label'); label'); both precision and recall is useful in cases where there is an imbalance in the observatio between the two classes. This is because the large number of class 1 (\u201cgood\u201d) examples means we are less interested in the skill of the model at predicting this correctly, e.g. high true positives, and instead are more concerned with how the model can predict true negatives (where credit class is \u201cbad\u201d). The confusion matrix above shows that the largest error group is False Positives, which is expected because the model has relatively few examples of negative examples compared to positive. This is why the precision is lower than the\u00a0recall A no-skill classifier is one that cannot discrimina between the classes and would predict a random or constant class in all cases. This creates a minimum precision called the \u201cno-skill\u201d line. Its value is the ratio of positive cases in the dataset. For our training data set it is 0.687. (Note it is not 0.7, due to the random selection of the training\u00a0s The Precision Recall metric is a useful measure of success of prediction when the classes are very imbalanced as they in this case. A curve focuses on the performanc of a classifier on the The curve shows the tradeoff between precision and recall for different probabilit thresholds which we refer to but do not investigat in detail in this The threshold is the probabilit at which a \u201cgood\u201d or \u201cbad\u201d label is applied. The default threshold is 0.5, which means that if the model calculates the probabilit of an observatio being \u201cgood\u201d as more than 0.5 then it applies the label \u201cgood\u201d, otherwise it applies the label \u201cbad\u201d. This threshold probabilit can be varied and the impact on model A high area under the curve represents both high recall and high precision. High scores for both show that the classifier is returning accurate results (high precision) as well as returning a majority of all positive results (high\u00a0reca A system with high recall but low precision returns many positive results, but most of its predicted labels are incorrect when compared to the A system with high precision but low recall is the opposite, returning very few positive results but most of them are correct. An ideal system with high precision and high recall will identify almost all positive cases, with all results AP summarizes a curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the\u00a0weight In\u00a0[30]: from import disp = X_test, y_test) no_skill = / len(y_test # the proportion of results that are \"good\" 1], [no_skill, no_skill], label='No Skill'); Summary\u00b6Us the 3 most signficant features we have the F1: 0.845 AUC:\u00a00.87 The model could be improved by finding the optimal number of features to\u00a0conside Improving the model by optimizing the number of features\u00b6T code below iterates through the available features from most significan to least, adding an additional factor in each\u00a0loop. Each loop generates a new logistic regression model and trains it on the same training data set. Its performanc is evalulated using the same validation data set. The results are summarized in the \u201cresults\u201d dataframe and Due to the very small amount of data, changing the random_sta value can produce results with different trends. Therefore the first recommenda to improve the model is to get more data. This is also In\u00a0[31]: %%capture X = # start with all the features max_featur = 10 i = 1 results = [] while i <= max_featur print(i) X = # all columns except the \"class\" column # select the features we want to use in the model cols = [j[0] for j in # is sorted descending # remake the training sets with the correct features X_train, X_test, y_train, y_test = y, # random state is specified, ensuring same groups for each test X_train, X_validati y_train, y_validati = y_train, logreg = y_train); y_pred = # use validation this time (not test) because we will compare different models disp = X_validati y_validati fscore = (2 * disp.preci * disp.recal / + disp.recal lr_f1 = y_pred) i, 'f1': lr_f1, 'AUC': i += 1 results = inplace=Tr In\u00a0[32]: results Out[32]: .dataframe tbody tr { middle; } .dataframe tbody tr th { top; } .dataframe thead th { text-align right; } f1 AUC i 1 0.852590 0.740326 2 0.850202 0.810601 3 0.859438 0.855195 4 0.832653 0.826140 5 0.825911 0.873120 6 0.825911 0.874651 7 0.811475 0.876348 8 0.832653 0.878845 9 0.826446 0.889659 10 0.825000 0.881342 Optimizati Results\u00b6Th results show that the highest F1 score (which places equal emphasis on Precision and Recall) of 0.859 is obtained using a logistic regression model that uses the three most significan features (Credit Amount, Checking Status and\u00a0Durati Due to the small data sets (700 examples in the training set, 150 in each of the validation and testing sets) the results can vary significan when a different random number seed is used to generate the 3\u00a0datasets Whilst the table above shows that AUC increases with number of features, the increase is small (0.9% from 5 to 10 features) and sensitive to the random number seeds - if a different seed is used, the same trend is not observed. Therefore, whilst curves are a useful tool in imbalanced datasets, the small size of the dataset being used in this analysis creates variation that wouldnt be observed in Model performanc using Test Data Set\u00b6We test the performanc of the model chosen in the optimizati stage. In this stage we use the testing dataset, because this provides a more reliable indication of how the model might perform with (new) production data compared to using the results from the In\u00a0[33]: cols = ['duration X_train, X_test, y_train, y_test = y, # random state is specified, ensuring same groups for each test X_train, X_validati y_train, y_validati = y_train, logreg = y_train) y_pred = # use validation this time (not test) because we will compare different models disp = X_test, y_test) #fscore = (2 * disp.preci * disp.recal / + disp.recal #fscore y_pred) Out[33]: Out[33]: Obtain more training\u00a0d This will reduce the influence of randomness on the model\u00a0scor Use a credit score instead of just a \u201cgood\u201d or The \u201cgood\u201d or \u201cbad\u201d status is probably the result of a threshold being applied to a metric. Accessing the underlying metric would provide more detail and help train a more Consider This would require domain expertise and if { var mathjaxscr = = = = ? \"innerHTML : \"text\")] = + \" config: + \" TeX: { extensions { autoNumber 'AMS' } },\" + \" jax: + \" extensions + \" displayAli 'center',\" + \" displayInd '0em',\" + \" showMathMe true,\" + \" tex2jax: { \" + \" inlineMath [ ['','$'] ], \" + \" displayMat [ ['$$','$$' ],\" + \" true,\" + \" preview: 'TeX',\" + \" }, \" + \" 'HTML-CSS' { \" + \" linebreaks { automatic: true, width: '95% container' }, \" + \" styles: { .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'black ! important' }\" + \" } \" + \"}); \"; (document. || }"},{"title":"Grep only inside particular\u00a0files","category":"snippet","url":"grep-particular-files-only.html","date":"18 May 2021","tags":"grep, linux ","body":"grep -inr --include package.js \\ 'shortcut\" {' . -A 3 It\u2019s the --include flag that does the -i \u2192 -n \u2192 print line\u00a0numbe -r \u2192 recursive from starting\u00a0p . \u2192 start in -A 3 \u2192 print the 3 lines below the found\u00a0line"},{"title":"View a List of Keyboard Mappings in\u00a0Vim","category":"snippet","url":"vim-debug-mapping.html","date":"18 May 2021","tags":"vim ","body":":map \u2192 show a list of the current keyboard mappings for normal, visual, select and operator pending\u00a0mo :map! \u2192 show a list of the current keyboard mappings for insert and Top put all the mappings into a convenient text\u00a0file: :redir! > vim_maps.t :map :map! :redir END source another\u00a0so"},{"title":"Note Taking and Knowledge\u00a0Systems","category":"snippet","url":"note-taking-and-knowledge-systems.html","date":"18 May 2021","tags":"zettelkasten, notes, knowledge-system ","body":"This blog post arrives at the conclusion that the only way to take good paper notes is to organise them and only summarise the content once the notebook is\u00a0full. Keep it simple, trust simple \u2192 robust \u2192 reliable Also, it links to"},{"title":"Building A Story\u00a0Brand","category":"Non-technical/Learning","url":"building-a-story-brand.html","date":"17 May 2021","tags":"reading, marketing, communication, book ","body":"\u201cBuilding a story brand\u201d by Donald Miller is one of the most helpful marketing books I\u2019ve found. It explains why things should be done, as well as how to do\u00a0them. As an engineer I love solutions to problems that start from first principles and this feels like that. It\u2019s strategy as well as\u00a0tactics These are my notes on an Table of Contents Premise Stories Hero Problem Guide Recap Plan A Call To\u00a0Action What is at\u00a0stake? Avoid\u00a0Fail End with\u00a0Succe People want your brand to participat in Implementa Websites Corporatio Marketing Road-map One-liner Create a lead\u00a0gener An automated email\u00a0camp Collect and tell stories Create a system to Premise People want to survive and thrive. They are the main character in the movie of their own life and they have problems. Life is busy and hard and complicate and they don\u2019t want to waste energy figuring out if you can help\u00a0them. Good marketing shows that the product has an obvious benefit for the customer. How does your product improve someones ability to \u201csurvive and\u00a0thrive Cut through noisy and challengin world by having a super quick and easy to understand message. Don\u2019t be clever, be clear. Don\u2019t make the customer spend calories trying to figure out how they will benefit from your\u00a0produ Stories are intuitive and leverage many psychologi features. They organise informatio in an intuitive way and are a great way to combat noise and gain attention. People are compelled to pay attention until all the \u201cstory gaps\u201d have been closed. Could a caveman look at your website and answer What do you offer? How will it improve my\u00a0life? What are the next steps? Story\u00a0Gaps A gap between a character and what they want. Will they find their way to success, overcome their Cadence and momentum are defined by the creation and fulfillmen of story\u00a0gaps If you fail to define something that the customer wants, you fail to open a story\u00a0gap. This makes the story uninterest because there isn\u2019t a question that requires an\u00a0answer. Story gaps work because we want things to resolve. It\u2019s like singing Twinkle Twinkle Little Star but stopping before the \u201care\u201d on the last line. You need to hear the last note of the melody for the tune to feel\u00a0compl Pare down the customers desire to a single focus. Make your brand known for single specific desire and helping people get\u00a0it. Don\u2019t clutter the story by diluting the hero\u2019s desire with other desires. You can eventually Apple When Apple released the Lisa computer in 1983, Jobs bought a 9 page ad in the New York times listing the computers features. When Jobs returned to Apple after being fired, (and after partly founding Pixar, which tells stories) Apple became a customer centric company, and their marketing was about the customers. 9 pages became 2 words - \u201cThink Different\u201d A message about their customers, and their customers need to survive and\u00a0thrive Apple isn\u2019t the hero in the \u201cThink Different\u201d brand, the customer is. It\u2019s the same with Nike - the athletes are the heros, and you can become a (hero) athlete with Nike\u2019s help. Apple plays a role more like Q in James Bond, giving the hero what they need to\u00a0win. Stories A Hero \u2192 has a Problem \u2192 and meets a Guide \u2192 who gives them a Plan \u2192 and calls them to Action \u2192 that ends in Success \u2192 and helps them avoid Failure. Hero The customer, not you. You are the guide. The customer is the main character in their life and is battling internal and external adversity it order to survive and thrive. Heroes have weaknesses the guide is usually the A story starts with a character who wants to overcome external challenges that pose internal and Your first task is to create a story gap that implicitly asks \u201cWill the hero get what they\u00a0need? You need to define the character\u2019 ambition at the beginning, so that the audience knows what\u2019s at stake and what kind of a story it is. This messaging implicitly tells them how they could benefit and why they should Problem Customers are more motivated to solve internal problems than\u00a0exter If you identify a customer\u2019s problem then you show that you understand them. This is a great hook because they will then relate to The more we talk about the problems our customers are experienci the more interest they will have in our brand. Every story needs a villain, which is the personifie problem. The germs are personifie envy is personifie etc. The diminished social status is personifie by someone with more status The villain should be relatable - readers should recognize that the villain should be\u00a0disdain should be a root source - frustratio is what the villain makes us feel. High taxes are the cause, and are therefore the\u00a0villai should be singular - just one villain, keep it simple to cut through the\u00a0noise. should be real - don\u2019t be a fear monger. Fight a real problem on behalf of There are three types of problem External - A barrier to stability that must be removed - My business isn\u2019t growing fast enough, profits are too\u00a0small. Internal - I have self-doubt Do I have what it takes to\u00a0succeed Philosophi - I deserve to be successful my hard work should be rewarded. Failure would be\u00a0unjust. The villain initiates an external problem that causes the hero to encounter an internal problem that is wrong or unjust. The purpose of the external problem is to manifest the internal problem. Customers should recognise and relate to both types of\u00a0problem Put your product in the context of a type of survival they they want. Otherwise there isn\u2019t a story gap. Translate the external problem into several Resources - Conserving or accumulati money or\u00a0time Social - Gaining status or a social\u00a0net Generosity - Most people are not as Darwinian as we\u2019ve been led to believe, they want to be empathetic and\u00a0caring Purpose - Give customers an opportunit to be generous and participat in something greater than themselves - \u201cthe chief desire of man is not pleasure, but\u00a0meanin The only reason that customers buy is because the external problem that your product would solve is causing an internal frustratio If you identify and articulate that frustratio and then clearly, confidentl and repeatedly offer to solve it along with the original external problem then you bond with your customer. You\u2019ll have positioned yourself more deeply into their internal narrative and substantia differenti your\u00a0brand Philosophi problems are important because people want to be involved in something larger than themselves It adds depth and meaning. Representi (and solving) a philosophi problem gives customers a way of expressing themselves that they wouldn\u2019t If you can resolve all three problems in the same transactio then customers will experience a wave of relief and pleasure, and love your\u00a0brand When Anakin Skywalker blows up the Deathstar by aiming the perfect shot, he defeats the external problem (the enemy army), his internal problem (self doubt) and the philosophi Your CTA is the action that must be taken to close the final story gap Checklist: Is there a single villain that your brand What external problem is that The internal problem is probably found by considerin how the external problem makes your customer\u00a0f What is unjust or wrong about the suffering caused by that\u00a0villa Our hero is being challenged - will they be able to solve their problem? The only way to find out is to engage with the brand. Guide Customers aren\u2019t looking for another hero, they are the hero in their life. They are the main character which the movie of their life revolves around. They\u2019re looking for a guide because they need\u00a0help. A persons life is made up of many acts - \u201cdoorways of no return\u201d. Each life is unique but we all have commonalit We are all on journeys with Story chapters are book-ended by events. These events are always instigated by external actors or events beyond their\u00a0cont Heroes need a guide who is trustworth and earns respect. If they didn\u2019t need a guide their wouldn\u2019t be narrative, or a problem. Everyone is looking for a guide to help them solve We wake up each morning as a hero. We are troubled by internal, external and philosophi issues. We know that we can\u2019t solve our issues on our own. This insight has consequenc and raises questions, it means our story isn\u2019t about us but about others. We can aspire to be someone else\u2019s guide, but not their hero. It shows why the search for meaning is innate, but can only be resolved by becoming a servant or\u00a0guide. Stop losing sleep over the success of your company and start losing sleep over the success of In stories, heroes are not the strongest characters they have self-doubt and are often ill-equipp They are often reluctant, and are thrown into the story by external events. They are \u201cchosen by\u00a0destiny In contrast, the guide has already \u201cbeen there and done that.\u201d They have already conquered the hero\u2019s internal and philosophi challenges in their own\u00a0backst The guide is the one with authority which the authority instinctiv recognises and accepts. The guide has much more authority than the hero but the main character is still the\u00a0hero. Those who realize that the epic story of life is not about them end up winning in the end. This is paradoxica Those who think they are a hero and win usually end up being remembered as a\u00a0villain. of a\u00a0guide Empathy and Authority are a precise one-two\u00a0pu 1. Empathy \u2192 Understand \u2192\u00a0Trust When we empathise with customers we show that we understand them, and understand People want to be seen, heard and understood This is the essence of\u00a0empathy Key phrases in your marketing copy could\u00a0be: we understand how it feels to \u2026 no-one should have to experience \u2026 like you, we are frustrated by\u00a0\u2026 Expressing empathy isn\u2019t difficult. Once you\u2019ve identified your customers internal problem, let them know that you understand and would like to help them find a\u00a0resoluti Brains like to conserve calories, energy, effort, time, so when a customer realises they have a lot in common with a brand, they fill in any gaps with trust. A customer will \u201cbatch\u201d their thoughts, which means they are thinking in chunks rather than in details. Commonalit whether in music taste or values, is a powerful 2.\u00a0Authori No one likes a know-it-al and no one wants to be preached at\u2026 But people do want you to establish competence When looking for a guide, a hero trusts someone who has demonstrab competence The guide should have some serious experience helping other heroes win their day, but doesn\u2019t need to be\u00a0perfect There are four ways to add authority (competenc to your marketing without Testimonia Logos Statistics Awards Meeting a brand is like meeting a\u00a0person, Can you help them live a better life? Can they associate their identity with your\u00a0brand Can I respect this\u00a0brand Can I trust this\u00a0brand Recap We started the narrative by identifyin something that the hero\u00a0wants Then we created intrigue and tension by defining the hero\u2019s problem. The audience wants to know if we can help them overcome the\u00a0proble Then we introduced ourselves as the guide and establishe authority, empathy and\u00a0trust. What next? Making a purchase always involves a small risk of wasting money. This risky element makes a purchase somewhat similar to starting a relationsh There is a potential downside, the customer might end up feeling foolish and regret Imagine a customer trying to cross a river to get to their purchase. They can hear the sound of a waterfall downstream If they try to cross it by making a purchase then there is a chance that something bad could happen. Put stones in the river so they know how to safely walk across, step by\u00a0step. The stones are the\u00a0plan. Plan In a movie, the guide gives the hero a plan. The plan tightens the focus of the movie and creates a \u201cpath of hope\u201d for the hero that might, possibly, lead to the resolution of the hero\u2019s problems. It creates a story gap and implicitly creates questions that the audience want to be\u00a0answere A good plan removes risk and explains what to do. If we don\u2019t guide customers, they experience a little bit of confusion and use that confusion as an excuse to not\u00a0purcha Even though the setup or purchase or after-purc steps are obvious to us, they are not obvious to customers. Give them a plan and they will feel more confident Heroes trust a guide who has a plan. People are looking for a philosophy then can embody, or a series of steps they can take to solve Customers want to know where you can take them. Unless you can take them somewhere they want to go, why would they listen? The marketing goal is that every potential customer knows where we want to take them. Define a desire for your customer, and your marketing story will have a powerful\u00a0h There are two kinds of plan. Both work by earning trust and offering the customer a clear path to\u00a0stabili Agreement\u00a0 Process\u00a0Pl Process\u00a0Pl The minimal (3 - 6) steps required to buy or get benefit from the product after purchase, or a mixture of both.\u00a0E.g. Make Allow us to create a Let\u2019s execute the plan\u00a0toget A process plan removes confusion from the customers journey. When they see the plan they think \u201coh that\u2019s not difficult, I can do that\u201d and then they\u00a0purch A post-purch process plan would alleviate confusion about how the customer would use the\u00a0produc Agreement\u00a0 Agreement plans are about alleviatin fears. It\u2019s a list of agreements you make with the customer that are designed to alleviate their fears of doing business with\u00a0you. An agreement plan can also work to highlight shared values. Give the agreement plan a good name and it can increase the perceived value of your product. \u201cthe plan\u201d, \u201cyour best nights sleep ever\u201d,\u00a0etc Agreement plans can work in the background they don\u2019t have to be on the landing page, though they could\u00a0be. Make an agreement plan by creating a list of all the things a customer could be fearful about when doing business with you (haggling for price, interactin with a pushy salesman, buying a defective product) and then create an promise that would nullify that\u00a0fear. A Call To\u00a0Action So far, we\u2019ve defined a desire, identified their challenges empathized with them, establishe our competency and given them a plan. Heroes only take action when challenged by an external force. They don\u2019t take action by themselves they must be challenged This is just how humans\u00a0are We are the external force that guides our customers to\u00a0success Heroes need to be challenged by external forces. Calls to action should be clear, and should be repeated over and over. Above the fold, in the center of the page. Also in the navbar. And also repeatedly as they scroll down the\u00a0page. Customers are bombarded with adverts and all day every day. They are ignoring things and filtering out noise all the time. So don\u2019t be shy or subtle. Be very clear. Make it very simple. If you have confidence in your product, make confident calls to\u00a0action. Direct\u00a0Cal Buy\u00a0Now Schedule an\u00a0Appoint Order\u00a0Now Call\u00a0Today Register\u00a0T Repeat the same (singular, simple) call to action again and again down the\u00a0page. Download our free PDF guide to growing your\u00a0busin Free informatio - advice, guides, Giving something away for\u00a0free Testimonia Samples Free-trial A good transition CTA does three things: Changes the customers perception of you - it establishe your expertise and\u00a0author Creates reciprocit you offer them something of value before you ask for their\u00a0mone Positions you as a guide for the next\u00a0steps Use both types of CTA (direct and transition in your messaging. Then customers will understand (simply and without burning calories) what you want them to do in order to solve What is at\u00a0stake? Stories live and die by a single question; \u201cWhat is at stake?\u201d If nothing is about to be gained or lost then nobody cares. If there is nothing at stake in the story then there is no story, it\u2019s just If there is no benefit to buying the product, then why buy\u00a0it? You have to show the customer the cost of not buying the product. Avoid\u00a0Fail The story remains interestin as long as the hero is teetering on the edge between success and failure. A hero in a story only ever has 2 motivation - to escape pain or experience something good. Life is like that too, our desire to avoid pain motivates us to make a\u00a0change. There needs to be meaningful and consequent stakes in the story, otherwise it isn\u2019t interestin Each scene needs to move the hero either closer to or further from their tragic A brand needs to answer the \u201cwhat if I don\u2019t buy\u201d question otherwise the customer can\u2019t answer the \u201cso what?\u201d question - the stakes need to be clearly, simply, concisely communicat Probably stake, not\u00a0stakes You can do it humorously or lightly. Don\u2019t make a big negative scary thing out of it - failure is salt to add flavor, not a main marketing ingredient Compare the fear to the peace and stability that could be achieved. If you show the pain \u201cbefore\u201d and contrast it to peace and stability \u201cafter\u201d then you\u2019ve opened and closed a story\u00a0loop Blog titles, email subjects, headlines, can all contain elements of potential failure to convey a sense of urgency. Bring up the negative stakes a\u00a0bit. People fear losing$100 more than they desire gaining $100 - loss and pain is more motivating than reward and\u00a0peace. Recipe Let the reader know they are vulnerable to a threat. \u201cX% of Y get\u00a0W\u201d Let the reader know that since they are vulnerable they should take action to reduce their vulnerabil \u201cMake sure this doesnt happen and Give the customer a clear, concise simple plan to reduce their vulnerabil \u201cWe offer the thing you\u00a0need\u201d Challenge people to take the next step right now. The CTA. \u201cCall us today to arrange Agitate a bit of fear, and then return the reader to peace and prosperity all within 1 or 2\u00a0paragrap What are you helping your customers avoid? It\u2019s only a little salt to add flavor. If it\u2019s too little, customers won\u2019t know why your product is important. If it\u2019s too strong it\u2019ll End with\u00a0Succe Humans are looking for resolution to their external, internal and philosophi problems. They can achieve this though status, and transcende (among other things). If you product or service can help people achieve this then it should be a central part of your brand\u00a0prom People want to change their lives and be taken into a new reality. Tell them how their lives are going to improve - peace, status, confidence People want a vision of a happy ending. Compare these two statements \u201cWe\u2019re going to put a man on the moon\u201d or \u201cWe would like a highly competitiv and successful space\u00a0prog Use this table to show how your customer\u2019s lives will change. It will give lots of good copy for Before your brand After your brand What do they have? What are they feeling? What\u2019s an average day like What is their status? Talk about the end vision really clearly. And use images of happy, successful powerful people enjoying the benefits of your product. Say the benefits loudly, confidentl clearly, Talk about your end-vision for their lives once they\u2019ve benefited from your\u00a0servi Show the customer a vision of how great their life will be if they do business with\u00a0you. Ultimately the end of the story should be a list of resolution to your customers problems. How do they feel and how have questions been resolved? Stories usually end in one of three\u00a0ways The hero gains status or power Offer access - get a free\u00a0coffe Offer a premium - skip the\u00a0line. Create scarcity - write \u201climited\u201d on\u00a0it. Identity associatio - wear a Rolex and be associated with what Rolex stands\u00a0for The hero becomes whole by being unified with someone The hero needed something they couldn\u2019t get themselves and external provision has saved\u00a0them Reduced anxiety, or more\u00a0secur Reduces effort, or More time, reaches an The hero has some internal realisatio (coming of age) that gives them confidence to overcome their circumstan or internal shortcomin and become \u201cwhole\u201d and wise. They can achieve inner peace and know they reached their potential. Inspiratio - chariots of fire - you can also run really\u00a0fas Acceptance - fashion brands doing positive body\u00a0image Transcende - greater purpose and meaning - People want your brand to participat in A hero needs someone else to step into their life, tell them they are different and special and better. That someone is a guide, that\u2019s\u00a0you Offering an aspiration identity to our customers adds a lot of value to our products and services. Realise that your customers want to transform and People are looking for a guide. Everybody wants to change either into someone better or someone more You are helping them become wiser, fitter, more equipped, accepted or\u00a0peacefu What does the customer want to\u00a0become? what is their what kind of person do they want to\u00a0be? How do they want their friends to talk about\u00a0them Being a guide is a position of the heart, not just a marketing tactic. Lose sleep over your customers problems instead of your business. Commit to solving their internal, external and philosophi problems, and give them a vision to aspire\u00a0to. Customers needs to be told very clearly how much other customers have changed and how far journey has taken them. Usually, the hero is deeply flawed right up until the final Not all elements of a story should be used\u00a0evenl Implementa Websites Above the fold, use the one-liner. It\u2019s one sentence saying what\u2019s in it for the customer. What problem do you solve, what aspiration identity do you offer. It will also give customers words they can use to tell others about your\u00a0busin Big obvious calls to action, in multiple places. Don\u2019t be\u00a0timid. Customers read in a Z\u00a0shape. People don\u2019t read websites, they scan them. Repeat the important things so that they are understood by quick readers. Use very few words. The fewer words you use, the more likely they will be\u00a0read. Perhaps 10 sentences on the entire landing\u00a0pa Increase the amount of text towards the bottom. The top needs to be short, fast, positive and\u00a0obviou Use a \u201cread more\u201d button to expand longer text, then customers have the option if they\u00a0want. Place a transition call to action next to the main call to action. \u201cDownload a guide\u201d next to \u201cbuy now\u201d. Put the transition CTA in less bright\u00a0col Repeatedly ask people to buy, twice above the fold and then in subsequent sections. If you don\u2019t tell people what to do then they won\u2019t do\u00a0it. Include images of success - people enjoying the benefits - resolved problems, aspiration achieved, closed story loops. A sense of health, well-being The flow of the landing page should follow the StoryBrand framework, albeit not\u00a0exactl Corporatio Not applicable but if you\u2019re a manager and you give your team a narrative they can fit themselves into I guess they\u2019re going to have more purpose, confidence and\u00a0meanin Five steps, Create a one-liner. Say how can you make their lives better in an engaging\u00a0w Create a lead generator and get qualified email addresses. A guide, a free 30 minute consultati a voucher,\u00a0e An automated email campaign - a weekly email, 3 with nurturing advice and then 1 with a CTA. All\u00a0automa Collect and tell stories of Almost all stories are about the transforma of the hero. Tell this story and people will understand what you are\u00a0offeri Create a system that generates referrals. If word of mouth is so powerful, 1.\u00a0One-lin The one-liner is a statement. It could be more than 1 sentence, but its supposed to be super short. It\u2019s the equivalent of a logline for a movie. Keep editing it until you find a version that\u00a0works Memorize it, put it on your website, and include it in every piece of marketing collateral you\u00a0create You could tell this to anybody and they would understand what you\u00a0do. Helps people realize they need your\u00a0servi Provoke imaginatio and\u00a0intrig It should include: A character (\u201cA busy mom\u201d, \u201cA\u00a0retiree A problem - don\u2019t miss an opportunit to talk about a customers challenges Define the problem as vitally important, it opens a story gap and customers will want you to close\u00a0it. A plan - hint at it. You can\u2019t explain it all in a\u00a0one-line Success - paint an image of life after a customer has bought your\u00a0servi Example answers to the question \u201cWhat do you\u00a0do?\u201d: \u201cWe provide busy moms with a short, meaningful workout they can use to stay healthy and have renewed energy\u201d vs \u201cI run a\u00a0gym\u201d. \u201cWe help retired couples who want to escape the cold avoid the hassle of a second mortgage while still enjoying the warm beautiful weather of Florida during the winter\u201d vs \u201cI got involved in real estate a few years ago and when we had out second kid we moved to Florida and\u00a0then\u2026. 2. Create a lead\u00a0gener A guide to\u00a0downloa A free webinar or online\u00a0cou Software demo or free\u00a0trial Free\u00a0sampl Live\u00a0event No need to reinvent the wheel, what are others\u00a0doi You can give away quite a lot of value that is easy to consume in an email or PDF - people will consume it quickly and will probably be happy to pay for a chance to learn in a more thorough and Downloadab guides etc should be about 3 pages in\u00a0length. Be generous, explain the \u201cwhy\u201d and give away as much of the \u201chow\u201d as\u00a0possibl 3. An automated email\u00a0camp Even if customers don\u2019t click the links in the email, they keep seeing your brand and they become familiar with it. When they need your services you will be the go-to brand in their mind. The relationsh and associatio is already\u00a0bu Nurturing emails and Call-To-Ac emails - 3:1 - put real value in the Don\u2019t be passive in the CTA emails - you want them to buy, so tell them, repeatedly and make it\u00a0easy. Talk about a\u00a0problem Describe a product you offer that solves the\u00a0proble Describe what life can look once the customer has bought your product and solved their\u00a0prob Call the customer to an action that leads directly to a\u00a0sale. Similar to a nurturing email, a direct action email also describes a problem and a solution, but in the direct email the solution is a product you sell and there is a strong call to action. A lot of the content can be taken from the 4. Collect and tell stories A great testimonia gives customers the gift of going second, and lowering the riskiness of\u00a0purchas A good showcase the value you\u00a0create the\u00a0result the\u00a0experi what transforma has\u00a0occurr Few things are more important to a good story than a hero that experience an external and internal and philosophi This is because everyone desires to be transforme in\u00a0someway People love businesses that help them transform in some\u00a0way. Questions to get a What was the problem you were having before you discovered our\u00a0produc What did the frustratio feel like as you experience that\u00a0probl What was different about out\u00a0produc Take us to the moment when you realized our product was actually working to solve your\u00a0probl Tell us what life looks like now that your problem is solved or being\u00a0solv 5. Create a system to Identify your existing Give them a reason to talk about you - create a video or a PDF that a customer can send to their friends to help them introduce your brand and explain the value you\u00a0delive Offer a reward - a discount, more access, extra samples,\u00a0e"},{"title":"Training \u2192 Knowledge \u2192 Confidence \u2192\u00a0Victory","category":"snippet","url":"trainging-knowledge-confidence-victory.html","date":"14 May 2021","tags":"quote, caesar, wisdom ","body":"\u201cWithout training, they lacked knowledge. Without knowledge, they lacked confidence Without confidence they - Julius\u00a0Cae"},{"title":"Forward\u00a0Email","category":"snippet","url":"email-forwarding.html","date":"14 May 2021","tags":"email ","body":"I know about ImprovMX, which used to be great because you could do a lot for free, but now you only get 1 domain for\u00a0free. ForwardEma are 3 times cheaper than ImprovMX, and I have 2 domains forwarding email. It\u2019s not particular private, but I can send and recieve from a domain, for\u00a0free."},{"title":"Nested Auto Commands for Overriding\u00a0Colorschemes","category":"snippet","url":"modifying-vim-colorschemes-correctly.html","date":"13 May 2021","tags":"vim ","body":"A snippet detailing how to use nested auto commands to apply custom modificati when a colorschem is loaded. Perhaps this will stop me :e-ing so\u00a0frequen"},{"title":"Vim\u00a0Snippets","category":"snippet","url":"vim-snippets.html","date":"13 May 2021","tags":"vim, netrw ","body":"A useful collection of gists by A gist about netrw."},{"title":"See where Vim is setting an\u00a0option","category":"snippet","url":"where-was-a-setting-set-.html","date":"12 May 2021","tags":"vim ","body":"See where an option was set in vim using the :verbose set textwidth?"},{"title":"My Life\u00a0Expectancy","category":"Non-technical/Journal","url":"my-life-expectancy.html","date":"12 May 2021","tags":"regression, statistics, life-expectancy, death ","body":"Key points: Weight doesn\u2019t matter. Be in \u201cexcellent health. Workout 3 or 4 times a week. Drink liquor (or red wine) 3 or 4 times a week (after each workout). Try to be happy, optimistic and relaxed. What are you working so hard for anyway? You need something to do, someone to love, something to hope for. Longevity Calculatio I was playing with the life expectancy calculator at and was surprised to find that their regression technique gives me a 50% chance of living to 95! I\u2019d expected a result closer to 80. After printing out a 90 year calendar, my next thought was to play with the calculator to find the maximum age I could have a 50% chance of reaching. Doing the factors below would apparently give me: 75% chance of living to 92 50% change of living to 101 25% chance of living to 107 Don\u2019t workout every day There is no additional benefit from working out more than 4 times a week. Once you\u2019re fit and working out 4 times a week it doesn\u2019t matter if you do more exercise. Being in \u201cexcellent health is better than being in \u201cvery good\u201d health though, so make the workouts count. I guess if it didn\u2019t make a difference then it wouldn\u2019t be excellent in the first place. Drink more alcohol I was surprised that having 3-4 drinks each week increases your life expectancy I thought that it was best to not drink any alcohol at all. I should be drinking red wine or liquor 3 or 4 times each week. Liquor increases longevity in men, but reduces longevity for women. Red wine increases longevity in women, but has no effect in men2. I guess I should drink some rum after each workout. Weight doesn\u2019t make a different According to their regression my expected longevity is unchanged within a weight range of 78kg - 90kg. I guess it\u2019s much more important that I\u2019m in \u201cexcellent health, working out 4 times a week and having some wine, rum, or whiskey after each workout. But it\u2019s just statistics The calculator only asks for quantifiab or physical attributes It doesn\u2019t consider emotional, relational or spiritual factors.Th is also a more detailed calculator at its calibrated for Canadian citizens. \u21a9Alcohol consumptio in later life and reaching longevity: the Netherland Cohort Study \u21a9"},{"title":"Globbing","category":"snippet","url":"globbing.html","date":"10 May 2021","tags":"linux ","body":"???? \u2192 4\u00a0chars * \u2192 any number of\u00a0chars [:upper:] \u21d4 [A-Z] same for [:lower:] and [:digit:] [:alpha:] \u21d4 [a-zA-Z] [:alnum:] \u21d4 [a-zA-Z0-9 ls -l [a-d] \u2192 part of a\u00a0range ^ and$ works like in\u00a0regex la a*.{doc,do \u2192 OR ls a*.(doc|do \u2192 OR"},{"title":"More VIM\u00a0Notes","category":"Technical/Developer Tools","url":"more-vim-notes.html","date":"10 May 2021","tags":"tips, vim ","body":"Table of Contents Quickfix\u00a0l A quickfix list is a set of positions in one or more\u00a0files A quckfix list is global. Not local to a A quickfix list is not the quickfix window. The window can show the list. The list is A changelist is local to its\u00a0buffer Registers 0 contains the content of the last\u00a0yank 1-9 contains the content you\u2019ve deleted or\u00a0changed _ blackhole register - send something here and it wont change any - contains any deleted or changed content smaller than 1\u00a0row. % contains the name of the current\u00a0fi In insert mode, =. Type any The output is inserted into the\u00a0buffer Substituti :&& \u2192 repeat the last substituti command with its\u00a0flags :~ \u2192 repeat the lat substituti with the same replacemen but with the last used search\u00a0pat Command\u00a0li q: - opens the command line window. Good for yanking and viewing : - open command history\u00a0li :UltiSnips - opens the ultiSnips file for the current buffers filetype. See which snippets are\u00a0define Delete stuff without leaving insert\u00a0mod - same as\u00a0backspa - delete previous\u00a0w - delete everything before cursor (on same\u00a0row) or - (un)indent a\u00a0row - delete next\u00a0word Text\u00a0objec gf - edit the file at the file path under the cursor (useful for\u00a0netrw? gx - open the file at the file path under the cursor (useful for\u00a0netrw? [m, ]m - move to the start or end of a\u00a0method @: - repeat the last\u00a0comma >> will indent a line. . will repeat the operation, so >>.. would indent a line 3\u00a0times. You can use this along with a count, which will do the indention for n number of lines (with the current line being the top line). 3>>.. will indent 3 lines 3 blocks to the\u00a0right. - up one line, and moves the cursor if it would go off the\u00a0screen - down one line, and moves the cursor if it would go off\u00a0screen - down one page, with cursor at top of\u00a0screen - up one page, with cursor at bottom of\u00a0screen Sources The Valuable Dev has a lot of great\u00a0tips Vim for Python has some great notes on linting and code completion plugins that I\u2019ve either copied or was more or less doing\u00a0alre"},{"title":"Vimscript\u00a0functions","category":"snippet","url":"create-custom-functions-in-vim.html","date":"5 May 2021","tags":"vim ","body":"Create a custom command and function to create a new file in\u00a0vim. command! -nargs=1 Ms call function! s:NewFile( echom a:fp execute \"e \" . \"~/foo/bar . a:fp . \".ext\" endfunctio Useful\u00a0hel :h %:h \u2192 :h expand() \u2192 expand wildcards, including question on SO"},{"title":"Better Text\u00a0Objects","category":"snippet","url":"vim-text-objects.html","date":"5 May 2021","tags":"vim ","body":"target more types of\u00a0object consistent if you\u2019re not inside the thing jump forward or\u00a0backwar look for the nth\u00a0occurr select white space Github Article about a\u00a0plugin"},{"title":"Delete stuff in Vim without leaving insert\u00a0mode:","category":"snippet","url":"delete-from-vim-insert-mode.html","date":"5 May 2021","tags":"vim ","body":" - same as\u00a0backspa - delete previous\u00a0w - delete everything before cursor (on same\u00a0row) or - (un)indent a\u00a0row - delete next word (create a mapping in\u00a0vimrc)"},{"title":"Global\u00a0Aliases","category":"snippet","url":"global-aliases.html","date":"5 May 2021","tags":"alias, linux ","body":"If you want to alias a bunch of arguments for a command, use alias -g foo=\"some complicate options\" grep some complicate options becomes: grep foo"},{"title":"Vim register for yanked\u00a0text","category":"snippet","url":"vim-yanked-text-buffer.html","date":"5 May 2021","tags":"vim, linux, text ","body":"It\u2019s annoying when you delete something and overwrite your yanked\u00a0tex Use numbered registers! \"0 to \"9 \"0 contains the most recent yank. \"1 contains the most recent deleted\u00a0te \"0p - paste the most recent yank, even if you deleted something after yanking\u00a0it"},{"title":"ChezMoi\u00a0shortcuts","category":"snippet","url":"chezmoi-shortcuts.html","date":"5 May 2021","tags":"dotfiles, alias ","body":"Chezmoi is a great tool for managing dotfiles. This is a shortcut to update the source state based on local\u00a0chan chezmoi status | cut -c 4- | xargs -I % -p sh -c 'chezmoi add ~/%' Github"},{"title":"Sleep","category":"snippet","url":"sleep-is-good.html","date":"4 May 2021","tags":"sleep, lifestyle ","body":"\u201cIt enhances your memory and makes you more creative. It makes you look more attractive It keeps you slim and lowers food cravings. It protects you from cancer and dementia. It wards off colds and the flu. It lowers your risk of heart attacks and stroke, not to mention diabetes. You\u2019ll even feel happier, less depressed, and less\u00a0anxio Why We Sleep by Dr. Matt\u00a0Walke"},{"title":"Bash Strict\u00a0Mode","category":"snippet","url":"bash-strict-mode.html","date":"4 May 2021","tags":"bash, linux ","body":"How to write robust bash\u00a0scrip Bash Strict\u00a0Mod"},{"title":"How to write an About\u00a0Page","category":"snippet","url":"how-to-write-an-about-page.html","date":"4 May 2021","tags":"writing ","body":"An often recommende blog post by Kaleigh Moore about writing a good about\u00a0page"},{"title":"Domain Name\u00a0Registrars","category":"snippet","url":"domain-name-registrars.html","date":"3 May 2021","tags":"web ","body":""},{"title":"The Honest Troubleshooting Code of\u00a0Conduct","category":"snippet","url":"honest-troubleshooting-code-of-conduct.html","date":"3 May 2021","body":"blog\u00a0post"},{"title":"Linux Filesystem Hierarchy\u00a0Standard","category":"snippet","url":"linux-etsy-dir.html","date":"2 May 2021","tags":"linux, filesystem ","body":"/etc (etsy) \u2192 \u201cetcetera\u201d or \u201ceditable text config\u201d \u2192 a place to put config\u00a0fil Originally the root directory had /boot for booting, /dev for devices\u2026 One dir for each type of thing. But this put config in many places. so etc/ Filesystem fhs-2.3"},{"title":"Browser\u00a0Security","category":"snippet","url":"browser-security.html","date":"30 April 2021","tags":"xss, cors, http ","body":"Blog post about CSRF CORS HTTP"},{"title":"HTML\u00a0Templates","category":"snippet","url":"html-templates.html","date":"30 April 2021","tags":"html, jam ","body":"cruip.com"},{"title":"Linus\u00a0Torvalds","category":"snippet","url":"interview-with-linus-torvalds.html","date":"29 April 2021","tags":"linux, interview, linus ","body":"From an Interview: I don\u2019t want to claim that programmin is an art, because it really is mostly just about \u2018good engineerin I\u2019m a big believer in Thomas Edison\u2019s \u2018one percent inspiratio and ninety-nin percent perspirati mantra. It\u2019s almost all about the little details and the But there is that occasional \u2018inspirati part, that \u2018good taste\u2019 thing that is about more than just solving some problem - solving it cleanly and nicely and yes,"},{"title":"Remote Procedure\u00a0Calls","category":"snippet","url":"rpc.html","date":"29 April 2021","tags":"rpc, linux ","body":"A RPC is when an executable causes a procedure (subroutin to execute on another computer, It\u2019s coded as if it were a normal (local) subroutine You don\u2019t explicitly code the details for the remote interactio You write the same code whether the subroutine is local or\u00a0remote."},{"title":"An interesting\u00a0blog","category":"snippet","url":"useful-blog.html","date":"29 April 2021","tags":"linux, shell, fzf, workflow, zsh, bash, builtin ","body":"Just found a really useful blog Interestin discussion about the difference between builtins Nice examples of using fzf to improve workflows."},{"title":"lsblk","category":"snippet","url":"lsblk-command.html","date":"28 April 2021","tags":"unix, cli ","body":"lsblk is a command to get info about Used when"},{"title":"Ranger File\u00a0Manager","category":"snippet","url":"ranger-file-manager.html","date":"28 April 2021","tags":"ranger, unix, vim, tools ","body":"A console based file manager with vi key\u00a0bindin Install it with brew install ranger Launch it with ranger"},{"title":"Vim\u00a0Regex","category":"snippet","url":"vim-regex.html","date":"28 April 2021","tags":"vim, regex ","body":"This is a great article about using regular expression in\u00a0Vim:"},{"title":"Where and when will the current Bitcoin market\u00a0peak?","category":"Technical/Cryptocurrencies","url":"when-bitcoin-top.html","date":"27 April 2021","tags":"bitcoin, finance, markets ","body":"Buying Checklist Cyclic bottom when price is close-ish to the 200 week moving average Selling Checklist Top Cap \u2248 Market Cap chart MVRV > 4 chart S2F deflection > 3, but noisy chart 0.875 \u00d7 Delta Cap \u2248 Realised Cap chart. HODL waves - 45% moved in the last 6 months chart 12 Month RSI > 90 chart 3-month coin days destroyed - check glassnode, STH and LTH chart Summary S2F model suggests a peak around the beginning of 2021Q4, in the region of $300,000. The rainbow chart seems to broadly agree with S2F. If the age-adjust 3-month coin days destroyed goes above 550,000 then get ready to sell. Willy Woos \u201cdouble top\u201d chart suggests a peak around$400,000. RationalRo comparison of bull runs suggests a market top around 14 September and a maximum price around $2,000,000 [sic]. RationalRo comparison of 12 months RSI suggests that the market top is reached shortly after a 12 month RSI exceeds 90. This seems less reliable than the above points Jurrien Timmer suggests a peak price around$100,000 if price shoots up unexpected quickly. I expect the market top to be significan higher. $100k will be a massive psychologi level. If the price does increase to this level before approximat July then the optimal sell price would therefore not be exactly$100,000. Market top is expected around September 2021 at the earliest. The behaviour of the market will change as its participan change. There hasn\u2019t been a bull run with significan institutio investors before. Sell using a cost averaging strategy. Awareness Buy when everyone is selling, sell when everyone is buying. Be brave when there is fear, if there is no fear then its about to get very messy. If everyone is super confident that prices are definitely going to go up, something bad is about to happen. A lack of uncertaint is a big warning bell. 14 September, $300,000. Threshold Values MVRV MVRV > 3 \u2192 Local top MVRV > 4 \u2192 Macro top MVRV has historical been one of the best on-chain predictors of market tops and bottoms. The ratio of Market Value to Realised Value is calculated by dividing Bitcoin\u2019s market cap by its realised cap. chart Top Cap Top Cap is 35 x Average Cap Market top when Top Cap is equal to Market Cap. Delta Cap Delta cap is Average Cap subtracted from Realised Cap. When Delta Cap is almost Realised Cap, it\u2019s a market top. When Delta cap touches Average Cap, it\u2019s a market bottom. Market Top when Delta cap is within 20%\u00ad15% of Realised Cap S2F Deflection Get key ratio values If s2F deflection > 3, but its noisy HODL Waves >45% of supply has been moved in the last 180 days (6 months) \u2192 Sell >70% of supply has been held over 180 days \u2192 Buy chart - hover the cursor over todays date and add up all the age brackets from 24hr to 3-6 months 12 month RSI > 90 14 month RSI > 95 \u2192 Sell 12 month RSI > 90 \u2192 Sell Noisy - defer to other metrics. SOPR Use 7 day average. 1.04 \u2192 Sell 0.97 \u2192 Buy Noisy - defer to other metrics. Realised Cap > NVT Cap Realised Cap should be lower than NVT Cap. Sell when Realised Cap almost exceeds NVT cap See chart below, there have been false positives. Realised Cap is lower than NVT cap during a bull market only. Noisy, could be a miss. Charts Stock to Flow Model Rainbow Model Top Cap Realised Cap, NVT Delta Cap 3 month coin days destroyed Double top Similariti to previous bull runs Version 1: Version 2: Halving model: 12 Month RSI comparison Bitcoin price history Lowest price forward model Key metrics and terms Average Cap The \u201cforever\u201d moving average of market cap. It is the cumulative sum total of daily market cap values divided by the age of the market in days. Top Cap Average Cap multipled by 35. NVT Cap A valuation using monetray velocity. Checkout CoinMetric for more info. MVRV The ratio of Market Value to Realised Value is calculated by dividing Bitcoin\u2019s market cap by its realised cap. Realised Cap The sum of the products of each UTXO and the market price of Bitcoin when the UTXO was generated. Market Cap The price of the most recent Bitcoin transactio multipled by the number of Bitcoin UTXO Unspent Transactio Outputs. These are kind of like unspent coins. If you have 1.5 BTC then you might have bought 2 and sold 0.5. The total value of UTXOs in your wallet will be 1.5. SOPR The Spent Output Profit Ratio is a measure of the average profit or loss on a coin. Is a coin is moved when the price is higher than when it was received the SOPR increases, if a coin is moved when the price is lower than when the coin was received then SOPR decreases. It won\u2019t be accurate for individual coins but in aggregate it gives an idea of whether coins are being sold at a loss or for profit. Market participan who have owned BTC for 3 months behave differentl to those that have held BTC for 3 years. A more experience investor will likely make more measured and less rash decisions. By segregatin the UTXOs according to age you can compare old coins and new coins, experience and inexperien investors (in aggregate) Weak hands will sell before stronger hands, and when market price decrease it\u2019s useful to know aggregate age data for the coins being sold. If coins are moving from young wallets then the selling is likely much less significan than if coins are being moved onto exchanges from old wallets. aSOPR The Adjusted Spent Output Profit Ratio is the same as SOPR but it ignores coins less than 1 hour old. If profits are taken by old coins, aSOPR trends higher. It will trend lower when older (and therefore profitable coins remain dormant. The higher aSOPR is, the more profit has been taken off the table. When aSOPR is less than 1, spent coints are moved at an aggregate loss. URPD UTXO Realised Price Distributi - If a lot of coins have moved within a particular price band, it is likely that there is strong price support and resistance at this price. This would be truer and more reliable in a mature market. Because the market for Bitcoin is expanding so rapidly and the price is so volatile, the attitudes and expectatio of market participan is also much more malleable than in traditiona finance. For example, what was considered a very high price 12 months ago would be considered a disaster today. RSI The Relative Strength Index is borrowed directly from traditiona finance. You can calculate it over different time periods. Miner Net Position Miner Net Position shows the degree to which, on aggregate, Bitcoin miners are profiting from the coins they\u2019ve generated from mining. Miners are expected to be among the most bullish of all market participan and therefore it is notable when they start moving coins from their mining wallets into exchange wallets. Stock-to-F A stock to flow model is used to measure the scarcity of a commodity. It\u2019s a calculatio based on the ratio of existing supply and how much is being produced. The higher the ratio, the longer it will take for supply to meet existing demand. Gold has a stock to flow ratio of 66, which means it would take 66 years at the current rate of production to produce the amount of gold currently in circulatio Silver has a S2F ratio of 74. BTC has a S2F of about 50. Background Over the past 8 years, Bitcoin has gone through phases of rapid price increase followed by periods of rapid decrease. The price has been driven by increasing market size and a decreasing rate of issuance, and has been so volatile (compared to traditiona finance) that \u201c1 month in cryptocurr markets is like 1 year in traditiona markets\u201d. However volatility is decreasing and we are seeing lower highs and higher lows during each subsequent market cycle. The single biggest factor driving multi-year market cycles appears to be the decreasing rate of supply increase (the issuance rate). The last 3 halvings1 seem to have provoked the last 3 bull cycles. We are in the third bull cycle now (April 2021) and I fully expect it to be followed by a bear cycle. As an amateur investor, I want to buy low and sell high. I\u2019d like to time the top and bottom of the market with reasonable accuracy, just like everyone else. But I\u2019m aware that my methods are less nuanced than profession traders and analysts - I have access to less data than them and I\u2019m not willing to put in as much effort as they are. I\u2019m happy to do this Pareto style - I\u2019ll give it 20% of my maximum effort and I\u2019ll be happy with 80% of an ideal result2. This is a review of what I consider to be the best sources of metrics and analysis that I\u2019ve come across. All the resources used in this article are attributed to the original author and have been made freely available on Twitter. I hope its OK for me to repost them here, if it\u2019s not then let me know and I\u2019ll edit the post. Hopefully helpful links This article assumes some familiarit with blockchain and financial markets. Some more general articles on this site are: Bitcoin compared to Gold How to buy bitcoin Analysts These insights, metrics and charts are the work of the following people and organisati Willy Woo Timothy Peterson PlanB Glass Node CoinMetric Jurrien Timmer Every 210,000 blocks, the number of bitcoin awarded to the miner for successful adding a block is halved. The last halving occurred in May 2020 and the rate of issuance halved from 12.5 BTC/block to 6.25 BTC/block \u21a9I realise this probably isn\u2019t, strictly, what Mr. Pareto was thinking when he published his research. I hope you get my intention. \u21a9"},{"title":"All Known Locations of an\u00a0Executable","category":"snippet","url":"where-command.html","date":"25 April 2021","tags":"unix, macos, cli, bash ","body":"where \u00a0the Expanding phrases: kr -> kind\u00a0regar Multi-line It really seems similar to what I\u2019m using UltiSnips\u00a0 I found this question on SO comparing abbreviati and snippets. TLDR: It\u2019s easier to add and maintain snippets than abbreviati and you have less boilerplat with snippets than abbreviati especially in complex\u00a0ca To fix the social sciences, look to the \u201cdark ages\u201d of medicine Emotional resilience in 3 virtues of a\u00a0programm Laziness - The quality that makes you go to great effort to reduce overall energy expenditur It makes you write labor-savi programs that other people will find useful and document what you wrote so you don\u2019t have to answer so many questions about\u00a0it. Impatience - The anger you feel when the computer is being lazy. This makes you write programs that don\u2019t just react to your needs, but actually anticipate them. Or at least pretend\u00a0to Hubris - The quality that makes you write (and maintain) programs that other people won\u2019t want to say bad things\u00a0abo Also, I read a quote somewhere saying the mark of a great program is having people use it in ways you didn\u2019t expect, or something like\u00a0that. Is WebAssembl magic performanc pixie\u00a0dust Yamauchi No.10 Family\u00a0Off A beautifull Improve and Extend Your Text Objects A Vim Guide for Adept\u00a0User How to manipulate multiple quickfix and What are digraphs and how to use\u00a0them. Useful keystrokes in INSERT\u00a0mod Useful keystrokes in VISUAL\u00a0mod Vim Using shell commands in\u00a0Vim. Deep dive in CORS The ps 1 - see which processes are running or sleeping. WCHAN tells you which kernel event a waiting processing is\u00a0awaitin"},{"title":"Learning - April\u00a02021","category":"Technical/Developer Tools","url":"learning-april-2021.html","date":"21 April 2021","tags":"learning, youtube, ansible, ssh, vagrant, google-cloud-platform, service-accounts, iam ","body":"Table of Contents Google Cloud Platform Ansible SSH Vagrant Google Cloud Platform It seems like I\u2019m looking for some general overview of how roles are managed, viewed, compared, and\u00a0inheri How can you tell if a users (or a service accounts) roles are adequate, or too much or too little for a particular task? And what\u2019s the difference between a user having some roles, and a user using a service account that has those\u00a0role It would also be nice to have some kind of adversaria test, that would identify how/if users or service accounts can create identities with more flexible permission that their\u00a0own. These short videos are good, but they\u2019re not a complete solution. I\u2019m not sure where to look\u00a0next. Ansible Based on Jeff Geerlings book. There are 15 episodes. Jeff seems like a great guy. I\u2019m going to try listen to one of these each\u00a0day. SSH This is also a very useful article. I made notes from it in another post. Vagrant Good for local developmen (Especiall when Not as good for cloud providers as\u00a0Terrafo No more"},{"title":"Tweets - April\u00a02021","category":"Non-technical/Journal","url":"tweets-april-2021.html","date":"21 April 2021","tags":"twitter ","body":"Table of Contents Front-End Mental\u00a0Mod Agency Razors Crypto Front-End 6 website for top landing page inspiratio onepagelov .com by @robhope \u2022 lapa .ninja by \u2022 landingfol .com by @dannypost saasframe .io by .com by @Cruip_com \u2022 saaspages .xyz by @Versoly\u2014 Jim Raptis (@d__rapti April 14, 2021 anyone interested in a fun @microacqu 90 or 180 days from start to finish to build and sell a tiny company in public?wou be really great M&A practice for my Jim Bisenius April 14, 2021 Mental\u00a0Mod Tobi's favorite example of FIRST PRINCIPLES is a Truck driver.His truck was sat still for 8 HOURS on THANKSGIVI waiting for his cargo to be unloaded when he realized\u2026\u201c not take the WHOLE trailer off the back of my ship rather than unloading + reloading each\u00a0item? George Mack May 18, 2020 LUTKE LEARNING 6 - TALENT STACK LED BY CURIOSITY > MBA He didn't have an MBA. He didn't grind 100-hour workweeks. Instead, he played video games (which led to coding) and he snowboarde (which led to an online snowboardi store). This 'Talent Stack' led to\u00a0Shopify George Mack May 18, 2020 A super long thread, worth reading it all: Josh Waitzkin might be the most INTERESTIN person alive.He doesn't have Twitter. And he barely uses the internet.I compiled my favorite 5 MENTAL MODELS of his below.THRE George Mack August 8, 2020 Agency 1/ HIGH AGENCY Once you SEE it - you can never UNSEE it. Arguedbly the most important personalit trait you can foster. I've thought about this concept every week for the last two years since I heard discuss it on @tferriss' podcast. THREAD\u2026\u2014 George Mack November 29, 2018 Razors THREAD: 15 of the most useful razors and rules I've found.Rule of thumb that George Mack January 16, 2021 Crypto Now let\u2019s compare this to the stock-to-f model. Below I added in the S2F model, which is the aforementi inflation rate regressed against price. /10 Jurrien Timmer April 13, 2021 #Bitcoin is looking strong at RSI 92. Still not above RSI 95 like 2017, 2013 and 2011 bull markets. I calculated BTC price needed for RSI 95 at April close:$92K. Let's see what the Coinbase IPO will do today\ud83d\ude80 PlanB April 14, 2021"},{"title":"SSH-Notes","category":"Technical/Developer Tools","url":"ssh-notes.html","date":"21 April 2021","tags":"ssh, linux, security ","body":"Table of Contents TLDR Setup SSH-Agent to prevent Authentica Passwords and\u00a0Keys Handshake Background Source TLDR Public key only on the remote\u00a0ser ssh-keygen -t rsa Generate a key pair and keep the private key privately on your local The new keys are added to ~/.ssh/id_ and You could reuse an existing key pair but if it gets compromise you\u2019ll need to reset cat to upload or copy a public\u00a0key Copy all the output from the relevant line: ssh-rsa /dev/null"},{"title":"The trouble with climbing higher is that eventually you lose sight of the\u00a0ground.","category":"snippet","url":"climbing-higher.html","date":"16 April 2021","tags":"advice, thoughts, meta ","body":"."},{"title":"I Leaked Credentials Onto A Public GitHub\u00a0Repo","category":"Technical/Engineering","url":"i-leaked-credentials-onto-a-public-github-repo.html","date":"15 April 2021","tags":"hack, github, service-account, keys, security ","body":"Table of Contents Don\u2019t post secrets to public Background The\u00a0hack Remediatio Questions Study Comments Don\u2019t post secrets to public I made this mistake a while ago, and in the interests of openness and learning from others, I\u2019d like to describe what happened. Maybe it\u2019ll help others avoid the mistake, and maybe I\u2019ll learn something from any conversati this Background Using Google Cloud Platform (GCP), I\u2019ve been doing some work across multiple compute instances. Thankfully the work wasn\u2019t business critical or on production systems. My account was isolated away from the rest of the\u00a0busine As the number of servers I was working with increased, I realised I needed to begin using some tools to automate server setup. This lead me to begin using Ansible, and once I\u2019d cobbled together a working playbook I pushed my Ansible project to my GitHub account\u2026 And accidental leaked the key for an account I\u2019d been\u00a0using The\u00a0hack Within a couple of minutes of pushing the repository to GitHub\u00a0I: Made the Stopped tracking the keys in git and removed them from the cache git rm -r --cached . Received an email from Google saying they\u2019d found OK, close call. The secret was leaked for less than 5 minutes. On my obscure I thought there was nothing to worry about.. But then I noticed some activity in the\u00a0consol Compute instances were being created, I could see the list growing rapidly. Over the next few minutes 195 compute instances and disks were being created, each with a unique name in zones across the world. The format of the name was Where type was either applicatio backup, jenkins, gke, prod, staging, worker, www, build, redis, or runner. Maybe some others too. The number seemed to be 5 random\u00a0dig Some of the instances were ephemeral. They all had delete protection enabled. I checked the details of a few of them and noticed some scripts that included references to\u00a0Monero. So I guess a Monero mining bot was being set\u00a0up. The logs showed that GKE and networking resources had also been requested, but the account which the stolen credential belonged to didn\u2019t have the necessary permission Our project also maxed out its quota of compute instances in multiple regions and\u00a0zones. Remediatio I deleted the account that had been leaked, and began quantifyin the damage. I wanted to know exactly what permission the key had, which resources could be created, and could the leaked account be used to create other accounts? No, it\u00a0can\u2019t. After looking around and becoming confident that it was only 195 compute instances with disks and delete protection that had been created, in regions and zones across the globe, I began to remove them. No other resources had It took me 10 minutes and some googling to create the Get all the compute instances and dump them into a file. I expected to run a script that iterated through the file line by line, setting variables based on the content of the current line: gcloud compute instances list --format zone)' > names.txt In Vim, find the rows that contain the instances that I don\u2019t want to delete, and remove these from the file. There are a handful of compute instances I want to keep, and 195 that I want to remove. :v/node- shows any rows that don\u2019t Loop through the file and for each row, which contains the instance name and its\u00a0zone, Remove Delete the\u00a0instan while IFS=, read -r name zone do gcloud compute instances update $name --zone$zone \\ && gcloud compute instances delete $name --zone$zone --quiet done < names.txt The --quiet flag is necessary because otherwise gcloud will ask me to confirm that I want to delete the Questions I\u2019m surprised by the speed with which the attacker found the leaked credential The repo did not belong to the clients account but my own, and I assume that my account is obscure enough to not be on any interestin lists. If my account is being scanned every few minutes, presumably all accounts are being How many resources are required to do that? I guess if one of these attacks works you can use the stolen compute to scan more repositori for more leaked credential It\u2019s easy to imagine scenarios where large corporatio that are already running complicate cloud infrastruc deployment wouldn\u2019t notice a few (200?) unauthoriz compute Study Service accounts on Google SSH crash\u00a0cour Vagrant crash\u00a0cour IFS= Comments There was some useful discussion about this article on\u00a0Lobste."},{"title":"Broot","category":"snippet","url":"broot.html","date":"13 April 2021","tags":"broot, macos, cli ","body":"Broot is a tool that shows the contents of a directory on one screen, even its got lots of files"},{"title":"Adding Keyboard\u00a0Navigation","category":"snippet","url":"adding-keyboard-navigation.html","date":"12 April 2021","tags":"blog, jam, jquery ","body":"I added keyboard navigation to my blog and it works really well. Now I find myself trying to use the same shortcuts on other\u00a0site"},{"title":"Ansible","category":"Technical/Developer Tools","url":"ansible.html","date":"12 April 2021","tags":"ansible, servers, ssh, automation ","body":"Background I\u2019ve been spending a lot of time lately working on nodes for various blockchain projects (Polkadot, Cardano, Tron, Binance Chain, Ethereum, \u2026). The rosetta api spec is super interestin but like most things in crypto the documentat is sometimes wrong or incomplete and there are bugs and Each of the nodes runs on a separate server, and we typically have one node for mainnet and another for testnet. I\u2019m working across mutiple servers, doing difficult stuff, and I want it to be as easy as\u00a0possibl I need to reduce friction and Accessing the servers is easy - I use Tmux with the continuum and resurrect plugins and maintain different sessions for each type of server. This makes accessing multiple servers during the same work day really simple and effortless But working on the servers is still\u00a0awkw On my dev machine I have zsh with syntax highlighti command completion and various tools, like z to make navigation supper easy. I also have a lot of aliases defined. E.g. .. \u2192 cd ... Working on a remote server should be as convenient and familiar as working on my local machine, so I want to find a way to configure a server the same way as my laptop, and I want to do it automatica so that it can be done many times, with no Ansible Ansible seems to be It\u2019s\u00a0free It\u2019s got all the features and capabiliti you\u2019re going to\u00a0need It\u2019s agentless - you don\u2019t need to install anything on the machine you want to control - you can use Ansible with anything that you can ssh\u00a0into. I used the following resources to get\u00a0starte This useful video gave me some orientatio and helped me figure out what I was aiming for and how to get started. Before watching it, I didn\u2019t know \u201cwhich way was\u00a0up\u201d. This blog post showed me how to create an inventory using the gcp_comput plugin. I spent a lot of time being unnecessar confused about service accounts. I guess until you have 1 success at understand something you don\u2019t know if you\u2019ve misunderst by a little or a\u00a0lot. Once you have an inventory of servers that you want to connect to, you still need to specify (and prepare for) how you will connect to them. I\u2019d hoped that the gcp_comput plugin would do some heavy lifting for me in this step, but it seems not. It can do lots of useful stuff like creating instances and specifying disk space and networks, but it won\u2019t really help you ssh into an instance. No matter\u00a0tho This blog post turned out to be just what I needed. I found it at the beginning of my search when I was trying to create an inventory, and discarded it as almost useful. Turns out that OS Login is the best way to ssh into a GCE instance and once you\u2019ve got your inventory taken care of, this blog post really\u00a0hel When I was installing python modules, I had some errors about pyenv shims being incorrect. The scripts were looking for versions that weren\u2019t present. Running pyenv reshash fixed it. Kind of magically, but\u00a0annoyi Setting up a service account and giving it the correct permission took more time and was more confusing than anything to do with\u00a0Ansib I found this blog post about setting up vim for yml files. The preferred way to install ansible on Mac is using pip. When you use OS Login the username you have when you ssh into the compute instance will change. This SO question explains\u00a0w Commands gcloud auth list ansible-co view|list| -i --graph ansible -i all -m ping"},{"title":"Github\u00a0Actions","category":"snippet","url":"github-actions-blog.html","date":"10 April 2021","tags":"github, blog ","body":"I should see if I can use GitHub actions to generate html from markdown and run some shell and"},{"title":"Socially Acceptable\u00a0Mistakes","category":"snippet","url":"socially-acceptable-mistakes.html","date":"10 April 2021","tags":"meta, thinking, advice ","body":"It\u2019s more socially acceptable to make mistakes and errors of omission"},{"title":"du is a tool for showing disk\u00a0usage.","category":"snippet","url":"du-command.html","date":"7 April 2021","tags":"cli, unix, macos ","body":"There is a similar tool, with a list of other similar tools here"},{"title":"Safe Bash\u00a0Scripting","category":"snippet","url":"safe-bash-scripting.html","date":"6 April 2021","tags":"bash ","body":"An example of a safe, good, robust bash file\u00a0skele"},{"title":"Running a Binary in\u00a0Debian","category":"snippet","url":"running-a-binary-in-debian.html","date":"5 April 2021","tags":"debian, binary, path ","body":"I was running a binary in Debian that was complainin about an environmen variable not existing. I moved the binary into a $PATH directory and logged in as a sudo user. Why did this solve the\u00a0proble"},{"title":"Over-Engineering this\u00a0blog","category":"Technical/Web","url":"over-engineering-this-blog.html","date":"5 April 2021","tags":"blog, javascript, self-reflection ","body":"Over the last few weeks I\u2019ve spent an unreasonab amount of time and energy making unnecessar improvemen to this\u00a0blog. Some of these Adding keyboard shortcuts (type ? to find out\u00a0which) Implementi then and then optimizing client side fuzzy\u00a0sear Using src-set to serve responsive images Lazy loading images to make this site load\u00a0faste Compressin page files using brotli and also gzip (Precompre Trying (and ultimately failing) to avoid a \u201cwhite flash\u201d when dark mode is chosen and a new page loads (Github discussion I\u2019m not really sure why I did it. It makes almost no difference to anyone but me. It felt a I like tinkering, and it\u2019s nice to build something that will continue to work with no maintenanc I tell myself that over the next few years I will gain the benefits of these features even when I\u2019ve forgotten I It\u2019s taught me a lot of JavaScript which is a great language to be familiar with - it\u2019s everywhere I would warmly encourage someone younger than myself to pursue interests for the sake of curiosity and fun. And there is a very high chance that even if no-one uses the shortcuts except me, my new javascript skills will come in useful But even if they do I\u2019m not sure its a good enough reason - things should be built when they solve a present problem, not for what-ifs and maybes. YAGNI. I wouldn\u2019t let myself do this in a profession capacity. There is a tension between being curious and I\u2019m not really sure that I need to justify myself. Its a hobby, I wanted to do it, I enjoy tinkering with web technologi and learning new\u00a0things But also, I lost sleep over this - I stayed up too late, and let it put pressure on other\u00a0thin I know that being curious, and making room to play with interestin things, has been one of the most useful approaches to personal developmen and up-skillin myself. But there must be a\u00a0limit.. There is a tension between wasting my time and taking a risk, and it will take a few years before I know for sure if these efforts were worthwhile or\u00a0not. If it\u2019s not fun, don\u2019t do\u00a0it. Successful business owners seem to be very good at leaving things alone once they\u2019re \u201cgood enough\u201d, and not being In fact, I think that being a perfection is antithetic to being an entreprene I am not a perfection I\u2019m just really curious and have a big appetite for\u00a0learni But this \u201cappetite for learning\u201d stops me from focussing. I let myself become distracted by adding new features to this blog, when instead I should zoom out a bit and think about working towards a more substantia and meaningful goal, to the exclusion of more minor\u00a0goal I think that good entreprene are very focussed, to a fault. I am not that focussed. I am too distracted by\u00a0life. It\u2019s a balancing act, there is a tension between being emotionall and physically present with my family and friends, and ignoring as many things as possible so that I can focus on doing something meaningful that is"},{"title":"Fuse\u00a0Search","category":"Technical/Web","url":"fuse-search.html","date":"5 April 2021","tags":"fuse, search, web ","body":"Adding search made the site feel faster and more accessible I\u2019ve reimplemen search on this site using fuse.js instead of tinySearch You can read about how I implemente tinysearch here. When I first implemente search I was surprised how much faster and more accessible the site began to feel. I could quickly access any content by typing a few words, I didn\u2019t need to scroll or follow a link.1 This means I can find content without having to think about how to get there - I don\u2019t need to break my flow or concentrat It might sound like a trivially small considerat but lowering friction or cognitive load in small ways can make the difference between using or not using something when you\u2019re already working hard or concentrat on something else. For example, if I want to look up my notes about using the nohup command, I can quickly go to the site, type / (the keyboard shortcut for search), type \u201cnohup\u201d and hit enter. This is all muscle-mem level impulses. I don\u2019t need to think about the content, think about its category or when I posted it, then scroll down and scan a list, or use a mouse to click on intermedia links. Win. Working at the speed of thought rather than the speed of input is a big deal. Why I switched from tinySearch to Fuse.js Before implementi fuse.js, this site had a search feature powered by TinySearch I wouldn\u2019t have had enough knowledge to implement fuse.js if I hadn\u2019t already learnt some JavaScript whilst setting-up tinySearch TinySearch had an example for Pelican Blogs, and a simple and clear readme. By using tinySearch first I saw an example of how to build the JSON array that becomes the search index, and how to implement the javascript that\u2019s required for client side search. Also, in the course of developing and this blog I\u2019ve become much more proficient and comfortabl with JavaScript (and jQuery) in general. Fuse.js is really quite simple to set up once you\u2019re familiar with JavaScript It\u2019s much more flexible than tinySearch you can choose search weights for different fields, accuracy thresholds and some parameters for the fuzzy search algorithm. The general approach is to instantiat an instance of Fuse by calling Fuse with a JSON array for it to parse, along with some options. You then give the instance a string and get back an array of results which you can do whatever you want with. The accuracy of the search results is higher with fuse.js and the speed is still acceptable I did have to do some optimizati of the search index that Fuse generates, though. Optimizing the search index The \u201cnormal\u201d search index that Fuse uses to return results is a JSON array of all the content of all the articles that you want to be able to search. You can generate it using a jinja template or any other way you want. (There simply needs to be a JSON array that the browser downloads and does a fuzzy search on). This gave me a file that was about 4MB. Once I asked Fuse to search the complete text of each article (not just the default first 600 chars, iirc) then speed really suffered. I optimized the index file in the following three ways: Removed any non-words. Some of my articles are jupyter notebooks that have been converted to articles (the plugin to do this is one of the reasons why I began using Pelican). When the index is built, lots of code and html gets included, which isn\u2019t helpful. Any \u201cwords\u201d that are more than 20 chars I just delete. Removed the 150 most common words. Any word that is in many articles is not useful for distinguis between different articles, so they can be deleted from the index. They don\u2019t add any meaning. I wrote a short pipeline of shell commands using tr, sort, uiq to generate a file with a list of the most common words. I then wrote a python script to update the original search index by removing all the common words. Shortened any long words by only keeping the first 12 characters If a word was 15 characters long, I simply removed that last 3 chars. I figured this would work fine because matching the first 12 characters would already be quite unique and give a good result. Doing these 3 optimizati reduced the file size by about 90%. Compressin the JSON using gzip or brotli makes the files even smaller, and now the amount of data transferre to the client seems reasonably small. (This is a static site, and therefore search has to happen client side.) The browser would still begin to lag as the search string length increased. It takes more time to search for a 10 character string than for a 5 character string, and initially fuse was doing a search every time a character was typed. I wanted the site to feel as fast as possible and thought that if search was paused whilst typing and occurred a short time after the last key was pressed this would be an improvemen I added a short delay of 200ms to the function call, and typing during the delay time resets the time. This reduced the lag and made the search tool feel responsive I learnt that this is called \u201cdebouncin There was some further complexity when I wanted to debounce characters used for searching, but not the navigation or keyboard shortcuts. Getting the debounce function to only run on some key presses was surprising complex. It taught me a lot of JavaScript though, and it\u2019s satisfying to have made a useful user interface. also immediatel gave me the idea to add keyboard shortcuts. Type ? to see what happened \u21a9"},{"title":"Creating\u00a0Slowly","category":"snippet","url":"creating-slowly.html","date":"1 April 2021","tags":"meta, thinking, advice ","body":"As a hacker, or creator, or whatever the best label is, I always want to create something (usually code) and have it\u00a0finishe But a strange creativity and productivi boost comes from dabbling, dipping in and\u00a0out. I think that if the technical challenges aren\u2019t too hard, then the main criteria for success is\u00a0creativ Creativity needs time away from the project, and sleep, to bubble up and let ideas\u00a0grow Ultimately the most successful path is usually the most interestin because success has more consequenc than failure. \u201cInteresti requires elements of novelty and surprise, and without creativity these elements can\u2019t flourish.\u00a0 Dabbling results in more creativity than 6+ hours of strenuous work, and is more likely to give you"},{"title":"Arrow syntax in\u00a0bash","category":"snippet","url":"arrow-syntax-in-bash.html","date":"1 April 2021","tags":"bash, syntax ","body":"bar << foo bar will stop reading input when it reached\u00a0\u201cf bar <<< \"foo\" foo is all the input. bar wont bar < <(foo:list process subscripti Kind of like piping in the output of Stack\u00a0Over"},{"title":"Pretty print JSON","category":"snippet","url":"pretty-print-json-in-typescript.html","date":"31 March 2021","tags":"typescript, json, syntax ","body":"null, 2)});"},{"title":"Vim: GoTo Tag\u00a0Definition","category":"Technical/Developer Tools","url":"vim-notes-goto-tag-definition.html","date":"31 March 2021","body":"Update (2021-03-3 Just use neovim.coc instead of YouComplet or Syntastic. It\u2019s faster, easier to setup, and works intuitivel ALE is still wonderful and useful, though there\u2019s a lot of overlap - coc can lint as well. Jump Lists and Change Lists If you\u2019re going to be jumping around to where things are defined, you will need to know how to jump back again. It seems there are two lists you need to be aware of, the jump list1 and the change list2. Jump List A list of locations that the cursor has jumped to. move up the jump list mode down the jump list Jumping to a definition or a search result Change List g; and g, \u2192 move up and down the change list A list of locations where a change was made. '. \u2192 go to the location of your last edit (. is a mark). '' \u2192 go back to where you were before your last jump Original Post: There are multiple ways of doing anything with vim, including going to where a function or object is defined, and I usually need to do something at least 3 times before I can do it without breaking my focus or train of thought. My memory is hazy but I remember spending a 1/2 day looking into this and considerin which solution I wanted to commit to.3 My options seemed to be between YouComplet and ALE. [Update!4] I can\u2019t remember everything I read and tried, but I trust my conclusion Looking in my .vimrc I see that I have x mapped to :YcmComple GoTo and it works just fine, even when a module is imported from somewhere outside the current project. The tool was working and ready to use, I just hadn\u2019t internaliz it yet. Commands to remember: x - GoTo definition - YCMs best guess at an \u2018intellige goto command, whether its a declaratio or a definition - Toggle tagbar :help jumplist \u21a9:help changelist \u21a9The more powerful the tool, the more worthwhile it is to take a closer look at what it can and can\u2019t do. \u21a9YCM and ALE work fine for goto definition and linting, but they don\u2019t give me satisfacto looks like it might offer some improvemen \u21a9"},{"title":"Useful\u00a0Business","category":"snippet","url":"useful-business.html","date":"30 March 2021","tags":"entrepreneur, saas ","body":"looks like a really"},{"title":"Frantic\u00a0Distraction","category":"Snippet","url":"frantic-distraction.html","date":"30 March 2021","tags":"meta, thinking ","body":"Frantic distractio via productive is exhausting and\u00a0useful"},{"title":"Rearrange splits in\u00a0Vim","category":"snippet","url":"vim-split-rearranging.html","date":"29 March 2021","tags":"vim ","body":"x - swap buffers, but keep arrangemen the\u00a0same H - make the current split cover the left of the\u00a0screen J, K, L covers the bottom, top, right of the\u00a0screen blog\u00a0post stack\u00a0over"},{"title":"Pelican Plugin\u00a0Guide","category":"snippet","url":"pelican-plugin-guide.html","date":"29 March 2021","tags":"pelican, plugin, guide ","body":"A guide about writing plugins for\u00a0Pelica Thanks @geographe"},{"title":"Read and Write the Same File in\u00a0Bash","category":"snippet","url":"read-and-write-same-file.html","date":"24 March 2021","tags":"shell, pipe, syntax, bash ","body":"I tried to read and write the same file in a pipeline, and got caught out by a race condition (why is the file empty?!). Do this\u00a0inste some_scrip < file > smscrpt.$$\\ && mv smscrpt.$$ file || rm smscrpt.$$|| removes the temporary file if it\u00a0errors.$$ is the process ID and ensures that you always have a unique temporary file\u00a0name."},{"title":"JSON\u00a0tools","category":"snippet","url":"json-tools.html","date":"24 March 2021","tags":"json ","body":"jj - A stream editor jq - A json processor python -m json.tool I like jq for pretty printing JSON output, jj for making JSON pretty or\u00a0condens This was really useful when optimizing the search index for this\u00a0blog."},{"title":"Docker\u00a0Commands","category":"snippet","url":"docker-commands.html","date":"17 March 2021","tags":"docker ","body":"docker run -d ... docker logs -f docker run -it ... docker run -itd docker container attach -> detach from container interactiv stack\u00a0over"},{"title":"Python\u00a0Notes","category":"Technical/Developer Tools","url":"python-notes-2.html","date":"17 March 2021","tags":"python, learning notes ","body":"__call__() In Python, every time you call a function or method such as my_functio or the interprete will replace the ( with .__call__( >>> def >>> return x+1 >>> 3 class Prefixer: def __init__(s prefix): self.prefi = prefix def __call__(s message): return self.prefi + message Then use prefixer like\u00a0this: >>> simonsays = says: \") >>> up high!\") 'Simon says: jump up high!' Every time you call a function or method, you\u2019re really just calling a built in __call__ method. There should be one, and preferably only one, obvious way to do\u00a0somethi It\u2019s in the \u2018zen of Python\u2019, which is a set of guidelines that help make design decisions. It\u2019s a choice that Python made, and other languages do There are different levels to languages and this applies more to the idiom level than the design pattern level. It applies even less at the architectu level where there can be several equally good ways of organizing business logic and Perl has the \u201cTMTOWTDI\u201d (tim towtdi) principle - \u201cThere\u2019s More Than One Way To Do It\u201d. Perl\u2019s philosophy is to give users more than one way to do\u00a0somethi"},{"title":"Adding\u00a0Search","category":"Technical/Web","url":"adding-search.html","date":"12 March 2021","tags":"blog, search, tinysearch, web ","body":"I\u2019ve added search to this blog. Results are generated as you type. Try it by typing / or cmd-k. If you look on the Pelican plugins index you\u2019ll see that Tipue search is the only search tool with a ready-made Pelican plugin, but unfortunat the project seems to have died and the projects website is now But searching a static site must be quite a common need and googling for alternativ gave me a few choices. Lunr.js seems to be the most popular, but it also seemed fairly complicate and like it was probably more than I needed. I went with Tiny Search because it seemed to do what I needed and was easy to setup. There\u2019s even an example for Pelican\u00a0bl One hurdle to success was minimising the false positives. The default settings seem to prioritise keeping the size of the index small (tiny) over giving a good user experience Maybe its because the amount of text on my site is significan less, or more, than the typical use case. Either way, after checking the project\u2019s issues on Github I found an issue that matched my problem perfectly. The solution is to increase the tiny_magic variable at build\u00a0time According to the Readme, this requires using a container and building the index using docker run.... Unfortunat the Dockerfile wouldn\u2019t complete without errors. Checking the issues again and adding to the discussion resulted in an alternativ Dockerfile being suggested, which works. Woohoo! I could then build the search index with a massive tiny_magic value\u00a0(204 Then something weird happened. I write in Vim and I use fzf to find and open files. I realised that fzf had stopped working. After some investigat I realised it was only not working in the blog project, and that fzf.vim calls the fzf CLI tool, which in turn calls the ripgrep tool. The underlying issue was that ripgrep wasn\u2019t working, and after a few hours (sob) of debugging, I found out that one of the things that makes rg special is that it ignores stuff in your .gitignore file. Sneakily, and without me noticing, the Docker image for constructi the tinysearch files had created a .gitignore file with a single entry. The entry was *, which selects everything So rg was ignoring everything and giving no results. Which meant I couldn\u2019t find and open\u00a0files I still don\u2019t know how (or which part of) the Dockerfile does this, so I\u2019ve created a file which contains the correct content, and after I generate a new search index I replace the new traitorous .gitignore with the contents of I\u2019ll come back to it later when/if I have a better understand of Dockerfile syntax, or\u00a0Rust. Adding search to the site made the content feel a lot closer and more accessible Once it was working I immediatel wanted to use some keyboard shortcuts to open the search box and select results. Kind of like does it. It feels really fast and\u00a0precis Googling for some jquery packages, and also some vanilla javaScript showed me enough to get things working. You can hit / or ctrl-k or cmd-k and bring up a search box that populates results as you\u00a0type! Only whole words are matched unfortunat but its still a super useful feature. The search index includes article content as well as article titles and categories I\u2019d like to tweak a few of the keyboard shortcut behaviours and add the contents of various pages (which aren\u2019t articles) to the search\u00a0ind Update I\u2019ve reimplemen search using fuse.js. You can read about it here"},{"title":"Pipe a Script File into\u00a0Bash","category":"snippet","url":"pipe-a-script-files-into-bash.html","date":"11 March 2021","tags":"bash, syntax, shell ","body":"Probably it\u2019s one you just curl-ed curl -sSfL | sh -s"},{"title":"Split Long\u00a0Strings","category":"snippet","url":"split-long-output-onto-multiple-lines.html","date":"10 March 2021","tags":"bash, linux ","body":"Split long strings (or command outputs) onto multiple lines Find and replace a particular char (maybe :) with a \\n. ... | tr ':' '\\n' ... | sed 's/:/\\n/g'"},{"title":"ripgrep\u00a0Regret","category":"snippet","url":"ripgrep-regret.html","date":"10 March 2021","tags":"ripgrep, fail ","body":"Without noticing, create a .gitignore file with a single * in\u00a0it. Spend a day trying to understand why ripgrep has stopped working for only 1\u00a0project. \ud83d\ude2d\ud83d\ude2d\ud83d\ude2d"},{"title":"Teaching Kids About\u00a0Money","category":"snippet","url":"teaching-kids-about-money.html","date":"5 March 2021","tags":"parenting, kids, money, teaching ","body":"Teaching my kids about money and work is having an\u00a0effect. Yesterday, my daughter made a painting for me and asked my to buy it using pretend\u00a0mo"},{"title":"Stop Prepending sudo to Docker\u00a0Commands","category":"snippet","url":"stop-prepending-sudo-to-docker-commands.html","date":"5 March 2021","tags":"sudo, docker, linux ","body":"sudo groupadd docker -> make the group sudo gpasswd -a$USER docker -> add $USER to the docker group newgrp docker -> activate the changes"},{"title":"cat and a new\u00a0line","category":"snippet","url":"cat-and-a-new-line.html","date":"5 March 2021","tags":"cat, linux, bash, shell ","body":"If you\u2019re cat-ing a file and the bash prompt doesn\u2019t start on a new line (cos the file you displayed using cat doesn\u2019t end with a new line char) the following will fix\u00a0it: cat ; echo"},{"title":"Cardano: Generating\u00a0Addresses","category":"Technical/Cryptocurrencies","url":"cardano-generating-addresses.html","date":"5 March 2021","body":"If many different customers are to deposit or send ADA (The unit of currency on the Cardano blockchain to a Cardano node, it will be necessary to determine which customer is responsibl for each transactio so that the correct customer account can be\u00a0updated As with many things involving blockchain this initially seemed like a simple requiremen but involved several hours of\u00a0work. Cardano wallets are generated using a parameter called The default value is 20, and is the number of unused addresses that the node will generate and return to a client using the REST API. If one of the addresses is used, the node will automatica generate another so that there are always 20 This is probably very convenient for personal use. If I want someone to send me some funds, I can make a simple api call using cURL and get a fresh address. But if you are running a service, weather its e-commerce or a financial service, its not really good enough. Some advice on the forums says to generate a wallet with a very large value such as 10,000 and just generate a new wallet when you run out of fresh addresses, but it still feels like a\u00a0compromi But lets explain our situation in more detail first. If a customer wants to send us some ADA, we want to give them a fresh address that\u2019s never been used before and that only they have. Then we know that any funds that arrive to that address are from a particular customer. However we don\u2019t know if the customer will actually use the address and transfer any funds. The address might remain unused or it might not. Neverthele that address is now reserved for them, and no one else can use\u00a0it. In this way, we might need to generate and maintain a list of thousands of addresses that are never used. Using for this seems like a bad\u00a0soluti Fortunatel has the answer, albeit in a fairly convoluted and obscured form. If you have the mnemonic that was used to generate a wallet originally you can generate 2^31 unique addresses like\u00a0so: Clone the repo and build the docker\u00a0ima git clone docker build -t . Get the mnemonic and generate a file containing a list of space separated words on one\u00a0row. Run the\u00a0follow export increment= && cat mnumonic.t | docker run --rm -i key Shelley | docker run --rm -i key child | docker run --rm -i key public | docker run --rm -i address payment --network- testnet > payment.ad && cat payment.ad ;echo"},{"title":"creating users with sudo\u00a0permissions","category":"snippet","url":"creating-users-with-sudo-permissions.html","date":"4 March 2021","tags":"sudo, linux, user, admin ","body":"adduser -m usermod -aG sudo CentOS: adduser -m passwd usermod -aG wheel (wheel is a usergroup with"},{"title":"Two Years Of\u00a0Vim","category":"Technical/Developer Tools","url":"two-years-of-vim.html","date":"4 March 2021","body":"I\u2019ve been feeling very comfortabl with my Vim + Tmux setup recently. Navigating around shells and files isn\u2019t taking much mental effort It\u2019s taken about 2 years of working full time with vim to get to the stage where the commands are so I pepper text files outside of vim (email, notes, etc) with vim keys accidental - j k x etc I can\u2019t remember what the command is to do something if I\u2019m not actually doing it. When I need to do an action, I do it from muscle memory and I only pay attention to the underlying key press if something goes\u00a0wrong This is noticeable when trying to find an unbound key combinatio for some new action, or when reading an article about vim and thinking \u201cthat\u2019s new\u201d when actually I\u2019ve been doing it A pleasant surprise has been that it doesn\u2019t take much effort to rebind a single command and retrain myself to use it. This is presumably because the mental effort for all the other commands has become negligible In the early days, retraining a key combinatio took a lot more effort because I was already making an effort to get used to doing things in\u00a0Vim. I can work even when my vision is blurry (and my speech slurred and my head heavy) because I can use text objects and navigation commands to get to where I know text is. I\u2019m not saying I should work when I\u2019m that tired, but I can, if I\u2019m already familiar with"},{"title":"Disk Full and Disk Usage\u00a0Commands","category":"snippet","url":"disk-full-and-usage-commands.html","date":"3 March 2021","tags":"linux, du, df, shell, cli ","body":"df -h Show disk\u00a0space du -hs . See how big the current dir\u00a0is"},{"title":"Git LFS","category":"snippet","url":"cloning-git-repos-using-lfs.html","date":"3 March 2021","tags":"git, git-lfs ","body":"Cloning large repos, or repos with large files in them, doesn\u2019t work with git clone ... you need to use git lfs clone ... So why is git lfs clone deprecated What\u2019s"},{"title":"Binance-Chain: Running a\u00a0node","category":"Technical/Cryptocurrencies","url":"binance-node-api.html","date":"3 March 2021","body":"I\u2019ve been setting up a binance-ch node. Unlike Polkadot or Cardano, I\u2019m not going to run it from a container until it\u2019s working reliably. The Binance docs show a couple of ways to install a node. I used the install.sh script and went with default values as much as possible. Installati My first attempts to sync a full node used the install.sh script, but the node wouldn\u2019t sync completely it would get stuck. I setup a new VM and did a manual install (\u201cOption Two\u201d) and so far the node has been syncing without any issues. You need to download the genesis file separately in this case. Also, be sure to download the node-binar repo using git lfs and not just git. It will look like it worked but bnbchaind wont have completely downloaded unless you use lfs It took me a while to realise that the documentat assumes that you have an environmen variable called BNCHOME. You can either create it using export (like you would for any environmen variable) or replace the environmen variable in the start node command with the file path: nohup bnbchaind start --home BNCHOME & Note: I\u2019m not sure if the bnbchaind needs the environmen variable to be set or not. It doesn\u2019t give errors if it isn\u2019t set, but I seem to be having more success when BNCHOME is defined. Syncing the node There are three ways to sync a node. Fast-sync isn\u2019t the fastest way to sync your node, hot-sync is. Using install.sh should put the correct default values in the file, but I needed to adjust ping_inter and pong_timeo to the recommende values. Surprises The documentat assumes you have familiarit with running tasks in the background of a terminal session, and that you\u2019re familiar with nohup. I wasn\u2019t - I\u2019d even forgotten what the & symbol does1 so I did some research and wrote some notesIt starts a process in the background You can move it to the foreground with fg or see a list of running jobs using jobs. You can move a running job to the background (like a vim session) using ctrl-z \u21a9"},{"title":"nohup","category":"snippet","url":"nohup.html","date":"2 March 2021","tags":"linux, cli ","body":"Use nohup to keep a curl process running even when the terminal (tty?) session autocloses at\u00a03am."},{"title":"Shell\u00a0Comparisons","category":"snippet","url":"shell-comparisons.html","date":"2 March 2021","tags":"zsh, bash, bsh, linux, cli ","body":"You can group shells into\u00a0group ksh - korn shell and\u00a0zshell sh - bourne shell and bash (the bourne again\u00a0shel Because zsh isn\u2019t a superset of\u00a0bash. bash is a superset of the bourne\u00a0she"},{"title":"nohup and Background\u00a0Processes","category":"Technical/Developer Tools","url":"nohup-and-background-processes.html","date":"2 March 2021","body":"Stop stuff\u00a0stop If you run a command in a terminal session and the terminal session is disconnect then the processes running in it will also be\u00a0termina I discovered this when I was trying to download a ~500GB database overnight. I logged in the next morning expecting to see a completed download, but found I only had half the\u00a0file. Use nohup to ignore HUP signals One solution to this seems to be to use nohup, a command that ignores the HUP signal. It stops your programme from stopping if the terminal session its running in is\u00a0stopped By convention the HUP signal is the method used by a terminal to warn dependent processes that it is about to\u00a0logout. You probably want to run nohup in the background You might want to prevent it from creating nohup.out. Close or redirect fd0 -\u00a0fd2 On Linux, nohup automatica closes stdin. If you\u2019re using MacOS or BSD this doesn\u2019t automatica happen, so you might want to redirect it yourself. This is because if a background process tries to read anything from stdin then it will pause itself whilst it waits for you to bring it to the foreground and type some input. This is probably a waste of\u00a0time. If nohup detects that you have redirected stdout and stderr then it won\u2019t create nohup.out. As with all commands, if you put & at the end of the command, it will run in the background You can bring it to the foreground by running fg, or see a list of jobs by running jobs. If you redirect input to /dev/null ( log.txt This will redirect stderr into a\u00a0file:$ asdfadsa 2> error.txt If you run a command that generates lots of error messages along with \u201cgood\u201d messages, you can redirect all the error messages (stderr) into /dev/null so that you can only see the useful stdout $grep -r hello /sys/ 2> /dev/null If you want to run a command and only see the errors, (stderr) then you can filter out all the stdout by redirectin the stdout messages to /dev/null:$ ping google.com 1> /dev/null Redirect all output into /dev/null if you want a command to run\u00a0quietl Redirect all the output. The command below redirects stdout to /dev/null (the default file descriptor is 1 if it isn\u2019t specified) and then redirects file descriptor 2 into file descriptor $grep -r hello /sys/ > /dev/null 2>&1 Read input from a file instead of the\u00a0termin 0>logfile Combining 2>&1 means send stderr wherever stdout is going. This means that you\u2019ve combined stdout and stderr into one data stream and you can\u2019t separate them anymore. It also means you can pipe stderr the same as you can stdout. Input You can redirect stdin similarly. If you run Ranges Searching Undo Splits Macros Other Verbs s - delete char under cursor and enter Insert\u00a0Mod r - replace char under\u00a0curs c/hello - change until next occurrence of\u00a0\u2018hello\u2019 Registers \"ayy yank the entire row into register a. \"Ay yank to register A and append the new text to the existing contents of the\u00a0regist :registers - preview the contents of Insert\u00a0Mod - delete back one\u00a0word. - delete back to the start of the line or start of cgn - if you are searching for a word (either by using / or * or #) and you want to change each instance of the search result, use gn to change or delete and then go to the next result. This will let you use the .dot operator to repeat both the steps (moving and\u00a0changi 0 - paste. if there are new-line chars Normal\u00a0Mod or Select a column of numbers you want to increment, then g will turn them into an Ranges :put =range(1,1 - insert a list of :for i in range(1,10 | put | endfor - use a loop to generate a long\u00a0list. Searching g# or g* for partial matches, like # or * for exact\u00a0matc Search for the word under the cursor, or similar: Press /. - this will copy and paste the word under the cursor into the search box. Edit it as\u00a0necessa After you\u2019ve done your search, to jump back to where your cursor was\u00a0before Find and replace whole words only: Find and replace either old-word1 or old_word2: g - show some stats about current bugger - word count, line count, char\u00a0count Undo g- and g+ - undo\u00a0branc Under changes within a period of time: :earlier 2d - undo changes in the last 2\u00a0days :later 5m - redo all changes in the last 5\u00a0minutes :earlier 3f - undo all changes in the last three buffer\u00a0wri s seconds, m minutes, h hours, d days, f saves @a - Use the global command to execute macro a on all lines of the current buffer containing - For every line containing \u201cgood\u201d substitute all \u201cbad\u201d with\u00a0\u201cugly Splits r - rotate the splits from left to right but only if they are split vertically R - rotate the splits from right to left. H - move the current split to the far left and make it full height. J - move the current split to the bottom of the screen and use the full\u00a0width :only - close all splits except the current\u00a0sp Macros @o - do the macro stored in buffer O on all lines that match the Other in Insert Mode will jump you into Command Mode for one command only and then put you back into Insert The .dot command only repeats commands that changes the buffer content. It wont repeat"},{"title":"Notes From \u201cPowerful\u00a0Python\u201d","category":"Technical/Developer Tools","url":"python-notes.html","date":"21 January 2021","body":"The parts of Aaron Maxwell\u2019s Powerful Python newsletter that I don\u2019t want to forget: Table of Contents Emergent Abstractio Practioner Engineer, Scientist Sentinel Values Levels of Python Code Read PEPs Emergent Abstractio Get used to expecting and letting abstractio emerge from projects. If you find yourself repeatedly solving similar problems in similar ways, what can you do that will simplify the code and the Is it a couple of convenienc methods on some helper class? The code below gives you three ways of instantiat the twitter API client within the same class: A generic \u201cnormal\u201d way A specialize way that looks for certain environmen variables A specialize way that looks for a configurat file import os import twitter # class ApiClient: def __init__(s consumer_k self.api = twitter.Ap consumer_k @classmeth def return cls( @classmeth def path): with open(path) as config_fil # ... return cls(...) # ... Practioner Engineer, Scientist Practioner - You can use a thing (a framework, a tool) Engineer - You can use a thing and if you needed to, you could recreate it Scientist - You can create frameworks and paradigms that have never existed before Aim for the engineer level. Sentinel Values Instead of setting your sentinel value to something that is not quite impossible like None or \"None\" set it to object() This is better because it creates a unique instance of the object class and there can be no ambiguity about where it came from. A sentinel value is a value you can set a variable to. It\u2019s special because it differs from all other legal or possible values that the variable could have. It\u2019s used as a signal or as a canary that something (bad or unexpected has happened. Levels of Python Code Syntax - understand what indentatio is important, when you need parenthesi colons, etc Idioms - the building blocks of a program. \u201cParagraph of code that follow common patterns, like for loops, __init__() methods (boilerpla or context managers. Design Patterns - Less well defined that Idioms, but more useful. More info. Creational Patterns, like Factories Structural Patterns, like Adapters or Proxies Behavioura Patterns, like Visitor or Strategy These tend to be the same across different languages. Architectu - the largest structures in your software system. The language itself doesn\u2019t make a lot of difference an applicatio would have the same architectu whether it is written in Python or Java. The interface between different components would be different, but the \u201corgans\u201d of the body would essentiall be the same. Read PEPs A Python Enhancemen Proposal is a document that\u2019s written to propose a new feature of Python. It fully details the proposed feature, the arguments for and against it, and lots of sample code. If the PEP is accepted into a future version of Python, the PEP becomes the authoritat document for that feature and how to use it. PEPs tend to be written by the best programmer in the world, so hang out with them. Abstractio is a principal of OOP \u21a9"},{"title":"Mental Models I Used To\u00a0Use","category":"Non-technical/Learning","url":"mental-models-i-used-to-use.html","date":"20 January 2021","body":"The rules1 and mental models that helped me succeed in one season or phase of life may not be the best for the next phase. Here is a list of a few mental models I remember being concious of in previous years. Probably I\u2019ve already forgotten some. Always ask \u201cwhy\u2026\u201d. Be obsessive about this. It\u2019s going to make things harder for a while before things get easier. You\u2019ll find difficult answers that you otherwise wouldn\u2019t. If you\u2019re only concerned with the present then its true that ignorance is bliss, but otherwise it\u2019s a liability. \u201cWhat if\u2026\u201d is another good question to ask a lot. Adapt to the situation, don\u2019t make it adapt to you if you have any choice. Be kind of like water, going around things and through the gaps. Look for the edges and the gaps, the parts that aren\u2019t well known. Let people talk as much as they want to. Shut up and listen. If they mean you harm or don\u2019t respect you then it\u2019ll become more obvious the more they keep talking. If they mean you well or they\u2019re saying something useful, you will benefit more from letting them talk more. Inversion - it can be hard to know if you should do something, but how would you feel if you didn\u2019t do it, or if it didn\u2019t happen? Regrets are inevitable everyone has them. Same as making mistakes. Let your regrets be for things that you did do, and not what you didn\u2019t do. If you are willing to try something, fail at it, and still be glad that you tried, then you should almost definitely do it. Commit to it and enjoy the experience Don\u2019t be scared, or at least, be scared and optimistic and happy2. There is beauty and luxury in being in such a bad spot that you are backed into a corner with seemingly no way out. Things become black and white, instead of shades of gray, and that will make priorities and options much clearer. You are likely to work very efficientl and effectivel in this scenario, and you will learn important things about yourself. Now that I\u2019m older and I have I can\u2019t ever let things become so bad that a situation becomes black and white. I have to navigate a world of grays. If they do become black and white, I\u2019ll already have a long list of failings. When I was younger, things were more fragile. My resources were smaller and things could quickly flip from good to bad. Enjoy the few benefits that a situation like that gives, because (hopefully once its gone its gone for good. The best way to solve a problem is to prevent it from occurring in the first place. Succeeding at this will bring its own challenges Take responsibi for things you are not responsibl for, kind of. Do it deliberate and for your own benefit, but don\u2019t forget that you are only pretending that it\u2019s your If you do this, you will force yourself to understand a situation more deeply and from other peoples perspectiv This will let you learn faster and help you in future. Keep this at arms length though - it\u2019s make-belie and you need to be able to switch it on and off. It\u2019s a toy for you to play with. This seems to be what \u201cextreme is. I think its important to have mental models that you\u2019re comfortabl with, because it lets you make decisions quickly and consistent But understand that the map is not the territory, and these are just tools in a toolbox.or policies \u21a9Courage isn\u2019t the absence of fear. It\u2019s being scared and doing the right thing anyway. \u21a9"},{"title":"What\u2019s So Different About\u00a0Now","category":"Non-technical/Social","url":"whats-so-different-about-now.html","date":"20 January 2021","body":"I think we are less aware of our ignorance than previous generation It is easy to implicitly assume that all useful informatio is available to us, and that we are therefore more informed than we really\u00a0are I think this is because the internet has made informatio more accessible and global air travel have made the world feel\u00a0small Whilst an individual would hopefully never pretend to know everything I think its easy to assume that the right informatio exists and is being used by the people to whom it is\u00a0relevan But the accessibil of all informatio has put us in a situation similar to informatio scarcity. We still need to actively search for the informatio we want, because the informatio that comes to us easily or for free is not equal to what we find when we apply\u00a0effo I can easily have so many short pieces of news or informatio that I am always slightly overwhelme The pace of modern communicat encourages me to never slow down enough to form my own questions or frame my own arguments. I can always find an answer to my questions, but when was the last time I checked that whoever gave it to me wasn\u2019t going to profit from\u00a0it?"},{"title":"Predicting the Future using Human Nature and\u00a0Technology","category":"Non-technical/Social","url":"what-happens-next.html","date":"20 January 2021","body":"Predicting the future sounds like a tough problem, but we try to do it all the time without realising\u00a0 We predict the future when we think about how risky or scary something is, or when we think about what\u2019s really going to change because of an announceme or press release. We try to predict the future when we\u2019re at the supermarke checkouts and we try to pick the queue that will move the fastest. I always seem to pick the wrong\u00a0one. There must be a million ways of trying to predicting the future but all the good ones are models which reduce complexity and emphasise key One of them could be comparing the influence of human nature and technology on the outcome, and then comparing the event to what\u2019s Human nature doesn\u2019t change, so if something is driven by fear or greed then it probably doesn\u2019t matter what century it occurs in. Technology is change, and if something is enabled or prevented due to technologi progress then the date is\u00a0importa What is driving the scenario? Is it human nature or technology Supermarke checkouts are mostly manual and require a couple of adults to work together, so human nature has a much bigger role on efficiency than tech. Young men will stack and pack quickly, old women will be the opposite. What types of shopping bags they have, or how they pay, or even how many items they\u2019re buying, are probably not going to lead you to the The same probably works for getting through"},{"title":"Financial Doom And\u00a0Gloom","category":"Non-technical/Social","url":"financial-doom-and-gloom.html","date":"19 January 2021","body":"Financial crises seem to happen fairly regularly so they shouldn\u2019t be unexpected But no-one seems particular concerned about our current financial system, at the moment our attention is controlled by other threats. I\u2019m concerned that a lot of money has been injected into the money supply but we haven\u2019t seen any inflation. And I am concerned that the price of stocks is no longer related to the value created by the company but instead by macro economics. It\u2019s a terrible time to be a value investor. This should be an alarming statement. Value investing should always be a decent way to make money unless markets are broken. If the price of something doesn\u2019t represent its value then a correction is inevitable Interest rates are really low at the moment so if you have spare money and you want to make it work for you then where do you put it? Not into a bank account, because interest rates are low1, and not into government debt, because the yield is so low. It has to be stocks if you want the value of your investment to increase meaningful But everyone is doing this which drives the price up, and because their price is increasing they increase even further. I think that the main reason for concern is super low interest rates and massive increases in the money supply, but there are a couple of other factors that are also contributi It\u2019s easier than ever for retail investors to participat in the stock market, and this seems like a good idea. However if retail investors have influence to effect prices, and they themselves can be manipulate or influenced regarding what or when they buy or sell, then that is likely a new kind of threat to financial stability. We\u2019ve never seen social media combined with quick, cheap investment services for amateurs before. Index funds are also more popular than ever2 - the efficacy of index investing relative to traditiona funds that use stock pickers is very high over medium or long time horizons because index funds are much cheaper. But if index funds become too large then they end up influencin the market in predictabl and rigid ways. Index funds cannot choose what they buy or how much they buy - they just track the index. If a company\u2019s stock crosses certain thresholds their stock has to be bought or sold. It seems like its possible to create feedback loops where funds have to buy more of a rising stock, which increases its scarcity and price, which then requires index funds to purchase more of the same stock. The amount of euros in existence in 2019 was 90% more than in 2010.3 But inflation between 2010 and 2020 is 13%.4 Why is that? If the price of something doesn\u2019t represent its value, then a correction is inevitable are interest rates low? Because confidence in the economy is low, so central bankers have to lower interest rates to make it 1. Cheaper for a business to borrow money to invest in their business and therefore easier for a business investment to be profitable and 2. So that its more attractive for investors to use their capital to invest in a business (which grows the economy) relative to depositing spare cash in a bank account (which is safer but a less efficient way to deploy capital). Interest rates affect the relative risk-rewar ratios of different investment strategies \u21a9Index Funds Are the New Kings of Wall Street \u21a9statista \u21a9"},{"title":"Debugging the more_categories plugin for\u00a0Pelican","category":"Technical/Developer Tools","url":"debugging-more-categories-pelican-plugin.html","date":"19 January 2021","body":"I\u2019ve realised that one of the plugins I use to make this blog is not working correctly. I use the plugin to: add subcategor assign multiple categories to articles. Subcategor aren\u2019t working and Pelican thinks each article just has categories than contain forward slashes. In his \u201cPowerful Python\u201d emails, Aaron Maxwell recommends looking at the source code for popular python libraries to see how really good Python is written, and how talented developers write code and solve problems. This is a good opportunit to look at the code that powers the plugin and see if if I can: Understand the source code Locate the source of the problem Fix the problem I don\u2019t know if Pelican is amazingly good quality or not, I get the feeling it could do with more developer resources, but I\u2019ve got a real reason and motivation to look at the underlying code so I\u2019m going to give it a shot. The documentat is sparse which doesn\u2019t help, I get the impression that whoever wrote it feels like Pelican is simple and it\u2019s obvious what\u2019s going on 1. It\u2019s not obvious to me. Pelican Plugins Every plugin has to have a register() function, here it is for the plugin: def register() I understand the idea of signals from Django, and generators are discussed a bit in the documentat So what else is happening\u2026 As I write down my understand of the plugin, I\u2019m aware that my understand is definitely incomplete and probably wrong. I hope that as I progress I will see the mistakes in what I\u2019ve already written. is called first, and it takes two arguments, generator and metadata. The entire function is 3 lines so here it is: def metadata): categories = = for name in categories = It looks like it gets the category from the metadata for each article. Presumably by the time this function is called the articles have already been parsed and a metadata object has already been created and populated with metadata about the articles, including categories The first row of splits up the categories if multiple categories are listed. metadata must be a dictionary and there must be a metadata dict for each article, otherwise you couldn\u2019t just get get the value assoiciate with the dictionary key and then split the string on commas. This means that this function is called once for each article. I don\u2019t know what text_type does yet. Maybe it ensures that the output is always a string. It\u2019s imported from six which I remember seeing being a dependecy of some other packages. .. Having checked the documentat for six it looks like I was right - it represents unicode textual data in both python2 and python3. Pelican was originally written in Python2 I guess. Next step is to write a new key-value pair to the metadata dictionary for each article. This plugin adds functional to python by enabling categories and not just a category for each article. It seems clear that adding a categories key to the metadata dict is an obvious way to do this. The value for the categories key is a list where each item is an instance of the Category class. This class is instantiat using two arguments, name which is the string from the previous row, and which is currently not understood .. printing the contents of shows that its a dictionary of all the settings. Easily assumed and good to confirm. I\u2019ll dig into the Category class in a moment, but first lets quickly cover the last row of the function. The category attribute of the articles metadata is simply updated with the first item in the categories list (categorie must be a list because it can be indexed.) class Category() This class is the only class defined by the plugin (which is only 96 lines of code). It has 6 methods, 5 of them are decorated, and it has no constants. The decorators are property [3], _name.sett [1] and [1]. URLWrapper is imported from and I don\u2019t know what that does beyond \u201cwrapping URLs\u201d. @property Decorators are functions that takes methods or functions as inputs. Using property along with setter decorators lets a class have a property assigned to it whilst ensuring that arbitrary conditions or logic is upheld. If the @property decorator is over a method called foo, then there would need to be a decorator called foo.setter on a method somewhere in the class. That doesn\u2019t seem entirely right though, because in our Category class, we have a @property decorator over a _name method, and also a @_name.set decorator over another method called _name. But the other methods with @property decorators (slug and ancestors) do not have any associated setter decorators or methods. The setter for _name seems to create parent categories if the string contains slashes: @_name.set def _name(self val): if '/' in val: parentname val = 1) self.paren = self.setti else: self.paren = None self.short = val.strip( Here, self.paren becomes an instance of the category class, that is instantiat using parentname and self.setti This is recursive to however many levels of subcategor are specified. The ancestors and as_dict methods seem more confusing. ancestors isn\u2019t called or mentioned within the class definition but is called from the function which is called after the get_catego function returns. I don\u2019t understand why it needs an @property decorator though. The class inherits from URLWrapper so that is probably the next best place to look\u2026 Indeed, looking at the definition of URLWrapper shows that the as_dict method is overriding the definition in the base class.I guess it\u2019s the \u201ccurse of knowledge\u201d \u21a9"},{"title":"Different Views For Different\u00a0Users","category":"Technical/Web","url":"different-views-for-different-users.html","date":"19 January 2021","body":"This blog serves a variety of purposes. It\u2019s partly a journal of how I\u2019m teaching myself to be a developer and a data scientist, and it\u2019s also a personal blog, with articles about my interests and experience It\u2019s unlikely that anyone is interested in every type of article, and I\u2019d like to make it easy for people to only read the content they\u2019re interested in. Therefore I thought I would separate the articles into two broad groups, technical and non-techni If you visit this blog for the first time by clicking a link to a technical article, the site will then only show you the technical articles on the blog. It\u2019s the same for non-techni articles. If you want, you can change these settings by clicking the paw icon1 in the navbar on the blog index page. I did this mainly because I could. I like playing around with the blog. The JAM stack feels accessible and its fun working with tailwind and with jQuery. I think that playing (being curious, lightheart and unhurried and not being concerned with failure) is really important. Especially for adults who don\u2019t usually do it much. Most of my successes or big opportunit have been the result of a process that started with playing around. Here is the list of requiremen I used when adding the feature: Requiremen If user lands on a page and DOESNT have local setting - create local setting based on type of article being read If user lands lands on a page and DOES have local setting which is contradict - reset local setting to \u201call\u201d If user lands on index and DOES have local settings, only show articles that match the setting Steps: Index page: check if local storage option exists, print to console the result Index page: make button group Index page: make correct button active on page load by using localstora Index page: update active button on page click Index page: articles when button clicked Index page: if local storage does exist, respect it Index page: add 3 stage switch to hamburger menu Index page: make hamburger menu behave intuitivel on small screens Index page: if local storage does not exist, pop up a modal asking for a choice Article page: check if local storage option exists, print to console the result Article page: if local storage doesn\u2019t exist, create it according to article type Article page: if local storage does exist and is contradict update article type to all It\u2019s a paw because cat\u2019s have paws and cat is like category. I might change this to something more intuitive in future, like making the icon an N if the user is only seeing non-techni posts, T for technical, and A for all posts. \u21a9"},{"title":"3 Different Types Of Programming\u00a0Problems","category":"Technical/Web","url":"different-types-of-problem.html","date":"18 January 2021","body":"Three categories of problem Last year when I was creating moneybar and pippip there were a few problems that took much more effort to solve than all the others. I think I could group problems into 3 buckets, based on how much time they take to solve. Type 1 takes less than 15 minutes to solve, type 2 takes between 15 and 45 minutes to solve, and type 3 takes more than 45 minutes (usualy much more). Type 3: When I start learning a hard thing (like web developmen almost everything is in the third bucket and it\u2019s exhausting You need to set aside big chunks of time, you need to be focussed and undistract calm and wide awake, and you need to be prepared for a long arduous journey. Probably your criteria for success should be \u201cam I dead?\u201d because then if you\u2019re asking the question you\u2019re guaranteed to be successful and keeping morale high is necessary for success. Type 2: Hopefully you can make good progress understand the basics and internaliz the relevant abstractio and your problems quickly1 become type 2 problems. They each take from 15 to 45 minutes to solve. Maybe this is because you know enough to break some big general problem into smaller problems (you are developing domain expertise) and your intuitions for how to solve the problem are becoming better so your first or second attempts are likely to be correct, rather than your fifth or sixth. Knowing how to google a problem so that you get the answer you need is also a really important skill, which requires intuiting how an English speaking expert would ask the question. This isn\u2019t trivial but I don\u2019t hear people discussing this often. When most of my coding problems are type 2, it feels like I\u2019m learning most efficientl and when I\u2019m most productive Type 1: After a while, the problems that need to be solved become type 1 problems. They take less than 15 minutes to solve, because: All the big problems have been solved and now you\u2019ve only got smaller problems left, and Your intuitions are good and your expertise has increased and you know where to look for answers.3 Exceptiona problems: But there seems to be a consistent exception to this model.4 Let\u2019s be silly and call them type W problems. These are the problems that eat up far too many hours, and are tiring to solve, even when you are (in most other respects) an expert. For me, these tend to relate to blob storage solutions for web apps deployed into production I can think of several factors why this is so, and I\u2019ll describe the specifics before generalisi When a web app runs in production the data is not stored on the web server because the things that make a web-server cheap and efficient are not the things that make a database or a file storage bucket cheap and efficient. Therefore they are stored somewhere else and you need some plumbing to join everything together. There are some abstractio involved to make this work easily and securely. However when developing locally, you are doing everything on your laptop. You have a web-server relational database and file system all in the same place. This is a big, fundamenta architectu difference between your developmen environmen and your production environmen As a general rule, these are supposed to be as similar as possible. These difference make it much easier to make something that works locally but doesn\u2019t work in production and it\u2019s very hard to test if a thing will work in production without deploying it to your staging environmen which you are likely less familiar with than your local developmen setup. Deploying to staging and debugging on staging is slower and harder than doing the same thing locally. Logging (and filtering) will likely be more important. Solving exceptiona problems So how do you solve these problems quickly and efficientl What is it about this problem that makes it so hard? Let\u2019s examine what makes the problem difficult to solve: Iteration cycles are slow - I can\u2019t test locally, I have to deploy to staging and this takes time. The problem occurs in a \u2018high friction\u2019 environmen - its difficult to dig around and figure out what\u2019s really going on when its hidden below 3 different layers of abstractio on a remote machine that I have limited access to via a web browser. I want to be able to dig and investigat quickly and easily using the same tools I use for writing and testing code locally. I\u2019ve taken great efforts to set up my local developmen environmen so that I can do this, and its stressful to switch to a different and more limited set of tools. The problem is the result of several things interactin at once, and I can\u2019t just test things one at a time. These things are probably very similar to the abstractio Thinking clearly, learning, buidling, solving problems, all rely on being able to separate or untangle a seemingly complex situation into its component parts so that you can figure out what causes what. If you can\u2019t isolate individual concerns or components you have a black box that is keeping you ignorant. In web developmen customized logging is usually a good way to being isolating and exploring particular components Having said all that, I think the best way to solve a problem is to prevent it from occurring in the first place, but I\u2019m not good enough to figure out how to do that, yet.on which timescale? Life is long, does it really matter if it takes 1 week or 1 month to learn something meaningful Momentum, and having fun, is important though. \u21a9from a personal growth point of view. I suppose from an employers point of view they want all problems solved fast, type 1 problems. \u21a9Open the right file, google the right query (and follow the link to stack overflow), make some changes, run your static type checker and linter, run your tests, and push. Done and on to the next item. \u21a9which is totally fine. It\u2019s just a mental model, and the map is not the territory \u21a9"},{"title":"Why I Want To Write\u00a0Regularly","category":"Non-technical/Learning","url":"why-i-want-to-write-regularly.html","date":"18 January 2021","body":"I\u2019ve started writing more frequently I want to do this because I often have thoughts which I\u2019d like to explore and develop further but rarely do. Writing forces me to organise my thoughts and look at how substantia they really are, or aren\u2019t. There is truth in the saying that \u201cto know a thing you need to be able to teach it\u201d 1 , and writing well has several similariti to teaching. Can I really copy a collection of thoughts from my head to yours? Powerful ideas are resilient and have many consequenc The older I get the more I believe that ideas matter2. They have so many subtle consequenc They are the first dominos. I don\u2019t expect writing regularly to become a permanent habit - it doesn\u2019t need to be. But I do want to focus on it for awhile so that I become significan better. It\u2019s a skill that has too many benefits to be ignored. The blogs I remember most are focussed and unapologet about their priorities Most of them have a lot of text and do not focus on design. They make it easy to read content and don\u2019t spend time or attention on header images or styling. Before I redesigned this blog I had default settings that asked me to supply an image for each post, and for a summary, and a suggested tweet. None of it was necessary and whilst they all tried to make the blog better they ended up making it harder to write. These peripheral features added complexity and distracted from the main thing. They\u2019re are still there if I want to use them but they are not set up to be used by default anymore. They\u2019ve been moved to the background and if I forget they exist then that\u2019s OK - it just shows they weren\u2019t as important in practice as I thought they would be. I was probably just having fun adding new features and working out how to build them. You Ain\u2019t Gonna Need It, mate.Wikip article, and some external validation \u21a9A compliment notion is that asking the right question is more important than finding the right answer. I guess asking the right question is always necessary, but finding the right answer is only sometimes sufficient Sometimes you can get the answer a bit wrong if you asked the right question, and still get enough benefits to avoid the problem. \u21a9"},{"title":"Python: Becoming A Better Python\u00a0Developer","category":"Technical/Developer Tools","url":"becoming-a-better-python-developer.html","date":"18 January 2021","body":"I\u2019ve been subscribed to Aaron Maxwell\u2019s \u201cPowerful Python\u201d newsletter for over a year and I really like it. His emails are opinionate and candid, and singularly focussed. He seems passionate about what he does and I like\u00a0that. Ultimately the emails are designed to drive sign-ups for his courses which I suspect would be very good, but there is a lot of value in the free emails. Thanks Aaron. I realised that the emails are sequential and each subscriber gets the same sequence of messages regardless of when they signed up. There is the \u2018first\u2019 message, and then the \u2018second\u2019, and they kind of progress and\u00a0flow. This means that there are more benefits to paying attention than for usual email subscripti Even though the emails arrive when I\u2019m at a supermarke or making dinner for my kids, it\u2019s good to try and read it After being subscribed for several months, I unsubscrib and resubscrib Now that I know how reliable and high quality this advice is I\u2019m going to prioritise working through the examples and doing some of what I missed the first time. I\u2019ve gone back to the beginning to reinforce the parts I know and to try again with what eluded me the first\u00a0time Three kinds of practice projects to become a A web app - use Django if you don\u2019t know which framework to user. Done\u00a0this. A command line tool - use the argparse module, because it\u2019s in the standard library. Haven\u2019t done this yet, I guess now is a good time to start. It seems like the simplest and quickest of the three kinds of project, and I can see how useful it could be - it lets you use the app in many different contexts, outside the python eco-system and anywhere command line tools can be A machine learning model - I\u2019ve already studied this, from theory (numpy) to frameworks (tensorflo I\u2019m happy to see it\u2019s\u00a0inclu"},{"title":"Using Vim with large\u00a0codebases","category":"Technical/Developer Tools","url":"vim-for-large-projects.html","date":"15 January 2021","body":"I use Vim as my text editor and IDE. I like that its free, open source and customizab Below are some of the most useful plugins and features I\u2019ve started using this year when I was building Moneybar and learning how to use\u00a0Django There\u2019s a copy of my .vimrc at the\u00a0end. I\u2019m happy to invest time and effort learning how to make the most of Vim and its plugins. I\u2019m confident that I\u2019ll still be using it twenty years from\u00a0now. Filetype plugins - if you want some settings to be active only for particular filetypes, like .py (python) or .txt (text) then create a file in Vim will look in this file when it opens a buffer of the correspond file type. Good for formatting options like line length, tab spaces, vim commands that are You can\u2019t activate plugins in these files though. All the plugins have to be activated in your .vimrc in the usual\u00a0way. - this plugin lets you runs tests without leaving vim. You can run the test that\u2019s nearest the cursor, or all the tests in the current buffer. It\u2019s very customizab I wish it could be a bit faster, but I could probably improve that myself by changing some\u00a0setti - The incredible Asyncronou Linting Engine (A.L.E) applies fixers and linters to various filetypes, when you want and how you want. Super useful for writing tidy code and catching mistakes before the code is\u00a0run. junegunn/f and - It took a little getting used to at first, but now I can\u2019t imagine not using a tool like this (this could be said about so many vim-relate things). Use fzf to switch between open buffers, open a new file, search for files using the filename, or search within all the files in the project for specific\u00a0t - This plugin opens a sidebar which contains a list of of functions and classes and methods (tags). You can use it to see which methods a class contains, and jump to the part of the buffer where a tag is\u00a0defined This is my .vimrc during January\u00a020 \" ========== Global ========== set nocompatib \" always put it at the top of .vimrc. effects mappings, undo, etc. set encoding=u \" utf-8 encoding set termguicol set t_Co=256 \" number of colors set noerrorbel vb t_vb= \" no error bells, yes screnflash set linespace= set scrolloff= \" minimum number of screen lines above and below the cursor set shortmess- \" show how many times a search result occurs in current buffer, and index of current match set hidden set number relativenu \" Line numbers set splitbelow set splitright \" set tabstop=8 softtabsto expandtab shiftwidth smarttab set undofile \" Maintain undo history between sessions set \" put all the undo files in this dir filetype on \" enables filetype detection filetype plugin indent on \" detection on, plugin on, indent on. To see the current status, type: :filetype syntax on \" syntax highlighti - try 'syntax on/enable' set noesckeys \" might break stuff, should make delay smaller set timeoutlen \" timeoutlen is used for mapping delays set ttimeoutle \" ttimeoutle is used for key code delays set incsearch ignorecase smartcase hlsearch highlight Search guibg=purp guifg='NON highlight Search cterm=none ctermbg=gr ctermfg=bl highlight CursorColu guibg=blue guifg=red highlight CursorColu ctermbg=re ctermfg=bl nnoremap // nnoremap # # nnoremap * * \" close buffers properly go to previous buffer, then delete the buffer you were just in. nnoremap bd :bp\\|bd # inoremap bd :bp\\|bd # \" Spell check set spelllang= nnoremap ss :setlocal spell! nnoremap sf z=1f :call Flash() ve :e$MYVIMRC vr :so $MYVIMRC \"+y if set \" copy to the system clipboard if \" X11 support set endif endif \" Go into NORMAL mode inoremap jk \" view working directory nnoremap pw :cd %:p:h \" toggle line wrap nnoremap lw :set nowrap!ln :set \" Insert current datetime nnoremap dt A ()hh \" map w to  nnoremap ` w \" Swap : and ; nnoremap ; : nnoremap : ; vnoremap ; : vnoremap : ; \" Navigation & movemement \" save buffer if it has been changed nnoremap ww :update \" save all changes nnoremap wa :wa \" close buffer nnoremap qq :bp\\|bd # nnoremap wq # \" switch buffers nnoremap + :bn nnoremap _ :bp \" Split navigation nmap h nmap j nmap k nmap l nmap ww nmap wq \" split (pane) resize nnoremap :resize +2 nnoremap :resize -2 nnoremap :vertical resize +2 nnoremap :vertical resize -2 \" open help in vertical split by default cabbrev vhelp vert help \" Natural cursor movement over wrapped lines nnoremap j gj nnoremap k gk \" Insert blank lines in normal mode nnoremap o ok nnoremap O Oj \"========= PLUGINS ========== call \" numbers as text objects Plug \"run shell commands async in vim8\" Plug let = 10 \" When using :python or :!python, access the packages in venv \" \" Plug \" force quickfix to be full widtth au FileType qf wincmd J \" testing - many languages and test runners Plug let test#strat = let = 'pytest' let = '-x' let = \"belowrigh nnoremap tn nnoremap tf :TestFile< nnoremap ts :TestSuite nnoremap tl :TestLast< nnoremap tg :TestVisit \" toggle the quickfix window function! if copen 15 setlocal else cclose endif endfunctio nnoremap cc :call \" generates an index (or tag) file of language objects found in source files \" jump to definition \" jump back \" g] see a list of multiple matches \" Plug \" (re)genera tags file in the bg Plug let = ['.json', \" sidebar that displays the tags of the current file, ordered by their scope Plug nnoremap nnoremap \" add python library code to tags file, goto def with let pyEnvLib =$VIRTUAL_E let pyEnvLib .= \" Async linting engine Plug let = 0 let = 0 \" ALE completion let = 0 set let = 1 nnoremap at :ALEToggle nnoremap af :ALEFix aj :ALENext ak \" iSort Plug \" track the snippets engine Plug \" Snippets are separated from the engine. Add this if you want them: Plug \" Trigger configurat Do not use if you use let let let \" If you want :UltiSnips to split your window. \" let Plug Plug Plug nnoremap x :YcmComple GoTo \" the subcommand add entries to Vim's 'jumplist' so you can use \" 'CTRL-O' to jump back to where you were before invoking the command (and \" 'CTRL-I' to jump forward; see ':h jumplist' for details) let = 0 let let = 1 let = 1 let = 1 let = 1 let = 1 let = 1 \" autoclose parens, brackets etc \" Plug \" vim-tmux focus events Plug \" Code folding \" Plug \" match m of n \" Plug \" adds vertical lines to easily show indent levels Plug \" Fugitive Plug \" Marks Plug \" Latex Vimtex Plug let g:tex_flav = 'latex' autocmd Filetype tex set updatetime let = 'open -a Preview' let = \\'specifie changed to'.\"\\n\". \\'You have \\'Missing number, treated as zero.'.\"\\n \\'There were undefined \\'Citation %.%# \\'Double space found.'.\"\\ let = 8 \" Rainbow parenthesi let blacklist = ['html', 'md', 'wiki'] autocmd BufWritePr * if &ft) < 0 | Plug let = 1 let g:rainbow_ = { \\'guifgs': ['green', 'magenta1' 'gold', 'red', \\'guis': \\} \" Set color scheme. set Plug \" colorschem colorschem badwolf let = 1 let = 1 let = 1 \" colorschem modificati highlight Comment ctermfg=cy guifg=cyan highlight pythonComm ctermfg=cy guifg=cyan highlight LineNr ctermfg=cy guifg=cyan hi nontext term=bold ctermfg=Cy guifg=#80a gui=bold hi vimLineCom term=bold ctermfg=Cy guifg=#80a gui=bold \" SpecialKey - use :set list to toggle visibility of EOL, CR, etc hi specialKey term=bold ctermfg=Cy guifg=#80a gui=bold \" colors for flashing cursorline and cursorcolu hi CursorLine cterm=NONE ctermbg=gr ctermfg=bl guibg=gree guifg=blac hi CursorColu cterm=NONE ctermbg=gr ctermfg=bl guibg=gree guifg=blac \" query what kind of syntax is this color? - wc nnoremap wc :echo \"hi<\" . . '> trans<' . .\"> lo<\" . . \">\" \" fuzzy file, buffer, tag finder set \" ensure you have the latest version Plug { 'do': { -> fzf#instal } } Plug nnoremap e :Files nnoremap r :Buffers t :Tags nnoremap ff :Rg \" nnoremap ff :Ag nnoremap la :BLines ll :Lines nnoremap ' :Marks nnoremap fh :Helptags< nnoremap fs :Snippets< nnoremap fc :Commits fb :BCommits< nnoremap hh :History h: :History:< nnoremap h/ :History/< \" let = --info=inl \" let --files --hidden\" let = 0 let g:fzf_layo = { 'down': '~50%' } \" let = '' let = 'right:0%' function! let joined_lin = join(a:lin \"\\n\") if len(a:line > 1 let joined_lin .= \"\\n\" endif let @+ = joined_lin endfunctio let g:fzf_acti = { \\ 'ctrl-t': 'tab split', \\ 'ctrl-x': 'split', \\ 'ctrl-v': 'vsplit', \\ 'ctrl-o': \\ } let g:fzf_colo = \\ { 'fg': ['fg', 'Normal'], \\ 'bg': ['bg', 'Normal'], \\ 'hl': ['fg', 'Comment'] \\ 'fg+': ['fg', 'CursorLin 'Normal'], \\ 'bg+': ['bg', 'CursorLin \\ 'hl+': ['fg', 'Statement \\ 'info': ['fg', 'PreProc'] \\ 'prompt': ['fg', \\ 'pointer': ['fg', 'Exception \\ 'marker': ['fg', 'Keyword'] \\ 'spinner': ['fg', 'Label'], \\ 'header': ['fg', 'Comment'] } \" grep in vim - shows results in a split window Plug \" session tracking Plug \" pairs of handy bracket mapping Plug \" Plug \" repeat commands from plugin mappings Plug \" vinegar Plug let = 3 \" CSV Plug \" nerdtree Plug nnoremap n let let = 1 \" Automatica delete the buffer of the file you just deleted let \" 2 - open nerdtree only if directory was given as startup argument let \" always focus file window after startup let \" Status bars Plug Plug let = 1 let = 0 let = 0 let let \" remove encoding status let = 1 let let = 1 let = 1 let = 1 let = 1 let = 1 let = 0 let = 0 let = 0 let = 0 let = \" comments Plug let = 1 let = 1 let = 'left' let = 0 let = 1 \" markdown. tabular is required Plug Plug let = ['python=p let = 0 let = 0 let = 1 let = 0 let g:tex_conc = \"\" let = 1 let = 4 let = 1 \" writing prose Plug Plug augroup pencil autocmd! autocmd FileType wiki,md,tx call pencil#ini autocmd FileType wiki,md,tx :PencilSof augroup END let = 'soft' autocmd! User GoyoEnter autocmd! User GoyoLeave \" Ensure :q to quit even when Goyo is active function! s:goyo_ent let b:quitting = 0 let = 0 autocmd QuitPre let b:quitting = 1 cabbrev q! let = 1 q! setlocal wrap endfunctio \" Quit Vim if this is the only remaining buffer function! s:goyo_lea if b:quitting && bufnr('\$') == 1 if qa! else qa endif endif endfunctio autocmd! User GoyoEnter call autocmd! User GoyoLeave call nnoremap g :Goyo \" python linting \" F7 checks flake8 Plug Plug \"Flagging Unnecessar Whitespace highlight BadWhitesp ctermbg=re guibg=dark Plug let = ['latex', 'html'] let = 1 let = [] \" javaScript Plug let = 1 let = 1 \" format .JSON files by using the jq cli tool com! JQ %!jq \" HTML/JINJA Plug Plug \" Plug let = \"*.html, *.xhtml, *.phtml\" call plug#end()"},{"title":"Using RSS","category":"Non-technical/Learning","url":"using-rss.html","date":"14 January 2021","body":"Updated: 10 Feb 2021 I found a blog post which is surprising similar to my thoughts on RSS feeds, but better presented and thought through. The post mentions the idea that \u201cRSS is about capturing the long tail of blogs that don\u2019t post frequently 1. This idea crystalise why I was so glad I\u2019d started using RSS feeds again. If readers use RSS, then authors don\u2019t need to concern themselves with attention. This removes pressure on the author to post frequently and lets them focus on quality over quantity. News feeds and ad supported platforms have fundamenta different mechanics and incentives With RSS I can let good quality content come to me, on its own schedule. I don\u2019t need to remember to look for it, and the authors don\u2019t need to remind me that they exist. Google Reader RSS is a very effective way of having good quality informatio come to you. Back in 2008, I used to use Google Reader to subscribe to RSS feeds. I was an aspiring photograph back then and I remember being subscribed to around 80 blogs. Each day I\u2019d read articles from whoever had posted something new, without needing to visit their websites or remember who they are or that I\u2019d subscribed to their blog. The authors didn\u2019t need to optimize their output according to an opaque and changing algorithm either - they didn\u2019t need to optimize article length, tags, post frequency, image inclusion or linked content. They could write how they wanted to, which I suspect leads to higher quality content. Social Media A few years later Google Reader was closed down, presumably because using RSS didn\u2019t fit with Googles advertisin model. I was unaware of it at the time but I imagine it sent shockwaves through blogging communitie and probably upended many businesses I mostly stopped reading blogs. Facebook was growing fast, Instagram felt new and exciting, and content was moving onto \u2018platforms or into walled gardens. And as they kept on growing the average quality of the content decreased. Twitter is like this now I think. There are some real diamonds to be found from time to time, but there\u2019s a lot of mud too. Mostly its just mud, but the occasional diamond can have outsized benefits. RSS isn\u2019t like this. I choose the contents of my \u2018news feed\u2019, and each article can be much longer than a Tweet, or a caption to a photo, or a status update. It\u2019s hard to write well and to create an interestin or useful blog post, and that makes it harder to dilute quality with entertaini distractio I have complete control about what content I see, and I can change it whenever I want. The process is designed around me. Reeder5 I used netNewsWir for a few weeks, but it couldn\u2019t sync between my laptop and phone, so I bought Reeder 5. It\u2019s got a few unusual design patterns, but it works well and has all the features I want. I\u2019ve been unsubscrib from email newsletter and subscribin to the RSS feed instead. It keeps my inbox quieter, and it feels good to have a \u2018separatio of concerns\u2019. It makes it easier to read interestin content without being \u21a9"},{"title":"Notes on learning\u00a0Django","category":"Technical/Web","url":"learning-to-django.html","date":"14 January 2021","body":"Table of Contents In the\u00a0beginn A personal The best\u00a0momen In the\u00a0beginn I came to web developmen via business analytics. I was working as an accountant and Excel wasn\u2019t good enough anymore, so I looked around for a way to get started and came across Jupyter Notebooks. Notebooks are said to be a kind of \u201cgateway drug\u201d to programmin and I think that\u2019s true. They\u2019re the easiest and fastest way to start programmin that I\u2019ve come\u00a0acros When you\u2019re working in a notebook, its easy to get data, wrangle it, and show some results. But as soon as you can create a chart or some summary table you inevitably wonder how you can show this to people more easily, and publishing the results to a website feels like the best most general and Unfortunat it\u2019s also the hardest, and so begins a long series of compromise and incrementa progress. Learn to use a dashboardi API, and learn to create static sites. But the end-goal, the ultimate solution, is a data driven web app, with saved user preference scalable performanc and automatica updated data\u00a0sourc A personal When I moved to the Netherland I wanted to use a personal finances dashboard to check weekly expenses. There wasn\u2019t a web-app that would do this (though there are a couple of apps that are trying) so I built my own dashboard. Then a few friends asked if they could use it too. They couldn\u2019t because it was just a dashboard and not a web app, but I thought this was a good reason to jump into It was a much bigger task than I anticipate (And that\u2019s OK.) It took several attempts and was super frustratin I would dabble for a few weeks, do a few tutorials, and then get completely lost when I tried to do something by myself. I\u2019d get disorienta working across many different files and trying to visualise which part of the model, or the cycle I was currently working\u00a0on I came to realise that the mental load seems so large at the beginning because is really a whole stack of technologi and abstractio combined (or stacked) together. Many of these have to be used together at the same time before you can see any evidence of success at all. I think the hardest things about Django are not actually Django. You\u2019ll need to comfortabl with classes and inheritanc You\u2019ll also need to be comfortabl with working across multiple files, and have some tools for searching across all you open buffers, or all the files in the project, at the same time. You\u2019ll also need to be comfortabl with version control (Git) and using the command line. Get familiar with stack traces\u00a0too If you\u2019re familiar enough with all these things, so that using them doesn\u2019t feel new, but ideally feels familiar and comfortabl then I think you\u2019ll make quite quick progress with\u00a0Djang Django uses the model. Models are how django maps Python objects to items in your database (oh yeah, you need to be familiar with SQL too\u2026), Views are where requests are processed (also Middleware and turned into Responses, which are then combined with templates (unless your building an API). You might notice I haven\u2019t mentioned what a Controller is - get used to informatio feeling incomplete whilst you\u2019re learning the ropes. It\u2019ll become clear soon\u00a0enoug The best\u00a0momen The \u2018curse of knowledge\u2019 states that once you\u2019ve learnt something you can\u2019t imagine or remember what it\u2019s like to not know it. Before that happens completely I want to record some of the \u2018ahah!\u2019 moments of For context, I stopped working as a freelance data scientist in April and after a few weeks wondering if django and PostgreSQL and python was the way to go (yes it is. use boring technology I began working full-time on what would become MoneyBar.n I called it \u2018myeuros\u2019 in the\u00a0beginn The learning curve felt steep. I wanted to do things \u201cright\u201d the first time because I wasn\u2019t building a toy, and although I felt that hindsight would show this to be a mistake in terms of efficiency I did it anyway because I have a hunch that following my compulsion sometimes makes life harder in the short term and better in the long\u00a0term. The best moments are usually preceded by the Adding a unique identifier to an existing I used pydanny\u2019s template. Honestly, by the time I\u2019d gone through the quickstart process and googled the nouns in all the questions (what is Sentry, what is Celery and what is a task que, what is whitenoise etc.) I was already tired. Play with it a few times and come back to\u00a0it. Anyway, I wanted to start with because the project template has that part kind of up and rnuning for you out of the box. uses the Django Allauth package, which is awesome, and reliable, and fully featured\u2026 and extremely abstracted Good luck looking at the module code and understand it if youre not an\u00a0expert. I wanted to give each user a unique ID - a UUID when they signed up. This would be used in query strings instead of usernames or incrementa keys. This was so hard the first time! And it turns out its not a trivial task, not if you already have a few users in your (test) database. Sure you can reset the database and start again, but experiment like this is fairly complex. Understand how the python model classes (the ORM) maps to the relations in the PostgreSQL databse was complex, and if I got confused, should I try to fix it by changing python Models, or editing migrations or working on the database directly? Getting started is one of the After I\u2019d figured out I started creating models for other simpler data (transacti and bank accounts I expect). This was much simpler and faster. I remember driving home one evening thinking that if I could get this far then success Testing\u00a0co Before long, testing each part of the app by hand when I added or changed a feature was no longer trivial. I needed to find some way of automatica creating users and checking that they could log in and access\u00a0vie I began working with pytest, and really found it hard to wrap my head around the idea accessing different parts of the app not by requests and responses but by accessing class I think its normal and good to code at the limit of your knowledge, where you know just enough to make a thing \u201cwork\u201d. But this approach falters when you want to then test what you wrote. Or at least, the measure of \u201cjust enough\u201d really changes when you require tests to be written. You don\u2019t just need to make it work, you need to understand why it works, so that you can write tests to assert that certain conditions pass and others\u00a0fai This feels really satisfying when it works, because you have proof that you really have grasped a bigger picture. There are far fewer (relevant) black boxes when you write tests. But it also makes learning slower, at least in the short term. It means you might have two get comfortabl with a handful of abstractio when you\u2019ve already solved the problem you started with. This is frustratin and it takes discipline to slow down, take a deeper look at the solution, and not just race on to the next\u00a0featu"},{"title":"Data Science vs Web Development: Larger Code\u00a0Bases","category":"Technical/Developer Tools","url":"larger-code-bases.html","date":"14 January 2021","body":"Code\u00a0Struc One of the most immediate and basic difference between working as a data scientist or as a web developer is the number of files the codebase is spread across and the amount of code within each\u00a0file. Web applicatio tend to be very modular - there are a lot of different things going on in a modern web app and generally they all need to be able to be modified or updated independen of each other. This requiremen encourages modular code base architectu with the code broken down into When working on a data science project you often have a well defined and quite narrow pipe line. Each stage of a pipeline has well defined inputs and\u00a0output This seems to have the consequenc of making data science projects tend towards a handful of files each with a substantia amount of unique (not boilerplat code. In web developmen there seems to be more boilerplat many more files spread across a tree of directorie and the average number of lines of code per file is IDE\u00a0featur These difference mean that code organizati tools and IDE features play very different roles within each industry. In web developmen you really need to be able to jump between different files (or buffers) quickly, and search for text across multiple files. Writing idiomatica becomes more important, and writing code within discreet testable units becomes essential so that things don\u2019t break without being\u00a0noti In data science, linting feels more optional, and searching for text within methods or functions outside the current module is\u00a0rarer. I didn\u2019t appreciate this until I paused my work as a Data Scientist and began building non-trivia web\u00a0apps."},{"title":"Test Driven\u00a0Development","category":"Technical/Developer Tools","url":"test-driven-development.html","date":"6 January 2021","tags":"python, django, testing, web-app ","body":"Test Driven Developmen was mind-bendi when I first grappled with it: \u201cWrite a test for the code before you write the code\u201d \u201cAssert that your code matches your expectatio by understand all the inputs and all the outputs for every function or method I write\u201d. Last summer I was building a web app and began to break things when adding new features. This soon led to lots of clicking around different pages to test if stuff was still working each time I made an update. This led to me thinking there must be a better way, which eventually brought me to Test Driven Developmen (TDD). It should have just led me to writing tests, which it did. But googling whatever I googled got me down the TDD rabbit hole rather than just the \u201cwrite some tests\u201d rabbit hole. Write tests for your code before you write the code. Write tests for bugs you\u2019ve fixed to check they stay fixed. Write tests as a kind of documentat to show what stuff is supposed to be doing. Errr\u2026 Django was a big enough pile of abstractio as it was. Views, ORMs, mixins, serializer Trying to add factories and fixtures into that took some getting used to. But eventually I made some progress, and now I quite enjoy running coverage reports to keep coverage close to 100%1. Some of the main things I\u2019ve learnt about writing tests: Use PyTest as much as possible rather than other testing libraries - its assert statements are more intuitive than Django\u2019s own testing framework, and you can use it in any Python codebase, not just Django. It has lots of extensions and seems good at getting the job done fairly easily. Write tests as you go. I haven\u2019t (yet) reached the elevated level of writing tests before I write the code to be tested, though I see why that would sometimes be useful. I do think writing tests sooner rather than later is best though, ideally as soon as you\u2019ve got a basic version of your feature working. Use Coverage to show you which code is covered by your tests, and which branches or edge cases are not. But be warned, it doesn\u2019t tell you if the test is useful or not, only that it passes and which methods or functions it uses. Fixtures are great for keeping tests fairly DRY. Freezegun is great for testing anything to do with dates and times. Static type checkers, like Mypy, get more attractive in proportion to codebase complexity and size. Which is fun and all, but testing for the sake of it doesn\u2019t necessaril stop bad things happening. Its very possible to write a test that covers the code you\u2019ve just written without ensuring that only the intended behaviours happen. \u21a9"},{"title":"Why Talk About\u00a0Jesus?","category":"Non-technical/Other","url":"faith-in-jesus.html","date":"5 January 2021","body":"Start with happiness It seems to me that I am much happier as a Christian than I would be if I weren\u2019t. By \u201cbeing a Christian\u201d I mean following Jesus - trying my best to act, think and speak like he want\u2019s me to because I\u2019m grateful that he did what I believe he did. I believe he died for my sins and is now alive, having been miraculous resurrecte by his father1 (who is now my father too). It\u2019s not just an intellectu exercise. I believe that Jesus is alive because I seem to have experience his companions and interventi in my day-to-day life. This is unusual, mind boggling, and makes things significan more complicate than if I thought he were dead. Nonetheles it seems to be true. It\u2019s not like I see supernatur interventi every day or anything, but there have been various times - too many to discard - when my prayers have been answered in practical ways (I\u2019ll leave out the intangible for now) that have surprised me and given me a lot of respect for the risen Jesus doing what its said he\u2019d do in the bible. Obvious answers There was this time I was in Yemen trying to get back to my room across town, at night and during a storm. I was lost and couldn\u2019t read Arabic to understand which bus to get. So I prayed, and two buses later I got off at the right stop. Another example, I was getting a haircut in Liverpool Street Station in London, and as I was sitting in the chair someone took my work bag. When I came to pay, I realised my laptop and wallet had gone. Trying to find a stolen bag in central London feels ridiculous Even so, I spent a few hours wondering around the alleys and parks looking for it, and I resigned myself to some awkward conversati and using most of my next salary to replace the stolen computer. To my surprise, my parents-in prayed about it and were really confident it would come back to me, which seemed super unlikely. A few days later I received a call from an office worker in the Gherkin building - my bag was under his desk and he wanted to know if I\u2019d like to collect it. I guess I\u2019ll try not to make the same mistake twice. The other example I tend to remember happened not long after I\u2019d first moved to Vienna. I was feeling lonely and isolated and I was wondering how on earth I was going to find some sort of normal that was healthy and sustainabl Try as I might, I wasn\u2019t enjoying things at all and was feeling stressed and overwhelme I remember walking up this steep hill in the 18th district towards my office, and repeatedly praying this really simple prayer \u2018God, please help, please help, please help..\u2019. It was that simple because I couldn\u2019t think of anything more useful to say. Nothing dramatic happened that day, or even that week as far as I can remember. But when I look back, it was a turning point when things started getting better instead of worse. I suppose you can call this last example an intangible answer to prayer. Maybe it is, but I think anyone whose grappled with overwhelmi loneliness or panic would say that the emotions become all too tangible at times. Being in a different emotional state seemed to make a tangible difference to just about every area of life. Eating, productivi at work, relationsh with friends and colleagues etc. This is only the beginning of why I think Jesus is alive and why I think Christiani is a real and living faith. It\u2019s not primarily a tradition, a worldview or a set of rules and ideals. Christiani is a relationsh with Jesus. He did lots of amazing things that have let me have a very practical relationsh with him. It\u2019s an almost unbelievab premise from which to live a life. It has so many implicatio And it holds up to scrutiny and my experience bear it out. Why write this? Despite Jesus\u2019 incredible works and their implicatio modern Christiani seems to be in a really confused and ineffectiv state. Ideas and thoughts about have become mixed up with cultural christiani or christian politics and traditions These are each different things, and unless we distinguis between them with the words we use, we are going to find it hard to think and communicat clearly. I suspect that we are in a negative cycle of imprecise thinking leading to imprecise articulati which leads to further imprecise thinking. Unless we can talk and think about one thing at a time, atomically if necessary, we take on the additional risks of reaching the wrong conclusion personally or arguing with others due to rather than actual disagreeme We should try to create a more precise vocabulary to navigate our Christian lives, so that we can think clearly about the experience and questions we have. Imprecise thinking is frustratin and conversati are less effective when there\u2019s an increased risk of disagreeme or what someone else means. This makes it harder to talk about our faith, which makes talking less common, and this creates room for or apathy, or missed opportunit or sadness.An many other things also. \u21a9from first principals \u21a9"},{"title":"API Design\u00a0Principles","category":"Technical/Developer Tools","url":"api-design.html","date":"4 January 2021","body":"Some super brief notes I made about API\u00a0design Background It\u2019s more of an art than a\u00a0science RESTful State Transfer) API design is an Alternativ API architectu SOAP (Simple Object Access Protocol) is a heavier\u00a0st GraphQL - doesnt overfetch. Graph query language made by\u00a0Faceboo APIs are everywhere (not just web APIs). They\u2019re an abstractio that hides an Django model managers are an API (and also part of Django\u2019s ORM), JavaScript is an API,\u00a0etc. RESTful\u00a0AP Web APIs (all REST APIs?) expose a databases to\u00a0clients A rest api is a URL route (endpoint) that returns JSON or XML. POST, GET, PUT, PATCH, DELETE, correspond to Create, Read, Update/Mod Delete (HTTP methods correspond to CRUD\u00a0metho HTTP METHODS: PUT (create or update) is idempotent POST is not idempotent (keep on PATCH - partial\u00a0up GET, HEAD, OPTIONS and TRACE methods are idempotent cos they are only designed for DELETE HEAD - almost identical to GET, but without any body. Good for checking what a request would return, i.e. Before downloadin a large amount of\u00a0data, OPTIONS - returns data describing what other methods and operations the server supports at the given URL. More loosely defined than other\u00a0verb Use HTTP verbs to make requests Use sensible resource names. Naming things is hard, so think about this a bit before starting. Use identifier in your URLs, not the query string. Good: /users/123 Poor: Use the hierarchic structure of the URL to imply the structure of the API. Design (names and structure of things) for the user/clien not for the database. Resource names should be nouns not\u00a0verbs Use plurals consistent not collection verbiage. Good: customers/ Use camel case or snake Short is better than long, but be\u00a0clear Spend time on design before writing\u00a0co Use HTTP response codes to Prefer JSON over XML. (Hotline does HTML..) XML requires schemas for validation and namespaces Don\u2019t support this complexity at the beginning (or ever) unless required. If it is required, make the XML as similar to JSON as\u00a0possibl Put links in the HTTP link header, or use a JSON representa of\u00a0this. Use the HTTP location header to contain a link on resource creation, or for GET with pagination use first, last, next,\u00a0prev Connectedn - return links in the response which link to useful resources. At minimum, a link to show how the data was received, or\u00a0posted. Idempotenc - clients making the same repeated requests create the same result on the server side. I.e. making repeated requests has the same result as making a similar request, server side. On the client side, a response code may change, of\u00a0course."},{"title":"Principles Of Object Orientated\u00a0Programming","category":"Technical/Developer Tools","url":"principles-of-oop.html","date":"4 January 2021","body":"I recently interviewe for a lead developer role at Lab Digital1 and thought it would be sensible to review some of the fundamenta aspects of Object Orientated Programmin (OOP). You might think that\u2019s a unusual way to prepare for an interview, and you\u2019d be right. Nothing close to these notes arose during the interview, but I find this stuff interestin If I\u2019m motivated enough to study it, then I think that\u2019s a good enough reason by itself, without a specific reason. These are some brief notes. Object Orientated Programmin has four key aspects: Encapsulat (Hiding informatio Abstractio (Hiding the Inheritanc Polymorphi 1. Encapsulat Each object keeps its state private, inside a class. Instance are kept private and accessor methods are made public. Other objects don\u2019t have direct access to this state. They can only call a list of public functions (methods). The object manages its own state via methods, no other class can touch it unless explicitly (not default) allowed. Private variables. Public methods. You can define classes within classes, and functions within functions. 2. Abstractio A natural extension of encapsulat A concept or idea that is not associated with any particular instance. Expresses the intent of the class, rather than a specific Programs are often extremely large and separate objects communicat with each other a lot. This makes maintainin large programs difficult, and abstractio tries to solve this. Applying abstractio means that each object should only expose a high-level mechanism for using it. This mechanism should hide internal implementa details. It should only reveal operations relevant for the other objects. This mechanism should be easy to use and should rarely change over time. Implementa changes \u2014 for example, a software update \u2014 rarely affect the abstractio you use. e.g. a coffee machine. It does a lot of stuff and makes quirky noises under the hood. But all you have to do is put in coffee and press a button. 3. Inheritanc In OOP, objects are often similar, sharing similar logic. But they are not 100% the same. Create a (child) class by deriving from another (parent) class. This way, we form a hierarchy. child class reuses all fields and methods of the parent class (common part) and can implement its own unique part using method or attribute overloadin 4. Polymorphi Gives a way to use a class exactly like its parent so there\u2019s no confusion with mixing types. But each child class keeps its own methods as they are. This typically happens by defining a (parent) interface to be reused. It outlines a bunch of common methods. Then, each child class implements its own version of these methods. Any time a collection (such as a list) or a method expects an instance of the parent (where common methods are outlined), the language takes care of evaluating the right implementa of the common method \u2014 regardless of which child is passed. I\u2019d like to be so familiar with the following features that I can use them without referring to notes: Getters and setters. Instance methods compared to class methods. Inheritanc mixins, and decorators The \u201cmagic\u201d within the Django source code that requires mypy to use extensions in order to do its static type checking correctly. Unfortunat I didn\u2019t get the job. They wanted a senior Python developer with experience with Infrastruc As Code, and also working at an agency. Can\u2019t win them all. \u21a9"},{"title":"Optimizing The Performance Of This\u00a0Blog","category":"Technical/Web","url":"site-performance.html","date":"4 January 2021","body":"I\u2019m coming to the end of redesignin this site. Now that the main changes have been made its fun (and good practice) to optimize the site so that it loads quickly and is optimized for SEO Lighthouse is a utility built into Chrome that runs a technical audit on a webpage and assesses a wide range of features. It also provides details about how to improve the\u00a0page. My site is hosted on Github Pages and is accessed via Cloudflare which gives me a lot of performanc gains including minified HTML and CSS, caching, and super fast server I\u2019m using Github Pages and Cloudflare for free and I think its amazing that I can get the benefits of these services without needed to pay anything. If someone knows where to look and can teach themselves using free resources, they could be read by anyone anywhere in the world. It\u2019s\u00a0amazi Below are the lighthouse results for the blog\u2019s index page and for a recent\u00a0pos"},{"title":"Unix: Utilities To Analyse And Update Multiple Text\u00a0Files","category":"Technical/Developer Tools","url":"using-unix-utilities-to-analyse-and-update-multiple-files.html","date":"4 January 2021","body":"As part of the redesign of this blog I wanted to make an article\u2019s category more meaningful Previously I simply picked a handful of categories and then assigned a single category to each post. This method becomes limiting when an article is relevant to Also, using nested categories seems like a good way of grouping similar content and allowing more nuanced filtering of\u00a0interes As I considered how to update the categories of existing articles, I realised this would be a good opportunit to practice analyzing and updating text files using Here is how I reviewed and updated the categories of my I use Pelican to generate the static files for this site. It converts markdown into HTML. Metadata for each article is set at the beginning of a file, the title is set by typing Title: ... and similarly the category is set by typing Category: ... on its own\u00a0line. To locate, analyse and update my existing categories I would therefore need to find all the markdown files which have a row that begins with Category: grep -h \u2018Category: **/*.md - prints each search\u00a0res grep -h \u2018Category: **/*.md | sort - prints and sorts each search\u00a0res grep -h \u2018Category: **/*.md | sort | uniq -c prints and sorts each search result, then counts how many occurrence of each unique result there\u00a0are. I had some repeat results though because some rows had white space at the end, so in order to make these the same, I needed to remove grep -h 'Category: **/*.md | sed | sort | uniq -c This gave me the 6 Category: 2 Category:D 16 2 2 15 15 8 Category:T Category is repeated and isn\u2019t\u00a0need grep -h 'Category: **/*.md | sed | sort | uniq -c | sort | sed This gives me the following output, which is\u00a0accepta 2 Data 2 Engineerin 2 Front-end 6 8 Tools 15 General 15 Startups 16 New\u00a0Catego The next stage was to begin updating these categories with the new, nested categories I\u2019ve decided to try splitting the categories into technical and I can imagine splitting Technical > Data even more in future, perhaps having Data Analytics, Data Science, and Data Engineerin as Technical Data Web Cryptocurr Not technical Family Self Career I cd into the directory containing the markdown files, and then to change all the articles with Category: Tools to Category: Tools I\u00a0did: grep -l 'Category: Tools' *.md | xargs sed -i 's/Categor Tools/g' If I want to see a list of files containing Category: General: grep -H 'Category: General' *.md If I want to see just the file names,\u00a0the grep -l 'Category: General' *.md Update Since writing this post I\u2019ve modified the categories a few times. The commands I run to switch out categories are as\u00a0follows export export newName=Li grep -l \"Category: | xargs sed -i '' Notes: Double quotes are not the same as single quotes. You need to use them if you want to access variables or commands inside a\u00a0string. .* is a wildcard operator allowing any number of characters It\u2019s required when an article belongs to"},{"title":"A New Blog\u00a0Design","category":"Technical/Web","url":"a-new-blog-design.html","date":"18 December 2020","body":"The blog has a new design! Out with the old, and in with well-writt HTML, an improved CSS framework, maintainab code, dark mode, and articles with This website was my first ever project using HTML and CSS, and the codebase for the original blog was terrible. It was poorly written and hard to maintain. I remember when I was first building it and trying to figure out what a