diff options
Diffstat (limited to 'blog/content/notes/tech')
15 files changed, 1143 insertions, 8 deletions
diff --git a/blog/content/notes/tech/about-apis.gmi b/blog/content/notes/tech/about-apis.gmi new file mode 100644 index 00000000..f1732cc9 --- /dev/null +++ b/blog/content/notes/tech/about-apis.gmi @@ -0,0 +1,26 @@ +# About APIs + +=> https://gist.github.com/kislayverma/d48b84db1ac5d737715e8319bd4dd368 The Jeff Bezos' API memo is one of the most talked stories about API programming. + +It is, in my opinion, also one of those things which are successful for some environments, but not for all. + +## The levels of API accessibility + +An "operation" in your application can be in one of the following levels of "API accessibility": + +* -oo The operation cannot be invoked in isolation easily. For instance, it is embedded in an MVC controller, mixed with form handling and HTML generation, and thus the best approach to invoke it programatically is to simulate a browser +* 0 The operation can be invoked, in-process, by calling a function or method, but requiring complex setup or using complex types (e.g. others than lists, maps, numbers and strings) +* 1 The operation can be invoked, in-process, by calling a function without complex setup and using plain types +* 2 The operation can be invoked, off-process, by calling a function without complex setup and using plain types +* 3 The operation can be invoked via a command line tool +* 4 The operation can be invoked via a network call + +Many proponents of APIs propose level 4 as the target. This obviously allows your operations to be integrated in separate processes via network calls, which is the most powerful way of API access. They will also reason that this will force your application to have a clean architecture with separation of concerns. + +Note also that doing proper testing will probably force your operations to be tested to be in levels 0-2, as otherwise it will be annoyingly complex to test them. + +We propose that the architecture benefits of level 4 are also present in levels 0-3, but achieving these levels requires much less effort than achieving level 4 (where you need to add a network protocol, handle aspects such as authentication/authorization, marshalling/unmarshalling, etc.), so unless you require level 4, you can stay in levels 0-3. Going to level 3 instead of 0 should be easy when creating new operations, so that's the level of API accessibility we recommend new code to adhere to by default. + +Note also that level 3 can provide many benefits of level 4, but with less development overhead, so it's a level we recommend considering explicitly, as it is often overlooked. + +Level -oo is typical of legacy applications. Note that we consider the distance between level -oo and the rest of levels much bigger than the distance between the rest of levels. diff --git a/blog/content/notes/tech/about-django.gmi b/blog/content/notes/tech/about-django.gmi new file mode 100644 index 00000000..15f73600 --- /dev/null +++ b/blog/content/notes/tech/about-django.gmi @@ -0,0 +1,115 @@ +# About Django + +Without more context, one of the technologies I recommend to everyone is Django. + +Django is a Python web framework with "batteries included". + +Web frameworks can provide more or less tools to write applications. Typically, frameworks that provide fewer tools are more flexible and give developers more freedom to develop their applications in the best possible way. Similarly, frameworks that provide more tools tend to guide you towards a specific way to writing applications, and typically, require more work if you want to deviate. + +In my opinion, many applications you might need to develop are very similar and have similar issues, and solving them ad-hoc for each project is a waste. Therefore, I lean towards using frameworks that provide more batteries in most cases. + +(Certainly, there are projects that clearly need special approaches, or which deviate enough from any generic web framework.) + +In fact, most of the complaints described in this document are caused by Django having too few batteries, not too many! + +=> https://github.com/alexpdp7/django-tws Django training wheels is my project in alpha to address some of those shortcomings. + +## The Django admin + +Besides including more batteries than most other frameworks, and being in general a well-engineered framework in my opinion, Django includes the admin. + +The admin is a declarative way to build administrative sites where some users edit data stored in the application database. + +Many similar tools exist, but I have not found any other tool that can do so much. + +* The Django admin handles multi-table relationships very well, including picking foreign key targets and editing related table data. For example, if a person entity has a "parent" related foreign key relationship, the Django admin provides a search functionality to pick a person's parent. If the person entity has a list of children, the Django admin provides a way to add and edit children from the person form. +* The Django admin has a simple, but useful for many scenarios permissions functionality, where editing certain entities is restricted to groups of users. + +The Django admin is frequently a big boost during the early development of database-backed applications, and sometimes it can provide value during a big part of the life of an application. + +Additionally, traditionally when working with frameworks without an equivalent facility, the friction of adding an interface to edit a piece of data can be large. Developers pressed for time might opt to hardcode the data in the source code of the application, requiring code changes to modify certain behaviors of the application. When the friction to add a user interface to edit such data is low, developers can configure the admin to let those users edit the data directly without going through the developers. + +## Django problems + +However, there are still many common issues for which batteries could exist, but that Django does not provide. + +### Django has no support or documentation about packaging Django projects + +Most Django projects have dependencies besides Django. In order to develop and deploy Django applications, you likely must install other dependencies. Django does not include documentation nor support to do this. + +Many different approaches and tools exist to manage Python project dependencies. Understandably, endorsing one particular approach in Django could be controversial. +So Django leaves the choice of approach up to users. Additionally, Django adds a few difficulties in Python project management, and users must figure out how to handle Django projects in their chosen approach. + +Several initiatives have tried to tackle this problem, notably: + +=> https://github.com/radiac/nanodjango nanodjango + +### Django settings are a partial solution + +Django provides settings to manage the configuration for a Django project. You implement Django settings by writing a Python module. + +For example, the default Django template includes the following snippet to configure the database connection: + +``` +DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.sqlite3', + 'NAME': BASE_DIR / 'db.sqlite3', + } +} +``` + +Besides assigning a setting directly like in the preceding snippet, you can use Python code to assign settings. + +This allows you to tackle many common issues, such as setting up a different database connection for development and production, while keeping the production database credentials away from the source code repository. There are many similar issues that you must tackle in nearly all projects. + +Several initiatives tackle some of those issues: + +=> https://github.com/jazzband/dj-database-url dj-database-url provides a way to configure the database connection through an environment variable. + +### Django does not explain a development database workflow + +Django provides migrations to handle schema changes. Migrations work well and are a valid solution to handle schema changes in production. + +However, while developing a Django application, you frequently need to make many temporary changes to the data definition until you find the right data definition. + +In my opinion, if you follow the Django documentation, then you might end up using migrations for those development schema changes. This is awkward and problematic, and there are procedures to develop database changes that work better. + +I would like a command that recreates your database, applying unmigrated model changes. This command could also have hooks to load sample data. (Likely, Python code and not fixtures.) + +### Django only tackles database-based, server-side-rendered, non highly interactive web applications + +While certainly a huge amount of applications: + +* Revolve around data stored in a relational database +* Are better implemented as server-side rendering applications +* Do not require very complex or real-time interactions + +There are certainly many applications that do not fit this mold. + +In my opinion, focusing on database-based applications is a good decision. Many Django features (like the admin) revolve around the database, and a framework oriented to other applications likely should be very different. + +However, more and more applications break the limits of server-side rendering, and while you can build such applications with Django, you need a lot of effort or finding additional libraries to use. + +For example: + +* Django REST framework provides a layer to provide REST APIs on top of the Django ORM. +* Projects exist to add support for Django for front end frameworks such as htmx or Hotwire. These frameworks are an intermediate step between traditional server-side-rendered applications and JavaScript front ends, enabling most of the benefits of JavaScript front ends within the traditional server-side rendering approach. + +Additionally, providing an API is also useful beyond JavaScript front ends. APIs are necessary for other purposes, such as implementing mobile apps to interact with your application, or just providing an API for programmatic access to your application. + +### Some common tasks should have more tutorial content + +The Django documentation is mostly for reference, covering all Django features, but with little content on how to use Django. The items I list below likely are documented on books, websites, forums, etc. If you know a good source for many of those, even if it is paid, feel free to let me know to add references. + +* Restricting users in the admin to a subset of the instances of a model. + +For example, users belong to organizations and users should only see instances of some model related to their organization. The FAQ contains "How do I limit admin access so that objects can only be edited by the users who created them?", which is a very similar question and points to the features you need to use to achieve these goals. These requirements are often related to requiring "extending the existing User model". + +* Having a search UI for reference fields instead of dropdowns. + +Many projects similar to the admin only offer dropdowns for reference fields. This does not work when the referenced objects are more than a couple. Django calls this raw_id_fields, and it is difficult to learn that this feature exists. + +## Further reading + +=> crud-is-an-important-unsolved-problem CRUD is an important unsolved problem diff --git a/blog/content/notes/tech/about-relational-databases.gmi b/blog/content/notes/tech/about-relational-databases.gmi new file mode 100644 index 00000000..d08071ac --- /dev/null +++ b/blog/content/notes/tech/about-relational-databases.gmi @@ -0,0 +1,27 @@ +# About relational databases + +## What is a relation? + +A common misconception is that the "relations" in a relational database are about relations between database tables. + +Actually, the relations in a relational database are the tables. + +A relation "relates" a set of values with another set of values. + +For example, a relation can relate the name of a person with their birth date and birth place. For example: + +(person name) => (birth date, birth place) +(Alice) => (1979-12-03, Barcelona) +(Bob) => (1995-03-04, Paris) +... + +Many computer languages have similar concepts: + +* Python mapping types such as dict +* C++ std::map +* Java java.util.Map +* C# System.Collections.Generic.Dictionary +* JavaScript Object +* PHP arrays + +Relations are a natural concept, so although non-relational data systems exist, most data can be stored as relations. diff --git a/blog/content/notes/tech/containers-might-not-be-the-right-answer.gmi b/blog/content/notes/tech/containers-might-not-be-the-right-answer.gmi new file mode 100644 index 00000000..de614983 --- /dev/null +++ b/blog/content/notes/tech/containers-might-not-be-the-right-answer.gmi @@ -0,0 +1,100 @@ +# Containers might not be the right answer + +Containers are everywhere, and I feel today they are the default answer to many problems for many people. + +=> https://www.jwz.org/blog/2014/05/so-this-happened/ Although the author of one of the handiest quotes does not want people to use it, I think that quote adequately describes the situation. Definitely, containers are no silver bullet. + +Containers are a good example of an “easy but not simple” technique. + +=> https://www.youtube.com/watch?v=SxdOUGdseq4 (See the "Simple Made Easy" - Rich Hickey 2011 talk.) + +Containers are easy because they automate getting arbitrary "isolated" Linux environments and running processes on them. Additionally, you can find container images for mostly everything on the Internet. For this reason, it is very easy to run much software using containers, often with a single command. + +However, containers are easy, but not simple. Containers combine many different techniques to achieve their ease, and thus, frequently you hit problems derived from any of the techniques containers use. + +However, Docker popularized many good ideas, both directly related to containerization and general ideas! There are still places where containers are the right answer. + +## Reasons not to use containers + +### Containers are Linux + +The fact that containers are a Linux technology is the most common point of complexity. + +One result of this is that using containers on any operating system that is not Linux, such as Windows or macOS, requires a Linux virtual machine. + +This has some accidental problems, like increased memory and CPU usage, and derived inconveniences like decreased battery life and fan noise. + +But the main issue is that adding this VM makes things more complex, mostly because you are adding networking to the mix, and some container features are not available or work worse, like bind mounts. + +Most issues can be worked around, but this requires more effort, or at least, more knowledge overhead of knowing how to avoid these issues. + +(On top of that, people who develop processes using Linux are not exposed to these issues, so they are likely to introduce issues without realizing what works on their Linux workstation does not work on macOS or Windows workstations.) + +### Container images are big and expensive + +Optimizing the size of container images can require significant effort. Popular public images are often optimized for size, but even with optimized images, storing and moving container images frequently requires much more bandwidth and storage than the alternatives. + +There are free ways to host private container images, but they are frequently limited in size, bandwidth, or both. You can easily run into Docker Hub limits, GitHub only provides 2gb of storage, etc. + +Building container images also can require significant resources. + +### Containers require specialized knowledge + +Using containers frequently requires learning quite a few things specific about containers. + +* Containerfile design is not obvious. Some questions, like ADD and COPY, or CMD and ENTRYPOINT are difficult and not well documented. + +* Container design is not obvious. Docker popularized "application containers", a fuzzy concept that is related to "single process containers", 12 factor architecture, and a few other ideas. Solving your problem might require good knowledge of application container design and use. + +* Container tools are complex, because containerization is difficult. Likely you need to know some intricate details of how Linux file permissions and users work, for example. + +Not using containers can mean avoiding having to think about these things, and being able to use the time you save to actually solve your problem. + +### Docker is not so good, targeting multiple container engines is not trivial + +Docker was the first popular container engine. Docker was a revolution and invented or popularized many great ideas. However, knowledge about containers was not well established when Docker was invented, and since then, better ways of doing many things have been discovered. + +Other tools such as Podman or Fedora Toolbx and adjacent tools such as Distrobox have introduced many improvements respect Docker, while still reusing and being compatible with many Docker concepts. + +However, creating processes and tools across these different tools can be difficult, despite their apparent compatibility. + +### In some scenarios, containers do not add much + +Mainly after the rise of the Go programming language, distributing binaries has become easier. Distributing binaries on Windows and macOS has always been simpler than distributing binaries on Linux. + +However, nowadays many programming languages can create binaries that can be downloaded and executed on most Linux distributions. + +One of the main benefits of Docker has been ease of distribution of software, but nowadays this is easy to achieve through binaries. + +### Beware container images + +Much software is distributed nowadays as container images. The abundance of container images means that learning how to use containers helps you run a wide variety of software distributed as a container image. + +However, many container images are not of great quality, nor are adequately updated. + +In some cases, you can find software that has a container image, but where the container image is not of sufficiently good quality and can cause issues down the road. + +## Reasons to use containers + +### Containers still provide isolation easily + +By default, running a container does not alter the system that runs the container significantly. + +This is a huge advantage, and in many cases, more difficult to accomplish without containers. + +### Making things work across different operating systems is not trivial either + +Some decisions in software, like programming language or dependency choices, influence greatly how easy running the software on different operating systems is. + +In many cases, making something run in a specific distribution and packaging as a container can be the most resource-efficient way to distribute software that works across Windows, macOS, and most Linux distributions. + +Finding the right combination that makes software portable can require significant effort, or even be unviable. + +### Some container-related software has good and unique ideas + +For example, the controversial Kubernetes still provides a distributed standardized operating system that can be managed in a declarative way. This is a powerful concept, and still the preferred way to package software for Kubernetes depends on container images. + +## What to use instead of container images + +* Binaries +* "Portable-friendly" development tools such as Go, uv, or Cargo. diff --git a/blog/content/notes/tech/crud-is-an-important-unsolved-problem.gmi b/blog/content/notes/tech/crud-is-an-important-unsolved-problem.gmi new file mode 100644 index 00000000..c338f785 --- /dev/null +++ b/blog/content/notes/tech/crud-is-an-important-unsolved-problem.gmi @@ -0,0 +1,78 @@ +# CRUD is an important unsolved problem + +The CRUD (initials of create, replace, update, and delete) term is used to describe the implementation of applications that provide a simple interface to a database. + +If you have used a computer for work, then you likely have used a system that allows to manipulate records of customers, orders, products, or any other information related to the business. + +Although programmers have been written a huge amount of CRUD systems for decades, my perception is that the costs of implementing CRUD systems is a major problem with important consequences. + +There are two major approaches to implementing CRUD systems: + +* Traditional programming: combining a relational database and most existing programming languages enables programmers to create CRUD systems. +* "No code" (or "low code"): many products and services enable non-programmers to describe their data structure and the user interface requiring less technical knowledge than the traditional programming approach. + +## About implementing CRUD systems with traditional programming + +The Python Django web framework coupled with a relational database requires writing the least code of all the platforms I have used to write CRUD systems. + +Most of what you need to do when using Django is to describe what you need, instead of implementing the mechanical parts of a CRUD system. + +Out of the box, Django provides: + +* List and detail views, including nested views. Many systems provide "flat" details view where you can edit a record, but not associated records. For example, they provide a detail view for customers where you can edit the customer name and other information, but any "multiple" information, such as multiple addresses or phone numbers, must be edited in a different view. This is frequently a huge issue, and it can require writing a significant amount of code in other systems. With Django, you can implement this by describing the associated data. +* Multi user authentication and role-based authentication. With Django and without programming any code, administrators can create groups, assign users to groups, and limit the kinds of records that each group can view or edit. +* Primitive change tracking. Out of the box, changes to records are tracked automatically and can be consulted. + +For most CRUD implementations, alternative platforms require significantly more effort to implement those features. + +Additionally, the entire stack is open source software that does not require paying licenses. + +(Surprisingly, in the past there existed even more sophisticated CRUD platforms. But sadly, most have disappeared.) + +## About implementing CRUD systems with no code + +A huge amount of systems provide similar functionality, in an even more friendly manner. + +They typically provide a user interface where you can create tables, add columns, and describe the user interface without programming. + +Some of those systems offer features comparable or superior to Django. + +However, because those systems focus on no code usage, frequently you hit roadblocks when using them. + +When you need a feature that they do not provide, it is either impossible to do it, or it requires programming in an unfriendly environment. + +Programming CRUD features can be complex. While traditional programming tools have evolved providing many features such as automated testing and advanced code revision control systems (rolling back bad changes and others), no code CRUD platforms do not reuse the wealth of programming tools that have been developed for traditional programming. + +Non-developers frequently face huge challenges going beyond the basics of what the tool provides, and developers struggle and suffer by working in environments that are more limiting compared to others. + +## The consequences of the high cost of development of CRUD systems + +In these conditions, most CRUD systems are expensive and do not work well. + +Organizations often resort to systems such as spreadsheets that can be productive, but have severe reliability concerns. + +No code CRUD systems often have significant costs and lock in their customers, because migrating costs can be astronomical. + +CRUD systems implemented with traditional programming often are costly to maintain and extend. + +In most cases, organizations cannot justify the costs of tailoring the CRUD system entirely to their needs, so they suffer from using CRUD systems that do not meet their needs. + +## Possible approaches + +### Improving existing traditional programming CRUD platforms + +I believe systems such as Django can still see many improvements. Likely, both the amount of technical knowledge to use these systems and the amount of effort to design CRUD systems can be reduced significantly. + +### Providing systems to transition from no code approaches to traditional programming + +No code approaches are wonderful, because giving end users the ability to describe what they need enables them to experiment and become productive very quickly. + +However, no code platforms cannot provide all features needed, and in many cases, end users will struggle past a certain point. + +Providing a way to migrate to a traditional programming approach would enable breaking this barrier and scaling systems more effectively. + +(Some no code platforms have APIs. With them, programmers can write code to extend the no code CRUD systems using traditional programming approaches. However, implementing functionalities through APIs has limitations and specific problems.) + +## Further reading + +=> about-django About Django diff --git a/blog/content/notes/tech/gadgets/pocket-computers.gmi b/blog/content/notes/tech/gadgets/pocket-computers.gmi index a31970ef..e2cd7a2f 100644 --- a/blog/content/notes/tech/gadgets/pocket-computers.gmi +++ b/blog/content/notes/tech/gadgets/pocket-computers.gmi @@ -8,16 +8,8 @@ The Blackberry KeyONE I used 2017-2021 was the last device I used that had this Since then, I believe the loss of physical keyboard phones means that some uses of the smartphone have disappeared. -=> https://www.bringbackblackberry.com/ - ## Current devices -=> https://www.unihertz.com/products/titan-pocket Titan Pocket (owned) - -* The keyboard feels worse than a Blackberry -* The keyboard has deteriorated with age (some keys require extra pressure to activate, meaning typing is painful) -* The keyboard software is insufficient for writing in Spanish and Catalan - => https://www.clicksphone.com/communicator Clicks Communicator => https://www.clicksphone.com/powerkeyboard Clicks Power Keyboard => https://keyphone.tech Keyphone @@ -29,6 +21,14 @@ Since then, I believe the loss of physical keyboard phones means that some uses => https://www.tindie.com/stores/zitaotech ZitaoTech refurbs keyboards, but they seem to be permanently out of stock. => https://linkapus.com/ The Q25 project is a project to put an Android phone inside a Blackberry Classic shell. +## Recent devices + +=> https://www.unihertz.com/products/titan-pocket Titan Pocket (owned) + +* The keyboard feels worse than a Blackberry and deteriorates with age (some keys require extra pressure to activate, meaning typing is painful) +* The keyboard software is insufficient for writing in Spanish and Catalan +* My battery swelled and I retired it + ## Obstacles ### Communication platforms without an open API diff --git a/blog/content/notes/tech/git-advice.gmi b/blog/content/notes/tech/git-advice.gmi new file mode 100644 index 00000000..8ec41105 --- /dev/null +++ b/blog/content/notes/tech/git-advice.gmi @@ -0,0 +1,21 @@ +# Git advice + +## Never use "git commit -m", use "git commit -v" + +Configure your system so that the EDITOR environment variable refers to your preferred editor. + +With "git commit -v" you can see your commit diff while writing your commit message. This helps you review that your commit is correct and write a better commit message. + +## Use gitignore properly + +=> https://git-scm.com/docs/gitignore gitignore + +Note that by default, Git defaults to $XDG_CONFIG_HOME/git/ignore or $HOME/.config/git/ignore. + +## Use the modern Git commands (or teach them) + +Particularly, "git checkout" has many functionalities that now can be handled by more focused commands like "git switch" and "git reset". + +If you have too much muscle memory and are used to them, then consider learning them only to teach other people so that they start with the safer commands. + +Many Git commands print suggestions that use the newer commands. diff --git a/blog/content/notes/tech/github-annoyances.gmi b/blog/content/notes/tech/github-annoyances.gmi new file mode 100644 index 00000000..a36214b9 --- /dev/null +++ b/blog/content/notes/tech/github-annoyances.gmi @@ -0,0 +1,7 @@ +# GitHub annoyances + +## The repository creation wizard can be confusing initially + +When creating a new repo, GitHub offers you to populate the repository with some files (a README, a .gitignore file, a license). + +In some situations, you have an existing directory in your computer with files that you want to be the initial contents of the repo. If you create a truly empty repo, then GitHub displays some instructions that can help pushing the contents of your existing directory to the new repo. If you use the GitHub features to populate the repo, then GitHub does not display these instructions and uploading your files requires more knowledge. diff --git a/blog/content/notes/tech/misc-python-stuff.gmi b/blog/content/notes/tech/misc-python-stuff.gmi new file mode 100644 index 00000000..f7cc33e7 --- /dev/null +++ b/blog/content/notes/tech/misc-python-stuff.gmi @@ -0,0 +1,62 @@ +# Misc Python stuff + +## Tools + +* Use uv for your software. It's modern and good. +* pipx is packaged by many distros and useful for installing Python software. Provide instructions for installing your software using pipx. +* Use pytest for testing. It's PEP-8 compliant unlike unittest in the standard library. doctest is good too. +* Use ruff for validation. + +## Python versions + +Try to support Python versions in popular LTS distros such as RHEL and its clones, and Debian and its derivates (like Ubuntu). + +## Writing command line tools + +* Use entry points so that when using pipx or uv to install your tool, they install the tool to the user's path. + +### Nice Python libraries + +* appdirs for using the proper configuration, cache, etc. directories on multiple platforms +* keyring for cross-platform secret storage +* tqdm for progress bars + +### Writing dependencyless Python + +If you can write your program without using dependencies, then it automatically becomes much easier to distribute and run. This is quite possible for many tools! And helps you replace problematic shell scripts. + +The standard library still includes a lot of batteries: + +* argparse is clunkier that third-party libraries, but it works well enough. The documentation provides a pattern for subcommands under "one particularly effective way of handling sub-commands...". argcomplete is a third party library that adds tab completion for argparse programs. Review the types that argparse provides support for, such as Path, enumerated choices, etc. +* compression allows working with many archive formats without shelling out to tar or (un)zip. +* concurrent.futures for executing things in parallel, esp. the map function combined with a ThreadPoolExecutor (for IO-bound tasks) or ProcessPoolExecutor (to avoid the GIL in tasks that use the CPU). +* getpass to read passwords from the terminal properly. Also to obtain the current user. +* hashlib to avoid shelling out to commands such as sha256sum. +* http.server is useful for simple web servers (and also as a quick web server in the command line). +* json is about the only structured format supported in the standard library. +* logging to print output with timestamps. +* pathlib for any kind of path manipulation, esp. the read|write_text|bytes methods that are so convenient. shutil still contains a few functions missing from pathlib, esp. in older Python versions. +* textwrap.dedent and str.[lr]strip for embedding multiline strings in code. +* urllib.request is clunkier than third-party libraries, but it's usable. + +For very simple stuff, tkinter can implement simple graphical tools and wsgiref can implement simple web apps (that you can even deploy with CGI). + +### Subprocess + +The main problem of using Python for scripting is that the subprocess functions do not default to check=True and shell=False. + +Likely many of your scripts will start with a subprocess wrapper suited to your script. + +You can use shlex.join to print commands you execute in a copy-pastable way. + +## Writing scrapers + +=> https://playwright.dev/python/ Use Playwright + +* Playwright automatically sets up headless browsers. +* Provides convenient abstractions for locating elements in a page (mostly no XPath required. It can match "intelligently" using text). +* Has a handy UI tool that records your actions in a browser and writes equivalent *readable* Playwright code. + +Further reading: + +=> https://new.pythonforengineers.com/blog/web-automation-dont-use-selenium-use-playwright/ Web automation: don't use Selenium, use Playwright diff --git a/blog/content/notes/tech/motivating-example-for-logical-replication-for-dynamic-ui.gmi b/blog/content/notes/tech/motivating-example-for-logical-replication-for-dynamic-ui.gmi new file mode 100644 index 00000000..daf5329e --- /dev/null +++ b/blog/content/notes/tech/motivating-example-for-logical-replication-for-dynamic-ui.gmi @@ -0,0 +1,61 @@ +# Motivating example for logical replication with dynamic UI + +(I'm almost sure what I write below is a horrible idea that will melt a PostgreSQL server with very few "real-time queries" at the same time. I'm very curious about how much load could PostgreSQL handle efficiently using this schema.) + +Suppose the following database schema (pseudo-SQL): + +```sql +create table chat_messages ( + id serial primary key, + posted timestamp not null, + channel text not null references chats(id), + author text not null references users(id), + message text not null +); +``` + +Imagine you could write an UI element that subscribed to the following publication: + +``` +create publication foo for table chat_messages where (channel in :list_of_channels_user_is_in and posted > :some_time_ago); +``` + +Without writing any additional code, the UI element would get instantly notified not only of all new messages, but also of editions, deletions, or messages moved in or out of the subscribed channels. I believe you could write a real-time UI element with much shorter and safer code than any alternative I can think of that only uses OSS code. (As far as I know, [ksqlDB](https://github.com/confluentinc/ksql) does a similar thing, but has non-OSS bits and seems much harder to deploy than PostgreSQL, besides you would also need to deploy PostgreSQL.) + +This has some caveats: + +* Publications cannot do "joins", and implementing any live UI element that requires joins would be much more complex. (And I'm not sure it would still be the best way to implement things.) +* This likely cannot be implemented efficiently without having all working set data in RAM (e.g. all the data involved in all subscriptions). + +My idea is writing: + +* A daemon that provides an API that can be used as in the following example: + +``` +subscription = subscribe("chat_messages", column("channel").in(list_of_channels) and column("posted").gt(some_time_ago)) +while update = subscription.next(): + for chat_message in sorted(update.all_current_results(), key=lambda chat_message: chat_message.posted): + print(chat_message.current_values, chat_messages.previous_values) + print(update.deleted_since_last_update_results()) +``` + +* Libraries for stacks such as "Django + HTMX", "GTK", etc. that allow to build UI elements that use the daemon API underneath, so you could write things like: + +``` +<ul class="channels"> + {% foreach channel in joined_channels %} + <li> + {{ channel.name }} + last message: {{ for chat_message in update_all_current_results() if chat_message.current_values.channel == channel | max(lambda chat_message: chat_message.current_values.posted) }} + </li> + {% end foreach %} +</ul> + +<ul class="current_chat_messages"> + {% foreach chat_message in update.all_current_results() if chat_message.current_values.channel == current_channel %} + <li>{{ chat_message.current_values.author }} {{ chat_message.current_values.message }}</li> + {% end foreach %} +</ul> +``` + +For stacks such as Django/HTMX, in simpler websites you could have websites that degrade gracefully out of the box without JS, just losing real-time updates. diff --git a/blog/content/notes/tech/prolog-vs-sql.gmi b/blog/content/notes/tech/prolog-vs-sql.gmi new file mode 100644 index 00000000..181ae7e3 --- /dev/null +++ b/blog/content/notes/tech/prolog-vs-sql.gmi @@ -0,0 +1,114 @@ +# Showing the similarities between SQL and Prolog + +SQL is a very common programming language, which sometimes is compared to the relatively more obscure Prolog language. Both are examples of declarative languages, where you define some facts, then you can ask questions about those facts, and the system answers the questions without you writing an explicit program. + +However, I could not find a good example of their similarities. This text presents the most typical Prolog example, and translates it to SQL. + +## A typical Prolog example + +"[x]" reads Prolog facts from file "x". We can use the special file "user" to read facts from the REPL, ending the facts with ctrl+d: + + +``` +$ swipl +?- [user]. +|: father(jim, julian). +|: father(julian, joe). +|: father(julian, jerome). +|: father(pete, perry). +|: ^D +true. +``` + +You should read "father(X,Y)" as "X is the father of Y". So Jim is the father of Julian, and so on. + +We can ask Prolog questions: + +``` +?- father(julian, jim). +false. +``` + +Is Julian the father of Jim? There is no known fact about this, so no. But Julian *is* the father of Joe: + +``` +?- father(julian, joe). +true. +``` + +More interestingly, you can ask who are Julian's children: + +``` +?- father(julian, X). +X = joe ; +X = jerome. +``` + +(You press ; to get further answers.) + +## A simple translation to SQL + +You can do pretty much the same with SQL, first define the facts as values in tables: + +``` +$ sqlite3 +sqlite> create table fatherhood(father, son); +sqlite> insert into fatherhood values ('jim', 'julian'); +sqlite> insert into fatherhood values ('julian', 'joe'); +sqlite> insert into fatherhood values ('julian', 'jerome'); +sqlite> insert into fatherhood values ('pete', 'perry'); +``` + +Then you can get the same answers: + +``` +sqlite> select * from fatherhood where father = 'julian' and son = 'jim'; +sqlite> select * from fatherhood where father = 'julian' and son = 'joe'; +julian|joe +sqlite> select * from fatherhood where father = 'julian'; +julian|joe +julian|jerome +``` + +## The next step in Prolog + +The typical example continues with some logic: + +``` +?- [user]. +|: grandfather(X,Y) :- father(X, Z), father(Z, Y). +|: ^D +true. +``` + +X is the grandfather of Y if X is the father of Z and Z is the father of Y. Then you can ask questions, and Prolog knows the answers: + +``` +?- grandfather(jim, X). +X = joe ; +X = jerome. + +?- grandfather(X, jerome). +X = jim ; +false. +``` + +## Can we do the same in SQL? + +You might not guess the answer on the first try, but the answer is not complex: you can do the same thing with SQL views: + +``` +sqlite> create view grandfatherhood as + ...> select fatherhood_1.father as grandfather, fatherhood_2.son as nephew + ...> from fatherhood as fatherhood_1 join fatherhood as fatherhood_2 on (fatherhood_1.son = fatherhood_2.father); +``` + +And if you ask the same questions, SQLite gives the same answers: + +``` +sqlite> select * from grandfatherhood where grandfather = 'jim'; +jim|jerome +jim|joe +sqlite> select * from grandfatherhood where nephew = 'jerome'; +jim|jerome +``` diff --git a/blog/content/notes/tech/python-modules-primer.gmi b/blog/content/notes/tech/python-modules-primer.gmi new file mode 100644 index 00000000..67d03e19 --- /dev/null +++ b/blog/content/notes/tech/python-modules-primer.gmi @@ -0,0 +1,229 @@ +# Python Modules Primer + +## Prerequisites + +These instructions assume a Linux environment. A macOS environment is similar, but not identical. A Windows environment is more different. + +## Previous knowledge + +### A refresher on the PATH variable + +If you execute the following command in your terminal: + +``` +$ echo hello +``` + +, the shell searches for the echo command in the directories listed in your PATH environment variable. You can display your PATH variable by running: + +``` +$ echo $PATH +/home/user/.local/bin:/home/user/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin +``` + +The contents of the PATH variable depend on your particular environment. + +If you run the following command: + +``` +$ which echo +/usr/bin/echo +``` + +The which command prints where the shell locates the echo command. + +### A refresher on shell scripts + +If you create a file named foo.sh with the following contents: + +``` +#!/bin/sh + +echo hello +``` + +, you define a "shell script". The first line indicates that this shell script is executed by using the /bin/sh command. The rest of the file are commands to be executed by the shell command. These commands behave as if you typed them into your terminal, so if you execute this script, the command "echo hello" will be executed, printing hello. + +If you try to run foo.sh like you run the echo command, by typing its name, it does not work: + +``` +$ foo.sh +bash: foo.sh: command not found... +``` + +, because the shell looks for the foo.sh in the directories listed in the PATH variable. Unless you created the foo.sh file in a directory like /usr/bin, the shell will not find the foo.sh command. + +A solution to this problem is to specify the path to the foo.sh file, instead of relying on the PATH variable. However, if you do this, you face a second problem. + +``` +$ ./foo.sh +bash: ./foo.sh: Permission denied +``` + +This happens because only files with the executable permission can be executed in this way. To solve this, add the executable permission; then it works: + +``` +$ chmod +x foo.sh +$ ./foo.sh +hello +``` + +## The import statement in Python + +### Importing from the Python standard library + +Run the following commands by using the Python REPL: + +``` +$ python3 +>>> import datetime +>>> datetime.datetime.now() +datetime.datetime(2023, 9, 11, 21, 53, 16, 331236) +``` + +import works in a similar way to running a command in the shell. +Python searches a number of directories looking for the datetime module. + +To see which directories are searched, run: + +``` +$ python3 +>>> import sys +>>> sys.path +['', '/usr/lib64/python39.zip', '/usr/lib64/python3.9', '/usr/lib64/python3.9/lib-dynload', '/home/alex/.local/lib/python3.9/site-packages', '/usr/lib64/python3.9/site-packages', '/usr/lib/python3.9/site-packages'] +``` + +sys.path is a list of the directories that the import command searches. +The contents of `sys.path` depend on your operating system and Python installation method. + +In my system, the /usr/lib64/python3.9 directory contains the datetime.py module. + +``` +$ head /usr/lib64/python3.9/datetime.py +"""Concrete date/time and related types. + +See http://www.iana.org/time-zones/repository/tz-link.html for +time zone and DST data sources. +""" + +__all__ = ("date", "datetime", "time", "timedelta", "timezone", "tzinfo", + "MINYEAR", "MAXYEAR") +... +``` + +/usr/lib64/python3.9 contains the modules in the Python standard library. + +### Importing your Python files + +If you create a file with the a.py name: + +``` +def f(): + return 2 +``` + +, and another with the b.py name: + +``` +import a + +print(a.f()) +``` + +, then: + +``` +$ python b.py +2 +``` + + +This works, because sys.path contains '', which means "the current directory". + +(sys.path is very similar to the PATH variable. However, sys.path contains the current directory by default, whereas PATH does not.) + +When "import a" is executed, then Python searches the directories in sys.path for an a.py file; it is found when checking the '' path. When "import datetime" is executed, Python searches in the current directory (because '' comes first in the path), doesn't find it, but then finds it in the following /usr/lib64/python3.9 directory. Python iterates over the sys.path directories, and loads the first matching file. + +## Installing libraries + +When writing Python software, sometimes it is enough with the modules included in the standard library. However, frequently you want to use other libraries. To use Python libraries, you must install them using the pip program. + +The pip program is not part of the python3 package in some Linux distributions, and comes from the python3-pip package. + +The pip program can download libraries from pypi.org, the Python package index, and install them. pip installs libraries to a "Python environment". + +Old versions of pip defaulted to installing libraries to the "system" Python environment. In a Linux system, the system Python environment is located in a directory such as /usr/lib64/python3.9. By default, normal Linux users cannot write to /usr, so installing a package would fail. + +Modern versions of pip detect that they cannot write to the "system" Python environment, and then redirect the install to the "user" Python environment. The "user" Python environment is in a directory such as ~/.local/lib/python3.9. + +You could use a command such as "sudo pip install" to grant pip the privileges required to write to /usr. However, this can make a Linux system unusable. Most Linux systems use software that uses the "system" Python environment. Altering the "system" Python environment can break such software. Do not run "sudo pip install" with root privileges unless you know why you need this. + +If you use a modern pip (or use the --user option), you can install libraries to the "user" Python environment. However, this is problematic because a Python environment can only contain a single version of a Python library. If you have two different Python programs that different versions of the same library, then these two programs cannot coexist in the "user" Python environment. + +In general, Python virtual environments are used to address this problem. + +## Creating Python virtual environments + +If you run: + +``` +$ python3 -m venv <some path> +``` + +This will create a directory with the path you specify, with the following contents: + +``` +<some path> +├── bin +│ ├── activate +│ ├── pip +│ ├── python +├── include +├── lib +│ └── python3.9 +``` + +The python and pip commands are copies of the same commands from the "system" Python environment. + +But these commands work differently from the "system" Python environment commands: + +``` +$ <some path>/bin/python +>>> import sys +>>> sys.path +['', '/usr/lib64/python39.zip', '/usr/lib64/python3.9', '/usr/lib64/python3.9/lib-dynload', '<some path>/lib64/python3.9/site-packages', '<some path>/lib/python3.9/site-packages'] +``` + +sys.path uses the lib directories in the virtual environment. + +When you use the `pip` program from the virtual environment, it installs the libraries to the virtual environment. + +You can create as many virtual environments as you need, and you can install different versions of libraries to each virtual environment. + +## Activating Python environments + +You can run the python and pip commands by specifying the full path, like we did when executing the foo.sh command earlier. + +By default, if you run python, the shell will invoke the python command from the "system" Python environment because it is in a directory included in the PATH variable. If you specify the full path, you override this. + +To save typing, the bin directory of a virtual environment contains an activate file. The activate file is a "special" shell script that must be invoked like this: + +``` +$ . <some path>/bin/activate +``` + +. is a special shell command that the activate script requires to work correctly. + +activate alters your path, so that the bin directory in your virtual environment comes first in your path. + +``` +$ echo $PATH +/home/user/.local/bin:/home/user/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin +$ . <some path>/bin/activate +(some path) $ echo $PATH +<some path>/bin:/home/user/.local/bin:/home/user/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin +``` + +, and thus if you run python, <some path>/bin/python will be executed instead of /usr/bin/python. + +Besides changing your prompt to indicate the virtual environment is activated, activate only alters your PATH. activate is not mandatory to use a virtual environment. For example, when running the Python command, if you specify the path of the Python executable in a virtual environment, the command will execute as if the virtual environment had been activated. Virtual environment management tools also have commands that can run commands inside a virtual environment without activating it. Activation can save time, but it is also more error-prone than more explicit means of using virtual environments. diff --git a/blog/content/notes/tech/so-you-want-to-play-with-functional-programming.gmi b/blog/content/notes/tech/so-you-want-to-play-with-functional-programming.gmi new file mode 100644 index 00000000..2e8abec5 --- /dev/null +++ b/blog/content/notes/tech/so-you-want-to-play-with-functional-programming.gmi @@ -0,0 +1,201 @@ +# So you want to play with functional programming + +If you are a programmer working on popular languages such as Python or Java, you are likely to have read articles about "functional programming". These articles can give you the idea that learning functional programming improves your skills as a programmer. I share this opinion. + +This article tries to help people who have read about functional programming figure out how to proceed. + +Note that this article expresses personal opinion. Particularly, I am not an expert in this topic: + +* I have programmed some Haskell (about 50 Project Euler problems, plus experimentation on and off during the years). +* I have studies of SML and functional programming. +* I have some minimal experience with Lisp. +* I have applied some functional programming techniques while being paid to write in non-functional programming languages. +* However, I have never been paid to write in any of those languages. + +Shortly after writing this, I was shown: + +=> https://technomancy.us/194 In which there is no such thing as a functional programming language + +I agree with most of that the article explains. I might extend this article with some similar ideas, but for the moment, I recommend reading that carefully before reading the rest of this article. + +## The basics of functional programming + +=> https://en.wikipedia.org/wiki/Functional_programming The Wikipedia article on functional programming is a great place to get started. + +The article describes a few concepts related to functional programming. I consider the following two the pillars of functional programming: + +* First-class and higher order functions. In languages with first-class functions, functions are values that you can use like other types such as integers. Higher-order functions are functions that take functions as arguments or return functions. +* Pure functions. Pure functions always return the same value for a given set of arguments. Pure functions also have no side effects; they do not modify anything in the system they run. For example, a function that creates a file is not pure. + +These concepts can be applied in most popular programming languages. + +For example, in Python: + +``` +def hello(): + print("hello") + +def twice(f): + f() + f() +``` + +twice is a higher order function because it takes a function as an argument. Functions are first-class functions because you can use hello as a value: + +``` +>>> twice(hello) +hello +hello +``` + +Similarly, you can write pure functions in almost any language. + +When you have first-class functions, you can define some higher-order functions that generalize some common code. Three very common higher-order functions are: + +* Filter applies a function to each element of a list, and returns a list composed of the elements for which the function returned true. +* Map applies a function to each element of a list, and returns a list of the result of the application of the function to each element. +* Fold. A fold starts from an initial value, then calls a function with the initial value and the first element of the list. Then it calls the function with the result of the previous call, and the next element of the list. This continues until the list end, returning the last result of the function. + +(For example, folding with the sum operator and an initial value of 0, sums the elements of a list.) + +Note that you can implement many list manipulations by composing filters, maps, and folds with different functions. (And by adding more higher-order functions, you can implement more list manipulations.) + +Also, you can manipulate other data structures with equivalent or other higher-order functions. + +Implementing code using higher-order functions and pure functions already has some interesting benefits. + +* Impure functions frequently require more mental overhead to understand, because you need to understand state. With pure functions, you do not have to think about state. +* To understand a program written as a composition of functions, you can start by understanding individual functions and then understand how they fit together. The same program written as a sequence of statements is often more difficult to understand. (However, sometimes the opposite effect occurs.) + +You can use these concepts in most popular programming languages. (Most popular languages also provide higher-order functions such as filters, maps, and folds.) + +So you can get started with functional programming by using the programming languages you already know: + +* Try to write as much code as possible as pure functions. +* Learn which higher-order functions your programming language provides. +* Learn how to implement higher-order functions. +* Write code by composing pure functions with higher-order functions. + +## The consequences of first-class functions, higher-order functions, and pure functions + +Writing code using these concepts often leads to: + +* Writing cumbersome code if the programming language you use lacks certain features. +* Unlocking additional functional programming techniques. + +Therefore, many programming languages provide features that make functional programming more straightforward, or features enabled by functional programming. Languages providing features related to functional programming are commonly named "functional programming languages". + +Although you can use functional programming with non-functional programming languages, this can often lead to: + +* Extra effort +* Not being able to use the full spectrum of functional programming features + +### The need for powerful type systems and type inference + +Higher-order functions often have complex type requirements. For example, to filter a list of a given type, you must pass a function that takes a single argument of that type and returns a boolean. If the arguments do not have the correct types, then the code does not work correctly. + +In languages with dynamic types, the program fails at runtime. In languages with static types, you frequently must specify the types, and higher-order functions often require complex types involving different function types. + +Functional programming languages frequently: + +* Have static types, to prevent frequent runtime failures. +* Automatically infer types instead of requiring programmers to declare them. (However, automatic type inference can cause issues in some scenarios, so frequently programming languages allow writing explicit types, or even require explicit types in some cases.) + +Because functional programs often use more complex types, functional programming languages often have more powerful type systems than non-functional programming languages. + +Derived from those properties, functional programming languages result in the "if it compiles, it works *correctly*" phenomenon. This phenomenon helps avoid incorrect programs. + +## Functional programming languages + +### Haskell + +Functional programming practitioners often recommend Haskell as a functional programming language. + +According to the Wikipedia, "Haskell is a general-purpose, statically-typed, purely functional programming language with type inference and lazy evaluation". Also, Haskell was designed by a committee whose purpose was "to consolidate existing functional languages into a common one to serve as a basis for future research in functional-language design". + +* Haskell is perhaps the language with more built-in functional programming features. As mentioned, Haskell is used for research about functional programming, therefore many new concepts appear in Haskell first. +* Haskell is also very strict about functional programming, so Haskell drives programmers more strongly towards avoiding non-functional programming. +* Haskell syntax is designed so Haskell programs can be extremely terse and contain almost no extraneous syntax. +* Haskell is a very popular language, with a very large ecosystem. You can take advantage of many existing libraries and tools for developing real-world programs faster. + +However, Haskell's benefits frequently also are negative for learning. + +* Haskell uses "lazy" evaluation, where most programming languages use "eager" evaluation. Haskell does not evaluate expressions until needed (and might not evaluate some expressions). Lazy evaluation can lead to efficiency and clearer programs. However, lazy evaluation can cause unexpected performance problems. + +=> https://wiki.haskell.org/Foldr_Foldl_Foldl%27 "Foldr Foldl Foldl'" explains how choosing incorrectly among different implementations of fold can lead to impactful performance problems. + +When writing Haskell code for learning, you can likely stumble into issues not present in languages that use eager evaluation. + +* Haskell is very strict about purity. To implement programs that have side effects, such as accessing files, you must use specific language features. Many articles try to explain those features, because many people have trouble understanding them. + +* Many libraries and tools in the ecosystem take advantage of powerful features enabled by Haskell. However, this might cause that using these libraries and tools require the understanding of the features they are based upon. + +Also, Haskell syntax is very terse, which leads to Haskell compilers not providing clear error messages. For example: + +``` +$ ghci +> let sum a b = a + b +> sum 2 2 +4 +> sum 2 2 2 + +<interactive>:3:1: error: + • Non type-variable argument in the constraint: Num (t1 -> t2) + (Use FlexibleContexts to permit this) + • When checking the inferred type + it :: forall {t1} {t2}. (Num t1, Num (t1 -> t2)) => t2 +``` + +In complex programs, programmers new to Haskell might have trouble identifying that a function has been called with an extra argument from that error message. + +Personally, Haskell is my favorite functional programming language. However, I learned Haskell after learning (with teachers and support from others) other functional programming languages. I think that Haskell is ideal to learn the most powerful concepts in functional programming, but it is not as ideal as a first functional programming language. + +(Note that these recommendations come from someone who only has implemented about 50 Project Euler problems in Haskell, and has experimented on and off with the language, but not been paid for it.) + +### Lisp + +Many programmers like Lisp and languages in the Lisp family, such as Scheme or Clojure. Lisp programmers often recommend Lisp to learn functional programming. + +Lisp is a very minimalistic, yet infinitely flexible language. Lisp is extensible, so you can add most programming language features to Lisp, including functional programming features. + +Therefore, you can do functional programming in Lisp, and also benefit from all other Lisp features. + +However, languages in the Lisp family tend to not have static typing and associated features, thus do not frequently exhibit the "if it compiles, it works *correctly*" phenomenon. + +Lisp has one of the simplest syntaxes of any programming language. The simple syntax of Lisp is directly tied to its power. Many favor the Lisp syntax and argue that the syntax makes Lisp better for learning programming. Personally, I find the Lisp syntax hard to read and write, and likely an additional difficulty on top of learning functional programming. + +I recommend learning Lisp because it is a unique programming language that can teach you many programming language concepts that are not present in many other languages. However, I do not recommend Lisp for learning functional programming (unless you already know Lisp). + +(Note that these recommendations come from someone who has some formal training on Lisp but only uses Lisp infrequently [as a recent Emacs user].) + +### The ML family of programming languages + +ML is a language that appeared in 1973. Since then, three dialects have become the most popular implementations of ML: + +* OCaml +* Standard ML +* F# (part of the .NET platform) + +Specifically, OCaml and F# have very strong ecosystems (OCaml because it is a popular and mature language, F# because as part of the .NET platform, it can use many .NET libraries and tools). + +Haskell is inspired by ML, but many of the Haskell features discussed above are not present in the ML languages: + +* MLs have eager evaluation, therefore avoiding the performance pitfalls of Haskell. +* MLs have simpler syntax, therefore frequently leading to clearer error messages. + +For example, compare the following snippet of OCaml to the previous error message example from Haskell: + +``` +$ utop # utop is a friendlier OCaml REPL +# let sum a b = a + b ;; +val sum : int -> int -> int = <fun> +# sum 2 2 ;; +- : int = 4 +# sum 2 2 2 ;; +Error: This function has type int -> int -> int + It is applied to too many arguments; maybe you forgot a `;'. +``` + +In my opinion, OCaml and F# are better languages for the initial learning of functional programming than Haskell. After learning an ML, you are likely more prepared to learn Haskell and more sophisticated functional programming. + +(Note that those recommendations come from someone who only has experimented with OCaml and F#, and learned SML formally.) diff --git a/blog/content/notes/tech/take-the-less-traveled-road.gmi b/blog/content/notes/tech/take-the-less-traveled-road.gmi new file mode 100644 index 00000000..e9107470 --- /dev/null +++ b/blog/content/notes/tech/take-the-less-traveled-road.gmi @@ -0,0 +1,29 @@ +# Take the less traveled road + +> Two roads diverged in a wood, and I— + +> I took the one less traveled by, + +> And that has made all the difference. + +=> https://www.gutenberg.org/cache/epub/59824/pg59824-images.html#THE_ROAD_NOT_TAKEN Robert Frost, The Road Not Taken + +=> https://xkcd.com/743/ Infrastructures, by Randall Munroe + +The prisoner's dilemma describes situations where people can choose to act in their own interest or collaborate with others. I believe the prisoner's dilemma is present everywhere in our daily life and can explain many behaviors and situations. + +One such scenario is when we choose services to use on the Internet. + +Frequently, we choose services that are easy, popular, and even have no cost. + +In the past, even non-technical people got shared hosting and ran WordPress to blog, with nearly total control about their communication. Nowadays, mostly everyone uses something like Twitter and cedes control to a company. + +Using Twitter is easier than self-hosting WordPress. Twitter is so popular, that your message will likely reach more users on Twitter than on an independent blog. And you can use Twitter for free, whereas shared hosting costs you money and time. + +Twitter became dominant and nowadays, many are frustrated by how Twitter has changed. + +Dominance implies lack of competition, which is nearly always bad for the consumer. + +Which means choosing which services to use frequently is a difficult choice. + +We all have limited time and energy, but I propose that whenever you can, you should take the less traveled road. The easy alternative likely benefits you in the short term, but can easily contribute to a monoculture that will damage everyone in the end, including you. diff --git a/blog/content/notes/tech/the-tragedy-of-the-geeks.gmi b/blog/content/notes/tech/the-tragedy-of-the-geeks.gmi new file mode 100644 index 00000000..e5ce1b10 --- /dev/null +++ b/blog/content/notes/tech/the-tragedy-of-the-geeks.gmi @@ -0,0 +1,65 @@ +# The tragedy of the geeks + +Since the first computer entered our home, I was hooked. This happened more than four decades ago, and continuously tinkering with computers has given me a well-paid and comfortable job. + +However, getting such jobs seems linked to spending a significant amount of your personal time practicing your skills. + +Many people seek careers related to computing because jobs have attractive conditions. However, they might later regret the time and energies spent trying to get into the field when they learn that getting a good job requires unexpected effort. + +This document tries to explain to people who want to work with computers this phenomenon, to help them make a better decision. + +## Tinkering + +Working with computers is the only career I can think of where all of the following are true at the same time: + +* You can work on personal projects that are very similar to the projects you would do in a job. +* There is a reputation of abundant well-paid job offers with good conditions. +* Working on personal projects sounds fun. + +This means that many of us end up spending a significant amount of time working on personal projects. This time investment increases our skills and the things we know. + +## Hiring + +Hiring is one of the most highly debated topics in this industry. + +Many people believe that many candidates cannot do the job. There are many stories about new hires who cannot write simple programs. + +Whether this is common or not is not as important as whether people making decisions believe there are large differences between candidates. When people who hire think that their hiring decision is going to have a large effect on them, then they want to make sure that they pick the right person. + +My perception is that most of the organizations that offer good job conditions (and many who do not) try to be very selective in hiring people. + +## Hiring tinkerers + +When you are hiring people, candidates who have spent significant time on personal projects tend to stand out over candidates who have not. + +This improved perception during the hiring process does not necessarily relate to improved performance on the job. However, I believe that people who tinker on their spare time tend to land better jobs. + +## Handing out advice + +Because there are good jobs working with computers, many people think about making a career in the industry. + +There are many curriculums and formal education programs, from shorter (typically one year) to longer (four or five years). + +Some of them provide advice to land a good job, and students who follow programs who do not, tend to ask for advice. In any case, one of the most frequent pieces of advice on the topic, is tinkering on your own time. + +I believe this is actually good advice, as in that it's more likely to be an efficient way to increase your prospects. + +However, remember that hiring is roughly a competitive process. An organization evaluates a group of candidates, and tries to pick the best one. + +So to stand out, if more candidates tinker (because this is effective advice), the more you need to tinker to stand out. + +I cannot estimate how much you need to tinker on your own time to land a good job, but my guess is that it is more than what someone wanting to get into the field expects. + +As long as this dynamic continues, the tinkering required to land a good job will increase. Only reduced competition can reduce the tinkering required, and reduced competition can happen by few factors, such as increased demand for workers, or a reduction in job seekers. + +## Breaking the cycle + +I cannot think of much that we can individually do to break the cycle. + +Maybe if people coming into the field are aware of this phenomenon, they will be able to make a better decision about what to do. + +If a sufficient amount of people decide that the time investment is not worthwhile, then perhaps the competition will decrease. And if people are well informed and decide to move forward, at least they will be less likely to become frustrated or regret their decision. + +## Further reading + +=> https://bertrandmeyer.com/2025/04/23/a-paean-to-programming/ A paean to programming, by Bertrand Meyer |