Some Perspective

Back in 2019 I pushed out a quick snippet talking about why software seems to always be broken. Well if anything has proven since that time to have not corrected itself, this would most certainly be it. But why? We have so many people focused in this field currently that its hard to imagine that we haven't gotten better at producing higher quality software at this point but here we are. I put together a quick sampling below of things that come to my mind pretty quickly when I ask myself what has happened in the past 50+ years.

Early Programmers were Amazing

Everyone should read some of the early books produced by names such as Brian Kerrighan and Dennis Ritchie, Thomas Cormen et al, W. Stevens, and my personal hero, John McCarthy. These people were truly world changing individuals that really understood how computers functioned at the lowest levels and how to write code that not only worked, but worked in a very optimized manner. Unfortunately, I would venture to guess that if I walked into a CS101 class today and asked all the students if they read books by any of these people or even knew who they were that outside of possibly K&R, they would have no idea.

How can you know know where you are going, if you don't know where you originated from? The concepts that these individuals, I mean giants, present are still true today, and yet many people don't know very much about them, or even more painful for me personally, want to know anything about them.

Forgotten, Lost, and Ignored Wisdom

I recently started reading the Lisp 1.5 Programmers Manual and was so overtaken by the knowledge that McCarthy was trying to convey. Understanding what his vision was at the time and how the Lisp Interpreter is designed. For a programmer, its like looking at Mona Lisa as you can see the true beauty in what he was creating. Unfortunately now I see no beauty in the same way when I look at a significant amount of source code. Not so say there isn't still a lot of really good code in existence, but I do see a lot of sloppy copy/pasted code that in many cases, unfortunately resides in production systems.

I believe we need to understand these concepts that were hacked out years ago in order to properly move forward and improve on what our senseis have taught us. Without a basic understanding and the leveraging of their guidance we are destined to continually reinvent the wheel at best, produce inferior software in many cases, or as Jonathan Blow mentioned in the talk I highlighted in my prior post, completely lose that skill to time as has occurred in the past.

Features Verses Quality

If ignorance of software engineering principals wasn't enough, there is now a growing pressure to simply get features shipped and out the door. I suspect everyone reading this who has developed commercial software within an organization has some version of this story running through their mind as they read this. Oh yeah, this one time... we had to get this feature out the door due to the business promises that were made. Its OK, if that happened to you.. know that you aren't alone.

We Need to Ship Next Week, We Made Commitments

This is the classic business comment that frequently rears its ugly head. Was the development team ever consulted before the commitments were made? Was there a change in the requirements during the development cycle? Was there scope creep that appeared in the middle of the cycle? These are all questions that should be asked as to how it reached this point.

It's Only A Small Bug

This is one of my personal favorites... its a small bug, we can live with it in the short term. If you ever hear this red flags should appear immediately. I can guarantee this is going to reappear in some form in the future if a release is cut with "a known small bug", and there will be a major push that will bubble up to the stack as something that needs to be corrected before you can call it a day at some point.

We can Fix that in the Next Release

The immediate follow up to "its a small bug" is this statement. We'll get that next release. Somehow during the initial meetings of every development cycle after this time though, this small bug is always so insignificant that it just seems to fall by the wayside in support of some new feature that just has to get out the door in the upcoming release, because.... hey we made promises.

Summary

Well there you have it, a brief introduction of things that immediately come to my mind as I ponder this subject of software quality today. I suspect that some who read this will consider my thoughts are that of a paranoid crazy open source developer who just doesn't get the modern development process. That CI/CD and agile processes have fixed all of this. I will just answer that with one word, bull. If it were the case, I wouldn't be writing this, and you wouldn't have agreed with one or more of the issues I presented.

Also, I can say that I've had the pleasure to have worked in an organization at one point where code quality was truly put in the fore front. I won't mention the organization but it proved to me that it's possible to write really good high quality code. Believe me, it can be done but everyone involved needs to have that vision. Software quality is really like security at some level, its only as good as its weakest link. We owe it to ourselves and our customers that as the developers we aren't that weakest link. Its how the giants that we hopefully admire and maybe even wanted to emulate when we started down this crazy path of developing software would want us to be.

Until the next blog, don't talk about it...

Shut up and Hack!

Background

Using LSP (Language Server Protocol) in Emacs brings some of the convenience items present in contemporary IDE's into the Emacs environment that we are all familiar with. Clojure has proven to me to be similar from a development standpoint where it is a relatively small core language with a large number of supporting libraries, similar in many ways to Common LISP.

Having the ability to leverage Emacs with LSP for Clojure development was a really high item on my list of integrations as I saw what clojure-lsp does in a FreeBSD and Debian Linux environment and I wanted to have those capabilities available to me in OpenBSD.

The Problem

The main issue faced when working with OpenBSD is generally its compatibility with various libraries used frequently in the Linux community. Linux users generally are happy when they get their code working in that environment, stopping at that point with porting to other environments. Sometimes FreeBSD will be ported to, but OpenBSD is generally left to fend for itself. Also the OpenBSD environment is not one that is as easy to get working due to its more locked down state.

In the case of clojure-lsp, the default binary found at https://github.com/snoe/clojure-lsp does not include support for OpenBSD for the required sqlite-jdbc driver within the download provided. The sqlite DB is used by LSP for maintaining the details of a project in a quickly accessible location I believe. (Disclaimer I haven't looked at the code to confirm this, it's an assumption) Once Emacs has been configured to leverage clojure-lsp, it tries to create a sqlite DB in a directory ".lsp" at the root of the project. When this is attempted in OpenBSD, the sqlite-jdbc driver isn't contained in the shipped binary, so even though the OS is detected, it doesn't have the proper jdbc binary available to make the connection to the DB, and fails. In Emacs it simply appears at the bottom of the screen as "Connecting" and continues to sit there spinning.

Luckily lsp-clojure by default logs to a file /tmp/lsp.out that contained information regarding this failure, giving us a clue to what is happening. Below is an example of the error.

{:type java.lang.Exception
 :message No native library is found for os.name=OpenBSD and
 os.arch=x86_64. 
 path=/org/sqlite/native/OpenBSD/x86_64
 :at [org.sqlite.SQLiteJDBCLoader loadSQLiteNativeLibrary
 SQLiteJDBCLoader.java 333]}

Its pretty apparent from the error that we have a library missing in the shipped binary for

Lets Make this Work

Getting lsp-clojure functional in OpenBSD requires a couple of things to happen.

• Rebuild sqlite-jdbc with support for OpenBSD
• Rebuild clojure-lsp with support for the sqlite-jdbc driver compiled

The next section walks through the process

Prerequisites

• Download the master zip file from the git repo for sqlite-jdbc: https://github.com/xerial/sqlite-jdbc
• Clone the git repo for clojure-lsp: https://github.com/snoe/clojure-lsp
• Ensure the following packages are installed for compiling
  • bash
  • gmake
  • lein
  • maven
• Emacs has been properly configured for LSP and lsp-clojure as noted on their respective web-sites.

Compiling an OpenBSD version of the sqlite-jdbc driver

• Unzip sqlite-jdbc-master.zip
• Edit the Makefile so it only compiles native
  • modify the line as shown in Makefile

#native-all: native win32 win64 mac64 linux32 linux64 linux-arm
#linux-armv7 linux-arm64 linux-android-arm linux-ppc64 alpine-linux64

native-all: native

• Using gmake compile sqlite-jdbc
  • The native compile will only build the host OS version
  • It was not possible to get the other OS versions to build easily, so they were ignored.

% gmake

Once its built, we need to have this installed in the local .m2 repository of the user's home directory. This is needed as when we build clojure-lsp we need to have this driver available. We use maven to perform the installation locally. This will run through some tests as part of the install process and at the end of the process an INFO line shows where it was installed.

$ mvn install 

[INFO] Scanning for projects...
[INFO] 
[INFO] -----------------------< org.xerial:sqlite-jdbc >-----------------------
[INFO] Building SQLite JDBC 3.31.2-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------

[INFO] --- maven-install-plugin:2.4:install (default-install) @ sqlite-jdbc ---
[INFO] Installing
/home/roger/Downloads/sqlite-jdbc-master/target/
sqlite-jdbc-3.31.2-SNAPSHOT.jar 
to /home/roger/.m2/repository/org/
xerial/sqlite-jdbc/3.31.2-SNAPSHOT/sqlite-jdbc-3.31.2-SNAPSHOT.jar
[INFO] Installing /home/roger/Downloads/sqlite-jdbc-master/pom.xml to 
/home/roger/.m2/repository/org/xerial/sqlite-jdbc/3.31.2-SNAPSHOT/
sqlite-jdbc-3.31.2-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  11.980 s
[INFO] Finished at: 2020-06-07T15:49:36-04:00
[INFO] ------------------------------------------------------------------------

Now that the driver is ready and installed, go to the git cloned directory for clojure-lsp and make the following changes to the deps.edn file. Make sure that the versions being built match your individual versions. By default Leiningen will look to the .m2 directory of the user running the build first, if it doesn't find the file there, it the searches these: clojars.org and Maven Central. By installing the library to the local cache, it will not go out to look for it. If it does try to look outside of the local repo, obviously this will fail.

;;org.xerial/sqlite-jdbc {:mvn/version "3.21.0.1"}
org.xerial/sqlite-jdbc {:mvn/version "3.31.2-SNAPSHOT"}

Now its time to build a new version of clojure-lsp with support for OpenBSD. By default lein does not allow SNAPSHOTS in release build libraries as dependencies, it will trigger an error. If a SNAPSHOT was used as above, its necessary to tell Leiningen this is OK before we build clojure-lsp.

% cd <CLOJURE_LSP_DIR>
% LEIN_SNAPSHOTS_IN_RELEASE=1
% export LEIN_SNAPSHOTS_IN_RELEASE
% <PATH TO LEIN> bin
WARNING: You have $CLASSPATH set, probably by accident.
It is strongly recommended to unset this before proceeding.
WARNING: seqable? already refers to: #'clojure.core/seqable? in
namespace: 
clojure.core.incubator, being replaced by: #'clojure.core
.incubator/seqable?
WARNING: seqable? already refers to: #'clojure.core/seqable? in
namespace: 
clostache.parser, being replaced by: #'clojure.core.incub
ator/seqable?
Compiling 1 source files to /home/roger/src/clojure/clojure-lsp/target/classes
Compiling clojure-lsp.clojure-core
Compiling clojure-lsp.crawler
Compiling clojure-lsp.db
Compiling clojure-lsp.handlers
Compiling clojure-lsp.interop
Compiling clojure-lsp.main
Compiling clojure-lsp.parser
Compiling clojure-lsp.refactor.edit
Compiling clojure-lsp.refactor.transform
Compiling clojure-lsp.shared
Created /home/roger/src/clojure/clojure-lsp/target/
clojure-lsp-release-20200514T134144.jar
Created /home/roger/src/clojure/clojure-lsp/target/
clojure-lsp-release-20200514T134144-standalone.jar
Creating standalone executable: /home/roger/src/clojure/clojure-lsp/
target/clojure-lsp
Re-aligning zip offsets

Copy the clojure-lsp file created as "standalone executable" to a location available in your PATH within your .emacs.

Open Emacs and load a clojure file from a project into Emacs.

When this is performed, it should launch the lsp process and you will see some message indicating STATUS:starting at the bottom of the Emacs window. After some time.. depending upon the processor, maybe 20-30 seconds, that should change to:

LSP :: clojure-lsp:\<PID\> initialized successfully

If a message like this appears, you did not set the environment variable properly

Release versions may not depend upon snapshots. 
Freeze snapshots to dated versions or set the
LEIN_SNAPSHOTS_IN_RELEASE environment variable to override.

Congratulations, you now have clojure-lsp working in OpenBSD!

If this doesn't happen and you have failures, or you see it sitting in the "starting" state for a long time, check the /tmp/lsp.out file to see what the error is.

When the driver is in place, once the nrepl server starts, it notices the DB was not present in the file system. Note, the only time that this particular warning was logged was when the system was able to load the OpenBSD version of the sqlite-jdbc driver.

INFO  clojure-lsp.main: :initialize 55577 ()
INFO  clojure-lsp.main: Shutting down
INFO  clojure-lsp.main: Server started
INFO  clojure-lsp.main: ====== LSP nrepl server started on port 32021
WARN  clojure-lsp.main: Initialize
WARN  clojure-lsp.db: Could not load db [SQLITE_ERROR] SQL error or
missing database (no such table: project)

Enjoy your new installation!

Until the next blog, don't talk about it...

Shut up and Hack!

Some History

I keep thinking about all of the killer applications and programming languages that have come along in recent years and the impacts that they made to the industry. Its also interesting to note how some of them fall off the radar over time that were once really popular. Ruby being the one that comes to mind in recent years. I suspect it really wasn't as great of a language but was instead propped up by an application, in this case Rails that used it. Having used Ruby for a number of years, I would say that I definitely have a preference for Python over Ruby and we now see Python moving into the DevOps world and becoming popular all over again there. That is in spite of the Python2 verses Python3 issues that keep raising their head. So Python went the way of Django and Ruby went the way of Rails, I could argue that Rails was more popular that Python for a number of years but now that has reversed. Anyone who has tried to update Rails or a number of Gems probably knows one of the major reasons why I believe Python is simply a better language to work with. Gems tend to be a nightmare, not when you first install them, but when you try to upgrade them down the road. So much so that people tended to not upgrade their Rails instances and before you know it, the version is no longer supported. I've seen in multiple times.

But what does this have to do with Clojurescript?

Think about how pervasive JavaScript has become to web development since it began back in the 90's. Currently it is pretty much impossible to have a web application that doesn't have some significant amount of JavaScript (JS) running asynchronous calls back to a web server that is probably leveraging REST to serve the queried information back to the client. That should give you some idea of how critical JS is to modern applications. Meanwhile there have been books written that have talked about how quickly JS was created, some stating as little as ten days. Not much time to get things right.

Anyone who has worked in JS for some time that isn't a full time JS developer, I put myself in that category, is likely to state that JS can be frustrating to work with and that errors seem to appear out of nowhere due to the syntax or just general semantics of the language.

So we have two things happening, an explosion of the language use coupled with a somewhat cantankerous language that is prone to errors. This is a recipe for exploits. So much so that in watching a YouTube video of Def-Con a few years ago they consider JS exploits basically like the kiddie pool for hackers. Definitely something that doesn't seem like we want to advocate using, but here we are and we are living with it daily. This doesn't mean JS is bad when programmed correctly, it simply means that programming it correctly can be tricky and is probably best left to dedicated front end developers for anything other than trivial use cases.

So what you would say to having a language that would let you compile your code behind the scenes into JS, but allow you to write something that looks exactly like Clojure to the point of many times being able to be shared by Clojure? In addition, when its compiled, it can be minimized for you automatically and even ran through the Google Closure compiler and optimized for performance. In addition to that, you can even "plug into" a running instance, update the code you program in, and the editor can then compile and restart the web instance updating the JS on the fly. Sound too good to be true, well its been available for a few years now, and its called Clojurescript.

In a future post, I will dig more into Clojurescript now that you have hopefully gotten a small taste of what it can do and are at least intrigued into digging into it a bit further.

Until the next blog, don't talk about it...

Shut up and Hack!

Something that I have been recently thinking a lot about is how bad software appears to be in recent times. You can't seem to go through a day without running into something that doesn't work as you would expect or worse, just not work at all. If you start to search the web you will see others that others have also started thinking about this same phenomena in recent years. I've watched a handful of YouTube videos in the past few weeks, that are all eluding the same thing. If you search for them, I'm sure you will find many. An example of one that I recently watched is from Jonathan Blow. His premise is basically that we have lost the understanding of the roots of development due to abstractions, and that this is something that has occurred over time for millennia with different domains, such as during the bronze age. Its an interesting talk and I see where he is coming from and I think he has some points that are valid, but he does come off as a bit of an alarmist, at least to me.

So as time permits, and I really have no schedule, I wanted to push out some posts on this topic. What I think is happening (again just my own opinions) and what I think are ways forward. I don't believe this is an Kobayshi Maru scenario that Jonathan seems to think, but it is a problem, one that is solvable.

Until the next blog, don't talk about it...

Shut up and Hack!

Python is sooo baller. Been trying to get this Haskell app to fire up a web server locally for my dev instance and I’m thinking forking, rebuilding, yada yada yada. I wonder if there is some pythonic way to do this so I don’t have to configure httpd or jump through so many hoops and sure enough...bam, one liner. Kick it off in Emacs and it logs to stdout as a bonus.

This is a huge time saver and for a development system is all that is required to have a good experience coding/compiling/viewing all within the confines of the Emacs editor. No need to switch screens to a terminal just to restart the web server as needed.

Until the next blog, don't talk about it...

Shut up and Hack!