Recently I was talking about LLVM with a colleague and the question arose of exactly why the OpenBSD project has an issue with the LLVM license change that occurred a couple of years ago. As I knew only a cursory level reason as to why this occurred, I started to do a little digging into the history and thought it may be of interest. Here is what I came up with.
Initially the LLVM project was placed under the NCSA license which is a very open license, much like the license that BSD uses in general. If anyone isn't aware of the BSD license model please take some time and research it separate from this entry if you want to fully understand it. There are many licensing models out there and if you think that the Apache 2, GNU, and ISC are all basically the same, I would suggest doing some reading, as that is where much of the reasoning behind this issue stems from, the nuances of the licenses. OpenBSD has a very clearly stated policy at https://www.openbsd.org/policy.html that outlines what they consider acceptable licenses for code that goes into their project.
It turns out that the Apache 2 license is free, but the change to it is what is actually in question. The one called out by OpenBSD in particular is that it was possible to change the license without the consent of the people who are listed as contributors in the NCSA license. In addition, what Theo expanded on was the LLVM team’s apparent “IANAL but let’s do it anyway” attitude that seems to indicate they really did not consult someone as to if it was possible to change the license mid stream as they did. The email in this case it isn’t a piece of software that a single person contributed to, and they would be available to provide consent to the license change, this software had been created by a team of contributors. What happens when someone at a time in the future who is listed on the NCSA license calls them out for not getting their permission to perform the license type change?
At the end of the day the OpenBSD project has been forced to stick with the last NCSA licensed version of LLVM. They generally have forked products in the past when this occurs and then will move forward in one of two ways. Either they will create their own version as in the case of LibreSSL and PF, or they will move to another piece of open source software as they did in the case of going from GCC to Clang. Unfortunately now this has bitten them there may be a tendency to write their own, but that is a huge undertaking. I tend to think they will not go the route they have with LibreSSL just due to the scale of building out a compiler toolchain, but one never knows.
This is how it ended for ipfilter. It occurred midstream in a release cycle, with no packet filter in place that could be used at the time. One had to be written from scratch and within the timelines. This is how PF was born. from the CVS commit message:
Remove ipf. Darren Reed has interpreted his (old, new, whichever) license in a way that makes ipf not free according to the rules we established over 5 years ago, at www.openbsd.org/goals.html (and those same basic rules govern the other *BSD projects too). Specifically, Darren says that modified versions are not permitted. But software which OpenBSD uses and redistributes must be free to all (be they people or companies), for any purpose they wish to use it, including modification, use, peeing on, or even integration into baby mulching machines or atomic bombs to be dropped on Australia. Furthermore, we know of a number of companies using ipf with modification like us, who are now in the same situation, and we hope that some of them will work with us to fill this gap that now exists in OpenBSD (temporarily, we hope)
Until the next blog, don't talk about it...
Using LSP (Language Server Protocol) in Emacs brings some of the convenience items present in contemporary IDE's into the Emacs environment that we are all familiar with. Clojure has proven to me to be similar from a development standpoint where it is a relatively small core language with a large number of supporting libraries, similar in many ways to Common LISP.
Having the ability to leverage Emacs with LSP for Clojure development was a really high item on my list of integrations as I saw what clojure-lsp does in a FreeBSD and Debian Linux environment and I wanted to have those capabilities available to me in OpenBSD.
The main issue faced when working with OpenBSD is generally its compatibility with various libraries used frequently in the Linux community. Linux users generally are happy when they get their code working in that environment, stopping at that point with porting to other environments. Sometimes FreeBSD will be ported to, but OpenBSD is generally left to fend for itself. Also the OpenBSD environment is not one that is as easy to get working due to its more locked down state.
In the case of clojure-lsp, the default binary found at https://github.com/snoe/clojure-lsp does not include support for OpenBSD for the required sqlite-jdbc driver within the download provided. The sqlite DB is used by LSP for maintaining the details of a project in a quickly accessible location I believe. (Disclaimer I haven't looked at the code to confirm this, it's an assumption) Once Emacs has been configured to leverage clojure-lsp, it tries to create a sqlite DB in a directory ".lsp" at the root of the project. When this is attempted in OpenBSD, the sqlite-jdbc driver isn't contained in the shipped binary, so even though the OS is detected, it doesn't have the proper jdbc binary available to make the connection to the DB, and fails. In Emacs it simply appears at the bottom of the screen as "Connecting" and continues to sit there spinning.
Luckily lsp-clojure by default logs to a file /tmp/lsp.out that contained information regarding this failure, giving us a clue to what is happening. Below is an example of the error.
{:type java.lang.Exception
:message No native library is found for os.name=OpenBSD and
os.arch=x86_64.
path=/org/sqlite/native/OpenBSD/x86_64
:at [org.sqlite.SQLiteJDBCLoader loadSQLiteNativeLibrary
SQLiteJDBCLoader.java 333]}Its pretty apparent from the error that we have a library missing in the shipped binary for
Getting lsp-clojure functional in OpenBSD requires a couple of things to happen.
The next section walks through the process
#native-all: native win32 win64 mac64 linux32 linux64 linux-arm
#linux-armv7 linux-arm64 linux-android-arm linux-ppc64 alpine-linux64
native-all: native% gmakeOnce its built, we need to have this installed in the local .m2 repository of the user's home directory. This is needed as when we build clojure-lsp we need to have this driver available. We use maven to perform the installation locally. This will run through some tests as part of the install process and at the end of the process an INFO line shows where it was installed.
$ mvn install
[INFO] Scanning for projects...
[INFO]
[INFO] -----------------------< org.xerial:sqlite-jdbc >-----------------------
[INFO] Building SQLite JDBC 3.31.2-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] --- maven-install-plugin:2.4:install (default-install) @ sqlite-jdbc ---
[INFO] Installing
/home/roger/Downloads/sqlite-jdbc-master/target/
sqlite-jdbc-3.31.2-SNAPSHOT.jar
to /home/roger/.m2/repository/org/
xerial/sqlite-jdbc/3.31.2-SNAPSHOT/sqlite-jdbc-3.31.2-SNAPSHOT.jar
[INFO] Installing /home/roger/Downloads/sqlite-jdbc-master/pom.xml to
/home/roger/.m2/repository/org/xerial/sqlite-jdbc/3.31.2-SNAPSHOT/
sqlite-jdbc-3.31.2-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11.980 s
[INFO] Finished at: 2020-06-07T15:49:36-04:00
[INFO] ------------------------------------------------------------------------
Now that the driver is ready and installed, go to the git cloned directory for clojure-lsp and make the following changes to the deps.edn file. Make sure that the versions being built match your individual versions. By default Leiningen will look to the .m2 directory of the user running the build first, if it doesn't find the file there, it the searches these: clojars.org and Maven Central. By installing the library to the local cache, it will not go out to look for it. If it does try to look outside of the local repo, obviously this will fail.
;;org.xerial/sqlite-jdbc {:mvn/version "3.21.0.1"}
org.xerial/sqlite-jdbc {:mvn/version "3.31.2-SNAPSHOT"}Now its time to build a new version of clojure-lsp with support for OpenBSD. By default lein does not allow SNAPSHOTS in release build libraries as dependencies, it will trigger an error. If a SNAPSHOT was used as above, its necessary to tell Leiningen this is OK before we build clojure-lsp.
% cd <CLOJURE_LSP_DIR>
% LEIN_SNAPSHOTS_IN_RELEASE=1
% export LEIN_SNAPSHOTS_IN_RELEASE
% <PATH TO LEIN> bin
WARNING: You have $CLASSPATH set, probably by accident.
It is strongly recommended to unset this before proceeding.
WARNING: seqable? already refers to: #'clojure.core/seqable? in
namespace:
clojure.core.incubator, being replaced by: #'clojure.core
.incubator/seqable?
WARNING: seqable? already refers to: #'clojure.core/seqable? in
namespace:
clostache.parser, being replaced by: #'clojure.core.incub
ator/seqable?
Compiling 1 source files to /home/roger/src/clojure/clojure-lsp/target/classes
Compiling clojure-lsp.clojure-core
Compiling clojure-lsp.crawler
Compiling clojure-lsp.db
Compiling clojure-lsp.handlers
Compiling clojure-lsp.interop
Compiling clojure-lsp.main
Compiling clojure-lsp.parser
Compiling clojure-lsp.refactor.edit
Compiling clojure-lsp.refactor.transform
Compiling clojure-lsp.shared
Created /home/roger/src/clojure/clojure-lsp/target/
clojure-lsp-release-20200514T134144.jar
Created /home/roger/src/clojure/clojure-lsp/target/
clojure-lsp-release-20200514T134144-standalone.jar
Creating standalone executable: /home/roger/src/clojure/clojure-lsp/
target/clojure-lsp
Re-aligning zip offsetsCopy the clojure-lsp file created as "standalone executable" to a location available in your PATH within your .emacs.
Open Emacs and load a clojure file from a project into Emacs.
When this is performed, it should launch the lsp process and you will see some message indicating STATUS:starting at the bottom of the Emacs window. After some time.. depending upon the processor, maybe 20-30 seconds, that should change to:
LSP :: clojure-lsp:\<PID\> initialized successfullyIf a message like this appears, you did not set the environment variable properly
Release versions may not depend upon snapshots.
Freeze snapshots to dated versions or set the
LEIN_SNAPSHOTS_IN_RELEASE environment variable to override.Congratulations, you now have clojure-lsp working in OpenBSD!
If this doesn't happen and you have failures, or you see it sitting in the "starting" state for a long time, check the /tmp/lsp.out file to see what the error is.
When the driver is in place, once the nrepl server starts, it notices the DB was not present in the file system. Note, the only time that this particular warning was logged was when the system was able to load the OpenBSD version of the sqlite-jdbc driver.
INFO clojure-lsp.main: :initialize 55577 ()
INFO clojure-lsp.main: Shutting down
INFO clojure-lsp.main: Server started
INFO clojure-lsp.main: ====== LSP nrepl server started on port 32021
WARN clojure-lsp.main: Initialize
WARN clojure-lsp.db: Could not load db [SQLITE_ERROR] SQL error or
missing database (no such table: project)Enjoy your new installation!
Until the next blog, don't talk about it...
How many times have you wanted to upgrade the OS but found that it's not going to be as simple as you wanted or thought? This is especially true when you are going to a major version upgrade, but sometimes can even impact you on minor version upgrades.
If you haven't seen this occur, take a look at trying to upgrade a CentOS or RHEL6 system to version 7.X. You will very quickly see something relating to "recommended upgrade procedure is to reinstall from scratch". There is very little that I find more annoying than that. Why can't an OS simply upgrade, when it doesn't require a change if file system or similar? Also, on most systems, once you do upgrade the is a lot of dust left laying around that is no longer used by the system. My only response to this stuff is that the developers are too lazy to come up with a system that actually handles this for you. If you are someone who like me is frustrated at this, take a look at OpenBSD.
Below is an upgrade of one of my OpenBSD systems that I copied the process right from the terminal to show how simple it can be when the developers care about making it simpler. OpenBSD has always been pretty simple, in the past a version upgrade was performed using a boot USB stick, but now its even simpler.
Log into the system as root, and execute "sysupgrade -r"
milliways$ doas sysupgrade -r
SHA256.sig 100% |*****************| 2141 00:00
Signature Verified
INSTALL.amd64 100% |************************| 43550 00:00
base66.tgz 100% |*************************| 236 MB 00:22
bsd 100% |*************************| 18250 KB 00:05
bsd.mp 100% |*************************| 18336 KB 00:05
bsd.rd 100% |*************************| 10058 KB 00:04
comp66.tgz 100% |*************************| 72109 KB 00:11
game66.tgz 100% |*************************| 2745 KB 00:02
man66.tgz 100% |*************************| 7418 KB 00:03
xbase66.tgz 100% |*************************| 22092 KB 00:06
xfont66.tgz 100% |*************************| 39342 KB 00:08
xserv66.tgz 100% |*************************| 15757 KB 00:05
xshare66.tgz 100% |*************************| 4482 KB 00:02
Verifying sets.
Fetching updated firmware.
Upgrading.
Connection to milliways.wilcis.com closed.The system reboots automatically, and then it installs any firmware upgrades necessary. Once it comes back up, simply run the syspatch command which will apply any security or enhancement changes to the new version
Log into the system and run syspatch:
milliways$ doas syspatch
doas (roger@milliways.wilcis.com) password:
Get/Verify syspatch66-001_bpf.tgz 100% |****| 102 KB 00:00
Installing patch 001_bpf
Get/Verify syspatch66-002_ber.tgz 100% |****| 660 KB 00:00
Installing patch 002_ber
Get/Verify syspatch66-003_bgpd.tgz 100% |***| 181 KB 00:00
Installing patch 003_bgpd
Get/Verify syspatch66-004_net8021... 100% |*| 64839 00:00
Installing patch 004_net80211
Get/Verify syspatch66-005_sysupgr... 100% |*| 3023 00:00
Installing patch 005_sysupgrade
Get/Verify syspatch66-006_ifioctl... 100% |*| 381 KB 00:00
Installing patch 006_ifioctl
Get/Verify syspatch66-007_inteldr... 100% |*| 21468 KB 00:06
Installing patch 007_inteldrm
Get/Verify syspatch66-008_mesa.tgz 100% |***| 5598 KB 00:04
Installing patch 008_mesa
Relinking to create unique kernel... done;
reboot to load the new kernel
Errata can be reviewed under /var/syspatchDust is always a problem in every OS I've seen. Dust being old files that are no longer used by the OS post upgrade, but are not removed as part of the upgrade. Most OS's don't even mention it, because.. well space is cheap apparently. They could be an attack vector though if someone had access to a box. OpenBSD also handles this by listing all of the old files that can be manually removed. Every release comes with a page of upgrading instructions that include a section called "Files to Remove". Simply go there, copy the sections and paste them into a terminal... done.
Normally after removing the cruft, I upgrade the installed packages using
$ doas pkg_add -uivReboot and you now have an upgraded system to the latest version... to coin Staples.... That was easy!
Until the next blog, don't talk about it...