Дойде време да ъпгрейдна от Debian Buster към BullsEye .

Новият стабилен Дебиан излезе това лято. Нямах особен зор да ъпгрейдвам, но ми беше в TODO списъка.

Първо ъпгрейднах лаптопа, където нямаше проблеми, но там и няма кой знае какво конфигурирано, понеже го ползвам рядко.

Десктопа ми беше сложен микс от:

• Buster (основна)
• Buster-Backports
• BullsEye
• Testing
• Unstable
• Custom repositories

На него се ползват 3 работни места с три видеокарти, всяко с по няколко монитора, собствени клавиатури и мишки, саундкарти, принтери и др.

Има активни Docker и LXC контейнери. Отделно има VirtualBox, systemd-nspawn, SnapD .. През годините се тестват разни мрежови конфигурации за каквото ли не. От bridges, firewalls, proxies, load balancers, уеб и файлови сървъри, файлови системи ..

Миксираната система се поддържаше от сложен микс apt_preferences, holds, sources.list.

Бих казал, че ъпгрейда мина много плавно, като нещата от които очаквах проблем( binary драйверите на NVidia, root on ZFS, KDE, multi seat) минаха гладко.

Единственият не очевиден проблем, който възникна (на този етап поне) беше с akonadiserver . Крашваше дори и след като му затърках напълно базата и конфигурацията. Проблемът се оказа, че са сложили apparmor policy, което предполагаше файловете му да са на същият дял като HOME директорията, а аз съм конфигурирал отделен потребителски VOLATILE дял за кешове и подобни, да не ми пълнят ZFS снапшотите.

Та въпросното policy не му позволяваше достъп до файловете, които имаше нужда да ползва и крашваше при всеки опит за старт.

Та бързият фикс беше:

aa-complain /etc/apparmor.d/usr.bin.akonadiserver

Правилното планиране направи връщането на конфигурацията лесно с един zfs rollback към snapshot-а, който направих преди ъпгрейда.

Ъпгрейда отне няколко часа, може би 5-6, като голямата част от тях бяха затова, защото държах да следя какво се случва отблизо и гледах да не си спестявам стъпки, които да ме доведат до пъти повече изгубено време впоследствие.

Поразгледах и разчистих и разни стари пакети и конфигурации.

Dear Debian,

OS ideology says - release early, release often.

Instead of doing "the freeze" I would go for RC releases. IMHO this would simplify the process a lot:

• in sync with the release practice of most of the OSS out there
• "testing" will not be blocked
• no need to do complicated things, like blocking package migrations, submission of new packages, etc.
• ...

Then, when you feel you're in a state for a new stable release just "name it".

A temporary intermediate version can be introduced:

• experimental
• unstable
• testing
• RC
• stable
• oldstable
• ...

Passing boot parameters to ScaleWay's baremetal C1 instance Linux kernel

Short story

Add tags like these to your server:

KEXEC_KERNEL=http://mirror.scaleway.com/kernel/armv7l-mainline-lts-4.9-4.9.93-rev1/vmlinuz
KEXEC_INITRD=http://mirror.scaleway.com/initrd/uInitrd-Linux-armv7l-v3.14.6
KEXEC_APPEND=vmalloc=512M

Longer story

The ScaleWay's "BareMetal" "C1" instance is a cheap EUR 3 / month cloud infrastructure instance. It has:

• 4 32bit armv7l cores
• 2 GB RAM
• 50 GB network attached storage
• 1 public IP included in the price

ScaleWay offers two lines of servers:

• BareMetal
• VirtualMachines (KVM based)

One important difference between the two is that:

• A VM can only be booted with as much storage as included in its offer
• Bare metal instances support attaching up-to 15x150 GB additional network block device drives ( charged EUR 1/month per 50 GB )

Another important difference is that currently in ScaleWay infrastructure, contra-logically:

• Only VMs can run custom kernels
• Bare metal servers come with e pre-build kernels and ScaleWay does not officially support changing these kernels. You can't even run the official kernel that comes with the chosen Linux distro.

Thus a problem arises you need to change something.

My case was that I wanted to use ZFS and it is not included in the official Linux kernel. It is rather build as a module. On standard Debian it is done easily by installing the zfs-dkms package.

It is possible to build the module for the C1 instance kernel by preparing the build env like described here:

• https://github.com/scaleway/kernel-tools#how-to-build-a-custom-kernel-module

The problem was, that ZFS on 32bit Linux:

• "May encounter stability problems"
• "May bump up against the virtual memory limit"

which is officially stated here:

• https://github.com/zfsonlinux/zfs/wiki/FAQ#32-bit-vs-64-bit-systems

I'm stil about to see the former but hit the latter quite fast, and as recommended I had to add the vmalloc=512M boot parameter.

Unfortunately Scaleway does not support passing parameters to their kernels. They however support KEXEC via the KEXEC_KERNEL and KEXEC_INITRD params as documented here:

• https://github.com/scaleway/initrd/blob/master/Linux/README.md#boot-options

and they support parameters to the KEXEC-ed kernel via the KEXEC_APPEND param.

So as I just needed to boot the same kernel and pass the parameter. So first I had to find where the current kernel and initrd are. This is done by installing "scaleway-cli":

• https://github.com/scaleway/scaleway-cli

I've just grabbed the pre-built amd64 deb packages, and then used the "scw" command to get info about the instance:

# list servers
$ scw ps 

# Show instance details 
$ scw inspect SERVER_ID

"bootscript": {
    "bootcmdargs": "LINUX_COMMON scaleway boot=local nbd.max_part=16",
    "initrd": "initrd/uInitrd-Linux-armv7l-v3.14.6",
    "kernel": "kernel/armv7l-mainline-lts-4.9-4.9.93-rev1",
    "dtb": "dtb/c1-armv7l-mainline-lts-4.9-4.9.93-rev1",
    ...

If you inspect a VM instance you will see that the kernel and initrd are referred by IP:

"bootscript": {
    "bootcmdargs": "LINUX_COMMON scaleway boot=local nbd.max_part=16",
    "initrd": "http://169.254.42.24/initrd/initrd-Linux-x86_64-v3.14.6.gz",
    "kernel": "http://169.254.42.24/kernel/x86_64-mainline-lts-4.4-4.4.127-rev1/vmlinuz-4.4.127"

And a google search showed me that the kernel and the initrd were available at:

• http://mirror.scaleway.com/kernel/
• http://mirror.scaleway.com/initrd/

I've had a problem by trying to use the image referred in the params above:

# DO NOT USE THIS ONE
KEXEC_INITRD=http://mirror.scaleway.com/initrd/uInitrd-Linux-armv7l-v3.14.6

and I've wasted a couple of hours until I realized that this image was in a different format, not usable for the KEXEC_INITRD . Then I've changed it to:

 KEXEC_INITRD=http://mirror.scaleway.com/initrd/initrd-Linux-armv7l-v3.14.6.gz

and this time it worked fine.

The kernel can be found via at least two different URLs:

KEXEC_KERNEL=http://mirror.scaleway.com/kernel/armv7l-mainline-lts-4.9-4.9.93-rev1/vmlinuz
             http://mirror.scaleway.com/kernel/armv7l/4.9.93-mainline-rev1/vmlinuz

And after the successfull boot I've just had to add:

KEXEC_APPEND=vmalloc=512M

And my ZFS module was no longer complaining about lack of virtual memory.

Let me add a few articles that were helpful:

• Running latest kernels on ARM Scaleway servers
• https://nixos.wiki/wiki/NixOS_on_ARM/Scaleway_C1
• How to boot a kernel with different cmdline? #378
• Linux kernel in ScaleWay community

I've wasted about a day while investigating this stuff. If you find it helpful and you think that I might have saved you a couple of hours you can decide to send me a small donation on this PayPal e-mail: krustev-paypal@krustev.net

GigaByte T1028M with Debian Buster

I bought this small netbook about 10 years ago(2008) and my grandma was using it for Skype in the last 5+ years. I got her a tablet recently as the netbook was getting unsupported and quite slow (WinXP, 32 bit only).

When I bought it I've installed Debian on it and kept the WinXP available on a small resized partition. It was a pain back then to get some of the hardware working, e.g. the touch screen and the Elantech touchpad.

When I got it back I decided to update it to an up-to date Debian and since Buster was coming in a few months it was the candidate. I expected everything to be working out-of-the box but unfortunately it wasn't. So I've decided to put down this little article as a reminder.

Elantech touchpad not working

The multitouch mouse was not working after default Debian Buster install so:

# Add the following kernel parameter to /etc/default/grub
i8042.noloop=1

Suspends on each 10 seconds before logging-in

# Make sure you have this in /etc/systemd/logind.conf
HandleLidSwitch=ignore

Hardware

• The processor is very slow - Intel Atom N270, 1.6 GHz, Single core with HyperThreading
• HDD is very very slow. I will probably change it with a small SSD laying around.
• Only 1GiB DDR2 RAM . Upgradeable to 2GiB (one DIMM socket only supporting 2GiB)
• I think the mouse is generating interrupts, so "paste" does not work in the GUI (currently Cinnamon)
• The touch screen worked with no additional configuration
• It has BlueTooth, SD CardReader, Internal 3g modem with a SIM card slot, Camera

Keyboard

Bulgarian phonetic keyboard layout does not work with the default keyboard type for a US locale. I had to choose Acer laptop .

BIOS

BIOS update to latest version - May 10th 2010, ver.FD , is supposed to fix some problems, e.g.:

• Add SSD flash drives support
• WiFi card On by default (confirmed)

To upgrade the BIOS I've used Grub chainload to FreeDOS .

Desktop environment and software

Desktop environment

I've tried all offered desktop environments to choose what works best.

• Gnome is very slow and eats too much memory
• KDE/Plasma is my favorite on the desktop. It mostly works but is sluggish from time to time on this tiny CPU.
• I've not experimented too much with Cinammon/LXDE

I've stuck with MATE as it works relatively fast and is lightweight. I was also able to set it up to my preferences with very little configuration .

Boot manager

• I've chosen lightdm as it seems fast and lighweight
• gdm is sluggish
• sddm not that much but still slower than lightdm

Web browsers

At the time of install chromium was not available in Buster. So I've set up a mixed Buster/Sid system in order to be able to use it from Sid. It seems that ffmpeg dependencies were blocking its Buster presence.

• Firefox works but
• Chromium looks faster. Heavy sites like Facebook are slow. Youtube plays videos in its 360p resolution fine. 480p also works.

Честит празник на системните администратори! :-)

Поради спиране на тока нощес имах привилегията да си ъпдейтна десктопа:

root@work:/# cat /etc/issue.net
Debian GNU/Linux wheezy/sid
root@work:/# last reboot | head -3
reboot   system boot  3.0.0-1-686-pae  Fri Jul 29 15:11 - 20:29  (05:17)    
reboot   system boot  2.6.38-2-686-big Fri Jul 29 13:25 - 15:09  (01:43)    
reboot   system boot  2.6.38-2-686-big Thu Apr  7 20:41 - 15:09 (112+18:28) 
root@work:/# uname -a
Linux work 3.0.0-1-686-pae #1 SMP Sun Jul 24 14:27:32 UTC 2011 i686 GNU/Linux

И макар и леко на патерици - Честита 20 годишнина на Линукс и честита ни 3-та версия! :-)

java.net.SocketException: Network is unreachable

I hit this about a week ago . First time I saw it was on my office desktop running Debian unstable. Since I was not doing too much Java on it I decided it was a problem with JConsole. I nearly lost a bet out of this:

I was pretty sure JConsole was able to attach to local processes even when they were started without any JMX options enabled. Borislav Tonchev was pretty sure it wasn't. I quickly wrote a Java class with its main method sleeping for 100 seconds and tried to attach to its process. Unfortunately I wasn't able to do so. At that point Borislav walked away with 10 bucks coming out of my pocket.

I was curious enough to check this stuff and at first appeared that Java didn't like the bsdgroups option my ext3 /tmp file system was mounted with. Trying the same thing on my home PC, with bsdgroups disabled showed this java.net.SocketException: Network is unreachable. At this point I was starting to loose ground. I decided to check the docs ( http://java.sun.com/javase/6/docs/technotes/guides/management/jconsole.html ) and they confirmed my point. I checked the documented behavior in a JVM running inside an Windows XP installation I have ( VirtualBox image for the corporate stuff in the office ) and Borislav unhappily brought my money back.

At this point I decided the exception under Debian was caused by a bug in JConsole - probably it was not maintained too much in recent releases as a similar tool appeared - VisualVM.

Several days after this long background, on Saturday, I've also hit the same exception on a production server running Tomcat. Pretty damn strange. I was not able to figure it out immediately. The actual problem was introduced in Debian in the beginning of December last year, with the netbase package setting:

# cat /etc/sysctl.d/bindv6only.conf
net.ipv6.bindv6only=1

This did not showed up on the server immediately, since the netbase upgrade did not apply the new setting. The exception appeared after a restart almost two months after the upgrade.

The workaround is to set the above to "0" as it was before, or to add the option -Djava.net.preferIPv4Stack=true to each Java process you start. I prefer the former as I did not want to configure every Java program (e.g. I use azureus/vuze) manually.

More information could be found in Debian bug #560044