Thursday, October 6, 2016

util-linux v2.29 -- what's new?

The release v2.29 (now rc1) is without dramatical changes, the small exception is libsmartcols where we have many improvements. 

The old good cal(1) is more user-friendly now. It's possible to specify month by name (e.g. "cal January 2017") and use relative placeholders, for example:

        cal now
        cal '1 year ago'
        cal '+2 months'

fdisk(8) allows to wipe newly created partitions -- the feature is possible to control by new command line option --wipe-partitions[==auto|never|default]. 
 The default in the interactive mode is to ask user when a filesystem or RAID signature is detected. The goal is to be sure that new block devices are usable without any collisions and extra wipefs(8) step (because users are lazy and mkfs-like programs are often no smart enough to wipe the device). 

findmnt --verify is probably the most attractive new feature for admins. The command scans /etc/fstab and tries to verify the configuration. The traditional way is to use "mount -a" for this purpose, but it's overkill. The new --verify does not call mount(2), but it checks parsability, LABEL/UUID/etc. translation to paths, mountpoints order, support for specified FS types. The option --verify together with --verbose provides many details. 

For example my ext4 filesystems:

# findmnt --verify --verbose -t ext4
/ 
   [ ] target exists
   [ ] LABEL=ROOT translated to /dev/sda4
   [ ] source /dev/sda4 exists
   [ ] FS type is ext4
   [W] recommended root FS passno is 1 (current is 2)
/boot
   [ ] target exists 
   [ ] UUID=c5490147-2a6c-4c8a-aa1b-33492034f927 translated to /dev/sda2
   [ ] source /dev/sda2 exists
   [ ] FS type is ext4
/home
   [ ] target exists
   [ ] UUID=196972ad-3b13-4bba-ac54-4cb3f7b409a4 translated to /dev/sda3
   [ ] source /dev/sda3 exists
   [ ] FS type is ext4
/home/misc
   [E] unreachable on boot required target: No such file or directory 
   [ ] UUID=e8ce5375-29d4-4e2f-a688-d3bae4b8d162 translated to /dev/sda5
   [ ] source /dev/sda5 exists
   [ ] FS type is ext4
 
0 parse errors, 1 error, 1 warning


When you create multiple loop block devices from one backing file then Linux kernel does not care about possible collisions and the same on-disk filesystem is maintained by multiple independent in-memory filesystem instances. The result is obvious -- data lost and filesystem damage.

Now mount(8) rejects requests to create another device and mount filesystem for the same backing file. The command losetup --nooverlap reuse loop device if already exists for the same backing file. All the functionality calculate with offset and sizelimit options of course, so it's fine to have multiple regions (partitions) in the same image file and mount all of them in the same time. The restriction is that the regions should not overlap. Thanks to Stanislav Brabec from Suse! 

Heiko Carstens from IBM (thanks!) has improved lscpu(1) for s390. Now it supports "drawer" topology level, static and dynamic MHz, machine type and a new option --physical. 

The most important libsmartcols change is probably better support for multi-line cells. Now the library supports custom cell wrap functions -- this allows to wrap your text in cells after words, line breaks, etc. See multi-line cells (WRAPNL column) output: 

TREE           ID PARENT WRAPNL
aaaa            1      0 aaa
├─bbb           2      1 bbbbb
│ ├─ee          5      2 hello
│ │                      baby
│ └─ffff        6      2 aaa
│                        bbb
│                        ccc
│                        ddd
├─ccccc         3      1 cccc
│ │                      CCCC
│ └─gggggg      7      3 eee
│   ├─hhh       8      7 fffff
│   │ └─iiiiii  9      8 g
│   │                    hhhhh
│   └─jj       10      7 ppppppppp
└─dddddd        4      1 dddddddd
                         DDDD
                         DD

The another change is support for user defined padding chars; we use this feature for LIBSMARTCOLS_DEBUG_PADDING=on|off, for example: 

   $ LIBSMARTCOLS_DEBUG=all LIBSMARTCOLS_DEBUG_PADDING=on findmnt 2> /dev/null

For me really important is that we have regression tests for all libsmartcols table and tree formatting code now :-) 

Igor Gnatenko from Red Hat (thanks!) continues to work on Python binding for libsmartcols, see https://github.com/ignatenkobrain/python-smartcols and see example below.

The idea is to use libsmartcols as output formatter for Fedora/RHEL dnf (package manager for RPM-based Linux distributions, yum replacement). This is also reason why libsmartcols has been massively extended and improved in the last releases. 

That's all. Thanks also to Werner Fink, Sami Kerola, Ruediger Meier and many others contributors! 

#!/bin/python3
 
import smartcols
 
tb = smartcols.Table()
name = tb.new_column("NAME")
name.tree = True
age = tb.new_column("AGE")
age.right = True
 
ggf = tb.new_line()
ggf[name] = "John"
ggf[age] = "70"
 
gfa = tb.new_line(ggf)
gfa[name] = "Donald"
gfa[age] = "50"
 
fa = tb.new_line(gfa)
fa[name] = "Benny"
fa[age] = "30"
 
ln = tb.new_line(fa)
ln[name] = "Arlen"
ln[age] = "5"
 
ln = tb.new_line(fa)
ln[name] = "Gerge"
ln[age] = "7"
 
fa = tb.new_line(gfa)
fa[name] = "Berry"
fa[age] = "32"
 
ln = tb.new_line(ggf)
ln[name] = "Alex"
ln[age] = "44"
 
print(tb)
  
NAME        AGE
John         70
├─Donald     50
│ ├─Benny    30
│ │ ├─Arlen   5
│ │ └─Gerge   7
│ └─Berry    32
└─Alex       44

Tuesday, March 22, 2016

util-linux v2.28 -- what's new?

The release v2.28 does not contain any dramatical changes and huge improvements. It's another release to keep users happy and absorb new features provided by kernel. We all love this kind of release, right? :-)

The fdisk programs (sfdisk, cfdisk and fdisk) have been improved to wipe old filesystem, RAID and partition tables from the device before libfdisk writes a new partition table.

The fdisk-like programs traditionally care about begin of the device, but it's insufficient. This new feature has been introduced to avoid collisions between new partition table and old unwanted signatures and it's possible to control it by --wipe[=auto|never|always]. For backward compatibility on non-terminals (non interactive fdisk execution) the feature is disabled by default.

We have a new command lsns, see http://karelzak.blogspot.cz/2015/12/lsns8-new-command-to-list-linux.html and sfdisk provides new functionality, see: http://karelzak.blogspot.cz/2015/09/whats-cooking-in-sfdisk-for-v228.html
 
Linux kernel 3.14 is really not a hot news, but standard Linux userspace still does not support DEADLINE scheduler. chrt since v2.28 supports the DEADLINE scheduling class and the new options --sched-runtime --sched-period and --sched-deadline.

The command logger supports RFC 5424 structured data through the new options --sd-id and --sd-param. For example:
   logger --rfc5424 --sd-id zoo@123     \  
          --sd-param tiger=\"hungry\"   \            
          --sd-param zebra=\"running\"  \            
          --sd-id manager@123           \            
          --sd-param onMeeting=\"yes\"  \            
          "this is message"
produces:
 <13>1 2015-10-01T14:07:59.168662+02:00 ws kzak - - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="218616"][zoo@123 tiger="hungry" zebra="running"][manager@123 onMeeting="yes"] this is message

The library libsmartcols has been massively improved (thanks to Igor Gnatenko for testing and reviews). Now it's possible to specify title for table, table supports multi-line cells, it's possible to print subset of table and the library supports continuous printing.

Igor works on Python binding, it's available at https://github.com/ignatenkobrain/python-smartcols.

The portability of the util-linux package is not our primary goal, but in many cases port code to the another libc or another operation system (if possible) is a way how to detect code disadvantages, obsolete functions etc. v2.28 is possible to compile on OSX and improved has been also support for kFreeBSD and GNU Hurd (of course you cannot compile Linux specific stuff, but build-system is smart enough to automatically disable utils irrelevant for your OS).

This is in connection with our regression tests suite, where many things have been improved to make the tests more stable in all random environments. (thanks to Ruediger Meier).

The complete list of all changes (~480 patches) is at
http://ftp.kernel.org/pub/linux/utils/util-linux/v2.28/v2.28-ReleaseNotes

and in details at: http://ftp.kernel.org/pub/linux/utils/util-linux/v2.28/v2.28-rc1-ChangeLog

Thanks to all (~40) contributors!