November 2023 - wlug - list.wlug.org.nz

Can AI Be Used to Fine-Tune Linux Kernel Performance?
by Peter Reutemann 21 Nov '23

21 Nov '23

'ZDNet: At the Linux Plumbers Conference, the invite-only meeting for the top Linux kernel developers, ByteDance Linux Kernel Engineer Cong Wang, proposed that we use AI and machine learning to tune the Linux kernel for the maximum results for specific workloads... There are thousands of parameters. Even for a Linux expert, tuning them for optimal performance is a long, hard job. And, of course, different workloads require different tunings for different sets of Linux kernel parameters... What ByteDance is working on is a first attempt to automate the entire Linux kernel parameter tuning process with minimal engineering efforts. Specifically, ByteDance is working on tuning Linux memory management. ByteDance has found that with machine learning algorithms, such as Bayesian optimization, automated tuning could even beat most Linux kernel engineers. Why? Well, the idea, as Wang wryly put it, "is not to put Linux kernel engineers out of business." No, the goal is "to liberate human engineers from tuning performance for each individual workload. While making better decisions with historical data, which humans often struggle with. And, last, but never least, find better solutions than those we come up with using our current trial and error, heuristic methods. In short, ByteDance's system optimizes resource usage by making real-time adjustments to things like CPU frequency scaling and memory management.' -- source: https://linux.slashdot.org/story/23/11/20/0153201 Cheers, Peter -- Peter Reutemann Dept. of Computer Science University of Waikato, Hamilton, NZ Mobile +64 22 190 2375 https://www.cs.waikato.ac.nz/~fracpete/ http://www.data-mining.co.nz/

1 0

Linux Foundation Announces Intent to Form 'High Performance Software Foundation'
by Peter Reutemann 20 Nov '23

20 Nov '23

'This week the Linux Foundation "announced the intention to form the High Performance Software Foundation. "Through a series of technical projects, the High Performance Software Foundation aims to build, promote, and advance a portable software stack for high performance computing by increasing adoption, lowering barriers to contribution, and supporting development efforts." As use of high performance computing becomes ubiquitous in scientific computing and digital engineering, and AI use cases multiply, more and more data centers deploy GPUs and other compute accelerators. The High Performance Software Foundation intends to leverage investments made by the United States Department of Energy's Exascale Computing Project, the EuroHPC Joint Undertaking, and other international projects in accelerated high performance computing to exploit the performance of this diversifying set of architectures. As an umbrella project under the Linux Foundation, HPSF intends to provide a neutral space for pivotal projects in the high performance software ecosystem, enabling industry, academia, and government entities to collaborate together on the scientific software stack. The High Performance Software Foundation already benefits from strong support across the high performance computing landscape, including leading companies and organizations like Amazon Web Services, Argonne National Laboratory, CEA, CIQ, Hewlett Packard Enterprise, Intel, Kitware, Lawrence Berkeley National Laboratory, Lawrence Livermore National Laboratory, Los Alamos National Laboratory, NVIDIA, Oak Ridge National Laboratory, Sandia National Laboratory, and the University of Oregon. Its first open source technical projects include: - Spack: the high performance computing package manager - Kokkos: a performance-portable programming model for writing modern C++ applications in a hardware-agnostic way. - AMReX: a performance-portable software framework designed to accelerate solving partial differential equations on block-structured, adaptively refined meshes. - WarpX: a performance-portable Particle-in-Cell code with advanced algorithms that won the 2022 Gordon Bell Prize - Trilinos: a collection of reusable scientific software libraries, known in particular for linear, non-linear, and transient solvers, as well as optimization and uncertainty quantification. - Apptainer: a container system and image format specifically designed for secure high-performance computing. - VTK-m: a toolkit of scientific visualization algorithms for accelerator architectures. - HPCToolkit: performance measurement and analysis tools for computers ranging from laptops to the world's largest GPU-accelerated supercomputers. - E4S: the Extreme-scale Scientific Software Stack - Charliecloud: high performance computing-tailored, lightweight, fully unprivileged container implementation.' -- source: https://tech.slashdot.org/story/23/11/19/1920250 Cheers, Peter -- Peter Reutemann Dept. of Computer Science University of Waikato, Hamilton, NZ Mobile +64 22 190 2375 https://www.cs.waikato.ac.nz/~fracpete/ http://www.data-mining.co.nz/

1 0

Governments turn to Open Source for sovereignty
by Peter Reutemann 20 Nov '23

20 Nov '23

Came across this news item on my Mastodon feed: 'The German government has launched a new Open Source software project called openDesk, which aims to reduce the country’s dependency on proprietary software vendors and support transparency and interoperability. openDesk is a collection of Open Source software modules that are important for day-to-day work in the public sector, such as text creation, file collaboration, project management, email, calendar and messaging. While the initial focus is on public administration in Germany, The source code is publicly available on a government-provided GitLab called openCoDE, making it available, free of charge, for use by other organizations, industries and geographical location. Discussions are taking place with other European countries; France and Austria in particular have already expressed interest. Commissioned by the German Federal Ministry of the Interior and Home Affairs (BMI), under the project leadership of a newly created Center for Digital Sovereignty, ZenDiS.' -- source: https://mastodon.social/@janvlug/111438616208935401 Cheers, Peter -- Peter Reutemann Dept. of Computer Science University of Waikato, Hamilton, NZ Mobile +64 22 190 2375 https://www.cs.waikato.ac.nz/~fracpete/ http://www.data-mining.co.nz/

2 1

Rust in Linux: Maturing with Support from Cisco, Samsung, Canonical
by Peter Reutemann 20 Nov '23

20 Nov '23

'ZDNet shares on update on "Rust in Linux: Where we are and where we're going next," citing a talk at the Linux Plumbers Conference in Richmond, Virginia by Linux/Rust developer Miguel Ojeda: In brief, Rust Linux is continuing to mature and is getting strong support from developers and vendors, such as Cisco, Samsung, and Canonical... Rust is taking the steps it needs to become — along with C — a fully-fledged member of the Linux language toolchain... That's not to say that we're ready to retire C for Rust just yet. In fact, that day is unlikely ever to come. But Rust is definitely on its way to becoming an important language for Linux development... As for the day-to-day work that's required to make Rust fully integrated with Linux, the "official" website of the initiative is now the self-explanatory Rust for Linux. This site is your one-stop source for all things Rust on Linux... However, the move forward has not been straightforward. Rust on Linux developers have discovered some problems along the way. For example, while deadlocks, when two or more threads are waiting on the other to finish, are safe in Rust, because they don't result in undefined behavior, they're not safe in the Linux kernel. The programmers are working on fixing this issue... A related issue is that there's growing interest in backporting Rust support into long-term support (LTS) versions of Linux, specifically 5.15 and 6.1. Some people are especially showing interest in the super LTS Linux 6.1 kernel. However, Linux doesn't generally allow backports into LTS Linuxes. So, if you really, really want fully featured Rust support in an older LTS kernel, you're going to need to pay for it in one way or the other. Another general rule that Rust developers have decided they're going to try to "break" is the rule against duplicate drivers. Usually, no one wants anyone wasting time reinventing the wheel, but some maintainers are open to the idea of experimenting with Rust, by starting simple with a driver they're already familiar with... These movements are small steps forward, but they're all critical for making Rust equal to C as a Linux programming language.' -- source: https://linux.slashdot.org/story/23/11/18/012228 Cheers, Peter -- Peter Reutemann Dept. of Computer Science University of Waikato, Hamilton, NZ Mobile +64 22 190 2375 https://www.cs.waikato.ac.nz/~fracpete/ http://www.data-mining.co.nz/

1 0

Raspberry Pi OS, elementary OS Will Default to Wayland
by Peter Reutemann 20 Nov '23

20 Nov '23

'Recently the Register pointed out that the new (Debian-based) Raspberry Pi OS 5.0 has "a completely new Wayland desktop environment replacing PIXEL, the older desktop based on LXDE and X.org, augmented with Mutter in its previous release." And when elementary OS 8 finally arrives, "the development team plans to finally shift to the Wayland display server by default," reports Linux magazine (adding "If you'd like to get early access to daily builds, you can do so by becoming an elementary OS sponsor on GitHub.") "This is a transition that we have been planning and working towards for several years," writes CEO/co-founder Danielle Foré, "and we're finally in the home stretch... Wayland will bring us improved performance, better app security, and opens the doors to support more complex display setups like mixed DPI multi-monitor setups." There are other things that we're experimenting with, like the possibility of an immutable OS, and there are more mundane things that will certainly happen like shipping Pipewire. You'll also see on the project board that we're looking to replace the onscreen keyboard and it's time to re-evaluate some things like SystemD Boot. You can expect lots more little features to be detailed over the coming months. Meanwhile, Linux Mint is getting "experimental" Wayland support next month. And also in December, Firefox will let Wayland support be enabled by default. And last month the Register noted a merge request for GNOME to remove the gnome-xorg.desktop file. "To put this in context, the Fedora project is considering a comparable change: removing or hiding the GNOME on X.org session from the login menu, which is already the plan for the Fedora KDE spin when it moves to KDE version 6, which is still in development."' -- source: https://tech.slashdot.org/story/23/11/18/039221 Cheers, Peter -- Peter Reutemann Dept. of Computer Science University of Waikato, Hamilton, NZ Mobile +64 22 190 2375 https://www.cs.waikato.ac.nz/~fracpete/ http://www.data-mining.co.nz/

1 0

The FSF Condemns Unauthorized Derivatives of GNU Licenses
by Peter Reutemann 20 Nov '23

20 Nov '23

'The Free Sofware Foundation issued a clarifying blog post this week, saying the organization is "pleased when people use GNU licenses to distribute and license software." But "we condemn the use of unauthorized, confusing derivatives of the licenses." Unfortunately, some authors engage in confusing practices by drafting licenses using existing terms and conditions of GNU free software licenses, without the intention of granting all four freedoms to users. For example, we have long seen attempts to add restrictions to the license text itself, placed in the LICENSE file, or included elsewhere in the program's release. An example is the so-called "Commons Clause," which, when applied to a free software license, affirms that the program is covered by the license. But, at the same time, is contradicting in its meaning by asserting that selling copies of the program or implementing a commercial service with the program is prohibited. The immediate consequence of the practice of inserting a restriction into a GNU license in this way is the confusion it causes for the community. Users still see the name of the original license, with its preamble and terms and conditions intact, transmitting a strong message that the purpose of the license is to enable users — grant users — their essential software freedoms. This message is clear from the license's text, and is bolstered from the renown accrued by the FSF and GNU trademarks, and their decades of free software advocacy. At the same time, these same users see a contradictory statement of the "Commons Clause," which is clearly contrary to the sprit of the free software movement and the Free Software Definition... [T]o make it even clearer that added restrictions are incompatible with our license, we gave users the right to delete such added restrictions [in 2007] and preserve the program's freedom. But we at the FSF have another legal tool against attempts to release programs under GNU General Public Licenses that have been wrongly altered to become nonfree licenses. The FSF holds copyrights and common law trademarks to the GNU family of General Public Licenses. Moreover, the FSF holds registered trademarks for "FSF," "Free Software Foundation," and "GNU." [...] We can't control the drafting by others of proprietary software licenses, but we can and do forbid doing this in a way that misleadingly associates those licenses with GNU or GNU licenses... [W]e are entitled to legally enforce our copyright and trademark for FSF licenses that have been altered by added restrictions to a verbatim GNU license... Licenses that confuse users about the freedoms they grant are damaging to the free software movement because they threaten to dilute the value and power of these licenses. When GNU licenses are misused through such confusing practices, it harms the renown accrued by the GNU project and the FSF over decades of free software advocacy. It is our duty to all computer users to stop these practices, and, if necessary, we will use our legal rights to this end. ' -- source: https://news.slashdot.org/story/23/11/19/1826218 Cheers, Peter -- Peter Reutemann Dept. of Computer Science University of Waikato, Hamilton, NZ Mobile +64 22 190 2375 https://www.cs.waikato.ac.nz/~fracpete/ http://www.data-mining.co.nz/

1 0

Microsoft Runs Ubuntu
by Lawrence D'Oliveiro 16 Nov '23

16 Nov '23

The November list of the top 500 supercomputers in the world, released a few days ago <https://top500.org/lists/top500/2023/11/> has, at number 3, a machine called “Eagle”, which is apparently part of Microsoft’s Azure cloud. Looking into the details of this <https://top500.org/system/180236/>, the OS is listed as “Ubuntu 22.04”. And there is another Microsoft-based super at number 15 <https://top500.org/system/180171/>, obviously older because its OS is listed as “Ubuntu 20.04”. And at number 23 <https://top500.org/system/180024/>, we have one running “Ubuntu 18.04 LTS”. Then a few more, even older, down in the 50s, all on Ubuntu 16.04 LTS. Conclusion: they won’t even use their own Windows-based Hyper-V virtualization architecture on high-performance installations. The ARM-based Fugaku machine, which was top of the list not so long ago, has dropped to number 4. Apart from that and two IBM POWER-based entries, the rest of the top 10 is currently all x86-based.

1 0

Firefox Going To Ship With Wayland Enabled By Default
by Peter Reutemann 16 Nov '23

16 Nov '23

'Michael Larabel reports via Phoronix: Guardrails have been in place where the Firefox browser has enabled Wayland by default (when running on recent GTK versions) but as of today that code has been removed... Firefox will try to move forward with stable releases where Wayland will ship by default! Mozilla Bug 1752398 to "ship the Wayland backend to release" has been closed this evening! After the ticket was open for the past two years, it's now deemed ready to hopefully ship enabled for Firefox 121! This patch drops the "early beta or earlier" check to let Wayland support be enabled by default when running on recent GTK versions (GTK 3.24.30 threshold). Firefox 121 is due for release around 19 December and if all continues to hold, it will finally ship with the Wayland back-end enabled by default as another big step forward. ' -- source: https://news.slashdot.org/story/23/11/15/2116232 Cheers, Peter -- Peter Reutemann Dept. of Computer Science University of Waikato, Hamilton, NZ Mobile +64 22 190 2375 https://www.cs.waikato.ac.nz/~fracpete/ http://www.data-mining.co.nz/

1 0

Proposed Acquisition Of Serato By Pioneer DJ Ruffling Feathers
by Lawrence D'Oliveiro 15 Nov '23

15 Nov '23

Serato is an NZ-based company that produces a software product that allows “scratching” of audio files on a computer. They provide special vinyl records that contain, not music, but a special time code that the software can recognize whether it is played forwards or backwards or fast or slow at whatever point, and keep the audio in sync. And if and when the vinyl wears out, you can just get new discs--your precious audio data remains pristine, free from the imperfections of being actually being encoded on vinyl. Now, an international company wants to buy them out <https://www.1news.co.nz/2023/11/15/kiwi-start-up-caught-up-in-controversial…>. And another international company, InMusic, is concerned enough about the deal here in little old NZ, to raise an objection. By the way, I find it a bit odd to find Serato described as a “start-up” in the subhead, but as a “software giant” just a few lines further down. I first heard of them about a quarter-century ago, so they are hardly newcomers. I suppose a hundred-million-odd turnover makes them a “giant” in NZ terms, yet the prospect of becoming part of a much larger company could make the present operation seem like a minnow in comparison. Or maybe not: maybe their present size already gives them a huge fraction of what is a fairly niche market. And sure enough, after all this time, there are open-source alternatives <https://alternativeto.net/software/serato-dj/>. I had a quick look at the info on Mixxx <https://mixxx.org/>, and they do mention “vinyl record control”, where you “use turntables with timecode vinyl records to control playback and scratch your digital music files as if they were pressed on vinyl”. Some more info on timecode vinyl is here <https://mixxx.org/news/2021-11-21-dvs-internals-pt1/>. Among the formats supported is the Serato one, and discs are apparently available from the company for “£10-20 each” <http://xwax.org/overview.html>. By the way, Debian includes packages in its standard repo for both Mixxx and xwax.

1 0

Intel Fixes High-Severity CPU Bug That Causes 'Very Strange Behavior'
by Peter Reutemann 15 Nov '23

15 Nov '23

'Ars Technica: Intel on Tuesday pushed microcode updates to fix a high-severity CPU bug that has the potential to be maliciously exploited against cloud-based hosts. The flaw, affecting virtually all modern Intel CPUs, causes them to "enter a glitch state where the normal rules don't apply," Tavis Ormandy, one of several security researchers inside Google who discovered the bug, reported. Once triggered, the glitch state results in unexpected and potentially serious behavior, most notably system crashes that occur even when untrusted code is executed within a guest account of a virtual machine, which, under most cloud security models, is assumed to be safe from such faults. Escalation of privileges is also a possibility. The bug, tracked under the common name Reptar and the designation CVE-2023-23583, is related to how affected CPUs manage prefixes, which change the behavior of instructions sent by running software. Intel x64 decoding generally allows redundant prefixes -- meaning those that don't make sense in a given context -- to be ignored without consequence. During testing in August, Ormandy noticed that the REX prefix was generating "unexpected results" when running on Intel CPUs that support a newer feature known as fast short repeat move, which was introduced in the Ice Lake architecture to fix microcoding bottlenecks. The unexpected behavior occurred when adding the redundant rex.r prefixes to the FSRM-optimized rep mov operation. [...] Intel's official bulletin lists two classes of affected products: those that were already fixed and those that are fixed using microcode updates released Tuesday. An exhaustive list of affected CPUs is available here. As usual, the microcode updates will be available from device or motherboard manufacturers. While individuals aren't likely to face any immediate threat from this vulnerability, they should check with the manufacturer for a fix. People with expertise in x86 instruction and decoding should read Ormandy's post in its entirety. For everyone else, the most important takeaway is this: "However, we simply don't know if we can control the corruption precisely enough to achieve privilege escalation." That means it's not possible for people outside of Intel to know the true extent of the vulnerability severity. That said, anytime code running inside a virtual machine can crash the hypervisor the VM runs on, cloud providers like Google, Microsoft, Amazon, and others are going to immediately take notice.' -- source: https://it.slashdot.org/story/23/11/14/2257235 Cheers, Peter -- Peter Reutemann Dept. of Computer Science University of Waikato, Hamilton, NZ Mobile +64 22 190 2375 https://www.cs.waikato.ac.nz/~fracpete/ http://www.data-mining.co.nz/

1 0