- wlug - list.wlug.org.nz

Are AI Web Crawlers 'Destroying Websites' In Their Hunt for Training Data?
by Peter Reutemann 01 Sep '25

01 Sep '25

Hi everyone I thought this news item rather fitting when I saw it this morning, considering Daniel's experience with these AI bots hammering the wlug wiki: '"AI web crawlers are strip-mining the web in their perpetual hunt for ever more content to feed into their Large Language Model mills," argues Steven J. Vaughan-Nichols at the Register. And "when AI searchbots, with Meta (52% of AI searchbot traffic), Google (23%), and OpenAI (20%) leading the way, clobber websites with as much as 30 Terabits in a single surge, they're damaging even the largest companies' site performance..." How much traffic do they account for? According to Cloudflare, a major content delivery network (CDN) force, 30% of global web traffic now comes from bots. Leading the way and growing fast? AI bots... Anyone who runs a website, though, knows there's a huge, honking difference between the old-style crawlers and today's AI crawlers. The new ones are site killers. Fastly warns that they're causing "performance degradation, service disruption, and increased operational costs." Why? Because they're hammering websites with traffic spikes that can reach up to ten or even twenty times normal levels within minutes. Moreover, AI crawlers are much more aggressive than standard crawlers. As the InMotionhosting web hosting company notes, they also tend to disregard crawl delays or bandwidth-saving guidelines and extract full page text, and sometimes attempt to follow dynamic links or scripts. The result? If you're using a shared server for your website, as many small businesses do, even if your site isn't being shaken down for content, other sites on the same hardware with the same Internet pipe may be getting hit. This means your site's performance drops through the floor even if an AI crawler isn't raiding your website... AI crawlers don't direct users back to the original sources. They kick our sites around, return nothing, and we're left trying to decide how we're to make a living in the AI-driven web world. Yes, of course, we can try to fend them off with logins, paywalls, CAPTCHA challenges, and sophisticated anti-bot technologies. You know one thing AI is good at? It's getting around those walls. As for robots.txt files, the old-school way of blocking crawlers? Many — most? — AI crawlers simply ignore them... There are efforts afoot to supplement robots.txt with llms.txt files. This is a proposed standard to provide LLM-friendly content that LLMs can access without compromising the site's performance. Not everyone is thrilled with this approach, though, and it may yet come to nothing. In the meantime, to combat excessive crawling, some infrastructure providers, such as Cloudflare, now offer default bot-blocking services to block AI crawlers and provide mechanisms to deter AI companies from accessing their data. ' -- source: https://tech.slashdot.org/story/25/08/31/1820249 Cheers, Peter

1 0

WLUG Wiki -- “Service Unavailable”?
by Lawrence D'Oliveiro 30 Aug '25

30 Aug '25

I was checking a link on the WLUG Wiki <https://wiki.wlug.org.nz/>, but it is currently returning “503 Service Unavailable”. I hope this is temporary ...

2 4

Bottles Project Receives EU-Backed Grant
by Peter Reutemann 18 Aug '25

18 Aug '25

'Over the past year, the Bottles project, a popular compatibility layer for running Windows software on Linux, has hit a bit of a rough patch, with financial struggles putting a serious damper on its development progress. It got to a point where, as we informed you back in early July, lead developer Mirko Brombin shared an honest update about the project’s tough financial situation and called on the open source community for support. Thankfully, that call didn’t go unheard—and now, it looks like those financial worries are behind them. The project secured a grant from the NLnet Foundation through its 2025 Commons Fund. This funding is a big deal, as it’ll help speed up development on Bottles Next, the project’s ambitious effort to deliver a sleeker, more modern experience for handling Windows apps in Linux environments.' -- source: https://linuxiac.com/bottles-project-receives-eu-backed-grant/ I had never heard of this project before today. Here is some more background: https://docs.usebottles.com/faq/why-bottles Cheers, Peter

1 0

Useful Commands Beginning With “ls”
by Lawrence D'Oliveiro 14 Aug '25

14 Aug '25

The “ls” command for listing directory contents has inspired a whole range of commands for listing other kinds of information. Here are a few I found in a quick search on my system: lsblk -- show block devices (disks, SSDs, SD cards, USB sticks etc). lscpu -- show information about your CPUs lsfd -- show information about open files (successor to lsof) lshw -- show detailed hardware information lsipc -- show state of Inter-Process Communication structures (message queues, shared memory, semaphores) lsirq -- show information about kernel hardware interrupt handling lslocks -- show information about filesystem locks lslogins -- show information about user IDs associated with processes on the system. This is not just regular user logins, but also user IDs allocated to system processes. lsmem -- show information about physical memory known to the kernel lsmod -- show state of loaded kernel modules lspci -- show information about PCI buses and connected devices. This will commonly include a lot of motherboard hardware, not just that in expansion slots. lsusb -- show information about connected USB devices. I think all these are from the util-linux package, which should be installed by default. Oh, and a bonus one that has nothing to do with system info as such: lsar -- handy tool for displaying the contents of various formats of archives, such as .zip, .tar and .7z files. Part of the “unar” package. Relevant manual pages (if these are not on your system): https://manpages.debian.org/lsblk(8) https://manpages.debian.org/lscpu(1) https://manpages.debian.org/lsfd(1) https://manpages.debian.org/lshw(1) https://manpages.debian.org/lsipc(1) https://manpages.debian.org/lsirq(1) https://manpages.debian.org/lslocks(8) https://manpages.debian.org/lslogins(1) https://manpages.debian.org/lsmem(1) https://manpages.debian.org/lsmod(8) https://manpages.debian.org/lspci(8) https://manpages.debian.org/lsusb(8) https://manpages.debian.org/lsar(1)

1 0

Bye-Bye Dialup USA
by Lawrence D'Oliveiro 10 Aug '25

10 Aug '25

US-based ISP America On-Line (AOL) will finally turn off its dialup Internet service at the end of September <https://www.tomshardware.com/service-providers/network-providers/aol-will-e…>, ending 34 years of operation. AOL was the biggest ISP in the US back in dialup days, but somehow never made a successful transition to broadband service like everyone else. Another peculiarity was that, unlike ISPs everywhere else in the world who offered their dialup service via standard protocols like SLIP (early days) or PPP (the most common later), AOL always stuck to its own proprietary protocol. The company was also infamous for loosing vast numbers of its sign-up CDs on the world via magazines, free mailouts and so on, in an effort to drum up business. There were endless jokes about what to do with this deluge of unwanted AOL CDs.

1 0

Blender 4.5 LTS Released
by Peter Reutemann 05 Aug '25

05 Aug '25

'NERDS.xyz reports: Blender 4.5 has arrived and it's a long-term support release. That means users get two full years of updates and bug fixes, making it a smart choice for anyone looking for stability in serious projects. Whether you're a solo artist or part of a studio pipeline, this version is built to last. Here's a list of key features and changes in this release: - Vulkan backend replaces OpenGL (faster, smoother UI) - Adaptive subdivision up to 14x faster with multithreading - New Geometry Nodes: Camera Info, Instance Bounds - GPU-accelerated compositor nodes with standardized inputs - New Boolean solver: Manifold (cleaner, faster mesh operations) - UV maps visible in Object Mode + improved selection behavior - Grease Pencil render pass and Geometry Nodes integration - Improved file import support: PLY, OBJ, STL, CSV, VDB - Deprecations: Collada, Big Endian, legacy .blend, Intel Mac support - Cycles OptiX now requires NVIDIA driver v535+ - New shader variants for add-on developers (POLYLINES_*, POINT_*) ~500 bug fixes across all major systems' -- source: https://tech.slashdot.org/story/25/07/15/1912208 Cheers, Peter

2 1

Microsoft Is Abandoning Windows 11 SE
by Lawrence D'Oliveiro 04 Aug '25

04 Aug '25

Seems like Microsoft’s attempt to fight back against Google’s Chromebooks in the education market has been a failure, and it is giving up <https://www.tomshardware.com/software/windows/microsoft-ending-windows-11-s…>. Put the blame on trying to squeeze the Windows quart into the pint-pot of low-cost, purpose-built educational machines: Microsoft hasn't given a formal reason as to why they are ending Windows 11 SE. Presumably, one of the biggest challenges was that despite being a version meant for classrooms, it was fundamentally based on the full version of Windows 11. This meant it struggled to run well on the low-cost devices that schools often buy, a space where Chrome OS thrives. Google’s offering continues to be a truly lightweight cloud-based operating system designed from the ground up for this purpose. Does this mean we stop including Chromebooks as part of the “desktop” market that Windows is supposedly king of?

1 0

Old Mailing List Archive Archive
by Lawrence D'Oliveiro 01 Aug '25

01 Aug '25

With the demise of the old Waikato-Uni-based WLUG list server went its archive of list messages. Luckily, that archive was itself archived at archive.org. Here <https://web.archive.org/web/20240711100644/https://list.waikato.ac.nz/archi…> is the last snapshot I could find of the list archive before it disappeared.

1 0

KDE Linux
by Peter Reutemann 01 Aug '25

01 Aug '25

Looks like it is finally available (initiative got announced in 2024): 'Linux has been used for decades powering the world’s supercomputers, data centers, web servers, and most smartphones. Now KDE makes it available for you too! Introducing KDE Linux, a free operating system that can turbocharge new computers and breathe new life into old ones.' -- source: https://kde.org/linux/ -- mastodon: https://mastodon.social/@nixCraft/114949288867476013 Cheers, Peter

2 1

Weta Digital’s Technology Division Sold To US Firm
by Lawrence D'Oliveiro 01 Aug '25

01 Aug '25

Let me see if I understand this article <https://www.nzherald.co.nz/business/sir-peter-jackson-to-sell-weta-digital-…> correctly: the part of Weta Digital that used to develop the in-house 3D graphics/effects tools that they use in all those movies is being sold to US-based Unity Software. The part that is left will be called “WetaFX”, and will become a customer of Unity for the use of those tools, no longer doing its own development in-house. By the way, I’ve been looking at the Unity site <https://unity.com/>, and it looks like all their products are rentware (subscription-based).

1 2