systemd supports a number of integration features that allow VMMs certain access to the inner state of VM guests for provisioning, synchronization and interaction, and many of them are little known, even though very very useful. In this talk I'd like to shed some light on many such integration points, such as SMBIOS type 11 based system credential provisioning; state propagation/readiness notification via AF_VSOCK; SSH support via AF_VSOCK, and so on.
Modern confidential computing technologies like AMD SEV-SNP and Intel TDX provide a reliable way to isolate guest workload and data in use from the virtualization or cloud infrastructure. Protecting data at rest is, however, not something you get ‘by default’. The task is particularly challenging for traditional operating systems where users expect to get full read/write experience.
The good news is that Linux OS already offers a number of great technologies which can be combined to achieve the goal: dm-verity and dm-integrity, LUKS, discoverable disk images and others. Doing it all right, however, is left as an “exercise to the reader”. In particular, the proposed solution must allow for meaningful remote attestation at any time in the lifetime of the guest.
The talk will focus on the recent developments in various upstream projects like systemd and dracut which are focused on making full disk encryption consumable by confidential computing guests running in a cloud.
It has been several years since the last rust-vmm update at FOSDEM, but the community has continued to grow. Our goal remains the same: to provide reusable Rust crates that make it easier and faster to build virtualization solutions.
This talk will present the main progress and achievements from the past few years. It reviews how rust-vmm crates integrate into a variety of projects such as Firecracker, Cloud Hypervisor, Dragonball, and libkrun. We will discuss the ongoing efforts to consolidate all crates into a single monorepo and how we expect this to simplify development and releases. The talk will also cover recent work supporting new architectures like RISC-V and additional operating systems. Finally, we will review the support for virtio and vhost-user devices that can be used by any VMM.
QEMU 10.2 will introduce MSHV as a new accelerator option for Linux hosts.
MSHV is a kernel driver maintained by Microsoft's Linux System Group that aims to expose HyperV capabilities to users in various virtualization topologies: on bare metal, in nested virtualization and most recently via a new model called "Direct Virtualization".
Direct virtualization will allow owners of an L1 VM to commit parts of their assigned resources (CPU, RAM, Peripherals) to virtual L2 guests, that are technically L1 siblings. Users can take advantage of the hypervisor's isolation boundaries without the performance and functional limitations of a nested guest. Untrusted code can be sandboxed with near-native performance and access to GPUs or NVMe controllers.
Adding support for MSHV acceleration to QEMU aims to broaden the reach of this technology to a Linux audience. The talk will cover the current state of the implementation, challenges that remain and future plans for both MSHV and QEMU.
VIRTIO is the open standard for virtual I/O, supported by a wide range of hypervisors and operating systems. Typically, device emulation is performed directly inside the Virtual Machine Monitor (VMM), like QEMU. However, modern virtualization stacks support multiple implementation models: keeping the device in the VMM, moving it to the kernel (vhost), offloading it to an external user-space process (vhost-user), or offloading it directly to the hardware (vDPA).
Each approach comes with specific trade-offs. Emulating in the VMM is straightforward but can be a bottleneck. In-kernel emulation offers high performance but increases the attack surface of the host system. External processes provide excellent isolation and flexibility, but introduce complexity. Finally, vDPA (vhost Data Path Acceleration) enables wire-speed performance with standard VIRTIO drivers, but introduces hardware dependencies.
So, how do we decide which approach is best for a specific use case?
In this talk, we will explore all four methods for emulating VIRTIO devices. We will analyze the architectural differences, discuss the pros and cons regarding performance and security, and provide guidance on how to choose the right architecture for your use case.
This talk shows how a Raspberry Pi can run a complete open-source cloud using OpenNebula. With MiniONE handling the installation and KVM doing the virtualization, a Raspberry Pi becomes a small but fully functional cloud node capable of running VMs, containers, lightweight Kubernetes clusters and edge services. The goal is simple: demonstrate that homelab users can build a full cloud stack with compute, networking, storage and orchestration on affordable hardware using only open-source tools. A short demo will show a VM launching on a Pi-based OpenNebula cloud, highlighting how the platform scales down to tiny devices while keeping the same clean and unified experience found on larger deployments.
To address the challenge of providing seamless Layer 2 connectivity and mobility for KubeVirt virtualized applications distributed across multiple clusters (for reasons like disaster recovery, scaling, or hybrid cloud), we integrated OpenPERouter, an open-source project that provides EVPN-based VXLAN overlays, solving the critical need for distributed L2 networking.
OpenPERouter's declarative APIs and dynamic BGP-EVPN control plane enable L2 networks to stretch transparently between clusters, maintaining VM MAC/IP consistency during migrations and disaster recovery. This architecture facilitates deterministic cross-cluster live migrations, better supports legacy workloads needing broadcast/multicast, and enables migrating workloads into the KubeVirt cluster while preserving original networking using open components. Routing domains are also supported for traffic segregation and to provide direct routed ingress to VMs, eliminating the need for Kubernetes services to expose ports.
Attendees will gain the practical knowledge to design and implement resilient, operationally safe, EVPN-based overlays with OpenPERouter, receiving actionable design patterns and configuration examples.
"GPU clouds" for AI application are the hot topic at the moment, but often these either end up being just big traditional HPC-style cluster deployments instead of actual cloud infrastructure or are built in secrecy by hyperscalers.
In this talk, we'll explore what makes a "GPU cloud" an actual cloud, how requirements differ from traditional cloud infrastructure, and most importantly, how you can build your own using open source technology - all the way from hardware selection (do you really need to buy the six-figures boxes?) over firmware (OpenBMC), networking (SONiC, VPP), storage (Ceph, SPDK), orchestration (K8s, but not the way you think), OS deployment (mkosi, UEFI HTTP netboot), virtualization (QEMU, vhost-user), performance tuning (NUMA, RDMA) to various managed services (load balancing, API gateways, Slurm etc.)
What if your container image were a few megabytes instead of hundreds of megabytes? WebAssembly (WASM) offers a radically lighter approach to running workloads on Kubernetes — right alongside your existing containers. In this talk, we'll dive deep into how WASM modules using the WebAssembly System Interface (WASI) integrate into Kubernetes through containerd shims like runwasi. Using a Rust example, we'll demonstrate the dramatic reduction in image size and startup time compared to traditional containers. We'll explore the current state of WebAssembly in the cloud-native ecosystem: what's production-ready today, and where you should wait before adopting. Beyond the basics, we'll look at real-world Cloud-Native Compute Foundation (CNCF) projects already running WASM in production and discuss the two areas where WebAssembly shines: plugin architectures that benefit from small, secure, sandboxed extensibility, and event-driven systems that can quickly scale from zero. Whether you're optimizing for resource efficiency or exploring new isolation patterns, this session provides insights into WebAssembly on Kubernetes and serves as a great starting point.
With KubeVirt, Virtual Machines become first-class citizens in Kubernetes, allowing VMs and containers to be managed through a unified control plane. As organizations evolve their infrastructure, many early adopters face the challenge of upgrading systems without disrupting essential services. To support this, KubeVirt provides powerful mobility capabilities for both compute and storage.
KubeVirt enables non-shared storage live migration through QEMU's block migration feature, which is orchestrated by libvirt and KubeVirt components. The core process involves copying the VM's disk data and memory state to the destination node while the VMI remains running.
Cross-Cluster Live Migration (CCLM) extends this capability across Kubernetes clusters, allowing a running VM to be moved seamlessly between clusters. This enhances flexibility and resilience in multi-cluster environments and is especially useful for load balancing, maintenance operations, and infrastructure consolidation – all without interrupting critical workloads. CCLM requires L2 network connectivity between clusters and compatible CPU architectures.
Storage Live Migration (SLM) allows you to move the VM’s disk data from one storage backend to another while the VM remains running. This is particularly valuable when rebalancing storage usage, retiring legacy systems, or adopting new storage classes – all without disrupting the applications inside the VM. SLM requires at least two compatible nodes.
Both CCLM and SLM work with any storage backend, including those using the ReadWriteOnce access mode.
After the session, you’ll be ready to migrate your running VMs – across clusters and across storage – with confidence, like a seasoned scheduler placing pods exactly where they need to be.
Lima (Linux Machines) is a command line tool to launch a local Linux virtual machine, with the primary focus on running containers on a laptop.
While Lima was originally made for promoting containers (particularly containerd) to Mac users, it has been known to be useful for a variety of other use cases as well. One of the most edgy use cases is to run an AI coding agent inside a VM, in order to isolate the agent from direct access to host files and commands. This setup ensures that even if an AI agent is deceived by malicious instructions searched from the Internet (e.g., fake package installations), any potential damage is confined within the VM, or limited to files specified to be mounted from the host.
This talk introduces the updates in Lima v2.0 (November 2025) that facilitates using Lima with AI: - Plugin infrastructure - GPU acceleration - MCP server - CLI improvements
Web site: https://lima-vm.io GitHub: https://github.com/lima-vm/lima
In this session, we will present a new extension to Prowler, the widely adopted open-source cloud security auditing tool, adding native support for the OpenNebula cloud management platform.
Our contribution delivers a modular, non-intrusive, and scalable auditing framework that integrates essential services and a growing catalogue of security checks aligned with established reference standards. This extension enables operators to detect misconfigurations and vulnerabilities more effectively, strengthening the overall security posture of OpenNebula deployments.
We will walk through the design and implementation of the tool, share validation results from real test scenarios, and outline how this effort helps democratize cloud security within the open-source ecosystem. Finally, we will discuss opportunities for community-driven collaboration to expand and evolve this new security auditing capability.
KubeVirt allows running VMs and containers on Kubernetes, but traditional Kubernetes networking - which uses NAT (Network Address Translation) to expose workloads outside the cluster - can still lead to complex, opaque, and brittle setups that prevent direct integration and reachability.
This presentation introduces a BGP-based solution to simplify KubeVirt networking. Kubernetes nodes dynamically exchange routes with the provider network, exposing workloads via their actual IPs, eliminating NAT and manual configurations.
This BGP approach simplifies network design, speeds up troubleshooting, and ensures consistent connectivity for virtualized workloads.
Attendees will learn practical, standard networking principles to simplify real-world Kubernetes environments and gain immediate, actionable insights to improve platform connectivity.
Platform engineering teams tackle complex, multi domain challenges, balancing governance and iterating quickly to enable developers. In this session we’ll detail how SUSE IT uses Kubewarden as a policy controller across both RKE2 and SUSE Virtualization environments. We’ll show how enforcing organizational policies with Kubewarden automatically integrates compliance and operational excellence into the core of the platform. We’ll discuss practical examples, e.g., how to restrict usage of resources, GPUs or VLANs to specific customers while providing the platform to a wider audience.
https://www.kubewarden.io/
https://www.rancher.com/
https://docs.rke2.io/
In this talk, we will present the current state of remote VM access in KubeVirt [0] and the challenges associated with it. We will discuss in-guest approaches such as running an RDP server on Windows or Linux, as well as host-side mechanisms like QEMU’s built-in VNC server exposed through KubeVirt’s virt-api. Finally, we will introduce a new proposal that leverages QEMU’s display-over-D-Bus interface [1], a direction that could enable additional vendors to build their own remote-display integrations.
[0] https://kubevirt.io/ [1] https://www.qemu.org/docs/master/interop/dbus-display.html
Serving AI models on a single GPU for multi tenant workload sounds challenging till you partition a GPU correctly.
This talk is a deep technical exploration of running AI inference workloads on modern GPUs across using Multi-Instance GPU (MIG) isolation.
We'll explore:
Whether you're building a multi-tenant inference platform, optimizing GPU utilization for your team, or exploring how to serve AI models cost-effectively, this talk provides practical configurations for your AI workloads.
Virtualization has transformed low-level debugging, system analysis, and malware research. By placing a thin hypervisor beneath the OS, developers gain a vantage point the OS cannot access. This blue-pill approach enables fine-grained control over CPU state, memory, interrupts, and hardware events without relying on OS components, supporting transparent breakpoints, VM-exit triggers, memory shadowing, and instruction tracing with minimal interference.
We present HyperDbg, an open-source hypervisor-based debugger. Leveraging the former characteristics, unlike kernel debuggers that depend on drivers, APIs, or software breakpoints, HyperDbg operates entirely below the OS, combining virtualization-based introspection with interactive debugging. It inspects memory, CPU execution, and traps events without OS cooperation, bypassing anti-debugging and anti-analysis techniques.
Using modern virtualization extensions like Mode Based Execution Control (MBEC) on top of Second Level Address Translation (SLAT), HyperDbg enforces breakpoints and traps through hardware transitions, independent of OS APIs or exceptions. This allows stealthy, artifact-free binary analysis, providing a powerful platform for reverse engineering and research. In its first iteration, HyperDbg introduced a hypervisor-powered kernel debugger. With the recent release of v0.15, HyperDbg enables cross-boundary debugging from kernel-mode into user-mode. For this talk, we will add special focus on how we implemented cross-boundary debugging, and how it enables users to intercept user-mode process execution using virtualization techniques.
Resources:
HyperDbg repository: https://github.com/HyperDbg/HyperDbg/
Documentation: https://docs.hyperdbg.org/
Kernel-mode debugger design: https://research.hyperdbg.org/debugger/kernel-debugger-design/
Research paper: https://dl.acm.org/doi/abs/10.1145/3548606.3560649
A number of industrial applications now demand hard real-time scheduling capabilities from the kernel of a Linux-based operating system, but scheduling measurements from the system itself cannot be completely trusted as they are referenced to the same clock as the kernel-under-test. Yet, if the system can output signals to hardware as it runs, their timing can be analysed by an external microcontroller, and a second "external" measurement obtained to compare with the system's own report.
Codethink wrote embedded Rust firmware to run on a Raspberry Pi Pico which analyses the timings of characters received on a UART serial port. On the other end of the serial port is our "Rusty Worker": a Rust program on a Linux system which uses the sched_setattr syscall to request that Linux schedules it with specified parameters, and then measures its own scheduling period and runtime. The single-threaded and interrupt-based architecture of this firmware allowed accurate external measurements of the Rusty Worker’s scheduling parameters at microsecond precision, and meant it was easy to extend to monitor the "petting" behaviour of a watchdog.
Rust was a natural choice for the verification firmware and Rusty Worker. Higher level or interpreted languages would increase non-determinism and reduce our confidence in the accuracy of the collected timing data, whereas C or C++ programs risk undefined behaviour unacceptable in a safety-related context. Yet, Rust really came into its own with the relative simplicity of the cargo toolchain for embedded targets, so reproducibly building the firmware with a self-built toolchain (and without access to the internet) was just as straightforward as building the Rusty Worker for a Linux target.
Having equipped our suite of bare-metal CI runners with KiCad-designed custom PCBs that feature a Raspberry Pi Pico and Debug Probe, we are able to run “soak” tests to collect thousands of self- and externally-measured deadline scheduling parameters for each iteration of our Codethink Trustable Reproducible Linux (CTRL OS), as soon as engineers push a new commit. We then use the open source Eclipse Trustable Software Framework (TSF) to facilitate automatic “real time” aggregation and statistical analysis of the external scheduling measurements for each commit, clearly communicating the results and trends to senior stakeholders and junior engineers alike.
TSF challenges us both to robustly, systematically, and continuously evidence our claims about Linux’s capabilities as a real-time operating system, and to scrutinise the software used for testing as strictly as the software under test. We are excited to share how we work towards these goals with external scheduling measurements and embedded Rust.
OS kernel development is often connected with time consuming testing process and non-trivial debug technics. Although emulators like QEMU and Bochs ease this work significantly, nothing can compare with convenience of userspace developer environment. Moving parts of the kernel to the userspace binary is not straightforward, especially if the kernel has almost no compatibility with POSIX and is written entirely in assembly. Still, sometimes it is doable. The talk shares experience, architecture and design decisions of compiling VFS, block, and some other subsystems of KolibriOS as Linux® interactive shell program and a FUSE filesystem. Implemented unit testing framework and coverage collection tool for assembly (flat assembler) programs are also discussed.
Talk about a project that implements a hardware-in-the-loop testing framework for validating Linux distributions on specific development boards. The system uses a universal testing harness that automatically detects target hardware platforms and adapts generic testing scripts to board-specific configurations with Claude AI help. Platform adaptation is achieved through specific configuration files that define board-specific parameters, enabling the same testing codebase to validate different hardware capabilities. The GitHub Actions CI/CD integration provides automated testing across multiple platforms, with matrix-based execution that flashes appropriate images and runs comprehensive validation including hardware-specific feature testing.
The ci-multiplatform project is a generic, OCI-based multi-architecture CI system designed to make cross-platform testing practical for open-source projects using GitLab CI. Originally created while enabling RISC-V support for Pixman (https://gitlab.freedesktop.org/pixman/pixman), it has since grown into an independent project under the RISE (RISC-V Software Ecosystem) umbrella: https://gitlab.com/riseproject/CI/ci-multiplatform, with a mirror on freedesktop.org: https://gitlab.freedesktop.org/pixman/ci-multiplatform
The project provides multi-arch layered OCI images (Base GNU, LLVM, Meson) based on Debian, GitLab Component-style templates, and fully automated downstream test pipelines for the included examples. It supports creating customized OCI images, building and testing across 16 Linux and Windows targets – including x86, ARM, RISC-V, MIPS, and PowerPC – using unprivileged GitLab runners with QEMU user-mode emulation. Architecture-specific options (e.g., RISC-V VLEN configuration) allow developers to exercise multiple virtual hardware profiles without any physical hardware, all within a convenient job-matrix workflow.
The talk covers how the system is engineered, tested, and validated across multiple GitLab instances, and what happens when unprivileged runners, QEMU quirks, toolchain differences, and architecture-specific behaviours all converge in a single pipeline. I will show how projects can adopt ci-multiplatform with minimal effort and turn multi-arch CI from a maintenance burden into a routine part of upstream development.
Testing is central to modern software quality, yet many widely used Fortran codebases still lack automated tests. Existing tests are often limited to coarse end-to-end regression checks that provide only partial confidence. With the growth of open-source Fortran tools, we can now bring unit testing and continuous validation to legacy and modern Fortran projects alike.
This talk surveys the current landscape of Fortran testing frameworks before focusing on three I have evaluated in practice — pFUnit, test-drive and veggies — and explaining why pFUnit is often the most robust choice. I will discuss its JUnit-inspired design, use of the preprocessor, and the compiler idiosyncrasies that can still make adoption challenging. I will examine the hurdles that make testing Fortran hard: global state, oversized subroutines, legacy dependencies, and compiler-specific behaviour.
I will then present community-oriented efforts to improve testing practices in Fortran, including development of an open-source Carpentries-style training course on testing in Fortran, with plans to expand into a broader introduction to sustainable Fortran development using open-source linting and documentation tools such as Fortitude and Ford.
Attendees will gain practical guidance for introducing effective testing into existing Fortran codebases, and insight into current efforts towards modern workflows that support reproducibility and continuous delivery.
ESPHome is a versatile framework to create custom firmware for various microcontrollers. In this talks we will look at how to automatically test the latest ESPHome firmware on an ESP32.
As ESPHome devices are used to interact with the real world, we will also look at how to test that the LUX sensors is able to detect light variations.
In order to test the ESP32 device, we are going to use lava on the command line, directly inside a gitlab runner.
The CI/CD server Jenkins provides powerful build-quality visualizations through plugins such as Warnings, Coverage, and Git Forensics. These plugins aggregate and visualize data from static analysis tools, coverage reports, software metrics, and Git history, enabling teams to track quality trends across builds. We have now brought this functionality to other widely used CI/CD platforms, including GitHub Actions and GitLab CI.
This talk presents portable, UI-independent implementations of these capabilities for GitHub Actions and GitLab CI: the quality monitor GitHub Action and the GitLab autograding-action. Both tools share a common architecture and codebase with the Jenkins plugins. They automatically analyze pull requests and branch pipelines, generate structured comments and concise Markdown summaries, and enforce configurable quality gates. The solutions are language-agnostic and integrate seamlessly with more than 150 static analysis, coverage, test, and metrics report formats—including Checkstyle, SpotBugs, SARIF, JUnit, JaCoCo, GoCov, and GCC. Additionally, both tools provide an autograding mode for educational use, enabling instructors to assess student submissions through a flexible, configurable point-based scoring system.
CI/CD with Gerrit, AI-Enhanced Review, and Hardware-in-the-Loop Testing in Jenkins Pipelines This presentation will explore advanced Continuous Integration (CI) strategies essential for open-source embedded systems development, moving beyond standard software testing to encompass physical hardware validation. We will begin by establishing the necessity of integrating rigorous unit and integration testing directly into the development workflow, demonstrating how to effectively define these steps within Jenkins Declarative Pipelines (DSL). The core of our approach involves deep integration with Gerrit Code Review, ensuring that tests and static analysis are triggered automatically upon every patch set creation, providing fast feedback to developers. A significant portion of the talk will focus on achieving true end-to-end validation through Hardware-in-the-Loop (HIL) testing. We will detail the implementation of Labgrid, an open-source tool used to manage and control remote hardware resources (such as embedded boards and IoT devices). This integration allows the Jenkins pipeline to reserve, provision, and execute automated, system-level tests directly on physical target devices before firmware changes are merged. Furthermore, we will introduce two critical elements for pipeline stability and code quality. Firstly, we will demonstrate the utility of an AI-Powered Error Explanation component (e.g., via the Explain Error Plugin). This feature leverages large language models to analyze complex Jenkins log files and pipeline failures, translating cryptic errors into human-readable insights and suggested fixes, which dramatically cuts down debugging time. Secondly, we will showcase the Warnings Next Generation (Warning-NG) Plugin, which serves as a central aggregator, collecting and visualizing issues and potential vulnerabilities reported by various static analysis tools, thereby enforcing strict, quantifiable quality gates within the CI process. Attendees will gain practical, cutting-edge insights into implementing a robust, AI and hardware-enhanced CI/CD workflow suitable for modern open-source projects.
The problem with the current most commonly accepted way of running code reviews using Pull Requests is that they have the nasty habit of blocking the flow of delivery. They introduce a cost of delay. Any delay reduces feedback. Consequently, it drives down quality.
The usual way to achieve fast, efficient and effective Continuous Code Reviews without disrupting the flow of delivery is through Pair Programming or Team Programming. However, for various valid reasons, these can be a cultural stretch for many teams and organisations.
In 2012, a novice team practising trunk-based development set in place a fairly uncommon but efficient alternative to implementing continuous code reviews on mainline without ever blocking the flow of delivery.
This team went from a bunch of rag-tags to becoming a reference team within the organisation, with auditors falling to the floor due to the high quality the team delivered. Target audience: software engineers, test engineers, infrastructure engineers, team leads, engineering managers, CTOs
With everything changing in tech at a frenetic pace, the emphasis on developer productivity has overshadowed the true essence of developer experience (DevEx). While frameworks like SPACE, getDX, and DORA metrics provide valuable insights, they often miss the mark on capturing developers' real, day-to-day experiences using tools and services, instead focusing strictly on the bottom line for the company. Meanwhile, developers and practitioners are job-hopping more than ever. This talk will explore the origins and evolution of "developer experience," dissect popular frameworks, and advocate for a more balanced approach that values the practitioner's perspective. At the end we will set a path towards integrating top-down metrics with bottom-up feedback, ensuring an approach to developer experience that fosters innovation and satisfaction.
Even with robust CI/CD, production rollouts can hit unexpected snags. While in Kubernetes Argo Rollouts excels at Progressive Delivery and automated rollbacks to mitigate deployment issues, what if we could go a step further?
This session explores how to elevate your release process by integrating Agentic AI and asynchronous coding agents, with Argo Rollouts canary deployments. We'll demonstrate how an intelligent agent can automatically analyze a rollout failure, pinpointing the root cause. Beyond diagnosis, these agents can take proactive steps on your behalf, suggesting and even implementing code fixes as new pull requests, which can be redeployed automatically after PR review. This approach moves us closer to truly self-healing deployments.
Join us to learn how to combine the power of Kubernetes and Argo Rollouts with the autonomous capabilities of Agentic AI, achieving a release experience that is not only seamless but also resilient.
We love ArgoCD, but it creates a classic "map vs. territory" problem. We treat Git as our "map", our single source of truth. But the cluster is the "territory", and it's often more complex than the map shows. This becomes a crisis with the 3 AM hotfix: an SRE fixes production, changing the territory. ArgoCD, loyal to the map, sees this as drift and helpfully overwrites the fix, re-breaking the cluster. The problem is that Git isn't our Truth, it's our Intention. This talk introduces a pragmatic solution: Cluster-Scoped Snapshotting. We’ll show a simple pattern that dumps the entire live cluster state (the "territory") into its own "reality" Git repo. To automate this, we wrote a small open-source tool called Kalco, but the pattern is the real takeaway. This "reality" repo gives us a powerful "pre-flight diff" in our CI pipeline, comparing our "intention" (the app repo) against the "truth" (the snapshot repo). This simple check lets us bootstrap existing clusters, create a complete audit log, and stop our pipeline before it merges a change that conflicts with a critical live fix.
We've all heard that we should test our software, but what happens when we don't? Sometimes, it leads to strange and unexplainable events.
Is 'testing more' always the right solution? What do these bugs reveal about software and its failures? And how can we use these lessons to build more resilient systems?
Let's explore together the most bizarre software bugs in history!
Forgotten files, incomplete system info, back and forward emails... Bug reporting can be messy process and sometimes wastes a lot of time on both developer and user side. However, a lot of it can be normalized and automated. In this talk we introduce CLI tool DebugPack that helps us to simplify the bug reporting process for our team and ensures that developers always have the necessary information for successful bug hunting.
https://gitlab.nic.cz/labs/bird-group/debugpack
Software backdoors aren’t a myth—they’re a recurring nightmare. Time and again, we’ve watched malicious code slip into open-source ecosystems. The notorious xz compromise grabbed headlines, but it wasn’t the first act in this drama. Earlier breaches included the PHP incident in 2021, as well as vulnerabilities in vsFTPd (CVE-2011-2523) and ProFTPD (CVE-2010-20103). And here’s the unsettling truth: these examples likely just scratch the surface. Why does it matter? Because a single backdoor in a widely used project turns into a hacker’s dream buffet—millions of machines served up for exploitation.
Tracking down and eliminating backdoors isn’t a quick win—it’s like diving headfirst into sprawling code jungles. Sounds epic? In reality, even for a veteran armed with reverse-engineering gear, it’s a grueling slog. So grueling that most people simply don’t bother. The good news? New tools such as ROSA (https://github.com/binsec/rosa) prove that large-scale backdoor detection can be automated—at least to a significant extent. Here’s the twist: traditional fuzzers like AFL++ (https://github.com/AFLplusplus/AFLplusplus) test programs with endless input variations to trigger crashes. It’s brute force, but brilliant for uncovering memory-safety flaws. Backdoors, however, play by different rules—they don’t crash; they lurk behind hidden triggers and perfectly valid behaviors. ROSA changes the game by training fuzzers to tell “normal” execution apart from “backdoored” behavior.
But there’s a catch: ROSA’s current use case is after-the-fact analysis, helping security experts vet full software releases (including binaries). Following the shift-left paradigm, our goal is to bring this detection magic into the CI pipeline—so we can stop backdoors before they ever land. Sounds great, but reality bites: ROSA produces false alarms and can require a significant test budget to find backdoors, which are a nightmare in CI. In this talk, we would like explore the methodological and technical upgrades needed to build a ROSA-based backdoor detection prototype that thrives in CI environments. Think reduced resources, and minimal noise—all within the tight resource windows CI jobs demand.
Modern automated testing environments generate vast amounts of test results, making failure analysis increasingly complex as both the number of tests and failures grow. This presentation introduces an AI-driven approach to failure aggregation, leveraging text embeddings and semantic similarity to efficiently group and analyze unique failures. The workflow integrates open-source, pre-trained models for text embedding (such as Sentence Transformers) and vector similarity search using PostgreSQL with pgvector, enabling scalable and low-barrier adoption.
In 2024, I left my job to build CDviz full-time—an open source platform for CI/CD observability using CDEvents, an emerging specification with minimal ecosystem adoption. This talk shares lessons from building production tooling on early-stage standards.
You'll see: - Why I chose to build on CDEvents despite limited adoption - Technical challenges: converting diverse tool events into a unified format - Architecture decisions: PostgreSQL/TimescaleDB for storage, Grafana for visualization - Live demo: CDviz tracking deployments with real metrics - What worked, what didn't, and lessons for building on emerging specs
This is a builder's story about creating interoperability tooling before the ecosystem is ready—and why standardization matters even when adoption is slow.
Testing VoIP infrastructure at scale is far from straightforward. These systems route calls and enrich the caller experience, with features such as playing prompts, interactive menus, and caller queues. With so many features and interactions, manually testing every scenario is impossible, so test automation is essential.
As a software tester for a real-world VoIP infrastructure, I built an automated framework using the open-source SIPSorcery library (https://github.com/sipsorcery-org/sipsorcery) to create programmable softphones that simulate complex call interactions. The talk covers the most interesting challenges faced, such as verifying who can be heard and seen in audio and video calls, mimicking specific physical phones, and making timing-sensitive tests run reliably.
Attendees will take away insights into the challenges of large-scale VoIP testing and practical strategies for designing automated tests that are reliable, repeatable, and maintainable.
In this talk, we present formal verification, a technique for mathematically verifying code and ensuring it is safe for any possible inputs.
We explore in particular the theorem prover Rocq, and how we use it to model and verify production code at Formal Land. We show the two primary methods to create a code model, by testing or proving the equivalence with the implementation, and the main classes of properties that are interesting to formally verify on a program.