After 20 years, Real-Time Linux (PREEMPT_RT) is finally – finally – in the mainline kernel. Linus Torvalds blessed the code while he was at Open Source Summit Europe. […] The real-time Linux code is now baked into all Linux distros as of the forthcoming Linux 6.12 kernel. This means Linux will soon start appearing in more mission-critical devices and industrial hardware. But it took its sweet time getting here. An RTOS is a specialized operating system designed to handle time-critical tasks with precision and reliability. Unlike general-purpose operating systems like Windows or macOS, an RTOS is built to respond to events and process data within strict time constraints, often measured in milliseconds or microseconds. As Steven Rostedt, a prominent real-time Linux developer and Google engineer, put it, “Real-time is the fastest worst-case scenario.” He means that the essential characteristic of an RTOS is its deterministic behavior. An RTOS guarantees that critical tasks will be completed within specified deadlines. […]

So, why is Real-Time Linux only now completely blessed in the kernel? “We actually would not push something up unless we thought it was ready,” Rostedt explained. “Almost everything was usually rewritten at least three times before it went into mainline because we had such a high bar for what would go in.” In addition, the path to the mainline wasn’t just about technical challenges. Politics and perception also played a role. “In the beginning, we couldn’t even mention real-time,” Rostedt recalled. “Everyone said, ‘Oh, we don’t care about real-time.’” Another problem was money. For many years funding for real-time Linux was erratic. In 2015, the Linux Foundation established the Real-Time Linux (RTL) collaborative project to coordinate efforts around mainlining PREEMPT_RT.

The final hurdle for full integration was reworking the kernel’s print_k function, a critical debugging tool dating back to 1991. Torvalds was particularly protective of print_k --He wrote the original code and still uses it for debugging. However, print_k also puts a hard delay in a Linux program whenever it’s called. That kind of slowdown is unacceptable in real-time systems. Rostedt explained: “Print_k has a thousand hacks to handle a thousand different situations. Whenever we modified print_k to do something, it would break one of these cases. The thing about print_k that’s great about debugging is you can know exactly where you were when a process crashed. When I would be hammering the system really, really hard, and the latency was mostly around maybe 30 microseconds, and then suddenly it would jump to five milliseconds.” That delay was the print_k message. After much work, many heated discussions, and several rejected proposals, a compromise was reached earlier this year. Torvalds is happy, the real-time Linux developers are happy, print_K users are happy, and, at long last, real-time Linux is real.

  • nebeker@programming.dev
    link
    fedilink
    English
    arrow-up
    28
    ·
    10 hours ago

    This is super interesting. I’ll admit I wasn’t even aware of this effort. Even real-time usage of Windows relies on a parallel kernel.

    This sounds like it’ll create a lot of cool opportunities and reduce friction.

    • refalo@programming.dev
      link
      fedilink
      arrow-up
      7
      arrow-down
      2
      ·
      9 hours ago

      But the RT patches have been available for 20 years… not sure why the fact that it is mainlined would suddenly expand its popularity? It might be easier to get started sure, but people doing RT were already going to such troubles anyway.

      • RustyWizard@programming.dev
        link
        fedilink
        arrow-up
        6
        arrow-down
        1
        ·
        6 hours ago

        It’s like saying GPS was available for decades before, why would putting it in everyone’s phone expand its popularity.

        For myself, I’m hoping the nerds and hackers that otherwise found it not worth the effort will start creating tools to manage real time better and start building them into the applications they write. That way you don’t need to pay an arm and a leg to RedHawk for the privilege of dynamically isolating CPUs or have to reboot the computer to modify kernel arguments a la RedHat MRG.

        • refalo@programming.dev
          link
          fedilink
          arrow-up
          1
          arrow-down
          2
          ·
          5 hours ago

          I don’t think that comparison is fair because I explicitly said that people who wanted RT are already going out of their way to get things done. The average desktop user (putting GPS in every phone) won’t benefit from it (or RT) and it could likely make their experience even worse.

          • RustyWizard@programming.dev
            link
            fedilink
            arrow-up
            5
            ·
            4 hours ago

            People who wanted GPS were also already using it by buying TomToms or other GPS devices…

            The average desktop user already benefits from the changes the RT folks have slowly been getting into the baseline.

            • refalo@programming.dev
              link
              fedilink
              arrow-up
              1
              arrow-down
              1
              ·
              3 hours ago

              People who want RT are not everyday people though.

              How does an average desktop user benefit from any RT changes?

              • FizzyOrange@programming.dev
                link
                fedilink
                arrow-up
                3
                ·
                1 hour ago

                You seem to have the idea that there are “people who want RT” and they’ll overcome any inconvenience to get it, therefore making RT more convenient to use won’t increase use of it.

                Clearly nonsense, and I think the GPS analogy is a good one. My mum isn’t “a person who wants GPS” and she would never have bought a GPS device in the 00s, but she uses one now because it’s conveniently already available in her phone.

              • RustyWizard@programming.dev
                link
                fedilink
                English
                arrow-up
                3
                arrow-down
                1
                ·
                2 hours ago

                The most obvious answer is gaming. Hard 60/144Hz deadlines is RT. But there are lots of changes that got into the kernel from the RT group, starting with getting rid of the BKL, which helped everyone.