{"article":{"slug":"software-sandboxing-the-basics","title":"Software sandboxing: The basics","subtitle":null,"summary":"A deep primer on software sandboxing: threat models, OS mechanisms, and practical patterns for isolating untrusted code—foundational reading for secure systems and agent runtimes.","content_type":"tutorial","language":"en","canonical_url":"https://blog.emilua.org/2025/01/12/software-sandboxing-basics/","author":{"name":"Emilua","url":"https://blog.emilua.org","person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Emilua Blog","url":"https://blog.emilua.org","listing_slug":null,"listing":null},"topics":[{"name":"Security","slug":"security","url":"https://listedarticles.com/topics/security"},{"name":"Systems Programming","slug":"systems-programming","url":"https://listedarticles.com/topics/systems-programming"},{"name":"Programming","slug":"programming","url":"https://listedarticles.com/topics/programming"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":9084,"reading_minutes":39,"published_at":"2025-01-12T00:00:00.000Z","added_at":"2026-09-21T03:09:30.817Z","updated_at":"2026-09-21T03:09:30.817Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/software-sandboxing-the-basics","markdown_url":"https://listedarticles.com/articles/software-sandboxing-the-basics.md","example":false,"citation":"Emilua, Emilua Blog. \"Software sandboxing: The basics.\" 12 Jan 2025. https://blog.emilua.org/2025/01/12/software-sandboxing-basics/ (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://blog.emilua.org/2025/01/12/software-sandboxing-basics/"},"body_markdown":"# Software sandboxing: The basics\n\nSoftware sandboxing: The basics\n\n- # Software sandboxing: The basics\n\n\n\n- [JSON Feed](https://blog.emilua.org/feed.json)\n\nDiving into the territory of software sandboxing is diving into mostly uncharted\nterritory. The necessary pieces to implement good sandboxing in your software\nare scattered all-around and the pioneers haven’t yet gathered enough knowledge\ninto an unified mappa mundi that can guide new sailors through some well\nunderstood safe routes. In this blog post I’ll offer my own share of experiences\nthat I have acquired while working on sandboxing support for Emilua. Writing\nstyle will suffer a little because I’ll err on the side of repeating myself too\nmuch to avoid any misunderstandings.\n\nDo keep in mind that some Lua code samples here require the\nunreleased Emilua 0.11 (just grab a recent commit from the repo’s development\nbranch).\n\nFirst, let’s get some informal (but useful) definition for sandboxing just to\nmake sure we’re on the same page. Here’s\n[the\ndefinition that was used by Julien Tinnes and Chris Evans at Hack In The Box\nMalaysia 2009](https://blog.cr0.org/2009/10/security-in-depth-for-linux-software.html):\n\nThe ability to restrict a process' privileges:\n\n- Programmatically;\n\n- Without administrative authority on the machine;\n\n- Discretionary privilege dropping.\n\nThat’s a very good definition to keep the ball rolling. Let’s quickly iterate\nover each point individually to make them crystal clear. However keep in mind\nthat the opinions I possess today are a little different from the\nopinions J. Tinnes and C. Evans had during the 2009 talk (especially around “is\nit okay to use superuser APIs?”), so my explanations will differ a little and\nguide you towards what I consider better practices for 2025.\n\n## Programmatic privilege dropping\n\nOSes present different interfaces to users and software developers. System\nadministrators traditionally rely on filesystem permissions to isolate services\n(UNIX daemons). If we allowed third-party programs to freely change such\npermissions then it’d nullify the policies the sysadmin was trying to enforce to\nbegin with.\n\nFurthermore third-party programs abstract their own virtual worlds and most of\nthe time UNIX filesystem permissions aren’t a good fit to model the security\npolicies such other virtual worlds require. Do you use UNIX permission modes to\ndefine who can see your Twitter feed or message you on Identi.ca? Filesystem\npermissions aren’t the only knobs sysadmins possess to restrict access rights,\nbut the reasoning developed here also apply to these other knobs.\n\nNonetheless a process inevitably runs on top of an OS and there are\nkernel-exposed resources the process interacts with (e.g. files). It’s this\ninterface that matters to the software developer. Web browsers such as Firefox\nrun DRM plugins and it’s desirable to run such third-party plugins without\nallowing them to have full access to every file that Firefox has access to\n(usually every file in the user’s HOME directory). Traditional tools such as\nsetuidgid can’t help here and their usefulness is limited as interfaces\nsysadmins turn to. setuidgid and similar tools aren’t interfaces intended for\nthe software developer to use.\n\nXKCD 1200: Authorization\n\nFor programmatic privilege dropping, traditional UNIX interfaces are a poor\nmatch, and OSes where this gap actually matters will provide extended interfaces\nthat go beyond traditional UNIX (e.g. FreeBSD’s Capsicum and Linux’s Seccomp).\n\n## Dropping privileges without root\n\nWhen good interfaces for sandboxing weren’t available, programmers found their\nway to create sandboxes anyway by abusing mechanisms available only to the\nsuperuser. The most emblematic technique in this class is a helper suid binary\nthat’ll configure a chroot jail.\n\nThe obvious problem with these approaches is that they aren’t available to all\nprograms. Allowing any program to install suid binaries defeat any security\nmeasures. Suid binaries equal to temporally raising privileges to full\nadministrative authority over the system. Privileges should only ever decrease,\nnever increase (principle of least privilege).\n\nAnother related concern here is to not design APIs that backfire by\nexponentially increasing the kernel attack surface. The Docker boom popularized\nLinux namespaces as a mechanism to cheaply isolate services. However within a\nnested user namespace, the process runs as superuser (within that namespace),\nand code paths within the kernel that would normally only be available to the\nsuperuser are now available to every user. We have over a decade of kernel code\nthat was never written with this premise in mind. This decision caused security\nproblems in the past, and it’s bound to happen again. To quote Andy Lutomirski:\n\nI consider the ability to use CLONE_NEWUSER to acquire CAP_NET_ADMIN over\nany network namespace and to thus access the network configuration API to be a\nhuge risk. For example, unprivileged users can program iptables. I’ll eat my hat\nif there are no privilege escalations in there.\n\n— Andy Lutomirski\n[https://lore.kernel.org/all/CALCETrWYRvqhyCwx5RX6L3TEYCfW0j6ThFUc+ASL7BpxgO5dEQ@mail.gmail.com/](https://lore.kernel.org/all/CALCETrWYRvqhyCwx5RX6L3TEYCfW0j6ThFUc+ASL7BpxgO5dEQ@mail.gmail.com/)\n\nIt’s fine to allow user namespaces as long as you restrict this interface to\ntrusted containerization tools (e.g. Docker). However Linux namespaces is a\nterrible interface for software sandboxing. Newer sandboxing interfaces in Linux\nsuch as Landlock were carefully designed to not exponentially increase the\nkernel attack surface as to avoid the disasters we’ve seen with Linux’s user\nnamespaces. [Moreover new ways to restrict\nnamespaces within Linux are still being developed and long-term it’s a bad bet\nto rely on them as a general sandboxing mechanism](https://lwn.net/Articles/903580/).\n\nThe first few years of software sandboxing research I’ve put into Emilua were\nsolely focused on Linux namespaces. After a lot of frustration the focus shifted\ntowards different solutions. Nowadays Emilua still offers support for Linux\nnamespaces, but the intended use-case now is the creation of containerization\ntools. For proper sandboxing within Emilua, you’ll use mechanisms other than\nLinux namespaces.\n\n## Discretionary privilege dropping\n\nActually sandboxes might also be defined as:\n\nA restricted, controlled execution environment that prevents potentially\nmalicious software […​] from accessing any system resources except those for\nwhich the software is authorized.\n\n— Committee on National Security Systems (CNSS) Glossary 2022\n[https://www.cnss.gov/CNSS/issuances/Instructions.cfm](https://www.cnss.gov/CNSS/issuances/Instructions.cfm)\n\nThere’s no actual consensus over what traits are required for some code to be\nconsidered sandboxed and definitions are usually very loose. These definitions\ndon’t require the properties we’ve been discussing so far. Therefore a different\nterm altogether might come in handy. J. Tinnes suggested “discretionary\nprivilege dropping”. That’s the type of sandboxing we’ll be looking into for\nthis article.\n\nDiscretionary privilege dropping doesn’t replace system administration\npolicies. Rather they complement each other and should be adopted in tandem.\n\n## Practical sandboxing: processes\n\nNow we’re hopefully on the same page. Sandbox for us mean the same thing:\ndiscretionary privilege dropping. How do we go from an unsandboxed program to a\nsandboxed one on existing real-world OSes? In every mainstream OS today, the\nprivilege boundary lies at the process level. Credentials are associated with\neach process and that’s what the kernel checks to decide whether the process can\nacquire new resources using ambient authority.\n\nLinux is actually different and associates credentials at the thread level, but\na design rooted at the thread level cannot work, and that’s why\n[glibc will do extra work to synchronize credentials\nacross threads even if the kernel is sloppy about\nit](https://ewontfix.com/17/). [GNOME\ndevelopers thought they could work at the thread level just to be proved wrong\nwith CVE-2023-43641](https://blogs.gnome.org/carlosg/2023/10/10/on-cve-2023-43641/).\n\nAdam Langley actually described a mechanism that in theory can work at the\nthread level, but in practice is economically too costly and I don’t think it’ll\never work:\n\nSo that’s what we do: each untrusted thread has a trusted helper thread running\nin the same process. This certainly presents a fairly hostile environment for\nthe trusted code to run in. For one, it can only trust its CPU registers - all\nmemory must be assumed to be hostile. Since C code will spill to the stack when\nneeded and may pass arguments on the stack, all the code for the trusted thread\nhas to carefully written in assembly.\n\nThe trusted thread can receive requests to make system calls from the untrusted\nthread over a socket pair, validate the system call number and perform them on\nits behalf. We can stop the untrusted thread from breaking out by only using CPU\nregisters and by refusing to let the untrusted code manipulate the VM in unsafe\nways with mmap, mprotect etc.\n\n— [https://www.imperialviolet.org/2009/08/26/seccomp.html](https://www.imperialviolet.org/2009/08/26/seccomp.html)\n\nLet’s not theorize over what alternative designs could work. For today,\nprocesses is what we got. Once we compartmentalise our program as separate\nprocesses, we can proceed to the next steps:\n\n- Assigning different privileges to each compartment (the processes).\n\n- Handling communication among the compartments.\n\nResearchers from FreeBSD’s Capsicum already had the right mental model to\ndevelop sandboxes for well over a decade:\n\nCompartmentalised application development is, of necessity, distributed\napplication development, with software components running in different processes\nand communicating via message passing.\n\n— Capsicum: practical capabilities for UNIX\nRobert N. M. Watson, Jonathan Anderson, Ben Laurie, and Kris Kennaway\n\nThe means to drop privileges are different on every platform, so we’ll skip this\nfor now and get back to it later. First let’s focus on the problem of\ndistributed application development.\n\n## The actor model and capability-based security\n\nThe actor model is one of the most well known patterns for the development of\ndistributed systems. Erlang is perhaps its most iconic user. However Erlang’s\ninterest in the actor model lies in high availability and fault\ntolerance. Nonetheless it’s still useful to look into widely used models even if\nwe’re not interested in high availability nor fault tolerance.\n\nIt’s common for many explanations of the actor model to quickly step into the\nworld of mathematics (which is fine). However many of them quickly become lost\ninto the world of abstraction and forget about computers entirely (which is not\nfine). So let’s just use a summary of the points we care about in the actor\nmodel:\n\n- Actors can manage their own internal state.\n\n- Actors can spawn other actors.\n\n- Actors can send messages to other actors.\n\n- Actors can include the addresses of other actors in messages.\n\nIf we summarize the actor model into concrete design choices within our\nprogramming language or framework, here’s what we care about:\n\n- There is a function to create actors. This function returns the address of the\nnew actor.\n\n- The address of an actor can be used to send messages.\n\n- The address of an actor can also be a message or part of a larger message.\n\n- There is a function to receive messages. This function read messages that are\nenqueued for the calling actor.\n\n- It’s possible to retrieve the address of the current actor.\n\n- Actors share no memory with each other.\n\n- An actor doesn’t run in parallel to itself. If an actor is currently running\nin thread A, it can’t also be running in thread B. However it’s fine for\nactors to jump from one thread to another (as in work-stealing threaded task\nschedulers). [That’s\nthe same property that Boost.Asio describe as strands](https://www.boost.org/doc/libs/1_87_0/doc/html/boost_asio/overview/core/strands.html).\n\nFor Emilua, this design translates into 3 functions:\n\nspawn_vm(module) -> actor\nactor.send(msg)\ninbox.receive() -> msg\n\nIf you can learn just 3 functions, you can code for the actor model. Let’s go\nover some examples now:\n\nCreating actors\n\n-- `module1.lua` will be the entry-point\n-- for the execution of `new_actor1`\nlocal new_actor1 = spawn_vm{\nmodule = 'module1' }\n\nlocal new_actor2 = spawn_vm{\nmodule = 'module2' }\n\nSending messages\n\nnew_actor1:send(1)\nnew_actor1:send('ping')\nnew_actor1:send{ arg1 = 1, arg2 = 2 }\n\nReceiving messages\n\nlocal inbox = require 'inbox'\n\nwhile true do\nlocal msg = inbox:receive()\nhandle(msg)\nend\n\nSending messages with addresses\n\nnew_actor1:send{ arg1 = 1, arg2 = 2, send_result_to = new_actor2 }\nnew_actor1:send{ arg1 = 3, arg2 = 4, send_result_to = inbox }\n\nIf we decide to use the actor model for sandboxing, then each process will be an\nactor. UNIX domain sockets can be used for actor messaging. Upon spawning a new\nactor, we setup socket inheritance so we can communicate with it. The socket\nwill be the actor address. We also need to be able to include the addresses of\nother actors in messages, but this is also covered because\n[it’s possible to send file\ndescriptors over UNIX domain sockets](https://blog.cloudflare.com/know-your-scm_rights/). The inbox file descriptor is never sent\nto other actors (i.e. we have a MPSC channel).\n\nEmilua has many implementations for the actor model, so we must explicitly\ninstruct it to use subprocesses upon spawning a new actor:\n\nCreating IPC-based actors\n\nlocal new_actor1 = spawn_vm{\nmodule = 'module1', subprocess = {} }\n\nlocal new_actor2 = spawn_vm{\nmodule = 'module2', subprocess = {} }\n\nThis design also solves another problem for our sandboxing concerns: handing\nresources over to restricted processes. “Everything is a file (descriptor)” is\none of the most well known phrases within the UNIX culture. If we can send file\ndescriptors then we have a really broad range of resources that we can work with\nfrom sandboxed processes. To mention just a few:\n\n- Files.\n\n- Directories.\n\n- Pipes.\n\n- Sockets.\n\n- Device nodes (e.g. /dev/random, GPU communication, …​).\n\n- Shared memory (memfds).\n\n- Process handles — pidfds, procdescs.\n\n- Sync objects (e.g. eventfd).\n\n- eBPF programs.\n\nThese are the resources we care about when we sandbox programs. These are the\nresources we’ll take into account when we develop our security models. If we can\nprove that we aren’t leaking file descriptors to the wrong actors then we can\nuse the actor model. Fortunately there’s a well researched model that solves\nthis problem for us: capability-based security. There’s even a programming\nlanguage based on the actor model and capability-based security:\n[the Pony programming language](https://www.ponylang.io/).\n\nThere’s only one small gap that we need to fill to combine both models:\ncapability based security assumes unforgeable tokens, but the actor model uses\naddresses (which are forgeable). In our case, this problem was already solved by\nthe use of channels instead of addresses. The API stays the same and nobody will\nnotice a thing. Now we can use capabilities to reason about questions such as:\n\n- Is it possible for actor A to have effective access to resource X?\n\n- How can we design a layout that makes it impossible for any sandboxed actor to\nsimultaneously have access to files and sockets?\n\nAs for the usage of file descriptors as capabilities, the rule of thumb would be\nto avoid ioctls, but we’ll be back to this topic later.\n\nThe actor model is simple to use, but very powerful. The ability to include the\naddresses of other actors in messages means arbitrarily variable topologies. On\nmost of my own projects, I restrict myself to tree topologies, but the moment a\ntree becomes unfit for my project, it’ll be easily replaced by a different\ntopology. So far I haven’t stumbled on a single sandboxed application that can’t\nbe modeled using actors.\n\nChromium Sandbox Topology Diagram\n\nIf you need guidelines on how to develop distributed applications using the\nactor model, you’ll enjoy several decades of R&D that’ve gone into it. Whether\nyou prefer books, small tutorials, face-to-face classes, study groups, or many\nother learning approaches, you’ll likely find something of use.\n\n## File descriptors as capabilities\n\nNow that we have messaging solved with the actor model, let’s jump into\nsandboxing (security models) again. There are other properties an object must\nhave so it can be modeled as a capability. A capability isn’t only a reference\nto a resource, but the associated access rights as well. Owning a capability is\nthe same as also having access rights to perform actions. With this in mind, we\nneed to ponder:\n\n- Can file descriptors be modeled as capabilities?\n\n- What precautions must we take to use file descriptors as capabilities?\n\nGenerally UNIX systems run permission checks to grant or deny access only when a\nnew file descriptor is created, not when existing file descriptors are\nused. This behavior is compatible with capabilities. Here’s the code for a\nsample program:\n\n#include \n#include \n\nint main()\n{\nint fd = open(\"/root\", O_RDONLY);\nif (fd == -1) {\nperror(\"open\");\n} else {\nprintf(\"success\\n\");\n}\n}\n\nAnd the output when I run the program as root:\n\nsuccess\n\nAnd the output when I run the program as any other user:\n\nopen: Permission denied\n\nThis is just the UNIX behavior I was mentioning. Now let’s run some shell command as root:\n\n# grep -Ee '^nobody:' &1 | cat\nPassword: su: Authentication token manipulation error\n\nThis example shows that we can easily trick suid binaries to read or write into\nany file descriptors that we have by using simple fd inheritance. For this\nexample, it wrote the string “Password: su: Authentication token manipulation\nerror” using the credentials of a privileged process. If the credentials of the\nwriter process had any importance for the security of the system, every UNIX\nsystem would be broken already. Thefefore new interfaces are always designed in\na way that the credentials of the writer process don’t matter at all.\n\nAs a recent example to further stress the point,\n[some of the last syscalls that Linux introduced\nwere related to filesystem mounting](https://lwn.net/Articles/759499/). The initial versions of the contributed\npatchset were rejected due to the use of the syscall write in operations\nthat’d use the credentials of the calling process for permission\nchecks. Eventually the contributor changed the design by using the new syscall\nfsconfig and the patchset was accepted.\n\nIt’s important to notice that kernel developers will respect the convention\nwhether suid binaries are allowed on our Linux distro or not. Even if we block\nsuid binaries from our OS entirely, we can still assume that no attacker will be\nable to gain new privileges by using our process as a proxy to perform some\ndangerous write operation (the attacker could just write into the file\ndescriptor directly instead and the effects would’ve been the same).\n\nThe exception to this rule are ioctls. Performing ioctls on fds received from\nuntrusted processes is always dangerous. Emilua relies on Boost.Asio for async\nIO, and Boost.Asio used to rely on FIONBIO when it shouldn’t. After a few\nemail exchanges, I managed to persuade Christopher Kohlhoff to change this\nbehavior and\n[now\nBoost.Asio will do the right thing as long as you’re at least on Boost 1.86](https://www.boost.org/doc/libs/1_86_0/doc/html/boost_asio/history.html).\nBy the way, even isatty() is — at least on Linux — implemented as an ioctl,\nso you really need to be careful about non-standard operations.\n\nAwesome. We can indeed model file descriptors as capabilities, but we weren’t\nthe first to reach this conclusion.\n\n## FreeBSD’s Capsicum\n\nCapsicum is an interface to better support the use of file descriptors as\ncapabilities that is part of FreeBSD since its 9.0 release. One of the\nfacilities offered by Capsicum is the function cap_enter. cap_enter drops\nprocess privileges by disabling ambient authority entirely.\n\nlocal new_actor3 = spawn_vm{\nmodule = 'module3',\nsubprocess = {\n-- the runtime will run this Lua\n-- code before it attempts to\n-- initialize complex libraries\n-- or acquire system resources\ninit = 'C.cap_enter()'\n} }\n\nThat’s it. One function call and we dropped privileges. All system accesses will\nhave to be performed through open file descriptors. If we don’t already have\naccess to some resource, the only way to get it now is through inbox. If we\nattempt to open files, open will fail because ambient authority is\ndisabled. If we attempt to connect a socket to some endpoint, the operation will\nfail because ambient authority is disabled. That’s the beauty of Capsicum: we\ndeny access to external resources because the names themselves that could be\nused to refer to resources become unavailable.\n\n[When\nthe Capsicum research was published, the following table was also presented](https://www.cl.cam.ac.uk/research/security/capsicum/papers/2010usenix-security-capsicum-website.pdf):\n\nTable 1. Sandboxing mechanisms employed by Chromium\n\nOperating system\nModel\nLine count\nDescription\n\nWindows\n\nACLs\n\n22350\n\nWindows ACLs and SIDs\n\nLinux\n\nchroot\n\n605\n\nsetuid root helper sandboxes renderer\n\nMac OS X\n\nSeatbelt\n\n560\n\nPath-based MAC sandbox\n\nLinux\n\nSELinux\n\n200\n\nRestricted sandbox type enforcement domain\n\nLinux\n\nseccomp\n\n11301\n\nseccomp and userspace syscall wrapper\n\nFreeBSD\n\nCapsicum\n\n100\n\nCapsicum sandboxing using cap_enter\n\nAt that time, its researchers have modified Chromium to make use of Capsicum and\ncompared how much effort was required to make use of each sandboxing mechanism\nwithin Chromium. Capsicum required only 100 lines of code. Compare that to\nseccomp’s 11301 lines or Windows' 22350 lines. The other mechanisms compared\ndidn’t actually restrict the sandboxes significantly and can be disregarded. If\nyou’re only going to study one sandboxing mechanism in your life, it should be\nCapsicum. To this date, I have yet to see a better sandboxing mechanism than\nCapsicum.\n\nCapsicum also provides finer grained access control to file descriptors. As an\nexample, one may use Capsicum to allow one process to wait on a semaphore, but\nnot to post on it. A file descriptor is usually created with all rights\nassigned. Then these rights can be reduced through the use of the function\ncap_rights_limit.\n\nlocal unix = require 'unix'\n\nlocal in_, out = unix.seqpacket.socket.pair()\nout:shutdown('receive')\nout = out:release() --< get file descriptor\n\n-- deny shutdown-send so one\n-- worker cannot shutdown the\n-- channel to everyone\nout:cap_rights_limit({'send'})\n\nfor i = 1, 20 do\nlocal worker = spawn_vm{\nmodule = 'worker',\nsubprocess = {\ninit = 'C.cap_enter()'} }\nworker:send(out)\nend\nout:close()\n\nlocal buf = byte_span.new(512)\nwhile true do\nlocal nread = in_:receive(buf)\nprint(buf:first(nread))\nend\n\n[Although\nopen() won’t work in Capsicum mode, openat() will, and Capsicum will make\nsure the relative paths are only resolved to a hierarchy beneath the given\ndirectory-fd](https://val.packett.cool/blog/use-openat/#and-back-to-the-regular-syscall-interfaces). If only we could force Linux syscalls to always include\nRESOLVE_BENEATH…​ but maybe we can? Keep reading until we’re back at this\ntopic.\n\n[I haven’t actually\nfaced many problems using Capsicum so the only complaint I’ve had was fixed long\nago](https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275330). There’s not much to talk about Capsicum. The system is incredibly simple\nto use yet powerful. This model will be the inspiration for all sandboxes\ndeveloped in the rest of this article no matter the OS.\n\n## Non-blocking IO on UNIX: it sucks!\n\nOnce we do receive a file descriptor from a sandbox, it’s time to operate\non it. However if we’re sloppy about it, our thread will block. We need to avoid\nblocking operations to dodge some DoS\nattempts. [The mess around non-blocking IO on\nUNIX has been long known](https://cr.yp.to/unix/nonblock.html). Believe me when I say I have my own share of comments\nto make here, but this article isn’t about async IO and the text is already\ngetting too long, so I’ll just present a boring summary of what you need to\nknow:\n\n- [close() may block according to POSIX](https://ewontfix.com/4/).\n\n- [Supposedly close() always succeed on\nLinux](https://lwn.net/Articles/576478/). Don’t bother checking for errors here.\n\n- [You\nmay need to create a thread to run close() if “slow-to-close” files are a\nproblem](https://uapi-group.org/kernel-features/#disabling-reception-of-scm_rights-for-af_unix-sockets).\n\n- Use fstat on the received file descriptor to check whether it’s a socket.\n\n- On sockets, use MSG_DONTWAIT on recv(). Boost.Asio still gets this wrong,\nbut I’m short on time to send bugfixes anytime soon.\n\n- On non-sockets, use a proactor (completion events opposed to readiness events)\nto perform IO operations (e.g. io_uring on Linux, POSIX AIO on FreeBSD, …​).\n\n- [On FreeBSD, you can\nnow use aio_read2() with AIO_OP2_FOFFSET to read without an offset](https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269638). Other\napproaches will likely fail with ENOTCAPABLE. Boost.Asio also gets this\nwrong, and I’m short on time to send bugfixes here too.\n\n- [On\nLinux, io_uring is widely distrusted and disabled](https://security.googleblog.com/2023/06/learnings-from-kctf-vrps-42-linux.html). Therefore you might just\nas well reject non-socket IO on Linux if the file descriptor was received from\na sandboxed process.\n\nBy now you should understand that there are actually two use cases:\n\n- Trusted process creates a resource (file descriptor) and sends it to a\ndistrusted sandboxed process.\n\n- The distrusted sandboxed process creates a resource and sends it elsewhere.\n\nIf the file descriptor was created by a trusted process then the range of\noptions we can work with widens. However state such as O_NONBLOCK is shared\namong all copies of a file descriptor and become a problem the moment the file\ndescriptor reaches the first sandboxed process. On Capsicum we can at least\nforbid F_SETFL and alleviate the problem slightly, but this approach only\nworks for FreeBSD.\n\n## Sandboxing existing code\n\nBy now you should have enough tools in your toolbox to sandbox all your future\ncode. However there’s a reason why we sandbox code. Projects grow high and large\nuntil it becomes impossible to ensure their code is…​ bug-free. A little bug in\njust one of the dozens of modules within a project shouldn’t equal to a fully\ncompromised system when the bug is exploited by a hacker. Damage should be\ncontained. Security policies implemented by OS tools external to your code can\nonly work at a program (or user) level. The program will still be fully\ncompromised and the hacker will have access to all data and credentials the\nprogram has access to.\n\nWhen your program deals with data that can be processed independently, there’s\nan opportunity to implement a safer approach. If you can run multiple instances\nof your program as different users on your OS, you can use existing security\nsolutions in your project. However if the concept of allocating defined portions\nof the data to fixed users doesn’t work for your project, you may need something\nmore complex or custom-tailored. When the relationships between the users are\nblurry and your project demands policies that are more dynamic, you may need\nsandboxes.\n\nAs a rule of thumb, every shell should have sandboxes. Shell are programs that\nact as the membrane that sits between the human operator and some virtual\nworld. Tablets, smartphones, and laptops display graphical shells to interact\nwith programs, windows, and files. Servers employ textual shells. Likewise web\nbrowsers act as the shells to the www world.\n\nI wouldn’t be surprised if Firefox and Chrome were the only software employing\ndiscretionary privilege dropping that you know. They are shells after all so it\nmatters to them. More than that, they’re very well funded projects. Sandboxing\nused to be very expensive (especially outside FreeBSD). However these shouldn’t\nbe the only software out there with builtin sandboxing support. Take Telegram,\nfor instance. The right media parsing bug could mean a hacker having access to\nall my chat history. What time does my son leave school? What people do I trust\nmy credit card info with? When will I go in a trip and leave my house\nunattended? These are just a few examples of the damage that might be done due\nto the lack of sandboxes in Telegram. Not only Telegram, but every instant\nmessenger should be employing sandboxes. Media parsing should always be\nperformed in dedicated sandboxes.\n\nThe first step into this direction is a realistic approach to real-world\nengineering: let’s not rewrite all code from scratch. Deal? The tricks you\nlearned earlier will still be useful, but from now on I’ll share tricks to work\non existing real-world code. Capsicum users refer to the ability to run\nunmodified code within sandboxes as oblivious sandboxing. Techniques for\noblivious sandboxing most often than not have nothing to do with discretionary\nprivilege dropping and can’t solve the problems we were mentioning just a\nsecond ago. However it’s possible to combine approaches from both worlds in the\nsame project so it’s important to study the techniques for oblivious\nsandboxing too.\n\nThe one place we need to look at to implement oblivious sandboxing is actually\npretty obvious: the ambient authority functions. In fact, that’s what projects\nsuch as [Super Capsicumizer\n9000](https://github.com/valpackett/capsicumizer) do. They inject a dynamic library into a process using LD_PRELOAD to\ninterpose ambient authority calls. This technique is actually yesterday news and\nprojects such as fakeroot have been using it for decades.\n\nSuper Capsicumizer 9000 demo: refusing to open /etc on gedit\n\nSuper Capsicumizer 9000 is actually a small experiment hacked together by a very\nvery small team. The experiment succeeded into opening old software built on top\nof complex libraries with a long history of changes. This is very\npromising. It’s a sign that maybe a single programmer working alone to interpose\njust a few functions for ambient authority access will have success in running\nlegacy code.\n\nProgrammers almost never do syscalls directly, and instead rely on libc to do\nthe syscalls on their behalf. That’s why this approach works so well. All you\nhave to do is to write a definition for the function from libc you want to\ninterpose. If you’re linking against the dynamic libc, your function will be\nloaded first and used instead. If you’re linking against the static libc,\nchances are that the libc symbol is actually a weak symbol so it’ll be dropped\nonce the static linker see your definition. Emilua has been using this approach\nto support dynamic and static executables on Linux and FreeBSD and so far\ngetaddrinfo was the only ambient authority function whose symbol lacked the\nattribute for weak symbols (please comment on the linked bug reports if you plan\nto build your own sandboxes using the same techniques or using Emilua):\n\n- [https://sourceware.org/bugzilla/show_bug.cgi?id=32509](https://sourceware.org/bugzilla/show_bug.cgi?id=32509).\n\n- [https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=283528](https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=283528).\n\nThe next step is choosing which functions to interpose. Functions from FreeBSD’s\nlibcasper are good first candidates. However for some reason libcasper doesn’t\ninterpose the functions it intends to replace so you’ll need to change names and\nparameters accordingly. libcasper functions (e.g. cap_getaddrinfo) always take\nan extra parameter. Another good source of inspiration to decide which functions\nto interpose is the library used by Super Capsicumizer 9000: libpreopen. Most of\nthe time, you’ll only need to interpose a few functions even for complex\nprojects.\n\nChromium renderers need very little authority. They need access to fontconfig to\nfind fonts on the system and to open those font files.\n\n— [https://www.imperialviolet.org/2009/08/26/seccomp.html](https://www.imperialviolet.org/2009/08/26/seccomp.html)\n\nEmilua 0.11 abstracts all these details into the module libc_service. The\nexample below shows how we use this module to override the behavior of open to\nreturn a rogue file descriptor when the subprocess try to open\n/dev/null. Actual sandboxing setup (i.e. privilege dropping within the new\nsubprocess) is omitted for brevity. The example also shows how to prefill the\ncode cache for the new subprocess so it won’t query the filesystem to fetch the\nLua code to execute.\n\nlocal libc_service = require 'libc_service'\nlocal stream = require 'stream'\nlocal pipe = require 'pipe'\nlocal fs = require 'filesystem'\n\nlocal master, slave = libc_service.new()\n\nslave.open = [[\nlocal real_open, path, flag, mode = ...\nlocal res, errno, fd = real_open(path, flag, mode)\nif fd then\nreturn fd\nelse\nreturn res, errno\nend\n]]\n\nlocal source_tree_cache = {}\nsource_tree_cache['a.lua'] = [[\nlocal stream = require 'stream'\nlocal file = require 'file'\nlocal fs = require 'filesystem'\n\nlocal f = file.stream.new()\nf:open(fs.path.new('/dev/null'), {'read_only'})\nf = stream.scanner.new{ stream = f }\nprint(f:get_line())\n]]\n\nspawn_vm{\nmodule = fs.path.new('/a.lua'),\nsubprocess = {\nsource_tree_cache = source_tree_cache,\nlibc_service = slave,\nstdout = 'share',\nstderr = 'share',\n}\n}\n\nspawn(function() pcall(function()\nwhile true do\nmaster:receive()\nif master.function_ ~= 'open' then\nmaster:use_slave_credentials()\ngoto continue\nend\nlocal p, f, m = master:arguments()\nif p ~= fs.path.new('/dev/null') then\nmaster:use_slave_credentials()\ngoto continue\nend\n\nlocal pi, po = pipe.pair()\npi = pi:release()\nspawn(function()\nstream.write_all(po, '/dev/null contents\\n')\npo:close()\nend):detach()\nmaster:send_with_fds(-2, {pi})\n::continue::\nend\nend) end):detach()\n\nEmilua uses UNIX sockets behind the scenes for communication between both\nprocesses. This approach allows one to implement fully dynamic security\npolicies. For instance, if you’re trying use Telegram’s tdlib to implement your\nown Telegram client, you could have the following rules for your secutiry\npolicy:\n\n- Only resolve name queries to pluto.web.telegram.org.\n\n- Only allow connect requests to the IP addresses we resolved in previous steps.\n\nThe little Lua script we send to be executed in the sandboxed side means we can\napply some simple call fixups at the call site to further broaden the use cases\nwe can tackle. For instance, when sandboxed code try to open a GUI connecting to\n/tmp/.X11-unix/X0, we can send a new file descriptor to an unrelated display\nserver and replace the socket from the original request with the new one using\ndup2 from the Lua script at the call site. In fact, we can do that:\n\nlocal libc_service = require 'libc_service'\nlocal stream = require 'stream'\nlocal system = require 'system'\nlocal pipe = require 'pipe'\nlocal unix = require 'unix'\nlocal fs = require 'filesystem'\n\nlocal preload_libc_path\ndo\nlocal pi, po = pipe.pair()\npo = po:release()\npi = stream.scanner.new{ stream = pi }\n\nsystem.spawn{\nprogram = 'pkg-config',\narguments = {'pkg-config', '--variable=libpath', 'emilua_preload_libc'},\nenvironment = system.environment,\nstdout = po,\n}\npo:close()\n\npreload_libc_path = tostring(pi:get_line())\nend\n\nlocal xephyrconnep\nfor i = 1, 20 do\nlocal pi, po = pipe.pair()\npo = po:release()\npi = stream.scanner.new{ stream = pi }\n\nlocal xephyr = system.spawn{\nprogram = 'Xephyr',\narguments = { 'Xephyr', ':' .. i, '-displayfd', '3' },\nenvironment = system.environment,\nextra_fds = {\n[3] = po,\n},\n}\npo:close()\n\nif pcall(function()\nlocal nr = tostring(pi:get_line())\nxephyrconnep = '/tmp/.X11-unix/X' .. nr\nreturn true\nend) then\nbreak\nend\nend\nif not xephyrconnep then\nprint('Failed to start Xephyr')\nsystem.exit(1)\nend\n\nlocal master, slave = libc_service.new()\n\nslave.connect_unix = [[\nlocal real_connect, fd, path = ...\nlocal res, errno, fd2 = real_connect(fd, path)\nif fd2 then\nC.dup2(fd2, fd)\nC.close(fd2)\nend\nreturn res, errno\n]]\n\nlocal guiappenv = system.environment\nguiappenv.DISPLAY = ':0'\nguiappenv.LD_PRELOAD = preload_libc_path\nguiappenv.EMILUA_LIBC_SERVICE_FD = '3'\nlocal guiapp = system.spawn{\nprogram = 'xterm',\narguments = { 'xterm' },\nenvironment = guiappenv,\nstdout = 'share',\nstderr = 'share',\nextra_fds = {\n[3] = slave,\n},\n}\nscope_cleanup_push(function() guiapp:wait() end)\n\nspawn(function() pcall(function()\nwhile true do\nmaster:receive()\nif\nmaster.function_ == 'connect_unix' and\n(\nmaster:arguments() == fs.path.new('\\0/tmp/.X11-unix/X0') or\nmaster:arguments() == fs.path.new('/tmp/.X11-unix/X0')\n)\nthen\nlocal xephyrconn = unix.stream.dial(xephyrconnep)\nmaster:send_with_fds(0, {xephyrconn:release()})\nelse\nmaster:use_slave_credentials()\nend\nend\nend) end):detach()\n\nThis example also shows that Emilua can make use of LD_PRELOAD to perform libc\ninterposition on existing programs such as xterm.\n\nAnother interesting approach that might prove useful to your projects is to use\nkcmp in your security policies. This technique would allow you to increase\nyour policies granularities even further by implementing different subpolicies\nfor each file descriptor.\n\nBy the way, now we can interpose openat() on Linux to make it work as in\nFreeBSD’s capability mode (but — as usual — don’t forget to forbid the actual\nsyscall):\n\nlocal libc_service = require 'libc_service'\n\nlocal master, slave = libc_service.new()\n\nslave.openat = [[\nlocal real_openat, dirfd, path, flags, mode, resolve = ...\nlocal res, errno, fd = real_openat(dirfd, path, flags, mode, resolve)\nif fd then\nreturn fd\nelse\nreturn res, errno\nend\n]]\n\nlocal worker = spawn_vm{\nmodule = 'module4',\nsubprocess = {\nlibc_service = slave,\n},\n}\n\npcall(function()\nwhile true do\nmaster:receive()\nif master.function_ == 'openat' then\nlocal path, flags, mode = master:arguments()\nflags[#flags + 1] = 'resolve_beneath'\nlocal dirfd = master:descriptors()\nlocal ok, res = pcall(function()\nreturn dirfd:openat(path, flags, mode)\nend)\nif ok then\nmaster:send_with_fds(-1, {res})\nelse\nmaster:send(-1, res)\nend\nelse\nmaster:use_slave_credentials()\nend\nend\nend)\n\nThese techniques are already probably more than what you need, but I have few\nmore tricks up in my sleeve to share, so let’s move on.\n\n## Sandboxing native plugins\n\nA common theme in the threat model of sandboxes is to assume that initially\ntrusted code becomes malicious once compromised. For instance, we might trust\nthat ffmpeg developers are well-intentioned and didn’t backdoored their\nproject. However ffmpeg is a complex project and legitimate bugs lurk around\njust waiting to be found. Some of these bugs might be exploitable by hackers. In\nthis case we can import ffmpeg as a library in our executable and only setup the\nsandbox right before we call ffmpeg functions on external data.\n\nHowever how’d we approach the sandboxing steps if we assumed the code to be\ncompromised from the start? Let’s take Telegram’s tdlib as an example. Suppose\nyou don’t trust tdlib at all. Under this threat model, even just loading the\nlibrary would be a dangerous operation. For such scenario, first we need to\nbuild tdlib under a secure environment (e.g. jails on FreeBSD, namespaces on\nLinux). Once we do have the built plugin, we can move to the next challenge.\n\nIf we disable ambient authority to sandbox the code, dlopen() will fail to\naccess the filesystem. To work around this issue, we can dlopen by file\ndescriptors instead. On FreeBSD, we can use fdlopen. On Linux, we can pass a\npath to /proc/self/fd/ as long as we never close the file descriptor to avoid\npath reuse by a different plugin (glibc will deduplicate plugins by\npath). That’s not a perfect solution, but it’s a start.\n\nOn some previous example, we saw code cache prefilling as an Emilua way to\ninstruct a policy to avoid filesystem queries. Emilua just follows the same\ntrend for plugins and exposes native_modules_cache to prefill the native\nplugin cache:\n\nspawn_vm{\nmodule = 'some_module',\nsubprocess = {\nnative_modules_cache = {\n'some_plugin'\n} } }\n\nThe biggest problem with plugins is that they might depend on not yet loaded\ndynamic libraries and dlopen would fail to load them. We can try to use the\nworkaround for libc-service that we saw in the previous section, but there are\ncleaner solutions for this problem. On Linux, we can simply use\nLandlock. Landlock would already be required for the /proc/self/fd/ trick\nanyway. [On FreeBSD, we can use rtld_set_var\nand LIBRARY_PATH_FDS](https://reviews.freebsd.org/D47351).\n\nspawn_vm{\nmodule = 'some_module',\nsubprocess = {\nnative_modules_cache = {\n'some_plugin'\n},\nld_library_directories = library_path_fds,\n} }\n\nFortunately we won’t need to worry about this for Tdlib so the code becomes\nslightly simpler. However it begs the question: if we don’t trust tdlib, why are\nwe trusting it with our data anyway? The question here is to evaluate if the\nthreat model even makes sense. In the case of tdlib, we aren’t giving tdlib’s\ndevelopers (Telegram developers) anything they don’t already have (data on\nTelegram servers). Running tdlib within a plugin prevents the Telegram company\nto have unrestricted access to data on our systems. The answer for tdlib might\nbe simple, but that’s not what matters in this story. The lesson that you should\ntake here is to evaluate whether your threat models even make sense.\n\nLet’s explore another threat model story. The Linux kernel can load compressed\ninitramfs images. Even if the Linux kernel uses a buggy unmaintained library\nfull of well known exploits to decompress the initramfs image, it doesn’t matter\nat all! We only use this library on data generated by a trusted user. If some\nadversarial actor had control over the initramfs images we load, the actor could\nalready do any damage he wished for even if the decompression library had zero\nexploitable bugs. However the story would be completely different if the library\nwere backdoored. Sometimes it’s more important to have auditable code written by\ntrusted individuals than supposedly better code written by individuals we aren’t\nsure we can trust.\n\n## Dropping privileges with Seccomp\n\nI’ve postponed this section for as long as I could because it’s awful. Seccomp\nis not a good mechanism for discretionary privilege dropping. Seccomp is a good\nmechanism for OS hardening. Seccomp is a simple programmable syscall filtering\nmechanism based on BPF programs. The BPF program must choose an action for each\nsyscall attempted by the process:\n\n- SECCOMP_RET_KILL_PROCESS.\n\n- SECCOMP_RET_KILL_THREAD.\n\n- SECCOMP_RET_TRAP.\n\n- SECCOMP_RET_ERRNO.\n\n- SECCOMP_RET_USER_NOTIF.\n\n- SECCOMP_RET_TRACE.\n\n- SECCOMP_RET_LOG.\n\n- SECCOMP_RET_ALLOW.\n\nThis mechanism can be used to deny access (e.g. SECCOMP_RET_ERRNO) to ambient\nauthority by disallowing syscalls that work on names (e.g. open, bind). All\nyou have to do is disable ambient authority, then your process will be properly\nsandboxed and you can use the lessons learned in previous sections for\ncompartmentalised application development. Using this mechanism you may either\nimplement whitelists or blacklists. These projects implement syscall blacklists:\n\n- [https://github.com/lxc/lxc/blob/v6.0.3/config/templates/common.seccomp](https://github.com/lxc/lxc/blob/v6.0.3/config/templates/common.seccomp)\n\n- [https://github.com/flatpak/flatpak/issues/4187#issuecomment-1075512546](https://github.com/flatpak/flatpak/issues/4187#issuecomment-1075512546)\n\n- [https://github.com/flatpak/flatpak/blob/1.16.0/common/flatpak-run.c#L1839](https://github.com/flatpak/flatpak/blob/1.16.0/common/flatpak-run.c#L1839)\n\nThe problem with blacklists is well known. New kernel versions may add new\nsyscalls. You don’t know today what syscalls will be added tomorrow. You don’t\nknow today if tomorrow’s syscall breaks your today’s policy. However the problem\nis even worse for\nseccomp. [Linux\nallows multiarch systems and syscall numbering varies wildly among arches](https://github.com/seccomp/libseccomp/blob/v2.5.5/src/syscalls.csv). Even\nif you block syscalls such as acct on your x86-64 program, a compromised\nsandbox could bypass the syscall filter by running a x86 executable. To our\ndelight, Linux somehow manages to make the problem even worse:\n\nThe arch field is not unique for all calling conventions. The x86-64 ABI and\nthe x32 ABI both use AUDIT_ARCH_X86_64 as arch, and they run on the same\nprocessors. Instead, the mask __X32_SYSCALL_BIT is used on the system call\nnumber to tell the two ABIs apart.\n\nThis means that a policy must either deny all syscalls with X32_SYSCALL_BIT\nor it must recognize syscalls with and without X32_SYSCALL_BIT set. A list\nof system calls to be denied based on nr that does not also contain nr\nvalues with X32_SYSCALL_BIT set can be bypassed by a malicious program that\nsets X32_SYSCALL_BIT.\n\n— [seccomp(2)](https://www.mankier.com/2/seccomp#Description-Filters)\n\nEven when we do migrate to whitelists, we’ll still be haunted by these\nimplementation details. Here are a few notable projects based on whitelists:\n\n- [https://github.com/moby/moby/blob/v27.4.1/profiles/seccomp/default.json](https://github.com/moby/moby/blob/v27.4.1/profiles/seccomp/default.json)\n\n- [https://github.com/containers/podman/blob/v5.3.1/vendor/github.com/containers/common/pkg/seccomp/seccomp.json](https://github.com/containers/podman/blob/v5.3.1/vendor/github.com/containers/common/pkg/seccomp/seccomp.json)\n\n- [https://gitlab.gnome.org/GNOME/localsearch/-/blob/3.8.2/src/libtracker-miners-common/tracker-seccomp.c#L141](https://gitlab.gnome.org/GNOME/localsearch/-/blob/3.8.2/src/libtracker-miners-common/tracker-seccomp.c#L141)\n\n- [https://android.googlesource.com/platform/bionic/+/704772bda034448165d071f68b6aeca716f4220e/libc/seccomp/seccomp_policy.cpp](https://android.googlesource.com/platform/bionic/+/704772bda034448165d071f68b6aeca716f4220e/libc/seccomp/seccomp_policy.cpp)\n\nOnce you do go through these lists to implement your own seccomp policies,\nyou’ll face another problem: Linux syscall parameter ordering changes among\narches. That’s why Docker uses multiple rules for the syscall clone. Why do we\nhave to become experts in Linux syscall conventions to do what FreeBSD users get\ndone in 10 seconds by just calling cap_enter()? Welcome to Linux.\n\nThe good news is that in theory we could implement some library with a single\nfunction to disable ambient authority and everyone would then just use this\nlibrary. I have a few ideas on how to implement this library, but so far no\ncustomer of mine was interested in this problem (at the same time I’m busy with\ndifferent projects).\n\nHowever there’s just another caveat you must keep in mind. Linux userspace\nrelies on the filesystem too much. Even if you interpose open for\ncompatibility with legacy code, legacy code will fail trying to access\n/proc/self when nested sandboxes are at play. It’s better to just allow open\nand filter filesystem access with Landlock instead. The problem now is that file\ndescriptors can’t be modeled as capabilities if a process can just reopen\n/proc/self/fd/ with a different mode. Landlock developers shared a long-term\ngoal to expose capabilities compatible with Capsicum, so maybe we’ll have a\nsolution to this problem in the future.\n\nSeccomp’s complexity might be disheartening for the compartmentalised\napplication developer, but the situation is even worse if you look into Linux\nnamespaces. For Linux, Seccomp + Landlock is what we have.\n\n## Easier Seccomp with Kafel\n\nKafel is a language and library for specifying syscall filtering policies. The\npolicies are compiled into BPF code that can be used with seccomp-filter.\n\n— [https://google.github.io/kafel/](https://google.github.io/kafel/)\n\nKafel is the most promising project to enable Seccomp policy reusing that I’ve\nseen during my quests. Using Kafel I’ve managed to define a few policy groups\nthat might be of interest to you:\n\nKafel policies\n\n// These policies are heavily influenced by Docker's default profile. Further\n// customization done on top:\n//\n// - Avoid syscalls that need root anyway. The policies here are mostly meant to\n// be used by unprivileged users (not containers with root inside). The\n// syscalls wouldn't be harmful, but would result in larger BPF programs that\n// in turn incur more overhead.\n// - Avoid rarely used syscalls that can be abused for yet more fingerprinting\n// on desktop applications. This category mostly contains syscalls useful for\n// profiling (e.g. mincore, cachestat).\n// - Split them into categories inspired by systemD's seccomp filter sets and\n// OpenBSD's pledge promises.\n\nPOLICY Aio {\nALLOW {\nio_cancel, io_destroy, io_getevents, io_pgetevents, io_setup, io_submit\n}\n}\n\nPOLICY BasicIo {\nALLOW {\nread, readv, tee, vmsplice, write, writev,\n\n// ioctl() is definitively not about generic/stream/basic I/O. ioctl()\n// is really a syscall in disguise that device drivers can use for\n// anything. However it's expected that any program doing file I/O or\n// socket I/O or TTY IO will eventually stumble on glibc using ioctl()\n// for some operations so let's go ahead and just include it in the\n// basic IO set to force other IO categories to include it too.\nioctl\n}\n}\n\nPOLICY Clock {\nALLOW {\nclock_getres, clock_gettime, gettimeofday, time, times\n}\n}\n\n// Compat quirks. This family of policies is a good candidate to be maintained\n// in a different repo.\nPOLICY CompatX86 {\nALLOW {\n// important for old ABI emulation\npersonality(persona) {\npersona == /*PER_LINUX=*/0 || persona == /*PER_LINUX32=*/8 ||\npersona == /*UNAME26=*/0x0020000 ||\npersona == /*PER_LINUX32|UNAME26=*/0x20008 ||\npersona == 0xffffffff\n},\n\n// Important for x86 family's ABI. We put it in here instead of\n// c-runtime because other archs don't need it. Ideally Kafel would\n// allow us to write arch_prctl@amd64 in c-runtime and the rule would\n// only be included when we're building for the amd64 arch.\narch_prctl\n}\n}\n\nPOLICY CompatDB32 {\nALLOW {\nremap_file_pages\n}\n}\n\nPOLICY CompatSystemd {\nALLOW {\n// SystemD uses this to get mount-id\nname_to_handle_at\n}\n}\n\nPOLICY CompatWine {\nALLOW {\nmodify_ldt\n}\n}\n\nPOLICY Credentials {\nALLOW {\ngetegid, geteuid, getgid, getgroups, getresgid, getresuid, getuid\n}\n}\n\nPOLICY CredentialsExtra {\nALLOW {\n// SystemD lists this syscall in the policy 'process' with the reasoning\n// that it's able to query arbitrary processes so it's a process\n// relationship related syscall. Following the same reasoning, we opt to\n// not include this syscall in the policy 'credentials' as other\n// syscalls in that category don't allow querying arbitrary\n// processes. However we also opt to not include capget in the category\n// 'process' given most usages of that policy won't need capget at all\n// and would just make the resulting BPF bigger.\ncapget\n}\n}\n\nPOLICY CredentialsMutation {\nALLOW {\ncapset, setfsgid, setfsuid, setgid, setgroups, setregid, setresgid,\nsetresuid, setreuid, setuid\n}\n}\n\n// Memory allocation, threading, syscall interaction (or libc support) and\n// functions that should always be available (e.g. exit_group to bail out as a\n// program's last resort).\n//\n// Do notice that actually opening a libc-based program requires access to much\n// more syscalls as the loader is going to scrape the filesystem for the\n// required libraries and do many operations to stich the program image\n// together. The idea here is to apply a filter that will allow the C runtime to\n// keep running after we already have the program image in RAM.\nPOLICY CRuntime {\nALLOW {\nbrk, exit, exit_group, futex, futex_requeue, futex_wait, futex_waitv,\nfutex_wake, get_robust_list, get_thread_area, gettid, madvise,\nmap_shadow_stack, membarrier, mmap, mprotect, mremap, munmap,\nrestart_syscall, rseq, sched_yield, set_robust_list, set_thread_area,\nset_tid_address,\n\n// glibc's malloc() has references to getrandom(), so it's included here\ngetrandom\n}\n}\n\n// These syscalls are already gated by YAMA's ptrace_scope or capabilities\n// (e.g. CAP_PERFMON). The usual reasoning would be that it's safe to permit\n// them, but:\n//\n// - They are really only useful for process inspection/debugging.\n// - For IPC usage, better mechanisms exist (e.g. one can memfd+seal+mmap to\n// have zero copy I/O between cooperating processes).\n// - They appeared in a few CVEs in the past.\nPOLICY Debug {\nALLOW {\nkcmp, pidfd_getfd, perf_event_open, process_madvise, process_mrelease,\nprocess_vm_readv, process_vm_writev, ptrace\n}\n}\n\nPOLICY FileDescriptors {\nALLOW {\nclose, close_range, dup, dup2, dup3, fcntl\n}\n}\n\n// This policy is split off from filesystem so a process could still perform\n// file IO on:\n//\n// - Already open files.\n// - Files received from UNIX sockets.\n// - Memfds.\nPOLICY FileIo {\nALLOW {\ncopy_file_range, fadvise64, fallocate, flock, ftruncate, lseek, pread64,\npreadv, preadv2, pwrite64, pwritev, pwritev2, readahead, sendfile,\nsplice\n}\n}\n\n// OpenBSD's pledge further breaks down this promise into rpath, wpath, cpath\n// and dpath, but Landlock would be more appropriate to mirror the intention of\n// such granular designs\nPOLICY Filesystem {\nALLOW {\naccess, chdir, creat, faccessat, faccessat2, fchdir, fgetxattr,\nflistxattr, fstat, fstatfs, getcwd, getdents, getdents64, getxattr,\ninotify_add_watch, inotify_init, inotify_init1, inotify_rm_watch,\nlgetxattr, link, linkat, listxattr, llistxattr, lstat, mkdir, mkdirat,\nmknod, mknodat, newfstatat, open, openat, openat2, readlink, readlinkat,\nrename, renameat, renameat2, rmdir, stat, statfs, statx, symlink,\nsymlinkat, truncate, umask, unlink, unlinkat\n}\n}\n\n// Allowed to make explicit changes to fields in struct stat relating to a file.\nPOLICY FilesystemAttr {\nALLOW {\nchmod, chown, fchmod, fchmodat, fchmodat2, fchown, fchownat,\nfremovexattr, fsetxattr, futimesat, lchown, lremovexattr, lsetxattr,\nremovexattr, setxattr, utime, utimensat, utimes\n}\n}\n\n// Event loop system calls.\nPOLICY IoEvent {\nALLOW {\nepoll_create, epoll_create1, epoll_ctl, epoll_ctl_old, epoll_pwait,\nepoll_pwait2, epoll_wait, epoll_wait_old, eventfd, eventfd2, poll,\nppoll, pselect6, select\n}\n}\n\n// io_uring nowadays is considered unsafe for general usage:\n// http://security.googleblog.com/2023/06/learnings-from-kctf-vrps-42-linux.html\nPOLICY IoUring {\nALLOW {\nio_uring_enter, io_uring_register, io_uring_setup\n}\n}\n\n// SysV IPC, POSIX Message Queues or other IPC.\nPOLICY Ipc {\nALLOW {\nmemfd_create, mq_getsetattr, mq_notify, mq_open, mq_timedreceive,\nmq_timedsend, mq_unlink, msgctl, msgget, msgrcv, msgsnd, pipe, pipe2,\nsemctl, semget, semop, semtimedop, shmat, shmctl, shmdt, shmget\n}\n}\n\n// Memory locking control.\nPOLICY Memlock {\nALLOW {\nmemfd_secret, mlock, mlock2, mlockall, munlock, munlockall\n}\n}\n\nPOLICY NetworkIo {\nALLOW {\nconnect, getpeername, getsockname, getsockopt, recvfrom, recvmmsg,\nrecvmsg, sendmmsg, sendmsg, sendto, setsockopt, shutdown\n}\n}\n\nPOLICY NetworkServer {\nALLOW {\naccept, accept4, bind, listen\n}\n}\n\nPOLICY NetworkSocketTcp {\nALLOW {\nsocket(domain, type, protocol) {\n(type & 0x7ff) == /*SOCK_STREAM=*/1 && protocol == 0 &&\n(domain == /*AF_INET=*/2 || domain == /*AF_INET6=*/10)\n}\n}\n}\n\nPOLICY NetworkSocketUdp {\nALLOW {\nsocket(domain, type, protocol) {\n(type & 0x7ff) == /*SOCK_DGRAM=*/2 && protocol == 0 &&\n(domain == /*AF_INET=*/2 || domain == /*AF_INET6=*/10)\n}\n}\n}\n\nPOLICY NetworkSocketUnix {\nALLOW {\nsocket(domain, type, protocol) {\ndomain == /*AF_UNIX=*/1 && protocol == 0\n},\nsocketpair(domain, type, protocol) {\ndomain == /*AF_UNIX=*/1 && protocol == 0\n}\n}\n}\n\n// System calls used for memory protection keys.\nPOLICY Pkey {\nALLOW {\npkey_alloc, pkey_free, pkey_mprotect\n}\n}\n\n// Process control, execution, namespacing, relationship operations.\n//\n// Most likely you'll ALWAYS need access to this set to sandbox other binaries:\n// . It's only\n// really practical to exclude this set from the seccomp filter if you're\n// sandboxing yourself (i.e. cooperatively dropping further privileges before\n// doing dangerous stuff). It's a shame that Linux doesn't offer this type of\n// transition-on-exec mechanism for seccomp nor cgroups. Folks from SELinux\n// already know just how important it is to support this kind of mechanism for\n// properly dropping privileges, and it'd be good for more kernel hackers to\n// learn this lesson as well.\nPOLICY Process {\nALLOW {\n// Where's clone2? ia64 is the only architecture that has clone2, but\n// ia64 doesn't implement seccomp. c.f.\n// acce2f71779c54086962fefce3833d886c655f62 in the kernel.\nclone, clone3, execve, execveat, fork, getpgid, getpgrp, getpid,\ngetppid, getrusage, getsid, kill, pidfd_open, pidfd_send_signal, prctl,\nrt_sigqueueinfo, rt_tgsigqueueinfo, setpgid, setsid, tgkill, tkill,\nvfork, wait4, waitid\n}\n}\n\nPOLICY Resources {\nALLOW {\ngetcpu, getpriority, getrlimit, ioprio_get, sched_getaffinity,\nsched_getattr, sched_getparam, sched_get_priority_max,\nsched_get_priority_min, sched_getscheduler, sched_rr_get_interval\n}\n}\n\n// Alter resource settings.\nPOLICY ResourcesMutation {\nALLOW {\nioprio_set, prlimit64, sched_setaffinity, sched_setattr, sched_setparam,\nsched_setscheduler, setpriority, setrlimit\n}\n}\n\nPOLICY Sandbox {\nALLOW {\nlandlock_add_rule, landlock_create_ruleset, landlock_restrict_self,\nseccomp\n}\n}\n\n// Process signal handling.\nPOLICY Signal {\nALLOW {\npause, rt_sigaction, rt_sigpending, rt_sigprocmask, rt_sigreturn,\nrt_sigsuspend, rt_sigtimedwait, sigaltstack, signalfd, signalfd4\n}\n}\n\n// Synchronize files and memory to storage.\nPOLICY Sync {\nALLOW {\nfdatasync, fsync, msync, sync, sync_file_range, syncfs\n}\n}\n\n// Schedule operations by time.\nPOLICY Timer {\nALLOW {\nalarm, getitimer, clock_nanosleep, nanosleep, setitimer, timer_create,\ntimer_delete, timer_getoverrun, timer_gettime, timer_settime,\ntimerfd_create, timerfd_gettime, timerfd_settime\n}\n}\n\nThese are the policies I’ve been using for almost a year. I’d make a few changes\nnowadays, but I haven’t gotten the time to it yet. Also do keep in mind that\nKafel’s syscall database is inexcusably poor so you’ll need a few changes (some\nsed-like preprocessing to replace the syscall names by their numbers) to make\nthe above work. Even when it does work, it’ll be omitting many syscalls that\nprojects using better syscall databases such as Docker handle. Did you notice\nthat we don’t mention clock_gettime64? That’s because Kafel’s syscall database\nis just too poor. Kafel won’t ever be adopted by projects such as Docker for as\nlong as it retains such poor syscall databases and poor multiarch support.\n\nHowever a lack of better syscall databases isn’t the only thing we could improve\nin Kafel. I’d like to see policy versioning and better policy composition\noperators, for instance. I’d be willing to develop these features, but again, no\ncurrent customer of mine was interested in this project, so my time will be\nspent in different projects.\n\n## The end?\n\nIn this article, I’ve gone just over the basics for sandboxing in Linux and\nFreeBSD. However there are more lessons that I’d like to share. I’ll save these\nfor a later article in the future. Topics that I’ll likely cover if I ever get\ninto the mood to write another blog post again:\n\n- More demos (for years I’ve been running graphical apps in some of my machines\nsolely in containers so I got quite a few demos to show).\n\n- More about the actor model, capability-based security, and access control\npolicies.\n\n- More about Linux kludges.\n\n- Maybe a comment or two on Windows and macOS.\n\n- PR_SET_DUMPABLE & /proc/sys/kernel/yama/ptrace_scope.\n\n- Containers (Emilua also works as a container runtime).\n\n- Sandboxing GUI applications.\n\n- Attack surfaces & safe parsers.\n\n- FreeBSD’s libnv.\n\n- Rights revocation & proxies.\n\n- Sandboxing patterns.\n\n- UNIX tricks for the C programmer that Emilua makes use of.\n\n© 2024 Emilua","body_html":"

Software sandboxing: The basics

\n

Software sandboxing: The basics

\n
  • # Software sandboxing: The basics
\n\n

Diving into the territory of software sandboxing is diving into mostly uncharted\nterritory. The necessary pieces to implement good sandboxing in your software\nare scattered all-around and the pioneers haven’t yet gathered enough knowledge\ninto an unified mappa mundi that can guide new sailors through some well\nunderstood safe routes. In this blog post I’ll offer my own share of experiences\nthat I have acquired while working on sandboxing support for Emilua. Writing\nstyle will suffer a little because I’ll err on the side of repeating myself too\nmuch to avoid any misunderstandings.

\n

Do keep in mind that some Lua code samples here require the\nunreleased Emilua 0.11 (just grab a recent commit from the repo’s development\nbranch).

\n

First, let’s get some informal (but useful) definition for sandboxing just to\nmake sure we’re on the same page. Here’s\nthe\ndefinition that was used by Julien Tinnes and Chris Evans at Hack In The Box\nMalaysia 2009:

\n

The ability to restrict a process' privileges:

\n
  • Programmatically;
  • Without administrative authority on the machine;
  • Discretionary privilege dropping.
\n

That’s a very good definition to keep the ball rolling. Let’s quickly iterate\nover each point individually to make them crystal clear. However keep in mind\nthat the opinions I possess today are a little different from the\nopinions J. Tinnes and C. Evans had during the 2009 talk (especially around “is\nit okay to use superuser APIs?”), so my explanations will differ a little and\nguide you towards what I consider better practices for 2025.

\n

Programmatic privilege dropping

\n

OSes present different interfaces to users and software developers. System\nadministrators traditionally rely on filesystem permissions to isolate services\n(UNIX daemons). If we allowed third-party programs to freely change such\npermissions then it’d nullify the policies the sysadmin was trying to enforce to\nbegin with.

\n

Furthermore third-party programs abstract their own virtual worlds and most of\nthe time UNIX filesystem permissions aren’t a good fit to model the security\npolicies such other virtual worlds require. Do you use UNIX permission modes to\ndefine who can see your Twitter feed or message you on Identi.ca? Filesystem\npermissions aren’t the only knobs sysadmins possess to restrict access rights,\nbut the reasoning developed here also apply to these other knobs.

\n

Nonetheless a process inevitably runs on top of an OS and there are\nkernel-exposed resources the process interacts with (e.g. files). It’s this\ninterface that matters to the software developer. Web browsers such as Firefox\nrun DRM plugins and it’s desirable to run such third-party plugins without\nallowing them to have full access to every file that Firefox has access to\n(usually every file in the user’s HOME directory). Traditional tools such as\nsetuidgid can’t help here and their usefulness is limited as interfaces\nsysadmins turn to. setuidgid and similar tools aren’t interfaces intended for\nthe software developer to use.

\n

XKCD 1200: Authorization

\n

For programmatic privilege dropping, traditional UNIX interfaces are a poor\nmatch, and OSes where this gap actually matters will provide extended interfaces\nthat go beyond traditional UNIX (e.g. FreeBSD’s Capsicum and Linux’s Seccomp).

\n

Dropping privileges without root

\n

When good interfaces for sandboxing weren’t available, programmers found their\nway to create sandboxes anyway by abusing mechanisms available only to the\nsuperuser. The most emblematic technique in this class is a helper suid binary\nthat’ll configure a chroot jail.

\n

The obvious problem with these approaches is that they aren’t available to all\nprograms. Allowing any program to install suid binaries defeat any security\nmeasures. Suid binaries equal to temporally raising privileges to full\nadministrative authority over the system. Privileges should only ever decrease,\nnever increase (principle of least privilege).

\n

Another related concern here is to not design APIs that backfire by\nexponentially increasing the kernel attack surface. The Docker boom popularized\nLinux namespaces as a mechanism to cheaply isolate services. However within a\nnested user namespace, the process runs as superuser (within that namespace),\nand code paths within the kernel that would normally only be available to the\nsuperuser are now available to every user. We have over a decade of kernel code\nthat was never written with this premise in mind. This decision caused security\nproblems in the past, and it’s bound to happen again. To quote Andy Lutomirski:

\n

I consider the ability to use CLONE_NEWUSER to acquire CAP_NET_ADMIN over\nany network namespace and to thus access the network configuration API to be a\nhuge risk. For example, unprivileged users can program iptables. I’ll eat my hat\nif there are no privilege escalations in there.

\n

— Andy Lutomirski\nhttps://lore.kernel.org/all/CALCETrWYRvqhyCwx5RX6L3TEYCfW0j6ThFUc+ASL7BpxgO5dEQ@mail.gmail.com/

\n

It’s fine to allow user namespaces as long as you restrict this interface to\ntrusted containerization tools (e.g. Docker). However Linux namespaces is a\nterrible interface for software sandboxing. Newer sandboxing interfaces in Linux\nsuch as Landlock were carefully designed to not exponentially increase the\nkernel attack surface as to avoid the disasters we’ve seen with Linux’s user\nnamespaces. Moreover new ways to restrict\nnamespaces within Linux are still being developed and long-term it’s a bad bet\nto rely on them as a general sandboxing mechanism.

\n

The first few years of software sandboxing research I’ve put into Emilua were\nsolely focused on Linux namespaces. After a lot of frustration the focus shifted\ntowards different solutions. Nowadays Emilua still offers support for Linux\nnamespaces, but the intended use-case now is the creation of containerization\ntools. For proper sandboxing within Emilua, you’ll use mechanisms other than\nLinux namespaces.

\n

Discretionary privilege dropping

\n

Actually sandboxes might also be defined as:

\n

A restricted, controlled execution environment that prevents potentially\nmalicious software […​] from accessing any system resources except those for\nwhich the software is authorized.

\n

— Committee on National Security Systems (CNSS) Glossary 2022\nhttps://www.cnss.gov/CNSS/issuances/Instructions.cfm

\n

There’s no actual consensus over what traits are required for some code to be\nconsidered sandboxed and definitions are usually very loose. These definitions\ndon’t require the properties we’ve been discussing so far. Therefore a different\nterm altogether might come in handy. J. Tinnes suggested “discretionary\nprivilege dropping”. That’s the type of sandboxing we’ll be looking into for\nthis article.

\n

Discretionary privilege dropping doesn’t replace system administration\npolicies. Rather they complement each other and should be adopted in tandem.

\n

Practical sandboxing: processes

\n

Now we’re hopefully on the same page. Sandbox for us mean the same thing:\ndiscretionary privilege dropping. How do we go from an unsandboxed program to a\nsandboxed one on existing real-world OSes? In every mainstream OS today, the\nprivilege boundary lies at the process level. Credentials are associated with\neach process and that’s what the kernel checks to decide whether the process can\nacquire new resources using ambient authority.

\n

Linux is actually different and associates credentials at the thread level, but\na design rooted at the thread level cannot work, and that’s why\nglibc will do extra work to synchronize credentials\nacross threads even if the kernel is sloppy about\nit. GNOME\ndevelopers thought they could work at the thread level just to be proved wrong\nwith CVE-2023-43641.

\n

Adam Langley actually described a mechanism that in theory can work at the\nthread level, but in practice is economically too costly and I don’t think it’ll\never work:

\n

So that’s what we do: each untrusted thread has a trusted helper thread running\nin the same process. This certainly presents a fairly hostile environment for\nthe trusted code to run in. For one, it can only trust its CPU registers - all\nmemory must be assumed to be hostile. Since C code will spill to the stack when\nneeded and may pass arguments on the stack, all the code for the trusted thread\nhas to carefully written in assembly.

\n

The trusted thread can receive requests to make system calls from the untrusted\nthread over a socket pair, validate the system call number and perform them on\nits behalf. We can stop the untrusted thread from breaking out by only using CPU\nregisters and by refusing to let the untrusted code manipulate the VM in unsafe\nways with mmap, mprotect etc.

\n

— https://www.imperialviolet.org/2009/08/26/seccomp.html

\n

Let’s not theorize over what alternative designs could work. For today,\nprocesses is what we got. Once we compartmentalise our program as separate\nprocesses, we can proceed to the next steps:

\n
  • Assigning different privileges to each compartment (the processes).
  • Handling communication among the compartments.
\n

Researchers from FreeBSD’s Capsicum already had the right mental model to\ndevelop sandboxes for well over a decade:

\n

Compartmentalised application development is, of necessity, distributed\napplication development, with software components running in different processes\nand communicating via message passing.

\n

— Capsicum: practical capabilities for UNIX\nRobert N. M. Watson, Jonathan Anderson, Ben Laurie, and Kris Kennaway

\n

The means to drop privileges are different on every platform, so we’ll skip this\nfor now and get back to it later. First let’s focus on the problem of\ndistributed application development.

\n

The actor model and capability-based security

\n

The actor model is one of the most well known patterns for the development of\ndistributed systems. Erlang is perhaps its most iconic user. However Erlang’s\ninterest in the actor model lies in high availability and fault\ntolerance. Nonetheless it’s still useful to look into widely used models even if\nwe’re not interested in high availability nor fault tolerance.

\n

It’s common for many explanations of the actor model to quickly step into the\nworld of mathematics (which is fine). However many of them quickly become lost\ninto the world of abstraction and forget about computers entirely (which is not\nfine). So let’s just use a summary of the points we care about in the actor\nmodel:

\n
  • Actors can manage their own internal state.
  • Actors can spawn other actors.
  • Actors can send messages to other actors.
  • Actors can include the addresses of other actors in messages.
\n

If we summarize the actor model into concrete design choices within our\nprogramming language or framework, here’s what we care about:

\n

  • There is a function to create actors. This function returns the address of the

    new actor.

  • The address of an actor can be used to send messages.
  • The address of an actor can also be a message or part of a larger message.

  • There is a function to receive messages. This function read messages that are

    enqueued for the calling actor.

  • It’s possible to retrieve the address of the current actor.
  • Actors share no memory with each other.

  • An actor doesn’t run in parallel to itself. If an actor is currently running

    in thread A, it can’t also be running in thread B. However it’s fine for\nactors to jump from one thread to another (as in work-stealing threaded task\nschedulers). That’s\nthe same property that Boost.Asio describe as strands.

\n

For Emilua, this design translates into 3 functions:

\n

spawn_vm(module) -> actor\nactor.send(msg)\ninbox.receive() -> msg

\n

If you can learn just 3 functions, you can code for the actor model. Let’s go\nover some examples now:

\n

Creating actors

\n

-- module1.lua will be the entry-point\n-- for the execution of new_actor1\nlocal new_actor1 = spawn_vm{\nmodule = 'module1' }

\n

local new_actor2 = spawn_vm{\nmodule = 'module2' }

\n

Sending messages

\n

new_actor1:send(1)\nnew_actor1:send('ping')\nnew_actor1:send{ arg1 = 1, arg2 = 2 }

\n

Receiving messages

\n

local inbox = require 'inbox'

\n

while true do\nlocal msg = inbox:receive()\nhandle(msg)\nend

\n

Sending messages with addresses

\n

new_actor1:send{ arg1 = 1, arg2 = 2, send_result_to = new_actor2 }\nnew_actor1:send{ arg1 = 3, arg2 = 4, send_result_to = inbox }

\n

If we decide to use the actor model for sandboxing, then each process will be an\nactor. UNIX domain sockets can be used for actor messaging. Upon spawning a new\nactor, we setup socket inheritance so we can communicate with it. The socket\nwill be the actor address. We also need to be able to include the addresses of\nother actors in messages, but this is also covered because\nit’s possible to send file\ndescriptors over UNIX domain sockets. The inbox file descriptor is never sent\nto other actors (i.e. we have a MPSC channel).

\n

Emilua has many implementations for the actor model, so we must explicitly\ninstruct it to use subprocesses upon spawning a new actor:

\n

Creating IPC-based actors

\n

local new_actor1 = spawn_vm{\nmodule = 'module1', subprocess = {} }

\n

local new_actor2 = spawn_vm{\nmodule = 'module2', subprocess = {} }

\n

This design also solves another problem for our sandboxing concerns: handing\nresources over to restricted processes. “Everything is a file (descriptor)” is\none of the most well known phrases within the UNIX culture. If we can send file\ndescriptors then we have a really broad range of resources that we can work with\nfrom sandboxed processes. To mention just a few:

\n
  • Files.
  • Directories.
  • Pipes.
  • Sockets.
  • Device nodes (e.g. /dev/random, GPU communication, …​).
  • Shared memory (memfds).
  • Process handles — pidfds, procdescs.
  • Sync objects (e.g. eventfd).
  • eBPF programs.
\n

These are the resources we care about when we sandbox programs. These are the\nresources we’ll take into account when we develop our security models. If we can\nprove that we aren’t leaking file descriptors to the wrong actors then we can\nuse the actor model. Fortunately there’s a well researched model that solves\nthis problem for us: capability-based security. There’s even a programming\nlanguage based on the actor model and capability-based security:\nthe Pony programming language.

\n

There’s only one small gap that we need to fill to combine both models:\ncapability based security assumes unforgeable tokens, but the actor model uses\naddresses (which are forgeable). In our case, this problem was already solved by\nthe use of channels instead of addresses. The API stays the same and nobody will\nnotice a thing. Now we can use capabilities to reason about questions such as:

\n
  • Is it possible for actor A to have effective access to resource X?

  • How can we design a layout that makes it impossible for any sandboxed actor to

    simultaneously have access to files and sockets?

\n

As for the usage of file descriptors as capabilities, the rule of thumb would be\nto avoid ioctls, but we’ll be back to this topic later.

\n

The actor model is simple to use, but very powerful. The ability to include the\naddresses of other actors in messages means arbitrarily variable topologies. On\nmost of my own projects, I restrict myself to tree topologies, but the moment a\ntree becomes unfit for my project, it’ll be easily replaced by a different\ntopology. So far I haven’t stumbled on a single sandboxed application that can’t\nbe modeled using actors.

\n

Chromium Sandbox Topology Diagram

\n

If you need guidelines on how to develop distributed applications using the\nactor model, you’ll enjoy several decades of R&D that’ve gone into it. Whether\nyou prefer books, small tutorials, face-to-face classes, study groups, or many\nother learning approaches, you’ll likely find something of use.

\n

File descriptors as capabilities

\n

Now that we have messaging solved with the actor model, let’s jump into\nsandboxing (security models) again. There are other properties an object must\nhave so it can be modeled as a capability. A capability isn’t only a reference\nto a resource, but the associated access rights as well. Owning a capability is\nthe same as also having access rights to perform actions. With this in mind, we\nneed to ponder:

\n
  • Can file descriptors be modeled as capabilities?
  • What precautions must we take to use file descriptors as capabilities?
\n

Generally UNIX systems run permission checks to grant or deny access only when a\nnew file descriptor is created, not when existing file descriptors are\nused. This behavior is compatible with capabilities. Here’s the code for a\nsample program:

\n

#include <fcntl.h>\n#include <stdio.h>

\n

int main()\n{\nint fd = open("/root", O_RDONLY);\nif (fd == -1) {\nperror("open");\n} else {\nprintf("success\\n");\n}\n}

\n

And the output when I run the program as root:

\n

success

\n

And the output when I run the program as any other user:

\n

open: Permission denied

\n

This is just the UNIX behavior I was mentioning. Now let’s run some shell command as root:

\n

grep -Ee '^nobody:' </etc/shadow

\n

nobody:!*:19642::::::

\n

And the same command as a different user:

\n

$ grep -Ee '^nobody:' </etc/shadow\n-bash: /etc/shadow: Permission denied

\n

Nothing surprising here. It’s just the same behavior. Now let’s run grep as an\nunprivileged user, but making sure it inherited a file descriptor opened by\nroot:

\n

setpriv --reuid=1000 --regid=1000 --init-groups grep -Ee '^nobody:' </etc/shadow

\n

nobody:!*:19642::::::

\n

As stated earlier, UNIX systems generally don’t run permission checks when\nperforming actions on existing file descriptors. That’s why grep succeeded to\nread the file contents in the example. This was true for the previous example\n(the action read on a regular file), but will it always be true? We could be\nafraid of new kernel versions. They could always introduce a new syscall that\nbreaks this convention. However early in UNIX history the concept of suid\nbinaries was introduced, and that’s a legacy that’ll keep haunting kernel\ndevelopers to make sure they don’t break this convention. Let’s explore suid\nbinaries now.

\n

In the last example we had the superuser using the syscall setresuid to change\nthe process credentials. Now we’ll walk into the opposite direction, creating a\nchild privileged process from an unprivileged one. This is allowed only for suid\nbinaries, so the privileged process will only ever run programs trusted by the\nsysadmin. One of such programs is su:

\n

$ setsid su </dev/null 2>&1 | cat\nPassword: su: Authentication token manipulation error

\n

This example shows that we can easily trick suid binaries to read or write into\nany file descriptors that we have by using simple fd inheritance. For this\nexample, it wrote the string “Password: su: Authentication token manipulation\nerror” using the credentials of a privileged process. If the credentials of the\nwriter process had any importance for the security of the system, every UNIX\nsystem would be broken already. Thefefore new interfaces are always designed in\na way that the credentials of the writer process don’t matter at all.

\n

As a recent example to further stress the point,\nsome of the last syscalls that Linux introduced\nwere related to filesystem mounting. The initial versions of the contributed\npatchset were rejected due to the use of the syscall write in operations\nthat’d use the credentials of the calling process for permission\nchecks. Eventually the contributor changed the design by using the new syscall\nfsconfig and the patchset was accepted.

\n

It’s important to notice that kernel developers will respect the convention\nwhether suid binaries are allowed on our Linux distro or not. Even if we block\nsuid binaries from our OS entirely, we can still assume that no attacker will be\nable to gain new privileges by using our process as a proxy to perform some\ndangerous write operation (the attacker could just write into the file\ndescriptor directly instead and the effects would’ve been the same).

\n

The exception to this rule are ioctls. Performing ioctls on fds received from\nuntrusted processes is always dangerous. Emilua relies on Boost.Asio for async\nIO, and Boost.Asio used to rely on FIONBIO when it shouldn’t. After a few\nemail exchanges, I managed to persuade Christopher Kohlhoff to change this\nbehavior and\nnow\nBoost.Asio will do the right thing as long as you’re at least on Boost 1.86.\nBy the way, even isatty() is — at least on Linux — implemented as an ioctl,\nso you really need to be careful about non-standard operations.

\n

Awesome. We can indeed model file descriptors as capabilities, but we weren’t\nthe first to reach this conclusion.

\n

FreeBSD’s Capsicum

\n

Capsicum is an interface to better support the use of file descriptors as\ncapabilities that is part of FreeBSD since its 9.0 release. One of the\nfacilities offered by Capsicum is the function cap_enter. cap_enter drops\nprocess privileges by disabling ambient authority entirely.

\n

local new_actor3 = spawn_vm{\nmodule = 'module3',\nsubprocess = {\n-- the runtime will run this Lua\n-- code before it attempts to\n-- initialize complex libraries\n-- or acquire system resources\ninit = 'C.cap_enter()'\n} }

\n

That’s it. One function call and we dropped privileges. All system accesses will\nhave to be performed through open file descriptors. If we don’t already have\naccess to some resource, the only way to get it now is through inbox. If we\nattempt to open files, open will fail because ambient authority is\ndisabled. If we attempt to connect a socket to some endpoint, the operation will\nfail because ambient authority is disabled. That’s the beauty of Capsicum: we\ndeny access to external resources because the names themselves that could be\nused to refer to resources become unavailable.

\n

When\nthe Capsicum research was published, the following table was also presented:

\n

Table 1. Sandboxing mechanisms employed by Chromium

\n

Operating system\nModel\nLine count\nDescription

\n

Windows

\n

ACLs

\n

22350

\n

Windows ACLs and SIDs

\n

Linux

\n

chroot

\n

605

\n

setuid root helper sandboxes renderer

\n

Mac OS X

\n

Seatbelt

\n

560

\n

Path-based MAC sandbox

\n

Linux

\n

SELinux

\n

200

\n

Restricted sandbox type enforcement domain

\n

Linux

\n

seccomp

\n

11301

\n

seccomp and userspace syscall wrapper

\n

FreeBSD

\n

Capsicum

\n

100

\n

Capsicum sandboxing using cap_enter

\n

At that time, its researchers have modified Chromium to make use of Capsicum and\ncompared how much effort was required to make use of each sandboxing mechanism\nwithin Chromium. Capsicum required only 100 lines of code. Compare that to\nseccomp’s 11301 lines or Windows' 22350 lines. The other mechanisms compared\ndidn’t actually restrict the sandboxes significantly and can be disregarded. If\nyou’re only going to study one sandboxing mechanism in your life, it should be\nCapsicum. To this date, I have yet to see a better sandboxing mechanism than\nCapsicum.

\n

Capsicum also provides finer grained access control to file descriptors. As an\nexample, one may use Capsicum to allow one process to wait on a semaphore, but\nnot to post on it. A file descriptor is usually created with all rights\nassigned. Then these rights can be reduced through the use of the function\ncap_rights_limit.

\n

local unix = require 'unix'

\n

local in_, out = unix.seqpacket.socket.pair()\nout:shutdown('receive')\nout = out:release() --< get file descriptor

\n

-- deny shutdown-send so one\n-- worker cannot shutdown the\n-- channel to everyone\nout:cap_rights_limit({'send'})

\n

for i = 1, 20 do\nlocal worker = spawn_vm{\nmodule = 'worker',\nsubprocess = {\ninit = 'C.cap_enter()'} }\nworker:send(out)\nend\nout:close()

\n

local buf = byte_span.new(512)\nwhile true do\nlocal nread = in_:receive(buf)\nprint(buf:first(nread))\nend

\n

Although\nopen() won’t work in Capsicum mode, openat() will, and Capsicum will make\nsure the relative paths are only resolved to a hierarchy beneath the given\ndirectory-fd. If only we could force Linux syscalls to always include\nRESOLVE_BENEATH…​ but maybe we can? Keep reading until we’re back at this\ntopic.

\n

I haven’t actually\nfaced many problems using Capsicum so the only complaint I’ve had was fixed long\nago. There’s not much to talk about Capsicum. The system is incredibly simple\nto use yet powerful. This model will be the inspiration for all sandboxes\ndeveloped in the rest of this article no matter the OS.

\n

Non-blocking IO on UNIX: it sucks!

\n

Once we do receive a file descriptor from a sandbox, it’s time to operate\non it. However if we’re sloppy about it, our thread will block. We need to avoid\nblocking operations to dodge some DoS\nattempts. The mess around non-blocking IO on\nUNIX has been long known. Believe me when I say I have my own share of comments\nto make here, but this article isn’t about async IO and the text is already\ngetting too long, so I’ll just present a boring summary of what you need to\nknow:

\n\n

By now you should understand that there are actually two use cases:

\n

  • Trusted process creates a resource (file descriptor) and sends it to a

    distrusted sandboxed process.

  • The distrusted sandboxed process creates a resource and sends it elsewhere.
\n

If the file descriptor was created by a trusted process then the range of\noptions we can work with widens. However state such as O_NONBLOCK is shared\namong all copies of a file descriptor and become a problem the moment the file\ndescriptor reaches the first sandboxed process. On Capsicum we can at least\nforbid F_SETFL and alleviate the problem slightly, but this approach only\nworks for FreeBSD.

\n

Sandboxing existing code

\n

By now you should have enough tools in your toolbox to sandbox all your future\ncode. However there’s a reason why we sandbox code. Projects grow high and large\nuntil it becomes impossible to ensure their code is…​ bug-free. A little bug in\njust one of the dozens of modules within a project shouldn’t equal to a fully\ncompromised system when the bug is exploited by a hacker. Damage should be\ncontained. Security policies implemented by OS tools external to your code can\nonly work at a program (or user) level. The program will still be fully\ncompromised and the hacker will have access to all data and credentials the\nprogram has access to.

\n

When your program deals with data that can be processed independently, there’s\nan opportunity to implement a safer approach. If you can run multiple instances\nof your program as different users on your OS, you can use existing security\nsolutions in your project. However if the concept of allocating defined portions\nof the data to fixed users doesn’t work for your project, you may need something\nmore complex or custom-tailored. When the relationships between the users are\nblurry and your project demands policies that are more dynamic, you may need\nsandboxes.

\n

As a rule of thumb, every shell should have sandboxes. Shell are programs that\nact as the membrane that sits between the human operator and some virtual\nworld. Tablets, smartphones, and laptops display graphical shells to interact\nwith programs, windows, and files. Servers employ textual shells. Likewise web\nbrowsers act as the shells to the www world.

\n

I wouldn’t be surprised if Firefox and Chrome were the only software employing\ndiscretionary privilege dropping that you know. They are shells after all so it\nmatters to them. More than that, they’re very well funded projects. Sandboxing\nused to be very expensive (especially outside FreeBSD). However these shouldn’t\nbe the only software out there with builtin sandboxing support. Take Telegram,\nfor instance. The right media parsing bug could mean a hacker having access to\nall my chat history. What time does my son leave school? What people do I trust\nmy credit card info with? When will I go in a trip and leave my house\nunattended? These are just a few examples of the damage that might be done due\nto the lack of sandboxes in Telegram. Not only Telegram, but every instant\nmessenger should be employing sandboxes. Media parsing should always be\nperformed in dedicated sandboxes.

\n

The first step into this direction is a realistic approach to real-world\nengineering: let’s not rewrite all code from scratch. Deal? The tricks you\nlearned earlier will still be useful, but from now on I’ll share tricks to work\non existing real-world code. Capsicum users refer to the ability to run\nunmodified code within sandboxes as oblivious sandboxing. Techniques for\noblivious sandboxing most often than not have nothing to do with discretionary\nprivilege dropping and can’t solve the problems we were mentioning just a\nsecond ago. However it’s possible to combine approaches from both worlds in the\nsame project so it’s important to study the techniques for oblivious\nsandboxing too.

\n

The one place we need to look at to implement oblivious sandboxing is actually\npretty obvious: the ambient authority functions. In fact, that’s what projects\nsuch as Super Capsicumizer\n9000 do. They inject a dynamic library into a process using LD_PRELOAD to\ninterpose ambient authority calls. This technique is actually yesterday news and\nprojects such as fakeroot have been using it for decades.

\n

Super Capsicumizer 9000 demo: refusing to open /etc on gedit

\n

Super Capsicumizer 9000 is actually a small experiment hacked together by a very\nvery small team. The experiment succeeded into opening old software built on top\nof complex libraries with a long history of changes. This is very\npromising. It’s a sign that maybe a single programmer working alone to interpose\njust a few functions for ambient authority access will have success in running\nlegacy code.

\n

Programmers almost never do syscalls directly, and instead rely on libc to do\nthe syscalls on their behalf. That’s why this approach works so well. All you\nhave to do is to write a definition for the function from libc you want to\ninterpose. If you’re linking against the dynamic libc, your function will be\nloaded first and used instead. If you’re linking against the static libc,\nchances are that the libc symbol is actually a weak symbol so it’ll be dropped\nonce the static linker see your definition. Emilua has been using this approach\nto support dynamic and static executables on Linux and FreeBSD and so far\ngetaddrinfo was the only ambient authority function whose symbol lacked the\nattribute for weak symbols (please comment on the linked bug reports if you plan\nto build your own sandboxes using the same techniques or using Emilua):

\n\n

The next step is choosing which functions to interpose. Functions from FreeBSD’s\nlibcasper are good first candidates. However for some reason libcasper doesn’t\ninterpose the functions it intends to replace so you’ll need to change names and\nparameters accordingly. libcasper functions (e.g. cap_getaddrinfo) always take\nan extra parameter. Another good source of inspiration to decide which functions\nto interpose is the library used by Super Capsicumizer 9000: libpreopen. Most of\nthe time, you’ll only need to interpose a few functions even for complex\nprojects.

\n

Chromium renderers need very little authority. They need access to fontconfig to\nfind fonts on the system and to open those font files.

\n

— https://www.imperialviolet.org/2009/08/26/seccomp.html

\n

Emilua 0.11 abstracts all these details into the module libc_service. The\nexample below shows how we use this module to override the behavior of open to\nreturn a rogue file descriptor when the subprocess try to open\n/dev/null. Actual sandboxing setup (i.e. privilege dropping within the new\nsubprocess) is omitted for brevity. The example also shows how to prefill the\ncode cache for the new subprocess so it won’t query the filesystem to fetch the\nLua code to execute.

\n

local libc_service = require 'libc_service'\nlocal stream = require 'stream'\nlocal pipe = require 'pipe'\nlocal fs = require 'filesystem'

\n

local master, slave = libc_service.new()

\n

slave.open = [[\nlocal real_open, path, flag, mode = ...\nlocal res, errno, fd = real_open(path, flag, mode)\nif fd then\nreturn fd\nelse\nreturn res, errno\nend\n]]

\n

local source_tree_cache = {}\nsource_tree_cache['a.lua'] = [[\nlocal stream = require 'stream'\nlocal file = require 'file'\nlocal fs = require 'filesystem'

\n

local f = file.stream.new()\nf:open(fs.path.new('/dev/null'), {'read_only'})\nf = stream.scanner.new{ stream = f }\nprint(f:get_line())\n]]

\n

spawn_vm{\nmodule = fs.path.new('/a.lua'),\nsubprocess = {\nsource_tree_cache = source_tree_cache,\nlibc_service = slave,\nstdout = 'share',\nstderr = 'share',\n}\n}

\n

spawn(function() pcall(function()\nwhile true do\nmaster:receive()\nif master.function_ ~= 'open' then\nmaster:use_slave_credentials()\ngoto continue\nend\nlocal p, f, m = master:arguments()\nif p ~= fs.path.new('/dev/null') then\nmaster:use_slave_credentials()\ngoto continue\nend

\n

local pi, po = pipe.pair()\npi = pi:release()\nspawn(function()\nstream.write_all(po, '/dev/null contents\\n')\npo:close()\nend):detach()\nmaster:send_with_fds(-2, {pi})\n::continue::\nend\nend) end):detach()

\n

Emilua uses UNIX sockets behind the scenes for communication between both\nprocesses. This approach allows one to implement fully dynamic security\npolicies. For instance, if you’re trying use Telegram’s tdlib to implement your\nown Telegram client, you could have the following rules for your secutiry\npolicy:

\n
  • Only resolve name queries to pluto.web.telegram.org.
  • Only allow connect requests to the IP addresses we resolved in previous steps.
\n

The little Lua script we send to be executed in the sandboxed side means we can\napply some simple call fixups at the call site to further broaden the use cases\nwe can tackle. For instance, when sandboxed code try to open a GUI connecting to\n/tmp/.X11-unix/X0, we can send a new file descriptor to an unrelated display\nserver and replace the socket from the original request with the new one using\ndup2 from the Lua script at the call site. In fact, we can do that:

\n

local libc_service = require 'libc_service'\nlocal stream = require 'stream'\nlocal system = require 'system'\nlocal pipe = require 'pipe'\nlocal unix = require 'unix'\nlocal fs = require 'filesystem'

\n

local preload_libc_path\ndo\nlocal pi, po = pipe.pair()\npo = po:release()\npi = stream.scanner.new{ stream = pi }

\n

system.spawn{\nprogram = 'pkg-config',\narguments = {'pkg-config', '--variable=libpath', 'emilua_preload_libc'},\nenvironment = system.environment,\nstdout = po,\n}\npo:close()

\n

preload_libc_path = tostring(pi:get_line())\nend

\n

local xephyrconnep\nfor i = 1, 20 do\nlocal pi, po = pipe.pair()\npo = po:release()\npi = stream.scanner.new{ stream = pi }

\n

local xephyr = system.spawn{\nprogram = 'Xephyr',\narguments = { 'Xephyr', ':' .. i, '-displayfd', '3' },\nenvironment = system.environment,\nextra_fds = {\n[3] = po,\n},\n}\npo:close()

\n

if pcall(function()\nlocal nr = tostring(pi:get_line())\nxephyrconnep = '/tmp/.X11-unix/X' .. nr\nreturn true\nend) then\nbreak\nend\nend\nif not xephyrconnep then\nprint('Failed to start Xephyr')\nsystem.exit(1)\nend

\n

local master, slave = libc_service.new()

\n

slave.connect_unix = [[\nlocal real_connect, fd, path = ...\nlocal res, errno, fd2 = real_connect(fd, path)\nif fd2 then\nC.dup2(fd2, fd)\nC.close(fd2)\nend\nreturn res, errno\n]]

\n

local guiappenv = system.environment\nguiappenv.DISPLAY = ':0'\nguiappenv.LD_PRELOAD = preload_libc_path\nguiappenv.EMILUA_LIBC_SERVICE_FD = '3'\nlocal guiapp = system.spawn{\nprogram = 'xterm',\narguments = { 'xterm' },\nenvironment = guiappenv,\nstdout = 'share',\nstderr = 'share',\nextra_fds = {\n[3] = slave,\n},\n}\nscope_cleanup_push(function() guiapp:wait() end)

\n

spawn(function() pcall(function()\nwhile true do\nmaster:receive()\nif\nmaster.function_ == 'connect_unix' and\n(\nmaster:arguments() == fs.path.new('\\0/tmp/.X11-unix/X0') or\nmaster:arguments() == fs.path.new('/tmp/.X11-unix/X0')\n)\nthen\nlocal xephyrconn = unix.stream.dial(xephyrconnep)\nmaster:send_with_fds(0, {xephyrconn:release()})\nelse\nmaster:use_slave_credentials()\nend\nend\nend) end):detach()

\n

This example also shows that Emilua can make use of LD_PRELOAD to perform libc\ninterposition on existing programs such as xterm.

\n

Another interesting approach that might prove useful to your projects is to use\nkcmp in your security policies. This technique would allow you to increase\nyour policies granularities even further by implementing different subpolicies\nfor each file descriptor.

\n

By the way, now we can interpose openat() on Linux to make it work as in\nFreeBSD’s capability mode (but — as usual — don’t forget to forbid the actual\nsyscall):

\n

local libc_service = require 'libc_service'

\n

local master, slave = libc_service.new()

\n

slave.openat = [[\nlocal real_openat, dirfd, path, flags, mode, resolve = ...\nlocal res, errno, fd = real_openat(dirfd, path, flags, mode, resolve)\nif fd then\nreturn fd\nelse\nreturn res, errno\nend\n]]

\n

local worker = spawn_vm{\nmodule = 'module4',\nsubprocess = {\nlibc_service = slave,\n},\n}

\n

pcall(function()\nwhile true do\nmaster:receive()\nif master.function_ == 'openat' then\nlocal path, flags, mode = master:arguments()\nflags[#flags + 1] = 'resolve_beneath'\nlocal dirfd = master:descriptors()\nlocal ok, res = pcall(function()\nreturn dirfd:openat(path, flags, mode)\nend)\nif ok then\nmaster:send_with_fds(-1, {res})\nelse\nmaster:send(-1, res)\nend\nelse\nmaster:use_slave_credentials()\nend\nend\nend)

\n

These techniques are already probably more than what you need, but I have few\nmore tricks up in my sleeve to share, so let’s move on.

\n

Sandboxing native plugins

\n

A common theme in the threat model of sandboxes is to assume that initially\ntrusted code becomes malicious once compromised. For instance, we might trust\nthat ffmpeg developers are well-intentioned and didn’t backdoored their\nproject. However ffmpeg is a complex project and legitimate bugs lurk around\njust waiting to be found. Some of these bugs might be exploitable by hackers. In\nthis case we can import ffmpeg as a library in our executable and only setup the\nsandbox right before we call ffmpeg functions on external data.

\n

However how’d we approach the sandboxing steps if we assumed the code to be\ncompromised from the start? Let’s take Telegram’s tdlib as an example. Suppose\nyou don’t trust tdlib at all. Under this threat model, even just loading the\nlibrary would be a dangerous operation. For such scenario, first we need to\nbuild tdlib under a secure environment (e.g. jails on FreeBSD, namespaces on\nLinux). Once we do have the built plugin, we can move to the next challenge.

\n

If we disable ambient authority to sandbox the code, dlopen() will fail to\naccess the filesystem. To work around this issue, we can dlopen by file\ndescriptors instead. On FreeBSD, we can use fdlopen. On Linux, we can pass a\npath to /proc/self/fd/ as long as we never close the file descriptor to avoid\npath reuse by a different plugin (glibc will deduplicate plugins by\npath). That’s not a perfect solution, but it’s a start.

\n

On some previous example, we saw code cache prefilling as an Emilua way to\ninstruct a policy to avoid filesystem queries. Emilua just follows the same\ntrend for plugins and exposes native_modules_cache to prefill the native\nplugin cache:

\n

spawn_vm{\nmodule = 'some_module',\nsubprocess = {\nnative_modules_cache = {\n'some_plugin'\n} } }

\n

The biggest problem with plugins is that they might depend on not yet loaded\ndynamic libraries and dlopen would fail to load them. We can try to use the\nworkaround for libc-service that we saw in the previous section, but there are\ncleaner solutions for this problem. On Linux, we can simply use\nLandlock. Landlock would already be required for the /proc/self/fd/ trick\nanyway. On FreeBSD, we can use rtld_set_var\nand LIBRARY_PATH_FDS.

\n

spawn_vm{\nmodule = 'some_module',\nsubprocess = {\nnative_modules_cache = {\n'some_plugin'\n},\nld_library_directories = library_path_fds,\n} }

\n

Fortunately we won’t need to worry about this for Tdlib so the code becomes\nslightly simpler. However it begs the question: if we don’t trust tdlib, why are\nwe trusting it with our data anyway? The question here is to evaluate if the\nthreat model even makes sense. In the case of tdlib, we aren’t giving tdlib’s\ndevelopers (Telegram developers) anything they don’t already have (data on\nTelegram servers). Running tdlib within a plugin prevents the Telegram company\nto have unrestricted access to data on our systems. The answer for tdlib might\nbe simple, but that’s not what matters in this story. The lesson that you should\ntake here is to evaluate whether your threat models even make sense.

\n

Let’s explore another threat model story. The Linux kernel can load compressed\ninitramfs images. Even if the Linux kernel uses a buggy unmaintained library\nfull of well known exploits to decompress the initramfs image, it doesn’t matter\nat all! We only use this library on data generated by a trusted user. If some\nadversarial actor had control over the initramfs images we load, the actor could\nalready do any damage he wished for even if the decompression library had zero\nexploitable bugs. However the story would be completely different if the library\nwere backdoored. Sometimes it’s more important to have auditable code written by\ntrusted individuals than supposedly better code written by individuals we aren’t\nsure we can trust.

\n

Dropping privileges with Seccomp

\n

I’ve postponed this section for as long as I could because it’s awful. Seccomp\nis not a good mechanism for discretionary privilege dropping. Seccomp is a good\nmechanism for OS hardening. Seccomp is a simple programmable syscall filtering\nmechanism based on BPF programs. The BPF program must choose an action for each\nsyscall attempted by the process:

\n
  • SECCOMP_RET_KILL_PROCESS.
  • SECCOMP_RET_KILL_THREAD.
  • SECCOMP_RET_TRAP.
  • SECCOMP_RET_ERRNO.
  • SECCOMP_RET_USER_NOTIF.
  • SECCOMP_RET_TRACE.
  • SECCOMP_RET_LOG.
  • SECCOMP_RET_ALLOW.
\n

This mechanism can be used to deny access (e.g. SECCOMP_RET_ERRNO) to ambient\nauthority by disallowing syscalls that work on names (e.g. open, bind). All\nyou have to do is disable ambient authority, then your process will be properly\nsandboxed and you can use the lessons learned in previous sections for\ncompartmentalised application development. Using this mechanism you may either\nimplement whitelists or blacklists. These projects implement syscall blacklists:

\n\n

The problem with blacklists is well known. New kernel versions may add new\nsyscalls. You don’t know today what syscalls will be added tomorrow. You don’t\nknow today if tomorrow’s syscall breaks your today’s policy. However the problem\nis even worse for\nseccomp. Linux\nallows multiarch systems and syscall numbering varies wildly among arches. Even\nif you block syscalls such as acct on your x86-64 program, a compromised\nsandbox could bypass the syscall filter by running a x86 executable. To our\ndelight, Linux somehow manages to make the problem even worse:

\n

The arch field is not unique for all calling conventions. The x86-64 ABI and\nthe x32 ABI both use AUDIT_ARCH_X86_64 as arch, and they run on the same\nprocessors. Instead, the mask __X32_SYSCALL_BIT is used on the system call\nnumber to tell the two ABIs apart.

\n

This means that a policy must either deny all syscalls with X32_SYSCALL_BIT\nor it must recognize syscalls with and without X32_SYSCALL_BIT set. A list\nof system calls to be denied based on nr that does not also contain nr\nvalues with X32_SYSCALL_BIT set can be bypassed by a malicious program that\nsets X32_SYSCALL_BIT.

\n

— seccomp(2)

\n

Even when we do migrate to whitelists, we’ll still be haunted by these\nimplementation details. Here are a few notable projects based on whitelists:

\n\n

Once you do go through these lists to implement your own seccomp policies,\nyou’ll face another problem: Linux syscall parameter ordering changes among\narches. That’s why Docker uses multiple rules for the syscall clone. Why do we\nhave to become experts in Linux syscall conventions to do what FreeBSD users get\ndone in 10 seconds by just calling cap_enter()? Welcome to Linux.

\n

The good news is that in theory we could implement some library with a single\nfunction to disable ambient authority and everyone would then just use this\nlibrary. I have a few ideas on how to implement this library, but so far no\ncustomer of mine was interested in this problem (at the same time I’m busy with\ndifferent projects).

\n

However there’s just another caveat you must keep in mind. Linux userspace\nrelies on the filesystem too much. Even if you interpose open for\ncompatibility with legacy code, legacy code will fail trying to access\n/proc/self when nested sandboxes are at play. It’s better to just allow open\nand filter filesystem access with Landlock instead. The problem now is that file\ndescriptors can’t be modeled as capabilities if a process can just reopen\n/proc/self/fd/ with a different mode. Landlock developers shared a long-term\ngoal to expose capabilities compatible with Capsicum, so maybe we’ll have a\nsolution to this problem in the future.

\n

Seccomp’s complexity might be disheartening for the compartmentalised\napplication developer, but the situation is even worse if you look into Linux\nnamespaces. For Linux, Seccomp + Landlock is what we have.

\n

Easier Seccomp with Kafel

\n

Kafel is a language and library for specifying syscall filtering policies. The\npolicies are compiled into BPF code that can be used with seccomp-filter.

\n

— https://google.github.io/kafel/

\n

Kafel is the most promising project to enable Seccomp policy reusing that I’ve\nseen during my quests. Using Kafel I’ve managed to define a few policy groups\nthat might be of interest to you:

\n

Kafel policies

\n

// These policies are heavily influenced by Docker's default profile. Further\n// customization done on top:\n//\n// - Avoid syscalls that need root anyway. The policies here are mostly meant to\n// be used by unprivileged users (not containers with root inside). The\n// syscalls wouldn't be harmful, but would result in larger BPF programs that\n// in turn incur more overhead.\n// - Avoid rarely used syscalls that can be abused for yet more fingerprinting\n// on desktop applications. This category mostly contains syscalls useful for\n// profiling (e.g. mincore, cachestat).\n// - Split them into categories inspired by systemD's seccomp filter sets and\n// OpenBSD's pledge promises.

\n

POLICY Aio {\nALLOW {\nio_cancel, io_destroy, io_getevents, io_pgetevents, io_setup, io_submit\n}\n}

\n

POLICY BasicIo {\nALLOW {\nread, readv, tee, vmsplice, write, writev,

\n

// ioctl() is definitively not about generic/stream/basic I/O. ioctl()\n// is really a syscall in disguise that device drivers can use for\n// anything. However it's expected that any program doing file I/O or\n// socket I/O or TTY IO will eventually stumble on glibc using ioctl()\n// for some operations so let's go ahead and just include it in the\n// basic IO set to force other IO categories to include it too.\nioctl\n}\n}

\n

POLICY Clock {\nALLOW {\nclock_getres, clock_gettime, gettimeofday, time, times\n}\n}

\n

// Compat quirks. This family of policies is a good candidate to be maintained\n// in a different repo.\nPOLICY CompatX86 {\nALLOW {\n// important for old ABI emulation\npersonality(persona) {\npersona == /PER_LINUX=/0 || persona == /PER_LINUX32=/8 ||\npersona == /UNAME26=/0x0020000 ||\npersona == /PER_LINUX32|UNAME26=/0x20008 ||\npersona == 0xffffffff\n},

\n

// Important for x86 family's ABI. We put it in here instead of\n// c-runtime because other archs don't need it. Ideally Kafel would\n// allow us to write arch_prctl@amd64 in c-runtime and the rule would\n// only be included when we're building for the amd64 arch.\narch_prctl\n}\n}

\n

POLICY CompatDB32 {\nALLOW {\nremap_file_pages\n}\n}

\n

POLICY CompatSystemd {\nALLOW {\n// SystemD uses this to get mount-id\nname_to_handle_at\n}\n}

\n

POLICY CompatWine {\nALLOW {\nmodify_ldt\n}\n}

\n

POLICY Credentials {\nALLOW {\ngetegid, geteuid, getgid, getgroups, getresgid, getresuid, getuid\n}\n}

\n

POLICY CredentialsExtra {\nALLOW {\n// SystemD lists this syscall in the policy 'process' with the reasoning\n// that it's able to query arbitrary processes so it's a process\n// relationship related syscall. Following the same reasoning, we opt to\n// not include this syscall in the policy 'credentials' as other\n// syscalls in that category don't allow querying arbitrary\n// processes. However we also opt to not include capget in the category\n// 'process' given most usages of that policy won't need capget at all\n// and would just make the resulting BPF bigger.\ncapget\n}\n}

\n

POLICY CredentialsMutation {\nALLOW {\ncapset, setfsgid, setfsuid, setgid, setgroups, setregid, setresgid,\nsetresuid, setreuid, setuid\n}\n}

\n

// Memory allocation, threading, syscall interaction (or libc support) and\n// functions that should always be available (e.g. exit_group to bail out as a\n// program's last resort).\n//\n// Do notice that actually opening a libc-based program requires access to much\n// more syscalls as the loader is going to scrape the filesystem for the\n// required libraries and do many operations to stich the program image\n// together. The idea here is to apply a filter that will allow the C runtime to\n// keep running after we already have the program image in RAM.\nPOLICY CRuntime {\nALLOW {\nbrk, exit, exit_group, futex, futex_requeue, futex_wait, futex_waitv,\nfutex_wake, get_robust_list, get_thread_area, gettid, madvise,\nmap_shadow_stack, membarrier, mmap, mprotect, mremap, munmap,\nrestart_syscall, rseq, sched_yield, set_robust_list, set_thread_area,\nset_tid_address,

\n

// glibc's malloc() has references to getrandom(), so it's included here\ngetrandom\n}\n}

\n

// These syscalls are already gated by YAMA's ptrace_scope or capabilities\n// (e.g. CAP_PERFMON). The usual reasoning would be that it's safe to permit\n// them, but:\n//\n// - They are really only useful for process inspection/debugging.\n// - For IPC usage, better mechanisms exist (e.g. one can memfd+seal+mmap to\n// have zero copy I/O between cooperating processes).\n// - They appeared in a few CVEs in the past.\nPOLICY Debug {\nALLOW {\nkcmp, pidfd_getfd, perf_event_open, process_madvise, process_mrelease,\nprocess_vm_readv, process_vm_writev, ptrace\n}\n}

\n

POLICY FileDescriptors {\nALLOW {\nclose, close_range, dup, dup2, dup3, fcntl\n}\n}

\n

// This policy is split off from filesystem so a process could still perform\n// file IO on:\n//\n// - Already open files.\n// - Files received from UNIX sockets.\n// - Memfds.\nPOLICY FileIo {\nALLOW {\ncopy_file_range, fadvise64, fallocate, flock, ftruncate, lseek, pread64,\npreadv, preadv2, pwrite64, pwritev, pwritev2, readahead, sendfile,\nsplice\n}\n}

\n

// OpenBSD's pledge further breaks down this promise into rpath, wpath, cpath\n// and dpath, but Landlock would be more appropriate to mirror the intention of\n// such granular designs\nPOLICY Filesystem {\nALLOW {\naccess, chdir, creat, faccessat, faccessat2, fchdir, fgetxattr,\nflistxattr, fstat, fstatfs, getcwd, getdents, getdents64, getxattr,\ninotify_add_watch, inotify_init, inotify_init1, inotify_rm_watch,\nlgetxattr, link, linkat, listxattr, llistxattr, lstat, mkdir, mkdirat,\nmknod, mknodat, newfstatat, open, openat, openat2, readlink, readlinkat,\nrename, renameat, renameat2, rmdir, stat, statfs, statx, symlink,\nsymlinkat, truncate, umask, unlink, unlinkat\n}\n}

\n

// Allowed to make explicit changes to fields in struct stat relating to a file.\nPOLICY FilesystemAttr {\nALLOW {\nchmod, chown, fchmod, fchmodat, fchmodat2, fchown, fchownat,\nfremovexattr, fsetxattr, futimesat, lchown, lremovexattr, lsetxattr,\nremovexattr, setxattr, utime, utimensat, utimes\n}\n}

\n

// Event loop system calls.\nPOLICY IoEvent {\nALLOW {\nepoll_create, epoll_create1, epoll_ctl, epoll_ctl_old, epoll_pwait,\nepoll_pwait2, epoll_wait, epoll_wait_old, eventfd, eventfd2, poll,\nppoll, pselect6, select\n}\n}

\n

// io_uring nowadays is considered unsafe for general usage:\n// http://security.googleblog.com/2023/06/learnings-from-kctf-vrps-42-linux.html\nPOLICY IoUring {\nALLOW {\nio_uring_enter, io_uring_register, io_uring_setup\n}\n}

\n

// SysV IPC, POSIX Message Queues or other IPC.\nPOLICY Ipc {\nALLOW {\nmemfd_create, mq_getsetattr, mq_notify, mq_open, mq_timedreceive,\nmq_timedsend, mq_unlink, msgctl, msgget, msgrcv, msgsnd, pipe, pipe2,\nsemctl, semget, semop, semtimedop, shmat, shmctl, shmdt, shmget\n}\n}

\n

// Memory locking control.\nPOLICY Memlock {\nALLOW {\nmemfd_secret, mlock, mlock2, mlockall, munlock, munlockall\n}\n}

\n

POLICY NetworkIo {\nALLOW {\nconnect, getpeername, getsockname, getsockopt, recvfrom, recvmmsg,\nrecvmsg, sendmmsg, sendmsg, sendto, setsockopt, shutdown\n}\n}

\n

POLICY NetworkServer {\nALLOW {\naccept, accept4, bind, listen\n}\n}

\n

POLICY NetworkSocketTcp {\nALLOW {\nsocket(domain, type, protocol) {\n(type & 0x7ff) == /SOCK_STREAM=/1 && protocol == 0 &&\n(domain == /AF_INET=/2 || domain == /AF_INET6=/10)\n}\n}\n}

\n

POLICY NetworkSocketUdp {\nALLOW {\nsocket(domain, type, protocol) {\n(type & 0x7ff) == /SOCK_DGRAM=/2 && protocol == 0 &&\n(domain == /AF_INET=/2 || domain == /AF_INET6=/10)\n}\n}\n}

\n

POLICY NetworkSocketUnix {\nALLOW {\nsocket(domain, type, protocol) {\ndomain == /AF_UNIX=/1 && protocol == 0\n},\nsocketpair(domain, type, protocol) {\ndomain == /AF_UNIX=/1 && protocol == 0\n}\n}\n}

\n

// System calls used for memory protection keys.\nPOLICY Pkey {\nALLOW {\npkey_alloc, pkey_free, pkey_mprotect\n}\n}

\n

// Process control, execution, namespacing, relationship operations.\n//\n// Most likely you'll ALWAYS need access to this set to sandbox other binaries:\n// https://lore.kernel.org/all/202010281500.855B950FE@keescook/T/. It's only\n// really practical to exclude this set from the seccomp filter if you're\n// sandboxing yourself (i.e. cooperatively dropping further privileges before\n// doing dangerous stuff). It's a shame that Linux doesn't offer this type of\n// transition-on-exec mechanism for seccomp nor cgroups. Folks from SELinux\n// already know just how important it is to support this kind of mechanism for\n// properly dropping privileges, and it'd be good for more kernel hackers to\n// learn this lesson as well.\nPOLICY Process {\nALLOW {\n// Where's clone2? ia64 is the only architecture that has clone2, but\n// ia64 doesn't implement seccomp. c.f.\n// acce2f71779c54086962fefce3833d886c655f62 in the kernel.\nclone, clone3, execve, execveat, fork, getpgid, getpgrp, getpid,\ngetppid, getrusage, getsid, kill, pidfd_open, pidfd_send_signal, prctl,\nrt_sigqueueinfo, rt_tgsigqueueinfo, setpgid, setsid, tgkill, tkill,\nvfork, wait4, waitid\n}\n}

\n

POLICY Resources {\nALLOW {\ngetcpu, getpriority, getrlimit, ioprio_get, sched_getaffinity,\nsched_getattr, sched_getparam, sched_get_priority_max,\nsched_get_priority_min, sched_getscheduler, sched_rr_get_interval\n}\n}

\n

// Alter resource settings.\nPOLICY ResourcesMutation {\nALLOW {\nioprio_set, prlimit64, sched_setaffinity, sched_setattr, sched_setparam,\nsched_setscheduler, setpriority, setrlimit\n}\n}

\n

POLICY Sandbox {\nALLOW {\nlandlock_add_rule, landlock_create_ruleset, landlock_restrict_self,\nseccomp\n}\n}

\n

// Process signal handling.\nPOLICY Signal {\nALLOW {\npause, rt_sigaction, rt_sigpending, rt_sigprocmask, rt_sigreturn,\nrt_sigsuspend, rt_sigtimedwait, sigaltstack, signalfd, signalfd4\n}\n}

\n

// Synchronize files and memory to storage.\nPOLICY Sync {\nALLOW {\nfdatasync, fsync, msync, sync, sync_file_range, syncfs\n}\n}

\n

// Schedule operations by time.\nPOLICY Timer {\nALLOW {\nalarm, getitimer, clock_nanosleep, nanosleep, setitimer, timer_create,\ntimer_delete, timer_getoverrun, timer_gettime, timer_settime,\ntimerfd_create, timerfd_gettime, timerfd_settime\n}\n}

\n

These are the policies I’ve been using for almost a year. I’d make a few changes\nnowadays, but I haven’t gotten the time to it yet. Also do keep in mind that\nKafel’s syscall database is inexcusably poor so you’ll need a few changes (some\nsed-like preprocessing to replace the syscall names by their numbers) to make\nthe above work. Even when it does work, it’ll be omitting many syscalls that\nprojects using better syscall databases such as Docker handle. Did you notice\nthat we don’t mention clock_gettime64? That’s because Kafel’s syscall database\nis just too poor. Kafel won’t ever be adopted by projects such as Docker for as\nlong as it retains such poor syscall databases and poor multiarch support.

\n

However a lack of better syscall databases isn’t the only thing we could improve\nin Kafel. I’d like to see policy versioning and better policy composition\noperators, for instance. I’d be willing to develop these features, but again, no\ncurrent customer of mine was interested in this project, so my time will be\nspent in different projects.

\n

The end?

\n

In this article, I’ve gone just over the basics for sandboxing in Linux and\nFreeBSD. However there are more lessons that I’d like to share. I’ll save these\nfor a later article in the future. Topics that I’ll likely cover if I ever get\ninto the mood to write another blog post again:

\n

  • More demos (for years I’ve been running graphical apps in some of my machines

    solely in containers so I got quite a few demos to show).

  • More about the actor model, capability-based security, and access control

    policies.

  • More about Linux kludges.
  • Maybe a comment or two on Windows and macOS.
  • PR_SET_DUMPABLE & /proc/sys/kernel/yama/ptrace_scope.
  • Containers (Emilua also works as a container runtime).
  • Sandboxing GUI applications.
  • Attack surfaces & safe parsers.
  • FreeBSD’s libnv.
  • Rights revocation & proxies.
  • Sandboxing patterns.
  • UNIX tricks for the C programmer that Emilua makes use of.
\n

© 2024 Emilua

","headings":[{"level":1,"text":"Software sandboxing: The basics","id":"software-sandboxing-the-basics"},{"level":2,"text":"Programmatic privilege dropping","id":"programmatic-privilege-dropping"},{"level":2,"text":"Dropping privileges without root","id":"dropping-privileges-without-root"},{"level":2,"text":"Discretionary privilege dropping","id":"discretionary-privilege-dropping"},{"level":2,"text":"Practical sandboxing: processes","id":"practical-sandboxing-processes"},{"level":2,"text":"The actor model and capability-based security","id":"the-actor-model-and-capability-based-security"},{"level":2,"text":"File descriptors as capabilities","id":"file-descriptors-as-capabilities"},{"level":1,"text":"grep -Ee '^nobody:'