Factored Operating Systems (fos): The Case for a Scalable
Operating System for Multicores
David Wentzlaff and Anant Agarwal
Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology
Cambridge, MA 02139
{wentzlaf, agarwal}@csail.mit.edu
Abstract
The next decade will afford us computer chips with 100’s to 1,000’s
of cores on a single piece of silicon. Contemporary operating sys-
tems have been designed to operate on a single core or small num-
ber of cores and hence are not well suited to manage and provide
operating system services at such large scale. If multicore trends
continue, the number of cores that an operating system will be
managing will continue to double every 18 months. The traditional
evolutionary approach of redesigning OS subsystems when there
is insufficient parallelism will cease to work because the rate of
increasing parallelism will far outpace the rate at which OS design-
ers will be capable of redesigning subsystems. The fundamental
design of operating systems and operating system data structures
must be rethought to put scalability as the prime design constraint.
This work begins by documenting the scalability problems of con-
temporary operating systems. These studies are used to motivate
the design of a factored operating system (fos). fos is a new operat-
ing system targeting manycore systems with scalability as the pri-
mary design constraint, where space sharing replaces time sharing
to increase scalability. We describe fos, which is built in a message
passing manner, out of a collection of Internet inspired services.
Each operating system service is factored into a set of commu-
nicating servers which in aggregate implement a system service.
These servers are designed much in the way that distributed Inter-
net services are designed, but instead of providing high level Inter-
net services, these servers provide traditional kernel services and
replace traditional kernel data structures in a factored, spatially dis-
tributed manner. fos replaces time sharing with space sharing. In
other words, fos’s servers are bound to distinct processing cores
and by doing so do not fight with end user applications for im-
plicit resources such as TLBs and caches. We describe how fos’s
design is well suited to attack the scalability challenge of future
multicores and discuss how traditional application-operating sys-
tems interfaces can be redesigned to improve scalability.
Categories and Subject Descriptors D.4.7 [Operating Systems]:
Organization and Design
General Terms Operating System Design, Multicore Computers
Keywords Multicore Operating Systems, Factored Operating
System
1. Introduction
The number of processor cores which fit onto a single chip micro-
processor is rapidly increasing. Within ten years, a single micropro-
cessor will contain 100’s - 1,000’s cores. Current operating systems
were designed for single processor or small number of processor
systems and were not designed to manage such scale of compu-
tational resources. Unlike the past, where new hardware genera-
tions brought higher clock frequency, larger caches, and more sin-
gle stream speculation, all of which are not huge changes to fun-
damental system organization, the multicore revolution promises
drastic changes in fundamental system architecture, primarily in
the fact that the number of general-purpose schedulable processing
elements is drastically increasing. The way that an operating sys-
tem manages 1,000 processors is so fundamentally different than
the manner in which it manages one or two processors that the en-
tire design of an operating system must be rethought. This work
investigates why simply scaling up traditional symmetric multipro-
cessor operating systems is not sufficient to attack this problem and
proposes how to build a factored operating system (fos) which em-
braces the 1,000 core multicore chip opportunity.
The growing ubiquity of multicore processors is being driven by
several factors. If single stream microprocessor performance were
to continue increasing exponentially, there would be little need
to contemplate parallelization of our computing systems. Unfortu-
nately, single stream performance of microprocessors has fallen off
the exponential trend due to the inability to detect and exploit paral-
lelism in sequential codes, the inability to further pipeline sequen-
tial processors, the inability to raise clock frequencies due to power
constraints, and the design complexity of high-performance sin-
gle stream microprocessors[4]. While single stream performance
may not be significantly increasing in the future, the opportunity
provided by semiconductor process scaling is continuing for the
foreseeable future. The ITRS road-map[1] and the continuation
of Moore’s Law[16] forecast exponential increases in the number
of transistors on a single microprocessor chip for at least another
decade. In order to turn these exponentially increasing transistor re-
sources into exponentially increasing performance, microprocessor
manufacturers have turned to integrating multiple processors onto a
single die. Current examples of this include Intel and AMD’s Quad-
core offerings, TI’s TMS320C6474 triple core offering, Freescale’s
6-core MSC8156, Tilera’s 64-core processor[25], and an 80-core
Intel prototype processor[22]. Road-maps by all major micropro-
cessor manufacturers suggest that the trend of integrating more
cores onto a single microprocessor will continue. Extrapolating the
doubling of transistor resources every 18-months, and that a 64-
core commercial processor was shipped in 2007, in just ten years,
we will be able to integrate over 6,000 processor cores on a single
microprocessor.
The fact that single stream performance has sizably increased
with past generations of microprocessors has shielded operating
system developers from qualitative hardware platform changes.
Unlike quantitative changes such as larger caches, larger TLBs,
higher clock frequency, and more instruction level parallelism, the
76

multicore phenomenon is a qualitative change which drastically
changes the playing field for operating system design. The primary
challenge of multicore operating system design is one of scalability.
Current symmetric multiprocessor (SMP) operating systems have
been designed to manage a relatively small number of cores. The
number of cores being managed has stayed relatively constant with
the vast majority of SMP systems being two processor systems.
With multicore chip designs, the number of cores will be expanding
at an exponential rate therefore any operating system designed to
run on multicores will need to embrace scalability and make it a
first order design constraint.
This work investigates the problems with scaling SMP OS’s to
high core counts. The first problem is that scaling SMP OS’s by
creating successively finer grain data structure locks is becoming
problematic. Unlike small core count systems, where only a small
portion of the code may need fine grain locking, in high core
count systems, any non-scalable portion of the design will quickly
become a performance problem by Ahmdal’s law. Also, in order
to build an OS which performs well on 100 and 10,000 cores, there
may be no optimal lock granularity as finer grain locking allows for
better scaling, but introduces potential lock overhead on small core
count machines. Last, retrofitting fine grain locking into an SMP
OS can be an error prone and challenging prospect.
A second challenge SMP OS’s face is that they rely on efficient
cache coherence for communications of data structures and locks.
It is doubtful that future multicore processors will have efficient
full-machine cache coherence as the abstraction of a global shared
memory space is inherently a global shared structure. Another
challenge for any scalable OS is the need to manage locality. Last,
the design of SMP OS’s traditionally execute the operating system
across the whole machine. While this has good locality benefits for
application and OS communications, it requires the cache system
on each core of a multicore system to contain the working set of
the application and OS.
This work utilizes the Linux 2.6 kernel as a vehicle to investi-
gate scaling of a prototypical SMP OS. We perform scaling studies
of the physical page allocation routines to see how varying core
count affects the performance of this parallelized code. We find
that the physical page allocator does not scale beyond 8 cores un-
der heavy load. We also study the cache performance interference
when operating system and application code are executed on the
same core.
We use these scalability studies to motivate the design of a fac-
tored operating system (fos). fos is a new scalable operating system
targeted at 1000+ core systems. The main feature of fos is that it
factors an OS into a set of services where each service is built to
resemble a distributed Internet server. Each system service is com-
posed of multiple server processes which are spatially distributed
across a multicore chip. These servers collaborate and exchange in-
formation, and in aggregate provide the overall system service. In
fos, each server is allocated to a specific core thereby removing the
need to time-multiplex processor cores and simplifying the design
of each service server.
fos not only distributes high-level services, but also, distributes
services and data-structures typically only found deep in OS ker-
nels such as physical page allocation, scheduling, memory manage-
ment, naming, and hardware multiplexing. Each system service is
constructed out of collaborating servers. The system service servers
execute on top of a microkernel. The fos-microkernel is platform
dependent, provides protection mechanisms but not protection pol-
icy, and implements a fast machine-dependent communication in-
frastructure.
Many of fos’s fundamental system services embrace the dis-
tributed Internet paradigm even further by allowing any core to con-
tact any server in a particular system service group. This is similar
to how a web client can access any webserver in a load balanced
web cluster, but for kernel data structures and services. Also, like
a spatially load balanced web cluster, the fos approach exploits lo-
cality by distributing servers spatially across a multicore. When an
application needs a system service, it only needs to communicate
with its local server thereby exploiting locality.
Implementing a kernel as a distributed set of servers has many
advantages. First, by breaking away from the SMP OS monolithic
kernel approach, the OS level communication is made explicit and
exposed thus removing the problem of hunting for poor perform-
ing shared memory or lock based code. Second, in fos, the number
of servers implementing a particular system service scales with the
number of cores being executed on, thus the computing available
for OS needs scales with the number of cores in the system. Third,
fos does not execute OS code on the same cores which are execut-
ing application code. Instead an application messages the particular
system service, which then executes the OS code and returns the
result. By partitioning where the OS and applications execute, the
working set of the OS and the working set of the application do not
interfere.
Another impediment to the scalability of modern day operating
systems is that the operating system interfaces provided to applica-
tions are inherently non-scalable. One way to increase scalability is
to provide interfaces which supply information to the user in a best
effort manner. For example, by allowing best-effort information, it
empowers the user to request information in a high performance
manner which may be slightly out of date. Alternatively, the user
can select the lower performing, but accurate interface.
fos’s design is inspired by microkernels such as Mach [2],
L3, and L4 [15] but has significant differences. These differences
include:
• fos distributes and parallelizes within a single system service
server.
• fos embraces the spatial nature of multicore processors by hav-
ing a spatially aware placement engine/scheduler.
• Because applications and OS execute on different cores, fos
does not need to take an expensive context switch when an
application messages a OS server.
Section 4 provides more detail on related research.
We currently have a prototype implementation of the fos-
microkernel. It runs on 16 core x86 64 hardware and a QEMU
simulation of up to 255 processors. The operating system is under
active development and currently consists of a bootloader and a
microkernel with a messaging layer. We now developing system
service servers.
This paper is organized as follows. Section 2 identifies the
scalability problems with contemporary SMP operating systems.
Section 3 describes the design of fos and how its design attacks
scalability problems. Section 4 describes related work and finally
we conclude
2. Scalability Problems of Contemporary
Operating Systems
This section investigates three main scalability problems with con-
temporary OS design, locks, locality aliasing and reliance on shared
memory. Case studies are utilized to illustrate how each of these
problems appears in a contemporary OS, Linux, on modern multi-
core x86 64 hardware. The results of these studies are utilized to
make recommendations for future operating systems.
2.1 Locks
Contemporary operating systems which execute on multi-processor
systems have evolved from uni-processor operating systems. The
77