🔗What is compute?

While storage and network are fairly intuitive concepts – where data lives and how it gets around – compute is about the more mysterious realm of data processing and ultimately computer programming. Don’t worry though, you don’t need any technical abilities to gain a deeper understanding of what’s happening behind the scenes when we use apps or interact with web sites.

As mentioned in the first part of this series, compute is basically what happens with data once it reaches its destination. Compute gives data relevance, by processing it according to the instructions provided by developers in their source code. When talking about what the ThreeFold Grid offers in terms of compute, it’s mostly about how the code that developers produce gets executed in order to provide the services we enjoy.

With ThreeFold, this actually happens for the most part using the same tools and systems provided by traditional cloud services, meaning that it’s highly compatible with existing cloud applications. What really makes the Grid different is that it’s been designed to provide a level of efficiency and security that simply isn’t possible without starting “from the ground up”. Before we get into specifics of compute offerings built into ThreeFold’s Zero-OS, let’s briefly explore what computing is.

🔗Playing with zeroes and ones

Everything a computer’s central processing unit does is based on a limited set of instructions that includes basic mathematical functions, logical operations, and ways to control the flow of a program. At their core, computers are really just automated adding machines that execute specified operations on whatever input they receive. It may be hard to believe that simple math creates all of the vibrant digital experience we enjoy, but at a certain low level, that’s the truth.

Take a messaging app as an example. Within a computer, all of the text, emojis, and images we see are represented by numbers. When we search for some text within a chat, matches are determined by checking if the numbers corresponding to the words we typed are equal to some numbers corresponding to words that appear in the chat history. This information is then converted into the list of results we see on screen.

In this context, compute is mostly responsible for organizing all of our messages based on the individual or group we were talking with, the time that the message happened, and whether or not we’ve opened it yet. Typically, a messaging app stores data both on our local devices and on a back end in the cloud. Compute is also how the service decides to copy messages you sent on one device to other devices where you have the same app installed.

Here is an important distinction about how the ThreeFold Grid operates. In a traditional cloud environment, it is impossible to know for sure what is happening on the backend, even if you are the one running the service. The administrative access retained by the cloud provider can be used for surveillance and tampering that can be very difficult or impossible to detect.

Zero OS offers a unique assurance that no one can abuse the Grid in this way, because it operates autonomously with no facilities provided for human intervention. In order to utilize compute capacity on the Grid, individuals and organizations deploy their choice of several “primitives,” or basic components, that provide an environment for code to be executed and create the magic we experience as online applications.

🔗ThreeFold Compute Primitives

Since we’re getting a bit into the technical weeds here, we’ll just take a brief look at each primitive and what makes it special.

🔗Zmachine

Zmachine is a lightweight virtual machine (VM) implementation that’s been developed in a uniquely efficient way by ThreeFold. VMs are essentially computers within computers, allowing multiple operating systems to run on a single physical system. A common example is running the Windows operating system on a Mac that’s already running MacOS.

In the cloud world, virtual machines are a classic way to manage complex deployments with flexibility and resilience – by packaging different components into different VMs that can be run side by side and moved between different physical servers with ease. While VMs have been replaced in many cases with the new container paradigm, which we’ll cover next, they are still widely used and provide a simpler solution than containers that’s sufficient for many use cases.

🔗Flist

The Flist is a next generation container format which extends the industry standard Docker container concept in an efficient and secure way. It is supported directly at the operating system level on Zero OS, and in its latest version on ThreeFold Grid 3.0, provides a unique advantage over traditional containers via the use of mini virtual machines.

Containers provide greater efficiency and flexibility than VMs, by sharing parts of the host operating system between containers rather than duplicating the full operating system for each component. This, however, means that containers typically come with less of an assurance that different workloads and their data remain isolated from each other. By employing mini VMs, the Flist concept provides better isolation than traditional container systems while retaining the advantages that containers offer over full VMs.

🔗Kubernetes

Containers usually represent small parts of a full solution, which are created, interconnected, and destroyed on an ongoing basis. Kubernetes is the industry standard container orchestration system, and it’s used to manage many of the largest and most complex cloud deployments. Kubernetes solutions are defined through configuration files that specify the types of containers, how updates are handled, and the network architecture both between containers and with the outside world.

Multiple physical servers or VMs are typically used for a Kubernetes “cluster” which provides redundancy and scalability. The ThreeFold Grid provides a lightweight Kubernetes VM that can be used to create clusters according to specific needs. Kubernetes users can, for the most part, simply bring their existing configuration files and recreate their deployments on a Grid based cluster seamlessly.

🔗Wrapping up

Taken together, the compute primitives offer a comprehensive alternative to existing cloud compute providers. Users of Docker, Kubernetes, and virtual machines can bring their deployments to the Grid with a minimal amount of migration effort. This is distinct from other decentralized cloud projects, which either offer a single solution like Kubernetes or a completely new environment that developers must adapt their code to run within.

Better yet, each compute offering on the ThreeFold Grid benefits from the unique efficiency and security features of Zero OS, as well as the unique aspects of the primitives themselves. Thanks again for joining me on this journey into the wonderful world of ThreeFold technology. In the next episode, we’ll cover the last element of the cloud infrastructure triad: network.

Welcome back for the second part of this introductory series on ThreeFold Grid technology. Last time we took a high level view of all that the Grid offers. This time, we’ll be zooming in on storage, covering the low level components and how they can provide a familiar experience like the cloud “drives” and “boxes” we use today.

The ThreeFold Grid offers several different storage primitives that can be combined in various ways to meet the needs of developers and end users of Grid capacity. Ultimately, all storage services are a way of presenting the underlying hardware to software workloads in various ways, according to the desired features. ThreeFold has created a few unique storage offerings, providing distinct benefits over legacy cloud infrastructure. From the start, every component has been designed for optimal efficiency and they work together to offer an unprecedented level of security.

Our most basic storage primitive is a “volume”, which simply offers some disk space to an application in a generic way. We’ll focus more on the innovative solutions that are built on the Zero Database, or Zdb. These are compatible with existing use cases but also represent a quantum leap forward for privacy and security. They are called Zdbfs, Zstor, and QSFS. Let’s take a look at each component and how they work together to provide a complete set of tools for developers building on the Grid. We’ll also see why this matters to end users of services that are hosted on the Grid.

🔗Zdb

Zdb is a low level offering that implements with a subset of features from the popular Redis protocol. It is a key-value store, which means that information is indexed like a dictionary. Each “word” in the database is associated with a “definition” or piece of data. Zdb is super fast and efficient, along with operating in an “append only” manner. This means that all new data is added to empty space following the end of existing data, never overwritten. It’s like writing in pen on paper without leaving any white space.

Append only has many advantages, including extending the life of certain hardware and offering archiving features out of the box. The disadvantage, of course, is that old data is retained even when no longer needed. However, this can be addressed by periodically “compacting” the data, to remove what’s no longer needed and restructure what remains with the same level of efficiency. We use Zdb as a base layer for more complex storage implementations that benefit from these features.

🔗Zdbfs

While Zdb has all of those neat properties, it requires that applications utilize a specific database interface. To expand its capabilities, we have the Zdb Filesystem. With Zdbfs, the back end Zdb is exposed through the most common interface used for storing and retrieving data: a filesystem. This is the same format we’re all familiar with from the file explorers on our computers. When running Zdbfs, a new “folder” appears on the system, allowing reading and writing to the connected Zdb without any concern for the underlying database. With Zdbfs, nearly all existing tools for working with data can be plugged in to and benefit from the advantages of Zdb.

🔗Zstor

So far, we’ve been discussing solutions that utilize storage space on a single node only. While this offers flexibility for developers with a variety of needs, it is also limited in the sense that the failure of a single node would result in the loss of all data. With that in mind, we offer the Zstor solution for spreading data across multiple nodes in a way that provides both resilience against failures and unprecedented security.

Zstor takes a single file as input, to be stored among a number of nodes as specified in its configuration. The system is flexible and able to cater to different needs for performance, redundancy, and geographical distribution. Understanding what happens behind the scenes requires a little math, but I’ll provide as simple an explanation as I can. Zstor has a very unique property: it does not actually upload any of the user’s data to the back end nodes, but it is able to reconstruct the data later based on the descriptive information that is stored.

🔗The magic of erasure encoding

Sounds like magic? Well, let’s explore briefly how this is possible. Zstor relies on a technology known as erasure encoding, which was originally designed to protect data against errors that sometimes occur in the normal operation of computer hardware. Rather than simply making copies of the data as backups, a clever scheme is used to efficiently offer the same benefit. With basic redundancy, four extra copies would be needed to accommodate four failures without a loss of data. Erasure encoding can accomplish the same failure tolerance using less than half the space needed for the original data.

While erasure encoding can be implemented by storing the data itself along with some extra data known as “parity”, it can also be implemented without storing the original data at all. As a simplified example, let’s say that we want to store the number 13. First, we take each digit individually, 1 and 3. Next, we calculate 1 + 3 = 4 and 3 - 1 = 2. By storing 4 and 2, along with the instructions to reverse these calculations, we have everything needed to get back to 13. If we add one more equation, say 1 - 3 = -2, we can restore the original data using any two of the numbers we’ve computed.

Each of our values would be stored on separate nodes, while the instructions to recombine them would be stored in yet another location. If an attacker were to compromise one of these nodes, they would only have a number that’s meaningless without the other elements. If a single node fails, the original data can be restored using the remaining nodes, and we can add a new node to bring us back to our desired state. By tuning these values, additional security or redundancy can be achieved. We might decide that we want 9 of 10 values to be present for reconstruction, representing a high level of security. Alternatively, a 2 of 10 arrangement would provide high redundancy, tolerating the failure of 8 nodes.

🔗Quantum Safe Filesystem

With all of these components working together, we have ThreeFold’s flagship storage solution, the Quantum Safe Filesystem or QSFS. It uses Zstor to backup data written to a Zdbfs. This provides the convenient and highly compatible filesystem interface along with the exceptional security and redundancy benefits of Zstor. We call it quantum safe, because even an attacker with a quantum computer would not be able to decode users’ data, if they managed to hack into one of the back end nodes—already an extraordinary feat given the exceptional security of Zero OS.

🔗Bringing it home

So, we’ve toured a good bit of technology which might be feeling a bit abstract at this point. Bringing this home, our front end experience with these technologies can feel no different than any of the “drives” and “boxes” we use to store our files in the cloud. In fact, ThreeFold has already developed a prototype file browser based on QSFS which has many cool features like editing documents and viewing media directly in the web interface. It’s one piece of a full suite of solutions we’re excited to showcase and invite the community to test soon.

I hope you found this piece informative and approachable. Thanks for joining me to learn and explore the wonderful world of ThreeFold technology. We’ll cover more aspects of what makes it all tick in future parts of this series. Do you have questions or feel like chatting about what’s possible with ThreeFold’s technology? We’d love to hear from you on our forum or in our Telegram group.

🔗An Intro to the ThreeFold Grid

If you are a farmer or member of the ThreeFold community, you may be wondering about the capacity provided by the ThreeFold Grid and how it’s relevant to individuals and businesses who currently rely on centralized corporate clouds. This can be hard to imagine, because the back end infrastructure that makes all of our apps and services tick is generally invisible to us.

This piece will be the first in a series where I’ll be working to understand what the technical components are and how ThreeFold provides an alternative with distinct benefits for individuals, businesses, and for the planet as a whole. But don’t worry, we’ll start from the ground level and build up understanding in a way that’s accessible to everyone.

🔗Infrastructure for a New Internet

So, what is the internet? In short, it is a network of interconnected devices that facilitates communication and commerce. These devices include our mobile phones and personal computers, but also the servers that store and process data to provide the end experiences we enjoy. Currently the vast majority of these servers live in massive data centers controlled by a handful of corporations. ThreeFold is changing this, by allowing anyone to connect computational and storage capacity to an open network that is available for all to use, known as the ThreeFold Grid. Our farmers do include some data centers operating on a smaller scale than major cloud providers, and also many individuals connecting nodes at their home or office all over the world.

On the ThreeFold Grid, servers are known as 3Nodes. A 3Node can be pretty much any modern computer system, offering capacity by running the Zero OS operating system. This is a major distinguishing factor, both from the traditional cloud computing world and also from other projects working to create decentralized networks in this space.

🔗The OS for a Next Generation Cloud

By starting from the operating system level, ThreeFold is able to provide a network with much stronger security and greater efficiency than anything that came before. Zero OS includes a collection of components, known as “primitives”, which provide developers with everything they need. These primitives can be used to migrate existing applications from the legacy cloud and also to create new solutions that leverage the Grid’s capacity in unique ways.

We’re all familiar with operating systems like Windows and MacOS that provide an interface to applications on our personal computers. The OS interacts with the underlying hardware to expose its capabilities in a way that’s useful for both the software it hosts and also the humans who utilize it. For example, when you launch an app on your phone, it is the OS that’s responsible for connecting it to data networks and providing access to peripherals like cameras or microphones. Things are a little different in the world of servers, where rather than through direct interactions, users benefit through the services provided to the applications we use.

To extend the example, when you open a messaging app it will connect to a server somewhere to check for new messages and relay any messages you are sending to the recipient. Here we can see a central issue that has inspired the design of ThreeFold’s technology: when our messages pass through servers controlled by others, it creates an open door for surveillance of all kinds.

ThreeFold farmers, on the other hand, have no administrative access to their nodes. Furthermore, each component offered by the ThreeFold Grid is built with the intention of allowing for secure and private storage, processing, and transmission of information. Of course, sharing is absolutely possible, but only when the creator of data or content chooses for it to happen.

🔗Three Essential Resources

In the world of cloud computing, we speak of three basic resources provided by servers: compute, storage, and network. The ThreeFold Grid provides various primitives in each of these categories, to fulfill all the same needs served by centralized cloud providers. While our offerings have the special properties discussed above, they are also highly compatible with the most popular tools and protocols currently used in the industry. This makes it relatively simple to migrate the back end applications, also known as “workloads”, that provide a kind of connective tissue for all of the digital experiences we enjoy on a daily basis.

Workloads are pieces of software that provide services. If you upload a file to some cloud storage service, there could be a number of individual workloads orchestrated together in order to ensure the data you want to save is successfully transmitted to and stored in the back end system. We can understand the need for each kind of capacity by looking at the journey that your file might take in such a scenario. Starting on your device, a network connection, some processing, and finally storage capacity are all involved in providing the conditions that allow you to retrieve that file later.

🔗Network

Networking takes place over cellular connections, wifi, and wired internet lines to move data from one place to another. For our purposes here, those details aren’t so important—we’ll focus on what happens once communication is completed by whatever means.

When you connect to the cloud storage service, there are network workloads that transmit your data to the appropriate services that process and store it. This might consist of some routing and also authentication that ensures that you and only you access the files that are associated with your account. On the Grid, we provide primitives for secure peer to peer communication and also for allowing public access to back end services.

🔗Compute

Compute is what happens with data once it reaches its destination. Computers are mostly just fancy adding machines that take numbers and turn them into other numbers according to a set of specifications, which we call software. In our example, the compute step might involve compressing the information so it takes up less space and also deriving some useful statistics like the file’s original size.

Of course, there are many details we’re glossing over here, and the compute resources are ultimately responsible for providing much of our end user experience and the interactivity of the apps we utilize. 3Nodes offer a compute environment that natively runs the vast majority of workloads in use today.

🔗Storage

Storage, of course, is the disk space where data resides. It can be a little strange to think about, actually, that all of the information you’ve stored online actually has at least one physical location where it exists as many small electrical charges. On the Grid, application developers can access the disks of nodes directly, in a way that’s optimized for energy efficiency.

We also have more advanced storage solutions that offer unparalleled security and low overhead redundancy, giving users the best possible guarantee that only they have access to their data and that it will actually be there when they go to retrieve it.

🔗Looking Forward

Taken together, network, storage, and compute are the raw elements that give life to the internet we know and love. On the ThreeFold Grid, these elements are expressed in a way that’s both familiar for developers but also pushes the limits of what is possible in terms of technical evolution.

Thanks so much for joining me on this introductory overview of what the Grid provides. Please join me next time for a deeper dive on some of what we’ve covered and more specifics about how ThreeFold technology represents a quantum leap for cloud computing.

🔗An Internet By the People

The ThreeFold Grid is formed by people from all around the world that connect Internet capacity to a peer-to-peer Infrastructure owned by humanity. Together, we expand a data-sovereign and planet-positive Internet infrastructure that puts people back in control of their data, and their digital lives. ThreeFold’s infrastructure can store all of humanity’s data – from personal data to public datasets and security archives – and can run all emerging technologies such as blockchain, IoT, and Big Data – with more efficiency and security.

ThreeFold flips the current data center model upside-down, and allows people around the globe to connect Internet capacity in the form of hardware, and earn income for it. This decentralized approach means that anyone can take part in the world’s largest and fastest growing economy – the Internet and Cloud. 

You can see now how this new Internet is by the people, but how about the latter? What does the ThreeFold Grid enable?

🔗An Internet For the People

Now anyone, anywhere can connect to the ThreeFold Grid and use capacity with some key advantages:

🔗A World Where we Own the Value of Our Data

Today, all of the data we create online sits inside of centralized data centers and we do not own it. ThreeFold turns terms & conditions on their head and gives us, the people, full control of our data rather than third parties. We can keep our data private, decide with whom to share it, and even decide on where it’s stored and located. And considering that our personal data is worth between US $1,000 and $20,000 per year, ThreeFold offers a direct way for us to own our digital life and the value attached to it.

🔗We Exchange Value Without Intermediaries

The “People’s Internet” is peer-to-peer. This means connections between senders and recipients are completely end-to-end, eliminating the need for any form of intermediaries from the model. Can you imagine what the world would look like without intermediaries? Let’s take an example like Airbnb.

Today, if you are looking for a place to stay on vacation, you will probably end up using the services of Airbnb. But did you know that Airbnb charges both consumers and property owners for using their service? In a peer-to-peer world, this wouldn’t be necessary. You’d simply access a database of property owners (possibly through a peer-to-peer marketplace) and be able to complete the transaction yourselves. No hidden fees, just a direct exchange of value!

🔗Applications Run on the Edge

In its essence, edge computing brings compute and storage closer to people (and the devices where it’s being gathered) rather than relying on a centralized data centers located on the other side of the planet. This is the essence of the ThreeFold Grid. So, data doesn’t suffer any latency issues (performance) and it’s far more efficient / money-saving. Further, the distributed nature of the Grid allows for true privacy and security. Edge computing is not just a growing trend but, because of the growing number of connected devices and global Internet usage, it’s becoming a real necessity.

🔗But what about the planet?

If you’ve been following us, you must already know that the Internet is responsible for 10% or more of the world’s energy consumption. And with decades of experience in the Internet and Cloud industry, ThreeFold co-founders knew how to turn this around. We developed a lightweight operating system and infrastructure, removing unnecessary layers of complexity from the traditional and Web3 models (actually, blockchains are making it even more complex). The ThreeFold technology is always geared towards maximum efficiency while maintaing performance. And with these advancements, we can already save up-to 90% energy on storage workloads.

Additionally, the distributed nature of the Grid uses less bandwidth (see above re: edge computing). It’s been reported that Netflix alone takes up 15% of the Internet’s bandwidth. What if Netflix were to leverage the ThreeFold Grid? If stored closer to watchers, we could reduce network usage considerably.

Our commitment to the planet goes beyond the technology. Through a partnership with Take Action Global (TAG), we aim to take three times more CO2 out of the air compared to what we put in. Imagine regenerating the earth while using the Internet?

At ThreeFold, we believe that it’s time for us to take back control. As the world turns more and more digital due to the pandemic and even just the natural progression of technology, it’s time that we not just realize the problems of today’s Internet, but that we do something about it.

This will happen! So join in, and spread the word! 

If you’ve seen The Social Dilemma, you know all too well that today, people are being turned into products. Everything we do online – all of the data we upload and all of our digital activities – is tracked and oftentimes used against us. Targeted advertising is just the beginning.

Personal data is used by algorithms to make incredible important decisions, like whether someone should receive financial or health support or benefits. In China, people are now even receiving “scores” for their behaviors. It sounds like Black Mirror, but it’s real life.

That’s the scary part. The good news is that things don’t have to be this way. ThreeFold is building the engine for an Internet where people can finally be autonomous with their data.

With today’s Internet, infrastructure is centralized. Companies provide services in exchange for your personal data, which is stored on servers in hyper-scale data centers owned by private companies.

In ThreeFold’s model, those servers are called 3Nodes, and rather than sitting inside centralized data centers, they sit inside the homes or offices (or small data centers) owned by independent people and organizations called Farmers. But the data inside is owned by the person or persons who put it there. They maintain access right as well. From a technical standpoint, data is dispersed across 3Nodes by an algorithm that makes it such that nobody can access it. In addition, connections are peer-to-peer, meaning data goes directly between sender and recipient rather than passing through intermediaries as it does today – bringing further levels of security.

Thus, people can finally have data autonomy, knowing their data is safe, and truly theirs.

Just like a brain is an interconnected network of neurons, the Internet (or internet) is a global interconnected network of computers and devices that communicate with each other via the Internet protocol suite (TCP/IP). The communications infrastructure of the Internet consists of hardware components and a system of software layers that control various aspects of the architecture.

To create a new Internet from scratch, ThreeFold had to start at the operating system (OS) level. An operating system is a software that manages computer hardware, software resources, and provides common services for computer programs. And this is how Zero OS was born

🔗What makes Zero OS special?

Zero OS is a stateless and lightweight operating system. Given it has no shell, remote or server interface, it is much safer than the big majority of the OSs on the market. When storing your data on Zero OS, it is fingerprinted and made available only when requested by you or those you provide access to.

The state of an application represents its state of being in a given moment in time. Think of a stateless transaction as vending machines, a single request and a response. In ThreeFold’s Internet infrastructure, this can be represented as data being accessible only when requested. The only way to access data on Zero OS is via the private key (or fingerprint) of a user. In that sense, the data is not accessible unless the data owner requests access to it.

🔗How does Zero OS apply to developers?

Thanks to its lightweight architecture, Zero OS is a perfect fit to run microservice architectures and emerging IT workloads such as IoT, Blockchains, Big Data, and more. A key benefit of microservice architecture is the freedom it offers you to choose the best runtime for the task at hand.

Let’s take the example of decentralizing service-oriented developing teams. A common challenge there is that most of the time they don’t really understand the applications they are exposing. But no one knows the data infrastructure of the applications better than those who craft them.

ThreeFold permits decentralized developer teams with the ability to share keys (access) to the developing environment on Zero OS and the gateways to the world wide web within the secure peer-to-peer overlay network. This is game changing. Also with all the automation frameworks provided, it creates the perfect environment to develop current and emerging technologies.

Put shortly, decentralized development and integration is now officially an organizational or cultural change, not a technical one anymore.

🔗Zero OS allows anyone to provide sovereign capacity

Given stateless systems separate the OS configuration and user data, Zero OS simplifies the maintenance and the deployment of digital workloads, while making it impossible to hack into user data on a hardware level.

3Nodes provide all the storage, compute and network primitives for a new internet infrastructure. Given they are booted with Zero OS, anyone can now participate in the Internet economy without jeopardizing user data sovereignty and security.

Conclusively, Zero OS is ultra secure, perfect to develop current and future technologies, and enables anyone to take part in the planet positive and data sovereign Internet brought by ThreeFold.

Learn more about Zero OS and the grid primitives here.