An overview of NSS Internals

A High-Level Overview to the Internals of Network Security Services (NSS)

Software developed by the Mozilla.org projects traditionally used its own implementation of security protocols and cryptographic algorithms, originally called Netscape Security Services, nowadays called Network Security Services (NSS). NSS is a library written in the C programming language. It’s free and open source software, and many other software projects have decided to use it. In order to support multiple operating systems (OS), it is based on a cross platform portability layer, called the Netscape Portable Runtime (NSPR), which provides cross platform application programming interfaces (APIs) for OS specific APIs like file system access, memory management, network communication, and multithreaded programming.

NSS offers lots of functionality; we’ll walk through the list of modules, design principles, and important relevant standards.

In order to allow interoperability between software and devices that perform cryptographic operations, NSS conforms to a standard called PKCS#11. (Note that it’s important to look at the number 11, as there are other PKCS standards with different numbers that define quite different topics.)

A software or hardware module conforming to the PKCS#11 standard implements an interface of C calls, which allow querying the characteristics and offered services of the module. Multiple elements of NSS’s own modules have been implemented with this interface, and NSS makes use of this interface when talking to those modules. This strategy allows NSS to work with many hardware devices (e.g., to speed up the calculations required for cryptographic operations, or to access smartcards that securely protect a secret key) and software modules (e.g., to allow to load such modules as a plugin that provides additional algorithms or stores key or trust information) that implement the PKCS#11 interface.

A core element of NSS is FreeBL, a base library providing hash functions, big number calculations, and cryptographic algorithms.

Softoken is an NSS module that exposes most FreeBL functionality as a PKCS#11 module.

Some cryptography uses the same secret key for both encrypting and decrypting, for example password based encryption (PBE). This is often sufficient if you encrypt data for yourself, but as soon as you need to exchange signed/encrypted data with communication partners, using public key encryption simplifies the key management. The environment that describes how to use public key encryption is called Public Key Infrastructure (PKI). The public keys that are exchanged between parties are transported using a container; the container is called a certificate, following standard X.509 version 3. A certificate contains lots of other details; for example, it contains a signature by a third party that expresses trust in the ownership relationship for the certificate. The trust assigned by the third party might be restricted to certain uses, which are listed in certificate extensions that are contained in the certificate.

Many (if not most) of the operations performed by NSS involve the use of X.509 certificates (often abbreviated as “cert”, unfortunately making it easy to confuse with the term “computer emergency response team“).

When checking whether a certificate is trusted or not, it’s necessary to find a relevant trust anchor (root certificate) that represents the signing capability of a trusted third party, usually called a Certificate Authority (CA). A trust anchor is just another X.509 certificate that is already known and has been deliberately marked as trusted by a software vendor, administrators inside an organizational infrastructure, or the software user. NSS ships a predefined set of CA certificates. This set, including their trust assignments, is provided by NSS as a software module, called CKBI (“built-in root certificates”), which also implements the PKCS#11 interface. On an organizational level the contents of the set are managed according to the Mozilla CA policy. On a technical level the set is a binary software module.

A cryptographic transaction, such as encryption or decryption related to a data exchange, usually involves working with the X.509 certs of your communication partners (peer). It’s also required that you safely keep your own secret keys that belong to your own certificates. You might want to protect the storage of your secret keys with PBE. You might decide to modify the default trust provided by NSS. All of this requires storing, looking up, and retrieving data. NSS simplifies performing these operations by offering storage and management APIs. NSS doesn’t require the programmer to manage individual files containing individual certificates or keys. Instead, NSS offers to use its own database(s). Once you have imported certificates and keys into the NSS database, you can easily look them up and use them again.

Because of NSS’s expectation to operate with an NSS database, it’s mandatory that you perform an initialization call, where you tell NSS which database you will be using. In the most simple scenario, the programmer will provide a directory on your filesystem as a parameter to the init function, and NSS is designed to do the rest. It will detect and open an existing database, or it can create a new one. Alternatively, should you decide that you don’t want to work with any persistent recording of certificates, you may initialize NSS in a no-database mode. Usually, NSS will flush all data to disk as soon as new data has been added to permanent storage. Storage consists of multiple files: a key database file, which contains your secret keys, and a certificate database file which contains the public portion of your own certificates, the certificates of peers or CAs, and a list of trust decisions (such as to not trust a built-in CA, or to explicitly trust other CAs). Examples for the database files are key3.db and cert8.db, where the numbers are file version numbers. A third file contains the list of external PKCS#11 modules that have been registered to be used by NSS. The file could be named secmod.db, but in newer database generations a file named pkcs11.txt is used.

Only NSS is allowed to access and manipulate these database files directly; a programmer using NSS must go through the APIs offered by NSS to manipulate the data stored in these files. The programmer’s task is to initialize NSS with the required parameters (such as a database), and NSS will then transparently manage the database files.

Most of the time certificates and keys are supposed to be stored in the NSS database. Therefore, after initial import or creation, the programmer usually doesn’t deal with their raw bytes. Instead, the programmer will use lookup functions, and NSS will provide an access handle that will be subsequently used by the application’s code. Those handles are reference counted. NSS will usually create an in-memory (RAM) presentation of certificates, once a certificate has been received from the network, read from disk, or looked up from the database, and prepare in-memory data structures that contain the certificate’s properties, as well as providing a handle for the programmer to use. Once the application is done with a handle, it should be released, allowing NSS to free the associated resources. When working with handles to private keys it’s usually difficult (and undesired) that an application gets access to the raw key data; therefore it may be difficult to extract such data from NSS. The usual minimum requirement is that private keys must be wrapped using a protective layer (such as password-based encryption). The intention is to make it easier to review code for security. The less code that has access to raw secret keys, the less code that must be reviewed.

NSS has only limited functionality to look up raw keys. The preferred approach is to use certificates, and to look up certificates by properties such as the contained subject name (information that describes the owner of the certificate). For example, while NSS supports random calculation (creation) of a new public/private key pair, it’s difficult to work with such a raw key pair. The usual approach is to create a certificate signing request (CSR) as soon as an application is done with the creation step, which will have created a handle to the key pair, and which can be used for the necessary related operations, like producing a proof-of-ownership of the private key, which is usually required when submitting the public key with a CSR to a CA. The usual follow up action is receiving a signed certificate from a CA. (However, it’s also possible to use NSS functionality to create a self-signed certificate, which, however, usually won’t be trusted by other parties.) Once received, it’s sufficient to tell NSS to import such a new certificate into the NSS database, and NSS will automatically perform a lookup of the embedded public key, be able to find the associated private key, and subsequently be able to treat it as a personal certificate. (A personal certificate is a certificate for which the private key is in possession, and which could be used for signing data or for decrypting data.) A unique nickname can/should be assigned to the certificate at the time of import, which can later be used to easily identify and retrieve it.

It’s important to note that NSS requires strict cleanup for all handles returned by NSS. The application should always call the appropriate dereference (destroy) functions once a handle is no longer needed. This is particularly important for applications that might need to close a database and reinitialize NSS using a different one, without restarting. Such an operation might fail at runtime if data elements are still being referenced.

In addition to the FreeBL, Softoken, and CKBI modules, there is an utility library for general operations (e.g., encoding/decoding between data formats, a list of standardized object identifiers (OID)). NSS has an SSL/TLS module that implements the Secure Sockets Layer/Transport Layer Security network protocols, an S/MIME module that implements CMS messaging used by secure email and some instant messaging implementations, a DBM library that implements the classic database storage, and finally a core NSS library for the big set of “everything else”. Newer generations of the database use the SQLite database to allow concurrent access by multiple applications.

All of the above are provided as shared libraries. The CRMF library, which is used to produce certain kinds of certificate requests, is available as a library for static linking only.

When dealing with certificates (X.509), file formats such as PKCS#12 (certificates and keys), PKCS#7 (signed data), and message formats as CMS, we should mention ASN.1, which is a syntax for storing structured data in a very efficient (small sized) presentation. It was originally developed for telecommunication systems at times where it was critical to minimize data as much as possible (although it still makes sense to use that principle today for good performance). In order to process data available in the ASN.1 format, the usual approach is to parse it and transfer it to a presentation that requires more space but is easier to work with, such as (nested) C data structures. Over the time NSS has received three different ASN.1 parser implementations, each having their own specific properties, advantages and disadvantages, which is why all of them are still being used (nobody has yet dared to replace the older with the newer ones because of risks for side effects). When using the ASN.1 parser(s), a template definition is passed to the parser, which will analyze the ASN.1 data stream accordingly. The templates are usually closely aligned to definitions found in RFC documents.

A data block described as DER is usually in ASN.1 format. You must know which data you are expecting, and use the correct template for parsing, based on the context of your software’s interaction. Data described as PEM is a base64 encoded presentation of DER, usually wrapped between human readable BEGIN/END lines. NSS prefers the binary presentation, but is often capable to use base64 or ASCII presentations, especially when importing data from files. A recent development adds support for loading external PEM files that contain private keys, in a software library called nss-pem, which is separately available, but should eventually become a core part of NSS.

Looking at the code level, NSS deals with blocks of raw data all the time. The common structure to store such an untyped block is SECItem, which contains a size and an untyped C pointer variable.

When dealing with memory, NSS makes use of arenas, which are an attempt to simplify management with the limited offerings of C (because there are no destructors). The idea is to group multiple memory allocations in order to simplify cleanup. Performing an operation often involves allocating many individual data items, and the code might be required to abort a task at many positions in the logic. An arena is requested once processing of a task starts, and all memory allocations that are logically associated to that task are requested from the associated arena. The implementation of arenas makes sure that all individual memory blocks are tracked. Once a task is done, regardless whether it completed or was aborted, the programmer simply needs to release the arena, and all individually allocated blocks will be released automatically. Often freeing is combined with immediately erasing (zeroing, zfree) the memory associated to the arena, in order to make it more difficult for attackers to extract keys from a memory dump.

NSS uses many C data structures. Often NSS has multiple implementations for the same or similar concepts. For example, there are multiple presentations of certificates, and the NSS internals (and sometimes even the application using NSS) might have to convert between them.

Key responsibilites of NSS are verification of signatures and certificates. In order to verify a digital signature, we have to look at the application data (e.g., a document that was signed), the signature data block (the digital signature), and a public key (as found in a certificate that is believed to be the signer, e.g., identified by metadata received together with the signature). The signature is verified if it can be shown that the signature data block must have been produced by the owner of the public key (because only that owner has the associated private key).

Verifying a certificate (A) requires some additional steps. First, you must identify the potential signer (B) of a certificate (A). This is done by reading the “issuer name” attribute of a certificate (A), and trying to find that issuer certificate (B) (by looking for a certificate that uses that name as its “subject name”). Then you attempt to verify the signature found in (A) using the public key found in (B). It might be necessary to try multiple certificates (B1, B2, …) each having the same subject name.

After succeeding, it might be necessary to repeat this procedure recursively. The goal is to eventually find a certificate B (or C or …) that has an appropriate trust assigned (e.g., because it can be found in the CKBI module and the user hasn’t made any overriding trust decisions, or it can be found in a NSS database file managed by the user or by the local environment).

After having successfully verified the signatures in a (chain of) issuer certificate(s), we’re still not done with verifying the certificate A. In a PKI it’s suggested/required to perform additional checks. For example: Certificates were valid at the time the signature was made, name in certificates matches the expected signer (check subject name, common name, email, based on application), the trust restrictions recorded inside the certificate (extensions) permit the use (e.g., encryption might be allowed, but not signing), and based on environment/application policy it might be required to perform a revocation check (OCSP or CRL), that asks the issuer(s) of the certificates whether there have been events that made it necessary to revoke the trust (revoke the validity of the cert).

Trust anchors contained in the CKBI module are usually self signed, which is defined as having identical subject name and issuer name fields. If a self-signed certificate is marked as explicitly trusted, NSS will skip checking the self-signature for validity.

NSS has multiple APIs to perform verification of certificates. There is a classic engine that is very stable and works fine in all simple scenarios, for example if all (B) candidate issuer certificates have the same subject and issuer names and differ by validity period; however, it works only in a limited amount of more advanced scenarios. Unfortunately, the world of certificates has become more complex in the recent past. New Certificate Authorities enter the global PKI market, and in order to get started with their business, they might make deals with established CAs and receive so-called cross-signing-certificates. As a result, when searching for a trust path from (A) to a trusted anchor (root) certificate (Z), the set of candidate issuer certificates might have different issuer names (referring to the second or higher issuer level). As a consequence, it will be necessary to try multiple different alternative routes while searching for (Z), in a recursive manner. Only the newer verification engine (internally named libPKIX) is capable of doing that properly.

It’s worth mentioning the Extended Validation (EV) principle, which is an effort by software vendors and CAs to define a stricter set of rules for issuing certificates for web site certificates. Instead of simply verifying that the requester of a certificate is in control of an administrative email address at the desired web site’s domain, it’s required that the CA performs a verification of real world identity documents (such as a company registration document with the country’s authority), and it’s also required that a browser software performs a revocation check with the CA, prior to granting validity to the certificate. In order to distinguish an EV certificate, CAs will embed a policy OID in the certificate, and the browser is expected to verify that a trust chain permits the end entity (EE) certificate to make use of the policy. Only the APIs of the newer libPKIX engine are capable of performing a policy verification.

That’s a good opportunity to talk about SSL/TLS connections to servers in general (not just EV, not just websites). Whenever this document mentions SSL, it refers to either SSL or TLS. (TLS is a newer version of SSL with enhanced features.)

When establishing an SSL connection to a server, (at least) a server certificate (and its trust chain) is exchanged from the server to the client (e.g., the browser), and the client verifies that the certificate can be verified (including matching the name of the expected destination server). Another part of the handshake between both parties is a key exchange. Because public key encryption is more expensive (more calculations required) than symmetric encryption (where both parties use the same key), a key agreement protocol will be executed, where the public and private keys are used to proof and verify the exchanged initial information. Once the key agreement is done, a symmetric encryption will be used (until a potential re-handshake on an existing channel). The combination of the hash and encryption algorithms used for a SSL connection is called a cipher suite.

NSS ships with a set of cipher suites that it supports at a technical level. In addition, NSS ships with a default policy that defines which cipher suites are enabled by default. An application is able to modify the policy used at program runtime, by using function calls to modify the set of enabled cipher suites.

If a programmer wants to influence how NSS verifies certificates or how NSS verifies the data presented in a SSL connection handshake, it is possible to register application-defined callback functions which will be called by NSS at the appropriate point of time, and which can be used to override the decisions made by NSS.

If you would like to use NSS as a toolkit that implements SSL, remember that you must init NSS first. But if you don’t care about modifying the default trust permanently (recorded on disk), you can use the no-database init calls. When creating the network socket for data exchange, note that you must use the operating system independent APIs provided by NSPR and NSS. It might be interesting to mention a property of the NSPR file descriptors, which are stacked in layers. This means you can define multiple layers that are involved in data processing. A file descriptor has a pointer to the first layer handling the data. That layer has a pointer to a potential second layer, which might have another pointer to a third layer, etc. Each layer defines its own functions for the open/close/read/write/poll/select (etc.) functions. When using an SSL network connection, you’ll already have two layers, the basic NSPR layer and an SSL library layer. The Mozilla applications define a third layer where application specific processing is performed. You can find more details in the NSPR reference documents.

NSS occassionally has to create outbound network connections, in addition to the connections requested by the application. Examples are retrieving OCSP (Online Certificate Status Protocol) information or downloading a CRL (Certificate Revocation List). However, NSS doesn’t have an implementation to work with network proxies. If you must support proxies in your application, you are able to register your own implementation of an http request callback interface, and NSS can use your application code that supports proxies.

When using hashing, encryption, and decryption functions, it is possible to stream data (as opposed to operating on a large buffer). Create a context handle while providing all the parameters required for the operation, then call an “update” function multiple times to pass subsets of the input to NSS. The data will be processed and either returned directly or sent to a callback function registered in the context. When done, you call a finalization function that will flush out any pending data and free the resources.

This line is a placeholder for future sections that should explain how libpkix works and is designed.

If you want to work with NSS, it’s often helpful to use the command line utilities that are provided by the NSS developers. There are tools for managing NSS databases, for dumping or verifying certificates, for registering PKCS#11 modules with a database, for processing CMS encrypted/signed messages, etc.

For example, if you wanted to create your own pair of keys and request a new certificate from a CA, you could use certutil to create an empty database, then use certutil to operate on your database and create a certificate request (which involves creating the desired key pair) and export it to a file, submit the request file to the CA, receive the file from the CA, and import the certificate into your database. You should assign a good nickname to a certificate when importing it, making it easier for you to refer to it later.

It should be noted that the first database format that can be accessed simultaneously by multiple applications is key4.db/cert9.db – database files with lower numbers will most likely experience unrecoverable corruption if you access them with multiple applications at the same time. In other words, if your browser or your server operates on an older NSS database format, don’t use the NSS tools to operate on it while the other software is executing. At the time of writing NSS and the Mozilla applications still use the older database file format by default, where each application has its own NSS database.

If you require a copy of a certificate stored in an NSS database, including its private key, you can use pk12util to export it to the PKCS#12 file format. If you require it in PEM format, you could use the openssl pkcs12 command (that’s not NSS) to convert the PKCS#12 file to PEM.

This line is a placeholder for how to prepare a database, how to dump a cert, and how to convert data.

You might have been motivated to work with NSS because it is used by the Mozilla applications such as Firefox, Thunderbird, etc. If you build the Mozilla application, it will automatically build the NSS library, too. However, if you want to work with the NSS command line tools, you will have to follow the standalone NSS build instructions, and build NSS outside of the Mozilla application sources.

The key database file will contain at least one symmetric key, which NSS will automatically create on demand, and which will be used to protect your secret (private) keys. The symmetric key can be protected with PBE by setting a master password on the database. As soon as you set a master password, an attacker stealing your key database will no longer be able to get access to your private key, unless the attacker would also succeed in stealing the master password.

Now you might be interest in how to get the NSS sources building testing