Introduction to Declarative Macros in Rust

This post will give a quick overview of declarative macros in Rust. If you are not familiar with the concepts of macros in Rust, then read Introduction to Macros in Rust first.

Declarative macros work similar to how pattern matching works. It involves taking inputs, then pattern matching to determine what logic to execute.

The macros definition for creating declarative macros, then contains rules which determine the Rust code that gets generated/executed based on the pattern.

This is why declarative macros are also sometimes referred to as Macros By Example, as the code that is generated depends on the example of patterns that matches.

Introduction to Macros In Rust

This guide will present an overview of the fundamentals a new developer approaching macros in Rust needs to know. If you have read other materials online and you still find the concept of macros in Rust vague, then this post is for you.

The guide will be broken into 4 series of posts. The first post, which is this post, will be a quick overview of macros in Rust. The following 3 posts will then look more deeply into the different types of macros in Rust and how to create them.

The posts in this series are:

• Introduction to macros in Rust (this post)
• Introduction to declarative macros in Rust
• Introduction to attribute-like procedural macros in Rust (yet to be published)
• Introduction to function-like procedural macros in Rust (yet to be published)
• Introduction to custom-derive procedural macros in Rust (yet to be published)

What are macros

Macros are mechanisms for generating or manipulating code programmatically at compile time. In essence, it allows us to treat source code as values. Meaning that, at compile time, we can take inputs: where the input could be another source code or some defined string pattern, and from these inputs, manipulate and generate source codes that form the final compiled code that will be executed.

Macros as a concept are not specific to Rust, as they can be found in languages like Lisp, C/C++, Scala, and a host of others. Although one clear distinction of macros in Rust is that it provides type safety. Meaning that it provides mechanisms to prevent generating programs or source code that are not type-safe.

That is what macros are in a nutshell, a mechanism for taking inputs (other source codes, or just string patterns) and using these to generate source code/program at compile time.

So now that we know what macros are, we talk about how macro usage looks like.

What does macros usage look like in Rust?

Before looking at how to create macros in Rust, it is instructive to see how already created macros are used.

There are two distinct ways of making use of macros in Rust. macros are either used via A syntax I call bang function syntax or via attribute syntax.

The bang function syntax is the syntax that looks like a function call but with an exclamation mark at the end of the function name. For example things like: println!(), dbg!(), format!().

These are used to invoke macros and are not normal function calls.

Attributes syntax, on the other hand, is the syntax used for annotation. It takes the form of #[..] or #![..] and they can be placed on various language elements like functions, structs, enums, etc. This is another syntax that can be used to invoke macros.

A real-world example of attribute syntax being used to invoke macros can be found in the web framework called rocket. For example in the code below, gotten from the documentation here where we see the definition of a handler for the / HTTP path:

#[get("/")]
fn index() -> &'static str {
    "Hello, world!"
}

The #[get("/")] syntax is invoking a macro defined by the rocket framework. This invocation means that at compile time, the custom get macro will take the function index() as input, process it, and generate the exact source code, the framework needs to be able to call this function in response to an HTTP GET request to the path /. Essentially fulfilling what a macro is: a mechanism to generate source code at compile time.

The exact syntax used to call a macro depends on the type of macro and how it has been created. Some macro types are called via the bang function syntax, while other kinds are called via the attribute syntax.

Types of Rust macros.

In Rust, there are two types of macros: Declarative macros and Procedural macros. There are then 3 subcategories of procedural macros: Custom-derive macros, Attribute-like macros, and Function-like macros.

Basically types of Rust macros:

• Declarative macros (also referred to as macro by example or mbe for short)
• Procedural macros
  • Function-like
  • Attribute-like macros
  • Custom derive macros

The first thing to note in this categorisation is what makes declarative macros different from procedural Macros.

The difference lies in their inputs and outputs, and also how they are created and used.

Declarative macros take what can be described as token fragments while procedural macros take TokenStream as inputs. Token fragments are like string patterns that can further be qualified by a fragment specifier that tells what kind they are. Fragment specifiers can be used to specify the inputs are block, expression, statement, etc. See Fragment Specifiers for an overview of the supported fragment specifiers.

Procedural macros, on the other hand, take in TokenStream which is a more low-level representation of the source code.

The other difference between declarative macros and procedural macros (also the difference between the 3 kinds of procedural macros) is how they are created and used.

Declarative macros and function-like procedural macros are used by a syntax that looks like function invocation but ends with !. for example do_work!(). Meaning when looking at usage it is not possible to tell if a declarative macro is being used or a function-like procedural macro is.

Attribute-like procedural macros on the other hand are used to create a custom annotation, so they are used with the syntax of applying attributes. That is: #[my_macro]. They can also be defined to take helper attributes. That is #[my_macro(attr)]

While custom derives macros are used to define new values that can be passed into the #[derive(...)] macro. For example, if MyMacro is defined to be a custom derive macro, it can then be used as #[derive(MyMacro)]

This completes the quick overview of macros in Rust. The following posts in the series will then go into the details of how these different types of macros are created and used and their different peculiarities.

How to query account balance on Solana using Rust.

This post shows how to query Solana's cluster rpc endpoint using Rust. Most of the examples of how to perform this task that I ran to uses Javascript/Typescript via the Solana Web3js library, so I decided to do a quick post showing how to do same but in Rust. It uses querying for the sol balance of a Solana account as the main example of interaction with the rpc endpoint.

The task is simple: given a Solana address, we retrieve the sol balance for that address.

To get started we will need the following dependencies:

[dependencies]
solana-client = "1.14.3" // update to the current latest version
solana-sdk = "1.14.3" // update to the current latest version

The code snippet below then shows how to call the rpc endpoint and retrieve the Sol balance for an account

use solana_client::rpc_client::RpcClient;
use solana_program::pubkey::Pubkey;
use std::str::FromStr;

fn main() {
 let rpc = RpcClient::new("https://api.devnet.solana.com");
 let pubkey_str = "4zMMC9srt5Ri5X14GAgXhaHii3GnPAEERYPJgZJDncDU";

 let balance = rpc
      .get_account(&Pubkey::from_str(pubkey_str).unwrap())
      .unwrap().lamports;

 println!("Sol balance of {pubkey_str} is {balance}");
}

Solana addresses are base58 encoded strings, hence they need to be first converted into an instance of Pubkey before they can be used in the rpc.get_account call, which expects a &[u8] byte array. To help with this conversion, Pubkey implements the std::str::FromStr trait. This wmeans, if the trait std::str::FromStr is in scope, then it is possible to convert a string to Pubkey, which is what was done in the code snippet above.

This post was inspired by a recent question I answered on Solana stack exchange here. The answer also mentions another verbose alternative that involves manually converting from base58 to bytes array.

IntoIterator and the for ... in Syntax in Rust

In Rust Iterator pattern with iter(), into_iter() and iter_mut() methods I explained why attempting to use a variable holding a Vec after iterating through it using the for … in syntax leads to a compilation error.

The post explains why the following code won't compile:

fn main() {
   let some_ints = vec![1,2,3,4,5];
  // iterating through a vec
   for i in some_ints {
       dbg!(i);
   }
// attempting to use the vec will 
// lead to compile error after the iteration
   dbg!(some_ints);
}

I then showed 3 methods that can be called before iterating using the for … in and how 2 of these methods allow the Vec to still be used even after iteration.

These 3 methods are into_iter(), iter(), and iter_mut(). That is:

#[test]
fn into_iter_demo() {
    // the .into_iter() method creates an iterator, v1_iter 
    // which takes ownership of the values being iterated.
    let mut v1_iter = v1.into_iter();

    assert_eq!(v1_iter.next(), Some(1));
    assert_eq!(v1_iter.next(), Some(2));
    assert_eq!(v1_iter.next(), Some(3));
    assert_eq!(v1_iter.next(), None);

    // If the line below is uncommented, the code won't compile anymore
    // this is because, after the iteration, v1 can no longer be used 
    // since the iteration moved ownership
    //dbg!(v1);
}

The two other methods that allow the Vec to still be used after iteration via for … in are:

#[test]
fn iter_demo() {
    let v1 = vec![1, 2, 3];
    // the .iter() method creates an iterator, 
    // v1_iter which borrows value immutably 
    let mut v1_iter = v1.iter();

    // iter() returns an iterator of slices.
    assert_eq!(v1_iter.next(), Some(&1));
    assert_eq!(v1_iter.next(), Some(&2));
    assert_eq!(v1_iter.next(), Some(&3));
    assert_eq!(v1_iter.next(), None);
   // because values were borrowed immutably, 
   // it is still possible to use 
   // the vec after iteration is done
    dbg!(v1);
}

And

#[test]
fn iter_mut_demo() {
    let mut v1 = vec![1, 2, 3];

    // the .iter_mut() method creates an iterator, 
    // v1_iter which borrows value and can mutate it. 
    let mut v1_iter = v1.iter_mut();

    // access the first item and multiple it by 2
    let item1 = v1_iter.next().unwrap();
    *item1 = *item1 * 2;

    // access the second item and multiple it by 2
    let item2 = v1_iter.next().unwrap();
    *item2 = *item2 * 2;

    // access the third item and multiple it by 2
    let item3 = v1_iter.next().unwrap();
    *item3 = *item3 * 2;

    // end of the iteration
    assert_eq!(v1_iter.next(), None);

    // this will print out [2,4,6]
    dbg!(v1);
}

In this post, we are going to dive a little bit deeper into understanding some of the machinery that makes the above work.

We start again by talking about the Iterator trait.

Rust Iterator pattern with iter(), into_iter() and iter_mut() methods

Let's create a vec of integers, iterate through and print the individual values, and then afterward, print out the whole vec. Here is the code:

fn main() {
   let some_ints = vec![1,2,3,4,5];
   for i in some_ints {
       dbg!(i);
   }

   dbg!(some_ints);
}

Simple right? Well nope. Trying to compile the above code will fail with the following errors:

error[E0382]: use of moved value: `some_ints`
   --> src/main.rs:9:9
    |
4   |    let some_ints = vec![1,2,3,4,5];
    |        --------- move occurs because `some_ints` has type `Vec<i32>`, 
    which does not implement the `Copy` trait
5   |    for i in some_ints {
    |             --------- `some_ints` moved due to this implicit call to `.into_iter()`
...
9   |    dbg!(some_ints);
    |         ^^^^^^^^^ value used here after move
    |
note: this function takes ownership of the receiver `self`, which moves `some_ints`
help: consider iterating over a slice of the `Vec<i32>`'s 
content to avoid moving into the `for` loop
    |
5   |    for i in &some_ints {
    |             +

For more information about this error, try `rustc --explain E0382`

Why is this the case? Why is the borrow checker preventing the use of a vec after a simple iteration?

Well, the answer to that question lies in Rust's implementation of the Iterator pattern - which by the way, is what makes it possible to use the for…in syntax.

Iterators are not special or unique to Rust. The concept can be found in a handful of languages. I wrote about the Iterator pattern as it exists in JavaScript in the post Iterables and Iterators in JavaScript.

The unique thing about the Iterator pattern in Rust is its interaction with the borrow checker.

If this interaction with the borrow checker is not taken into consideration then it is possible to bump into certain confusing compile errors while attempting to use the iterator pattern.

So to get started answering the question of why the borrow checker prevents what looks like a legit code above, let's take a look at the Iterator pattern and what is special about it in Rust.

Introduction to Digital Signature for the Working Developer

In this post, we would be looking at digital signatures. It will be a quick, straight-to-the-point overview of some of the essential things a developer should know about digital signatures. How to think about them and what problems they solve. It is targeted at the working developer who needs to be familiar enough with digital signatures to use them, but who does not need to know the gory details of how they are implemented, or how they work internally.

This post contains the following sections: 

1. What is a digital signature 
2. Components of Digital Signature Schemes 
3. Digital Signature Algorithms 
4. Last note: Digital Signatures versus Message Authenticated Codes.

Introduction to Asymmetric Encryption for the Working Developer

In this post, we are going to look at asymmetric encryption. As mentioned in the previous post on symmetric encryption, encryption is the cryptographic primitive that guarantees confidentiality, by which we mean the ability for two or more authorized parties to communicate, without an unauthorized person being able to decipher the messages being communicated.

In symmetric encryption, both parties need to use the same key for encryption and decryption. This leads to the problem of how to securely get the communicating parties to use the same key. A solution to this is the Key exchange protocol that was discussed in Introduction to Key Exchange for the Working Developer

Asymmetric encryption is different. Unlike symmetric encryption, it does not have the requirement that the same key needs to be used for encryption and decryption. Hence why it is called asymmetric. It makes use of key pairs instead. One is called the private key, the other called the public key.

In this post, we will be looking at how to use asymmetric encryption and what other general information to be aware of. As always it is targeted at the working developer who needs a grounded understanding of these cryptographic primitives without necessarily covering the internals of these primitives.

A Neophyte's Introduction to the async/await landscape in Rust

The truth of the matter is, the async/await features in other programming languages seem to be more straightforward than it is in Rust. For example in JavaScript, you have the syntax that you use, and that is about it. But in Rust, there seem to be more moving parts.

Tokio? Future? There is a futures crate? Runtimes? How do all these fit into the async/await picture in Rust?

This is a short post to help clarify some of the terms a developer would encounter when they start exploring asynchronous programming in Rust. It would explain the main moving parts, which are: the standard library's std::future::Future, the futures crates, and asynchronous runtimes like tokio and async-std.

Let get started:

Introduction to Key Exchange for the Working Developer

In Introduction to Symmetric Encryption for the Working Developer, we saw how to encrypt and decrypt data. Being symmetric encryption, both the process of encryption and decryption needed the same key. This poses a challenge, which was not addressed in the post: How do one get across the same key to both parties who want to engage in encrypted communication?

Basically how do we perform key exchange? How do we get the same key to parties who want to encrypt and decrypt messages?

An approach is perhaps for the parties involved to first meet in person and exchange the key?

Another approach could be to use another communication channel to first share the key? But then again, if the other communication channel is digital, and it uses symmetric encryption, how should its encryption key be also exchanged? This becomes a catch 22 real quick.

This is the problem Key exchange schemes seek to solve, and thankfully there is a secure way to have two parties exchange secret keys without the need to physically meet in person. This is what this post is about. We will be looking at two popular mechanisms for key exchange: Diffie-Hellman key exchange procedures and application of RSA for key exchange.

As always, as with the other posts in this series, the idea is to provide the basic information needed by the working developer to be able to understand and use these cryptographic primitives without going into the thick of their internal details or implementation.

This post contains the following sections

• Entering the realm of Public Key Cryptography
• Introduction to Diffie-Hellman Key Exchange: An Intuition.
• Whirlwind tour of the Mathematics
• Diffie-Hellman in code
• Diffie-Hellman Standards
• Using RSA for key exchange
• Conclusion and References

Introduction to Authenticated Encryption for the Working Developer

In Introduction to Symmetric Encryption for the Working Developer, I presented an overview of symmetric encryption. Which as explained in the post, is a process for establishing confidentiality during communication between two parties.

The only problem is that just encrypting data using symmetric encryption does not provide all the security requirements needed during secure communication.

The issue is that, it is still possible for an eavesdropper to tamper with the message even without being able to make sense of the ciphertexts, and the receiving party won't be made aware that the message they are decrypting has been tampered with and is not exactly the original one that was sent.

Hence even though encryption gives us confidentiality, done alone, it does not guarantee the other security requirements like authenticity/integrity that is often needed - i.e. it does not guarantee that a message came from the right party (authenticity) and it has not been modified in any way (integrity).

Encryption alone is not enough, we need authenticated encryption. This post will introduce the concept of authenticated encryption.

Introduction to Symmetric Encryption for the Working Developer

Encryption is all about establishing confidentiality. That is, making sure only authorised parties have access to specific data. It involves transforming data into a form that is inscrutable to unauthorised parties but with the ability for authorised parties to transform the data back into its legible form.

Think about two parties communicating without wanting other persons privy to the messages. This is achieved via the cryptographic primitive of Encryption.

In this post, I will provide a whirlwind tour of what encryption is for the working developer. The goal is to provide the basic information needed to be able to properly wield this cryptography tool without going into the inner workings of the algorithm. It is part of the Cryptography101 series of posts.

This post contains the following

• What Encryption is and its main components
• What Encryption is not
• Symmetric Encryption vs Asymmetric Encryption.
• Block Cipher vs Stream Cipher.
• Overview of encryption Standards
• Components of AES
  • Key
  • Padding
  • Mode