SCTP Multi-Homing in Linux

After exploring how SCTP multi-homing works, it's time to see how to use this feature in Linux. This post will show how to implement multi-homing for the client-server application, used up to now. Baseline code for the development will be the 'one-to-many_advanced' branch, used in SCTP specific socket functions in Linux post. It already uses SCTP's advanced interface so only a few modifications are required to get the job done.

Implementing multi-homing means that the server should bind to more than one IP address and the client - to connect to more than one destination IP address. Both applications can also operate on single IP address. Mixed scenario is also possible, e.g. a multi-homed server can communicate with non-multi-homed client.

The code in this post uses one-to-many styles sockets, however the approach is the same for one-to-one sockets. I am pretty sure that you will be able to handle the latter case by yourself, so I will not discuss it. Anyway do not hesitate to leave a comment, in case you have got any questions.

How to Implement Multi-Homing

You should remember from SCTP specific socket functions in Linux post that sctp_connectx() and sctp_bindx() can work with more than one IP address. This means that it is mandatory to use these functions in a multi-homed application. Everything else, like link monitoring and path switch-over) is done by the kernel.

On the server side sctp_bindx() is called to specify which IP addresses and ports should be used. The addresses are passed via addrs parameter (which is a pointer to an array of struct sockaddr) and addrcnt is the elements count.

On the client side, sctp_connectx() is used to connect to a multi-homed server. This function has also got addrs parameter (pointer to array of struct sockaddr) and addrcnt (the elements count). If specific local IP addresses should be used for the association establishment, sctp_bindx() should also be called to specify them. If random local port number should be used (which is a good idea in most of the cases) sin_port in struct sockaddr should be set to zero.

Code Review

To explore multi-homing, proper network environment is needed. The setup from the previous post is good enough, so I will use it again.  The code for this post is in one-to-many_multihoming branch. Check it out:

git checkout one-to-many_multihoming

Both client and server have got command line parameters for the server port, local and peer IP addresses. In real world applications it's very important to implement user friendly handling of those parameters. However the main purpose of this project is to demonstrate how SCTP is used and I try to keep the code as focused on this topic as possible. For this reason I decided to sacrifice the proper command line arguments handling and implemented a really basic solution. For example the list of local/remote addresses is represented by a semi-colon separated string. If a space/tab is put between the addresses - the application will return an error. I hope you will excuse this sloppiness for the sake of simplicity.

The code which parses the command line arguments is based on getopt. I consider it not relevant to this post and I will not discuss it at all.

Common Code

In common.h there are two new functions - validate_port_number(), which checks if the port is valid and parse_addresses() which splits semi-colon separated list of IP addresses to array of strings. Both functions are helpers for the command line parser. As already discussed, parse_addresses() doesn't handle whitespaces at all, so don't try anything fancy with it.

The server hasn't got any mandatory parameters. It has got default port number and will bind to all available IP addresses. For the client one or more server IP addresses should be provided. All other parameters are optional.

The Server

The most important modification in the server is the call to sctp_bindx(). In the implementation without multi-homing (one-to-many_advanced branch) it was like this:

if(sctp_bindx(server_fd, (struct sockaddr*)&bind_addr, 1, SCTP_BINDX_ADD_ADDR) == -1) {

The address count is hardcoded to one and bind_addr is single struct sockaddr. Now the server should be able to bind to more than one address, so the call to sctp_bindx() now is reworked to:

if(sctp_bindx(server_fd, (struct sockaddr*)&local_addrs, local_addr_count, SCTP_BINDX_ADD_ADDR) == -1) {

Here local_addrs is an array of struct sockaddr-s and local_addr_count is the number of elements in it.

The list of local addresses is an optional parameter for the server. If the user doesn't provide any, the server binds to all possible addresses. This job is done in generate_default_addresses():

int generate_default_addresses(struct sockaddr_in* bind_addrs, int* count, const int server_port)
{
    memset(bind_addrs, 0, sizeof(struct sockaddr_in));
    bind_addrs->sin_family = ADDR_FAMILY;
    bind_addrs->sin_port = htons(server_port);
    bind_addrs->sin_addr.s_addr = INADDR_ANY;

    *count = 1;

    return 0;
}

The rest of the changes mainly involve command line parameters handling, so I will not review them.

The Client

The command line arguments handling and IP address parsing in the client is identical to the one in the server. Small difference is that the client requires list of remote IP addresses, which are used to connect to the server.

Usually a client doesn't need to bind to specific IP address and port. However when multi-homing is used, the client might have to bind only to specific subset of IP addresses. This is accomplished with a call to sctp_bindx():

if(local_ipaddr_list != NULL) {
    if(sctp_bindx(client_fd, (struct sockaddr*)&local_addrs, local_addr_count, SCTP_BINDX_ADD_ADDR) == -1) {
        perror("bind");
        return 9;
    }
}

This way the SCTP stack will announce only the listed IP addresses to its peer. Pay attention to the parse_addresses() function call:

if(local_ipaddr_list != NULL) {
    if(parse_addresses(local_ipaddr_list, strlen(local_ipaddr_list), 0, local_addrs, MAX_MULTIHOMING_ADDRESSES, &local_addr_count)) {
        return 7;
    }
}

The third parameter is the port number filled in sin_port field of struct sockaddr_in. Setting it to 0 lets the SCTP stack to use random port number for the association. I recommend this approach, unless you really need to use specific port number.

After the socket is bound, a connection to the server needs to be initiated. There are two options here - either connect to single address and let the server announce its IP addresses or pass more than one IP address to sctp_connectx(). The advantage of the latter approach is redundancy - if the primary IP address of the server is unreachable, the association will be established via one of the others. sctp_connectx() also accepts array of struct sockaddr as input parameter:

sctp_assoc_t assoc_id = 0;
if(sctp_connectx(client_fd, (struct sockaddr*)&remote_addrs, remote_addr_count, &assoc_id) == -1) {
    perror("sctp_connectx");
    return 11;
}

The array with the struct sockaddr-s is remote_addrs. The number of the elements in the array is remote_addr_count.

For message sending sctp_sendmsg() is used. Its declaration is:

int sctp_sendmsg(int sd, const void * msg, size_t len,
                        struct sockaddr *to, socklen_t tolen,
                        uint32_t ppid, uint32_t flags,
                        uint16_t stream_no, uint32_t timetolive,
                        uint32_t context);

The fourth parameter (struct sockaddr *to) is the destination IP address. The preferred destination IP address for the message can be controlled with this parameter. If this address is not reachable the SCTP stack will select another and will still try to deliver the message.

Message receiving hasn't got any multi-homing specifics.

A Few Words About One-to-One Style Sockets

Everything discussed up to now is also valid for one-to-one sockets. Binding is done with sctp_bindx() and connecting to the server - with sctp_connectx(). For sending data sctp_send() can be used, which is the SCTP alternative of send(). Here is its declaration:

int sctp_send(int sd, const void * msg, size_t len,
                    const struct sctp_sndrcvinfo *sinfo,
                    uint32_t flags);

It is like sctp_sendmsg(), but without the address parameter. This function is very convenient for the cases when you don't have to pick destination IP addresses and you don't want to use notifications.

Run the Code

Now let's test the client and the server. Run the server on alpha:

./server '192.168.35.10;192.168.45.10' 4444

And the client on beta:

./client -l '192.168.35.11;192.168.45.11' -r '192.168.35.10;192.168.45.10' -p 4444

They will exchange some messages and the client will quit. You can investigate the association with Wireshark. What you see should be similar to the results in the previous post.

Conclusion

In many posts I have advertised multi-homing as one of the SCTP's killer features. I avoided this topic many times and promised dedicated posts for it. I did this on purpose and I hope you are now convinced that with the appropriate foundations multi-homing is easy to implement. In many cases you might not need multi-homing support in your application, but my advise is to design your code in a way which allows you to enable it with minimum effort. If your app uses SCTP, in 99% of the cases someone will want to use multi-homing with it.

Thanks for reading and don't hesitate to comment, if you have got any questions!