Learn how to add your own custom commands to your QNX network driver using lessons learned from a real project described below.
QNX defines a set of IOCTL commands that a driver must implement in order to support PTP. These are defined in usr/include/netdrvr/ptp.h.
#define PTP_GET_RX_TIMESTAMP 0x100 /* get RX timestamp */ #define PTP_GET_TX_TIMESTAMP 0x101 /* get TX timestamp */ #define PTP_GET_TIME 0x102 /* get time */ #define PTP_SET_TIME 0x103 /* set time */ #define PTP_SET_COMPENSATION 0x104 /* set compensation */ #define PTP_GET_COMPENSATION 0x105 /* get compensation */
Along with the IOCTL codes above there are data structures that define the format of the data transferred by each command. For example, the get/set time commands use the following:
/* get/set time */ typedef struct { int32_t sec; /* ptp clock seconds */ int32_t nsec; /* ptp clock nanoseconds */ } ptp_time_t;
These commands are used by the PTP daemon to implement the PTP protocol.
We wanted to add the following commands to extend the PTP capability:
- Enable 1PPS (1 Hz pulse-per-second) output
- Disable 1PPS
- Set Tx and Rx latency adjustment
This was done by defining our own IOCTL codes and data structures:
#define PTP_ENABLE_PPS 0x200 /* PPS output enable */ #define PTP_DISABLE_PPS 0x201 /* PPS output disable */ #define PTP_SET_LATENCY 0x202 /* Set Tx/Rx latency adjustment */ typedef struct { uint32_t tx; /* Tx path latency (in nanoseconds) */ uint32_t rx; /* Rx path latency (in nanoseconds) */ } ptp_latency_t;
The PTP IOCTL commands are sent by the application to the driver using the driver-specific command SIOCSDRVSPEC, which takes a struct ifdrv object as an argument. This is defined in usr/include/net/if.h.
struct ifdrv { char ifd_name[IFNAMSIZ]; /* if name, e.g. "en0" */ unsigned long ifd_cmd; /* command code */ size_t ifd_len; /* length of data buffer */ void *ifd_data; /* pointer to data buffer */ };
We created a user-space utility to issue the commands. A network device does not have an associated device node in the file system so an IOCTL is sent via a socket created as follows:
int s = socket(AF_INET, SOCK_DGRAM, 0);
The PPS enable/disable commands do not require any data so these were straight-forward to implement. All that was required was a struct ifdrv object.
static int enable_pps(int s, char *if_name, int enable) { struct ifdrv ifd; strncpy(ifd.ifd_name, if_name, sizeof(ifd.ifd_name)); ifd.ifd_cmd = (enable) ? PTP_ENABLE_PPS : PTP_DISABLE_PPS; ifd.ifd_len = 0; ifd.ifd_data = NULL; if (ioctl(s, SIOCGDRVSPEC, &ifd) == -1) { warn("SIOCGDRVSPEC %s", ifd.ifd_name); return (-1); } return 0; }
At the destination the driver’s IOCTL handler extracts the pointer to the IOCTL data. The OS takes care of ensuring that the pointer makes sense in the receiving address space.
int lan743x_ioctl(struct ifnet *ifp, unsigned long cmd, caddr_t data) { lan743x_dev_t *lan743x; struct ifdrv *ifd; int result = EOK; if (ifp == NULL) { return EINVAL; } lan743x = ifp->if_softc; switch (cmd) { case SIOCSDRVSPEC: /* Driver-specific set command */ case SIOCGDRVSPEC: /* Driver-specific get command */ ifd = (struct ifdrv *)data; result = lan743x_drvspec_ioctl(lan743x, ifd); break; ...
The situation becomes more complicated when the IOCTL command includes a data buffer. The pointer to this buffer is in the context of the calling address space (the user-space utility) but will not make sense at the driver, which operates in the context of the io-pkt network stack process.
The QNX ioctl() reference page has the following statement about commands with data buffers:
In QNX Neutrino 6.5.0 and later, ioctl() handles embedded pointers, so you don’t have to use ioctl_socket() instead.
This lead me to believe that the buffer address would be automatically translated. This turned out to be an incorrect assumption but more about that later.
The latency adjustment command in the user-space application was set up as follows:
static int ptp_latency_set(int s, char *if_name, unsigned tx_latency, unsigned rx_latency) { struct { struct ifdrv ifd; ptp_latency_t latency; } cmd; struct ifdrv *ifd = &cmd.ifd; ptp_latency_t *adj = &cmd.latency; strncpy(ifd->ifd_name, if_name, sizeof(ifd->ifd_name)); ifd->ifd_cmd = PTP_SET_LATENCY; ifd->ifd_data = adj; ifd->ifd_len = sizeof(*adj); adj->tx = tx_latency; adj->rx = rx_latency; if (ioctl(s, SIOCSDRVSPEC, &cmd) == -1) { warn("SIOCSDRVSPEC %s", ifd->ifd_name); return (-1); } return 0; }
At the driver the latency values were extracted as follows:
#define IFD_DATA(ifd) (((uint8_t *)ifd) + sizeof(*ifd)) static int lan743x_ptp_copyin(void *dst, const void *src, size_t length) { int result = EOK; if (ISSTACK) { result = copyin(src, dst, length); } else { memcpy(dst, src, length); } return result; } static int lan743x_ptp_latency_update(lan743x_dev_t *lan743x, struct ifdrv *ifd) { ptp_latency_t latency; if (ifd->ifd_len != sizeof(latency)) { return EINVAL; } uint32_t *data = (void *)IFD_DATA(ifd); /* * Get the adjustment value. */ int result = lan743x_ptp_copyin(&latency, IFD_DATA(ifd), sizeof(latency)); if (result == EOK) { uint32_t tx_latency = latency.tx; uint32_t rx_latency = latency.rx; ... } return EOK; }
When this code was run no errors were reported but the PTP latency were garbage. Investigating further we discovered the latency values were not being copied into the buffer in the driver address space.
Going back to the ioctl() documentation we looked more closely at the following:
Some ioctl() commands map to calls to fcntl(), tcgetattr(), tcsetattr(), and tcsetsid(). Other commands are transformed into devctl() commands, and the rest are simply passed to devctl(). Here’s a summary (for details, see the Devctl and Ioctl Commands reference):
The SIOCSDRVSPEC is one of the commands passed to devctl(). This command is defined as follows:
int devctl( int filedes, int dcmd, void *dev_data_ptr, size_t n_bytes, int *dev_info_ptr );
The first three parameters are those passed to ioctl(). The n_bytes parameter is defined as
The size of the data to be sent to the driver, or the maximum size of the data to be received from the driver.
Changing the ioctl() command above to the following devctl() command fixed the problem and passed the latency data values to the driver:
if (devctl(s, SIOCSDRVSPEC, &cmd, sizeof(cmd), NULL) == -1) { warn("SIOCSDRVSPEC %s", ifd->ifd_name); return (-1); }
To understand what was going we contacted QNX technical support who provided the following useful explanation of what was happening “under the hood”:
API call order:
ioctl -> (special handling for spcific cmds or ioctl_handler) ->devctl -> MsgSend (for one part iov)/MsgSendv (for multi part iov)
This tells us that the devctl() constructs a message to send to the network stack. We can do the same in our application. The following code replaces the devctl() call shown above.
io_devctl_t msg; iov_t iov[2]; msg.i.type = _IO_DEVCTL; msg.i.combine_len = sizeof(msg.i); msg.i.dcmd = SIOCSDRVSPEC; msg.i.nbytes = sizeof(cmd); msg.i.zero = 0; // Setup data to the device. SETIOV(&iov[0], &msg.i, sizeof(msg.i)); SETIOV(&iov[1], &cmd, sizeof(cmd)); if (MsgSendv(s, iov, 2, NULL, 0) == -1) { warn("SIOCSDRVSPEC %s", ifd->ifd_name); return -1; }
This clearly reveals the message passing architecture of QNX operating system.
