EK9 Network Types

These EK9 Network Types are all related to network communications or protocols.

The Types

Worked Example

For each of the network types above there are examples. There are some explanations of the examples; but assuming you've read many of the other sections on EK9; the syntax and rationale/approach should start to be familiar. If you've not read the other sections on Built in Types, Collection Types and Standard Types, you should read those first.

UDP Examples

There are two server examples and two client examples for UDP; each of which shows different ways that UDP could be used. These range from imperative type examples to more functional examples utilising Stream pipelines. The initial packet size of a UDPPacket is 512 bytes, you can alter the size of the packet but this may make it more likely that packets will be dropped by the network. You must remember that UDP packets are not guaranteed to be delivered (this is not an EK9 thing, it is the very nature of UDP). Whilst this may seem strange; it does work well for some types of application that can accept some data loss.

UDP Server (1)

Below is a simple imperative code style of processing (echoing) UDP messages (until the message 'end' is received). It listens on port 4445 and blocks to receive() packets, it echos the packet back.

#!ek9
defines module introduction
  defines program
    UDPServer1()
      stdout <- Stdout()
      udpConnection <- UDP(NetworkProperties(port: 4445))

      while udpConnection?
        packet <- udpConnection.receive()
        if packet?
          if packet.content == "end"
            endPacket <- UDPPacket(packet.properties, "OK will stop Server")
            udpConnection.send(endPacket)
            udpConnection.close()
          else
            udpConnection.send(packet)

      stdout.println("UDP Server Complete")

//EOF

If you wanted to use the iterator syntax, the code above could have been written as follows:

  • while udpConnection?
  •    packet udpConnection.receive()
  • ...
  • while udpConnection.hasNext()
  •    packet udpConnection.next()

If you are from a Java/C/C++/C# or Python background this will probably be the way you would have thought of coding up a simple UDP server. However, there is an alternative approach that those with a more functional background may prefer - take a look you may also find this approach appealing.

UDP Server (2)

This example has broadly the same functionality as Server (1) above, except that it:

As you can see with this example, the EK9 Stream pipeline used to continuously accept incoming UDP packets then pass them through a function (just echos content back) and then send what it echoed back to 'Standard Out'.

#!ek9
defines module introduction
  defines function
    UDPPacketHandler() as abstract
      -> incoming as UDPPacket
      <- outgoing as UDPPacket?

  defines program
    UDPServer2()
      ->
        port as Integer
        packetSize as Integer
        signal as String

      stdout <- Stdout()
      udpConnection <- UDP(NetworkProperties(port, packetSize))

      //This time stop server with abort signal not content 'end'.
      abortHandler <- (udpConnection) is SignalHandler as function
        result: 0
        stderr <- Stderr()
        stderr.println("Received abort Signal closing connection")
        udpConnection.close()

      Signals().register(signal, abortHandler)

      echoFunction <- (udpConnection) is UDPPacketHandler
        outgoing = incoming
        udpConnection.send(outgoing)

      cat udpConnection | map by echoFunction > stdout
      stdout.println("UDP Server Complete")

//EOF

Clearly there is a bit more code to this example, but then it is doing more. Handling command line parameters (though these could have come from Environment Variables if you prefer). It also terminates via signal handling rather than message content.

Importantly it uses a dynamic function that extends an abstract function to process the incoming packet. With this approach it is quite possible to be able to create any amount of processing both before the packet content is echoed back and after echoing.

For example; incoming packets could be filtered before echoing to check for 'band words' and those could be recorded together with the originating hostname/ip. The number of times specific words were sent in could be accumulated and statistics gathered. Content length above a certain size could be ignored and not responded to.

In short the functional approach enables more flexibility and future extension, rather than mixing all that varied logic inside the imperative processing loop. But this is an aside to the functionality in the UDP class.

UDP Client (1)

This example shows how a client using UDP could connect to one of the servers defined above.

#!ek9
defines module introduction
  defines program
    UDPClient1()
      stdout <- Stdout()
      stderr <- Stderr()

      try //use a try block with a closable resource, auto close
        -> client <- UDP(NetworkProperties(timeout: 250ms))

        packet1 <- UDPPacket(NetworkProperties("localhost", 4445), "A Test")
        stdout.println(`Sending [${packet1}]`)
        client.send(packet1)
        stdout.println("Waiting for response")
        response1 <- client.receive()
        if response1?
          stdout.println(`Received [${response1}]`)
        else
          stderr.println(`There was no response [${client.lastErrorMessage()}]`)

        packet2 <- UDPPacket(NetworkProperties(host: "localhost", port: 4445), "end")
        stdout.println(`Sending [${packet2}]`)
        client.send(packet2)
        stdout.println("Waiting for response")
        response2 <- client.receive()
        if response2?
          stdout.println(`Received [${response2}]`)
        else
          stderr.println(`There was no response [${client.lastErrorMessage()}]`)

      stdout.println("UDP Client Complete")
//EOF

As you can see this example also uses the UDP network class, but this time no port parameter is given. Also note that the UDP instance is both created and a timeout is set to receive packets. This is done within the incoming try parameter, this means that the resource will be automatically closed.

The detail of where to send the content for the UDP message is all bound in the creation of the UDPPacket, this is then sent out via the UDP class.

Note that the example above shows that the client is expecting a response, depending on the server logic, this may not always be the case. But this client is expecting some unreliability in message transmission as so has set a timeout of 250 milliseconds and does check if the response packet is valid. In the case of not receiving a packet withing the timeout the UDP class will return and empty packet.

As you can see the client code above is again quite imperative/procedural in nature.

UDP Client (2)

This example shows how a client using UDP could connect to one of the servers defined above. But this example is more functional and also processes a number of command line parameters.

#!ek9
defines module introduction
  defines type
    List of String

  defines function
    packetConverter() as abstract
      -> content as String
      <- outgoing as UDPPacket?

  defines program
    UDPClient2()
      ->
        hostname as String
        port as Integer
        timeout as Millisecond
      try
        -> udpConnection <- UDP(NetworkProperties(timeout))

        //This is the nearest we get to a lambda, inline dynamic function
        //You must use () for the single line you are allowed.
        cat ["A Test", "Middle Item", "end" ]
          | map (hostname, port) extends packetConverter (outgoing: UDPPacket(NetworkProperties(hostname, port), content))
          > udpConnection

      Stdout().println("UDP Client Complete")
//EOF

Just like the second server example this client expects a number of command line parameters, the initial setup of the UDP connection is the same as in the first client example. But this time the messages to be sent are just held in a List of String. These messages are then pipelined via a dynamic function that just wraps them in a UDPPacket with the correct hostname address and port. They are then pushed out via the UDP connection using the > operator.

Interestingly the dynamic function is simple and has been inlined. It could have been held as a function delegate or created using a higher level function. Here it has just been inlined.

Note in this case, the client is not bothered whether the server responded or not. Because the UDP connection was opened in the try incoming parameter block it will be automatically closed.

There is something about the functional approach that is more elegant in my opinion. Though I'm less keen on the inline dynamic function.

UDP Summary

As you can see from the above examples you can use a procedural or functional approach to working with UDP. The UDP protocol is quite useful for a range of application as long as you accept its limitations.

TCP Examples

There are a number of examples for TCP (both client and server); each of which shows different ways that TCP could be used. The mechanism for TCP is different from UDP - TCP is connection oriented and more reliable than UDP.

TCP Server (1)

The TCP server example below is again written in a procedural manner. The key conceptual difference between UDP and TCP is the connection oriented nature and the need to be able to service multiple client connections at the concurrently.

TCP server connections can be fully available on a port via the network or just local (useful for just controlling local applications). The TCP constructor takes a NetworkProperties record which has a range of properties to control the back log (the number of connections that are waiting to connect). It also can limit the number of concurrent connections to service at the same time. This is the first time multi-threading has been mentioned.

As an aside it is also possible to configure a read timeout on the connection so that if clients are slow in sending data, the connection can be dropped so another client can be serviced.

#!ek9
defines module introduction
  defines program  
    TCPServer1()
      stdout <- Stdout()
      stderr <- Stderr()

      stdout.println(`Version of program package is ${Package().version}`)

      tcpServer <- TCP(NetworkProperties(4446, 5, 4, true))

      if tcpServer?
        handler <- (tcpServer, stderr) extends TCPHandler as function
          stderr.println("Connection made")
          while input?
            line <- input.next()
            stderr.println(`Received [${line}]`)
            output.println(line.trim().upperCase())
            if line == "end"
              input.close()
            if line == "shutdown"
              tcpServer.close()
          stderr.println("No more input on connection")

        stdout.println("Awaiting connections")
        if tcpServer.accept(handler)
          stdout.println("TCP Server completed and shutdown")
        else
          stderr.println(`Was not able to service requests gracefully [${tcpServer}]`)
      else
        stderr.println(`Failed to get TCP server running [${tcpServer}]`)
//EOF

Note the the TCPHandler is a built-in abstract function with the following signature:

...
    defines function
      TCPHandler as abstract
        ->
          input as StringInput
          output as StringOutput
...
    

So when you define your handler, you get the input and output arguments inferred within your function implementation.

The TCPHandler plays a really important role in processing the actual requests made by TCP clients. The code in this handler will be called by multiple clients at the same time and therefore should not hold state unless it is protected and design to allow multiple (possibly) simultaneous accesses (more on this later).

The line tcpServer.accept(handler) puts the software into a mode where it now blocks and waits for clients to initiate a connection (on port 4446 in this example).

Once a client does connect; the handler(input, output) is called and the client requests can be serviced by using the input and output as these provide the connection from and to the client. This servicing will continue until the client disconnects (or in the case of this example sends "end").

The important point to note here; is that the tcpServer goes back to listening for more connections while the handler is still running. This is where the multi-threaded nature comes in.

Note that if the client sends "shutdown" in this example that actual server will close is listening port and hence will complete and exit (this would not normally be coded like this - see the next example for an alternative).

TCP Server (2)

#!ek9
defines module introduction

  defines function           
    serviceConnections()
      ->
        tcpServer as TCP
        handler as TCPHandler
      <-
        response as String: `TCP Server on ${tcpServer} completed and shutdown`

      if not tcpServer.accept(handler)
        response: `Was not able to service requests gracefully on ${tcpServer}`

    createShutdownHandler
      ->
        controlServer as TCP
        processingServer as TCP
        shutdownCommand as String
      <-
        rtn as TCPHandler: (controlServer, processingServer, shutdownCommand) is TCPHandler as function
          while input?
            command <- input.next()
            if command == shutdownCommand
              output.println("Shutting down")
              processingServer.close()
              controlServer.close()
            else
              output.println(`Ignoring [${command}]`)

    createHandler()
      <- rtn as TCPHandler: () is TCPHandler as function
          cat input | map toUpperCase > output

    toUpperCase()
      -> input as String
      <- output as String: input.upperCase()

    shutdownFunction()
      -> tcpClient as TCP
      <- response as String: `Connection failed ${tcpClient}`

      if tcpClient?
        try
          -> connection <- tcpClient.connect()
          if connection?
            connection.output().println("SHUTDOWN")
            response: `Received response [${connection.input().next()} from ${tcpClient}]`
          else
            response: `Failed to connect on [${tcpClient}]`
      else
        response: `Failed to get TCP client running [${tcpClient}]`

  defines program
    TCPServer2()
      ->
        processingPort as Integer
        controlPort as Integer
        shutdownCommand as String

      stdout <- Stdout()

      //Alternative layout as there are a number of properties being set.
      processingServer <- TCP(
        NetworkProperties(
          port: processingPort,
          timeout: 250ms,
          backlog: 2,
          maxConcurrent: 4,
          localOnly: false)
        )

      controlServer <- TCP(NetworkProperties(port: controlPort, localOnly: true))

      if controlServer? and processingServer?

        shutdownControl <- (controlServer, processingServer, shutdownCommand) extends serverFunction
          response: serviceConnections(
            tcpServer: controlServer,
            handler: createShutdownHandler(controlServer, processingServer, shutdownCommand)
          )

        processing <- (processingServer) is serverFunction
          response: serviceConnections(
            tcpServer: processingServer,
            handler: createHandler()
          )

        //Now run both of those serverFunctions concurrently.
        cat [shutdownControl, processing] | async > stdout
        //Once the serverFunctions both finish this program will finally exit.
      else
        Stderr().println(`Failed to start servers, control: ${controlServer} processing: ${processingServer}`)
//EOF

This server example is much longer, but does a few things that are more realistic for a real application. If you review the program TCPServer2, you can see that again command line parameters are processed.

The next thing the program does is create two TCP server connections.

The first (processingServer) is a TCP server that processes connections from the network with a read timeout of 250 milliseconds with a backlog of 2 connections and concurrent processing for up to 4 clients simultaneously. Note the use of a different layout and named parameters. This makes the configuration a little easier to see with so many parameters.

The second (controlServer) is also a TCP server that only processes local connections to shut the whole application down (much like Tomcat works).

Following this; two dynamic functions (serverFunction) are created that returns serviceConnections. Importantly that function accepts a TCPHandler that can deal with connected clients.

Finally the two serverFunctions are added to a List, this list is then used in a Streams pipeline. The important part here is that the pipeline uses the async command to run both of the serverFunctions concurrently. Clearly for this example that is critical.

The program then effectively blocks in the tcpServer.accept(handler) for both TCP servers, as client connections are made the appropriate handler deals with this client communications.

The main processing function is implemented in the createHandler function. It simply processes the input content from the client, converts it to upper case and sends that data to the output.

The createShutdownHandler function deals with accepting input and checking it is the same as the shutdown command. If the command matches then both of the TCP server sockets are closed. This then causes all processing to be shutdown. It is only at this point the main program line 'cat servers | async > stdout' finally completes as those functions return.

The important take away from the example above is that network communications (and signal handling) is inherently multi-process/threaded/concurrent in its very essence. EK9 provides a way to set the stage with the provision of handlers and functions and then enables concurrent processing to take place via the async part of the stream pipeline and deals with all the joining of calls. There is no call back hell, 'multi-tasking' is implied but its implementation is not explicit. So the implementation detail of how the two TCP services are actually managed by EK9 is an implementation detail and not one for the developer.

TCP Client (1)

The example below shows a procedural style TCP client application. The connection to the server is just closed at the end of the try block. The server will deal with this by detecting there is no more input and there it will close all its connections to this client.

#!ek9
defines module introduction
  defines program
    TCPClient1()
      stdout <- Stdout()
      stderr <- Stderr()

      tcpClient <- TCP(NetworkProperties("localhost", 4446))

      if tcpClient?
        stdout.println(`Will attempt to connect to server ${tcpClient}`)
        try
          -> connection <- tcpClient.connect()
          if connection?
            connection.output().println("Short Message")
            stdout.println(`Received response [${connection.input().next()}]`)
            //End of the try will just close the connection.
          else
            stderr.println(`Failed to connect [${tcpClient}]`)
      else
        stderr.println(`Failed to get TCP client running [${tcpClient}]`)

//EOF

The above example is quite short and to the point, the next client example has a little more functionality.

TCP Client (2)

More of a functional approach and is designed to connect to the TCP Server (2) example above, ie connect to both the processing port and the control port to shut the server application down.

#!ek9
defines module introduction

  defines function  
    shutdownFunction()
      -> tcpClient as TCP
      <- response as String: `Connection failed ${tcpClient}`

      if tcpClient?
        try
          -> connection <- tcpClient.connect()
          if connection?
            connection.output().println("SHUTDOWN")
            response: `Received response [${connection.input().next()} from ${tcpClient}]`
          else
            response: `Failed to connect on [${tcpClient}]`
      else
        response: `Failed to get TCP client running [${tcpClient}]`
        
    processingFunction() as abstract
      -> input as String
      <- response as String?

  defines program
    TCPClient2()
      ->
        processingPort as Integer
        controlPort as Integer

      stdout <- Stdout()
      processingClient <- TCP(NetworkProperties("localhost", processingPort))

      if processingClient?
        try
          -> connection <- processingClient.connect()
          sender <- (connection) is processingFunction as function
            connection.output().println(input)
            response: connection.input().next()

          cat ["one", "two", "Alpha", "Beta", "Zeta", "end", "begin"] | map with sender > stdout

      cat Optional(TCP(NetworkProperties("localhost", controlPort))) | map with shutdownFunction > stdout

//EOF

There are a couple of interesting ideas in this example, the first is using a sender function to accept incoming String values send them to the processing server and get the server response and send it along a pipeline and in this case to stdout.

The second interesting point is to actually send a TCP object through a processing pipeline into a shutdown function; that sends "SHUTDOWN" to that TCP server by connecting, it then sends the result of what happened down the pipeline. This might seem like a strange approach, but it is just another different way of treating Objects and functions in a more functional manner.

TCP Summary

Both client and server TCP connections can be constructed and utility in both a procedural and functional manner. There is a reasonable level of control of the TCP connectivity and the possibility to handle multiple concurrent connections.

HTTP

EK9 does have Web Services; these are built on top of the HTTP server. The built-in HTTP server is not designed to be as massively configurable as implementations like Tomcat, Jetty, NGINX or IIS. It is expected that you would front any EK9 web implementation with a caching proxy server and would deploy instances of your application via a range of scalable containers.

EK9 Web Services can serve simple html content; but are really designed to serve dynamic content. It does however have built in logic and types to be able to support full REST services.

The built-in types HTTPRequest and HTTPResponse have all the necessary structures to support the REST maturity model up to level 2. To move to level 3 (HATEOAS) the developer must define and provide the content (HAL/ATOM or bespoke). This means that etag and last modified headers are 'baked in'. So verbs like PUT, PATCH and DELETE can be used to EK9 mandates the use of headers to ensure up-to-date versions are being referenced. RESTFULAPI has more details on this if you are new to REST.

Conclusion

EK9 has included networking types in as part of the standard/network API right from the start because they are a key component in any modern programming language. It must be remembered that when developing any sort of networking type application UDP/TCP/HTTP there will be multiple concurrent threads of execution and that networks can and do fail.

Designing your applications to be as stateless and pure as possible is recommended. However, there are times when concurrent access to shared data is necessary and EK9 provides a MutexLock for this situation.

Next Steps

This section has covered a number of programs, but these are explained in more detail in the next section on programs.