1. Network programming

Access to the network is as common nowadays just like access to the local file system. Since Java 1.0 Java offers a network API for developing client-server applications. The Java library can establish encrypted connections and also brings support for the Hypertext Transfer Protocol (HTTP). The exercises in this chapter are about getting resources from a web server and developing a small client-server application with its own protocol.


  • know the URL class

  • know input/output streams

  • be able to read internet resources

  • be able to implement client and server with Socket and `ServerSocket

Data types used in this chapter:

1.1. URL and URLConnection

In Java, the obvious class URL represents a URL and URI a URI. An HTTP connection can be opened via URL, internally this is handled by the class URLConnection.

Both classes are not very comfortable for modern HTTP calls; only in Java 11 a new package java.net.http has been added with HttpClient in the core. Java enterprise frameworks have more solutions, and there are many alternatives in the open source universe as well:

  • Client API in Jakarta EE

  • WebClient in Spring Webflux

  • OkHttp

  • Apache HttpClient

  • Feign and Retrofit

1.1.1. Download remote images via URL ⭐

The URL class provides a method that returns an InputStream so that the bytes of the resource can be read.

Captain CiaoCiao likes to relax with pictures of well-shaped ships on shiphub.com. He would like to have some pictures on his storage device so that he has something to daydream about on long voyages.


  • For a given URL, write a program that downloads the resource and stores it on the local file system.

  • The file name should be based on the URL.

1.1.2. Read remote text file from URL ⭐

The Center for Systems Science and Engineering (CSSE) at Johns Hopkins University publishes a CSV file of covid disease data around the world every day at https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_daily_reports. For March 1, 2021, the URL on the server is: https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/03-01-2021.csv.

The file starts with

,,,Afghanistan,2020-10-22 04:24:27,33.93911,67.709953,40510,1501,33824,5185,Afghanistan,104.06300129769207,3.7052579609972844
,,,Albania,2020-10-22 04:24:27,41.1533,20.1683,17948,462,10341,7145,Albania,623.6708596844812,2.574102964118565
,,,Algeria,2020-10-22 04:24:27,28.0339,1.6596,55081,1880,38482,14719,Algeria,125.60932701190256,3.4131551714747372


  • Create a new class CoronaData with a new method String findByDateAndSearchTerm(LocalDate date, String search).

    • findByDateAndSearchTerm(…​) should build a URL object with the date. Note that the filename has the order month, day, year.

    • Open an input stream to the generated URL, read the stream line by line, and filter out all lines that do not contain the passed substring search.

    • The end result is a string with all lines containing the search word, no need for a CSV parser.


  • The call findByDateAndSearchTerm( LocalDate.now().minusDays( 1 ), "Miguel" ) returns from the remote CSV document all Corona numbers from yesterday that contain "Miguel".

  • Sample return:

    8113,San Miguel,Colorado,US,2020-10-22 04:24:27,38.00450883,-108.4020725,100,0,0,100,"San Miguel, Colorado, US",1222.643354933366,0.0
    35047,San Miguel,New Mexico,US,2020-10-22 04:24:27,35.48014807,-104.8163562,151,0,0,151,"San Miguel, New Mexico, US",553.5799391428677,0.0

If the CORONA numbers go down it is quite possible that CSSE will stop publishing new documents. The old documents should stay.

1.2. HTTP client

Although URLConnection can be used to make an HTTP request, changing the HTTP method (GET, POST, PUT …​) and setting headers, this is not comfortable. Therefore, the HTTP client was added in Java 11. With the new API, HTTP resources can be obtained more elegantly over the network. Moreover, the HTTP client supports HTTP/1.1 and HTTP/2 as well as synchronous and asynchronous programming models. We will use this API to solve some exercises; those who cannot use Java 11 can find a similar library for Java 8 at https://github.com/AsyncHttpClient/async-http-client. In the next chapter on file formats, we will also return to the HTTP client in the exercises, since JSON or XML are often exchanged.

1.2.1. Top news from Hacker News ⭐⭐

Hacker News is a website with up-to-date discussion of technology trends. Articles can be accessed via a web service, and documentation can be found at https://github.com/HackerNews/API. Two endpoints are:


  • Create a new class HackerNews.

  • Implement a new method long[] hackerNewsTopStories() which will

  • Implement a new method String news(long id) that returns as string the complete JSON document.


  • The class with the two methods can be used like this:

    System.out.println( Arrays.toString( hackerNewsTopStories() ) );
    String newsInJson = news( 24857356 );
    System.out.println( newsInJson );

1.3. Socket and ServerSocket

Operating systems provide sockets for TCP/UDP communication; Java can use them via the classes

  • java.net.Socket and java.net.ServerSocket for TCP and

  • java.net.DatagramSocket for UDP.

Objects of Socket and ServerSocket are created via the constructor or even better via the factories javax.net.SocketFactory and javax.net.ServerSocketFactory; for UDP there is no factory.

1.3.1. Implement a swear server and a client ⭐⭐

Bonny Brain is going to participate in the next swearing contest soon. She wants to prepare perfectly. A server should run an application that manages slurs and clients can connect to this slur server and search for sentences.


  • Write a server and client.

  • Customize the server to accept multiple connections; use a thread pool.

  • The thread pool should use a maximum number of threads to prevent denial-of-service (DOS) attacks. If the maximum number of concurrent connections is exhausted, a client must wait until another connection becomes available.


  • After starting the server and client, an interaction may look like this:

    You, sir, are an oxygen thief!
    You, sir, are an oxygen thief!
    Stop trying to be a smart ass, you're just an ass.

1.3.2. Implement a port scanner ⭐⭐

Bonny Brain installs the new Ay! OS, but important analysis tools are missing. It needs a tool that detects and reports the occupied TCP/UDP ports.


  • Write a program that tries to register a ServerSocket and DatagramSocket on all TCP/UDP ports from 0 to 49151; if it succeeds, the port is free, otherwise it is busy.

  • Display the occupied ports on the console, and in addition, for the known ports, a description of the usual service that occupies that port.


  • The output might look like this:

    Protocol   Port       Service
    TCP         135       EPMAP
    UPD         137       NetBIOS Name Service
    UPD         138       NetBIOS Datagram Service
    TCP         139       NetBIOS Session Service
    TCP         445       Microsoft-DS Active Directory
    TCP         843       Adobe Flash
    UPD        1900       Simple Service Discovery Protocol (SSDP)
    UPD        3702       Web Services Dynamic Discovery
    TCP        5040
    UPD        5050
    UPD        5353       Multicast DNS
    UPD        5355       Link-Local Multicast Name Resolution (LLMNR)
    TCP        5939
    TCP        6463
    TCP        6942
    TCP       17500       Dropbox
    UPD       17500       Dropbox
    TCP       17600
    TCP       27017       MongoDB
    UPD       42420

A network interface connects computers via a computer network. In the following we always assume a TCP/UDP interface. The network interface does not have to be physical, but can also be implemented in software. Like the loopback interface with the IP (IPv4) or ::1 (IPv6). Typical network interfaces continue to exist for the LAN or WLAN. Operating system tools can display all network interfaces, such as ipconfig /all under Windows or ip a under Linux. In Java the network interfaces can be retrieved via java.net.NetworkInterface. Each network interface has its own IP address.

To register a server socket there are two possibilities: either the service only accepts requests from a special local InetAddress or the service accepts requests from all local addresses. So in principle it is possible to bind the same port several times on one network card, because on one network card any number of network interfaces can be configured, because are have distinguishable IP addresses.

The solution of the task can perform a simple test and register the socket on all network interfaces; if this fails, a service was already active on one of the network interfaces. This is enough for us as a criterion that on some network interface the port is busy.

1.4. Suggested solutions

1.4.1. Download remote images via URL

public static void downloadImage( URL url ) throws IOException {
  try ( InputStream inputStream = url.openStream() ) {
    String filename = url.toString().replaceAll( "[^a-zA-Z0-9_.-]",
                                                 "_" );
    Files.copy( inputStream, Paths.get( filename ),
                StandardCopyOption.REPLACE_EXISTING );

The solution involves three steps.

  1. The URL object returns an InputStream for the bytes of the resource with openStream(). Since you should close what you open, we put the opening of the stream in a try-with-resource block.

  2. The name of the target file is derived from the URL. We have to be a bit careful with the filename, because not every character in the URL is always a valid character for a filename. The Wikipedia page https://en.wikipedia.org/wiki/Filename#Comparison_of_filename_limitations summarizes the special characters for some file systems. Also not all characters may appear in a URL, but even the simple path separator / will cause problems. Therefore we replace all problematic characters. The regular expression [^a-zA-Z0-9_.-] chosen in the solution replaces all characters that are not letters, digits, _, . or - with an underscore. This gives a safe filename for each file system. However, the URLs are not unique because, for example, the URLs https://www.penisland.net/?; and https://www.penisland.net/?, would become http___www.penisland.net___.

  3. The second step elegantly uses the copy(…​) method of the Files class. There are two of these: one for reading from a file and writing to an OutputStream, and one for reading all data from an InputStream and writing to a file — this is the version we use. All bytes are read from the InputStream and written to the new destination file. The third parameter at copy(…​) is a vararg and stands for attributes: StandardCopyOption.REPLACE_EXISTING states, that existing files will be overwritten, otherwise there will be an exception at existing files.

1.4.2. Read remote text file from URL

private static final String URL_TEMPLATE =
    + "csse_covid_19_data/csse_covid_19_daily_reports/%s.csv";

public static String findByDateAndSearchTerm( LocalDate date, String search ) {
  DateTimeFormatter MMddyyyy = DateTimeFormatter.ofPattern( "MM-dd-yyyy" );
  String url = String.format( URL_TEMPLATE, date.format( MMddyyyy ) );

  try ( InputStream    is  = new URL( url ).openStream();
        Reader         isr = new InputStreamReader(is, StandardCharsets.UTF_8);
        BufferedReader br  = new BufferedReader( isr ) ) {
    return br.lines()
             .filter( line -> line.contains( search ) )
             .collect( Collectors.joining( "\n" ) );
  catch ( MalformedURLException e ) {
    System.err.println( "Malformed URL format of " + url );
  catch ( IOException e ) {
  return "";

The first step is to build the URL. The URL contains the date, which we receive as a parameter. However, it is not possible to call toString() method directly on the LocalDate object, because that returns a string according to the ISO 8601 notation; the smart Center for Systems Science and Engineering uses a different order when specifying the date segments. The program therefore builds up its own date formatting with the DateTimeFormatter by placing the month first, followed by the day and then the year.

From this dynamically generated string a URL object is built and the central method openStream() is called, which returns an InputStream. Since we want to read character by character, we convert this binary InputStream into a Reader. For line-by-line reading, the BufferedReader is well suited, in particular the lines() method is handy, which returns a Stream<String>. Using a filter, we leave the lines in the stream that contain the search string, and finally concatenate all the lines into one big string, which we return.

Two possible exceptions are possible: the URL may be formed incorrectly, in which case there is a MalformedURLException, or there are connection errors or errors during the read. We catch these exceptions, print messages on the screen and return an empty string, since nothing was found even in the error case.

1.4.3. Top news from Hacker News

private static final HttpClient client = HttpClient.newHttpClient();

public static long[] hackerNewsTopStories() {
  String url = "https://hacker-news.firebaseio.com/v0/topstories.json";
  HttpRequest request = HttpRequest
      .newBuilder( URI.create( url ) )
      .timeout( Duration.ofSeconds( 5 ) )

  try {
    HttpResponse<InputStream> response =
        client.send( request, HttpResponse.BodyHandlers.ofInputStream() );
    var scanner = new Scanner( response.body() ).useDelimiter( "[,\\[\\]]" );
    return scanner.tokens().mapToLong( Long::parseLong ).toArray();
  catch ( IOException | InterruptedException e ) {
    return new long[ 0 ];
public static String news( long id ) {
  String url = "https://hacker-news.firebaseio.com/v0/item/" + id + ".json";
  HttpRequest request = HttpRequest
      .newBuilder( URI.create( url ) )
      .timeout( Duration.ofSeconds( 5 ) )

  try {
    return client.send( request, HttpResponse.BodyHandlers.ofString() )
  catch ( IOException | InterruptedException e ) {
    return "";

For our example, we create a HttpClient as a class variable. Since our two methods are static, it is convenient to preconfigure the object. If object methods should access the HttpClient, and this still from several threads, then each thread should use its own HttpClient object.

The send(…​) method has two parameters:

<T> HttpResponse<T> send(HttpRequest request,
                         HttpResponse.BodyHandler<T> responseBodyHandler)
      throws IOException, InterruptedException;

The first thing to do is to build and configure a HttpRequest for the individual request. This can be done with HttpRequest.newBuilder() and HttpRequest.newBuilder(URI); the second version saves the later call to uri(…​). The timeout is optional and is set to 5 seconds by the program. The final build() method returns the HttpRequest object, which is executed by the HttpClient. This is the principle of the API: Different HttpRequest calls run over the HttpClient once it is built and configured.

The second parameter of the send(…​) method determines how the result should be fetched. For the following Scanner an InputStream is handy, which HttpResponse.BodyHandlers.ofInputStream() requests. From the HttpResponse<InputStream> response we read the content with body(), where the body is not the data for now, but just an InputStream of that data. This InputStream configures the Scanner. Encoding is not necessary in our case, because the JSON document consists of pure ASCII characters. However, the Scanner is configured with a delimiter that sees [ and , and ] as delimiters. The Scanner provides the useful method tokens() which gives us a stream of all tokens, in our case the numbers. mapToLong(…​) converts any textual representation of a number into a LongStream, and toArray() returns all elements of the stream as an array. If there are any errors, they are caught, reported, and an empty array is returned.

The second method, news(long id), builds the HttpRequest with a concrete id for a message. The implementation is a bit simpler because we don`t need to parse the output, and HttpResponse.BodyHandlers.ofString() determines that a string is returned as the result.

JSON documents from web services will generally be converted to Java objects. We’ll look at how to do this in the next chapter.

1.4.4. Implement a swear server and a client

The server’s job is to respond to incoming connections and pick out any insults that have a partial string.

public class SlangingMatchServer {

  private static final int PORT              = 10_000;
  private static final int MAXIMUM_POOL_SIZE = 10_000;

  public static void main( String[] args ) throws IOException {
    Executor executor = new ThreadPoolExecutor( 0, MAXIMUM_POOL_SIZE,
                                                60, TimeUnit.SECONDS,
                                                new SynchronousQueue<>() );

    try ( ServerSocket serverSocket =
              ServerSocketFactory.getDefault().createServerSocket( PORT ) ) {
      System.out.println( "Server running at port "+serverSocket.getLocalPort() );

      while ( Thread.currentThread().isInterrupted() ) {
        Socket socket = serverSocket.accept();
        executor.execute( () -> handleConnection( socket ) );

  private static void handleConnection( Socket socket ) {
    try ( socket;
          Scanner requestReader =
              new Scanner( socket.getInputStream(), StandardCharsets.UTF_8 );
          PrintWriter responseWriter =
              new PrintWriter( socket.getOutputStream(),
                               true, StandardCharsets.UTF_8 ) ) {
      String request = requestReader.nextLine();
      responseWriter.println( searchInsult( request ) );
    catch ( IOException e ) {

  private static String searchInsult( String search ) {
    return Stream.of( "You, sir, are an oxygen thief!",
                      "Stop trying to be a smart ass, you're just an ass.",
                      "Shock me, say something intelligent." )
                 .filter( s -> s.toLowerCase().contains( search.toLowerCase() ) )
                 .collect( Collectors.joining( "\n" ) );

The main(…​) method prepares a server socket on the predefined port 10,000, enters an infinite loop that can in principle be terminated with an interrupt, and waits for an incoming connection. The thread blocks at accept(), and the block does not unblock until there is an incoming client. In that case, we call our own handleCollection(..) method over a thread pool. The ThreadPoolExecutor is the thread pool, and so far we could always set it to Executors.newCachedThreadPool(), but then the number of concurrent connections is unlimited. Via the constructor of ThreadPoolExecutor the number of concurrent threads can be specified.

If accept() returns, the return is the client socket, and that becomes the argument of handleConnection(…​). We move the call to handleConnection(…​) to a Runnable and that is passed to the Executor, and so executed in the background. While the thread handles the client-server communication in the background, the infinite loop goes back to accept() to quickly serve the next interested party. At this point we are not allowed to close the socket, so it must not say:

try ( Socket socket = serverSocket.accept() ) {
  executor.execute( () -> handleConnection( socket ) );

The processing is asynchronous! The close() from the try-with-resources would otherwise be right after sending via execute(…​) and consequently the Socket would be closed quickly while handleConnection(…​) has just started communicating.

Lets have a look at handleConnection(…​): A Socket is AutoCloseable, and the try-with-resources closes the Socket at the end. We have the unusual case here that we don`t actually need to declare a new resource variable, but the abbreviated notation is only allowed since Java 9.

In communication, input and output streams are necessary. These are also resources to be closed via try-with resources. A string is read from the input stream, searched for the insult word, and the result is written back. The protocol requires strings to go over the wire, so the InputStream and OutputStream are upgraded to character-oriented types. The Scanner can be built in the constructor with an InputStream and an encoding and can then read a line with nextLine(). We write the output to a PrintWriter; again, the constructor can accept the OutputStream and the encoding. The second argument true is important, because it controls the flushing of the buffer on an end-of-line character. println (…​) writes the result to the PrintWriter, the line feed signals the flushing of the buffer. The catch block ends the try-with-resources, and all resources are closed and the socket is returned to the operating system as a native resource.

The utility method searchInsult(…​) checks if the search word is contained in the given strings, and concatenates all results with a newline.

Java 8 Backport

We have the unusual case here that we can directly close the passed socket in try-with-resources: try (socket; …​). This syntax is only possible since Java 9. For Java 8, we need to declare a resource variable, something like this: try (socket __ = socket; …​).

The client has a similar logic, but of course it doesn’t have to listen for incoming connections, it establishes them.

public class SlangingMatchClient {

  private static final String HOST = "localhost";
  private static final int    PORT = 10_000;

  public static void main( String[] args ) throws IOException {
    while ( true ) {
      String request = new Scanner( System.in ).nextLine();
      remoteSearchInsult( request );

  private static void remoteSearchInsult( String search ) throws IOException {
    try ( Socket socket = SocketFactory.getDefault().createSocket( HOST, PORT );
          PrintWriter requestWriter =
              new PrintWriter( socket.getOutputStream(), true,
                               StandardCharsets.UTF_8 );
          BufferedReader responseReader = new BufferedReader(
              new InputStreamReader( socket.getInputStream(),
                                     StandardCharsets.UTF_8 ) ) ) {
      requestWriter.println( search );
        responseReader.lines().collect( Collectors.joining( "\n" ) )

The main(…​) method contains an infinite loop, asks the user for a string and passes it to to remoteSearchInsult(String). The new method is responsible for communication with the server.

In the first step, the socket factory returns a socket object for localhost and the desired port. Sockets are native resources that must be returned to the operating system at the end; closing them is handled as usual by a try-with-resources block, which also closes the input/output streams.

Writing the string is again done by PrintWriter. The client uses a BufferedReader for reading, since it has the advantage of providing a stream of lines with the lines() method. The lines read in are joined with a Collector and printed. You cannot connect a Reader directly to an InputStream, so the InputStreamReader decorator is needed to enable the Reader API on an InputStream.

1.4.5. Implement a port scanner

The PortScanner class accesses its own Protocol enumeration, with TCP and UDP constants, a packet-visible object method isAvailable(int), the abstract helper method openSocket(int), and the static method serviceName(int).

enum Protocol {
  TCP {
    @Override AutoCloseable openSocket( int port ) throws IOException {
      return ServerSocketFactory.getDefault().createServerSocket( port );
  UDP {
    @Override AutoCloseable openSocket( int port ) throws IOException {
      return new DatagramSocket( port );

  abstract AutoCloseable openSocket( int port ) throws IOException;

  boolean isAvailable( int port ) {
    try ( AutoCloseable __ = openSocket( port ) ) { return true; }
    catch ( Exception e ) { return false; }

  private static final String COMPRESSED_SERVICE_NAMES = """
      7 Echo\n13 Daytime\n20 FTP\n21 FTP\n22 SSH\n23 Telnet\n25 SMTP
      53 DNS\n80 HTTP\n135 EPMAP\n137 NetBIOS Name Service
      138 NetBIOS Datagram Service\n139 NetBIOS Session Service
      445 Microsoft-DS Active Directory\n843 Adobe Flash
      1900 Simple Service Discovery Protocol (SSDP)
      3702 Web Services Dynamic Discovery\n5353 Multicast DNS
      5355 Link-Local Multicast Name Resolution (LLMNR)
      17500 Dropbox\n27017 MongoDB""";

  private static final Map<Integer, String> SERVICE_NAMES =
          .map( Scanner::new )
          .collect( Collectors.toMap( Scanner::nextInt, Scanner::nextLine ) );

  static String serviceName( int port ) {
    return Optional.ofNullable( SERVICE_NAMES.get( port ) ).orElse( "" );

The Protocol enumeration type declares an abstract method openSocket(int) for opening the connection, since for TCP and UDP the code is different; the two enumeration elements implement the abstract method accordingly. In the case of TCP, the application builds a ServerSocket via the ServerSocketFactory or just a DatagramSocket via the constructor of DatagramSocket. Although ServerSocket and DatagramSocket are different types, both implement the AutoCloseable interface, and openSocket(int) also returns this type, because only this type is relevant for isAvailable(int).

It is the task of isAvailable(int) to find out if the port is already in use. To do this, it calls openSocket(…​), and if there was no exception, the port was free, and the connection can be closed right away; this is what try-with-resources on AutoCloseable takes care of.

serviceName(int) accesses a previously built Map. Inside the class there is a constant COMPRESSED_SERVICE_NAMES, which could easily come from a file. The string contains the port number and, separated by spaces, a short description, which in turn is terminated with a newline. Text blocks save us the line break at the end of the line. A Stream expression prepares the Map by breaking the String into lines, this is where the lines() method is useful. The Stream then consists of lines, which are passed into the constructor of the Scanner object, so that afterwards nextInt() returns the key for the Map`and `nextLine() the short description associated with the port number. serviceName(int) can access this associative store SERVICE_NAMES and returns an empty string if there is no description associated with the port number. In another modeling, of course, we could return Optional.empty() directly, but at this point the empty string is handy.

The main(…​) method makes use of the methods of PortScanner:

final int MIN_SYSTEM_PORT     =     0;
//    final int MAX_SYSTEM_PORT     =  1023;
//    final int MIN_REGISTERED_PORT =  1024;
final int MAX_REGISTERED_PORT = 49151;

System.out.println( "Protocol   Port       Service" );
for ( int port = MIN_SYSTEM_PORT; port <= MAX_REGISTERED_PORT; port++ ) {
  for ( Protocol protocol : Protocol.values() )
    if ( ! protocol.isAvailable( port ) )
      System.out.printf( "%s       %5d      %s%n",
                         protocol, port,    Protocol.serviceName( port ) );

First, we declare constants for the boundaries of the port ranges that our port scanner should run. They represent the boundaries of the port ranges that our port scanner should run. In the code example we manage to use only two constants for the upper and lower limits, because after MAX_SYSTEM_PORT we continue with MIN_REGISTERED_PORT. We use a for loop to use all registered ports from 0 to 49151. Protocol.values() returns an array with the two enumeration elements TCP and UDP, and if the isAvailable(…​) method declared on the enumeration shows a blocked port, this prints the console output.

Java 8 Backport

The lines() method has been around since Java 11, and an alternative for Java 8 is the expression Pattern.compile( "\n" ).splitAsStream(string). Instead of the text blocks, the multiline string would have to be formed with \n and concatenations.