Packaging in EK9

This section will discuss the package construct in more detail.

The package construct is a flexible and gradual way to control what your programs will consist of. It is a little like the pom.xml in maven, but with a more gradual approach to creating a project model.
It does not matter if you don't know what a pom is or the npm - package.json. In short they are both mechanisms to help with packaging your software in some form or other, and also pulling in other packages you may need..

The problem we are attempting to solve

If you've only ever done small scale development or very limited applications/system utilities you're probably wondering what problem we are attempting to solve here with packages. In fact if you are still only doing small scale local development just for yourself - you don't even need to use packages.

What makes up your whole program

The first issue package aims to solve, is the way to describe what source files should be included for compilation. Clearly with EK9 it is possible to put all your source in one file and just compile that.

If you've looked at the examples, you can see you can actually get quite a long way with that approach. But in time you will probably want to break the code up in to several files. Hopefully you will use different module name spaces to give the code more structure.
This is a matter of taste, practice, experience, tooling - it's very easy to go from one extreme to another.

Which files to compile

So now we have our first issue, which files do we need to compile?

Depending on your approach to software development; you may start of with a 'grand plan' and fill in the blanks (i.e. you see the big picture and do lots of structure up front).
Alternatively you may just 'start coding' and see where it takes you! With the EK9 language you can take either approach.

Personally I prefer the latter, creating prototypes, pulling bits from here and there, refactoring, duplicating and then resolving duplications. So I tend to take a very dynamic and fluid approach to development and like to develop 'unit tests' to exercise the code written (sometimes before it is written). The EK9 package has been developed in a staged manner to support this approach.

By default the EK9 compiler will just compile the single file you gave it on the command like. Normally this will have at least one program you want to run - but not always.

But if there are multiple files that are needed. This is where the package construct starts to be useful. It enables the developer to 'list' the files they want to compile, as shown below.

Define the list of files to be compiled

Assumes the content below is stored in a file called ConstantRef1.ek9. This is a bit like a 'Makefile' used by other languages like 'C' and 'C++' for example. 

#!ek9
defines module introduction

  defines package

    description <- "Example of different constants in different modules."

    //We only want this file and these three to be compiled up.
    includeFiles <- [
      "ConstantRef2.ek9",
      "ConstantRef3.ek9",
      "ConstantRef4.ek9"
      ]
      ...
//EOF

Now for projects that require just a few source files this approach is quite suitable, it has enabled the developer to take a simple application with one source file, refactor it (over a period of time). Adding features and capabilities and then break those features and capabilities up and use a small number of related source files. This approach might work for a handful of source files, but it soon becomes a burden to remember to add in the new source code files you have created.

This approach is really aimed at small utility programs and tools that are really just a bit too big for a single source file. Maybe there are some reusable elements needed by several little utilities and some constructs need to be shared.

Just include all EK9 source files

Once your project gets to a certain size, you will probably have it all in a directory on the filesystem that is dedicated just to that project. Smaller separate applications might have all been stored together but were actually separate and shared very little. But now you've reached the point where you want a dedicated directory for all the related source code. Moreover you want subdirectories for specific code that are best stored near each other.

You now need to make a slight modification to your package construct. You will be relieved to know that you can stop listing all the source files now in the includeFiles directive. Now you can just write the following: 

#!ek9
defines module introduction
  defines package

    description <- "Example of different constants in different modules."

    //No need for includeFiles for source code anymore.
    applyStandardIncludes <- true
    ...
//EOF

This single directive above means that any and all files that have the .ek9 file suffix will now be included in the compilation build. This applies to the current directory where the source file with the package directive is and all sub directories (though not quite true) it will exclude all files in sub directories of /dev unless the type of compilation is a development build. The different types of build will be discussed later.

You may ask, why not just do this as the default. The main reason is 'different use cases'. Sometimes you just want to knock up a quick utility (maybe a single source file). Why mess about with package when a simple program and a couple of functions are all that is needed? See EK9 purpose for more details on this reasoning, EK9 is trying to make easy things 'easy' and hard things possible and 'fairly easy'.

This 'directory' based approach with special named subdirectories is more the like 'maven' or 'gradle' approach. At this point all your code only uses the standard libraries shipped with EK9 and as such has no external 'dependencies'. Those 'dependencies' and 'libraries' are the subject of the next paragraph.

What about libraries

This is the second issue developers face; in general you won't want or need to write all the code for your application yourself or even with your team. You will probably find someone else has written code that you would find useful. That code will most probably have been made available by that developer (or developers) in the EK9 repository.

Conversely, maybe you've developed a number of constructs and what to package those up as a reusable library and make that available either within your own development group/company or to everyone via opensource.

You must now update your package directive again to state what libraries (dependencies) you need included in your project.

Now if the developers of those dependencies are the sort of developers that really want you to use their packages - they will have probably provided lots of sample code in the /dev directory. Take a look in your library $HOME/.ek9/lib/{package-name}/dev. There should be some good examples of how the developer intended you to use the package they have provided.
You'd expect quite a few tests and examples to be located in the dev/ directory with lots of documentation in the source code. These examples and snippets should really help you get started with how you use that library.

Defining dependencies
#!ek9
defines module introduction

  defines package

    description <- "Example of different constants in different modules."

    //No need for includeFiles for source code anymore.
    applyStandardIncludes <- true

    deps <- {
      "ekopen.network.support.utils": "1.6.1-9",
      "ekopen.net.handy.tools": "3.2.1-0"
      }
      ...
//EOF

In the above example we have declared that our project has two dependencies. Also note that we have declared the specific version of the dependency. EK9 will also bring in any transitive dependencies. EK9 is very strict on both dependency naming and version numbering.
There is more in the dependency naming and the version numbering in later paragraphs.

Dependencies and Libraries

If you want to use or publish packages of constructs then the issue of version numbering and dependency naming has to be addressed.

Dependency Naming and Version Numbers

There are two main spaces for deploying modules in the EK9 repository.

Wild West

The naming of public modules in the 'ekopen' space is a 'free for all', first come first served, no corporates that can't mix it with freelance devs. No legal cases, no complaints, whoever gets there first gets the name space. But they must have a depth of at least four parts i.e ekopen.network.support.utils. The developer that issues the first package to ekopen.network.support then controls all the final module names under that space. They can publish and deploy as many modules under that space as they like and only they can do this.

Civilisation

This is the naming of public modules where there is a link to a controlled DNS domain. This is controlled so that individuals or corporate entities can publish packages with their associated name. For example only the entity that controls the ek9lang.org domain name can publish a package like org.ek9lang.core.functions. The mechanism of domain validation is a standard one. The developer wishing to publish to a controlled space like this has to create a suitable TXT DNS entry. The EK9 repository will then check that and that developer will be able publish to that name space.

In Summary

If you are or represent a corporate entity stay in Civilisation, do not stray into the Wild West. In Civilisation you will be able to retain control of your corporate brand naming and main package names and module names. If you are a developer in the Wild West and like the free for all - then don't complain about naming clashes - that's what it is like. If you don't like it; register a DNS domain with some provider and use Civilisation.

Version Numbering

Version number are the third issue developers face; EK9 uses a form of strict Semantic Versioning as follows:

There are no other supported variations. See features for more information on why you might want a feature name in your version number.

The Public Interface

All constructs defined in the packaged module name are the public interface. So as an example: Using ekopen.net.handy.tools in the example above, if class CRC and function reverse were defined in module ekopen.net.handy.tools; then they would be the public interface.
If you wished to retain the same major version number - it is not possible to alter these constructs or their parameters in any way. Moreover no other non public interface constructs can be used as parameters into or out of that public interface. Any transitive dependencies that the module pulled in must also be major version compatible.

This may appear overly strict or draconian, but it is necessary to ensure dependency consistency. A developer needs to be assured that version 4.5.6-21 can be used in place of 4.4.1-9 and that not only will the public code interface be the same but also the transitive dependencies will also remain major version compatible.

So in summary:

Dependency Exclusions

There will come a time when a transitive dependency (the fourth issue developers face) is pulled in and it has known defects; you need to avoid that specific buggy version. You can use the directive below to be explicit about excluding a dependency when it has been pulled in. 

#!ek9
defines module introduction

  defines package

    description <- "Example of different constants in different modules."

    //No need for includeFiles for source code anymore.
    applyStandardIncludes <- true

    deps <- {
      "ekopen.network.support.utils": "1.6.1-9",
      "ekopen.net.handy.tools": "3.2.1-0"
      }

    excludeDeps <- {
      "ekopen.some.bad.dependency.pack": "ekopen.org.supertools.util"
      }
      ...
//EOF

As you can see above the directive is very simple, it is just a dictionary (map) of dependency names. The above statement means: "exclude dependency 'ekopen.some.bad.dependency.pack' when it has been pulled in as a transitive dependency from 'ekopen.org.supertools.util'". It does not have to be a direct transitive dependency it could be a dependency of a dependency etc.

Version Numbers for the module and program you are developing

In general, if/when the software you are developing has become more than just a bit of tooling for yourself; and other users are now using it you will probably want to give your software a version number.

This becomes critical for managing releases/defect/improvements - your customers or people that just use your software will want to know when's the next release. Will version x.y.z have this feature or that feature.
The version directive is now added to the package declaration, but you should probably align this with some sort of source code repository like git or something like that.

Now add in your version number, as shown below we've decided this version is 2.3.14-0. Now each time you build, patch, make minor improvements or major improvements you must alter the release number.
But don't worry the EK9 compiler can be used to automate that - so there is no need to do it by hand. 

#!ek9
defines module introduction

  defines package

    version <- 2.3.14-0

    description <- "Example of different constants in different modules."

    //No need for includeFiles for source code anymore.
    applyStandardIncludes <- true

    deps <- {
      "ekopen.network.support.utils": "1.6.1-9",
      "ekopen.net.handy.tools": "3.2.1-0"
      }

    excludeDeps <- {
      "ekopen.some.bad.dependency.pack": "ekopen.org.supertools.util"
      }
      ...
//EOF

Now maybe you're thinking it's going to be a pain altering that version number all the time. EK9 has a mechanism to deal with that for you. Once you've put a version number in the EK9 compiler can alter that for you as part of a build pipeline.
In general when you get to the point of supplying your software to others, you need a reliable/repeatable and controlled build mechanism (rather than just building it on your local PC!).

By using something like GitHub Actions, AWS, Azure or Hudson/Jenkins you can define phases of a build - and this can include alterations to the version being built. See the command line for details on the command line options for updating the version number.

Publishing an open source module

You may feel that the code you've developed, either just a range of functions, classes or other constructs could be useful to other people. You may already have a git repository with the code on and may other developers have contributed to that. Now you want to make specific versions available via the EK9 repository.

As mentioned earlier - decide Wild West or Civilisation and name your module accordingly. But now we need to add a couple more directives (as well as naming the module to be unique). 

#!ek9
defines module ekopen.math.simple.constants

  defines package

    publicAccess <- true

    version <- 2.3.14-0

    description <- "Example of different constants in different modules."

    tags <- [
      "mathematics",
      "constants",
      ]

    license <- "MIT"

    applyStandardIncludes <- true

    includeFiles <- [
      "License.txt",
      "Copyright.txt",
      "Authors.txt"
      ]

    //certain files like .gitignore we don't want to package
    applyStandardExcludes <- true

    deps <- {
      "ekopen.network.support.utils": "1.6.1-9",
      "ekopen.net.handy.tools": "3.2.1-0"
      }

    excludeDeps<- {
      "ekopen.some.bad.dependency.pack": "ekopen.org.supertools.util"
      }
      ...
//EOF

By adding in the tags the packaged module will be searchable (the fifth issue developers face - how to easily find useful stuff) and by adding publicAccess it has been made public to the open source community. We've also stated what license the software is. By using the directive applyStandardExcludes we can exclude specific files. You will probably have noticed the the includeFiles has been reintroduced; this in conjunction with applyStandardIncludes will ensure that all ek9 source code in addition to those three files all get packaged.

By changing the name of the module to ekopen.math.simple.constants (assuming ekopen.math.simple is still available) the software can now be packaged and uploaded to the EK9 repository. As the developer(s) that now own ekopen.math.simple you can publish any number of packages to under that structure. No other developers can publish with that module naming.

It's the same process if you use and control a domain name, only your development team can publish to that module name.

Importantly once a version of a module is published - that's it; it cannot be retracted/deleted or altered (other than publishing a new version). Others may now have referenced it and if you could retract the software you would break their builds.

How to publish

Publishing to a EK9 public repository is not instant - the content being published must be scanned for viruses and also ensure it means the minimum standards (i.e. size, minimal/acceptable profanities). Note that when they are published the EK9 server will also compile them to check they will actually compile.

There are a number of stages and steps the 'repository' will take to ensure that the code provided is safe and meets and number of basic standards.

Summary

Hopefully you will have seen how you can easily progress from small programs in single EK9 source files, through to medium, large and very large suites of software. These start of simply; but can have dependencies added and can themselves be wrapped up and published. Each aspect of this just requires the progression of augmenting the package construct with a little more detail as your applications/projects grow for small simple programs through to large modules/applications.

Final Steps!

The final section is on the command line compiler tool.